Missing "do_lower_case" action for special token (e.g. mask_token)
Author: makcedward
Pull request: no
Labels: name: wontfix

## 🐛 Bug

Model I am using (Bert, XLNet....): 'bert-base-uncased'

Language I am using the model on (English, Chinese....): English

After upgrading to 2.2.1 version, the BERT tokenizer cannot tokenize special word while it works in 2.1.1 version.

According to [here](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_bert.py#L71), 'bert-base-uncased' should perform lower case operation. Inputs follow this config to [perform lower case operation](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_utils.py#L615), while no corresponding action for [special tokens](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_utils.py#L1100). Eventually, it tokenizes '[MASK]' to 3 subwords (e.g. [, mask and ]) rather than skip the tokenization operation in [here](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_utils.py#L658).

Error occurs after this [commit](https://github.com/huggingface/transformers/commit/7246d3c2f93c4461f3ec8ada7a26a002d8f196ea).





## To Reproduce

Steps to reproduce the behavior:
```
import torch
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
tokenizer.tokenize('The quick brown [MASK] jumps over the lazy dog.')
```

## Expected behavior

Expected output is 
['the', 'quick', 'brown', **'[MASK]'**, 'jumps', 'over', 'the', 'lazy', 'dog', '.']
while actual output is 
['the', 'quick', 'brown', **'[', 'mask', ']'**, 'jumps', 'over', 'the', 'lazy', 'dog', '.']


Comments:
With Transformers **2.2.0**, it works as expected!
```
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> from transformers import BertTokenizer
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/vidiemme/anaconda3/envs/huggingface/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
2019-12-05 10:19:00.776555: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-05 10:19:00.799189: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3408000000 Hz
2019-12-05 10:19:00.799911: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55971fe06640 executing computations on platform Host. Devices:
2019-12-05 10:19:00.799929: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
>>> tokenizer.tokenize('The quick brown [MASK] jumps over the lazy dog.')
['the', 'quick', 'brown', '[MASK]', 'jumps', 'over', 'the', 'lazy', 'dog', '.']
```
With Transformers **2.2.1**, the bug you've highlighted occurs to me too!

> ##  Bug
> Model I am using (Bert, XLNet....): 'bert-base-uncased'
> 
> Language I am using the model on (English, Chinese....): English
> 
> After upgrading to 2.2.1 version, the BERT tokenizer cannot tokenize special word while it works in 2.1.1 version.
> 
> According to [here](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_bert.py#L71), 'bert-base-uncased' should perform lower case operation. Inputs follow this config to [perform lower case operation](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_utils.py#L615), while no corresponding action for [special tokens](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_utils.py#L1100). Eventually, it tokenizes '[MASK]' to 3 subwords (e.g. [, mask and ]) rather than skip the tokenization operation in [here](https://github.com/huggingface/transformers/blob/master/transformers/tokenization_utils.py#L658).
> 
> Error occurs after this [commit](https://github.com/huggingface/transformers/commit/7246d3c2f93c4461f3ec8ada7a26a002d8f196ea).
> 
> ## To Reproduce
> Steps to reproduce the behavior:
> 
> ```
> import torch
> from transformers import BertTokenizer
> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
> tokenizer.tokenize('The quick brown [MASK] jumps over the lazy dog.')
> ```
> 
> ## Expected behavior
> Expected output is
> ['the', 'quick', 'brown', **'[mask]'**, 'jumps', 'over', 'the', 'lazy', 'dog', '.']
> while actual output is
> ['the', 'quick', 'brown', **'[', 'mask', ']'**, 'jumps', 'over', 'the', 'lazy', 'dog', '.']

I've tracked it yesterday evening and I confirm all of that too. `PreTrainedTokenizer.add_tokens` forces added tokens to lower-case but tokens coming from BertTokenizer constructor aren't lower-cased.

Yet considering a viable patch, I tend to think there is an issue linked to current design of Tokenizers with respect to flags such as `do_lower_case`.

For example, current BertTokenizer is:

```python
class BertTokenizer(PreTrainedTokenizer):
...

    def __init__(self, vocab_file, do_lower_case=True, do_basic_tokenize=True, never_split=None,
                 unk_token="[UNK]", sep_token="[SEP]", pad_token="[PAD]", cls_token="[CLS]",
                 mask_token="[MASK]", tokenize_chinese_chars=True, **kwargs):

        super(BertTokenizer, self).__init__(unk_token=unk_token, sep_token=sep_token,
                                            pad_token=pad_token, cls_token=cls_token,
                                            mask_token=mask_token, **kwargs)
```

So `BertTokenizer` knows about `do_lower_case` but not the super class `PreTrainedTokenizer`. Moreover, by default `do_lower_case` is True but all tokens are defined in upper_case.

Then, in `PreTrainedTokenizer`, there are some `if self.init_kwargs.get('do_lower_case', False):` in different places of the code to force text or added_tokens to lower_case before tokenization. But this means you inject a knowledge of `lower_case` in a class that doesn't know it by construction. It works but as we see in the case of token case, it's error-prone and not so robust. Moreover, if there were several flags, it would become even harder to track.

A solution could be to provide a simple callback system in `PreTrainedTokenizer` with callbacks `prepare_tokens` and `prepare_text` provided by the implementing Tokenizer class which takes into account its own flags.

Yet it requires a bigger modification of code and a bit more reflection (I can propose a PR on this if we agree on something).
For now, an immediate solution to current issue would be to force BertTokenizer to lower_case its tokens by construction: 
```python
class BertTokenizer(PreTrainedTokenizer):
...

    def __init__(self, vocab_file, do_lower_case=True, do_basic_tokenize=True, never_split=None,
                 unk_token="[UNK]", sep_token="[SEP]", pad_token="[PAD]", cls_token="[CLS]",
                 mask_token="[MASK]", tokenize_chinese_chars=True, **kwargs):

        if do_lower_case:
            unk_token, sep_token, pad_token, cls_token, mask_token = unk_token.lower(), sep_token.lower(), pad_token.lower(), cls_token.lower(), mask_token.lower()

        super(BertTokenizer, self).__init__(unk_token=unk_token, sep_token=sep_token,
                                            pad_token=pad_token, cls_token=cls_token,
                                            mask_token=mask_token, **kwargs)
```

WDYT?

Should have been fixed with #2051

I confirm it should solve the issue! It introduces a bit more external logic about `do_lower_case` in `PreTrainedTokenizer` as I explained. It's not critical but keep in mind for the future, there are solutions to improve that in the code ;)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.