Special Tokens are Split by BPE
Author: wcollins-ebsco
Pull request: no

## 🐛 Bug

When I load 'distilbert-base-uncased' DistilBertTokenizer (with do_basic_tokenize=False) and call tokenize() on a string that includes special tokens, the special tokens are broken up by BPE.

Model I am using (Bert, XLNet....): DistilBertForSequenceClassification

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Load pretrained DistilBertTokenizer
2. call tokenize on a string including special tokens

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased', do_lower_case=True, do_basic_tokenize=False)
print(tokenizer.special_tokens_map)
```
{'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}

```
text = '%s Hrabri (Brave) was the lead boat of the Hrabri-class submarines; built by the Vickers-Armstrong Naval Yard in the United Kingdom, for the Kingdom of Serbs, Croats and Slovenes (later Yugoslavia) %s' % (tokenizer.cls_token, tokenizer.sep_token)
print(text)
```
[CLS] Hrabri (Brave) was the lead boat of the Hrabri-class submarines; built by the Vickers-Armstrong Naval Yard in the United Kingdom, for the Kingdom of Serbs, Croats and Slovenes (later Yugoslavia) [SEP]

```
tokens = tokenizer.tokenize(text)
print(' '.join(tokens))
```
[ cl ##s ] hr ##ab ##ri ( brave ) was the lead boat of the hr ##ab ##ri - class submarines ; built by the vickers - armstrong naval yard in the united kingdom , for the kingdom of serbs , croats and slovene ##s ( later yugoslavia ) [ sep ]

## Expected behavior
```
tokens = tokenizer.tokenize(text)
print(' '.join(tokens))
```
[CLS] hr ##ab ##ri ( brave ) was the lead boat of the hr ##ab ##ri - class submarines ; built by the vickers - armstrong naval yard in the united kingdom , for the kingdom of serbs , croats and slovene ##s ( later yugoslavia ) [SEP]

## Environment

* OS: Windows
* Python version: 3.6.3
* PyTorch version: 0.4.1
* PyTorch Transformers version (or branch): 2.2.1
* Using GPU ? no
* Distributed of parallel setup ? no
* Any other relevant information:

## Additional context



Comments:
Hello! Indeed this is a known issue with version 2.2.1. You can either revert to 2.2.0 or install from source (`pip install git+https://github.com/huggingface/transformers`) until we push a new version (2.2.2) which should happen before the end of the week.

I confirm that reverting to 2.2.0 solves the problem

Bumping to 2.2.2 (released today) should solve the problem too!

I can confirm that 2.2.2 fixes the issue. This question can be closed.

Thanks! This is a great library!

It really is! Can you close this question? That way the overview of open issues is a lot clearer. Thanks.

Hi, I still get the same issue with version `2.5.1` (installed from source).
The `<MASK>` token seems to be split into it's individual characters when an input string is encoded. 

I trained the `roberta` model from scratch on my own dataset as described in https://huggingface.co/blog/how-to-train .
I ran the following lines to test my trained model on the masked token prediction task.
```
config = RobertaConfig.from_json_file("drive/My Drive/doerrberto-small-v1/config.json")
model = RobertaForMaskedLM(config)
state_dict = torch.load("drive/My Drive/doerrberto-small-v1/pytorch_model.bin")
model.load_state_dict(state_dict)
tokenizer = RobertaTokenizer("drive/My Drive/doerrberto-small-v1/vocab.json", "drive/My Drive/doerrberto-small-v1/merges.txt")

fill_mask = pipeline(
    "fill-mask",
    model=model,
    tokenizer=tokenizer
    )

result = fill_mask(sentence)
```

This was when I encountered the `ValueError: only one element tensors can be converted to Python scalars` error. I then confirmed that this error was generated due to incorrect encoding of `<MASK>` token.

Any help will be appreciated. Thanks!

@aksub99 `RobertaTokenizer`'s mask token is actually `<mask>` not `<MASK>`

You can also just use `tokenizer.mask_token`

@julien-c Thanks for pointing that out, but I had used `tokenizer.mask_token` while testing. Sorry for the typo in my previous comment.
That still gave me the same errors.
This is my complete testing code snippet and it's output.
Code:
```
import torch
from transformers import RobertaConfig, RobertaForMaskedLM, pipeline, RobertaTokenizer

config = RobertaConfig.from_json_file("drive/My Drive/doerrberto-small-v1/config.json")
model = RobertaForMaskedLM(config)
state_dict = torch.load("drive/My Drive/doerrberto-small-v1/pytorch_model.bin")
model.load_state_dict(state_dict)
tokenizer = RobertaTokenizer("drive/My Drive/doerrberto-small-v1/vocab.json", "drive/My Drive/doerrberto-small-v1/merges.txt")

fill_mask = pipeline(
    "fill-mask",
    model=model,
    tokenizer=tokenizer
    )


sentence = "I {} you".format(tokenizer.mask_token)
print(sentence)

token_ids = tokenizer.encode(sentence, return_tensors='pt')
print(token_ids.squeeze())

print(tokenizer.mask_token_id)

```
Output:
```
I <mask> you
tensor([   0,  387,  225,   32,   81, 3229,   34,  377,    2])
4
```
Clearly, the `<mask>` is being split into it's individual characters.

Does the `doerrberto-small-v1` vocabulary file contain a mask token? Can you do `tokenizer.encode(tokenizer.mask_token)`, and does it return the `tokenizer.mask_token_id` in-between model-specific tokens?

@LysandreJik Yes, the `doerrberto-small-v1` vocabulary file does contain a mask token and is associated with an ID of 4. 
`tokenizer.encode(tokenizer.mask_token)` gives out `[0, 225, 32, 81, 3229, 34, 2]` which means that the mask token is again being split up.
Sorry, could you explain what you mean by "in-between model-specific tokens"? 

Seems broken in GPT2 ?


```python
tok = transformers.AutoTokenizer.from_pretrained("gpt2")
tok.cls_token = "
"
sample = "My name is Barrack Obama
I like pizza"
print(tok.tokenize(sample))
```
>`['My', 'Ġname', 'Ġis', 'ĠBarr', 'ack', 'ĠObama', '<', '|', 'cl', 's', '|', '>', 'I', 'Ġlike', 'Ġpizza']`


It does show up in the special tokens:
```python
>>> print(tok)
PreTrainedTokenizerFast(name_or_path='gpt2', vocab_size=50257, model_max_len=1024, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '
', 'eos_token': '
', 'unk_token': '
', 'cls_token': '
'})
```

@LysandreJik maybe

Using `
` does work however. It's just when you add new special tokens that the tokenizer doesn't use them.

```python
>>> tok.tokenize("An attempt with eot
Will it work")
['An', 'Ġattempt', 'Ġwith', 'Ġe', 'ot', '
', 'Will', 'Ġit', 'Ġwork']
```