added_tokens.json is used for splitting texts
Author: chuanli11
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

name: Should Fix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Run this script to save a pre-trained vocab, add some vocab using `added_tokens.json`. Then create a new tokenizer with the combined vocabs and use it to tokenize a sentence.

```
import os

from transformers import BertTokenizer

model_name = 'bert-base-uncased'
tokenizer_path = 'tmp'

if not os.path.exists(tokenizer_path):
    os.makedirs(tokenizer_path)

tokenizer = BertTokenizer.from_pretrained(model_name)

tokenizer.save_vocabulary(tokenizer_path)

with open(tokenizer_path + '/added_tokens.json', 'w') as f:
    f.write('{"ver": 30522, "rw": 30523}')

tokenizer = BertTokenizer.from_pretrained(tokenizer_path)

s = "i want to overwrite ubuntu with windows"

a = tokenizer.tokenize(s)
print(a)
```
Output run 1:
```
['i', 'want', 'to', 'o', '##ve', 'rw', 'rite', 'u', '##bu', '##nt', '##u', 'with', 'windows']
```

Ouptut run 2:
```
['i', 'want', 'to', 'o', 'ver', 'write', 'u', '##bu', '##nt', '##u', 'with', 'windows']
```
Cause of the problem:

`added_tokens.json` is [merged](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils.py#L668) with `all_special_tokens`, and then used to [split](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils.py#L815) the input text. Since the merged tokens is stored in an unordered set, the splitting process is non-deterministic for each run. 

For example, these are how the text is split in different runs:

Split Run 1 (use `rw`):
```
['i want to ove', 'rw', 'rite ubuntu with windows']
```

Split Run 2 (use `ver`):
```
['i want to o', 'ver', 'write ubuntu with windows']
```
Possible solution:
Instead of `self.unique_added_tokens_encoder`, use `set(self.all_special_tokens)` to [split the text](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils.py#L814)

## Expected behavior
Using `added_tokens.json` to split the text seems to be a bug to me. I expect only a small set of special tokens in the `special_tokens_map.json` should be used for this purpose.

In general it is helpful to let a tokenizer behave deterministically across multiple runs. Otherwise it will be bad for certain downstream task such as [sentence embedding ](https://github.com/UKPLab/sentence-transformers) because one sentence can be encoded in many different ways. This is in particular problematic if the number of added_vocab is big. 


## Environment info

wget https://download.pytorch.org/whl/cu100/torch-1.4.0%2Bcu100-cp36-cp36m-linux_x86_64.whl
wget https://files.pythonhosted.org/packages/7e/90/6141bf41f5655c78e24f40f710fdd4f8a8aff6c8b7c6f0328240f649bdbe/torchvision-0.5.0-cp36-cp36m-manylinux1_x86_64.whl

virtualenv -p /usr/bin/python3.6 venv && . venv/bin/activate && find . -maxdepth 1 -name "*.whl" | xargs pip install && pip install -r requirements.txt

requirements.txt:
transformers==2.5.1
tensorboardX==2.0
scikit-learn==0.22.2
     
- `transformers` version: 2.5.1
- Platform: Ubuntu
- Python version: 3.6.9
- PyTorch version (GPU?): Y
- Tensorflow version (GPU?): N
- Using GPU in script?: Y
- Using distributed or parallel set-up in script?: N


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.