Tokenizer is working different from expected functionality. 
Author: pyturn
Pull request: no

Hi, 

I updated the vocabulary for pre-trained tokenizer. Pretrained tokenizer was taken from this model - https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract

When I am using the updated tokenizer , it is creating the sub tokens of the word which are present in vocabulary dictionary. And those sub-tokens are also not starting with '##' , which is creating confusion. 

`input_ids = TOKENIZER.encode('vancomycin')`
`TOKENIZER.decode(input_ids)`
`TOKENIZER.decode([16100]) #checked through manual rules that same word is present in vocabulary.`  

**Sanpshot of the Code**


![image](https://user-images.githubusercontent.com/25935364/108026165-d566aa00-704d-11eb-9a89-27dfb0d786c6.png)






## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform: Windows
- Python version:  3.8.5
- PyTorch version (GPU?): 1.7.1 , Yes,  Cuda 10.1
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: 



Comments:
Hi! Could you provide a reproducible example? I don't understand what's your `TOKENIZER` here. Thanks

Hi @LysandreJik , 

Thanks for your response , I have done deep-dive and modified code a bit to reproduce the same issue. I am extending the vocabulary of tokenizer and using some automated logic to add the new words. I don't not have complete control on the words which are adding (but, I am making sure those are not the noise.)

```python
from transformers import AutoTokenizer, AutoModelForMaskedLM
model_checkpoint_tok = "microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext"
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint_tok)
enc_ids = tokenizer.encode('vancomycin')
tokenizer.decode(enc_ids)
```
```out
[CLS] vancomycin [SEP]
```

Adding the new vocabulary through some automated logic, I added thousands of words, To reproduce the same issue, I am manually adding one of the word in next line

```python
tokenizer.add_tokens(['vanco'])
enc_ids = tokenizer.encode('vancomycin')
tokenizer.decode(enc_ids)
```
```out
`Output : '[CLS] vanco mycin [SEP]'
```

The word 'vancomycin' is present in vocabulary, The word 'vanco' got added in vocabulary (by some automated logic), Now when I am again tokenizing "vancomycin" , It is splitting it into token id's of 'vanco' and 'mycin' (Mycin is not exactly present, So it got splitted into subwords). 

My question is - 

- Is it the expected functionality of tokenizer ( I know that its a Subword tokenization technique) , I am not sure but what I guess is there should be some word boundary detection technique and if the exact word in word boundary is not present, then only sub word tokenization should happen?   


 
Please suggest how I can handle these scenarios ?

Please Find Below Screenshot for the same - 

![image](https://user-images.githubusercontent.com/25935364/108313945-1f32ca00-71df-11eb-9f73-5640faefe5eb.png)


Hi @LysandreJik , Are there any updates on this ? 

Hi! That isn't a bug of itself, but expected behavior (which could be better documented). The tokens added to the tokenizers don't get added to the "vocabulary" that resulted from the tokenizer training.

Instead, they get added to the "added tokens", and these tokens take priority over the vocabulary tokens. It is, unfortunately, complex or near impossible to add tokens to the vocabulary itself in a way that all tokenizers could benefit. This is particularly complex in the case of subwords tokenizers, which is the case of the BERT tokenizer here.

You can check this thread https://github.com/huggingface/tokenizers/issues/370 for a similar issue.

Hi @LysandreJik , So, if we add the tokens and then continually pretrain the Bert Model (By Fine Tuning on Masked Language Modeling) on specific corpus , Then , Does the model learn embeddings for these added tokens ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.