How to bypass "Special tokens have been added in the vocabulary..." warning?
Author: stas00
Pull request: no

Is there a way to avoid always getting:

> Special tokens have been added in the vocabulary, make sure the associated word embedding are fine-tuned or trained

@ https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils_base.py#L1614

(other than turning logging off)

As far as I can see from stepping through that code, there are always special tokens (e.g. eos, pad, etc.), i.e. there is nothing special about it. What purpose does this warning serve when loading a tokenizer? 

I'm not sure how the end user can act on the suggestion:

> make sure the associated word embedding are fine-tuned or trained

when they just want to run, say, the `generate` function on a pre-trained model, other than just learning to ignore this warning and not paying heed to when a warning is really saying something crucial.

Thoughts?


Comments:
Hello @stas00 ,

Can you show us a sample of your code ?

Did you explicitely add special tokens to your tokenizer ?

From my understanding this warning appears when the method _sanitize_special_tokens_ of the tokenizer returns a strictly positive integer.

The docstring of the method is:
```
"""
        Make sure that all the special tokens attributes of the tokenizer (:obj:`tokenizer.mask_token`,
        :obj:`tokenizer.cls_token`, etc.) are in the vocabulary.
        Add the missing ones to the vocabulary if needed.
        Return:
            :obj:`int`: The number of tokens added in the vocaulary during the operation.
"""
```

So this warning appears when you add special tokens to the vocabulary **after** loading the tokenizer. If you use a model trained on the first version of the tokenizer (before adding the new tokens), you might feed it tokens it has not been trained on, which would lead to a random embedding and worse performance.

If you load your model and your tokenizer with the same training, for example:
```
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
```
then I suggest not providing special tokens, as the basic ones are already present.

Please let me know if it helps.

You're absolutely correct, the tokenizer was adding special tokens (copied from another tokenizer, but wasn't really needing them), so I removed them now and the warning is gone.

And, yes, I forgot to add the code - wasn't my best!

Much appreciating your follow up, @nassim-yagoub 

The same warning also happens with this code:
```
import torch
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('vinai/bertweet-base')
model = AutoModel.from_pretrained('vinai/bertweet-base')
```
```
python --version
Python 3.9.13
```
I do not understand where tokens are added to the vocabulary after loading the tokenizer.

@sbocconi  Any idea how to solve the warnings in the case of BERTTweet?


No unfortunately @codepujan, it has been a while I have not used this functionality

This worked for me:

transformers.logging.set_verbosity_error()

@rbehzadan's comment worked!


+1 I always get this warning and cannot understand what I am supposed to do.

cc @ArthurZucker Maybe we can downgrade this warning level? 

yeah I don't think it help anyone.