BartTokenizer add_tokens feature.
Author: charitharaghavaraju
Pull request: no

Hi @LysandreJik ,

I am working on audio captioning, and the ground truth captions are tokenized using the BartTokenizer. I have observed that some of the words in the captions are not tokenized correctly. For instance, the word 'rumbling'. There is no such word in the tokenizer, and its tokenizing is ['Ġr', 'umbling']. I have tried to add the tokens(the word 'Ġrumbling')and change the model token embeddings. But instead of tokenizing the word correctly, it is still tokenizing as ['Ġr', 'umbling']. Did I miss anything here? I have also faced the same issues with the some other words too!

Here is my code!

```
tokenizer = AutoTokenizer.from_pretrained("facebook/bart-base", use_fast=True)
tokenizer.add_tokens(['Ġrumbling'])
model = BartForConditionalGeneration.from_pretrained("facebook/bart-base")
model.resize_token_embeddings(len(tokenizer))
print(tokenizer.is_fast)
ou_e = 'the rain falls down while someone is pounding a car passes by and the thunder is rumbling'
tok_e = tokenizer(ou_e, max_length=64, return_tensors='pt', padding='max_length')
seq = tokenizer.tokenize(ou_e)
print(seq)
summary_ids = model.generate(tok_e['input_ids'], num_beams=4, min_length=5, max_length=100)
summary = tokenizer.batch_decode(summary_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
print(summary)
```

Comments:
Thanks for opening an issue @Charithavarma!

@ydshieh, could you take a look here?

Could confirm the issue. Also occur for slow bart tokenizer. I can see the word is added to the tokenizers, but the output don't change.

```python
from transformers import AutoTokenizer

tokenizer_slow = AutoTokenizer.from_pretrained("facebook/bart-base", use_fast=False)
tokenizer_fast = AutoTokenizer.from_pretrained("facebook/bart-base", use_fast=True)

print(len(tokenizer_slow.get_vocab()))
print(len(tokenizer_fast.vocab))

print('Ġrumbling' in tokenizer_slow.get_vocab())
print('Ġrumbling' in tokenizer_fast.vocab)

tokenizer_slow.add_tokens(['Ġrumbling'], special_tokens=True)
tokenizer_fast.add_tokens(['Ġrumbling'], special_tokens=True)

print(len(tokenizer_slow.get_vocab()))
print(len(tokenizer_fast.vocab))

print('Ġrumbling' in tokenizer_slow.get_vocab())
print('Ġrumbling' in tokenizer_fast.vocab)


text = 'the rain falls down while someone is pounding a car passes by and the thunder is rumbling'
seq = tokenizer_slow.tokenize(text)
print(seq)
```
gives
```bash
50265
50265
False
False
50266
50266
True
True
['the', 'Ġrain', 'Ġfalls', 'Ġdown', 'Ġwhile', 'Ġsomeone', 'Ġis', 'Ġpounding', 'Ġa', 'Ġcar', 'Ġpasses', 'Ġby', 'Ġand', 'Ġthe', 'Ġthunder', 'Ġis', 'Ġr', 'umbling']
```

Hi @Charithavarma,

In transformers, an added token will be a token that will be preserved before the tokenizer is applied. In this case, since in the initial sentence the spaces are `" "` and not `"Ġ"`, the `'Ġrumbling'` token is not identified anywhere.

To achieve what you want to do I advise you to try to add the token `" rumbling"`.

Let me know if it solves your issue! :hugs: 

Thank you, @SaulLu ! Is this documented somewhere (I believe so) 🙏 .

By searching a little I realize that the current documentation is not very explicit on this point. I propose to detail it a little in the PR https://github.com/huggingface/transformers/pull/18687 :relaxed: 

Hi @SaulLu,

Thank you. It solved my problem. But the performance of the BART in my model was reduced!

Is it possible to use the manual tokenizer instead of this BART Tokenizer? Is it compatible?


@Charithavarma If you want to use the trained model `facebook/bart-base`, it's always good to use the corresponding tokenizer. If you change the tokenizer (for example, here you add a new token, where the tokenization of sentences may change too - for some examples), it is normal that the model performance is affected (as it never sees the word/token `rumbling` before).

If adding new tokens is really important in your task, you probably would consider finetuning the original model with this changed tokenizer.