Adding new tokens to RobertaTokenizer gives very strange results - probably a bug.
Author: Oxi84
Pull request: no

 I am trying to add new tokens to Roberta tokenizer, but the results are rather strange.  

    from transformers import RobertaTokenizerFast
    t = RobertaTokenizerFast.from_pretrained('roberta-base')
    print(t.tokenize("Apples are too_big for turtles."))
    t.add_tokens(["too_big","turtles"])
    print(t.tokenize("Apples are too_big for turtles."))

The result before adding the tokens are:

    ['App', 'les', 'Ġare', 'Ġtoo', '_', 'big', 'Ġfor', 'Ġturtles', '.']

and this is ok, but after I add new tokes the results are:

     ['App', 'les', 'Ġare', 'Ġ', 'too_big', 'Ġfor', 'Ġ', 'turtles', '.']

and they should be:

    ['App', 'les', 'Ġare', ''Ġtoo_big', 'Ġfor', 'Ġturtles', '.']

Is this a bug? Normally i doubt the tokenizer ever has Ġ alone without anything else as a token.





Comments:
For additional context, the non-fast `RobertaTokenizer` returns the following results.

```python
>>> print(tokenizer.tokenize("Apples are too_big for turtles."))
['App', 'les', 'Ġare', 'Ġtoo', '_', 'big', 'Ġfor', 'Ġturtles', '.']
>>> tokenizer.add_tokens(["too_big","turtles"])
>>> print(tokenizer.tokenize("Apples are too_big for turtles."))
['App', 'les', 'Ġare', 'too_big', 'for', 'turtles', '.']
```

cc @SaulLu

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @Oxi84! 

Sorry for the late reply. As far as added tokens are concerned, you can set attributes when using a fast tokenizer (for the moment it won't work with a slow one).

https://github.com/huggingface/transformers/blob/31616b8d613dcb7ac69b562d51b42d0db379f72f/src/transformers/tokenization_utils_base.py#L77-L91

So if the default behaviour doesn't suit you, you can for example change it to get the same behavior as the slow tokenizer:
```python
from transformers import RobertaTokenizerFast, AddedToken
t = RobertaTokenizerFast.from_pretrained('roberta-base')

t.add_tokens([AddedToken("too_big", lstrip=True, rstrip=True), AddedToken("turtles", lstrip=True, rstrip=True)])
print(t.convert_ids_to_tokens(t.encode("Apples are too_big for turtles.")))
# ['
', 'App', 'les', 'Ġare', 'too_big', 'for', 'turtles', '.', '
']
```

Or if you want to be close to your initial proposal, you'll have to add a space at the beginning of the added tokens:
```python
from transformers import RobertaTokenizerFast, AddedToken
t = RobertaTokenizerFast.from_pretrained('roberta-base')

t.add_tokens([" too_big", " turtles", "too_big","turtles"])
print(t.convert_ids_to_tokens(t.encode("Apples are too_big for turtles.")))
# ['
', 'App', 'les', 'Ġare', ' too_big', 'Ġfor', ' turtles', '.', '
']
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.