tokenizer return_special_tokens does not work correctly with custom special tokens
Author: armancohan
Pull request: no
Labels: name: bug

The tokenizer's returned `special_tokens_mask` does not take into account the newly added `special_tokens`.

### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-5.4.0-92-generic-x86_64-with-glibc2.31
- Python version: 3.9.12
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.11.0 (False)
```

### Who can help?

@LysandreJik 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import transformers
print(transformers.__version__)
tokenizer = transformers.AutoTokenizer.from_pretrained('roberta-base')

special_tokens_dict = {"additional_special_tokens": ["<test1>", "<test2>"]}
tokenizer.add_special_tokens(special_tokens_dict)

processed = tokenizer("this <test1> that <test2> this", return_special_tokens_mask=True)
tokens = tokenizer.convert_ids_to_tokens(processed.input_ids)

for i in range(len(processed.input_ids)):
    print(f"{processed.input_ids[i]}\t{tokens[i]}\t{processed.special_tokens_mask[i]}")
```

### Expected behavior

```shell
Returned output:

0       
     1
9226    this    0
1437    Ġ       0
50265   <test1> 0
14      Ġthat   0
1437    Ġ       0
50266   <test2> 0
42      Ġthis   0
2       
    1


Expected output:


0       
     1
9226    this    0
1437    Ġ       0
50265   <test1> 1
14      Ġthat   0
1437    Ġ       0
50266   <test2> 1
42      Ġthis   0
2       
    1



Comments:
I also found that -  even when you add these specialized tokens it does not tokenise correctly.

I got it working by using the `tokenizer.get_special_tokens_mask` and passing the `already_has_special_tokens=True`, instead of using the `__call__` function of the tokenizer. 
In any case, the interface for `tokenizer(..., return_special_tokens_mask=True)` is confusing, and one has to look at the actual [source code](https://github.com/huggingface/transformers/blob/aaee4038c3c34faea58c84e04fc88297e2be6cb2/src/transformers/tokenization_utils_base.py#L3007) to figure out `return_special_tokens_mask=True` doesn't work for additional special tokens.

This happens for both fast and slow tokenizers.
The problem for fast tokenizers seems to originate in the [rust implementation](https://github.com/huggingface/transformers/blob/31ec2cb2badfbdd4c1ac9c6c9b8a74e974984206/src/transformers/tokenization_utils_fast.py#L425) (?)

The problem for slow tokenizers seems to be as @armancohan pointed out that `already_has_special_tokens` is not set to True [here](https://github.com/huggingface/transformers/blob/aaee4038c3c34faea58c84e04fc88297e2be6cb2/src/transformers/tokenization_utils_base.py#L3009).

Thanks for this @armancohan, saved me quite some time.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi, I encountered the same problem here. Thanks @armancohan for the fast solutions. Should we open a MR to fix the issue?