`additional_special_tokens` is replaced instead of being updated
Author: ydshieh
Pull request: no



### Reproduction

```
from transformers import T5Tokenizer
from transformers.tokenization_utils import AddedToken

tokenizer = T5Tokenizer.from_pretrained("t5-small", extra_ids=0, additional_special_tokens=["new_token_1"])

print(tokenizer.additional_special_tokens)
print(tokenizer.added_tokens_encoder)

tokenizer.add_special_tokens({"additional_special_tokens": ["new_token_2"]})
print(tokenizer.additional_special_tokens)
print(tokenizer.added_tokens_encoder)

tokenizer.add_special_tokens({"additional_special_tokens": ["new_token_3"]})
print(tokenizer.additional_special_tokens)
print(tokenizer.added_tokens_encoder)
```

gives 

```
['new_token_1']
{'new_token_1': 32000}
['new_token_2']
{'new_token_1': 32000, 'new_token_2': 32001}
['new_token_3']
{'new_token_1': 32000, 'new_token_2': 32001, 'new_token_3': 32002}
```

### Expected behavior

We should get 

['new_token_1', 'new_token_2', 'new_token_3']

Comments:
I also saw this issue, and found it very unintuitive. Should be addressed in V5 imo. Cc @LysandreJik @SaulLu 

To share what has been discussed on slack. 

I think the naming is confusing but the current behavior of the method is useful because we need to be able to completely change the list of tokens associated with additional special tokens. Maybe calling this method `set_special_tokens` would be less confusing?

Basically, the difficulty here is that I think it's not necessarily obvious what types of tokens a tokenizer can have.

A tokenizer has a vocabulary (a dictionary mapping tokens to ids) that consists of:
1. The vocabulary learned with a tokenization algorithm (BPE, wordpiece or Unigram) 
2. The vocabulary corresponding to the tokens added afterwards and which will be isolated upstream of the tokenization algorithm. They are called `added_tokens`.

Afterwards, some tokens can be special in the sense that they will be associated with a specific role for the model (e.g. the begin of sentence token, the mask token). The additional special tokens list is here to be able to flag more tokens as special for the models that need them [_Even if for the model it is not safe to go and find a special token based on an index in a list but that's another subject_].


The problem is that we need some consistency between `additional_special_tokens` and `added_tokens_encoder`. I don't mean we can achieve this 100%, but in this example, if the role of `additional_special_tokens` is to `completely change the list of tokens associated with additional special tokens`, we can't have `added_tokens_encoder` still keep the original added tokens. This doesn't make any sense and can cause to strange errors like the wrong length of the full tokenizer, which is defined as
```python
    def __len__(self):
        """
        Size of the full vocabulary with the added tokens.
        """
        return self.vocab_size + len(self.added_tokens_encoder)
```

I understood what was bothering you :smile: , that's why in my previous message I tried to show that the notion of "special token" is different from the notion of "added token". So from my point of view, we _don't absolutely need_ some consistency between `additional_special_tokens` and `added_tokens_encoder`.

In the end, what you propose is maybe the desirable behavior of the tokenizer but before making this breaking change I would like to explain why I think that the length of the full tokenizer is not wrong now. 

Currently, the approach is that it's not because we change a flag on a token (e.g. removing a token from the `additional_special_tokens` list)  that we necessarily want to remove it from the vocabulary. Indeed, if we do so, it means that we can potentially reassign these ids to new tokens, which is not an action without consequences either. From my point of view, if we are willing to change the current behavior we need to answer the following questions: in which situations would a user use the `add_special_tokens` method? Can it be used once the `additional_special_tokens`, `bos_token`, etc are already set? If yes, does the user want to exclude those previous tokens from the vocabulary? Can the effect of the method be error-prone (on the result of the model)?

Finally, what I want to say is that I was told to avoid breaking changes as much as possible in transformers. That's why I spent a lot of time trying to figure out the intention behind the effect of each method. In this case, I think there is a valid reason why these methods act the way they do. Maybe today the usage has changed a lot and this effect has finally more disadvantages than advantages. For this particular discussion, I'm not (yet) convinced that's the case.

---------------
To illustrate why I think that the length of the full tokenizer is not currently wrong.

```python
from transformers import T5Tokenizer
from transformers.tokenization_utils import AddedToken

tokenizer = T5Tokenizer.from_pretrained("t5-small", extra_ids=0, additional_special_tokens=["new_token_1"])
text = "this is a text with new_token_1, new_token_2 and new_token_3 "

print(tokenizer.additional_special_tokens)
print(tokenizer.added_tokens_encoder)
print(tokenizer.convert_ids_to_tokens(tokenizer.encode(text)))
print("***")

tokenizer.add_special_tokens({"additional_special_tokens": ["new_token_2"]})
print(tokenizer.additional_special_tokens)
print(tokenizer.added_tokens_encoder)
print(tokenizer.convert_ids_to_tokens(tokenizer.encode(text)))
print("***")

tokenizer.add_special_tokens({"additional_special_tokens": ["new_token_3"]})
print(tokenizer.additional_special_tokens)
print(tokenizer.added_tokens_encoder)
print(tokenizer.convert_ids_to_tokens(tokenizer.encode(text)))
```

```
['new_token_1']
{'new_token_1': 32000}
['▁this', '▁is', '▁', 'a', '▁text', '▁with', 'new_token_1', '▁', ',', '▁new', '_', 'to', 'ken', '_', '2', '▁and', '▁new', '_', 'to', 'ken', '_', '3', '
']
***
['new_token_2']
{'new_token_1': 32000, 'new_token_2': 32001}
['▁this', '▁is', '▁', 'a', '▁text', '▁with', 'new_token_1', '▁', ',', 'new_token_2', '▁and', '▁new', '_', 'to', 'ken', '_', '3', '
']

***
['new_token_3']
{'new_token_1': 32000, 'new_token_2': 32001, 'new_token_3': 32002}
['▁this', '▁is', '▁', 'a', '▁text', '▁with', 'new_token_1', '▁', ',', 'new_token_2', '▁and', 'new_token_3', '
']
```
The last tokenization shows that `new_token_2` and `new_token_3` still are in the vocabulary even if they are not "special tokens" flagged.


Oh! I get your point @SaulLu now ! Thank you for the patience to correct my desire to break everything.


> Oh! I get your point @SaulLu now ! Thank you for the patience to correct my desire to break everything.

No worries, actually I took a long time to understand this subtlety of the code and I'm glad that it can be useful! In any case, this discussion shows that tokenizers are not easy to understand and that we can surely improve this aspect!