Adding the same special token twice cause the additional_special_tokens to be set to an empty list inside tokenizer
Author: dblakely
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-5.4.0-166-generic-x86_64-with-glibc2.31
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): NA
- Jax version: NA
- JaxLib version: NA
- Using GPU in script?: NA
- Using distributed or parallel set-up in script?: NA

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The issue is that if you re-add a particular special token to the tokenizer, it'll make that token no longer an "additional_special_token" (the token will still exist in the vocabulary, but it'll cease to be treated as a special token).

Here's an example to reproduce the issue:
```python
from transformers import LlamaTokenizer

base_model = "meta-llama/Llama-2-7b-chat-hf"

# Create tokenizer and add an additional special token
tokenizer_1 = LlamaTokenizer.from_pretrained(base_model)
tokenizer_1.add_special_tokens({"additional_special_tokens": ["<tok>"]})
print(tokenizer_1.additional_special_tokens) # outputs `['<tok>']` as expected
tokenizer_1_checkpoint = "tokenizer_1"
tokenizer_1.save_pretrained(tokenizer_1_checkpoint)

# Load the above tokenizer and add the same special token a second time
tokenizer_2 = LlamaTokenizer.from_pretrained(tokenizer_1_checkpoint)
tokenizer_2.add_special_tokens({"additional_special_tokens": ["<tok>"]})
print(tokenizer_2.additional_special_tokens) # outputs `[]` which seems incorrect
```

This might seem contrived, but I have a training script where I add a special token by default to the tokenizer to ensure that it's there (I don't always know a priori that it's already been added). However, I found that if it is already there, the behavior of the tokenizer changes a bit. For example, the tokenizer will cease to remove the token when decoding automatically. After digging into it, I realized it's because the `additional_special_tokens` list was getting reset.

### Expected behavior

I'd expect adding the exact same special token a second time to have no effect whatsoever on the tokenizer.

Comments:
On going through the source code for `add_special_tokens` in the link [here](https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L873), it seems like this function has a param `replace_additional_special_tokens` which defaults to `True`.

This is an excerpt from the docstring for this param:

If `True`, the existing list of additional special tokens will be replaced by the list provided in `special_tokens_dict`. Otherwise, `self._additional_special_tokens` is just extended. In the former case, the tokens will NOT be removed from the tokenizer's full vocabulary - they are only being flagged as non-special tokens.

So, what happened is after you added the same token again, as you expected, it wasn't added, which is why an empty list was outputted.

So, it seems like what you wanna do is explicitly set this param to `False`.

Also, you can execute `print(tokenizer_2)` to verify that the token you added prior to saving is retained. This is what I see using your code sample:

```
LlamaTokenizer(name_or_path='tokenizer_1', vocab_size=32000, model_max_length=1000000000000000019884624838656, is_fast=False, padding_side='right', truncation_side='right', special_tokens={'bos_token': '
', 'eos_token': '
', 'unk_token': '<unk>'}, clean_up_tokenization_spaces=False),  added_tokens_decoder={
	0: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	1: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	2: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
	32000: AddedToken("<tok>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
```
Clearly your special token is retained after loading.

I actually discovered that as well and that is how I solved the problem on my end. But I still felt like the situation seemed like a bug--they initialize an empty set of additional special tokens but then don't end up adding a special token that already exists. And so the set remains empty, which in effect removes the special token from the set. My guess is that this wasn't intended.

Yep, I can reproduce and that's indeed a good catch. I have no idea why this is happening yet, opening a pr now 🤗 

Sorry! Christmas delays 🤗 making sure this is fixed for next release