[MBart50] Inconsistent decoding with additional special tokens between slow and fast tokenizers 
Author: fleonce
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-6.2.0-25-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Load a non-fast Tokenizer for mBART
2. Add an additional special token to it
3. Encode and then decode input containing previously added special token

```python3
from transformers import MBart50Tokenizer

tk = MBart50Tokenizer.from_pretrained('facebook/mbart-large-50')
tk.add_tokens('<token>', True)
print(tk.decode(tk("This is my example sentence with a special <token> token")["input_ids"]))
>>> 'en_XXThis is my example sentence with a special <token> token
'
```
This differs from the fast tokenizers' decoding scheme, as it will correctly decode the input with a space after `en_XX`. I believe this is due to the implementation for `legacy_added_tokens` in https://github.com/huggingface/transformers/blob/3cefac1d974db5e2825a0cb2b842883a628be7a0/src/transformers/tokenization_utils.py#L1002-L1022
and more specifically the second part of the set definition for `legacy_added_tokens` that accounts for special tokens that have been added manually after loading (?)

When disabling the special handling for `legacy_added_tokens`, the tokenization output would be correct, so I was primarily wondering for what reason this was added and whether removing this would potentially break other tokenizers.

### Expected behavior

```python3
fast_tk = MBart50TokenizerFast.from_pretrained('facebook/mbart-large-50')
fast_tk.add_tokens('<token>', True)
print(fast_tk.decode(fast_tk("This is my example sentence with a special <token> token")["input_ids"])))
>>> 'en_XX This is my example sentence with a special <token> token
'
```
The decoding should match the fast tokenizers' output (?), at least I would assume so.

Comments:
When confirmed this is unexpected behaviour I would be willing to research behaviour of other tokenizers in the same setting and submit a pull request addressing the issue :-)

Thanks @fleonce for your issue! @ArthurZucker will take a look at your issue as soon as he's back from leave, in about a week. Thanks a lot!

Hey @fleonce the issue is rather with https://github.com/huggingface/transformers/blob/b1292bca6923cfbc9cb3f70cb55df57e4e17e630/src/transformers/models/mbart50/tokenization_mbart50.py#L249

the `legacy_added_tokens` was added in #23909  to be BC. 
I am almost certain that if you checkout an earlier commit the same issue would be present since the `convert_tokens_to_string` method is what is wrong for me here since it takes : `['en_XX', '▁This', '▁is', '▁my', '▁example', '▁sentence', '▁with', '▁a', '▁special']` and outputs `'en_XXThis is my example sentence with a special'`. If you want to open a PR for a fix feel free to do so! 
This might affect tokenizer with the exact same `convert_tokens_to_string`

In this case, the `_decode` might need a tiny rework ! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.