T5 tokenizer adds whitespace after added token
Author: harshil-shah
Pull request: no

### System Info

- `transformers` version: 4.33.2
- Platform: Linux-6.2.0-33-generic-x86_64-with-glibc2.35
- Python version: 3.11.5
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hi,

When adding a token to the T5 tokenizer and then tokenizing a string, it seems that the encoding step is inserting an unwanted space after the added token.

```python
from transformers import AddedToken, T5TokenizerFast

tokenizer = T5TokenizerFast.from_pretrained("google/flan-t5-small")
tokenizer.add_tokens(["<"])

print(tokenizer.encode("<body>"))  # [32100, 643, 3155, 1]
print(tokenizer.decode(tokenizer.encode("<body>")))  # < body>

print(tokenizer.convert_ids_to_tokens(tokenizer.encode("<body>")))  # ['<', '▁body', '>', '
']
```

It's unclear why the model is using the token `"▁body"` when `"body"` is also in the vocabulary? And even if `"body"` weren't in the vocabulary, I'd still expect `convert_ids_to_tokens` to give back something like `["<", "b", "o", "d", "y", ">", "
"]`.

### Expected behavior

The following script should print `<body>
`.

```python
from transformers import AddedToken, T5TokenizerFast

tokenizer = T5TokenizerFast.from_pretrained("google/flan-t5-small")
tokenizer.add_tokens(["<"])

print(tokenizer.decode(tokenizer.encode("<body>")))
```

I saw https://github.com/huggingface/transformers/pull/24565 but this doesn't seem to have solved it for this case?

Comments:
Hey, this is the same as  #25881 , the fix to `rust` has not been done yet and is more involved. I'll try to get to it! 

Thank you!

Linked the fix PR 😉 

The PR is in a good state, should be mergeable this week. It uncovers more "inconsistencies" with slow and fast, but I'll document all of this there! 😉 You can already do something like:
```python 
from tokenizers.pre_tokenizers import Metaspace
.... # tokenizer.from_pretrained etc
tokenizer._tokenizer.pre_tokenizer = Metaspace(add_prefix_space = True, replacement='▁', prepend_scheme = "first") 
```


@ArthurZucker Even after following the step in your previous comment, it still seems to be producing incorrect output for certain inputs:
```py
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained('t5-base', use_fast=True)
print(tok.encode("
test
", add_special_tokens=False)) # Broken

from tokenizers.pre_tokenizers import Metaspace
tok._tokenizer.pre_tokenizer = Metaspace(add_prefix_space = True, replacement='▁', prepend_scheme = "first")
print(tok.encode("
test
", add_special_tokens=False)) # Should be fixed, but isn't
```
In both cases, `[1, 794, 1]` is printed which corresponds to `['
', '▁test', '
']`... but it should be `[1, 4377, 1]` which corresponds to `['
', 'test', '
']`. This can be achieved with the slow tokenizer with legacy set to false:
```py
from transformers import AutoTokenizer
slow = AutoTokenizer.from_pretrained('t5-base', use_fast=False, legacy=False)
print(slow.encode("
test
", add_special_tokens=False)) # [1, 4377, 1]
```

I've also tested saving and loading the tokenizer again (see [here](https://huggingface.co/Xenova/t5-tokenizer-new)), but that has the same problem. I'm using `tokenizers==0.15.0` and `transformers==4.36.1` (latest).


It is worth noting that it does fix other problems, like `"Hey 
. how are you"`:
- Old (incorrect): `['▁Hey', '▁', '
', '▁', '.', '▁how', '▁are', '▁you']`
- New (correct):   `['▁Hey', '▁', '
', '.', '▁how', '▁are', '▁you']`


Indeed. That's a different issue which also comes from the `extract_and_normalize` piece of code. I'll see if there is a quick fix thanks for reporting 

Also note that the template processors usually use this: https://github.com/huggingface/transformers/blob/e6dcf8abd6f65bb4b6dfc1831b20d9ba49ce00e2/src/transformers/models/llama/tokenization_llama_fast.py#L160
with a prefix space before the sequence. 

> Also note that the template processors usually use this:
> 
> https://github.com/huggingface/transformers/blob/e6dcf8abd6f65bb4b6dfc1831b20d9ba49ce00e2/src/transformers/models/llama/tokenization_llama_fast.py#L160
> 
> 
> with a prefix space before the sequence.

Even with `add_special_tokens=False`? 👀