Empty string token gets added for certain numbers with T5 and LLama for fast and slow tokenizers
Author: SumanthRH
Pull request: no

### System Info

`transformers` 4.35.0, `python` 3.11.5

Can also reproduce on the latest dev version 4.36.0.dev0

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

An empty string token (or, equivalently, a whitespace after decoding) gets added for T5 and Llama tokenizers (both slow and fast tokenizers). This seems to be related to the fix #26678 - however the problem is seen in *both* slow and fast tokenizers. Here's a code snippet to reproduce :

```
tok = AutoTokenizer.from_pretrained("meta-llama/LLama-2-7b-hf", use_fast=True)
print(tok.batch_decode(tok.encode("there are 410 people"))) # ['
', 'there', 'are', '', '4', '1', '0', 'people']
tok = AutoTokenizer.from_pretrained("meta-llama/LLama-2-7b-hf", use_fast=False)
print(tok.batch_decode(tok.encode("there are 410 people"))) # ['
', 'there', 'are', '', '4', '1', '0', 'people']
```

Notice the extra token (I'm using `batch_decode` to see the individual tokens, but if you use `decode`, you'll see an extra whitespace). I see the same issue with `t5-base`, but the issue goes away for certain numbers with t5 (400, for example). Also, I used the `transformers` fork in PR #26678 (after building from source), but even with the `legacy` flag, the problem remains, so I believe this is a separate issue.

### Expected behavior

No extra empty string token/whitespace in decoded output

Comments:
Hey! As you can read[ from the documentation](https://huggingface.co/docs/transformers/main/model_doc/llama#transformers.LlamaTokenizer.legacy), if you do not set the flag `legacy` to `False`, you won't get the fix for the slow tokenizers. For Fast tokenizers, this can already be fixed by doing:
```python 
from tokenizers import Metaspace
tokenizer._tokenizer.pre_tokenizer = Metaspace(add_prefix_space = True, replacement = "▁", prepend_scheme = "first")
```
for T5 and for Llama the normnalizer should be set to None. 
The proper fix is coming in #26678. 🤗 

Ah I got my mistake! I was actually referring to the empty string token that gets added in the middle, not the whitespace that gets added in the beginning. I always saw the empty string token being present in `tok.encode`, and I naively assumed that meant that `tok.decode` would add an extra whitespace (turns out to NOT be the case). My main confusion stemmed from the fact that tokenization for numbers differs across T5 and Llama. It looks like for Llama, the tokenizer implicitly adds whitespaces between tokens that are words, but doesn't do so with numbers!
```
tok = AutoTokenizer.from_pretrained("meta-llama/LLama-2-7b-hf", use_fast=False)
print(tok.encode("there are 4 people"), add_special_tokens=False) # [727, 526, 29871, 29946, 2305]
print(tok.batch_decode([727, 526, 29871, 29946, 2305])) # ['there', 'are', '', '4',  'people']
print(tok.decode([727, 526, 29871, 29946, 2305])) # orig string "there are 4 people"
```
None of the tokens have whitespace, unlike GPT2 (which gives `['there', ' are', ' 4', ' people']`). And if you remove the whitespace token (29871 ID) you get:
```
print(tok.decode([727, 526, 29946, 2305])) # "there are4 people"
```
Looks like this difference in the pretokenization (and detokenizer) in adding a whitespace is causing this behaviour. Please correct me if I'm wrong!

No both Llama and T5 add a prefix token to the sentence they have as an input. Main differences:
- different training 
- different algorithm (unigram vs BPE) 
- different stripping mechanism (T5 replaces any occurences of spaces to a single space)
😉 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.