LlamaTokenizer: Slow implementation opts for whitespace-lead token (different from fast)
Author: lbeurerkellner
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.15.0-75-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help?

@ArthurZucker @youn

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Reproduction

Comparing slow and fast `LlamaTokenizer` instances with `huggyllama/llama-7b`.

```
from transformers import AutoTokenizer

model = "huggyllama/llama-7b"

fast = AutoTokenizer.from_pretrained(model)
slow = AutoTokenizer.from_pretrained(model, use_fast=False)

# use tokenize()
print(fast.tokenize("
uns"), slow.tokenize("
uns"))
# -> (['▁
', 'uns'], ['
', '▁uns'])

# use __call__
print(fast(f"{fast.bos_token}uns", add_special_tokens=False), slow(f"{slow.bos_token}uns", add_special_tokens=False))
# -> ({'input_ids': [1, 6948], 'token_type_ids': [0, 0], 'attention_mask': [1, 1]},
#     {'input_ids': [1, 9644], 'attention_mask': [1, 1]})

# round-tripping
print(fast.convert_tokens_to_string(fast.tokenize("
uns")), fast.convert_tokens_to_string(slow.tokenize("
uns")))
# -> ('
uns', '
 uns')
```

### Expected behavior

It looks like the slow LlamaTokenizer wrongly tokenises `uns`. I would not expect the additional whitespace when round-tripping or when tokenising in the first place.

Thanks a lot in advance.

Comments:
Thanks for reporting, will have a look

Hi @ArthurZucker! Are you currently working on this? If not, I think I could fix it pretty quickly :)

Sure! Feel free to take it! 😉 I'll have a look soon otherwise


@ArthurZucker @lbeurerkellner I have done some debugging and I have a few observations. Firstly I have checked other tokenizers that use `LlamaTokenizer` or `LlamaTokenizerFast` and the results are pretty weird:

1) the issue is not with `uns` but with any word after a special token like `
`. Why this is happening is pretty straightforward

```
# 
 is added to Trie so there is a split after its encounter in the text
tokens = self.tokens_trie.split(text) # tokenization_utils.py:517
```
So it seems like it was a deliberate decision to split special tokens like this? 

1) because of the above split, all slow tokenizers based on `LLaMaTokenizer` return `['
', '▁uns']`

2) more interesting thing is that most of the tokenizers based on `LlamaTokenizerFast` split text into `['▁
', 'uns']` (e.g `fxmarty/tiny-llama-fast-tokenizer`). But for example `openlm-research/open_llama_3b` which is one of the most downloaded llama based models outputs `['
', '▁uns']` even thought it has the same tokenizer config like the one from fxmarty.

```LlamaTokenizerFast(name_or_path='openlm-research/open_llama_3b', vocab_size=32000, model_max_length=2048, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token': AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=True), 'eos_token': AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=True), 'unk_token': AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True)}, clean_up_tokenization_spaces=False)```

the fast is working properly! As suspected, this is linked to #24622 and #24565. I am working on a fix for all our spm based models. 

For other tokenizers, I wouldn’t refer to them since a lot are outdated/don’t include some fixes

Actually this is fixed, the output is now `['▁
', 'uns'] ['
', 'uns']`. The fast  just works that way for tokenization, but the output is the same. Use 
```python 
slow = AutoTokenizer.from_pretrained(model, use_fast=False, legacy = False)
```