Discrepancy between `LlamaTokenizer` and `LlamaTokenizerFast` outputs
Author: PyroGenesis
Pull request: no

### System Info

- `transformers` version: 4.32.0
- Platform: Windows-10-10.0.20348-SP0
- Python version: 3.10.11
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

It seems like the `LlamaTokenizer` and `LlamaTokenizerFast` tokenize text differently. Specifically, it looks like the `LlamaTokenizer` eats some characters instead of tokenizing them.

Code:
```py
import torch
from transformers import LlamaForCausalLM, LlamaTokenizer, LlamaTokenizerFast

# I tried legacy=False here too, the results are the same
tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
tokenizer_fast = LlamaTokenizerFast.from_pretrained("meta-llama/Llama-2-7b-chat-hf")

# This functions transforms the user prompt into the prompt template mentioned here:
# https://huggingface.co/blog/llama2#how-to-prompt-llama-2
def createPromptTemplate(prompt):
    system_prompt = 'Test system prompt'
    
    return '\n'.join([
        "

 
",
        system_prompt,
        "
",
        "",
        f"{prompt} 
"
    ])

input_prompt = createPromptTemplate("Test user prompt.")

print("Input:")
print(input_prompt)
print('\nLlamaTokenizer')
print(tokenizer.tokenize(input_prompt))
print('\nLlamaTokenizerFast')
print(tokenizer_fast.tokenize(input_prompt))
```

Output:
```
Input:


 

Test system prompt



Test user prompt. 


LlamaTokenizer
['
', 'INST', ']', '▁<<', 'SY', 'S', '>>', '<0x0A>', 'Test', '▁system', '▁prompt', '<0x0A>', '<', '</', 'SY', 'S', '>>', '<0x0A>', '<0x0A>', 'Test', '▁user', '▁prompt', '.', '▁[', '/', 'INST', ']']

LlamaTokenizerFast
['
', '▁[', 'INST', ']', '▁<<', 'SY', 'S', '>>', '<0x0A>', 'Test', '▁system', '▁prompt', '<0x0A>', '<', '</', 'SY', 'S', '>>', '<0x0A>', '<0x0A>', 'Test', '▁user', '▁prompt', '.', '▁[', '/', 'INST', ']']
```

### Expected behavior

I expected the tokenization of `LlamaTokenizer` to keep all the characters in the input.

Comments:
On main I am getting this: 
```python 
LlamaTokenizer
['
', '[', 'INST', ']', '▁<<', 'SY', 'S', '>>', '<0x0A>', 'Test', '▁system', '▁prompt', '<0x0A>', '<', '</', 'SY', 'S', '>>', '<0x0A>', '<0x0A>', 'Test', '▁user', '▁prompt', '.', '▁[', '/', 'INST', ']']

LlamaTokenizerFast
['
', '▁[', 'INST', ']', '▁<<', 'SY', 'S', '>>', '<0x0A>', 'Test', '▁system', '▁prompt', '<0x0A>', '<', '</', 'SY', 'S', '>>', '<0x0A>', '<0x0A>', 'Test', '▁user', '▁prompt', '.', '▁[', '/', 'INST', ']']
```
The fast version is a bit wrong as it does not benefit from the fix that prevents adding an extra space after special tokens. As you can see: 
```diff
'[', 'INST', ']'
+ '▁[', 'INST', ']'
```
You should not manually add the token but use `add_special_token = True` 😉 

Oh yes, I didn't realize that the Fast one is incorrect too. 

About adding tokens, I did not add any new tokens, just used the pretrained tokenizer directly. And correct me if I am wrong, but the `add_special_token` is True by default for the tokenizer call right?

Also, since the `LlamaTokenizer` on main is working correctly for you, can you tell me how to update to the version on main? Or do I need to wait for a new release?

What I mean by `add_special_token = True` is that in the snippet you shared, you added the `
` token manually. A foolproof way to add it si to use `tokenizer.encode(text, add_special_tokens = True)`. Will produce the same outputs for both the fast and slow 😉 

Ok so first of all, I downloaded the bleeding edge of the `transformers` library (using `pip install git+https://github.com/huggingface/transformers`) and now my results from `LlamaTokenizer` match yours:

```
LlamaTokenizer
['
', '[', 'INST', ']', '▁<<', 'SY', 'S', '>>', '<0x0A>', 'Test', '▁system', '▁prompt', '<0x0A>', '<', '</', 'SY', 'S', '>>', '<0x0A>', '<0x0A>', 'Test', '▁user', '▁prompt', '.', '▁[', '/', 'INST', ']']
```
---
I also now understand what you mean by using `add_special_token`. If I run through the below input:
```

 

Test system prompt



Test user prompt. 

```
with the code :
```py
tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf", legacy=True)
print(tokenizer.decode(tokenizer(input_prompt, add_special_tokens=True).input_ids))
```
I get the output:
```


 

Test system prompt



Test user prompt. 

```

But if I run the same code with `legacy=False` it puts a space after the `
` token instead:
```

 
 

Test system prompt



Test user prompt. 

```
Is this intentional? I'm just trying to follow [the prompt format for llama2](https://huggingface.co/blog/llama2#how-to-prompt-llama-2) as closely as possible

---

Lastly `LlamaTokenizerFast` seems to add a space after the `
` token, no matter whether I use `legacy=True` or `legacy=False`.
Is this intentional too?

Few things here. 
What you should be looking at is not `tokenizer.decode(tokenizer.encode(xxx))` but the `input_ids`. In this case you'll see that the extra space is not part of the `input_ids`. It is added by the tokenizer when decoding. You should notice that it is added both by the fast and the slow tokenizer. 

Is this expected? Yes:
- when you add special token, you actually add the ids directly, instead of adding the tokens before the prompt. This means that the prompt is tokenizer with an additional prefix space. While I do understand this is not exactly what you want, the actual prompt format need the extra space. See [here](https://github.com/facebookresearch/llama/blob/main/llama/generation.py#L244)
```python
 self.tokenizer.encode(f"{B_INST} {(prompt['content']).strip()} {E_INST} {(answer['content']).strip()} ",bos=True,eos=True)
```

So the blog is a little bit missleading but the code in `transformers` also uses this (see [here](https://github.com/ArthurZucker/transformers/blob/d7732c60c9fc61201b8a120a836e27fb3d8b3577/src/transformers/models/llama/tokenization_llama.py#L429-L432))

Lastly, the `fast` tokenizer does not use the `legacy` flag (yet 😉 )

I think I understand it correctly now. That extra space is the difference of how the decoder works rather than a difference in the encoder. I can confirm using this code:
```py
tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf", legacy=False)
tokenizer_legacy = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf", legacy=True)
tokenizer_fast = LlamaTokenizerFast.from_pretrained("meta-llama/Llama-2-7b-chat-hf")

print('LlamaTokenizer')
print(tokenizer(input_prompt, add_special_tokens=True).input_ids)
print('\nLlamaTokenizer Legacy')
print(tokenizer(input_prompt, add_special_tokens=True).input_ids)
print('\nLlamaTokenizerFast')
print(tokenizer(input_prompt, add_special_tokens=True).input_ids)
```

which gives me the same output for all tokenizers:
```
LlamaTokenizer
[1, 518, 25580, 29962, 3532, 14816, 29903, 6778, 13, 3057, 1788, 9508, 13, 29966, 829, 14816, 29903, 6778, 13, 13, 3057, 1404, 9508, 29889, 518, 29914, 25580, 29962]

LlamaTokenizer Legacy
[1, 518, 25580, 29962, 3532, 14816, 29903, 6778, 13, 3057, 1788, 9508, 13, 29966, 829, 14816, 29903, 6778, 13, 13, 3057, 1404, 9508, 29889, 518, 29914, 25580, 29962]

LlamaTokenizerFast
[1, 518, 25580, 29962, 3532, 14816, 29903, 6778, 13, 3057, 1788, 9508, 13, 29966, 829, 14816, 29903, 6778, 13, 13, 3057, 1404, 9508, 29889, 518, 29914, 25580, 29962]
```

Thanks for also letting me know that the actual prompt format does have the prefix space, it just wasn't present in the blog (because the blog was written during the legacy decoder behavior?)

Thank you for taking the time to explain all of this to me. I'm a little new to tokenizers and I appreciate it!