Llama tokenizers behaviours different with special tokens.
Author: ShomyLiu
Pull request: no
Labels: name: Core: Tokenization

name: Should Fix

### System Info

- `transformers` version: 4.39.1
- Platform: Linux-3.10.0-1062.9.1.el7.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

the code sample is :
```
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("./llama2", use_fast=False)

# add three extra different type tokens:
tok.add_tokens(['|<im_start1>|'], special_tokens=False)
tok.add_tokens(['|<im_start2>|'], special_tokens=True)
tok.add_special_tokens({"pad_token": '|<im_start3>|'})

print(tok)
# output:
"""
LlamaTokenizer(name_or_path='./llama2/', vocab_size=32000, model_max_length=1000000000000000019884624838656, is_fast=False, padding_side='right', truncation_side='right', special_tokens={'bos_token': '
', 'eos_token': '
', 'unk_token': '<unk>', 'pad_token': '|<im_start3>|', 'additional_special_tokens': ['|<im_start2>|']}, clean_up_tokenization_spaces=False),  added_tokens_decoder={
        0: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
        1: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
        2: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
        32000: AddedToken("|<im_start1>|", rstrip=False, lstrip=False, single_word=False, normalized=True, special=False),
        32001: AddedToken("|<im_start2>|", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
        32002: AddedToken("|<im_start3>|", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
}
"""

print(tok.tokenize("|<im_start1>|system"))
# ['▁', '|<im_start1>|', 'system']

print(tok.tokenize("|<im_start2>|system"))
# ['|<im_start2>|', 'system']

print(tok.tokenize("|<im_start3>|system"))
# ['|<im_start3>|', 'system']

print(tok.decode(tok("|<im_start1>|system").input_ids))
# '
  |<im_start1>| system'

print(tok.decode(tok("|<im_start2>|system").input_ids))
# '
 |<im_start2>| system'

print(tok.decode(tok("|<im_start3>|system").input_ids))
# '
|<im_start3>|system'
```

We can see that only the last example of using '|<im_start3>|' could recover the original input, and there will be a space in the other two examples.
So it is confusing, and I don't know what is right? (maybe the |<im_start3>| is the expected behavior?)
- what is the difference of the following two methods:
 ```
    tok.add_tokens(['|<im_start2>|'], special_tokens=True)
    tok.add_special_tokens({"pad_token": '|<im_start3>|'})
 ```
- Why there are extra space in the above three tokens? 
- How to change the behaviors of '|<im_start1>|' and '|<im_start2>|' to the same as '|<im_start3>|'? 

Would you mind taking care of this issue? @ArthurZucker 
Thanks a lot. 


### Expected behavior

I expect the behaviors of '|<im_start3>|'. 



Comments:
Hey! 
The extra spaces come from the `spaces_between_special_tokens` option that is set to `True` I believe. We'll try to get rid of this. 

Related to #29452 as well

@ArthurZucker  Hi, thanks for the reply. 
However, it does not work to set `spaces_between_special_tokens `, since the default value is `False`. 
You could execute the above code to reproduce the output (just with the llama2-tokenizer).

Thanks again.

Can you share an official repo `"./llama2"` is not on my path 

Yeah, just the Llama2 repo:
```
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b", use_fast=False)
```

Updated, even if I set `add_bos_token=False` the output is similar, there will be some space between `|<im_start>|` and `system`.
```
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b", use_fast=False, add_bos_token=False)
```
the output:
```
['▁', '|<im_start1>|', 'system']
['|<im_start2>|', 'system']
['|<im_start3>|', 'system']
|<im_start1>| system
|<im_start2>| system
|<im_start3>|system
```
seems that only if that a token must be `special tokens` (with special name not in the additional_special_tokens list) could recover the origin input.
 



1. `print(tok.tokenize("|<im_start1>|system"))` gives `['▁', '|<im_start1>|', 'system']`. That is expected, because "added tokens" tokens that are not special are simply treated as tokens added. So the prefix space is prepended. The space after the added token will be fixed by #28881 
2. Using `tok.convert_tokens_to_string(tok.convert_ids_to_tokens(tok("|<im_start1>|system").input_ids))` shows that the issue lies with the `decode` function. `print(tok.decode(tok("|<im_start2>|system").input_ids, spaces_between_special_tokens=False))` gives the expected results. `"
|<im_start2>|system"`
3. The different with `tok.add_special_tokens({"pad_token": '|<im_start3>|'})` is the `legacy_added_tokens`. Adding with `add_special_tokens` updated the `self.additional_special_tokens` list with ` ['|<im_start2>|']`. I'll open a PR for this, it's not really expected IMO

In all cases, `spaces_between_special_tokens =True` as it is the default. 

This is all for BC with previous code. I'll roll out the changes to deprecate `"spaces_between_special_tokens"` . Thanks for reporting @ShomyLiu 

Get it, thanks for the explanation. I first wrongly set the `spaces_between_special_tokens =False` in the tokenizer initialized code. 
```
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b", use_fast=False, spaces_between_special_tokens=False ).
```
It turned out to be the `decode function` problem.
Thanks a lot.  

BTW, if setting `legacy=True` the behaviors will be different again.

yeah `legacy` is legacy, it's even more wrong