Fast tokenizer breaks added tokens
Author: geronimi73
Pull request: no

### System Info

- `transformers` version: 4.35.0.dev0
- Platform: Linux-6.2.0-35-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.25.0.dev0
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("models/llama2-7b", use_fast=False)

# add tokens for chatml
tokenizer.add_tokens(["
"])
tokenizer.add_special_tokens({"eos_token": "
"})

messages = [ {"role": "user", "content": "question"},
  {"role": "assistant", "content": "answer"} ]

# https://huggingface.co/docs/transformers/main/chat_templating
tokenizer.chat_template = "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'
' + message['role'] + '\n' + message['content'] + '
' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '
assistant\n' }}{% endif %}"

chat = tokenizer.apply_chat_template(messages, tokenize=False)
chat_tokenized = tokenizer(chat, add_special_tokens=False)["input_ids"]

for token in chat_tokenized:
	print(f"{token} - \"{tokenizer.decode(token)}\"")
```

**output**: first occurence of `
` is correctly tokenized, second one is split

```
32000 - "
"
1792 - "user"
13 - "
"
12470 - "question"
32001 - "
"
29871 - ""
13 - "
"
**29966 - "<"
29989 - "|"
326 - "im"
29918 - "_"
2962 - "start"
29989 - "|"
29958 - ">"**
465 - "ass"
22137 - "istant"
13 - "
"
12011 - "answer"
32001 - "
"
29871 - ""
13 - "
"
```


### Expected behavior

```
32000 - "
"
1404 - "user"
13 - "<0x0A>"
12470 - "question"
32001 - "
"
29871 - ""
13 - "<0x0A>"
32000 - "
"
20255 - "assistant"
13 - "<0x0A>"
12011 - "answer"
32001 - "
"
29871 - ""
13 - "<0x0A>"
```

this is the correct output of the slow tokenizer `AutoTokenizer.from_pretrained("models/llama2-7b", use_fast=False)`

1. why does this happen with fast but not slow?
2. any other solution than **not** using the fast tokenizer?

i guess this is known, sorry if I missed it in the existing issues


Comments:
Hi @geronimi73, thanks for raising an issue! 

@ArthurZucker is off for this week and is the main person who knows and works with the tokenizers, so you might have to wait until then to have an answer. 

@Rocketknight1 any chance you know what's happening? 

Hi @geronimi73, I'll wait for @ArthurZucker to return to give a full answer here, but in the meantime I think the issue is that when you add a normal token, the tokenizer may split it. If you want to preserve an important control token like `
` you should make it a special token. Try doing this instead:

```python
tokenizer.add_special_tokens({"additional_special_tokens": ["
"]})
tokenizer.add_special_tokens({"eos_token": "
"})
```

Well, it's partly true partly wrong 😅 
When you add a token, if it is not special, it will be normalized by default. I'll add the `add_tokens` function to the doc it seems that it was removed. But anyway, the Llama normalizer adds a `SPIECE_UNDERLINE` at the beginning of the special tokens, which will thus be a different token. AddedTokens (special or not) should never be splitted, but the content of the added tokens is affected by the normalizer

ok, thanks!

it's somehow working now. 

just to **sum this up** for others who are struggling with this too:
- I raised this issue because the fast tokenizer breaks the ChatML tag `
` into several tokens even though it was added with `tokenizer.add_tokens(["
"])`, slow tokenizer works fine
- @ArthurZucker explains above, Llama normalizer adds a SPIECE_UNDERLINE; indeed, fast tokenizer encodes `
` correctly when token is added with ..
```python
tokenizer.add_tokens(
	AddedToken("
",normalized=False))
)

```
- but, new problem. decoding now **adds a space** after added tokens, example
```python
tokenizer = AutoTokenizer.from_pretrained("../models/llama2-7b", use_fast=True, legacy=False)

tokenizer.add_tokens(
	AddedToken("
",normalized=False, rstrip=True, lstrip=False)
)
tokenizer.add_special_tokens({"eos_token": "
"})

# https://huggingface.co/docs/transformers/main/chat_templating
tokenizer.chat_template = "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'
' + message['role'] + '\n' + message['content'] + '
' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '
assistant\n' }}{% endif %}"

messages=[
    {"role": "user", "content": "Hi there!"},
    {"role": "assistant", "content": "Nice to meet you!"}
]

chat = tokenizer.apply_chat_template(messages, tokenize=False)
chat_tokenized = tokenizer(chat, add_special_tokens=False)["input_ids"]

print("INPUT")
print(chat)
print("-"*30)
print("DECODE(ENCODE(INPUT))")
print(tokenizer.decode(chat_tokenized))

# INPUT
# 
user
# Hi there!

# 
assistant
# Nice to meet you!


# ------------------------------
# DECODE(ENCODE(INPUT))
# 
 user
# Hi there!
 
# 
 assistant
# Nice to meet you!
 
```

- **fix all of the above**: use slow tokenizer `use_fast=False, legacy=False`, add tokens with `tokenizer.add_tokens(["
"])`, decode with `spaces_between_special_tokens=False` like this
```python
tokenizer = AutoTokenizer.from_pretrained("../models/llama2-7b", use_fast=False, legacy=False)
tokenizer.add_tokens(["
"])
...
chat_tokenized = tokenizer(chat, add_special_tokens=False)["input_ids"]
print(tokenizer.decode(chat_tokenized, spaces_between_special_tokens=False))
```
- using `transformers 4.35.0` btw

Thanks for the great explanation! 
Regarding the space added after added tokens, this PR will fix it: https://github.com/huggingface/tokenizers/pull/1357 😉 I'll have to change the Llama paradigm a little bit to make sure it's compatible 

feel free to play with #26678 as well 🤗