Added tokens incorrectly splitted when they immediately follow a subword without a leading whitespace
Author: bilelomrani1
Pull request: no

### System Info

- `transformers` version: 4.34.0.dev0
- Platform: macOS-14.0-x86_64-i386-64bit
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am experimenting with the new Chat Template feature, my use case involves using ChatML with the Llama-base models for further finetuning. I added the `
` and `
` tokens

```python
from transformers import AutoTokenizer

chat = [
    {"role": "user", "content": "Hi there!"},
    {"role": "assistant", "content": "Nice to meet you!"}
]
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer.chat_template = "{% for message in messages %}{{'
' + message['role'] + '\n' + message['content'] + '
' + '\n'}}{% endfor %}"
tokenizer.add_tokens(["
", "
"])
tokenizer.use_default_system_prompt = False
encodings = tokenizer.apply_chat_template(chat, tokenize=True)

print([tokenizer.decode(i) for i in encodings])
```

The output is 

```python
['
',
 'user',
 '\n',
 'Hi',
 'there',
 '!',
 '<',
 '|',
 'im',
 '_',
 'end',
 '|',
 '>',
 '\n',
 '<',
 '|',
 'im',
 '_',
 'start',
 '|',
 '>',
 'ass',
 'istant',
 '\n',
 'N',
 'ice',
 'to',
 'meet',
 'you',
 '!',
 '<',
 '|',
 'im',
 '_',
 'end',
 '|',
 '>',
 '\n']
```

As you can see, the string is not properly tokenized, the tokens `
` and `
` are splitted into subwords. This is not specific to chat template but this is how I encountered the issue. When prefixing the ChatML tokens with a whitespace, the tokenizer properly tokenizes the string:

```python
tokenizer.chat_template = "{% for message in messages %}{{' 
' + message['role'] + '\n' + message['content'] + ' 
' + '\n'}}{% endfor %}"
tokenizer.add_tokens(["
", "
"])
tokenizer.use_default_system_prompt = False
encodings = tokenizer.apply_chat_template(chat, tokenize=True)
print([tokenizer.decode(i) for i in encodings])

>> ['',
 '
',
 'user',
 '\n',
 'Hi',
 'there',
 '!',
 '
',
 '\n',
 '
',
 'ass',
 'istant',
 '\n',
 'N',
 'ice',
 'to',
 'meet',
 'you',
 '!',
 '
',
 '\n']
```


### Expected behavior

The added tokens should be properly tokenized, even if there is no leading whitespace in front of them. The expected output is:

```python
['
',
 'user',
 '\n',
 'Hi',
 'there',
 '!',
 '
',
 '\n',
 '
',
 'ass',
 'istant',
 '\n',
 'N',
 'ice',
 'to',
 'meet',
 'you',
 '!',
 '
',
 '\n']
```

Comments:
Hey, if you add the tokens using the following it will work as expected:
```python 
>>> from transformers import AutoTokenizer, AddedToken
>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
>>> tokenizer.add_tokens([AddedToken("
", normalized = False, special = False), AddedToken("
", normalized = False, special = False)])
>>> tokenizer.tokenize("Hey
how
")
['▁Hey', '
', '▁how', '
']
```
if you do not want the space after the token, you should set `legacy=False`:
```python 
>>> from transformers import AutoTokenizer, AddedToken
>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", use_fast = False, legacy = False)
>>> tokenizer.add_tokens([AddedToken("
", normalized = False, special = False), AddedToken("
", normalized = False, special = False)])
>>> tokenizer.tokenize("Hey
how
")
['▁Hey', '
', 'how', '
']
```

Hi @ArthurZucker I was not aware of this feature, thank you it now works as expected!