[Phi-3-mini-128k-instruct] Difference in slow and fast tokenization after adding new tokens
Author: jpmann
Pull request: no

### System Info

Transformers versions: 4.37.2, 4.40.1

### Who can help?

@ArthurZucker @youn

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

> Code
```python3
import transformers
print(f"Tranformers version: {transformers.__version__}")

from transformers import AutoTokenizer

model_name = "microsoft/Phi-3-mini-128k-instruct"
new_tokens_to_add = ["token_1"]

print("FAST TOKENIZER OUTPUT")
fast_tokenizer = AutoTokenizer.from_pretrained(model_name)
# Uncomment below line to verify the type of tokenizer
# print(type(fast_tokenizer))
fast_tokenizer.add_tokens(new_tokens_to_add)  # not adding them as special tokens
# token_1 is split into multiple sub tokens
enc_tokens = fast_tokenizer.encode("\ntoken_1", add_special_tokens=False)
print(enc_tokens)
# decode the tokens
print(repr(fast_tokenizer.decode(enc_tokens)))

print("\nSLOW_TOKENIZER_OUTPUT")
slow_tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)
# Uncomment below line to verify the type of tokenizer
# print(type(slow_tokenizer))
slow_tokenizer.add_tokens(new_tokens_to_add)  # not adding them as special tokens
# token_1 is NOT split into multiple sub tokens
enc_tokens = slow_tokenizer.encode("\ntoken_1", add_special_tokens=False)
print(enc_tokens)
# decode the tokens
print(repr(slow_tokenizer.decode(enc_tokens)))
```

>Output of the above code snippet
```
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Tranformers version: 4.40.1
FAST TOKENIZER OUTPUT
[29871, 13, 6979, 29918, 29896]
'\ntoken_1'

SLOW_TOKENIZER_OUTPUT
[29871, 13, 32011]
'\n token_1'
```

### Expected behavior

As can be seen in the above code output, the `fast tokenizer` is sub-splitting the token in the vocabulary which is not intuitive whereas the `slow tokenizer` is working correctly.

Comments:
cc @ArthurZucker 

Hey! 
Printing the tokenizer shows:
```python3
LlamaTokenizer(name_or_path='microsoft/Phi-3-mini-128k-instruct', vocab_size=32000, model_max_length=131072, is_fast=False, padding_side='left', truncation_side='right', special_tokens={'bos_token': '
', 'eos_token': '
', 'unk_token': '<unk>', 'pad_token': '
'}, clean_up_tokenization_spaces=False),  added_tokens_decoder={
.........
        32010: AddedToken("
", rstrip=True, lstrip=False, single_word=False, normalized=False, special=True),
        32011: AddedToken("token_1", rstrip=False, lstrip=False, single_word=False, normalized=True, special=False),
}
```
your token was added with normalization.

You are also not using the `from_slow` flag:
```python 
fast_tokenizer = AutoTokenizer.from_pretrained(model_name, from_slow=True)
fast_tokenizer.add_tokens(new_tokens_to_add)
fast_tokenizer.encode("\ntoken_1", add_special_tokens=False)
```

closing as it was fixed by #28881 and has a lot of similar issues