super weird tokenizer behaviour
Author: Hannibal046
Pull request: no

### System Info

- `transformers` version: 4.38.0
- Platform: Linux-5.15.0-1050-azure-x86_64-with-glibc2.31
- Python version: 3.9.18
- Huggingface_hub version: 0.21.1
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hi, I would like to report a super weird behaviour of Tokenizer.

Here is the code I got:
```python
from transformers import AutoTokenizer
from functools import partial
import datasets

def a_random_map_function(sample,tokenizer):
    assert tokenizer.eos_token_id in tokenizer("hello"+tokenizer.eos_token).input_ids ## wrong

if __name__ == '__main__':
    tokenizer = AutoTokenizer.from_pretrained("allenai/tulu-2-7b")
    assert tokenizer.eos_token_id in tokenizer("hello"+tokenizer.eos_token).input_ids ## right
    map_function = partial(a_random_map_function,tokenizer=tokenizer)

    dataset = datasets.load_dataset("kilt_tasks", "fever")

    lm_datasets = dataset.map(
        map_function,
        batched=False,
        num_proc=16,
    )    
```

it seems that tokenizer in the map function changed somewhere...

### Expected behavior

no assertion error

Comments:
Hey! That seems a bit hard to debug but with a print here is what I can see:

```python 
def a_random_map_function(sample,tokenizer):
    print(tokenizer("hello"+tokenizer.eos_token).input_ids)
    assert tokenizer.eos_token_id in tokenizer("hello"+tokenizer.eos_token).input_ids
```

```python
[1, 22172, 829, 29879, 29958] 
In [12]: tokenizer.decode([1, 22172, 829, 29879, 29958])
2024-03-01 02:26:11.430701: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Out[12]: '
 hello
'
```
I am pretty sur the the reason this is failing is because :
```python 
LlamaTokenizerFast(name_or_path='allenai/tulu-2-7b', vocab_size=32000, model_max_length=2048, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token': '
', 'eos_token': '
', 'unk_token': '<unk>'}, clean_up_tokenization_spaces=False),  added_tokens_decoder={
        0: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
        1: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
        2: AddedToken("
", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),
}
```
normalized is set to True.
I use `meta-llama/Llama-2-7b-hf` and it worked like a charm.
This is a duplicate of a few issues related to `normalized` in Llama so closing!