behaviour of Tokenizer
Author: Hannibal046
Pull request: no

### System Info

- `transformers` version: 4.38.0
- Platform: Linux-5.15.0-1050-azure-x86_64-with-glibc2.31
- Python version: 3.9.18
- Huggingface_hub version: 0.21.1
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hi, @ArthurZucker 

This issue is related to https://github.com/huggingface/transformers/issues/29375

```python
from  transformers import AutoTokenizer
from functools import partial
import datasets

def a_random_map_function(sample,tokenizer):
    print(tokenizer("hello"+tokenizer.eos_token).input_ids)
    assert tokenizer.eos_token_id in tokenizer("hello"+tokenizer.eos_token).input_ids

if __name__ == '__main__':
    tokenizer = AutoTokenizer.from_pretrained("allenai/tulu-2-7b")
    print(tokenizer("hello"+tokenizer.eos_token).input_ids)
    assert tokenizer.eos_token_id in tokenizer("hello"+tokenizer.eos_token).input_ids
    map_function = partial(a_random_map_function,tokenizer=tokenizer)

    dataset = datasets.load_dataset("kilt_tasks", "fever")

    lm_datasets = dataset.map(
        map_function,
        batched=False,
        num_proc=16,
    )    
```

Sorry for the unclear description in that issue. But the problem I want to report is not **normalization** problem, but the **inconsistency** behaviour after the `map` function. As you can see here:
![image](https://github.com/huggingface/transformers/assets/38466901/4dec5998-293b-4071-b52a-f89336ff370b)


### Expected behavior

Behaviour are consistent all the time.

Comments:
BTW,  changing from `allenai/tulu-2-7b` to `meta-llama/llama-2-7b-hf`. It works. But it is weird for `allenai/tulu-2-7b` because (1) it is essentially an instruction-tuned llama-2 (2) it behaves normally under the `__main__`.

Hey! I think that this issue is related to parallelism. Some interaction with the underlying `tokenizers` rather than transformers. As I mentioned, if you use `AutoTokenizer.from_pretrained("...", use_fast=False)` It works. 
If you also add the tokens as non-normalized, it will work. 

But this might not be related to the tokenizer but to the way it is used? cc @narsil I am not super familiar with the underlying ray but recommended usage is not to wrap around partial no? 

Hi, thanks for response.
After further investigation, I confirm that this phenomenon would only appear when:
1. using fast tokenizer
2. num_proc > 1 when using datasets.map

The fast tokenizer has it's own parallelism code which is probably the source of this

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.