Inconsistent tokenization across different runs (`WhisperTokenizerFast`)
Author: xenova
Pull request: no

### System Info

- `transformers` version: 4.34.0.dev0
- Platform: Linux-5.15.109+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu118 (False)
- Tensorflow version (GPU?): 2.12.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.2 (cpu)
- Jax version: 0.4.14
- JaxLib version: 0.4.14
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker @Narsil 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Running this code
```py
from transformers import AutoTokenizer
# Load first tokenizer
a=AutoTokenizer.from_pretrained('NbAiLab/nb-whisper-large-beta')

# Save and load first tokenizer again
a.save_pretrained('temp')
b=AutoTokenizer.from_pretrained('temp')

# Load second tokenizer (NOTE: this is just a `.save_pretrained()` of `NbAiLab/nb-whisper-large-beta`)
c=AutoTokenizer.from_pretrained('Xenova/nb-whisper-large-beta')

# Save and load second tokenizer again
c.save_pretrained('temp2')
d=AutoTokenizer.from_pretrained('temp2')

# All these should output the same
a("Hello World"), b("Hello World"), c("Hello World"), c("Hello World")
```

Produces many different outputs, for example:
1. Run 1
```
({'input_ids': [50258, 50288, 50359, 50363, 2425, 3937, 50257], 'attention_mask': [1, 1, 1, 1, 1, 1, 1]},
 {'input_ids': [50258, 50288, 50359, 50363, 2425, 26363, 348, 50257], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]},
 {'input_ids': [50258, 50288, 50359, 50363, 2425, 3937, 50257], 'attention_mask': [1, 1, 1, 1, 1, 1, 1]},
 {'input_ids': [50258, 50288, 50359, 50363, 2425, 26363, 348, 50257], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]})
```
2. Run 2
```
({'input_ids': [50258, 50288, 50359, 50363, 2425, 3937, 50257], 'attention_mask': [1, 1, 1, 1, 1, 1, 1]},
 {'input_ids': [50258, 50288, 50359, 50363, 2425, 3937, 50257], 'attention_mask': [1, 1, 1, 1, 1, 1, 1]},
 {'input_ids': [50258, 50288, 50359, 50363, 2425, 26363, 348, 50257], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]},
 {'input_ids': [50258, 50288, 50359, 50363, 2425, 3937, 50257], 'attention_mask': [1, 1, 1, 1, 1, 1, 1]})
```

Video:


https://github.com/huggingface/transformers/assets/26504141/c85b26e3-8436-4352-8ca7-dade32c61c99





### Expected behavior

The tokenization should be consistent across runs.

Comments:
This model uses `dropout: 0.1` 

This means that by default it will ignore merges 10% of the time, leading to different tokenization.
You can ignore the dropout by setting it to 0. If the model was trained with it though it's expected that sometimes it should see different tokens, therefore the generations shouldn't change much (at least in theory)

> This model uses dropout: 0.1

Ahh I somehow missed that 🙄 What's the recommended way to set it to 0? I don't immediately see any notes about this in the [docs](https://huggingface.co/docs/transformers/v4.33.0/en/model_doc/whisper#transformers.WhisperTokenizerFast). I've also tried adding `dropout=0` as a kwargs in `from_pretrained` but that didn't work.

After a bit of digging I found you can override it with `tokenizer.backend_tokenizer.model.dropout=0`. Closing the issue!

Thanks for digging!