Slow tokenizer decode
Author: peregilk
Pull request: no

### System Info

transformers 4.34.0.dev0. Running this on tpu v4-8. Might happen on other plattforms as well.

### Who can help?

@ArthurZucker

### Reproduction

Decoding is extremely slow using Transformers 4.34.0.dev0.

A small script to reproduce:
```
import argparse, time
from transformers import AutoTokenizer

def measure_tokenization_speed(tokenizer, sentences):
    start_time = time.time()
    outputs = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt")
    end_time = time.time()
    print(f"Time taken for encoding: {end_time - start_time} seconds")
    return outputs["input_ids"]

def measure_detokenization_speed(tokenizer, input_ids):
    start_time = time.time()
    decoded_sentences = tokenizer.batch_decode(input_ids)
    end_time = time.time()
    print(f"Time taken for decoding: {end_time - start_time} seconds")

def main(args):
    tokenizer = AutoTokenizer.from_pretrained("openai/whisper-medium", use_fast=True)

    # Create an array of 1000 sentences
    sentences = ["This is a sample sentence."] * 1000

    input_ids = measure_tokenization_speed(tokenizer, sentences)
    measure_detokenization_speed(tokenizer, input_ids)

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Measure the speed of HuggingFace tokenizer.")
    args = parser.parse_args()
    main(args)
```
tpu v4-8 (transformers 4.34.0.dev0)
Time taken for encoding: 1.1659502983093262 seconds
Time taken for decoding: 39.807389974594116 seconds

tpu v4-8 (transformers 4.30.1)
Time taken for encoding: 1.2527313232421875 seconds
Time taken for decoding: 1.8215229511260986 seconds


### Expected behavior

Decoder should take approximately as long as encoding.

Comments:
Hey! That is indeed a long decoding time! I believe that #26299 fixes this! 

Running on a local MacBook with `transformers` on main now that https://github.com/huggingface/transformers/pull/26299 is merged:
```
Time taken for encoding: 0.6956140995025635 seconds
Time taken for decoding: 0.010346174240112305 seconds
```
Looks like we've recovered performance! Hope this is good for you @peregilk.

Yes. It absolutely did!

I think a related issue came back. I'm facing same problem using transformers==4.34 and tokenizers==0.14.1. With transformers=4.33 (and tokenizers==0.13.3) the decoding step is 4-5x times faster.

Hey @augustomaillo - thanks for reporting! Could you confirm what the speed is decoding using the latest version of the library? There have been a number of changes to `tokenizers` and the Whisper tokenizer since v4.34:
```bash
pip install --upgrade transformers tokenizers
```

Thanks!

@sanchit-gandhi I've tested several `tokenizers` versions, but decoding time was still slow. I ended up using `WhisperTokenizerFast` class to decode. The observed speed up is huge and my problems are solved. Is there some big difference between `WhisperTokenizer` and `WhisperTokenizerFast`? If not, why keep both?


I think you have a few misconception: 
- `tokenizers` is unrelated to `WhisperTokenizer`
- `transformers` version should be the latest as mentioned by @sanchit-gandhi 

Even in `transformers` latest version `WhisperTokenizer` is very slow.

That can be an issue 😉 


I re-ran the provided script (in the issue) and here is what I got:
```python
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 283k/283k [00:00<00:00, 1.74MB/s]
vocab.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 836k/836k [00:00<00:00, 10.5MB/s]
tokenizer.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.48M/2.48M [00:00<00:00, 8.57MB/s]
merges.txt: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 494k/494k [00:00<00:00, 33.8MB/s]
normalizer.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 52.7k/52.7k [00:00<00:00, 629kB/s]
added_tokens.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 34.6k/34.6k [00:00<00:00, 36.6MB/s]
special_tokens_map.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.19k/2.19k [00:00<00:00, 7.08MB/s]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Time taken for encoding: 0.006536960601806641 seconds
Time taken for decoding: 0.008824825286865234 seconds
```
which is pretty good. 
It's not "slow" but it's expected to be slower than parallelised tokenizers 


Do you have a reproducer? 

I was using `WhisperTokenizer` class, without `use_fast=True`. But in older versions it was fast. I can reproduce using the code provided in the issue with `use_fast=False`

`tokenizer = AutoTokenizer.from_pretrained("openai/whisper-medium", use_fast=False)`

In `transformers==4.33.0` I got:

```
Time taken for encoding: 1.7099077701568604 seconds
Time taken for decoding: 0.016812801361083984 seconds
```

Which is ok for decoding (my goal). When I updated transformers to latest version (4.40.1), the same script (`use_fast=False`) gives me:

```
Time taken for encoding: 0.05837678909301758 seconds
Time taken for decoding: 3.31003475189209 seconds
```

In this latest version, changing to `use_fast=True` I got:

```
Time taken for encoding: 0.04278135299682617 seconds
Time taken for decoding: 0.01867818832397461 seconds
```
Which is fast as the default behavior in 4.33.  

Looking at these values it **don't seem like an issue** properly, but a behavioral change as the library evolved, problably well described in documentation updates. 
Thank you for the help!

Mmm I am not sure, but 3sec for decoding looks really slow. We can dig a bit I think, can potentially be related to the many added tokens. 
I won't have the bandwidth to dig as the fast tokenizer is working as expected,