Add WhisperTokenizerFast
Author: jonatanklosko
Pull request: yes

Adds the fast version of Whisper tokenizer. The Whisper tokenizer is essentially GPT2 tokenizer with special tokens. The main difference is the additional normalizer (which I mirrored from the slow tokenizer) and language/task-dependent prefix tokens.

One of the tokenizer tests is failing, it's because there is no `tokenizer.json` file in the `openai/whisper-*` (specifically the `tiny` checkpoint). I added a converter, so now it is possible to load fast tokenizer from existing checkpoints and export `tokenizer.json`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ArthurZucker thanks for the help! I think now the steps are to update the unknown token in multilangual checkpoints and add `tokenizer.json` to the repos. Let me know if there's anything I can help with :)

Feel free to open community PR on the model' (hub) linking to this PR (github) 🚀 

@ArthurZucker sure! I've just created https://huggingface.co/openai/whisper-tiny/discussions/5, let me know if it looks as expected and I will open a matching PR on the other checkpoints too.

FTR I generated the `tokenizer.json` with:

```python
import sys
sys.path.reverse()
sys.path.append("/Users/jonatanklosko/git/transformers/src")
sys.path.reverse()

from transformers import WhisperTokenizerFast

tokenizer = WhisperTokenizerFast.from_pretrained("/Users/jonatanklosko/git/hf/whisper-tiny/")
tokenizer.save_pretrained("/Users/jonatanklosko/git/hf/whisper-tiny/")
```

I also updated the unknown token configuration manually.

Changing the unknown token in configuration leads to a weird behaviour when loading the slow tokenizer, see an example in the PR. Any ideas why that is?

So the issue is that the multilingual tokenizer doesn't have `
` in the initial vocabulary, so it would need to be added from special tokens map. However, when loading special tokens we have this check:

https://github.com/huggingface/transformers/blob/7119bb052a3f492b9af3afe4f3f13132445eba6e/src/transformers/tokenization_utils.py#L419-L420

and since `eos_token` and `unk_token` are both `
`, we end up not adding them to the vocabulary.

To address this we would need to add `"
": 50257` to `vocab.json` and remove it from `added_tokens.json`. Note that this is the case in the English checkpoints (except with 50256).

The question is if this hurts compatibility; when loading the slow tokenizer both of these files would be used to load the vocabulary, so moving the entry from one to the other should be alright?

Yep, I think the idea is to make the multilingual added tokens match the ones that we have for english. I forgot to mention but yes, we have to add `"
` to the vocabulary instead of `''`. This should normally do the trick (with also the modification of the content of the unknown token. 

Ah, so we should actually replace it, so that `
` gets the id that currently `""` has, and we keep `""` just to make sure the ids are not shifted at any point?

```
"
": 50256,
"": 50257,
```

and not:

```
"": 50256,
"
": 50257,
```

@ArthurZucker I updated the PR on the checkpoint. I tried the remaining failing tests locally pointing tokenizer to the updated revision and they passed, so I think we are good on this side.

Note that the only difference is that originally EOS (`
`) was 50257 and now it is 50256, not sure if that's something to worry about.

The EOS toke id appears multiple times in the `config.json` so we need to adjust it too. Let me know if that's the way to go, or if we should swap them back :)

> Note that the only difference is that originally EOS (
) was 50257 and now it is 50256, not sure if that's something to worry about.

Ah, this can be an issue I think. We have to keep it at 50257! So let's leave `''` in the vocab (it is also in the original repo) and we just need `{"
": 50257}` this to be in the `added_special_tokens`. See [this repo](https://github.com/openai/whisper/tree/main/whisper/assets/multilingual) which contains most of what we need 

@ArthurZucker we need `
` in the `vocab` rather than `added_tokens` as per https://github.com/huggingface/transformers/pull/21222#issuecomment-1401119817.

Note that this means unknown token changes from 50256 to 50257, but hopefully that's less invasive.

Yeah! That's better

Ok, so I think the `openai/whisper-tiny` PR ready too, if there's anything else let me know :)

I merged your PR on the hub, now let's fix the failing tests! 

@ArthurZucker all green!

Will ask for a final review from @sgugger 

@ArthurZucker it looks like the new failures come from the GenerationConfig missing some attributes, also looking at `openai/whisper-tiny` the `forced_decoder_ids` have a `null` token and don't match what we have in `config.json`.

Hey, `null` token is fine! I added that for the refactoring, it allows the model to automatically predict the language

OKay the error comes from the `tiny_random_testing` where configuration files are created from the config, and thus don't have any of the parameters related to generation. The `return_timestamps` is set to `True` but it should not if there are not generation config. 
Feel free to skip these tests for now, unless @ydshieh you have an alternative solution

> OKay the error comes from the `tiny_random_testing` where configuration files are created from the config, and thus don't have any of the parameters related to generation. The `return_timestamps` is set to `True` but it should not if there are not generation config. Feel free to skip these tests for now, unless @ydshieh you have an alternative solution

The CI is currently running and I can't see which test you are mentioning. I will check later once the CI results is available.

PRs for other checkpoints:

* [whisper-tiny.en](https://huggingface.co/openai/whisper-tiny.en/discussions/10)
* [whisper-small.en](https://huggingface.co/openai/whisper-small.en/discussions/6)
* [whisper-base.en](https://huggingface.co/openai/whisper-base.en/discussions/5)
* [whisper-medium.en](https://huggingface.co/openai/whisper-medium.en/discussions/5)
* [whisper-small](https://huggingface.co/openai/whisper-small/discussions/11)
* [whisper-base](https://huggingface.co/openai/whisper-base/discussions/7)
* [whisper-medium](https://huggingface.co/openai/whisper-medium/discussions/7)
* [whisper-large](https://huggingface.co/openai/whisper-large/discussions/20)

Hey @ydshieh, the tests are aforementioned tests are not skipped, but you can see the previous CI failure [here](https://app.circleci.com/pipelines/github/huggingface/transformers/56124/workflows/d31aa74b-175d-4c79-a237-cd342ded9900/jobs/677380).

Hi, @jonatanklosko  could you rebase on main branch? You will need to resolve the conflicts. Let me know if you need help on this. Sorry for being late here.

@jonatanklosko Thank you. I will take a look on Monday if the pipeline testing is still failing!

@ydaigo perfect, thanks :)

Hey @jonatanklosko can you rebase on main to or resolve the merge conflicts?

@ArthurZucker done and everything passes now :)