Add WhisperModel to transformers
Author: ArthurZucker
Pull request: yes
Labels: name: New model

# What does this PR do?
Adds Whisper to transformers


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Going forward, let's check generation with:
```python
#!/usr/bin/env python3
import whisper
import jiwer
import numpy as np
import torch
from datasets import load_dataset
from transformers import WhisperForConditionalGeneration, WhisperProcessor, WhisperTokenizer
from whisper.normalizers import EnglishTextNormalizer

normalizer = EnglishTextNormalizer()


model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-base.en")
processor = WhisperProcessor.from_pretrained("openai/whisper-base.en")

device = "cuda"
model = model.to(device).eval()


def map_fn(batch):
    arrays = [x["array"] for x in batch["audio"]]

    # -> here is a bug
    input_features = processor.feature_extractor(arrays, padding="max_length", max_length=480_000, return_tensors="pt").input_features
    input_features = input_features.to(device)

    model.config.use_cache = False
    sequences = model.generate(input_features, max_length=224, forced_bos_token_id=50362, decoder_start_token_id=50257)
    results = processor.tokenizer.batch_decode(sequences, skip_special_tokens=True)

    batch["hypotheses"] = [normalizer(result) for result in results]
    batch["reference"] = [normalizer(text) for text in batch["text"]]
    return batch


ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.map(map_fn, batch_size=16, remove_columns=ds.column_names, batched=True)

wer = jiwer.wer(list(ds["reference"]), list(ds["hypotheses"]))
print("Wer", wer)
```

Failing tests are related to the `RAG` model that re-uses the generate function.

Ready I think @patrickvonplaten 

Hey @patrickvonplaten and @sgugger the PR is ready for a final review! 🤗 

Could we also add a script that runs each checkpoint in a 5-linear as discussed on Slack here?
These code snippets could then be added to the respective model cards

Okay, so here is a simple example : 

```python
>>> model = WhisperForConditionalGeneration.from_pretrained(f"openai/whisper-large")
>>> processor = WhisperProcessor.from_pretrained(f"openai/whisper-large")

>>> ds = load_dataset("common_voice", "ja", split="test", streaming=True)
>>> ds = ds.cast_column("audio", datasets.Audio(sampling_rate=16_000))
>>> ds_iter = iter(ds)
>>> input_speech = next(ds_iter)["audio"]["array"]
>>> inputs = processor(input_speech, return_tensors = "pt")

>>> predicted_ids = model.generate(**inputs)
>>> processor.tokenizer.batch_decode(predicted_ids, skip_special_tokens=True, normalize = True)[0]
'i borrowed a phone from kimura san'

>>> forced_decoder_ids = processor.get_decoder_prompt_ids(language = "ja", task = "transcribe")
>>> predicted_ids = model.generate(**inputs, forced_decoder_ids=forced_decoder_ids)
>>> processor.tokenizer.batch_decode(predicted_ids, skip_special_tokens=True)[0]
"木村さんに電話を貸してもらいました"

>>> forced_decoder_ids = processor.get_decoder_prompt_ids(language = "en", task = "transcribe")
>>> predicted_ids = model.generate(**inputs, forced_decoder_ids=forced_decoder_ids)
>>> processor.tokenizer.batch_decode(predicted_ids, skip_special_tokens=True)[0]
' Kimura san ni denwa wo kaite moraimashita'
```

2 final things:

- Add 2 tests for batched generation
- Make sure the tokenizer has a pad_token_id => it should be identical to the eos_token_id since there is no official one. We don't want to trigger a warning every time we run generation in batch
- Also make sure that `config.pad_token_id` is correctly set.

cc @sanchit-gandhi we have to remember this when doing fine-tuning experiments! Whisper has `pad_token_id == eos_token_id` which means that during training we need to make sure in our general training scripts that we don't replace the `eos_token_id` with `-100` and thus ignore it in the loss. Instead we should only replace the "not-first" pad_token_id with `-100` (we have the same for GPT2 BTW) 

Hello!
I apologize for interrupting the development process. But I'm following this thread, because I'm really looking forward to the Whisper at HF, and here I also see words about fine tuning. It will be very cool if you can make good fine tuning and code examples!

I myself am already trying to finetune in different ways, but so far the model is only being unlearned.

In any case, thanks for your work and good luck! ❤️ 

> Hello! I apologize for interrupting the development process. But I'm following this thread, because I'm really looking forward to the Whisper at HF, and here I also see words about fine tuning. It will be very cool if you can make good fine tuning and code examples!
> 
> I myself am already trying to finetune in different ways, but so far the model is only being unlearned.
> 
> In any case, thanks for your work and good luck! heart

Hey @ArtyomZemlyak, 

This is a major focus of our right now! We've already done some experiments - you can check it here: 
https://openreview.net/forum?id=9OL2fIfDLK (we've fine-tuned whisper on a bunch of open-source datasets)

We hope to have a well-functioning fine-tuning script by early next week (we plan on doing a blog post + google colab)

Merging to unblock TF PR

Awesome, sorry for the delay!