Make whisper-event checkpoints compliant to support `return_timestamp`
Author: Vaibhavs10
Pull request: no

### System Info

- `transformers` version: 4.27.0.dev0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 1.13.1+cu116 (False)
- Tensorflow version (GPU?): 2.11.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

### Who can help?

@sanchit-gandhi @ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Inferring a Whisper checkpoint fine-tuned before the `TimestampstampProcessor` was introduced into transformers returns a rather un-informed error message `AttributeError: 'GenerationConfig' object has no attribute 'no_timestamps_token_id'`

Minimum steps to reproduce this:
```python
from transformers.pipelines import AutomaticSpeechRecognitionPipeline, pipeline
from datasets import load_dataset

cv11 = load_dataset("mozilla-foundation/common_voice_11_0", "hi", split="test", streaming=True)

pipe = pipeline(model="sanchit-gandhi/whisper-small-hi", return_timestamps=True)

test_sample = {"raw": next(iter(cv11))["audio"]["array"],
"sampling_rate": next(iter(cv11))["audio"]["sampling_rate"]}

pipe(test_sample)
```

Colab/ notebook: [here](https://github.com/Vaibhavs10/scratchpad/blob/main/pipeline_backward_compatability_test.ipynb)

The above snippet throws an error as mentioned above. This problem effects the majority (727) of the checkpoints fine-tuned during the Whisper Event.

P.S. This has been reported by multiple community members, so not just me.

### Expected behavior

We should ideally make the `return_timestamp` functionality backwards compatible or throw a more informative error message.

Sorry if there already is a way to do this and I am just misinformed. 

Comments:
Well if you are using `return_timestamps = True` you are asking for it 😅
This functionality was introduced after. Let's tell our users that they have to set it in the Generation config (when we pop). Otherwise the `generate` function should be able to set a default value if multilingual or not

Hey hey! - Sorry I did not do a good job at explaining the intent. For a typical developer who doesn't have any clue of how these checkpoints were fine-tuned and just wants to use a checkpoint on the hub for downstream inference only, this poses a challenge.

For them, they'd typically just take a checkpoint throw it into the pipe and expect the pipeline to do its magic - transcribe and provide the timestamps.

So my ask here is the following:
1. Is there a way to make the checkpoints trained during the Whisper event compliant with the most recent changes?
2. Can we add a more informative Error message so that an average developer knows what to do next?

IMO point 1 is really important as our library of fine-tuned models is one of the distinguishing factors for us. It'd be less than ideal if we ask the community to have to fine-tune their checkpoints again to be able to get timestamps.

Hope this makes more sense!

For 1. I think we can open a PR on all of the whisper models that are from the event to add the required generation config WDYT? 
2. This can of course be done on either `generate` in whisper modelling or in the logits processor!

Makes a lot of sense thanks for reporting! 👍🏻 


> 1. I think we can open a PR on all of the whisper models that are from the event to add the required generation config WDYT?

Just to be clear, if I add the `no_timestamps_token_id` to config, it would work with timestamps with re-finetuning?

The model should already be able to produce timestamps without finetuning (as it is knowledge from the pretrained model) but might not be as good as the original pretrained model. 
You need more than just `no_timestamps_token_id`. You have to use the `generation_config` in full that is available on the openai checkpoints. 
This is required as it is a new behaviour 

Hey @ArthurZucker -> Can you maybe provide the steps one needs to take to make the checkpoints compatible? We can then potentially run autoPR on all the Whisper checkpoints produced during the whisper-event.

You can just do something like 
```python 
from transformers import GenerationConfig, WhisperForConditionalGeneration
model = WhisperForConditionalGeneration.from_pretrained("your_pretrained_checkpoint")
generation_config = GenerationConfig.from_pretrained("openai/whisper-base") # if you are using a multilingual model
model.generation_config = generation_config
model.push_to_hub("your_pretrained_checkpoint", use_auth_token = "your_token_if_not_logged_in", create_pr = True)
```


Would it not be easier to make changes in the codebase to make it robust to the changes we made to generate (switching to generate config and adding timestamp prediction)? What we have is currently backwards breaking 🚨 and something we want to avoid

That makes sense, then I'll refrain from the Auto-PR and wait for these changes to be merged into `main`. Thank you @sanchit-gandhi & @ArthurZucker <3

The main issue is that the `generation_config.no_timestamps_token_id` is kind of linked to the model (english or not). We are lucky that all the models are multilingual, but we can't default 2 values, and breaking changes it is, but we kind of have to. 

I will add it to the `config` of whisper, will be easier to deal with that!


Edit: I think opening PR to the relevant repositories will help (easier to generate the `generation_config`. Also this is not a problem for backward compatibility, as timestamps is a new feature, and is not part of any release yet. However #21937 is indeed a problem and will be fixed by #21965. In the mean time, will also add a warning in case `return_timestamps` is used when the generation config is not properly setup, that will refer to the solution I shared here! 

please stop "fixing" things

> You can just do something like
> 
> ```python
> from transformers import GenerationConfig, WhisperForConditionalGeneration
> model = WhisperForConditionalGeneration.from_pretrained("your_pretrained_checkpoint")
> generation_config = GenerationConfig.from_pretrained("openai/whisper-base") # if you are using a multilingual model
> model.generation_config = generation_config
> model.push_to_hub("your_pretrained_checkpoint", use_auth_token = "your_token_if_not_logged_in", create_pr = True)
> ```

still cannot generate timestamp with this setting, i also checked PR #21334 . Is return_timestamps supported now? how could i use it properly?

I'm also facing the same problem, any ideas?