[Whisper] Index error with model weights like base, large-v2
Author: kurianbenoy-sentient
Pull request: no

### System Info

- `transformers` version: 4.27.0.dev0
- Platform: Linux-5.4.0-66-generic-x86_64-with-glibc2.31
- Python version: 3.10.9
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 1.13.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help?

@sanchit-gandhi @Narsil 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. I have been using Whisper from transformers library with my own custom script. The full code is attached below for reference:

```
import argparse
import base64
import os
import tempfile

import torch
from transformers import (
    AutoFeatureExtractor,
    AutoTokenizer,
    WhisperForConditionalGeneration,
    WhisperProcessor,
    pipeline,
)
from utils.b64 import convert_to_base64

model_dir = "openai/whisper-base"


class Whisper:
    def __init__(self):
        self.tokenizer = AutoTokenizer.from_pretrained(model_dir)
        self.feature_extractor = AutoFeatureExtractor.from_pretrained(model_dir)
        self.processor = WhisperProcessor.from_pretrained(model_dir)
        self.device = "cuda" if torch.cuda.is_available() else "cpu"
        # print([x for x in Path(model_dir).iterdir()])
        self.model = WhisperForConditionalGeneration.from_pretrained(model_dir).to(self.device)
        self.is_timestamp = False

    def predict_raw(self, payload):
            if payload is None:
                return {"inputerror": "JSON expected"}

            if "wav_base64" not in payload:
                return {"inputerror": "Missing key wav_base64 in payload."}


            afs = payload["wav_base64"]
            if not isinstance(afs, str):
                return {"inputerror": "Audio file to passed as input in base64 format"}

            if "timestamps" not in payload:
                timestamp = self.is_timestamp
            elif "timestamps" in payload and type(payload["timestamps"]) != str:
                return {"inputerror": "timestamps should be of string datatype"}
            elif "timestamps" in payload and payload ["timestamps"] != "true":
                return {"inputerror": "timestamps payload should be of Value True"}
            else:
                timestamp = True


            lang = payload.get("language")
            print(lang)
            afs = base64.b64decode(afs)

            dno = torch.cuda.current_device() if self.device == "cuda" else -1
            with tempfile.NamedTemporaryFile() as a_file:
                a_file.write(afs)
                pipe = pipeline(
                    task="automatic-speech-recognition",
                    model=self.model,
                    tokenizer=self.tokenizer,
                    feature_extractor=self.feature_extractor,
                    framework="pt",
                    chunk_length_s=30,
                    generate_kwargs={"max_new_tokens": 1024},
                    device=dno,
                    return_timestamps=timestamp
                )
                if lang:
                    self.model.config.forced_decoder_ids = self.processor.get_decoder_prompt_ids(
                        task="transcribe", language=lang
                    )
                if timestamp:
                    results = pipe(a_file.name)
                    timestamp_info = [{"text": x["text"], "start": x["timestamp"][0],"end": x["timestamp"][1]} for x in results["chunks"]]
                    return {"text": results["text"], "timestamps": timestamp_info}
                else:
                    return pipe(a_file.name)
                    
 
if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "audio_file",
        type=str,
        help="Input the path to audio file you want to transcribe",
    )
    parser.add_argument(
        "--language",
        type=str,
        help="Input the language",
    )
    args = parser.parse_args()
    kwargs = vars(args)
    b64 = convert_to_base64(kwargs["audio_file"])
    if kwargs["language"]:
        payload = {"wav_base64": b64, "language": kwargs["language"]}
    payload = {"wav_base64": b64, "timestamps": "true"}
    model = Whisper()
    print(model.predict_raw(payload))
    payload = {"wav_base64": b64}
    print(model.predict_raw(payload))
```

2. When I am using model weights of `base` and `large-v2`. The code is getting an error as below:

```
Traceback (most recent call last):
  File "/root/whisper_project/microservice-ai-whisper/deploy/debug.py", line 102, in <module>
    print(model.predict_raw(payload))
  File "/root/whisper_project/microservice-ai-whisper/deploy/debug.py", line 75, in predict_raw
    results = pipe(a_file.name)
  File "/root/mambaforge/lib/python3.10/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 272, in __call__
    return super().__call__(inputs, **kwargs)
  File "/root/mambaforge/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1101, in __call__
    return next(
  File "/root/mambaforge/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
  File "/root/mambaforge/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 266, in __next__
    processed = self.infer(next(self.iterator), **self.params)
  File "/root/mambaforge/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1015, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/root/mambaforge/lib/python3.10/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 445, in _forward
    tokens = self.model.generate(
  File "/root/mambaforge/lib/python3.10/site-packages/transformers/models/whisper/modeling_whisper.py", line 1543, in generate
    return super().generate(
  File "/root/mambaforge/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/root/mambaforge/lib/python3.10/site-packages/transformers/generation/utils.py", line 1406, in generate
    return self.greedy_search(
  File "/root/mambaforge/lib/python3.10/site-packages/transformers/generation/utils.py", line 2211, in greedy_search
    next_token_logits = outputs.logits[:, -1, :]
IndexError: index -1 is out of bounds for dimension 1 with size 0
```

3. I noticed this issue only with some model weights. The exact same code works with medium and large-v1 model weights

### Expected behavior

Get the transcribed output along with corresponding timestamps

Comments:
Do you have the sample audio that triggers the bug ?
I couldn't reproduce locally.

Aslo, is there any particular reason for having such a complex script ? I can see several things that could slow down this code more than it should.

`pipe = pipeline(task="automatic-speech-recognition", model=model_dir, device=device)` should work out of the box.

Here for instance the device is set during prediction, which is fcalled repetively could slow down things a bit (once the device is set it shouldn't move afterwards. Even the pipeline creation, is not big, but still could be done ahead of time.


Cheers ! 

@Narsil my apologies. I got this error with specific weights in transformers library v4.26.1. 

You can reproduce this error with the file in the below link.

https://drive.google.com/file/d/1ubw9PLlo5NwB7xwTwfVnuRWKH8L5XPIx/view?usp=sharing

I am using whisper models for offline inference. So simply passing the pipeline directly won't work:

```
pipe = pipeline(task="automatic-speech-recognition", model=model_dir, device=device)
```

That's why I have to manually pass the pipeline:

```
pipe = pipeline(
                    task="automatic-speech-recognition",
                    model=self.model,
                    tokenizer=self.tokenizer,
                    feature_extractor=self.feature_extractor,
                    framework="pt",
                    chunk_length_s=30,
                    generate_kwargs={"max_new_tokens": 1024},
                    device=dno,
                    return_timestamps=timestamp
                )
```

Thanks for pointing the issue in device_no, I will set it in such a way it won't always be called each time when method is called. 

While in the latest version in master I am getting the error: v4.27.0-dev for all model weights, when I use the above code:

```
Traceback (most recent call last):                                                                                                                      File "/root/whisper_project/microservice-ai-whisper/deploy/Whisper.py", line 99, in <module>                                                            print(model.predict_raw(payload))        
ile "/root/whisper_project/microservice-ai-whisper/deploy/Whisper.py", line 72, in predict_raw
    results = pipe(a_file.name)                                                                                                                         File "/root/mambaforge/lib/python3.10/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 272, in __call__                      return super().__call__(inputs, **kwargs)                                                                                                           File "/root/mambaforge/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1101, in __call__                                             return next(                                                                                                                                        File "/root/mambaforge/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 124, in __next__                                          item = next(self.iterator)                                                                                                                          File "/root/mambaforge/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 266, in __next__                                          processed = self.infer(next(self.iterator), **self.params)                                                                                          File "/root/mambaforge/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1015, in forward                                              model_outputs = self._forward(model_inputs, **forward_params)                                                                                       File "/root/mambaforge/lib/python3.10/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 445, in _forward                      tokens = self.model.generate(                                                                                                                       File "/root/mambaforge/lib/python3.10/site-packages/transformers/models/whisper/modeling_whisper.py", line 1534, in generate                            logits_processor = [WhisperTimeStampLogitsProcessor(generation_config)]                                                                             File "/root/mambaforge/lib/python3.10/site-packages/transformers/generation/logits_process.py", line 935, in __init__                                   self.no_timestamps_token_id = generate_config.no_timestamps_token_id                                                                              AttributeError: 'GenerationConfig' object has no attribute 'no_timestamps_token_id'   
```

![image](https://user-images.githubusercontent.com/101088788/223036652-fa889119-fbe1-41ed-87b1-91db3cfd29ee.png)


Ok this works:

```python
from transformers import pipeline

pipe = pipeline(
    task="automatic-speech-recognition",
    model="openai/whisper-large-v2",
    chunk_length_s=30,
    device=0,
    return_timestamps=True,
)

out = pipe("sample.wav")
print(out)
```

Then you can definitely use the same simplicity with local values. Just save the pipeline

`pipe.save_pretrained("whisper-local")`

And then 
```python
from transformers import pipeline

pipe = pipeline(
    task="automatic-speech-recognition",
    model="./whisper_local",
    chunk_length_s=30,
    device=0,
    return_timestamps=True,
)
```

Should work.

No for the latest error, I'm guessing this has to do with recent changes in the configuration of whisper.
Is that possible @ArthurZucker  ?

Thank you @Narsil for techniques to reduce complexity. 

I didn't we can save pipeline also locally.

Yes, if you are using main, and want to use timestamps, you should make sure that the model has the `no_timestamp_id`. 
This is because this is a new feature, and for proper generation, we recommend setting a `generation_config`. 
Also, if you check both models, they have a different `configuration.forced_decoder_ids`.  This is probably what was causing the difference in behaviour between the `large` and `base` (for example).




@ArthurZucker how can I make sure that model has the `no_timestamp_id`?

You should save/push the update `generation_config`. 
The simplest you can do is the following: 
```python 
from transformers import GenerationConfig, WhisperForConditionalGeneration
model = WhisperForConditionalGeneration.from_pretrained("your_pretrained_checkpoint")
generation_config = GenerationConfig.from_pretrained("openai/whisper-base") # if you are using a multilingual model
model.generation_config = generation_config
model.push_to_hub("your_pretrained_checkpoint", use_auth_token = "your_token_if_not_logged_in", create_pr = True)
```

Thanks for your answer. Hope you will resolve this issue before taking it in next stable version. I am closing this issue for now.