`input_embeds` - `embed_pos` dimension mismatch when using `max_length` in Whisper (v3) model
Author: MorenoLaQuatra
Pull request: no

### System Info

- `transformers` version: 4.35.0
- Platform: Linux-6.2.0-36-generic-x86_64-with-glibc2.35
- Python version: 3.10.9
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.0+cu118 (True)
- Tensorflow version (GPU?): 2.13.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@sanchit-gandhi

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Basically in whisper model I think the positional embeddings do not take in charge the maximum length set externally.

Simply execute the following script:

```python
import torch

from transformers import WhisperProcessor, WhisperForConditionalGeneration
from datasets import load_dataset

model_tag = "openai/whisper-large-v3"
device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

# load model and processor
processor = WhisperProcessor.from_pretrained("openai/whisper-large-v3")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3", torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True)
model.config.forced_decoder_ids = None

model = model.to(device)

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = ds[0]["audio"]
input_features = processor(sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt", return_attention_mask=True, padding="max_length", truncation=True, max_length=5 * sample["sampling_rate"])
item = {
    "input_features": input_features.input_features.squeeze(0).to(torch_dtype),
    "attention_mask": input_features.attention_mask.squeeze(0).to(torch_dtype),
}

# move to device
item = {k: v.to(device) for k, v in item.items()}

# generate token ids
predicted_ids = model.generate(
    input_features=item["input_features"].unsqueeze(0),
    attention_mask=item["attention_mask"].unsqueeze(0),
)
# decode token ids to text
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=False)
print(transcription)
```

It returns the following error:
`RuntimeError: The size of tensor a (250) must match the size of tensor b (1500) at non-singleton dimension 1`

How to solve:
Change [this line](https://github.com/huggingface/transformers/blob/eb30a49b2028f2411514c10e432792ad581fc08b/src/transformers/models/whisper/modeling_whisper.py#L1123C22-L1123C22) with this:

```python
embed_pos = self.embed_positions.weight[:inputs_embeds.size(1), :]
```

I don't know if it is general enough tho.




### Expected behavior

Should do the model inference and print the transcription. 

Comments:
Hello, sorry for the late reply, this is a duplicate of #25744! 🤗 

The Whisper model is trained on a context of fixed size (30 seconds = 1500 spectrogram frames = default max spectrogram length). It will give you gibberish if you go beyond this, since the embeddings will be randomly initialised. If you go below this, you get sub-optimal outputs, since the model deviates from how it was trained. 

Thus, we follow the original OpenAI codebase, and always fix the max spectrogram length to 1500 frames, and expect this in the embeddings.

Got it, thanks for the explaination, I was not aware of the problem. Indeed I feel a little bit "strange" the need to pad to 30s but as @sanchit-gandhi said, it may lead to worse results. 

I'm going to close the issue, however, do you have any evidence of the degraded performance for <30s unpadded audio?

If someone is actually asking, those are some results on a subset of 1000 samples taken from CV 13 test set in English. This is the `openai/whisper-large-v3` model. without any special text normalization (just to have an idea). 

*No explicit set of language="en" in the generate function*:
- Max Length: 10 seconds, WER: 31.67%
- Max Length: 20 seconds, WER: 29.48%
- Max Length: 30 seconds, WER: 24.94%

*Explicitly setting language='en' in the generate function*:
- Max Length: 10 seconds, WER: 18.54%
- Max Length: 20 seconds, WER: 17.69%
- Max Length: 30 seconds, WER: 17.47%

That's super interesting that the language detection of `large-v3` is so off! I wonder whether general inference pipelines would benefit from having a more accurate language detection as a pre-processing stage, then passing the detected language and audio to Whisper for transcription? E.g. we could use https://huggingface.co/facebook/mms-lid-126

Interesting to see a near 1% WER degradation for a smaller context window -> this does suggest that we should keep it pinned to 30s to prevent 'silent errors' creeping in if users aren't aware the model has a fixed context length.

Thanks so much for the results @MorenoLaQuatra! A very interesting discussion :)

Thank you for stimulating the discussion. Language induction is indeed something I'm investigating in my research 🚀