Whisper demo code for model + processor API is broken
Author: piotyrus
Pull request: no
Labels: name: bug

### System Info

When running a copy-paste code of '4.48.3', log mel spectogram values are not padded to the length required by the model. This is the exact error message:
```
ValueError: Whisper expects the mel input features to be of length 3000, but found 585. Make sure to pad the input mel features to 3000.
```

It appears that the demo code is missing a logic to pad the log mel spectogram values to the multiple of 3000.

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the code on CPU as in https://huggingface.co/openai/whisper-large-v3:
```
import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
from datasets import Audio, load_dataset


device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model_id = "openai/whisper-large-v3"

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True
)
model.to(device)

processor = AutoProcessor.from_pretrained(model_id)

dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
dataset = dataset.cast_column("audio", Audio(processor.feature_extractor.sampling_rate))
sample = dataset[0]["audio"]

inputs = processor(
    sample["array"],
    sampling_rate=sample["sampling_rate"],
    return_tensors="pt",
    truncation=False,
    padding="longest",
    return_attention_mask=True,
)
inputs = inputs.to(device, dtype=torch_dtype)

gen_kwargs = {
    "max_new_tokens": 448,
    "num_beams": 1,
    "condition_on_prev_tokens": False,
    "compression_ratio_threshold": 1.35,  # zlib compression ratio threshold (in token space)
    "temperature": (0.0, 0.2, 0.4, 0.6, 0.8, 1.0),
    "logprob_threshold": -1.0,
    "no_speech_threshold": 0.6,
    "return_timestamps": True,
}

pred_ids = model.generate(**inputs, **gen_kwargs)
pred_text = processor.batch_decode(pred_ids, skip_special_tokens=True, decode_with_timestamps=False)

print(pred_text)
```

### Expected behavior

correct transcription of an audio sample

Comments:
Hi @piotyrus ! Yes, that’s correct. We need to use `padding="max_length"`, which is also the default padding mode for `WhisperFeatureExtractor`, as shown below.
```python
inputs = processor(
    sample["array"],
    sampling_rate=sample["sampling_rate"],
    return_tensors="pt",
    truncation=False,
    padding="max_length",
    return_attention_mask=True,
)
```

I think there are a few more issues that popped out before I could get this sample to work:
1. padding="max_length" is necessary, as you have pointed out.
2. AutoProcessor API default feature size to 80. It should read value (config.num_mel_bins=128) based on config associated with the model_id.
3. max_length parameter needs to be passed to the processor as 30 * sampling_rate.
4. Padding occurs before log-mel spectogram calculation, which is wasteful and cumbersome. In NLP, it would be akin to extending the string before tokenization instead of appending pad_id token. Ideally, we should be able to pad in the log_mel space (max_length=3000, same as the expected input tensor dimensions of (batch_size=1, feature_size=128, sequence_length=3000).
5. "max_new_tokens" needs to be reduced to 445, to account for 3 prompt tokens. Total prompt + generated tokens are limited to config.max_length=448 tokens

Overall, here's a diff that works for me:
```diff
$ diff whisper_demo_hf.py whisper_demo_modified.py 
16c16
< processor = AutoProcessor.from_pretrained(model_id)
---
> processor = AutoProcessor.from_pretrained(model_id, feature_size=model.config.num_mel_bins)
21a22
> audio_seconds=model.config.max_source_positions * model.model.encoder.conv1.stride[0] * model.model.encoder.conv2.stride[0] // 100
27c28,29
<     padding="longest",
---
>     padding="max_length",
>     max_length=audio_seconds*sample["sampling_rate"],
33c35
<     "max_new_tokens": 448,
---
>     "max_new_tokens": 445,

```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.