Whisper model max_length doesn't work as expected
Author: RohitMidha23
Pull request: no

### System Info

- `transformers` version: 4.36.0.dev0
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+cu118 (True)
- Tensorflow version (GPU?): 2.14.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (gpu)
- Jax version: 0.4.20
- JaxLib version: 0.4.20
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@sanchit-gandhi 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import WhisperForConditionalGeneration

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3")
max_label_length = model.config.max_length
print(max_label_length)
```

Output: 20 

This same for `openai/whisper-small` returns 448. 

As suggested [here](https://discuss.huggingface.co/t/open-to-the-community-whisper-fine-tuning-event/26681/21) by you, we should ignore all the samples where length is >= max_length. 

This is not viable in the case of `openai/whisper-large-v3` as lot of transcripts are going to be > 20 tokens. 

[Here](https://github.com/huggingface/datasets/issues/5391#issuecomment-1372180962) you've mentioned that we can change the max_length of the model. 

I've tried that as well. 
```
model.generation_config.max_length = 1024
```
The issue comes when we have transcripts that are longer than 448 tokens. We get an error that says: 
```
The size of tensor a (921) must match the size of tensor b (448) at non-singleton dimension 1
```

**NOTE** The rest of the code follows from the Whisper Fine Tuning Blog post.

### Expected behavior

1. Ignoring the transcripts > 448 seems like a hacky solution. 
2. Setting the `max_length` of the `config` of the model *and* the `generation_config` don't actually seem to change anything while training. 

Is there a workaround to this? What are we to do with transcripts that are longer than 448 tokens? 

If this is not the expected behaviour and I'm missing something, please mark the same. 

Thanks!

Comments:
@sanchit-gandhi @sgugger @ArthurZucker @patrickvonplaten  any update on this? 

@RohitMidha23, please provide a reproducible code snippet so that we can reproduce this bug? 

Hey @RohitMidha23 - the `max_length` attribute is not stored in the Whisper config on the HF Hub: https://huggingface.co/openai/whisper-large-v3/blob/9ee8ba8d7e3eb15fafd5be628011c3399f85a2de/config.json#L36

That means when we try to access the `max_length`, we get the *default* max length of 20 tokens: https://github.com/huggingface/transformers/blob/ee292615559834ae2ba5b3aae3abe3f54bc81ac2/src/transformers/configuration_utils.py#L287

The correct max length is 448 tokens, which I've opened a PR to update: https://huggingface.co/openai/whisper-large-v3/discussions/38

Note that the current recommended method for storing/accessing generation related parameters is in the generation config. Here, you'll find that the max length is indeed set correctly:

```python
from transformers import WhisperForConditionalGeneration

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3")
print(model.generation_config.max_length)
```
**Print Output:**
```
448
```

=> going forwards, it is recommended that you prioritise the values in the generation config over the config for generation parameters. The PR I've opened to update the config is purely for backwards compatibility.

Note that you cannot train Whisper on sequence longer than 448 tokens. This is because Whisper has a maximum of 448 positional embeddings in the decoder, and so a hard-coded max length of 448. You should filter any tokens longer than this in your training script (_c.f._ https://discuss.huggingface.co/t/open-to-the-community-whisper-fine-tuning-event/26681/21?u=sanchit-gandhi and https://github.com/huggingface/distil-whisper/blob/914dcdf3919552d5a3826a9d5db99b059ddcc16e/training/run_distillation.py#L1189)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@sanchit-gandhi despite implementing your suggestion [here](https://discuss.huggingface.co/t/trainer-runtimeerror-the-size-of-tensor-a-462-must-match-the-size-of-tensor-b-448-at-non-singleton-dimension-1/26010/2) in the whisper finetuning code the max_length error still isnt going away. 
i still encounter crashes with the error:
```
The size of tensor a (450) must match the size of tensor b (448) at non-singleton dimension 1
```
i am using the latest release of torch, datasets and transformers

the following is my code for reference:

```
def prepare_dataset(batch):
        # process audio
        sample = batch[audio_column_name]
        inputs = feature_extractor(
            sample["array"], sampling_rate=sample["sampling_rate"], return_attention_mask=forward_attention_mask
            )
        # process audio length
        batch[model_input_name] = inputs.get(model_input_name)[0]
        batch["input_length"] = len(sample["array"])
        if forward_attention_mask:
            batch["attention_mask"] = inputs.get("attention_mask")[0]

        # process targets
        input_str = batch[text_column_name].lower() if do_lower_case else batch[text_column_name]
        batch["labels"] = tokenizer(input_str).input_ids
        batch["labels_length"] = len(tokenizer(batch["sentence"], add_special_tokens=False).input_ids)
        return batch

    with training_args.main_process_first(desc="dataset map pre-processing"):
        vectorized_datasets = raw_datasets.map(
            prepare_dataset,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=next(iter(raw_datasets.values())).column_names,
            desc="preprocess train dataset",
        )

    # filter data that is shorter than min_input_length or longer than
    # max_input_length
    def is_audio_in_length_range(length):
        return length > min_input_length and length < max_input_length
    
    def filter_labels(labels_length):
        """Filter label sequences longer than max length (448)"""
        return labels_length < 448
    
    vectorized_datasets = vectorized_datasets.filter(
        is_audio_in_length_range,
        num_proc=1,#num_proc=data_args.preprocessing_num_workers,
        input_columns=["input_length"],
        # remove_columns=next(iter(raw_datasets.values())).column_names
    )
    vectorized_datasets = vectorized_datasets.filter(
        filter_labels,
        num_proc=1,#num_proc=data_args.preprocessing_num_workers,
        input_columns=["labels_length"],
        # remove_columns=next(iter(raw_datasets.values())).column_names
```