Bug: The elements of the batch contain different keys. Cannot batch them ...
Author: asusdisciple
Pull request: no
Labels: name: Usage

name: bug

name: Audio

### System Info

DGX Station
Ubuntu 20.04
Transformers 4.44.2
torch 2.4.1

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

So I ran into this bug which I did not experience in prior versions of transformers (Unfortunately I did not document which was the last one working). What I basically do in transcribe_audio() is to call different model architectures depending on their name:

```
def transcribe_audio(audiopath: List[str], model, repo_name: str, parameters: {}, lang: str) -> List[str]:
  tmp = model(audiopath,
                      generate_kwargs={"language": lang}
                      )
  
          res = [i["text"] for i in tmp]
          return res

```
The bug:

```
You have passed task=transcribe, but also have set `forced_decoder_ids` to [[1, None], [2, 50359]] which creates a conflict. `forced_decoder_ids` will be ignored in favor of task=transcribe.
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.

Traceback (most recent call last):
  File "/raid/.../Whisper/main.py", line 139, in <module>
    transcript = transcribe_audio(filenames, m, repo, par, lang.value)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/raid/.../Whisper/benchmark.py", line 247, in transcribe_audio
    tmp = model(audiopath,
          ^^^^^^^^^^^^^^^^
  File "/raid/.../Whisper/venv2/lib/python3.12/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 284, in __call__
    return super().__call__(inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/raid/.../Whisper/venv2/lib/python3.12/site-packages/transformers/pipelines/base.py", line 1238, in __call__
    outputs = list(final_iterator)
              ^^^^^^^^^^^^^^^^^^^^
  File "/raid/.../Whisper/venv2/lib/python3.12/site-packages/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
           ^^^^^^^^^^^^^^^^^^^
  File "/raid/.../Whisper/venv2/lib/python3.12/site-packages/transformers/pipelines/pt_utils.py", line 269, in __next__
    processed = self.infer(next(self.iterator), **self.params)
                           ^^^^^^^^^^^^^^^^^^^
  File "/raid/.../Whisper/venv2/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "/raid/.../Whisper/venv2/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 673, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/raid/.../Whisper/venv2/lib/python3.12/site-packages/torch/utils/data/_utils/fetch.py", line 43, in fetch
    return self.collate_fn(data)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/raid/.../Whisper/venv2/lib/python3.12/site-packages/transformers/pipelines/base.py", line 175, in inner
    raise ValueError(
ValueError: The elements of the batch contain different keys. Cannot batch them ({'input_features', 'is_last'} != {'num_frames', 'input_features', 'is_last'})
```

This error is only raised when I initialize a new model. So I loop through different models, whisper-v2, whisper-v3, ... and initialize a new model at the beginng of each loop. First loop with inference runs through, then in the second loop after a few inferences this bug appears. Do you have an idea what this can relate to? I just overwrite my old model object with the new one. I also empty any torch cache before the new init.

### Expected behavior

Just to work like before, looping through models and doing inference. There should not be any internal bug like this.

Comments:
cc @ylacombe 

Hello I found the root of the problem:

I used a` transformers pipeline `with a `WhisperForConditionalGeneration` model and explicitly set the `batch_size` parameter.
However when you set the` batch_size` parameter without setting` chunk_length_s `this error occurs. After setting `chunk_length_s` to some value the error did not appear anymore.

I thinks this should not be the case right? From what I understood not using the chunk_length_s parameter should result in using default sequential batching: https://huggingface.co/blog/asr-chunking

Or is this only true for the ASR pipeline? Should I use that instead of the vanilla pipeline or isnt ASR inferred by the model type anyway?

Hey @asusdisciple, thanks for opening this issue and digging in to find the root of the problem!
Do you think you could write a **small* self-sufficient and reproducible code, so that I can better identify and correct the root of the issue ?
Thanks!

For reference, can't seem to be able to reproduce, more details [here](https://github.com/huggingface/transformers/issues/33415#issuecomment-2345587578)

Hey @asusdisciple, I was able to reproduce this issue! More details [here](https://github.com/huggingface/transformers/issues/33415#issuecomment-2353433444)!

TL;DR: the bug happens when, in the same batch, there are samples that are longer and shorter than 30s, which is the fixed length on which Whisper was trained. You can already avoid this bug by passing chunk_length_s=30.0 to the pipeline, as you rightfully indicated. 

Thanks for sticking to the topic @ylacombe ! Do you think this should be fixed or addressed with a more detailed warning message?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.