[Bug] whisper pipeline inference bug on transformers master branch
Author: WeichenXu123
Pull request: no

### System Info

OS:
ubuntu 20.04

transformer version: master branch.
`pip install git+https://github.com/huggingface/transformers`

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Run following code:

```python
import transformers
from packaging.version import Version
import pathlib


def whisper_pipeline():
    task = "automatic-speech-recognition"
    architecture = "openai/whisper-tiny"
    model = transformers.WhisperForConditionalGeneration.from_pretrained(architecture)
    tokenizer = transformers.WhisperTokenizer.from_pretrained(architecture)
    feature_extractor = transformers.WhisperFeatureExtractor.from_pretrained(architecture)
    if Version(transformers.__version__) > Version("4.30.2"):
        model.generation_config.alignment_heads = [[2, 2], [3, 0], [3, 2], [3, 3], [3, 4], [3, 5]]
    return transformers.pipeline(
        task=task, model=model, tokenizer=tokenizer, feature_extractor=feature_extractor
    )

def raw_audio_file():
    # The dataset file comes from https://github.com/mlflow/mlflow/blob/master/tests/datasets/apollo11_launch.wav
    datasets_path = "/path/to/apollo11_launch.wav"
    return pathlib.Path(datasets_path).read_bytes()


inference_config = {
    "return_timestamps": "word",
    "chunk_length_s": 60,
    "batch_size": 16,
}
whisper = whisper_pipeline()
raw_audio_file_data = raw_audio_file()
prediction = whisper(raw_audio_file_data, return_timestamps="word", chunk_length_s=60, batch_size=16)
```

The last line raises error like:

```
>>> prediction = whisper(raw_audio_file_data, return_timestamps="word", chunk_length_s=60, batch_size=16)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/weichen.xu/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 356, in __call__
    return super().__call__(inputs, **kwargs)
  File "/home/weichen.xu/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1132, in __call__
    return next(
  File "/home/weichen.xu/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 124, in __next__
    item = next(self.iterator)
  File "/home/weichen.xu/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/pipelines/pt_utils.py", line 266, in __next__
    processed = self.infer(next(self.iterator), **self.params)
  File "/home/weichen.xu/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1046, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/home/weichen.xu/miniconda3/envs/py310/lib/python3.10/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 551, in _forward
    generate_kwargs["num_frames"] = stride[0] // self.feature_extractor.hop_length
TypeError: unsupported operand type(s) for //: 'tuple' and 'int'
>>> 
```

Note this error only happens on transformer github master branch.
For released version, above code works well.




### Expected behavior

My example code should not raise error.

Comments:
CC @BenWilson2

This causes MLflow CI failure https://github.com/mlflow-automation/mlflow/actions/runs/6223078500/job/16896947205#step:13:7352

This is the commit and line that is problematic. 

https://github.com/huggingface/transformers/commit/95fe0f5d806ff1b981f1870f290a4d9aaa53a5d4#diff-ccc8e98fcaa81fdf6317a652438a309bcede0bbe336774288c2fcf91d9f11082R551

The `stride[0]` object is a tuple, not an int. Should this be `stride[0][0]`?

CC @xenova @ArthurZucker

Thanks for the ping. My hunch is that this is due to `batch_size` being larger than 1. Just to confirm, does the same thing happen if you remove that argument?

> Thanks for the ping. My hunch is that this is due to `batch_size` being larger than 1. Just to confirm, does the same thing happen if you remove that argument?

Yes It only happens when batch > 1

Hi, I have the same issue (it was not happening before) is there any solution / workaround? This is how i use the pipeline:
`return self.model(filepath,
                            return_timestamps="word", 
                            chunk_length_s=30,
                            batch_size=32,
                            ignore_warning=True)`
Thanks

cc @sanchit-gandhi 

Thanks for the comprehensive issues description @WeichenXu123! Opened a PR here: #26699. Will discuss how to fix this best with @xenova and let you know when merged!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@josebruzzoni

I've had the same issue.
@WeichenXu123 's replies were very helpful, thanks man!

First try setting batch size to 1 if that's not a problem.

Second, you can try going into the location that the error message says in the 3rd from last row.
For me it says ""/home/nofreewill/.local/lib/python3.10/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 552, in _forward"
So I opened it, went to line 552 and changed according to @WeichenXu123 's suggestion:
generate_kwargs["num_frames"] = _stride[0]_ // self.feature_extractor.hop_length
generate_kwargs["num_frames"] = **stride[0][0]** // self.feature_extractor.hop_length

And it works now with batch size > 1 as well

also have the same issue, any update on this @sanchit-gandhi ?

@timnlupo
have you tried my solution?

You only need to write:
[0]
in the file where I said in order to try this out.

Just open the file with the location that the error message says and go to the line that it mentions and write
stride[0][0] instead of stride[0]

Another workaround in addition to `batch_size=1` is to disable chunking: `chunk_length_s=0`, but this is probably not feasable for the most.
When I read the discussion on the aforementioned PR i see that the maintainers considered this relation already.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Any updates ? @sanchit-gandhi ?  Thanks!

Good news: this was fixed by https://github.com/huggingface/transformers/pull/28114 🥳 

So what is the solution to using batch_size to 1 ? this makes the whole process much slower....

A safe change without breaking anything : 

1. Go to your env's `automatic_speech_recognition.py`, `line 552`

Replace the existing code with the following code.

```python
if stride is not None:
    if len(stride[0]) and isinstance(stride[0], tuple):
        print(f"<manual change> {stride[0]}") # just in case to check if the chunk is created in the format (duration, start time, end time)
        generate_kwargs["num_frames"] = stride[0][0] // self.feature_extractor.hop_length
    else:
        generate_kwargs["num_frames"] = stride[0] // self.feature_extractor.hop_length
```