Whisper Large v3 Word Level Timestamps Error
Author: souvikqb
Pull request: no

### System Info


- `transformers` version: 4.36.0.dev0
- Platform: Linux-5.15.0-75-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.19.0
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes

### Who can help?

@sanchit-gandhi 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
from datasets import load_dataset


device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model_id = "openai/whisper-large-v3"

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
)
model.to(device)

processor = AutoProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=128,
    chunk_length_s=30,
    batch_size=16,
    return_timestamps=True,
    torch_dtype=torch_dtype,
    device=device,
)

dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
sample = dataset[0]["audio"]

result = pipe(sample)
print(result["text"])

sample = 'Denoised_Audio.wav'

result = pipe(sample, return_timestamps="word")
print(result["chunks"])
```

Error - 

![image](https://github.com/huggingface/transformers/assets/135606016/d1dc0e4f-d1ae-4021-9fe5-aac7e75c2119)


### Expected behavior

Expecting to get back Word Level timestamps

Just to be clear the file is **NOT** Corrupted by any chance. It still gives out the text transcript. 

Only facing issue in word level transcription

Comments:
I'm having the same issue, and seems to be related to the batch size. At least it works for me if i set `batch_size=1`.

Nevertheless, it would good to be able to use a higher batch size.

Thanks for the ping - will be fixed by #26699. Will try and fix this asap!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Fixed by https://github.com/huggingface/transformers/pull/28114 - feel free to install Transformers on `main` to get the update: https://huggingface.co/docs/transformers/installation#install-from-source

Otherwise, it'll be in the next release!