[Whisper] Word-level timestamps broken for short-form audio
Author: sanchit-gandhi
Pull request: no
Labels: name: Good Second Issue

name: bug

name: Audio

### System Info

- `transformers` version: 4.40.0.dev0
- Platform: Linux-5.4.0-166-generic-x86_64-with-glibc2.31
- Python version: 3.11.8
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.2
- Accelerate version: 0.29.2
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.4.0.dev20240404+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

@kamilakesbi 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Word-level timestamps for the Whisper large/large-v2/large-v3/distil-large-v3 models are broken:

```python
import torch
from transformers import pipeline
from datasets import load_dataset

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

pipe = pipeline(
  "automatic-speech-recognition",
  model="openai/whisper-large-v3",
  device=device,
  torch_dtype=torch_dtype,
)
dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = dataset[0]["audio"]

prediction = pipe(sample, return_timestamps="word")
print(prediction)
```
**Print Output:**
```
{'text': ' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.', 
'chunks': [{'text': ' Mr.', 'timestamp': (29.98, 29.98)}, {'text': ' Quilter', 'timestamp': (29.98, 29.98)}, {'text': ' is', 'timestamp': (29.98, 29.98)}, {'text': ' the', 'timestamp': (29.98, 29.98)}, {'text': ' apostle', 'timestamp': (29.98, 29.98)}, {'text': ' of', 'timestamp': (29.98, 29.98)}, {'text': ' the', 'timestamp': (29.98, 29.98)}, {'text': ' middle', 'timestamp': (29.98, 29.98)}, {'text': ' classes,', 'timestamp': (29.98, 29.98)}, {'text': ' and', 'timestamp': (29.98, 29.98)}, {'text': ' we', 'timestamp': (29.98, 29.98)}, {'text': ' are', 'timestamp': (29.98, 29.98)}, {'text': ' glad', 'timestamp': (29.98, 29.98)}, {'text': ' to', 'timestamp': (29.98, 29.98)}, {'text': ' welcome', 'timestamp': (29.98, 29.98)}, {'text': ' his', 'timestamp': (29.98, 29.98)}, {'text': ' gospel.', 'timestamp': (29.98, 29.98)}]}
```

### Expected behavior

Compare to the word-level timestamps for Whisper tiny:
```python
import torch
from transformers import pipeline
from datasets import load_dataset

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

pipe = pipeline(
  "automatic-speech-recognition",
  model="openai/whisper-tiny",
  device=device,
  torch_dtype=torch_dtype,
)
dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = dataset[0]["audio"]

prediction = pipe(sample, return_timestamps="word")
print(prediction)
```
**Print Output:**
```
{'text': ' Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.', 
'chunks': [{'text': ' Mr.', 'timestamp': (0.44, 0.96)}, {'text': ' Quilter', 'timestamp': (0.96, 1.22)}, {'text': ' is', 'timestamp': (1.22, 1.5)}, {'text': ' the', 'timestamp': (1.5, 1.72)}, {'text': ' apostle', 'timestamp': (1.72, 2.0)}, {'text': ' of', 'timestamp': (2.0, 2.34)}, {'text': ' the', 'timestamp': (2.34, 2.5)}, {'text': ' middle', 'timestamp': (2.5, 2.66)}, {'text': ' classes', 'timestamp': (2.66, 3.18)}, {'text': ' and', 'timestamp': (3.18, 3.58)}, {'text': ' we', 'timestamp': (3.58, 3.68)}, {'text': ' are', 'timestamp': (3.68, 3.8)}, {'text': ' glad', 'timestamp': (3.8, 4.1)}, {'text': ' to', 'timestamp': (4.1, 4.32)}, {'text': ' welcome', 'timestamp': (4.32, 4.58)}, {'text': ' his', 'timestamp': (4.58, 4.94)}, {'text': ' gospel.', 'timestamp': (4.94, 6.6)}]}
```

Comments:
Could be related: #25607

Worth also checking whether there are any generation config parameters that are set incorrectly in the large models vs the tiny one!

Hi @sanchit-gandhi, 

It seems that setting `chunk_length_s > 0` as input to the pipeline solves the problem. 
I will probably open a PR to fix this.