Incorrect Whisper long-form decoding timestamps 
Author: kamilakesbi
Pull request: yes
Labels: name: Audio

# What does this PR do?

As mentionned in issue #31942,  when performing long form generation, timestamps are currently reset to zero after 30 seconds of audio. 

This error occurs both with AutomaticSpeechRecognitionPipeline and WhisperForConditionalGeneration. 

Here, I propose a solution to make it work with `WhisperForConditionalGeneration`. With this PR, the following code snippet should give the right output: 

```
import numpy as np
import json
from transformers import AutoProcessor, WhisperForConditionalGeneration
import torch
from datasets import load_dataset

device = "cuda"
torch_dtype = torch.bfloat16

processor = AutoProcessor.from_pretrained("distil-whisper/distil-large-v3")
model = WhisperForConditionalGeneration.from_pretrained("distil-whisper/distil-large-v3", torch_dtype=torch.float16)
model = model.to("cuda")

dataset = load_dataset(
    "hf-internal-testing/librispeech_asr_dummy", "clean", split="validation"
)
sample = dataset[0]["audio"]
sample = np.concatenate([sample["array"]] * 10)

inputs = processor(sample, return_tensors="pt", truncation=False, sampling_rate=16_000)
inputs = inputs.to("cuda", torch.float16)

output = model.generate(**inputs, return_timestamps=True, return_segments = True)

result = processor.batch_decode(output, skip_special_tokens=True, output_offsets = True)
```

## Explanation of the solution: 

When performing long form generation with Whisper, the utterance level timestamps are returned as output to generate when we specify `return_segments = True` and `return_timestamps=True`.

The problem arise at the decoding level: `batch_decode` currently doesn't support long form generation (can't take as input `segments`). When specifying output_offsets, it outputs the wrong timestamps. 

Solution: we can update `batch_decode` to use the timestamps returned by the generate method. 

# Who can review: 

@sanchit-gandhi @ylacombe 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32003). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @ylacombe for your review! 

There was indeed a bug in the snippet I shared with you, I've just updated it :) 

I'll add a test to make sure everything works properly. 

cc @amyeroberts this should be ready for final review :) 

Thanks for the review @amyeroberts, this can be merged :) 

While this PR goes some way to addressing the aforementioned issue, it only does so by changing the fundamental design of the processor. 

Specifically, the processor's `.batch_decode` method is designed to work explicitly on **tensors** of token ids. We take the dictionary output from `.generate`, and pass the token ids (sequences) to the `.batch_decode` method for decoding:

```py
output = model.generate(**inputs, return_timestamps=True, return_segments=True)
result = processor.batch_decode(output["sequences"], skip_special_tokens=True, output_offsets=True)
```

In doing so, the processor's `.batch_decode` method can call the tokenizer's `.batch_decode` method directly. We know the token ids are passed as an arg, and the rest of the args/kwargs can be forwarded directly to the tokenizer for decoding:
https://github.com/huggingface/transformers/blob/673440d073d5f534a6d6bedeeca94869afd8d0a7/src/transformers/models/whisper/processing_whisper.py#L82-L87

This PR breaks this design. It relies on the user passing a **dictionary** of model outputs to the processor's `.batch_decode` method, e.g. as shown in the code example in the PR description:
```diff
output = model.generate(**inputs, return_timestamps=True, return_segments=True)
- result = processor.batch_decode(output["sequences"], skip_special_tokens=True, output_offsets=True)
+ result = processor.batch_decode(output, skip_special_tokens=True, output_offsets=True)
```

It achieves this by extracting the token ids out of this dictionary, and then passing a new set of args/kwargs to the tokenizer:
https://github.com/huggingface/transformers/blob/0fdea8607d7e01eb0e38a1ebeb7feee30a22f0cf/src/transformers/models/whisper/processing_whisper.py#L82-L94

There are two shortcomings in this design:
1. Because we do extra pre-processing in the processor class, only the processor receives the fix from this PR, not the tokenizer. Take the example where the user runs the feature extractor + tokenizer API (instead of the processor one). The user would still get the wrong outputs after this PR.
2. The original code example where the issue was flagged is **not** fixed. In the original code example, we pass the tensor of token ids to the `.batch_decode` method:
```python
result = processor.batch_decode(output["sequences"], skip_special_tokens=True, output_offsets=True)
```
For this PR to work, we need to pass the dictionary of outputs to the `.batch_decode` method:
```python
result = processor.batch_decode(output, skip_special_tokens=True, output_offsets=True)
```
This means the fix **will not** benefit users who are using existing code examples. It will only benefit users who run the new design where dictionaries of model outputs are permitted to be passed to the `.batch_decode` method, which is a new design that we do not want to promote.

The correct fix should require changes to `_compute_offsets` only, as detailed in #32131.

I propose that we either:
1. Revert this PR and open a new one with the tokenizer changes in #32131
2. Proceed with #32131, which reverts the processing + tokenizing changes from this PR and implements the updated logic to `_compute_offsets`

WDYT @kamilakesbi @ylacombe?

Hi @sanchit-gandhi, 

Thanks for the explanation! I agree with you that this PR breaks the design and your suggested solution is better :) thanks for implementing it :) 

Reverting this PR in favour of https://github.com/huggingface/transformers/pull/32131. Doing this now so it doesn't creep into the up-coming release.