[Whisper] Fix whisper tokenizer
Author: eustlb
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Fixes #34472

## What's happening

A special case in timestamp offsets was not handled.
When predicting timestamps, Whisper follows two strategies (see [here](https://github.com/openai/whisper/blob/5979f03701209bb035a0a466f14131aeb1116cbb/whisper/transcribe.py#L367C17-L375C62)):
1. single timestamp at the end: predicted sequence ends with 
<eos> → no speech after t1, seek to* end of the 30sec segment
2. double timestamp at the end: predicted sequence ends with 

<eos> →  seek to* t1 

>*Note: Whisper works on 30sec windows of audio. Above "seek to" means sliding this 30sec window to a new start position.

Case 1 is correctly handled in `_retrieve_segments` that is responsible of this seeking process during generation, making the generated segments correct (see snippet below) while it is not correctly handled in the tokenizer. 

<details>
  <summary>Snippet 🔧</summary>
  
```python
from datasets import load_dataset
from transformers import WhisperForConditionalGeneration, AutoProcessor
import numpy as np

# load model + processor
processor = AutoProcessor.from_pretrained("openai/whisper-small.en")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small.en")

# load dataset
dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
sample = dataset[0]["audio"]["array"]
sampling_rate = dataset[0]["audio"]["sampling_rate"]

sample = [*sample[:15 * sampling_rate], *np.zeros(16 * sampling_rate).tolist(), *sample[15 * sampling_rate:]]
sample = np.array(sample)

# pre-process
inputs = processor(
    sample,
    sampling_rate=16_000,
    padding="longest",
    truncation=False,
    return_attention_mask=True,
    return_tensors="pt",
)

# inference
output = model.generate(**inputs, return_timestamps=True, return_segments=True)

# this is correct
print("=" * 10, "this is correct", "=" * 10)
for seg in output["segments"][0]:
    print(f"{seg['start'].item():.2f} -> {seg['end'].item():.2f}: {processor.decode(seg['tokens'])}")

# this is wrong
# pass token ids to processor's decode method
print("=" * 10, "this is wrong", "=" * 10)
result = processor.batch_decode(output["sequences"], skip_special_tokens=True, output_offsets=True)
print("\n".join([f"{chunk['timestamp'][0]:.2f} -> {chunk['timestamp'][1]:.2f} : {chunk['text']}" for chunk in result[0]["offsets"]]))
```

Returns:
```
========== this is correct ==========
0.00 -> 6.38:  Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.
6.38 -> 11.32:  Nor is Mr. Quilter's manner less interesting than his matter.
11.32 -> 15.00:  He tells us that at this festive season of the year,
30.00 -> 36.76:  With Christmas and roast beef looming before us, similes drawn from eating and its results
36.76 -> 39.80:  occur most readily to the mind.
39.80 -> 45.38:  He has grave doubts whether Sir Frederick Layton's work is really Greek after all and
45.38 -> 49.00:  can discover in it but little of rocky Ithaca.
49.00 -> 56.28:  Lenell's pictures are a sort of up-guards-and-atom paintings, and Mason's exquisite ittles
56.28 -> 64.12:  are as national as a jingo poem. Mr. Burkett fosters landscape's smile at one much in
64.12 -> 70.76:  the same way that Mr. Karker used to flash his teeth. And Mr. John Collier gives his
70.76 -> 77.16:  sitter a cheerful slap on the back before he says, like a shampoo or in a Turkish bath,
77.16 -> 78.16:  Next Man
========== this is wrong ==========
0.00 -> 6.38 :  Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.
6.38 -> 11.32 :  Nor is Mr. Quilter's manner less interesting than his matter.
11.32 -> 15.00 :  He tells us that at this festive season of the year,
15.00 -> 21.76 :  With Christmas and roast beef looming before us, similes drawn from eating and its results
21.76 -> 24.80 :  occur most readily to the mind.
24.80 -> 30.38 :  He has grave doubts whether Sir Frederick Layton's work is really Greek after all and
30.38 -> 34.00 :  can discover in it but little of rocky Ithaca.
34.00 -> 41.28 :  Lenell's pictures are a sort of up-guards-and-atom paintings, and Mason's exquisite ittles
41.28 -> 49.12 :  are as national as a jingo poem. Mr. Burkett fosters landscape's smile at one much in
49.12 -> 55.76 :  the same way that Mr. Karker used to flash his teeth. And Mr. John Collier gives his
55.76 -> 62.16 :  sitter a cheerful slap on the back before he says, like a shampoo or in a Turkish bath,
62.16 -> 63.16 :  Next Man
```
</details>

## Changes 

### on the existing
Most important change here is the one in `_retrieve_segments`. When finishing generation with 

 (case 2), we **must** add it to the tokens of the returned segment. Indeed, otherwise, when we will decode the concatenated sequence, we won't have any way to differentiate a single ending token (case 1) from a double ending token. 

### added 
A new `test_small_longform_timestamps_generation` test since this edge case was not catched before. IMO it's an important one to add since we can compare directly with OAI's expected output so it's pretty robust. Code to reproduce the expected output can be found [here](https://github.com/eustlb/reproduce-whisper-expected-outputs/blob/main/test_small_longform_timestamps_generation.py). This new test catch an edge case we did not test before: single timestamp ending (here at 15.0s), triggering seeking to end of segment (here 30.0s). Moreover, we test here both the segments through `generated_ids["segments"]` that are built at generation time and the offsets with `processor.batch_decode(generated_ids["sequences"], skip_special_tokens=True, output_offsets=True)` that are reconstructed from the concatenated segments.
 





Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34537). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.