Timestamps in Whisper processor
Author: JeffreyWardman
Pull request: no

### Feature request

output_word_offsets argument in Whisper's processor.decode() function.

I want to get the timestamp of the start and end of each word.

### Motivation

I cannot use Whisper until it accommodates for word timestamps and long audio.

### Your contribution

With guidance, happy to submit it but will need guidance. Can do this in a month's time.

Comments:
cc @sanchit-gandhi or @ArthurZucker 

Related to #19887, in which timestamps for Whisper were discussed. Is this on your timeline @ArthurZucker as part of the Whisper integration? Otherwise I'll add it to my TODO's!

Hey, really sorry for being so late. Will focus on that next week! I'll ping you once a draft PR is ready! 🤗 

BTW, you can already have the `timestamp` generation using the model : 
```python 


```
```
tensor([[50258, 50265, 50359, 50364,  1456,  1804,  1021,   871,   368,   635,
         32400,   368,   635, 32400,  1030,  4666,  2795,    70,  3201,   339,
           892,  1531,   287,   311,    68,   368, 10384,  2023, 20071,    13,
         50639, 50257]])
``` Where the timestamp tokens are `>50363`. You can also use a custom logit processor to be sure that they are correctly generated. 

Moreover, the original paper used a simple rule that associates 0.02 seconds to each tokens, which means that without removing the special tokens you can already get the per_word timestamps. 😉 

@ArthurZucker can you provide an example of how to get the timestamp tokens like above with `WhisperForConditionalGeneration` by any chance?

Of course. BTW it is included in[ this notebook](https://colab.research.google.com/drive/1rS1L4YSJqKUH_3YxIQHBI982zso23wor#scrollTo=Ca4YYdtATxzo)
```python 
from datasets import load_dataset
from transformers import WhisperForConditionalGeneration, WhisperProcessor
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small").to(device)
processor = WhisperProcessor.from_pretrained("openai/whisper-small")
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
audio_sample = ds[3]
speech_data = audio_sample["audio"]["array"]
speech_file = audio_sample["file"] # used as an example for the pipeline
inputs = processor.feature_extractor(speech_data, return_tensors="pt", sampling_rate=16_000).input_features
generate_ids = model.generate(inputs, return_timestamps=True, task="translate")
print(generate_ids)
```
```python
tensor([[50258, 50266, 50358, 50364,   634,   575, 12525, 22618,  1968,  6144,
         35617, 20084,  1756,   311,   589,   307,   534, 10281,   934,   439,
           293, 50676, 50676,   393,  4411,   294,   309,   457,   707,   295,
         33301,   286,   392,  6628,    13, 50836, 50257]])
>>> processor.tokenizer.decode(generate_ids[0], decode_with_timestamps=True)



<|0.00|> He has grave doubts whether Sir Frederick Layton's work is really Greek after all and<|6.24|><|6.24|> can discover in it but little of rocky Ithaca.<|9.44|>

```

Great. Thanks! I had a quick test and for this: `each` (1123) occurs at 7.0 seconds in the audio. However with each token representing 0.02s you can see it's at 0.34s. So it doesn't look like using the tokens like so cannot find breaks at a per word level.
```
tensor([[50257, 50363,  1649,   257,  3440,  1332,   318,  2067,    11,   281,
          4554,   286,   257,  2836,  1398,   481,   307,  2727,   329,  1123,
         50681, 50681,
```

That is not exactly the way to compute the time. You should be using the `tokenizer.decode(..., output_offset = True)`.
Also the 0.02s rule is to convert from a timestamp token to time. Here the end is at `50681`, you substract the timestamp begin so you have `50681 - 50363 = 318` which you then multiply by `0.02`, you get `6.36`s. 

When I use `generate(..., decode_with_timestamps=True)`
...followed by `decode(pid, decode_with_timestamps=True)`,
I get timestamps but very wrong transcription.
If I leave `decode_with_timestamps=True` in, but use `output_offset=True` instead of `decode_with_timestamps=True` I get the same low-accuracy, and no timestamp output.

Does anyone have an example of doing this correctly?
This is my current snippet:
```python3
for predwav in predwavs:
    aa,sr=librosa.load(predwav, sr=16000)
    sample=aa.astype(np.float64)
        # import ipdb; ipdb.set_trace(context=16); pass

    input_features = processor(sample, sampling_rate=sr, return_tensors="pt").input_features 
    # generate token ids
    predicted_ids = model.generate(input_features, return_timestamps=True)
    # predicted_ids = model.generate(input_features)
    # decode token ids to text
    transcription = processor.batch_decode(predicted_ids, skip_special_tokens=False)
    print(f"Transcription: {transcription}")
    print("Timestamp info:")
    for pidi, pid in enumerate(predicted_ids):
        # timestamps = processor.tokenizer.decode(pid, decode_with_timestamps=True)
        timestamps = processor.tokenizer.decode(pid, output_offset=True)
        print(f"Predicted id [{pidi}]: {timestamps}")
    
    transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
    print(f"Transcription: {transcription}")
    import sys; sys.exit()
```

Update:
Okay, it doesn't error with `output_offset` -- but the correct is `output_offsets`.
I went through the code and, if you use that (True), the return changes to a dict with the keys 'text' and 'offsets':
pdict = processor.tokenizer.decode(pid, output_offsets=True)
> pdict['offsets']
[{'text': ' There is a time...', 'timestamp': (0.0, 2.6)}, {'text': ' ...of a subconscious development.', 'timestamp': (14.6, 17.6)}, {'text': " It don't work.", 'timestamp': (20.6, 22.6)}, {'text': 'Bureau work.', 'timestamp': (23.400000000000002, 24.400000000000002)}, {'text': ' The branch, the branch.', 'timestamp': (25.6, 27.6)}]

I still haven't resolved the accuracy-destruction issue.

which version of transformers are you using? Do you want to open a separate issue with the accuracy issue? 

Okay, posted as a new issue. Thanks! :)
https://github.com/huggingface/transformers/issues/30815

I came up with a new potential capability (idea) which might balance accuracy/degradation of timestamp inclusion:
https://github.com/huggingface/transformers/issues/30815#issuecomment-2254296338