Whisper Timestamp processor and prediction
Author: ArthurZucker
Pull request: yes

# What does this PR do?
This will add support for correct `timestamp` prediction in the generation, and should update the ASR pipeline to use these when generating on longer audio file. 
The rough idea is that when the timestamps are generated, the model is  more *aware* of the timing and generates `
` tokens to fill in the silence. So the token-to-time is approximatly a linear regression, and provides valuable information for matching begining and end of chunk of a longer audio. 
By using both the fact that **timestamp**  tokens always come in pair when separating two sentences, and the approximate **toke-to-time** (see [here](https://github.com/openai/whisper/blob/main/whisper/transcribe.py#L134)) we should increase the performances and also have the timestamp prediction.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Looking forward to this PR

Example output, HF vs openai : 
Model : `openai/whisper-medium`.
```python 
{'chunks': [{'text': " Je m'appelle Claude.", 'timestamp': (0.0, 2.0)},
            {'text': ' Je te coupe, Plow.', 'timestamp': (2.0, 4.0)},
            {'text': " Let's just try it again.", 'timestamp': (8.0, 10.0)},
            {'text': " Je m'appelle Claude.", 'timestamp': (10.0, 12.0)},
            {'text': ' Je te plie, Mlew.', 'timestamp': (12.0, 14.0)},
            {'text': " Huh. It's not quite what I'm saying.",'timestamp': (16.0, 20.0)},
            {'text': ' Really?', 'timestamp': (20.0, 22.0)},
            {'text': ' Sounds exactly the same to me.','timestamp': (22.0, 24.0)},
            {'text': ' It does? Really?', 'timestamp': (24.0, 26.0)},
            {'text': ' Yeah.', 'timestamp': (26.0, 28.0)},
            {'text': " Let's try it again. Really listen.",'timestamp': (29.0, 30.88)},
            {'text': ' Got it.', 'timestamp': (30.88, 32.48)},
            {'text': " Je m'appelle Claude.",'timestamp': (32.480, 35.24)},
            {'text': ' Je te flou-flee.', 'timestamp': (35.24, 37.28)},
            {'text': ' Oh, mon Dieu.', 'timestamp': (39.28, 40.28)},
            {'text': ' Oh, de fouf.', 'timestamp': (40.28, 41.88)},
            {'text': " Je m'appelle Claude.",'timestamp': (43.48, 46.6)},
            {'text': ' Je te call blue.', 'timestamp': (46.6, 48.24)},
            {'text': ' No!', 'timestamp': (48.24, 50.44)},
            {'text': ' Okay, maybe if we just break it down.','timestamp': (50.44, 53.28)},
            {'text': " Okay, let's just try it one syllable at a time.",'timestamp': (53.28, 56.08)},
            {'text': ' Okay, so repeat after me.', 'timestamp': (56.08, 58.08)},
            {'text': ' Pardon me.', 'timestamp': (58.0, 59.0)},
            {'text': ' Je...', 'timestamp': (59.0, 60.0)},
            {'text': ' Je...', 'timestamp': (60.0, 61.0)},
            {'text': ' Ma...', 'timestamp': (61.0, 62.0)},
            {'text': ' Ma...', 'timestamp': (62.0, 63.0)},
            {'text': ' Pelle.', 'timestamp': (63.0, 64.0)},
            {'text': ' Pelle.', 'timestamp': (64.0, 65.0)},
            {'text': ' Great!', 'timestamp': (65.0, 66.0)},
            {'text': ' Okay, faster.', 'timestamp': (66.0, 67.0)},
            {'text': ' Je...', 'timestamp': (67.0, 68.0)},
            {'text': ' Je...', 'timestamp': (68.0, 69.0)},
            {'text': ' Ma...', 'timestamp': (69.0, 70.0)},
            {'text': ' Pelle.', 'timestamp': (70.0, 71.0)},
            {'text': ' Pelle.', 'timestamp': (71.0, 72.0)},
            {'text': " Je m'appelle.", 'timestamp': (72.0, 73.0)},
            {'text': ' Mais pour pour?', 'timestamp': (73.0, 74.0)},
            {'text': " It's too hard.", 'timestamp': (74.0, 75.0)},
            {'text': " I can't teach you.", 'timestamp': (75.0, 76.0)},
            {'text': ' What are you doing?', 'timestamp': (76.0, 77.0)},
            {'text': ' I have to go before I put your head through a wall.','timestamp': (77.0, 78.0)},
            {'text': " Don't go!", 'timestamp': (78.0, 79.0)},
            {'text': " Don't go!", 'timestamp': (79.0, 80.0)},
            {'text': ' I need you!', 'timestamp': (80.0, 81.0)},
            {'text': ' My addition is tomorrow!', 'timestamp': (81.0, 82.0)},
            {'text': ' Cha-blu-bla!', 'timestamp': (82.0, 83.0)},
            {'text': ' Mille-la-pille!', 'timestamp': (83.0, 84.0)},
            {'text': ' Oum-bla!', 'timestamp': (84.0, 85.0)},
            {'text': ' Hola!', 'timestamp': (82.56, 83.4)}],
 'text': " Je m'appelle Claude. Je te coupe, Plow. Let's just try it again. Je "
         "m'appelle Claude. Je te plie, Mlew. Huh. It's not quite what I'm "
         'saying. Really? Sounds exactly the same to me. It does? Really? '
         "Yeah. Let's try it again. Really listen. Got it. Je m'appelle "
         "Claude. Je te flou-flee. Oh, mon Dieu. Oh, de fouf. Je m'appelle "
         'Claude. Je te call blue. No! Okay, maybe if we just break it down. '
         "Okay, let's just try it one syllable at a time. Okay, so repeat "
         'after me. Pardon me. Je... Je... Ma... Ma... Pelle. Pelle. Great! '
         "Okay, faster. Je... Je... Ma... Pelle. Pelle. Je m'appelle. Mais "
         "pour pour? It's too hard. I can't teach you. What are you doing? I "
         "have to go before I put your head through a wall. Don't go! Don't "
         'go! I need you! My addition is tomorrow! Cha-blu-bla! '
         'Mille-la-pille! Oum-bla! Hola! Boo.'}
```
```
[(" Je m'appelle Claude.", 0.0, 2.0),
 (' Je te coupe, Plow.', 2.0, 4.0),
 (" Let's just try it again.", 8.0, 10.0),
 (" Je m'appelle Claude.", 10.0, 12.0),
 (' Je te plie, Mlew.', 12.0, 14.0),
 (" Huh. It's not quite what I'm saying.", 16.0, 20.0),
 (' Really?', 20.0, 22.0),
 (' Sounds exactly the same to me.', 22.0, 24.0),
 (' It does? Really?', 24.0, 26.0),
 (' Yeah.', 26.0, 28.0),
 (" Let's try it again. Really listen.", 28.0, 30.0),
 (' Got it.', 30.0, 32.0),
 (" Je m'appelle Claude.", 32.0, 34.0),
 (' Je te plie, Mlew.', 34.0, 36.0),
 (' Oh, mon Dieu.', 38.0, 40.0),
 (' Oh, de fouf.', 40.0, 42.0),
 (" Je m'appelle Claude.", 42.0, 44.0),
 (' Je te coupe, Mlew.', 44.0, 46.0),
 (' No!', 46.0, 48.0),
 (' Okay.', 48.0, 50.0),
 (' Maybe if we just break it down.', 50.0, 52.0),
 (" Okay, let's just try it one syllable at a time.", 52.0, 54.0),
 (' Okay, so repeat after me.', 54.0, 56.0),
 (" Je m'appelle.", 56.0, 60.0),
 (' Great. Okay, faster.', 60.0, 62.0),
 (" Je m'appelle.", 62.0, 64.0),
 (" Je m'appelle.", 64.0, 66.0),
 (' Mais pour pour?', 66.0, 68.0),
 (" It's too hard. I can't teach you.", 70.0, 72.0),
 (' What are you doing?', 72.0, 74.0),
 (' I have to go before I put your head through a wall.', 74.0, 76.0),
 (" Don't go. I need you.", 76.0, 78.0),
 (' My audition is tomorrow.', 78.0, 80.0),
 (' Cha-blah-blah.', 80.0, 82.0),
 (' Mela-pi.', 82.0, 84.0),
 (' Hola!', 84.0, 86.0),
 (' Boo!', 86.0, 114.0)]
```
Note that the differences in the text are related to the logit processors that they updated. But overall it is very similar, but 3x faster 😉 

Looking close!!!
🙏🏾🤞🏾 this review gets pushed through!! 

> Looking close!!! 🙏🏾🤞🏾 this review gets pushed through!!

I'm too thirsty lol. Love yall and appreciate all the work being done! 

> > Looking close!!! 🙏🏾🤞🏾 this review gets pushed through!!
> 
> I'm too thirsty lol. Love yall and appreciate all the work being done!

@Narsil 🫠👀

🙌🏾

Really appreciate the constant updates to get this finalized! Thanks! 

This should just need a little code cleaning / documenting and will be good for a final review! 

Wow! That was a lot of refactoring. Ready for a final review @Narsil 

@TheExGenesis the current implementation should not really be different with the `Trie` approach. Problem with the `Trie` was that it does not really keep track of the longest, but rather the first longest common sequence. It also assumed that the entire sequence had to be present (an extra loop would have had to be added). 

We can still discuss cases with a term appearing twice, in the current implementation the last occurence would be chosen for merge. Do you have a specific example in mind? 

@ArthurZucker I'm choosing the first occurrence rather than the last one and it's working well. Otherwise, if an expression is the actual prefix, and occurs later in the sequence, the beginning of the sequence gets eaten. Also, I think you should be discounting stride_right as well when incrementing chunk time. I apologize for not making specific code recommendations right now, I'm a little short on time and working in my own messy environment.

I think it is pretty random and would need a heuristic for small sequences. If you merge on a single term, you should probably be using just a little bit more chunks length. Have you tried both versions? 😉 

> I think it is pretty random and would need a heuristic for small sequences. If you merge on a single term, you should probably be using just a little bit more chunks length. Have you tried both versions? 😉

I'm sorry I don't understand, are you responding to the first or second point? If to the second point, you really want the timestamps to be accurate otherwise they won't match up with the audio.

No I was talking about the first point, did you try taking the last occurence as well? Just wondering if you have some kind of experimental benchmark on this. 

The `stride_right` is not used, based on [this](https://huggingface.co/blog/asr-chunking), the stride right is part of the speech that is disregarded. It is not very intuitive, but basically the stride right does not influence the beginning time of the next sequence.

> No I was talking about the first point, did you try taking the last occurence as well? Just wondering if you have some kind of experimental benchmark on this.
> 
> The `stride_right` is not used, based on [this](https://huggingface.co/blog/asr-chunking), the stride right is part of the speech that is disregarded. It is not very intuitive, but basically the stride right does not influence the beginning time of the next sequence.

Yeah I tried taking the last occurrence, it ate like 10s of audio, and then I changed it.

re stride_right - [chunk_iter does use stride_left and stride_right ](https://github.com/huggingface/transformers/blob/15573920bccf879d621e99e21367e216017adf7d/src/transformers/pipelines/automatic_speech_recognition.py#L59), and I verified this empirically, the timestamps are only correct when I take both left and right strides into account. 

Oh okay thanks. Would be awesome if you have a sample audio on which I could work on 😉 
I think you are making a good point, `chunk_iter` is indeed stepping w.r.t the stride right! My bad. 

> Oh okay thanks. Would be awesome if you have a sample audio on which I could work on 😉 I think you are making a good point, `chunk_iter` is indeed stepping w.r.t the stride right! My bad.

All good :) I've been using the first few minutes of [this podcast](https://www.iheart.com/podcast/256-global-voices-podcast-31091854/episode/bangladeshs-new-years-celebration-of-diversity-63758318/)