Refactor whisper asr pipeline to include language too.
Author: Narsil
Pull request: yes

# What does this PR do?

## Why a refacto of this magnitude ?

- Earlier iterations tried to keep in line with `pipeline`, which started a cascade of `if ... else ` just for `whisper`.
- Those cascade of functions where all loosing the `language` information which would make it tough to include
  `language` in the return.
- Since we have chunking, potentially we have several `language` codes within a single file making this a segmentation problem, not a classification problem. (So using `detect_language(..) -> str` not really applicable to the pipeline).


Given this problem space, it was simpler to reimplement the whole thing within the `tokenizer` akin to `_build_conversational_inputs` which can allow lots of specifities within each model.
Here the new `_decode_asr` doesn't contain any information outside the tokenizer.



## How does it work ?

Hopefully inline comments should be the most comprehensive tools.
The tokenizer, needs `stride`, expressed in `seconds` and `time_precision` to convert from token space to seconds space.
We could do this outside of `tokenizer` if necessary (everything kept in tokenizer space) but since `time_precision` is already used as args of some methods, we can use this, so the output of this function doesn't need to be converted back.

=========== Overview ============
 - iterate over all outputs
 - all tokens within output
 - Each token can be
   - language token
   - special token
   - timestamp token
   - text token
 - We accumulate the text tokens.
 - We split on end timestamps
 - Lots of complexity comes from stride and timestamps

Most importantly we need to handle strides, and timestamps within strides. In order to handle those, we're simply not splitting on them, and handling merges later.

All the merges is relatively simple, we find the maximum overlapping sequence.
Small optimization: The overlap sequence might contain errors/conflicts. We're choosing from the previous sequence on the left side of the overlap, and next sequence on right side of the overlap. Since those tokens should correspond to the same audio, midway should be correct most of the time. 

## Benchmark

Courtesy of @ArthurZucker 
```python
# Whisper HF captioning
from evaluate import load
from datasets import load_dataset
import numpy as np
from transformers import pipeline
import time
import whisper


libri = load_dataset("librispeech_asr", "clean", split="test")


batch = len(libri)
start_dataset = 0

# batch = 10
# start = 0 * batch

models = ["tiny", "tiny.en", "base", "base.en", "small", "small.en", "medium", "medium.en", "large", "large-v2"]
# models = ["tiny.en"]
# models = ["large-v2"]

for model_name in models:
    speech_recognizer = pipeline(
        task="automatic-speech-recognition", model=f"openai/whisper-{model_name}", framework="pt", device=1
    )
    model = whisper.load_model(f"{model_name}" + "-v1" if model_name == "large" else f"{model_name}").to(
        device="cuda:2"
    )
    for offset in range(start_dataset, len(libri), batch):

        iterator = range(offset, offset + batch)
        speech_file = np.concatenate([libri[i]["audio"]["array"] for i in iterator])
        labels = " ".join([libri[i]["text"] for i in iterator])

        start = time.time()
        hf = speech_recognizer(
            [speech_file],
            return_timestamps=True,
            chunk_length_s=30,
            stride_length_s=[4, 4],
            batch_size=32,
            ignore_warning=True,
            num_workers=1,
        )[0]
        end = time.time()
        # print(res)
        print(f"model : {model_name}\nhf time : ", end - start)
        # print(res["text"])
        with open("hf.txt", "w") as f:
            f.write(hf["text"])

        norm_labels = speech_recognizer.tokenizer._normalize(labels)
        norm_res = speech_recognizer.tokenizer._normalize(hf["text"])

        wer = load("wer")
        hf_wer = wer.compute(predictions=[norm_res], references=[norm_labels])
        print("hf wer :", hf_wer)
```

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts and @NielsRogge
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Still not finished (I'm seeing weird drops in WER when changing parameter combinations) but as a starter @sgugger if you have early feedback I'd take it.

Ok, I found that whisper doesn't play well **at all** without `return_timestamps:

```python
# Whisper HF captioning
from evaluate import load
from datasets import load_dataset
import numpy as np
from transformers import pipeline
import time
import whisper


libri = load_dataset("librispeech_asr", "clean", split="test")
model_name = "openai/whisper-large-v2"

speech_recognizer = pipeline(task="automatic-speech-recognition", model=model_name, framework="pt", device=2)

name = model_name.split("/")[-1][len("whisper") + 1 :]
model = whisper.load_model(f"{name}-v1" if name == "large" else name).to(device="cuda:3")

# Faulty example
start = 10
end = 15
iterator = range(start, end)
speech_file = np.concatenate([libri[i]["audio"]["array"] for i in iterator])
labels = " ".join(libri[i]["text"] for i in iterator)

for return_timestamps in [True, False]:
    print(f"========Timestamps ({return_timestamps})=========")
    start = time.time()
    hf = speech_recognizer(
        [speech_file],
        return_timestamps=return_timestamps,
        chunk_length_s=30,
        stride_length_s=[4, 4],
        batch_size=32,
        num_workers=1,
    )[0]
    end = time.time()
    print(f"model : {model_name}\nhf time : ", end - start)

    norm_labels = speech_recognizer.tokenizer._normalize(labels)
    norm_res = speech_recognizer.tokenizer._normalize(hf["text"])
    print("HF TEXT:", hf["text"])

    wer = load("wer")
    hf_wer = wer.compute(predictions=[norm_res], references=[norm_labels])
    print("hf wer :", hf_wer)

    start = time.time()
    openai = model.transcribe(np.asarray(speech_file, dtype=np.float32), without_timestamps=not return_timestamps)
    end = time.time()
    norm_open = speech_recognizer.tokenizer._normalize(openai["text"])
    print("openai time : ", end - start)
    openai_wer = wer.compute(predictions=[norm_open], references=[norm_labels])
    print("openai wer :", openai_wer)
    print("Openai TEXT:", openai["text"])

    # # if hf_wer > 1.5 * openai_wer:
    # #     import ipdb

    # #     ipdb.set_trace()
```

If you check, the output is really different in both cases. Whisper seems to stop outputting tokens when not using timestamps instead of continuing.

> Can we do the changes in the tests in a followup PR so that the diffs only show potential new tests, but no changes otherwise?

There's less change than the diff leads to believe. I only removed the tests linked to a function I had removed, and there's a few tiny changes in the actual timestamps being outputted. Happy to make the modifications, it will leave a function unused here but would be the tests modifications easier to spot.

Folks I'm not sure where I'm wrong doing

```python
 predicted_ids = model.generate(input_features,
                                               max_length=model.config.max_target_positions,
                                               num_beams=5,
                                               length_penalty=1.0,
                                               repetition_penalty=1.2,
                                               do_sample=do_sample,
                                               temperature=temperature,
                                               return_timestamps=return_timestamps,
                                               early_stopping=early_stopping)
            
            if return_timestamps is False:
                # Decode the predicted IDs to text and append to results
                transcription_chunk = processor.batch_decode(predicted_ids,
                                                             task=task,
                                                             skip_special_tokens=skip_special_tokens,
                                                             return_timestamps=return_timestamps)[0]
                transcriptions.append(transcription_chunk)
            else:
                # Extract and decode the timing information
                timestamps = processor.batch_decode(predicted_ids,
                                                    task=task,
                                                    skip_special_tokens=skip_special_tokens,
                                                    return_timestamps=return_timestamps)[0]
```

Should I get `timestamps` information there when passing the kwargs  `return_timestamps` directly in the `batch_decode`?

My second attempt to retrieve timestamps

```python
             predicted_ids = model.generate(input_features,
                                               max_length=model.config.max_target_positions,
                                               num_beams=5,
                                               length_penalty=1.0,
                                               repetition_penalty=1.2,
                                               do_sample=do_sample,
                                               temperature=temperature,
                                               early_stopping=early_stopping)
            
               # Decode the outputs including the timestamps
                transcription_ids = predicted_ids[0].cpu().numpy().tolist()
                transcription_tokens = processor.batch_decode([transcription_ids])[0]

                segments = processor.tokenizer.convert_ids_to_tokens(transcription_ids)

                # Extract timing information from timestamp tokens
                transcription_chunk = []
                current_text = []
                current_start_time = None

                print(segments)
                for segment in segments:
                    if segment.startswith("
"):
                        continue
                    if segment.startswith("
"):
                        if current_text and current_start_time is not None:
                            end_time = (segments.index(segment) - 1) * HOP_LENGTH / sampling_rate
                            transcription_chunk.append({
                                "start_time": current_start_time,
                                "end_time": end_time,
                                "text": " ".join(current_text)
                            })
                            current_text = []
                            current_start_time = None
                    elif segment.startswith("<|") and segment.endswith("|>"):
                        continue
                    else:
                        if not current_text:
                            current_start_time = segments.index(segment) * HOP_LENGTH / sampling_rate
                        current_text.append(segment.replace("\u2581", " "))  # Replace special underscore token
```

This appraoch also failed, I cannot see in `segments` the timestamp tokens outputted.

THIRD ATTEMPT: Successful! I had to add `output_hidden_states`, `output_attentions` and the dicationary `return_dict_in_generate`

```python
                 predicted_ids = model.generate(input_features,
                                            max_length=model.config.max_target_positions,
                                            num_beams=5,
                                            length_penalty=1.0,
                                            repetition_penalty=1.2,
                                            do_sample=do_sample,
                                            temperature=temperature,
                                            return_dict_in_generate=True,
                                            output_scores=True,
                                            output_hidden_states=True,
                                            output_attentions=True,
                                            early_stopping=early_stopping)
            
                # Decode the outputs, including the timestamps
                decoded_transcription = processor.batch_decode(predicted_ids['sequences'], skip_special_tokens=False)[0]
                decoded_tokens = processor.tokenizer.tokenize(decoded_transcription)
                
                # Extract timestamps
                transcription_chunk = []
                current_text = []
                current_start_time = None
                for token in decoded_tokens:
                    if token.startswith("<|"):
                        if token not in {"
", "
", "
", "
", "
"}:
                            if current_text:
                                end_time = start_idx / sampling_rate + processor.tokenizer.convert_tokens_to_ids([token])[0] * HOP_LENGTH / sampling_rate
                                transcription_chunk.append({
                                    "start_time": current_start_time,
                                    "end_time": end_time,
                                    "text": " ".join(current_text)
                                })
                                current_text = []
                            current_start_time = start_idx / sampling_rate + processor.tokenizer.convert_tokens_to_ids([token])[0] * HOP_LENGTH / sampling_rate
                    else:
                        if not current_text:
                            current_start_time = start_idx / sampling_rate
                        current_text.append(token.replace("Ġ", " "))  # Clean up token format

                if current_text:
                    end_time = len(audio_chunk) / sampling_rate
                    transcription_chunk.append({
                        "start_time": current_start_time,
                        "end_time": end_time,
                        "text": " ".join(current_text)
                    })
```