[Whisper] Finalize batched SOTA long-form generation
Author: patrickvonplaten
Pull request: yes

🚨🚨 ***Disclaimer***: **All the credit of this PR goes to the original codebase: https://github.com/openai/whisper/blob/main/whisper/transcribe.py (this PR is to 90% copied from there). The reason it's added here (instead of contributing to https://github.com/openai/whisper) is because the original repo fundamentally doesn't work with batched inference and because the original repo doesn't make use of Flash Attention and other PyTorch-specific optimizations** 🚨🚨

# Batched Long-Form Generation for Whisper

Whisper is the *de-facto* open-source model that is used in production use cases. It's still the most performant open-source model out there and was shipped with a variety of sizes ranging from 30M params to 1.5B params. 

For English, especially the slightly smaller checkpoints such as `whisper-medium.en` and `whisper-small.en` yield impressive performances.

Most benchmarks only evaluate on short-form (input is shorter than 30 seconds), such as [our ASR leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard), but for most industry use cases (meeting transcriptions, one actually requires long-form transcription (input is minutes or even hours long)

As stated [in section 4.5 of the paper](https://cdn.openai.com/papers/whisper.pdf), long-form generation requires a couple of heuristics to function well. These include:
- init time stamp constrain
- condition on prev input
- voice activity detection
- temperature fallback when it is detected that the model yields repetitive outputs or low-prob outputs

These features help to make long-form transcription more performant and more robust. So far our offer in transformers mainly focused on chunked transcription and vanilla "sequential" decoding which was added [here](https://github.com/huggingface/transformers/pull/27492). 

While chunked transcription gives already good results for long-form (see [here](https://github.com/huggingface/transformers/pull/27492)), "sequential" decoding improves the results significantly. This PR adds all features of "sequantial" decoding and allows it to be used for batched generation.
Compared to "chunked generation", this PR now improves:
`whisper-tiny.en` by 25-30%
`whisper-small.en` by 20%
`whisper-large-v2` by 10% (not conditioning on prev input) 
*numbers are retrieved by comparing results below with results of "chunked generation" [here](https://github.com/huggingface/transformers/pull/27492)*

=> Smaller model profit more from improved decoding strategies! 

The main reason why this feature is implemented in Transformers is because the original code base doesn't support batch inference (see [here](https://github.com/openai/whisper/discussions/662)) and is quite slow compared as it doesn't make use of batching, nor pure fp16 nor FA2.

## Speed Results:

- Transcribing all of [these four long-form](https://huggingface.co/collections/distil-whisper/long-form-test-sets-652ec038902fe76a6d0d65bb) datasets with a high batch size and whisper-large-v2 is **4.5 times** faster with `transformers` compared to the [original codebase](https://github.com/huggingface/transformers/pull/27492) while yielding more or less the same results (measured with Torch 2.1, CUDA 12.1 on RTX4090):
- Original Whisper (batch size 1): **9h2min**
- Transformers (batch size 1): **5h20min**
- Transformers (batch size 16): **2h23min**
- Transformers (chunked, batch size = 16): **2h15min**.

=> Batched sequential generation is therefore only 5% slower than chunked generation, but gives much improved WER results.

## Usage

One activates long-form generation by simply passing `input_features` that are longer than 30 seconds. **Note** when running in `batch_size>1` mode, one should pass an `attention_mask` as well so that the input length of each audio stream can be known:

(the following command takes ~5min on a RTX4090 since 120h of audio are decoded)
```py
#!/usr/bin/env python3
from transformers import WhisperForConditionalGeneration, AutoProcessor
from datasets import load_dataset, Audio
import torch
import numpy as np

processor = AutoProcessor.from_pretrained("openai/whisper-small.en")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small.en", torch_dtype=torch.float16)
model.to("cuda")

# rertieve 8 long audio sequences
ds = load_dataset("distil-whisper/earnings21", "full")["test"]
ds = ds.cast_column("audio", Audio(sampling_rate=16000))
ds = ds[:8] # take batch size of 8

raw_audio = [x["array"].astype(np.float32) for x in ds["audio"]]

# process input, make sure to pass `padding='longest'` and `return_attention_mask=True`
inputs = processor(raw_audio, return_tensors="pt", truncation=False, padding="longest", return_attention_mask=True, sampling_rate=16_000)
inputs = inputs.to("cuda", torch.float16)

# activate `temperature_fallback` and repetition detection filters and condition on prev text
result = model.generate(**inputs, condition_on_prev_tokens=False, temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0), logprob_threshold=-1.0, compression_ratio_threshold=1.35, return_timestamps=True)

decoded = processor.batch_decode(result, skip_special_tokens=True)
print(decoded)
```


## WER Results:

First, it is to be noted that results can sometimes strongly vary between runs (1-2% absolute WER) when temperature fallback is activated due to the random nature when sampling and because long-form transcription is strongly dependent on previous inputs. This is much less the case though when NOT conditioning on previous input.

In summary, Transformers results match the results of the original code base. They match very closely when not conditioning on previous input, but have a higher variance when conditioning on previous input.
It is important to understand that an array of different hyper-parameters is used here for inference:

```
        gen_kwargs = {
            "condition_on_prev_tokens": True/False,
            "max_length": 448,
            "return_timestamps": True,
            "num_beams": data_args.num_beams,
            "top_k": 0,
            "compression_ratio_threshold": 1.35, # different compression threshold is used
            "temperature": (0.0, 0.2, 0.4, 0.6, 0.8, 1.0),
            "logprob_threshold": -1.0,
            "no_speech_threshold": 0.6,
        }
```

whereas especially `compression_ratio_threshold`, `temperature`, `condition_on_prev_tokens`, `max_length, `logprob_threshold` and `no_speech_threshold` quite strongly influence the WER. In addition, different hardware can lead to different results.

**Note**: All of the above parameters are **exactly** implement as in the original codebase except fo `"compression_ratio_threshold"`. While the original codebase computes the compression ratio based on the decoded string of a generation, here we compute it directly on the predicted IDS because:
- High repetition in actual output generations is a better proxy for repetition compared to high repetition of letters/words
- It allows for a cleaner design where we don't have to pass the tokenizer into `generate`.

You can unfold the following to have in-detail overview of all results of this PR compared to the original codebase.
In a nutshell, one can see that when **not** conditioning on the previous input, results more or less match exactly (only very minor differences). However, when conditioning on the previous input, the results because much more brittle (re-running the same experiment will then lead to different results). Here we see that the original codebase gives slightly better results for the bigger models, but slighly worse for smaller models. Re-running the experiment might give different results though.

<details>

<summary>All results - click me</summary>

Results for all models (except v3) on long-form.

**[This PR] When not conditioning on previous input:**

| Name                | rev16/test_wer | meanwhile/test_wer | earnings22/test_wer | earnings21/test_wer | **Avg**
|---------------------|----------------|--------------------|---------------------|---------------------|---------------------|
| whisper-large-v2    | 11.3           | 4.8                | 13.7                | 10.5                | 10.08
| whisper-medium.en   | 11.3           | 5.6                | 13.8                | 10.8                | 10.38
| whisper-medium      | 11.3           | 5.6                | 13.8                | 10.6                | 10.33
| whisper-small.en    | 11.8           | 6.6                | 14.6                | 11.4                | 11.1
| whisper-small       | 12.1           | 7.5                | 15.1                | 11.4                | 11.53
| whisper-base.en     | 13.5           | 10.5               | 17.3                | 13.1                | 13.6
| whisper-base        | 20.5           | 13.4               | 18.5                | 14.3                | 16.68
| whisper-tiny.en     | 15.3           | 13.5               | 21.4                | 15.9                | 16.53
| whisper-tiny        | 17.1           | 16.4               | 24.1                | 18.4                | 19.00

**[Original OAI] When not conditioning on previous input:**

Name                  | rev16/test_wer | meanwhile/test_wer | earnings22/test_wer | earnings21/test_wer | **Avg**
----------------------|----------------|--------------------|---------------------|---------------------|---------------------
whisper-large-v2      | 11.4           | 4.8                | 13.8                | 10.5 | 10.13
whisper-medium.en     |         11.2       | 5.6                |        13.97             | 10.8 | 10.39
whisper-medium        | 11.4           | 5.6                | 13.8                | 10.6 | 10.35
whisper-small.en      | 12.0           | 6.6                | 14.7                | 11.4 | 11.18
whisper-small         | 12.0           | 7.5                | 15.0                | 11.4 | 11.48
whisper-base.en       | 13.5           | 10.5               | 17.4                | 13.2 | 13.65
whisper-base          | 14.5           | 13.6               | 18.5                | 14.4 | 15.25
whisper-tiny.en       | 15.3           | 13.6               | 21.7                | 15.9 | 16.63
whisper-tiny          | 17.1           | 17.4               | 24.3                | 18.3 | 19.28

**[This PR] When conditioning on previous input:**

To transform the table in the provided image to the same simplified format, I'll remove the "RTX4090-openai/" and "-Longform-cond-input-test" from the Name column, and round the numerical values to one decimal place. Here is the updated table in text format:

Name              | rev16/test_wer | meanwhile/test_wer | earnings22/test_wer | earnings21/test_wer | **Avg**
------------------|----------------|--------------------|---------------------|---------------------|---------------------
whisper-large-v2  | 12.3           | 4.3                | 15.0                | 11.3 | 10.73
whisper-large     | 12.88          | 5.1                | 14.09                | 12.5 | 10.42
whisper-medium.en | 11.4           | 6.0                | 13.7                | 10.6 | 10.95
whisper-medium    | 11.5           | 5.6                | 14.1                | 10.6 | 10.45
whisper-small.en  | 11.8           | 6.4                | 14.3                | 11.0 | 10.88
whisper-small     | 11.9           | 7.2                | 14.5                | 11.1 | 11.18
whisper-base.en   | 12.8           | 9.5                | 17.7                | 12.3 | 13.08
whisper-base      | 14.0           | 16.6               | 19.0                | 13.8 | 15.85
whisper-tiny.en   | 14.7           | 13.0               | 20.6                | 15.2 | 15.88
whisper-tiny      | 16.5           | 16.2               | 24.5                | 18.5 | 18.93

**[Original OAI] When conditioning on previous input:**

Name                | rev16/test_wer | meanwhile/test_wer | earnings22/test_wer | earnings21/test_wer | **Avg**
--------------------|----------------|--------------------|---------------------|---------------------|---------------------
whisper-large-v2    | 11.4           | 5.5                | 13.8                | 10.5 | 10.3
whisper-large       | 10.5           | 5.3                | 13.2                | 9.7 | 9.68
whisper-medium.en   | 11.2           | 5.5                | 13.9                | 10.8 | 10.35
whisper-medium      | 11.9           | 5.6                | 14.0                | 11.1 | 10.65
whisper-small.en    | 13.5           | 6.4                | 14.8                | 11.1 | 11.45
whisper-small       | 12.2           | 7.4                | 14.6                | 11.0 | 11.3
whisper-base.en     | 13.5           | 10.0               | 17.0                | 12.5 | 13.25
whisper-base        | 14.4           | 13.9               | 18.5                | 14.4 | 15.3
whisper-tiny.en     | 15.3           | 13.5               | 21.6                | 15.9 | 16.58
whisper-tiny        | 17.2           | 18.1               | 24.1                | 18.2 | 19.4

The code to run Transformers Whisper is based on:
```py
 raw_audio = [x["array"].astype(np.float32) for x in batch[data_args.audio_column_name]]
        inputs = processor(raw_audio, truncation=False, padding="longest", return_attention_mask=True, sampling_rate=SAMPLING_RATE, return_tensors="pt")

        if inputs.input_features.shape[-1] < 3000:
            inputs = processor(raw_audio, return_tensors="pt")

        inputs.to("cuda", DTYPE)

        gen_kwargs = {
            "condition_on_prev_tokens": data_args.condition_on_prev_tokens,
            "max_length": 448,
            "return_timestamps": True,
            "num_beams": 1,
            "top_k": 0,
            "compression_ratio_threshold": 1.35, # different compression threshold is used
            "temperature": (0.0, 0.2, 0.4, 0.6, 0.8, 1.0),
            "logprob_threshold": -1.0,
            "no_speech_threshold": 0.6,
        }

        result = model.generate(**inputs,  **gen_kwargs)
        decoded = processor.batch_decode(result, skip_special_tokens=True)
```
<details>

<summary>The whole inference code</summary>

```py
import transformers
import whisper
from transformers import HfArgumentParser, is_wandb_available
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
from whisper.normalizers import EnglishTextNormalizer

SAMPLING_RATE = 16_000

logger = logging.getLogger(__name__)
metric = evaluate.load("wer")


@dataclass
class DataTrainingArguments:
    """
    Arguments pertaining to what data we are going to input our model for training and eval.
    """

    dataset_name: str = field(
        default=None,
        metadata={
            "help": "The name of the dataset to use (via the datasets library). Load and combine "
            "multiple datasets by separating dataset hours by a '+' symbol."
        },
    )
    model_name_or_path: str = field(
        default=None,
        metadata={
            "help": "The name of the model to use (via the transformers library). "
        },
    )
    condition_on_prev_tokens: bool = field(
        default=False,
        metadata={"help": "Whether to condition on previous tokens or not"},
    )
    num_beams: int = field(
        default=1,
        metadata={"help": "The number of beams used for evluation."},
    )
    batch_size: int = field(
        default=1,
        metadata={"help": "Batch size at which the model should be evaluated at."}
    )
    use_fp16: bool = field(
        default=True,
        metadata={"help": "Whether to run the model in fp16 or not"}
    )
    dataset_config_name: Optional[str] = field(
        default=None,
        metadata={"help": "The configuration name of the dataset to use (via the datasets library)."},
    )
    dataset_split_name: Optional[str] = field(
        default=None,
        metadata={"help": "The split name of the dataset to use (via the datasets library)."},
    )
    dataset_cache_dir: Optional[str] = field(
        default=None,
        metadata={"help": "Path to cache directory for saving and loading datasets"},
    )
    audio_column_name: str = field(
        default="audio",
        metadata={"help": "The name of the dataset column containing the audio data. Defaults to 'audio'"},
    )
    text_column_name: str = field(
        default=None,
        metadata={"help": "The name of the dataset column containing the text data. Defaults to `text`."},
    )
    wandb_project: str = field(
        default="distil-whisper-speed-benchmark",
        metadata={"help": "The name of the wandb project."},
    )
    wandb_name: str = field(
        default=None,
        metadata={"help": "The name of the wandb run."},
    )
    wandb_job_type: str = field(
        default="distil-whisper",
        metadata={"help": "The name of the wandb job type."},
    )
    wandb_dir: str = field(
        default=None,
        metadata={"help": "The absolute path to save the wandb logs."},
    )
    save_code_to_wandb: bool =
        default=True,
        metadata={"help": "Whether to use Datasets' streaming mode to load and the data."},
    )


def write_metric(summary_writer, eval_metrics, step, prefix="eval"):
    for metric_name, value in eval_metrics.items():
        summary_writer.scalar(f"{prefix}/{metric_name}", value, step)


def write_wandb_metric(wandb_logger, metrics, train_time, prefix):
    log_metrics = {}
    for k, v in metrics.items():
        log_metrics[f"{prefix}/{k}"] = v
    log_metrics[f"{prefix}/time"] = train_time
    wandb_logger.log(log_metrics)  # TODO(SG): bug with wandb means we can't log the step count

def compute_metrics(pred_str, label_str, normalizer):
    # normalize everything and re-compute the WER
    norm_pred_str = [normalizer(pred) for pred in pred_str]
    norm_label_str = [normalizer(label) for label in label_str]

    wer = 100 * metric.compute(predictions=norm_pred_str, references=norm_label_str)

    return wer

def convert_dataset_str_to_list(
    dataset_names, dataset_config_names, splits=None, text_column_names=None, dataset_hours=None, default_split="train"
):
    if isinstance(dataset_names, str):
        dataset_names = dataset_names.split("+")

        # we assume that all the datasets we're using derive from the distil-whisper org on the Hub - prepend the org name if necessary
        for i in range(len(dataset_names)):
            ds_name = dataset_names[i]
            dataset_names[i] = f"distil-whisper/{ds_name}" if "/" not in ds_name else ds_name

        dataset_config_names = dataset_config_names.split("+")
        splits = splits.split("+") if splits is not None else None
        text_column_names = text_column_names.split("+") if text_column_names is not None else None
        dataset_hours = dataset_hours.split("+") if dataset_hours is not None else None

    # basic checks to ensure we've got the right number of datasets/configs/splits/columns/probs
    if len(dataset_names) != len(dataset_config_names):
        raise ValueError(
            f"Ensure one config is passed for each dataset, got {len(dataset_names)} datasets and"
            f" {len(dataset_config_names)} configs."
        )

    if splits is not None and len(splits) != len(dataset_names):
        raise ValueError(
            f"Ensure one split is passed for each dataset, got {len(dataset_names)} datasets and {len(splits)} splits."
        )

    if text_column_names is not None and len(text_column_names) != len(dataset_names):
        raise ValueError(
            f"Ensure one text column name is passed for each dataset, got {len(dataset_names)} datasets and"
            f" {len(text_column_names)} text column names."
        )

    if dataset_hours is not None:
        if len(dataset_hours) != len(dataset_names):
            raise ValueError(
                f"Ensure one probability is passed for each dataset, got {len(dataset_names)} datasets and "
                f"{len(dataset_hours)} hours."
            )
        dataset_hours = [float(ds_hours) for ds_hours in dataset_hours]
    else:
        dataset_hours = [None] * len(dataset_names)

    text_column_names = (
        text_column_names if text_column_names is not None else ["text" for _ in range(len(dataset_names))]
    )
    splits = splits if splits is not None else [default_split for _ in range(len(dataset_names))]

    dataset_names_dict = []
    for i, ds_name in enumerate(dataset_names):
        dataset_names_dict.append(
            {
                "name": ds_name,
                "config": dataset_config_names[i],
                "split": splits[i],
                "text_column_name": text_column_names[i],
                "hour

def main():
    # 1. Parse input arguments
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.
    parser = HfArgumentParser([DataTrainingArguments])

    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        data_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))[0]
    else:
        data_args = parser.parse_args_into_dataclasses()[0]

    # 2. Setup logging
    # Make one log on every process with the configuration for debugging.
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )

    has_wandb = is_wandb_available()
    if has_wandb:
        import wandb as wandb_logger
        import wandb

        # Set up wandb run
        wandb_logger.init(
            project=data_args.wandb_project,
            name=data_args.wandb_name,
            job_type=data_args.wandb_job_type,
            dir=data_args.wandb_dir,
            save_code=data_args.save_code_to_wandb,
        )
        wandb_logger.log({"torch_version": str(torch.__version__)})
        wandb_logger.log({"transformers_version": str(transformers.__version__)})
        wandb_logger.log({"batch_size": 1})
    else:
        raise ValueError("Wandb logging requires wandb to be installed. Run `pip install wandb` to enable.")

    # 3. Load dataset
    raw_datasets = IterableDatasetDict() if data_args.streaming else DatasetDict()

    # Convert lists of dataset names/configs/splits to a dict
    # names: "librispeech_asr+gigaspeech", configs: "all+l", splits: "validation.clean+validation"
    # -> [{"name: "librispeech_asr": "config": "all", "split": "validation.clean"}, {"name: "gigaspeech": "config": "l", "split": "validation"}
    dataset_names_dict = convert_dataset_str_to_list(
        data_args.dataset_name,
        data_args.dataset_config_name,
        splits=data_args.dataset_split_name,
        text_column_names=data_args.text_column_name,
    )

    if len(dataset_names_dict) == 1:
        # load a single eval set
        dataset_dict = dataset_names_dict[0]
        raw_datasets["eval"] = load_dataset(
            dataset_dict["name"],
            dataset_dict["config"],
            split=dataset_dict["split"],
            cache_dir=data_args.dataset_cache_dir,
            streaming=data_args.streaming,
        )
        if dataset_dict["text_column_name"] not in list(raw_datasets["eval"].features.keys()):
            raise ValueError(
                f"--text column name {dataset_dict['text_column_name']} not found in the evaluation "
                f"dataset {dataset_dict['name']}. Ensure `text_column_name` is set to the correct column "
                f"for the target text. Should be one of {' '.join(list(raw_datasets['eval'].features.keys()))}"
            )
        if dataset_dict["text_column_name"] != "text":
            raw_datasets["eval"] = raw_datasets["eval"].rename_column(dataset_dict["text_column_name"], "text")
    else:
        # load multiple eval sets
        for dataset_dict in tqdm(dataset_names_dict, desc="Loading datasets..."):
            # Clean-up the dataset name for pretty logging
            # ("distil-whisper/librispeech_asr", "validation.clean") -> "librispeech_asr/validation-clean"
            pretty_name = f"{dataset_dict['name'].split('/')[-1]}/{dataset_dict['split'].replace('.', '-')}"
            raw_datasets[pretty_name] = load_dataset(
                dataset_dict["name"],
                dataset_dict["config"],
                split=dataset_dict["split"],
                cache_dir=data_args.dataset_cach
    # so we just need to set the correct target sampling rate.
    raw_datasets = raw_datasets.cast_column(
        data_args.audio_column_name,
        datasets.features.Audio(SAMPLING_RATE),
    )

    # 5. Load model & normalizer
    processor = AutoProcessor.from_pretrained(data_args.model_name_or_path)
    model = AutoModelForSpeechSeq2Seq.from_pretrained(data_args.model_name_or_path, low_cpu_mem_usage=True, torch_dtype=DTYPE)
    model.generation_config.max_initial_timestamp_index = 50
    model.cuda()

    normalizer = EnglishTextNormalizer()

    # 6. Run evaluation
    def evaluate(batch):
        # batch_size has to be 1 for openai/whisper
        raw_audio = [x["array"].astype(np.float32) for x in batch[data_args.audio_column_name]]
        inputs = processor(raw_audio, truncation=False, padding="longest", return_attention_mask=True, sampling_rate=SAMPLING_RATE, return_tensors="pt")

        if inputs.input_features.shape[-1] < 3000:
            inputs = processor(raw_audio, return_tensors="pt")

        inputs.to("cuda", DTYPE)

        gen_kwargs = {
            "condition_on_prev_tokens": data_args.condition_on_prev_tokens,
            "max_length": 448,
            "return_timestamps": True,
            "num_beams": data_args.num_beams,
            "top_k": 0,
            "compression_ratio_threshold": 1.35, # different compression threshold is used
            "temperature": (0.0, 0.2, 0.4, 0.6, 0.8, 1.0),
            "logprob_threshold": -1.0,
            "no_speech_threshold": 0.6,
        }

        result = model.generate(**inputs,  **gen_kwargs)
        decoded = processor.batch_decode(result, skip_special_tokens=True)

        batch["transcription"] = decoded
        batch["reference"] = batch["text"]

        return batch

    result_datasets = DatasetDict()
    for split in raw_datasets:
        map_fn = partial(
            raw_datasets[split].map,
            function=evaluate,
            remove_columns=raw_datasets[split].features.keys(),
            batch_size=data_args.batch_size,
            batched=True,
        )

        result_datasets[split] = (
            map_fn(num_proc=1, desc="benchmark eval dataset")
            if not data_args.streaming
            else map_fn()
        )

    # 7. Compute WER and upload
    count = 0
    for split in result_datasets:
        transcriptions = []
        references = []
        all_wers = []

        if data_args.streaming:
            result_iter = iter(result_datasets[split])

        for result in result_iter:
            transcriptions.append(result["transcription"])
            references.append(result["reference"])
            try:
                all_wers.append(compute_metrics(transcriptions[-1:], references[-1:], normalizer))
            except:
                all_wers.append(None)

            count += 1
            print(f"Processed {count} samples...")

        log_stats = {
            f"{split}_wer": compute_metrics(transcriptions, references, normalizer),
            f"{split}_all_wer": all_wers,
        }
        wandb_logger.log(log_stats)

    print("Done!")


if __name__ == "__main__":
    main()
```

</details>


The code to run the OpenAI model is based on:

```py
raw_audio = batch[data_args.audio_column_name][0]["array"]
raw_audio = raw_audio.astype(np.float32)

out_dict = model.transcribe(raw_audio, condition_on_previous_text=data_args.condition_on_prev_tokens, language="en")

batch["transcription"] = [out_dict["text"]]
batch["reference"] = batch["text"]
```


<details>

<summary>The whole inference code</summary>

```py
"""
Evaluating a Whisper model on one or more evaluation datasets.
"""
# You can also adapt this script for your own speech recognition validation. Pointers for this are left as comments.

from dataclasses import dataclass
import logging
import os
from typing import Optional
import numpy as np
import sys
from dataclasses import field
from functools import partial

import datasets
import evaluate
import torch
from datasets import DatasetDict, IterableDatasetDict, load_dataset
from tqdm import tqdm
import transformers
import whisper
from transformers import HfArgumentParser, is_wandb_available
from whisper.normalizers import EnglishTextNormalizer

SAMPLING_RATE = 16_000

logger = logging.getLogger(__name__)
metric = evaluate.load("wer")


@dataclass
class DataTrainingArguments:
    """
    Arguments pertaining to what data we are going to input our model for training and eval.
    """

    dataset_name: str = field(
        default=None,
        metadata={
            "help": "The name of the dataset to use (via the datasets library). Load and combine "
            "multiple datasets by separating dataset hours by a '+' symbol."
        },
    )
    model_name_or_path: str = field(
        default=None,
        metadata={
            "help": "The name of the model to use (via the transformers library). "
        },
    )
    condition_on_prev_tokens: bool = field(
        default=False,
        metadata={"help": "Whether to condition on previous tokens or not"},
    )
    num_beams: int = field(
        default=1,
        metadata={"help": "The number of beams used for evluation."},
    )
    dataset_config_name: Optional[str] = field(
        default=None,
        metadata={"help": "The configuration name of the dataset to use (via the datasets library)."},
    )
    dataset_split_name: Optional[str] = field(
        default=None,
        metadata={"help": "The split name of the dataset to use (via the datasets library)."},
    )
    dataset_cache_dir: Optional[str] = field(
        default=None,
        metadata={"help": "Path to cache directory for saving and loading datasets"},
    )
    audio_column_name: str = field(
        default="audio",
        metadata={"help": "The name of the dataset column containing the audio data. Defaults to 'audio'"},
    )
    text_column_name: str = field(
        default=None,
        metadata={"help": "The name of the dataset column containing the text data. Defaults to `text`."},
    )
    wandb_project: str = field(
        default="distil-whisper-speed-benchmark",
        metadata={"help": "The name of the wandb project."},
    )
    wandb_name: str = field(
        default=None,
        metadata={"help": "The name of the wandb run."},
    )
    wandb_job_type: str = field(
        default="distil-whisper",
        metadata={"help": "The name of the wandb job type."},
    )
    wandb_dir: str = field(
        default=None,
        metadata={"help": "The absolute path to save the wandb logs."},
    )
    save_code_to_wandb: bool = field(
        default=True,
        metadata={
            "help": (
                "Whether to save main script to wandb. This is valuable for improving"
                " experiment reproducibility and to diff code across experiments in"
                " the UI."
            )
        },
    )
    streaming: bool = field(
        default=True,
        metadata={"help": "Whether to use Datasets' streaming mode to load and the data."},
    )
    max_eval_samples: Optional[int] = field(
        default=None,
        metadata={"help": "For debugging purposes, truncate the number of eval examples to this value if set."},
    )


def write_metric(summary_writer, eval_metrics, step, prefix="eval"):
    for metric_name, value in eval_metrics.items():
        summary_writer.scalar(f"{prefix}/{metric_name}", value, step)


def write_wandb_metric(wandb_logger, metrics, train_time, prefix):
    log_metrics = {}
    for k, v in metrics.items():
        log_metrics[f"{prefix}/{k}"] = v
    log_metrics[f"{prefix}/time"] = train_time
    wandb_logger.log(log_metrics)  # TODO(SG): bug with wandb means we can't log the step count

def compute_metrics(pred_str, label_str, normalizer):
    # normalize everything and re-compute the WER
    norm_pred_str = [normalizer(pred) for pred in pred_str]
    norm_label_str = [normalizer(label) for label in label_str]

    wer = 100 * metric.compute(predictions=norm_pred_str, references=norm_label_str)

    return wer



def convert_dataset_str_to_list(
    dataset_names, dataset_config_names, splits=None, text_column_names=None, dataset_hours=None, default_split="train"
):
    if isinstance(dataset_names, str):
        dataset_names = dataset_names.split("+")

        # we assume that all the datasets we're using derive from the distil-whisper org on the Hub - prepend the org name if necessary
        for i in range(len(dataset_names)):
            ds_name = dataset_names[i]
            dataset_names[i] = f"distil-whisper/{ds_name}" if "/" not in ds_name else ds_name

        dataset_config_names = dataset_config_names.split("+")
        splits = splits.split("+") if splits is not None else None
        text_column_names = text_column_names.split("+") if text_column_names is not None else None
        dataset_hours = dataset_hours.split("+") if dataset_hours is not None else None

    # basic checks to ensure we've got the right number of datasets/configs/splits/columns/probs
    if len(dataset_names) != len(dataset_config_names):
        raise ValueError(
            f"Ensure one config is passed for each dataset, got {len(dataset_names)} datasets and"
            f" {len(dataset_config_names)} configs."
        )

    if splits is not None and len(splits) != len(dataset_names):
        raise ValueError(
            f"Ensure one split is passed for each dataset, got {len(dataset_names)} datasets and {len(splits)} splits."
        )

    if text_column_names is not None and len(text_column_names) != len(dataset_names):
        raise ValueError(
            f"Ensure one text column name is passed for each dataset, got {len(dataset_names)} datasets and"
            f" {len(text_column_names)} text column names."
        )

    if dataset_hours is not None:
        if len(dataset_hours) != len(dataset_names):
            raise ValueError(
                f"Ensure one probability is passed for each dataset, got {len(dataset_names)} datasets and "
                f"{len(dataset_hours)} hours."
            )
        dataset_hours = [float(ds_hours) for ds_hours in dataset_hours]
    else:
        dataset_hours = [None] * len(dataset_names)

    text_column_names = (
        text_column_names if text_column_names is not None else ["text" for _ in range(len(dataset_names))]
    )
    splits = splits if splits is not None else [default_split for _ in range(len(dataset_names))]

    dataset_names_dict = []
    for i, ds_name in enumerate(dataset_names):
        dataset_names_dict.append(
            {
                "name": ds_name,
                "config": dataset_config_names[i],
                "split": splits[i],
                "text_column_name": text_column_names[i],
                "hours": dataset_hours[i],
            }
        )
    return dataset_names_dict




def main():
    # 1. Parse input arguments
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.
    parser = HfArgumentParser([DataTrainingArguments])

    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        data_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))[0]
    else:
        data_args = parser.parse_args_into_dataclasses()[0]

    # 2. Setup logging
    # Make one log on every process with the configuration for debugging.
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )

    has_wandb = is_wandb_available()
    if has_wandb:
        import wandb as wandb_logger
        import wandb

        # Set up wandb run
        wandb_logger.init(
            project=data_args.wandb_project,
            name=data_args.wandb_name,
            job_type=data_args.wandb_job_type,
            dir=data_args.wandb_dir,
            save_code=data_args.save_code_to_wandb,
        )
        wandb_logger.log({"torch_version": str(torch.__version__)})
        wandb_logger.log({"transformers_version": str(transformers.__version__)})
        wandb_logger.log({"batch_size": 1})
    else:
        raise ValueError("Wandb logging requires wandb to be installed. Run `pip install wandb` to enable.")

    # 3. Load dataset
    raw_datasets = IterableDatasetDict() if data_args.streaming else DatasetDict()

    # Convert lists of dataset names/configs/splits to a dict
    # names: "librispeech_asr+gigaspeech", configs: "all+l", splits: "validation.clean+validation"
    # -> [{"name: "librispeech_asr": "config": "all", "split": "validation.clean"}, {"name: "gigaspeech": "config": "l", "split": "validation"}
    dataset_names_dict = convert_dataset_str_to_list(
        data_args.dataset_name,
        data_args.dataset_config_name,
        splits=data_args.dataset_split_name,
        text_column_names=data_args.text_column_name,
    )

    if len(dataset_names_dict) == 1:
        # load a single eval set
        dataset_dict = dataset_names_dict[0]
        raw_datasets["eval"] = load_dataset(
            dataset_dict["name"],
            dataset_dict["config"],
            split=dataset_dict["split"],
            cache_dir=data_args.dataset_cache_dir,
            streaming=data_args.streaming,
        )
        if dataset_dict["text_column_name"] not in list(raw_datasets["eval"].features.keys()):
            raise ValueError(
                f"--text column name {dataset_dict['text_column_name']} not found in the evaluation "
                f"dataset {dataset_dict['name']}. Ensure `text_column_name` is set to the correct column "
                f"for the target text. Should be one of {' '.join(list(raw_datasets['eval'].features.keys()))}"
            )
        if dataset_dict["text_column_name"] != "text":
            raw_datasets["eval"] = raw_datasets["eval"].rename_column(dataset_dict["text_column_name"], "text")
    else:
        # load multiple eval sets
        for dataset_dict in tqdm(dataset_names_dict, desc="Loading datasets..."):
            # Clean-up the dataset name for pretty logging
            # ("distil-whisper/librispeech_asr", "validation.clean") -> "librispeech_asr/validation-clean"
            pretty_name = f"{dataset_dict['name'].split('/')[-1]}/{dataset_dict['split'].replace('.', '-')}"
            raw_datasets[pretty_name] = load_dataset(
                dataset_dict["name"],
                dataset_dict["config"],
                split=dataset_dict["split"],
                cache_dir=data_args.dataset_cache_dir,
                streaming=data_args.streaming,
            )
            if dataset_dict["text_column_name"] not in list(raw_datasets[pretty_name].features.keys()):
                raise ValueError(
                    f"`--text_column_name` {dataset_dict['text_column_name']} not found in the evaluation "
                    f"dataset {dataset_dict['name']}. Ensure `text_column_name` is set to the correct column "
                    f"for the target text. Should be one of {' '.join(list(raw_datasets[pretty_name].features.keys()))}"
                )
            if dataset_dict["text_column_name"] != "text":
                raw_datasets[pretty_name] = raw_datasets[pretty_name].rename_column(
                    dataset_dict["text_column_name"], "text"
                )

    # 4. Resample speech dataset: `datasets` takes care of automatically loading and resampling the audio,
    # so we just need to set the correct target sampling rate.
    raw_datasets = raw_datasets.cast_column(
        data_args.audio_column_name,
        datasets.features.Audio(SAMPLING_RATE),
    )

    # 5. Load model & normalizer
    model_name = data_args.model_name_or_path.split("/")[-1].split("whisper-")[-1]

    model = whisper.load_model(model_name)
    model.cuda()

    normalizer = EnglishTextNormalizer()

    # 6. Run evaluation
    def evaluate(batch):
        # batch_size has to be 1 for openai/whisper
        raw_audio = batch[data_args.audio_column_name][0]["array"]
        raw_audio = raw_audio.astype(np.float32)

        # generate
        out_dict = model.transcribe(raw_audio, condition_on_previous_text=data_args.condition_on_prev_tokens, language="en")

        batch["transcription"] = [out_dict["text"]]
        batch["reference"] = batch["text"]

        return batch

    result_datasets = DatasetDict()
    for split in raw_datasets:
        map_fn = partial(
            raw_datasets[split].map,
            function=evaluate,
            remove_columns=raw_datasets[split].features.keys(),
            batch_size=1,
            batched=True,
        )

        result_datasets[split] = (
            map_fn(num_proc=1, desc="benchmark eval dataset")
            if not data_args.streaming
            else map_fn()
        )

    # 7. Compute WER and upload
    count = 0
    for split in result_datasets:
        transcriptions = []
        references = []

        if data_args.streaming:
            result_iter = iter(result_datasets[split])

        for result in result_iter:
            transcriptions.append(result["transcription"])
            references.append(result["reference"])

            count += 1
            print(f"Processed {count} samples...")

        log_stats = {f"{split}_wer": compute_metrics(transcriptions, references, normalizer)}
        wandb_logger.log(log_stats)

    print("Done!")


if __name__ == "__main__":
    main()
```

</details>
</details>

## Review

To make this review a bit easier, let me explain first why we need so much code. Very much simplified we need the following loops for long-form generation with temperature fallback in Whisper:

```
# pseudo-code
for segment in audio:
    for temperature in (0.0, 0.2, 0.4, 0.6, 0.8. 1.0):
        tokens = model.generate(segment, temperature=temperature)

        for batch_idx in tokens.shape[0]:
            # find slices that need temperature fallback
            needs_fallback = ...

        if not any(needs_fallback):
            break (out of temperature loop)
```

Temperature fallback is already quite tricky as it means we have to dynamically re-generate certain segments. This coupled with batched generation is even trickier as it means we often only have to do this for some segments in the batch, but not all. Hence we need a couple of nested for-loops here.

Because the generation code has become so complex, I moved all the Whisper-specific generation code int `generation_whisper.py`. I've added new tests making sure that everything works correctly an ran all slow tests to make sure I didn't break anything. 
Failing slow tests for Whisper are due to unrelated issues such as this: https://github.com/huggingface/transformers/pull/27492 one.

To make the whole Whisper generation function easier to read, I've split the big generation function into multiple static private functions to help readability.

## Next steps

- [ ] Once approved & before merging, all of the official Whisper model cards should be updated as a couple of generation parameters should be added to the model's generation config
- [ ] While SOTA batched long-form generation is now functional, it's far from easy to understand what's going on here. Since Whisper is so highly used & important, I'm planning on writing some in-detail blog post / docs after this PR is merged
- [ ] The way we deal with `forced_decoder_input_ids` is quite suboptimal which can also be seen from this issue: https://github.com/huggingface/transformers/issues/28228 . In a follow-up PR this should be refactored so that we only make use of `decoder_input_ids` instead of relying on `decoder_input_ids`
- [ ] Contact OAI to see if a joint blog post could make sense here


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27658). All of your documentation changes will be reflected on that endpoint.

Follow-up of: https://github.com/huggingface/transformers/pull/27492

Failing test is a timeout which should not happen when running again

> This was a fun PR to review 🔥 Really cool feature -- and a lot of impactful work done here 💪
> 
> Questions:
> 
> 1. I was recaping the features from section 4.5 in the paper at the end of the review, and I can't find the part corresponding to "initial timestamp constraint". Are these changes in the PR?
> 2. I see that you dynamically remove rows for a given batch as their computation becomes redundant (no more data or no need for fallback), which is something that I have been considering for the main `generate()` method. Did you notice a significant speed increase?
> 
> Suggestion for a future PR: move Whisper's logits processors into their own file under `models/whisper/logits_process.py` and, more importantly, add their documentation to the Whisper doc page. That way, all Whisper functionality becomes more self-contained, instead of being mixed up with generalist methods 🤗

Thanks for the extensive review and for diving into the complex heuristics here 😅 

1.) Yes, so we already make sure of the initial_timestamp_contraint actually [here](https://github.com/huggingface/transformers/blob/6c78bbcb8320d316434262ef003251ca997db0d1/src/transformers/generation/logits_process.py#L1867) and have it (incorrectly) set in our generation configs [here](https://huggingface.co/openai/whisper-large-v2/blob/696465c62215e36a9ab3f9b7672fe7749f1a1df5/generation_config.json#L215) . The correct value is 50 which is what I've hardcoded in my decoding experiments for now (see [here](https://github.com/patrickvonplaten/whisper-long-form/blob/b599d4df1e88c255cb0e59386eb2a778d51f009f/run_whisper_transformers.py#L328)), but I'll update the model cards once this PR is merged.

2.) Yes, nice observation! I've ran most experiments on a RTX4090 or A100 and tbh didn't really see a huge speed-up as soon as a row is removed (but probably also because these GPUs are quite good at parallelization. Think on a T4 I'd see a bigger speed up). But it should def give a speed-up! The downside here is that it's not great for torch compile (cc @ArthurZucker) but I think this is a bit outside of this PR and could be added in a future PR.

> Suggestion for a future PR: move Whisper's logits processors into their own file under models/whisper/logits_process.py and, more importantly, add their documentation to the Whisper doc page. That way, all Whisper functionality becomes more self-contained, instead of being mixed up with generalist methods 🤗
=> Agree! I'm happy to move the logits processors actually directly into the whisper model directory in this PR if that's cleaner (cc @sanchit-gandhi @ArthurZucker) 


> Looks great! Thanks for working on this complex batched generation logic. The only difference I'm observing is how we handle timestamps > 30s.
> 
> In the original Whisper package, we always offset predicted timestamps above 30s such that they correspond to the actual audio timing, e.g.
> 
> ```python
> import torch
> from datasets import load_dataset
> from whisper import load_model, transcribe
> from whisper.utils import WriteVTT
> 
> model = load_model("tiny.en")
> 
> dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
> sample = dataset[0]["audio"]["array"]
> sample = torch.from_numpy(sample).float()
> 
> pred_out = transcribe(model, audio=sample, condition_on_previous_text=False, temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0))
> 
> writer = WriteVTT(output_dir="./transcription")
> 
> for start, end, text in writer.iterate_result(pred_out):
>     print(f"{start} --> {end}  {text}")
> ```
> 
> **Print Output:**
> 
> ```
> 00:00.000 --> 00:06.480  Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.
> 00:06.480 --> 00:11.280  Nor is Mr. Quilter's manner less interesting than his matter.
> 00:11.280 --> 00:16.840  He tells us that at this festive season of the year, with Christmas and roast beef looming
> 00:16.840 --> 00:23.760  before us, similes drawn from eating and its results occur most readily to the mind.
> 00:23.760 --> 00:29.440  He has grave doubts whether Sir Frederick Layton's work is really Greek after all, and
> 00:29.440 --> 00:33.760  can discover in it but little of rocky Ithaca.
> 00:33.760 --> 00:39.800  Linnell's pictures are a sort of up-gards and atom paintings, and Mason's exquisite
> 00:39.800 --> 00:44.720  idles are as national as a jingo poem.
> 00:44.720 --> 00:50.360  Mr. Burkett Foster's landscapes smile at one much in the same way that Mr. Carker used
> 00:50.360 --> 00:52.960  to flash his teeth.
> 00:52.960 --> 00:57.600  Mr. John Collier gives his sitter a cheerful slap in the back.
> 00:57.600 --> 01:01.240  Before he says, like a shampooer and a Turkish bath,
> 01:01.240 --> 01:02.080  next man.
> ```
> 
> Whereas currently in Transformers, any timestamps above 30s are reset back to zero:
> 
> ```python
> from datasets import load_dataset
> from transformers import WhisperProcessor, WhisperForConditionalGeneration
> 
> dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
> sample = dataset[0]["audio"]["array"]
> 
> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en")
> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny.en")
> 
> inputs = processor(sample, return_tensors="pt", truncation=False, padding="longest", return_attention_mask=True, sampling_rate=16_000)
> 
> pred_ids = model.generate(**inputs, temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0), logprob_threshold=-1.0, no_speech_threshold=0.6, compression_ratio_threshold=1.35)
> 
> pred_str = processor.batch_decode(pred_ids, skip_special_tokens=True, decode_with_timestamps=True)[0]
> 
> cur_segment = pred_str[0]
> prev_char = pred_str[0]
> for char in pred_str[1:]:
>     if prev_char == ">" and char == "<":
>         print(cur_segment)
>         cur_segment = char
>     else:
>         cur_segment += char
>     prev_char = char
> ```
> 
> **Print Output:**
> 
> ```
> <|0.00|> Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.<|6.48|>
> <|6.48|> Nor is Mr. Quilter's manner less interesting than his matter.<|11.28|>
> <|11.28|> He tells us that at this festive season of the year, with Christmas and roast beef looming<|16.84|>
> <|16.84|> before us, similes drawn from eating and its results occur most readily to the mind.<|23.76|>
> <|0.00|> He has grave doubts whether Sir Frederick Layton's work is really Greek after all, and<|5.68|>
> <|5.68|> can discover in it but little of rocky Ithaca.<|10.00|>
> <|10.00|> Linnell's pictures are a sort of up-gards and atom paintings, and Mason's exquisite<|16.04|>
> <|16.04|> idles are as national as a jingo poem.<|20.96|>
> <|20.96|> Mr. Burkett Foster's landscapes smile at one much in the same way that Mr. Carker used<|26.60|>
> <|26.60|> to flash his teeth.<|29.20|>
> <|0.00|> Mr. John Collier gives his sitter a cheerful slap in the back.<|4.68|>
> ```
> 
> This can be rectified by keeping track of the time offset for each element in the batch: https://github.com/openai/whisper/blob/ba3f3cd54b0e5b8ce1ab3de13e32122d0d5f98ab/whisper/transcribe.py#L271

Nice observation. I've changed a couple lines of code in Whisper's tokenizer and now we're getting the same formatting:

```
<|0.00|> Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.<|6.48|>
<|6.48|> Nor is Mr. Quilter's manner less interesting than his matter.<|11.28|>
<|11.28|> He tells us that at this festive season of the year, with Christmas and roast beef looming<|16.84|>
<|16.84|> before us, similes drawn from eating and its results occur most readily to the mind.<|23.76|>
<|23.76|> He has grave doubts whether Sir Frederick Layton's work is really Greek after all, and<|29.44|>
<|29.44|> can discover in it but little of rocky Ithaca.<|33.76|>
<|33.76|> Linnell's pictures are a sort of up-gards and atom paintings, and Mason's exquisite<|39.80|>
<|39.80|> idles are as national as a jingo poem.<|44.72|>
<|44.72|> Mr. Burkett Foster's landscapes smile at one much in the same way that Mr. Carker used<|50.36|>
<|50.36|> to flash his teeth.<|52.96|>
<|52.96|> Mr. John Collier gives his sitter a cheerful slap in the back.<|57.64|>
```

I don't see how the failing test can be related to changes in Whisper's generate or tokenizer method. I'm also not able to reproduce them locally.

```
FAILED tests/test_modeling_utils.py::ModelUtilsTest::test_model_from_pretrained_with_different_pretrained_model_name - AssertionError: False is not true
FAILED tests/test_modeling_utils.py::ModelUtilsTest::test_unexpected_keys_warnings - AssertionError: "were not used when initializing ModelWithHead: ['added_key']" not found in ''
[Ensure a warning is shown when the input_ids start with a pad_token_id.] SUBFAIL tests/test_modeling_utils.py::ModelUtilsTest::test_warn_if_padding_and_no_attention_mask - AssertionError: 'We strongly recommend passing in an `attention_mask`' not found in ''
[Ensure a warning is shown when the input_ids end with a pad_token_id.] SUBFAIL tests/test_modeling_utils.py::ModelUtilsTest::test_warn_if_padding_and_no_attention_mask - AssertionError: 'We strongly recommend passing in an `attention_mask`' not found in ''
[Ensure that the warning is shown at most once.] SUBFAIL tests/test_modeling_utils.py::ModelUtilsTest::test_warn_if_padding_and_no_attention_mask - AssertionError: 0 != 1
[Ensure a different warning is shown when the pad_token_id is equal to the bos_token_id.] SUBFAIL tests/test_modeling_utils.py::ModelUtilsTest::test_warn_if_padding_and_no_attention_mask - AssertionError: 'You may ignore this warning if your `pad_token_id`' not found in ''
```

@patrickvonplaten They're not related and they're affecting other PRs e.g. [this PR](https://github.com/huggingface/transformers/pull/28214). Internal thread: https://huggingface.slack.com/archives/C01NE71C4F7/p1705082667041509

@patrickvonplaten A fix has been merged into main which resolves the flaky logging tests:  https://github.com/huggingface/transformers/commit/0754217c82e5c640c6269d4d0ddc99203b3fd99b

All slow tests of Whisper are passing -> merging! 

@sanchit-gandhi open PRs for the model repos to change generation configs:
Pr created at https://huggingface.co/openai/whisper-large-v3/discussions/69
Pr created at https://huggingface.co/openai/whisper-large-v2/discussions/95
Pr created at https://huggingface.co/openai/whisper-base.en/discussions/17
Pr created at https://huggingface.co/openai/whisper-tiny.en/discussions/24
Pr created at https://huggingface.co/openai/whisper-small/discussions/37
Pr created at https://huggingface.co/openai/whisper-base/discussions/29
Pr created at https://huggingface.co/openai/whisper-tiny/discussions/39
Pr created at https://huggingface.co/openai/whisper-small.en/discussions/16
Pr created at https://huggingface.co/openai/whisper-medium/discussions/32
Pr created at https://huggingface.co/openai/whisper-large/discussions/48
Pr created at https://huggingface.co/openai/whisper-medium.en/discussions/16
Pr created at https://huggingface.co/distil-whisper/distil-medium.en/discussions/11
Pr created at https://huggingface.co/distil-whisper/distil-large-v2/discussions/23
Pr created at https://huggingface.co/distil-whisper/distil-small.en/discussions/8

=> Would be great if you could check and merge.

Great new feature. Is there a way to get word-level timestamps when using the above long-form transcription? I know it can give segment-level timestamps, but word-level timestamps would be very useful. 

> a way to get word-level timestamps when using the above long-form transcription? I know it can give segment-level timestamps, but word-level timestamps would be very useful.

That would be a great feature addition

thank you very much for the contribution @patrickvonplaten 
i am pretty new to coding.  Since this PR is marked as merged, does it mean if I use
```
pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=128,
    chunk_length_s=30,
    batch_size=16,
    return_timestamps=True,
    torch_dtype=torch_dtype,
    device=device,
)
```

as given in https://huggingface.co/openai/whisper-large-v3

I will automatically use the batched sequential method here, or is something else required?

the reason I ask is, I followed your code in the first post and successfully processed the data from datasets module
however, when I try to read a mp3 file using librosa 

```
raw_audio, sampling_rate = librosa.load(file_path, sr=16000, mono=True)
inputs = processor([raw_audio], return_tensors="pt", truncation=False, padding="longest", return_attention_mask=True, sampling_rate=16_000)
result = model.generate(**inputs, condition_on_prev_tokens=False, temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0), logprob_threshold=-1.0, compression_ratio_threshold=1.35, return_timestamps=True)
decoded = processor.batch_decode(result, skip_special_tokens=True)
```
I get an error 
```
    result = model.generate(**inputs, condition_on_prev_tokens=False, temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0), logprob_threshold=-1.0, compression_ratio_threshold=1.35, return_timestamps=True)
  File "C:\Users\r\AppData\Roaming\Python\Python311\site-packages\transformers\models\whisper\generation_whisper.py", line 614, in generate
    decoder_input_ids, kwargs = self._prepare_decoder_input_ids(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\r\AppData\Roaming\Python\Python311\site-packages\transformers\models\whisper\generation_whisper.py", line 1322, in _prepare_decoder_input_ids
    decoder_input_ids = torch.cat([t * one_tensor for t in init_tokens], dim=-1)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\r\AppData\Roaming\Python\Python311\site-packages\transformers\models\whisper\generation_whisper.py", line 1322, in <listcomp>
    decoder_input_ids = torch.cat([t * one_tensor for t in init_tokens], dim=-1)

TypeError: unsupported operand type(s) for *: 'NoneType' and 'Tensor'
```
I suspected it had something to do with how librosa read my file but I printed the output for raw_audio and it was an array of float32 which seems to be what is needed

Hey @revantemp3,

Could you maybe open a new issue and ping me and @sanchit-gandhi on it? I'm sadly not able to reproduce the error when running the code linked [here](https://github.com/huggingface/transformers/pull/27658#issue-2006844016).

Hey @patrickvonplaten ,  can your approach also work for short-form transcription, what is for audio <30 seconds? 

Just wanted to understand if there are any other reasons beyond just sticking to legacy code for not using your method for <30 second audio? 

I don't understand, if you have less than 30sec you can just pass it to the model and it will predict everything in one go, that is the default mode

> I don't understand, if you have less than 30sec you can just pass it to the model and it will predict everything in one go, that is the default mode

Being new to transformers and trying to understand how to deal with repetitions, I had a similar question. Does generating with fallback not apply for shortform to automatically try different temperatures? Beam search seems to be helping but trying to wrap my ahead around why fallback and all it's associated parameters only apply to longform. Or is it just that you should do the fallback yourself if it's shortform?

I think you could try to play with `is_shortform = total_input_frames <= num_segment_frames`  but otherwise no, temperature fallback does not apply to short form

> I think you could try to play with `is_shortform = total_input_frames <= num_segment_frames` but otherwise no, temperature fallback does not apply to short form

Conceptually why is that? Why wouldn't checking the compression ratio and retrying apply to short audio? I believe faster-whisper does it for all audio lengths, haven't looked at the upstream openai implementation yet.

If openai does not do it then this would be a feature request and it is a design choice. Feel free to open an issue for the feature request! 🤗 

> If openai does not do it then this would be a feature request and it is a design choice.

It looks like all code paths lead to [decode_with_fallback](https://github.com/openai/whisper/blob/main/whisper/transcribe.py#L178) in the OpenAI implementation. I don't see any distinguishing between short and long audio.



Interesting. That seems new to me, cc @sanchit-gandhi  and @ylacombe might be worth adding this path as well for short generation WDYT?  

Indeed, the original codebase always uses fallback, for both short and long-form generation. It should be quite straightforward to update the Transformers codebase to also do this. Could you open a new issue to discuss this proposal @Kimahriman, such that we can better keep track of it? 

Made https://github.com/huggingface/transformers/issues/29508