[Whisper] Computing features on GPU in batch mode for whisper feature extractor.   
Author: vaibhavagg303
Pull request: yes

# What does this PR do?

This pull request introduces a feature enabling the computation of audio features for the Whisper model in batch mode on the GPU. This enhancement substantially reduces latency during feature extraction, thereby enhancing overall performance.
cc: @yashjogi

@sanchit-gandhi @ArthurZucker @hollance  Please look into it.

fixes #29901

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @kamilakesbi for viz

https://github.com/huggingface/transformers/issues/29901
The details regarding the time improvement are mentioned in this issue. It reduces the complete inference time for Whisper-small by **around 25%**, which is significant improvement considering the overall latency of Whisper models. And we also see a significant time decrease in extracting features from 1.5 seconds ( without batch and CPU ) to 0.02 seconds ( by using batch and GPU ). 
@sanchit-gandhi @ArthurZucker 

Thanks for the benchmark link @vaibhavagg303, that's most helpful! Once follow-up question to your benchmark: you mention that torch gpu stft "cuts the average time for batches of 8 from 1.5 seconds to 0.25 seconds". How does the compute time change for bsz=1, since this is also a common use-case for computing features in short-form mode (<30s).

As a follow-up PR: we'll need to update the ASR pipeline class to compute batched input features to leverage this speed-up, since currently it uses bsz=1 always for the feature extraction
https://github.com/huggingface/transformers/blob/9b0a8ea7d1d6226b76cfdc645ce65e21157e2b50/src/transformers/pipelines/automatic_speech_recognition.py#L66

> Thanks for the benchmark link @vaibhavagg303, that's most helpful! Once follow-up question to your benchmark: you mention that torch gpu stft "cuts the average time for batches of 8 from 1.5 seconds to 0.25 seconds". How does the compute time change for bsz=1, since this is also a common use-case for computing features in short-form mode (<30s).

This is the average computation time for **8 audios.**
| Mode | Time |
|-------|------|
| CPU Batch Size 1 |  1.5 seconds |
| CPU Batch Size 8 |  0.25 seconds |
| GPU Batch Size 8 | 0.02 seconds |

Thanks @yashjogi. To clarify my question above, what's the speed up of `CPU Batch Size 1` vs `GPU Batch Size 1`?



> Thanks @yashjogi. To clarify my question above, what's the speed up of `CPU Batch Size 1` vs `GPU Batch Size 1`?

It's around `0.02291` seconds for `GPU Batch Size 1` @sanchit-gandhi 

Also, we found out that this is a huge bottleneck for training Whisper -- this simple change reduced the training time by almost 9 times, depending on the CPU resources of the machine we're training on. This means that it would significantly decrease the time **to train distil-whisper** as well! https://github.com/huggingface/distil-whisper/blob/main/training/run_distillation.py
@sanchit-gandhi 

Hi @ArthurZucker, can you please review and approve these changes? 

Thanks, @sanchit-gandhi  @ArthurZucker

Thanks for your contribution @vaibhavagg303!