OpenAI/Whisper-large-v2 - Transcription & ONNX inference
Author: Kirankumar2609
Pull request: no

### System Info

- `transformers` version: 4.26.0.dev0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.9.12
- Huggingface_hub version: 0.11.0
- PyTorch version (GPU?): 1.11.0 (False)
- Tensorflow version (GPU?): 2.10.1 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@sanchit-gandhi 



### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

[OpenAI-Whisper_ONNX_Implementation.zip](https://github.com/huggingface/transformers/files/10317927/OpenAI-Whisper_ONNX_Implementation.zip)


### Expected behavior

Audio is not transcribed/translated post the 30 second mark. If the method mentioned in https://huggingface.co/openai/whisper-large-v2/discussions/7#6398809b11095028d87b16a2 is followed, there is no issue. But if the method mentioned in the model card is followed, the above error arises. I need this to be solved as the ONNX version requires input ('input_features', 'decoder_input_ids') in the form of arrays. 

Also, if I use the model.onnx (as shown in the attached zip file) for prediction, it returns an array with float values. Can you help in decoding those values to the transcribed/translated text.



Comments:
Hey @Kirankumar2609! Thanks for linking the reproducible code snippet!

The fact that audios are not transcribed beyond the 30s mark is not a bug with the Whisper model. Rather, it's a pre-defined characteristic of the system. OpenAI designed Whisper such that all input audio sequences are padded / truncated to 30s prior to being passed to the model. This way, the model is only ever required to deal with inputs of fixed length (30s).

<details>

<summary> 

Excerpt from [blog post](https://huggingface.co/blog/fine-tune-whisper#load-whisperfeatureextractor):

</summary>

> Samples shorter than 30s are padded to 30s by appending zeros to the end of the sequence (zeros in an audio signal corresponding to no signal or silence). Samples longer than 30s are truncated to 30s. Since all elements in the batch are padded/truncated to a maximum length in the input space, we don't require an attention mask when forwarding the audio inputs to the Whisper model. Whisper is unique in this regard - with most audio models, you can expect to provide an attention mask that details where sequences have been padded, and thus where they should be ignored in the self-attention mechanism. Whisper is trained to operate without an attention mask and infer directly from the speech signals where to ignore the inputs.

</details>

So, when audio samples longer than 30s are not transcribed it's due to the fact that the audio inputs are being truncated to 30s. This is somewhat suboptimal for a generalisable ASR system: ideally, we want a system that can handle audio inputs of arbitrary length! This is where `pipeline` comes it. `pipeline` chunks the audio samples into 30s blocks, generates the transcriptions for each chunk, and uses a novel 'stitching' algorithm to piece the transcriptions together. This way, we can transcribe audios of arbitrary length!

The code snippet you've provided performs **one forward pass** of the ONNX Whisper model. That is why you're required to pass the `decoder_input_ids` as well as the `input_features`. For **auto-regressive generation**, we only require the `input_features`. We perform one forward pass of the encoder and auto-regressively generate using the decoder. Could you ask in the [optimum](https://github.com/huggingface/optimum) repository if you require help getting this to work with the exported ONNX model please? The corresponding transformers code can be found here: https://github.com/openai/whisper/discussions/654

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.