what audio input files are supported by ASR model "openai/whisper-large-v2" on huggingface?
Author: pythonvijay
Pull request: no

Hi,

I am trying to use hugging face pipeline to invoke the model "openai/whisper-large-v2' to perform ASR task. Theerafter, I am wrapping the model within gradio as below but when I submit, I get an error. 

Model card of this model says "To transcribe audio samples, the model has to be used alongside a [WhisperProcessor]"

How do I know what audio input is supported by this model and also how can I convert my audio sample (.m4a file) default generated by macbook voice memos app to a format supported by this model?

code as here:

```
from transformers import pipeline
import gradio as gr

#model = pipeline(model = "facebook/wav2vec2-base-960h")
model = pipeline( model= "openai/whisper-large-v2")

def transcribe_audio(mic=None, file = None):
    if mic is not None:
        audio = mic
    elif file is not None:
        audio = file
    else:
        return "You must either provide a mic recording or a file"
    transcription = model(audio)["text"]
    return transcription


gr.Interface(
    fn=transcribe_audio,
    inputs=[
        gr.Audio(source="microphone", type="filepath", optional=True),
        gr.Audio(source="upload", type="filepath", optional=True),
    ],
    outputs="text",
).launch(share = True)
```

Comments:
Hello @pythonvijay !
I had a look at the hub & docs , in this case, the model [Usage section](https://huggingface.co/openai/whisper-large-v2#usage)  to backtrack your challenge. 

1. As I'm understanding from [code snippets](https://huggingface.co/openai/whisper-large-v2#english-to-english), you don´t actually use HF `pipeline` to load the model, and you need the `Whisperprocessor` beforehand. The WhisperProcessor is used to:

Pre-process the audio inputs (converting them to log-Mel spectrograms for the model)
Post-process the model outputs (converting them from tokens to text) 

Have you tried something like this?

```
from transformers import WhisperProcessor, WhisperForConditionalGeneration


# load model and processor
processor = WhisperProcessor.from_pretrained("openai/whisper-large-v2")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v2")
model.config.forced_decoder_ids = None
```

2. Regarding the audio files, what I have done to try to clarify this is to search for the dataset the model was [evaluated with](https://huggingface.co/openai/whisper-large-v2#evaluation) , and ended up [finding](https://huggingface.co/datasets/librispeech_asr#data-fields) that apparently is a `.flac` format [file](https://en.wikipedia.org/wiki/FLAC).  On the other hand, there are [spaces](https://huggingface.co/spaces/innev/whisper-Base) built that accept `.wav` and `.mp3` file formats so Im assuming that the input for inference could be diverse. 

Hope this helps! 🎤🎤

cc @sanchit-gandhi 

Hey @pythonvijay and @SoyGema!

There are two ways of using the Whisper model with HF Transformers:

### 1. model + processor

Here, the **user** is responsible for loading the input audio as a numpy array. Hence, any file format you are able to read as a numpy array is valid. 

In the following code-snippet, we show how to do this with the [`soundfile`](https://pysoundfile.readthedocs.io/en/latest/#read-write-functions) library, which supports a number of input file formats. Overall, `model` + `processor` requires you to specify the 3 distinct stages of inference (pre-processing, model forward, post-processing)

```python
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import soundfile as sf

# load model and processor from pre-trained
processor = WhisperProcessor.from_pretrained("openai/whisper-large-v2")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v2")

# load audio file: user is responsible for loading the audio files themselves
array, sampling_rate = sf.read("audio.mp3", samplerate=processor.feature_extractor.sampling_rate

# pre-process to get the input features
input_features = processor(array, sampling_rate=sampling_rate, return_tensors="pt").input_features 

# generate token ids by running model forward sequentially
predicted_ids = model.generate(input_features, max_new_tokens=256)

# post-process token ids to text
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
print(transcription)
```

### 2. pipeline class

Takes care of loading the audio file for you with [`ffmpeg`](https://www.ffmpeg.org/general.html#File-Formats), which also supports a number of different file formats. Overall, [`pipeline`](https://huggingface.co/docs/transformers/v4.33.0/en/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline) is a higher-level helper function that allows you to run the models with minimal lines of code:

```python
from transformers import pipeline

pipe = pipeline("automatic-speech-recognition", model="openai/whisper-large-v2")
transcription = pipe("audio.mp3")
```

Hi Sanchit,

Thanks. The code looks intuitive. However, I get the following error, which should be self explanatory;

`AttributeError: 'WhisperProcessor' object has no attribute 'sampling_rate`














++

```
# load audio file: user is responsible for loading the audio files themselves
array, sampling_rate = sf.read("audio.mp3", samplerate=processor.sampling_rate)
```

When I try the using pipeline(), I get a file not found error (which I am sure will happen in the aforementioned code aswell, which got stuck at 'sampling-rate' attribute error . I defined the path of my audio file pretty much the way we define the path of a text file and assigned it to a variable 'audio'.

Is there a different way to read the audio file?


Thanks for flagging @pythonvijay - fixed the `model` + `processor` code snippet as required.

Nope, it should work specifying either the absolute or relative path to your audio file. Can you try first downloading a dummy audio file with the bash command:
```bash
wget https://huggingface.co/spaces/speechbox/whisper-restore-punctuation/resolve/main/sample1.flac
```

And then run the pipeline as:
```python
from transformers import pipeline

pipe = pipeline("automatic-speech-recognition", model="openai/whisper-large-v2")
transcription = pipe("./sample1.flac")
```



Hi Sanchit,

When you fixed, did you change something in the pipeline code as I dont see any change in the visible code?

here is the error I get:

`File ["<ipython-input-7-b38d5899e442>"](https://localhost:8080/#), line 4
    wget https://huggingface.co/spaces/speechbox/whisper-restore-punctuation/resolve/main/sample1.flac
         ^
SyntaxError: invalid syntax`

No change to the pipeline code. Note that you need to execute the `wget` command in **bash** (as specified previously), or pre-pend it with a `!` to work in a colab cell.

Bash:
```
wget https://huggingface.co/spaces/speechbox/whisper-restore-punctuation/resolve/main/sample1.flac
```

ipython/jupyter/colab:
```
!wget https://huggingface.co/spaces/speechbox/whisper-restore-punctuation/resolve/main/sample1.flac
```

Thanks Sanchit,

Got that.

Can you please advise what am I doing wrong while passing my own .flac file?

It gives en error

```
audio = '/Users/vijay/Downloads/audio.flac'
transcription = pipe("audio.flac")
```
`FileNotFoundError: [Errno 2] No such file or directory: 'audio.flac'`


did you try: 
```python
audio = '/Users/vijay/Downloads/audio.flac'
transcription = pipe(audio)
```

Sorry, that was my bad as I overlooked that audio was a variable.

Somehow, I still get the same error when I pass audio as a variable in pipe;

```
from transformers import pipeline
pipe = pipeline("automatic-speech-recognition", model="openai/whisper-large-v2")
audio = '/Users/vijay/Downloads/audio.flac'
transcription = pipe(audio)
```
`FileNotFoundError: [Errno 2] No such file or directory: '/Users/vijay/Downloads/audio.flac'`

Also, if I process 
`!wget https://huggingface.co/spaces/speechbox/whisper-restore-punctuation/resolve/main/sample1.flac`
using model + processor (which somehow seems more intuitive to me as I can see what is happening at each step), I get a type error as follows:

```
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import soundfile as sf

# load model and processor from pre-trained
processor = WhisperProcessor.from_pretrained("openai/whisper-large-v2")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v2")

!wget https://huggingface.co/spaces/speechbox/whisper-restore-punctuation/resolve/main/sample1.flac

# load audio file: user is responsible for loading the audio files themselves
array, sampling_rate = sf.read("./sample1.flac",  samplerate=processor.feature_extractor.sampling_rate)

# pre-process to get the input features
input_features = processor(array, sampling_rate=sampling_rate, return_tensors="pt").input_features

# generate token ids by running model forward sequentially
predicted_ids = model.generate(input_features, max_new_tokens=256)

# post-process token ids to text
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
print(transcription)
```

error:

```
---> 11 array, sampling_rate = sf.read("./sample1.flac",  samplerate=processor.feature_extractor.sampling_rate)
     12 
     13 # pre-process to get the input features

2 frames
[/usr/local/lib/python3.10/dist-packages/soundfile.py](https://localhost:8080/#) in _create_info_struct(file, mode, samplerate, channels, format, subtype, endian)
   1481         if any(arg is not None for arg in (
   1482                 samplerate, channels, original_format, subtype, endian)):
-> 1483             raise TypeError("Not allowed for existing files (except 'RAW'): "
   1484                             "samplerate, channels, format, subtype, endian")
   1485     return info

TypeError: Not allowed for existing files (except 'RAW'): samplerate, channels, format, subtype, endian
```

The `FileNotFoundError` means that your variable `audio` is not pointing to the correct path to your audio file. Please double check that you are setting `audio` to the correct path.

The error you've posted regarding the `soundfile` library not being able to convert the sampling rate on-the-fly should be taken up with the `soundfile` repository. As mentioned in the first comment, it is the users' responsibility to load the audio file if using the `model` + `processor` approach. If you don't want to load the audio yourself, you are advised to use the `pipeline`.

If you want to try a different approach, have a go with this code snippet. Note that you will need to pass the correct path to your audio file for this to work and ensure that you have `ffmpeg` installed on your system: https://ffmpeg.org/download.html

```python
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from transformers.pipelines.audio_utils import ffmpeg_read

# load model and processor from pre-trained
processor = WhisperProcessor.from_pretrained("openai/whisper-large-v2")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v2")

# load audio file: user is responsible for loading the audio files themselves
audio_path = "./sample1.flac"
sampling_rate = processor.feature_extractor.sampling_rate

# load as bytes
with open(audio_path, "rb") as f:
    inputs = f.read()

# read bytes as array
inputs = ffmpeg_read(inputs, sampling_rate=sampling_rate)

# pre-process to get the input features
input_features = processor(inputs, sampling_rate=sampling_rate, return_tensors="pt").input_features

# generate token ids by running model forward sequentially
predicted_ids = model.generate(input_features, max_new_tokens=256)

# post-process token ids to text
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
print(transcription)
```

Specifically on the path of the audio file, I am unsure what is the right way to read the file:

First, I tried this:

```
from google.colab import files
uploaded = files.upload()

transcription = pipe(uploaded)
```

error:
`ValueError: When passing a dictionary to AutomaticSpeechRecognitionPipeline, the dict needs to contain a "raw" key containing the numpy array representing the audio and a "sampling_rate" key, containing the sampling_rate associated with that array`

Next, I tried this (defining the path):

```
file_id = "1RjF9CRj8xTQakH5jex1ppBrTw1heHwPS"
url = 'https://drive.google.com/uc?id={}'.format(file_id)

transcription = pipe(url)
```
error

`ValueError: Malformed soundfile`

file reading (whether audio, pdf ) is something I am struggling with....everything else seems straight forward

as an aside, if there is a good resource to sharpen skills on how to access various data file in colab such that they are in the desired format for pipeline(), please do suggest


Hey @SoyGema - for the audio path, you don't need to _read_ the audio file. You just need to set it to the path where it is saved.

As for uploading files to Google Colab, this is not really the best place to get such help, since it's not a `transformers` related question. One easy way I would suggest is that you upload the file to the Hugging Face Hub: https://huggingface.co/docs/hub/models-uploading

And then clone the repository locally on your Google Cloud runtime: https://huggingface.co/docs/hub/repositories-getting-started#cloning-repositories

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @pythonvijay - did the above comments help answer your question? Feel free to close this issue if so! Otherwise, on-hand to answer any follow-ups! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.