audio pipeline utility ffmpeg_microphone_live doesn't work in Google Colab
Author: crcdng
Pull request: no

### System Info

Googel Colab 2023/07/21 / Chrome 115.0.5790.114 / macOS 13.5 
Default MacBook microphone

### Who can help?

@sanchit-gandhi

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. make sure microphone works and access to microphone is enabled in Colab
2. follow https://huggingface.co/learn/audio-course/chapter7/voice-assistant


### Expected behavior

Colab cell `launch_fn(debug=True)` should output scores while listening for the wake word as described in the tutorial. Instead, nothing happens. No error is shown and no audio is recorded. This makes the `transcribe` cell in the tutorial crash.

Comments:
Hey @crcdng! Thanks for flagging this - could you share details of how you achieved step 1 (make sure microphone works and access to microphone is enabled in Colab)? Once I can reproduce I can take a look into how we can get this working!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> Hey @crcdng! Thanks for flagging this - could you share details of how you achieved step 1 (make sure microphone works and access to microphone is enabled in Colab)? Once I can reproduce I can take a look into how we can get this working!

Testing the microphone on my machine and then looking at the colab permissions:

<img width="314" alt="Screenshot 2023-08-29 at 14 31 37" src="https://github.com/huggingface/transformers/assets/830492/c631d8df-550f-49f7-9257-93284eaa7e79">



Hmm yes I don't think Colab is able to take the output from your local microphone and send it to the Colab device. Have you tried running the script locally? I actually wrote it using a CPU only device on a MBP laptop, so it should work on your local CPU. There are some hacky ways you can try to get your microphone output to work with Colab if you'd like to try this, e.g. https://ricardodeazambuja.com/deep_learning/2019/03/09/audio_and_video_google_colab/

> Hmm yes I don't think Colab is able to take the output from your local microphone and send it to the Colab device. Have you tried running the script locally? I actually wrote it using a CPU only device on a MBP laptop, so it should work on your local CPU. There are some hacky ways you can try to get your microphone output to work with Colab if you'd like to try this, e.g. https://ricardodeazambuja.com/deep_learning/2019/03/09/audio_and_video_google_colab/

EDIT: I haven't been able to run the code locally either using VSCode / with Jupyter Notebook Support. 

I get a 

`ValueError: We expect a numpy ndarray as input`

in cell `launch_fn(debug=True)` 

Is it possible that `ffmpeg_microphone_live` needs more work?

First response: 
(Yeah, I have seen the "hacky" method as well and was wondering if it could be integrated in this section of the Audio course? Alternatively, maybe a note at the beginning of https://huggingface.co/learn/audio-course/chapter7/voice-assistant#creating-a-voice-assistant such as "Due to the requirement for an active microphone, it is recommended to try this section on a local machine instead of Google Colab." )     

## ffmpeg repro

Sorry to hear that @crcdng! Could you post the full stack trace please? I can then take a deeper look! For reference, I'm using this reduced version of the code to test ASR inference:

Cell 0:
```python
from transformers import pipeline
import torch
from transformers.pipelines.audio_utils import ffmpeg_microphone_live

device = "cuda:0" if torch.cuda.is_available() else "cpu"

transcriber = pipeline(
    "automatic-speech-recognition", model="openai/whisper-base.en", device=device
)

import sys


def transcribe(chunk_length_s=15.0, stream_chunk_s=1.0):
    sampling_rate = transcriber.feature_extractor.sampling_rate

    mic = ffmpeg_microphone_live(
        sampling_rate=sampling_rate,
        chunk_length_s=chunk_length_s,
        stream_chunk_s=stream_chunk_s,
    )

    print("Start speaking...")
    for item in transcriber(mic, generate_kwargs={"max_new_tokens": 128}):
        sys.stdout.write("\033[K")
        print(item["text"], end="\r")
        if not item["partial"][0]:
            break
```

Cell 1:
```
transcribe()
```

## Google Colab compatibility

I'll add a note to the start of the section - that's a great suggestion.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I updated the course docs to suggest using a local device. Do you have a reproducible code snippet and error trace for the ffmpeg issues @crcdng? The code is working using the reproducer I gave above: https://github.com/huggingface/transformers/issues/25183#issuecomment-1703099362

Hi @sanchit-gandhi 
I ran the code snippet(saved as python live_transcription.py) you shared on my Local machine(CPU). Unfortunately, it is not working for me. It seems like the program ends before `print("Start speaking...")` I am guessing I might be setting up the environment on my machine wrongly?

**System Info**
Intel Core i7 10th Gen
Default MSI Laptop Microphone

**Environment Setup & to replicate error **
pip install git+https://github.com/huggingface/transformers.git
python live_transcription.py

**Who can help?**
@sanchit-gandhi

**Information**
 The official example scripts

**Error**

USER@MSI MINGW64 ~/Desktop/Medical STT & TTS (master)
$ C:/Users/USER/AppData/Local/Programs/Python/Python38/python.exe "c:/Users/USER/Desktop/Medical STT & TTS/live_transcription.py"
Loading the tokenizer from the `special_tokens_map.json` and the `added_tokens.json` will be removed in `transformers 5`,  it is kept for forward compatibility, but it is recommended to update your `tokenizer_config.json` by uploading it again. You will see the new `added_tokens_decoder` attribute that will store 
the relevant information.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

Hey @ss8319! Do you get any error message when the program hangs/crashes? The logs you've sent are just warnings about tokenizer updates, rather than error logs! Also, are you calling `transcribe()` in your python file? If using a Python file, your code should be:
```python
from transformers import pipeline
import torch
from transformers.pipelines.audio_utils import ffmpeg_microphone_live
import sys

device = "cuda:0" if torch.cuda.is_available() else "cpu"

transcriber = pipeline(
    "automatic-speech-recognition", model="openai/whisper-base.en", device=device
)


def transcribe(chunk_length_s=15.0, stream_chunk_s=1.0):
    sampling_rate = transcriber.feature_extractor.sampling_rate

    mic = ffmpeg_microphone_live(
        sampling_rate=sampling_rate,
        chunk_length_s=chunk_length_s,
        stream_chunk_s=stream_chunk_s,
    )

    print("Start speaking...")
    for item in transcriber(mic, generate_kwargs={"max_new_tokens": 128}):
        sys.stdout.write("\033[K")
        print(item["text"], end="\r")
        if not item["partial"][0]:
            break

transcribe()
```

Thanks @sanchit-gandhi! Yea I forgot to add transcribe(). Thanks for catching that. Its showing a different error now. Doing pip install ffmpeg didn't help.

```USER@MSI MINGW64 ~/Desktop/Medical STT & TTS (master)
$ python live_transcription.py
Loading the tokenizer from the `special_tokens_map.json` and the `added_tokens.json` will be removed in `transformers 5`,  it is kept for forward compatibility, but it is recommended to update your `tokenizer_config.json` by uploading it again. You will see the new `added_tokens_decoder` attribute that will store 
the relevant information.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Start speaking...
Traceback (most recent call last):
  File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\pipelines\audio_utils.py", line 217, in _ffmpeg_stream
    with subprocess.Popen(ffmpeg_command, stdout=subprocess.PIPE, bufsize=bufsize) as ffmpeg_process:    
  File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 858, in __init__   
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\subprocess.py", line 1311, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "live_transcription.py", line 30, in <module>
    transcribe()
  File "live_transcription.py", line 23, in transcribe
    for item in transcriber(mic, generate_kwargs={"max_new_tokens": 128}):
  File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\pipelines\pt_utils.py", line 124, in __next__
    item = next(self.iterator)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\pipelines\pt_utils.py", line 266, in __next__
    processed = self.infer(next(self.iterator), **self.params)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 681, in __next__
    data = self._next_data()
  File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 721, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\utils\data\_utils\fetch.py", line 32, in fetch
    data.append(next(self.dataset_iter))
  File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\pipelines\pt_utils.py", line 180, in __next__
    self.subiterator = self.infer(next(self.iterator), **self.params)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\pipelines\audio_utils.py", line 161, in ffmpeg_microphone_live
    for item in chunk_bytes_iter(microphone, chunk_len, stride=(stride_left, stride_right), stream=True):  File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\pipelines\audio_utils.py", line 188, in chunk_bytes_iter
    for raw in iterator:
  File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\pipelines\audio_utils.py", line 94, in ffmpeg_microphone
    for item in iterator:
  File "C:\Users\USER\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\pipelines\audio_utils.py", line 224, in _ffmpeg_stream
    raise ValueError("ffmpeg was not found but is required to stream audio files from filename") from error
ValueError: ffmpeg was not found but is required to stream audio files from filename```

Ah yes - `ffmpeg` isn't a Python package, but rather a system one: https://ffmpeg.org/download.html

Could you try installing it via the official installation instructions?

Not work to me too, this skips in `for item in pipe(mic)`
Windows 11
ffmpeg was installed

Hey @CWKSC! Sorry just to clarify - how is the line in the code 'skipped'? Does the code give an error when you run it? Or does it just run and hang without an error? 

> Hey @CWKSC! Sorry just to clarify - how is the line in the code 'skipped'? Does the code give an error when you run it? Or does it just run and hang without an error?

No error, no hang

I am following the tutorial: https://huggingface.co/learn/audio-course/chapter7/voice-assistant#creating-a-voice-assistant

```python
from transformers import pipeline
import torch

device = "cuda:0" if torch.cuda.is_available() else "cpu"

classifier = pipeline(
    "audio-classification", model="MIT/ast-finetuned-speech-commands-v2", device=device
)

from transformers.pipelines.audio_utils import ffmpeg_microphone_live

def launch_fn(
    wake_word="marvin",
    prob_threshold=0.5,
    chunk_length_s=2.0,
    stream_chunk_s=0.25,
    debug=False,
):
    if wake_word not in classifier.model.config.label2id.keys():
        raise ValueError(
            f"Wake word {wake_word} not in set of valid class labels, pick a wake word in the set {classifier.model.config.label2id.keys()}."
        )

    sampling_rate = classifier.feature_extractor.sampling_rate

    mic = ffmpeg_microphone_live(
        sampling_rate=sampling_rate,
        chunk_length_s=chunk_length_s,
        stream_chunk_s=stream_chunk_s,
    )

    print("Listening for wake word...")
    for prediction in classifier(mic):
        prediction = prediction[0]
        if debug:
            print(prediction)
        if prediction["label"] == wake_word:
            if prediction["score"] > prob_threshold:
                return True

launch_fn(debug=True)
```

It skip in `for prediction in classifier(mic):`

Also here is my build venv script:

```powershell
py -3.9 -m venv venv
.\venv\Scripts\activate

python -m pip install --upgrade pip

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

pip install huggingface_hub
pip install transformers
pip install ipython
```

`classifier(mic)` return a `transformers.pipelines.pt_utils.PipelineIterator` object

And this looks like immediate return

If I `print(list(classifier(mic)))`, it output `[]`, and then: 

```python
print(next(classifier(mic)))
```

```python
Traceback (most recent call last):
  File "...", line 43, in <module>
    launch_fn(debug=True)
  File "...", line 33, in launch_fn
    print(classifier(mic).__next__())
  File "...\venv\lib\site-packages\transformers\pipelines\pt_utils.py", line 124, in __next__
    item = next(self.iterator)
AttributeError: 'PipelineIterator' object has no attribute 'iterator'
```

Maybe my microphone isn't recognized? I don't know any further setup.


What version of Transformers are you using @CWKSC? Could you paste the output of:
```
transformers-cli env
```

Otherwise, it might be a case that your microphone is indeed not being recognised. For now, let's just try printing out the values from the microphone function:
```python
from transformers.pipelines.audio_utils import ffmpeg_microphone_live

microphone = ffmpeg_microphone_live(sampling_rate=16000, chunk_length_s=5)

for chunk in microphone:
    print(chunk)
```

=> let's check that this returns some values before putting it in combination with the model. If this returns nothing, then we have an error with our ffmpeg read function

```powershell
transformers-cli env
```

```powershell
Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.34.0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.9.13
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

The output said platform is `Windows-10-10.0.22621-SP0`, but actually, I am Windows 11.

![image](https://github.com/huggingface/transformers/assets/53114952/28677954-ed04-4caa-b788-3fae7df41b51)

The script returns nothing.

I tried install transformers from source.

```powershell
pip install git+https://github.com/huggingface/transformers
```

No different.

Has anyone else on Windows tried this? 

I am having the same problem here, using windows 11
tried 
```python
from transformers.pipelines.audio_utils import ffmpeg_microphone_live

microphone = ffmpeg_microphone_live(sampling_rate=16000, chunk_length_s=5)

for chunk in microphone:
    print(chunk)
```
didn't print anything

Thanks both for the info! cc'ing @Narsil here - looks like the ffmpeg utilities for the Transformers pipeline are not working in Windows. Might it be related to these parameters that we set for the ffmpeg microphone? https://github.com/huggingface/transformers/blob/14b04b4b9c483d94fadd2b5479ed9430bae8ac84/src/transformers/pipelines/audio_utils.py#L73-L75

Do you have any advice for testing with a Windows system too if ones own computer runs Mac/Linux? Did you have a framework for testing these functions with Windows?

Had the same problem with the recording not working. Just solved it for testing purpoces.

You need to specify the input device to the ffmpeg_microphone() function:

Find your microphone name using cmd: _ffmpeg -list_devices true -f dshow -i dummy_
Copy the name and edit it in line 75 in ->  ...\.venv\Lib\site-packages\transformers\pipelines\audio_utils.py
Instead of "default" put "audio=_input_device_name_"
```python
    elif system == "Windows":
        format_ = "dshow"
        input_ = "audio=Microphone (High Definition Audio Device)"

``` 

Nice fix @Teapack1! Would you like to open a PR to fix this in Transformers? We can call `ffmpeg -list_devices true -f dshow -I dummy` and store the microphone name as a variable, then set this to the `input_` in the audio utils code: https://github.com/huggingface/transformers/blob/14b04b4b9c483d94fadd2b5479ed9430bae8ac84/src/transformers/pipelines/audio_utils.py#L73-L75

Having some problems on Mac currently, trying to follow[ this guide](https://huggingface.co/learn/audio-course/chapter7/voice-assistant). Right now, I can get output, but it doesn't seem to actually be listening to the microphone and only predicting one class for speech classification, no matter what I say, after granting Terminal access to the microphone: 

```
/Users/victor/anaconda3/envs/transformers/lib/python3.9/site-packages/transformers/models/audio_spectrogram_transformer/feature_extraction_audio_spectrogram_transformer.py:96: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /private/var/folders/sy/f16zz6x50xz3113nwtb9bvq00000gp/T/abs_9d63z49rj_/croot/pytorch_1681837279022/work/torch/csrc/utils/tensor_numpy.cpp:205.)
  waveform = torch.from_numpy(waveform).unsqueeze(0)
{'score': 0.05440586432814598, 'label': 'no'}
{'score': 0.05816075950860977, 'label': 'up'}
{'score': 0.07136523723602295, 'label': 'up'}
{'score': 0.09769058227539062, 'label': 'follow'}
{'score': 0.14641302824020386, 'label': 'follow'}
{'score': 0.1379959136247635, 'label': 'follow'}
{'score': 0.1379959136247635, 'label': 'follow'}
{'score': 0.1379959136247635, 'label': 'follow'}
{'score': 0.1379959136247635, 'label': 'follow'}
{'score': 0.1379959136247635, 'label': 'follow'}
{'score': 0.1379959136247635, 'label': 'follow'}
{'score': 0.1379959136247635, 'label': 'follow'}
{'score': 0.1379959136247635, 'label': 'follow'}
{'score': 0.1379959136247635, 'label': 'follow'}
{'score': 0.1379959136247635, 'label': 'follow'}
{'score': 0.1379959136247635, 'label': 'follow'}
```

Curious if anyone else had this issue? Not sure if the warning message has anything to do with this. Is `ffmpeg_microphone` supposed to support MacOS?

Edit: it seems like the problem lies with [setting the input for ffmpeg on MacOS to `:1` instead](https://github.com/huggingface/transformers/blob/fb78769b9c053876ed7ae152ee995b0439a4462a/src/transformers/pipelines/audio_utils.py#L68). 

> ```python
> def transcribe(chunk_length_s=15.0, stream_chunk_s=1.0):
>     sampling_rate = transcriber.feature_extractor.sampling_rate
> 
>     mic = ffmpeg_microphone_live(
>         sampling_rate=sampling_rate,
>         chunk_length_s=chunk_length_s,
>         stream_chunk_s=stream_chunk_s,
>     )
> 
>     print("Start speaking...")
>     for item in transcriber(mic, generate_kwargs={"max_new_tokens": 128}):
>         sys.stdout.write("\033[K")
>         print(item["text"], end="\r")
>         if not item["partial"][0]:
>             break
> 
> transcribe()
> ```

ty its help me 👍 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.