Loading audio in video from video URLs fail with chat template
Author: merveenoyan
Pull request: no
Labels: name: bug

### System Info

4.54.0.dev0 but also older versions

### Who can help?

@zucchini-nlp [this line](https://github.com/huggingface/transformers/blob/a52478253bbe522a420e88ea3940d4d98a935300/src/transformers/audio_utils.py#L60) in audio utils tries to load the audio in the video, technically expects a url to wav or mp4 but fails, tries to open the video file as audio hence the error. [in this line](https://github.com/huggingface/transformers/blob/a52478253bbe522a420e88ea3940d4d98a935300/src/transformers/processing_utils.py#L1597) you directly pass video filename (url here) to `load_audio`, so issue is fundamentally with chat templates and not per the model. Loading from path works ([here](https://colab.research.google.com/drive/10u49NTDod48kPc7bf-6YCHAdhRqdoZSL?usp=sharing)). I can take a stab, I want to confirm if this is in any way intended, the same code exists in [Qwen-Omni model docs](https://huggingface.co/docs/transformers/main/model_doc/qwen2_5_omni) but with loading from path.

here's the trace: 
```
/usr/local/lib/python3.11/dist-packages/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py:347: FutureWarning: `video_fps` is deprecated and will be removed in version 4.58 for `Qwen2_5OmniProcessor.apply_chat_template`. Use `fps` instead.
  return super().apply_chat_template(conversations, chat_template, **kwargs)
---------------------------------------------------------------------------
LibsndfileError                           Traceback (most recent call last)
[/tmp/ipython-input-2-2033772580.py](https://localhost:8080/#) in <cell line: 0>()
     15 ]
     16 
---> 17 inputs = processor.apply_chat_template(
     18     conversations,
     19     load_audio_from_video=True,

8 frames
[/usr/local/lib/python3.11/dist-packages/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py](https://localhost:8080/#) in apply_chat_template(self, conversations, chat_template, **kwargs)
    345                     + "Audio output mode only works when using default system prompt 'You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech.'"
    346                 )
--> 347         return super().apply_chat_template(conversations, chat_template, **kwargs)
    348 
    349     @property

[/usr/local/lib/python3.11/dist-packages/transformers/utils/deprecation.py](https://localhost:8080/#) in wrapped_func(*args, **kwargs)
    170                 warnings.warn(message, FutureWarning, stacklevel=2)
    171 
--> 172             return func(*args, **kwargs)
    173 
    174         return wrapped_func

[/usr/local/lib/python3.11/dist-packages/transformers/processing_utils.py](https://localhost:8080/#) in apply_chat_template(self, conversation, chat_template, **kwargs)
   1595                     else:
   1596                         for fname in video_fnames:
-> 1597                             batch_audios.append(load_audio(fname, sampling_rate=mm_load_kwargs["sampling_rate"]))
   1598 
   1599                     for fname in video_fnames:

[/usr/local/lib/python3.11/dist-packages/transformers/audio_utils.py](https://localhost:8080/#) in load_audio(audio, sampling_rate, timeout)
     58         # Load audio from URL (e.g https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/translate_to_chinese.wav)
     59         if audio.startswith("http://") or audio.startswith("https://"):
---> 60             audio = librosa.load(BytesIO(requests.get(audio, timeout=timeout).content), sr=sampling_rate)[0]
     61         elif os.path.isfile(audio):
     62             audio = librosa.load(audio, sr=sampling_rate)[0]

[/usr/local/lib/python3.11/dist-packages/librosa/core/audio.py](https://localhost:8080/#) in load(path, sr, mono, offset, duration, dtype, res_type)
    184                 y, sr_native = __audioread_load(path, offset, duration, dtype)
    185             else:
--> 186                 raise exc
    187 
    188     # Final cleanup for dtype and contiguity

[/usr/local/lib/python3.11/dist-packages/librosa/core/audio.py](https://localhost:8080/#) in load(path, sr, mono, offset, duration, dtype, res_type)
    174         # Otherwise try soundfile first, and then fall back if necessary
    175         try:
--> 176             y, sr_native = __soundfile_load(path, offset, duration, dtype)
    177 
    178         except sf.SoundFileRuntimeError as exc:

[/usr/local/lib/python3.11/dist-packages/librosa/core/audio.py](https://localhost:8080/#) in __soundfile_load(path, offset, duration, dtype)
    207     else:
    208         # Otherwise, create the soundfile object
--> 209         context = sf.SoundFile(path)
    210 
    211     with context as sf_desc:

[/usr/local/lib/python3.11/dist-packages/soundfile.py](https://localhost:8080/#) in __init__(self, file, mode, samplerate, channels, subtype, endian, format, closefd, compression_level, bitrate_mode)
    688         self._info = _create_info_struct(file, mode, samplerate, channels,
    689                                          format, subtype, endian)
--> 690         self._file = self._open(file, mode_int, closefd)
    691         if set(mode).issuperset('r+') and self.seekable():
    692             # Move write position to 0 (like in Python file objects)

[/usr/local/lib/python3.11/dist-packages/soundfile.py](https://localhost:8080/#) in _open(self, file, mode_int, closefd)
   1263             # get the actual error code
   1264             err = _snd.sf_error(file_ptr)
-> 1265             raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
   1266         if mode_int == _snd.SFM_WRITE:
   1267             # Due to a bug in libsndfile version <= 1.0.25, frames != 0

LibsndfileError: Error opening <_io.BytesIO object at 0x7b13bd3af600>: Format not recognised./usr/local/lib/python3.11/dist-packages/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py:347: FutureWarning: `video_fps` is deprecated and will be removed in version 4.58 for `Qwen2_5OmniProcessor.apply_chat_template`. Use `fps` instead.
  return super().apply_chat_template(conversations, chat_template, **kwargs)
---------------------------------------------------------------------------
LibsndfileError                           Traceback (most recent call last)
[/tmp/ipython-input-2-2033772580.py](https://localhost:8080/#) in <cell line: 0>()
     15 ]
     16 
---> 17 inputs = processor.apply_chat_template(
     18     conversations,
     19     load_audio_from_video=True,

8 frames
[/usr/local/lib/python3.11/dist-packages/transformers/models/qwen2_5_omni/processing_qwen2_5_omni.py](https://localhost:8080/#) in apply_chat_template(self, conversations, chat_template, **kwargs)
    345                     + "Audio output mode only works when using default system prompt 'You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech.'"
    346                 )
--> 347         return super().apply_chat_template(conversations, chat_template, **kwargs)
    348 
    349     @property

[/usr/local/lib/python3.11/dist-packages/transformers/utils/deprecation.py](https://localhost:8080/#) in wrapped_func(*args, **kwargs)
    170                 warnings.warn(message, FutureWarning, stacklevel=2)
    171 
--> 172             return func(*args, **kwargs)
    173 
    174         return wrapped_func

[/usr/local/lib/python3.11/dist-packages/transformers/processing_utils.py](https://localhost:8080/#) in apply_chat_template(self, conversation, chat_template, **kwargs)
   1595                     else:
   1596                         for fname in video_fnames:
-> 1597                             batch_audios.append(load_audio(fname, sampling_rate=mm_load_kwargs["sampling_rate"]))
   1598 
   1599                     for fname in video_fnames:

[/usr/local/lib/python3.11/dist-packages/transformers/audio_utils.py](https://localhost:8080/#) in load_audio(audio, sampling_rate, timeout)
     58         # Load audio from URL (e.g https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/translate_to_chinese.wav)
     59         if audio.startswith("http://") or audio.startswith("https://"):
---> 60             audio = librosa.load(BytesIO(requests.get(audio, timeout=timeout).content), sr=sampling_rate)[0]
     61         elif os.path.isfile(audio):
     62             audio = librosa.load(audio, sr=sampling_rate)[0]

[/usr/local/lib/python3.11/dist-packages/librosa/core/audio.py](https://localhost:8080/#) in load(path, sr, mono, offset, duration, dtype, res_type)
    184                 y, sr_native = __audioread_load(path, offset, duration, dtype)
    185             else:
--> 186                 raise exc
    187 
    188     # Final cleanup for dtype and contiguity

[/usr/local/lib/python3.11/dist-packages/librosa/core/audio.py](https://localhost:8080/#) in load(path, sr, mono, offset, duration, dtype, res_type)
    174         # Otherwise try soundfile first, and then fall back if necessary
    175         try:
--> 176             y, sr_native = __soundfile_load(path, offset, duration, dtype)
    177 
    178         except sf.SoundFileRuntimeError as exc:

[/usr/local/lib/python3.11/dist-packages/librosa/core/audio.py](https://localhost:8080/#) in __soundfile_load(path, offset, duration, dtype)
    207     else:
    208         # Otherwise, create the soundfile object
--> 209         context = sf.SoundFile(path)
    210 
    211     with context as sf_desc:

[/usr/local/lib/python3.11/dist-packages/soundfile.py](https://localhost:8080/#) in __init__(self, file, mode, samplerate, channels, subtype, endian, format, closefd, compression_level, bitrate_mode)
    688         self._info = _create_info_struct(file, mode, samplerate, channels,
    689                                          format, subtype, endian)
--> 690         self._file = self._open(file, mode_int, closefd)
    691         if set(mode).issuperset('r+') and self.seekable():
    692             # Move write position to 0 (like in Python file objects)

[/usr/local/lib/python3.11/dist-packages/soundfile.py](https://localhost:8080/#) in _open(self, file, mode_int, closefd)
   1263             # get the actual error code
   1264             err = _snd.sf_error(file_ptr)
-> 1265             raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
   1266         if mode_int == _snd.SFM_WRITE:
   1267             # Due to a bug in libsndfile version <= 1.0.25, frames != 0

LibsndfileError: Error opening <_io.BytesIO object at 0x7b13bd3af600>: Format not recognised.
```

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Here's the link to notebook: https://colab.research.google.com/drive/174ZQ1brFQxMwGZsVpQQLPYGhACuKOYEH?usp=sharing

### Expected behavior

It should work the same as loading from path

Comments:
Thanks for reporting! Yeah, I think `librosa` can't open audio from video urls though I haven't tested it. In general `librosa` isn't the best choice when it comes to audio track from video files, so I am planning to move to `torchcodec` by default. It will be the default for all video files

I am still benchmarking it and testing if it works with different video formats, will do the same for audio tracks in videos. In the meanwhile, maybe we can just ask users to not use `URL` when loading audio from video?

@zucchini-nlp sounds good!

I would like to ask if the issue about loading audio tracks from video files has been resolved?
I am using Qwen2.5 omni and encountered a similar problem: when I set use_audio_in_video=True, it throws an error.

In progress. If you're using a video bu URL or incompatible video format, you can either pre-load the audio yourself or use `qwen-omni-utils` as per [docs](https://huggingface.co/Qwen/Qwen2.5-Omni-7B#%F0%9F%A4%97--transformers-usage)

I checked the code for `process_audio_info` in `qwen-omni-utils`, and as far as I know, librosa shouldn’t be able to read audio directly from mp4 files, right?

[http://github.com/QwenLM/Qwen2.5-Omni/blob/main/qwen-omni-utils/src/qwen_omni_utils/v2_5/audio_process.py#L66](https://github.com/QwenLM/Qwen2.5-Omni/blob/main/qwen-omni-utils/src/qwen_omni_utils/v2_5/audio_process.py#L66)

```

 elif use_audio_in_video and ele["type"] == "video":
                    if "video" in ele or "video_url" in ele:
                        path = ele.get("video", ele.get("video_url"))
                        audio_start = ele.get("video_start", 0.0)
                        audio_end = ele.get("video_end", None)
                        assert _check_if_video_has_audio(
                            path
                        ), "Video must has audio track when use_audio_in_video=True"
                        if path.startswith("http://") or path.startswith("https://"):
                            data = audioread.ffdec.FFmpegAudioFile(path)
                        elif path.startswith("file://"):
                            data = path[len("file://") :]
                        else:
                            data = path
                    else:
                        raise ValueError("Unknown video {}".format(ele))
                else:
                    continue
                audios.append(
                    librosa.load(
                        data,
                        sr=SAMPLE_RATE,
                        offset=audio_start,
                        duration=(audio_end - audio_start) if audio_end is not None else None,
                    )[0]
                )
```

Librosa isn't super reliable when it comes to reading audio from videos, though it can decode the audio track in common cases. At least I remember testing it and it worked with demo videos

We'll use `torchcodec` for that which is guaranteed to support all types of videos as per their docs, I will submit a PR soon

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.