Repository: speechbrain/speechbrain

Issue:
[Bug]: CommonVoice sorting is likely broken with certain torchaudio versions
### Describe the bug

CommonVoice is a dataset in MP3 format. It seems that a recent `torchaudio` version (linked issue reports `0.12`, but my `0.11` environment seems to reproduce) more or less intentionally broke querying the length of MP3 files when using `torchaudio.info(path).num_frames`, as it seems to always return 0, see <https://github.com/pytorch/audio/issues/2524>.

It seems that a PR was merged at some point to solve that issue but the minimum torchaudio version we require is probably still affected.

SpeechBrain uses this to generated the CSVs. As a result, they contain a `0.0` duration column.

`train.py` uses that column to sort data.

### Expected behaviour

`duration` should never be zero.

### To Reproduce

Run any preprocessing step in a CommonVoice recipe, then inspect `results/<seed>/save/*.csv`.

### Versions

_No response_

### Relevant log output

Example contents of CSV:

```
ID,duration,wav,spk_id,wrd
common_voice_fr_19642221,0.0,/users/sdelangen/CommonVoice8.0/fr/clips/common_voice_fr_19642221.mp3,4e64587f8dd935d70e50607fe60f73b5badcd6e967db2c331a6c92ca00e9d1ddde6ff2385cd1a90c6d5ceea960e708248914dcb6cdc69fe6ea7edd0208a3ff5e,C EST LE PLUS VASTE ET LE PLUS PEUPLÉ DE LA COMMUNE
common_voice_fr_19642222,0.0,/users/sdelangen/CommonVoice8.0/fr/clips/common_voice_fr_19642222.mp3,4e64587f8dd935d70e50607fe60f73b5badcd6e967db2c331a6c92ca00e9d1ddde6ff2385cd1a90c6d5ceea960e708248914dcb6cdc69fe6ea7edd0208a3ff5e,CES SEPT QUATUORS ONT ÉTÉ ÉDITÉS PAR MARC LOMBARD
common_voice_fr_19642223,0.0,/users/sdelangen/CommonVoice8.0/fr/clips/common_voice_fr_19642223.mp3,
```


### Additional context

_No response_


Discussion:
MP3 `.info` seems to be correct again with `torchaudio == 2.0.0` (and potentially earlier). However, on that version, it doesn't seem to be actually any faster than `.load(path, normalize=False)[0].size(1)`, so it's likely that it does the same thing.

So for MP3 formats, working around it in SB by doing a full load doesn't seem too unreasonable. This should not be generalized to other formats, which aren't as awful and generally have a more trivial way to check for length :)
`DVoice` is likely also affected as it seems to use MP3 too (if that isn't an artifact from the recipe being based on CommonVoice).

Resolving patch:
diff --git a/recipes/CommonVoice/common_voice_prepare.py b/recipes/CommonVoice/common_voice_prepare.py
--- a/recipes/CommonVoice/common_voice_prepare.py
+++ b/recipes/CommonVoice/common_voice_prepare.py
@@ -18,6 +18,7 @@
 import functools
 
 from speechbrain.utils.parallel import parallel_map
+from speechbrain.dataio.dataio import read_audio_info
 
 logger = logging.getLogger(__name__)
 
@@ -180,7 +181,7 @@ def process_line(line, data_folder, language, accented_letters):
 
     # Reading the signal (to retrieve duration in seconds)
     if os.path.isfile(mp3_path):
-        info = torchaudio.info(mp3_path)
+        info = read_audio_info(mp3_path)
     else:
         msg = "\tError loading: %s" % (str(len(file_name)))
         logger.info(msg)
diff --git a/recipes/DVoice/dvoice_prepare.py b/recipes/DVoice/dvoice_prepare.py
--- a/recipes/DVoice/dvoice_prepare.py
+++ b/recipes/DVoice/dvoice_prepare.py
@@ -19,6 +19,7 @@
 from tqdm import tqdm
 import numpy as np
 import glob
+from speechbrain.dataio.dataio import read_audio_info
 
 logger = logging.getLogger(__name__)
 
@@ -195,7 +196,7 @@ def alffa_public_prepare(language, data_folder):
         for j in range(len(wavs)):
             if wavs[j].split("/")[-1] == file_name + ".wav":
                 wav = wavs[j]
-                info = torchaudio.info(wav)
+                info = read_audio_info(wav)
                 duration = info.num_frames / info.sample_rate
                 dic = {
                     "wav": wavs[j].replace(data_folder + "/", ""),
@@ -254,7 +255,7 @@ def swahili_prepare(data_folder):
         for j in range(len(wavs_alffa)):
             if wavs_alffa[j].split("/")[-1] == file_name + ".wav":
                 wav = wavs_alffa[j]
-                info = torchaudio.info(wav)
+                info = read_audio_info(wav)
                 duration = info.num_frames / info.sample_rate
                 dic = {
                     "wav": wavs_alffa[j].replace(data_folder + "/", ""),
diff --git a/speechbrain/dataio/dataio.py b/speechbrain/dataio/dataio.py
--- a/speechbrain/dataio/dataio.py
+++ b/speechbrain/dataio/dataio.py
@@ -160,6 +160,64 @@ def load_data_csv(csv_path, replacements={}):
     return result
 
 
+def read_audio_info(path) -> "torchaudio.backend.common.AudioMetaData":
+    """Retrieves audio metadata from a file path. Behaves identically to
+    torchaudio.info, but attempts to fix metadata (such as frame count) that is
+    otherwise broken with certain torchaudio version and codec combinations.
+
+    Note that this may cause full file traversal in certain cases!
+
+    Arguments
+    ----------
+    path : str
+        Path to the audio file to examine.
+
+    Returns
+    -------
+    torchaudio.backend.common.AudioMetaData
+        Same value as returned by `torchaudio.info`, but may eventually have
+        `num_frames` corrected if it otherwise would have been `== 0`.
+
+    NOTE
+    ----
+    Some codecs, such as MP3, require full file traversal for accurate length
+    information to be retrieved.
+    In these cases, you may as well read the entire audio file to avoid doubling
+    the processing time.
+    """
+
+    _path_no_ext, path_ext = os.path.splitext(path)
+
+    if path_ext == ".mp3":
+        # Additionally, certain affected versions of torchaudio fail to
+        # autodetect mp3.
+        # HACK: here, we check for the file extension to force mp3 detection,
+        # which prevents an error from occuring in torchaudio.
+        info = torchaudio.info(path, format="mp3")
+    else:
+        info = torchaudio.info(path)
+
+    # Certain file formats, such as MP3, do not provide a reliable way to
+    # query file duration from metadata (when there is any).
+    # For MP3, certain versions of torchaudio began returning num_frames == 0.
+    #
+    # https://github.com/speechbrain/speechbrain/issues/1925
+    # https://github.com/pytorch/audio/issues/2524
+    #
+    # Accomodate for these cases here: if `num_frames == 0` then maybe something
+    # has gone wrong.
+    # If some file really had `num_frames == 0` then we are not doing harm
+    # double-checking anyway. If I am wrong and you are reading this comment
+    # because of it: sorry
+    if info.num_frames == 0:
+        channels_data, sample_rate = torchaudio.load(path, normalize=False)
+
+        info.num_frames = channels_data.size(1)
+        info.sample_rate = sample_rate  # because we might as well
+
+    return info
+
+
 def read_audio(waveforms_obj):
     """General audio loading, based on a custom notation.
 


Test patch:
diff --git a/tests/unittests/test_data_io.py b/tests/unittests/test_data_io.py
--- a/tests/unittests/test_data_io.py
+++ b/tests/unittests/test_data_io.py
@@ -2,6 +2,28 @@
 import os
 
 
+def test_read_audio_info(tmpdir, device):
+    from speechbrain.dataio.dataio import read_audio_info, write_audio
+
+    test_waveform = torch.rand(32000, device=device)
+
+    for ext in ["wav", "ogg", "mp3"]:
+        audio_path = os.path.join(tmpdir, f"test.{ext}")
+        write_audio(audio_path, test_waveform.cpu(), 16000)
+        info = read_audio_info(audio_path)
+
+        # NOTE: This wide check is introduced for codecs that cannot handle
+        # exact frame counts, such as mp3, which appears to operate over chunks
+        # of 384 frames.
+        # This remains an exact read of the metadata, however, which is what we
+        # want to test and is the reason for `read_audio_info` to exist (see
+        # comments there).
+        assert (
+            32000 <= info.num_frames <= 35000
+        ), f"expected consistent len for codec {ext}"
+        assert info.sample_rate == 16000
+
+
 def test_read_audio(tmpdir, device):
     from speechbrain.dataio.dataio import read_audio, write_audio