Create Audio feature
State: closed | Author: albertvillanova (MEMBER)

Create `Audio` feature to handle raw audio files.

Some decisions to be further discussed:
- I have chosen `soundfile` as the audio library; another interesting library is `librosa`, but this requires `soundfile` (see [here](https://github.com/librosa/librosa/blob/main/setup.cfg#L53)). If we require some more advanced functionalities, we could eventually switch the library.
- I have implemented the audio feature as an extra: `pip install datasets[audio]`. For the moment, the typical datasets user uses only text datasets, and there is no need for them for additional package requirements for audio/image if they do not need them.
- For tests, I require audio dependencies (so that all audio functionalities are checked with our CI test suite); I exclude Linux platforms, which require an additional library to be installed with the distribution package manager
  - I also require `pytest-datadir`, which allow to have (audio) data files for tests
- The audio data contain: array and sample_rate.
- The array is reshaped as 1D array (expected input for `Wav2Vec2`).

Note that to install `soundfile` on Linux, you need to install `libsndfile` using your distribution’s package manager, for example `sudo apt-get install libsndfile1`.

## Requirements Specification
- Access example with audio loading and resampling:
  ```python
  ds[0]["audio"]
  ```
- Map with audio loading & resampling:
  ```python
  def preprocess(batch):
       batch["input_values"] = processor(batch["audio"]).input_values
       return batch

  ds = ds.map(preprocess)
  ```
- Map without audio loading and resampling:
  ```python
  def preprocess(batch):
       batch["labels"] = processor(batch["target_text"]).input_values
       return batch

  ds = ds.map(preprocess)
  ```
- Additional requirement specification (see https://github.com/huggingface/datasets/pull/2324#pullrequestreview-768864998): Cast audio column to change sampling sate:
  ```python
  ds = ds.cast_column("audio", Audio(sampling_rate=16_000))
  ```

Comments: For optimal storage, it would be better to:
- store only the audio file path in the cache Arrow file
- perform decoding of the audio file (into audio array and sample rate) on the fly, while loading the dataset from cache (or by adding a convenient `load_audio` function)

Thanks a lot @lhoestq for your helpful insights! 🤗 

Just one step before having a first running example to benchmark.

Decision to make: how to call the function `dataset.features.decode_example`:
- The usual approach until now in speech applications: call it in a subsequent `.map` function
  - Pros: multiprocessing can be used out of the box
  - Cons: large disk storage required for caching decoded audio files, although having it cached will enhance speed for further usage
- Approach suggested by @lhoestq (see above: https://github.com/huggingface/datasets/pull/2324#discussion_r660758683): doing it in formatting
  - Pros: no large disk storage required, as it will be done on the fly while iterating on the dataset
  - Cons: it is not cached; need to implement multiprocessing for this case
- Other pros/cons for the previous options?
- Other options?

cc: @lhoestq @patrickvonplaten @anton-l 

@albertvillanova I'm in two minds about this, to be honest. For example, if we consider CommonVoice, which is encoded in lossy `mp3`:

- If we decompress `mp3` into raw `wav` arrays,  loading a batch will speed up about 40x.
- However, a 60gb English mp3 dataset will blow up to about 600gb raw (iirc), which is why loading on-the-fly (optionally?) could be very beneficial as well.

Users can do the conversion from mp3 to wav by themselves if they want to using `map`.

IMO it's better if we can keep the decoding part with the minimal features to be both easy to understand and flexible, i.e. just having the on-the-fly decoding of the audio data (with the sampling rate parameter)

Decompressing from mp3 to wav sounds like an optimization that depends on the problem that the user wants to solve, the constrains from its environment (disk space, IO speed), and other parameters (optimal training speed for example). Therefore I would leave this to the user to decide whether it has to do it or not.

Let me know what you think about this

@albertvillanova, In my opinion the pros strongly outweigh the cons in the @lhoestq's suggestion which is why I think we should go forward with it. 

The cons:
- "the operation won't be cached" is not to important as the user will most likely access just a couple of audio array to see how it looks like and then for the "full" feature extraction she/he will make use of `.map(...)` anyways which means that the result will be cached. 
- Regarding the multi-processing - if I understand correctly it'll follow the same logic here -> the user will only access some audio arrays for testing playing around with the model but use `.map(...)` for larger operations where multi-processing would still work as before.

The advantages mostly solve the main poinpoints being:
- exploding disk space
- bad user experience since the audio is not loaded on the go

=> So I'm very much in favor of the "direct-access" feature

Update: I've retaken this issue.

If the decoding logic is implemented when "examples are accessed", then if afterwards we use the `.map`, it tries to apply the decoding twice (as maps iterates over the examples, thus "accessing them", before trying to apply the map function)...

I'm thinking on some other approach...

I have reimplemented the previous approach, so that we can discuss about it: examples are decoded when accessed.

What about creating a new specific formatting, just for decoding? This would be only active within a context manager.

Hi @lhoestq, as we discussed, I've followed your suggestion of implementing the decoding step within the formatting logic: extract-decode-format. Feel free to tell me what you think.

@patrickvonplaten and @anton-l, could you have a look at the use case in the test (https://github.com/huggingface/datasets/pull/2324/files#diff-58e348f6e4deaa5f3119e420a5d48ebb82875a78c28628831748fb54f59b2c78R34-R50) and tell me if this is aligned with your needs? Thanks.

Hi @lhoestq, if you validate this approach, we could merge the Audio feature this (or early next) week.

Sure it looks nice this way :) Feel free to continue !

As discussed, we should pay attention when applying `map` to a dataset with `Audio` feature, in order to avoid decoding the audio data twice.

One proposed solution is to pass `input_columns` to `map`. Just, note that the field containing the Audio feature should not be passed in `input_columns` (not possible, for example, to map the audio file path to a new directory).

I suggest again (3rd time, sorry, lol) using a formatting context manager (as we already use for PyTorch/TensorFlow: https://huggingface.co/docs/datasets/torch_tensorflow.html).

Above (https://github.com/huggingface/datasets/pull/2324#issuecomment-915244003), I suggested to define a formatting just for decoding: the decoding of the audio data is only performed if this specific formatting is set (`ds.set_format("decoding")`) or within a context manager (`with ds.formatted_as("decoding"): ...`)

Now, I would like also to suggest an alternative formatting for **non-decoding** (if decoding is the default behavior), for a use case like this:
```python
def change_dir(example):
    example["audio"] = "dir/" + example["audio"]


with ds.formatted_as("no_decoding"):
    print(ds[0])  # {"audio": "path/to/file.wav"}
    ds.map(change_dir)
    print(ds[0])  # {"audio": "dir/path/to/file.wav"}

print(ds[0])  # {"audio": {"path": "dir/path/to/file.wav", "array": np.array([1., 2., 3...]), "sampling_rate": 44100}}
```

Please, just tell me what you think.
CC: @lhoestq @patrickvonplaten @anton-l 

> As discussed, we should pay attention when applying `map` to a dataset with `Audio` feature, in order to avoid decoding the audio data twice.
> 
> One proposed solution is to pass `input_columns` to `map`. Just, note that the field containing the Audio feature should not be passed in `input_columns` (not possible, for example, to map the audio file path to a new directory).
> 
> I suggest again (3rd time, sorry, lol) using a formatting context manager (as we already use for PyTorch/TensorFlow: https://huggingface.co/docs/datasets/torch_tensorflow.html).
> 
> Above ([#2324 (comment)](https://github.com/huggingface/datasets/pull/2324#issuecomment-915244003)), I suggested to define a formatting just for decoding: the decoding of the audio data is only performed if this specific formatting is set (`ds.set_format("decoding")`) or within a context manager (`with ds.formatted_as("decoding"): ...`)
> 
> Now, I would like also to suggest an alternative formatting for **non-decoding** (if decoding is the default behavior), for a use case like this:
> 
> ```python
> def change_dir(example):
>     example["audio"] = "dir/" + example["audio"]
> 
> 
> with ds.formatted_as("no_decoding"):
>     print(ds[0])  # {"audio": "path/to/file.wav"}
>     ds.map(change_dir)
>     print(ds[0])  # {"audio": "dir/path/to/file.wav"}
> 
> print(ds[0])  # {"audio": {"path": "dir/path/to/file.wav", "array": np.array([1., 2., 3...]), "sampling_rate": 44100}}
> ```
> 
> Please, just tell me what you think.
> CC: @lhoestq @patrickvonplaten @anton-l

I'm fine with a context manager! There is no way to **not** decode the audio if its key is not accessed no?

E.g.

```python
def load(batch):
    batch["speech_array"] = torchaudio.load(batch["file"])
    return batch

ds.map(load)
```

does *e.g.* not access the "audio" key `batch["audio"}` but there is no way to not decode it without major changes no? 

=> I'm happy with both the context manager and using `input_colmuns`. Both of those solutions are equally good to me if a "not-access-key-no-decoding" solution is just not feasible. I let you guys decide :-)

> 
> There is no way to **not** decode the audio if its key is not accessed no?
> 
> E.g...
> 
> does _e.g._ not access the "audio" key `batch["audio"}` but there is no way to not decode it without major changes no?

@patrickvonplaten I think therefore we should rethink the implementation of the Audio feature: its goal is to enrich/simplify the user experience when working with audio files. If on the other hand, you see that the current implementation may be problematic/unsatisfying/not-optimal, then we miss the point of creating this feature. This feature should be useful to users, not inconvenient.

> > There is no way to **not** decode the audio if its key is not accessed no?
> > E.g...
> > does _e.g._ not access the "audio" key `batch["audio"}` but there is no way to not decode it without major changes no?
> 
> @patrickvonplaten I think therefore we should rethink the implementation of the Audio feature: its goal is to enrich/simplify the user experience when working with audio files. If on the other hand, you see that the current implementation may be problematic/unsatisfying/not-optimal, then we miss the point of creating this feature. This feature should be useful to users, not inconvenient.

Thanks a lot for the message! I'm discussing a bit with @anton-l at the moment - will share our results as soon as possible

Current implementation: see use cases in file https://github.com/huggingface/datasets/blob/0f80e6eaa6f596ff6287eb33587e2d9c69af0e73/tests/features/test_audio.py

Automatic decoding:
- when directly accessing an example or a batch
  ```python
  dset[0]
  dset[:2]
  ```
- during map, only if audio field is accessed:
  ```python
  def process_audio_sampling_rate(example):
      example["double_sampling_rate"] = 2 * example["audio"]["sampling_rate"]
      return example

  decoded_dset = dset.map(process_audio_sampling_rate)
  ```

No automatic decoding:
- during map if audio field is not accessed:
  ```python
  def process_text(example):
      example["text"] = example["text"] + " World!"
      return example

  decoded_dset = dset.map(process_text)
  ```

The types of example and batch are kept as usual, `dict[str, Any]` and `dict[str, list[Any]]` respectively.

CC: @patrickvonplaten @anton-l @lhoestq 

That's awesome! Thanks so much for your work on this @albertvillanova!

Oh and maybe have a test to make sure that casting the Audio feature to change the sampling rate works as expected ?

@lhoestq the test for the resampling is already in place in `test_audio_resampling`: 
https://github.com/huggingface/datasets/pull/2324/files#diff-58e348f6e4deaa5f3119e420a5d48ebb82875a78c28628831748fb54f59b2c78R48-R56

Please note that we should agree in the API: see 53d6d73

This is just a proposal implementation:
- Create a new method named `cast_column`, which performs a shallow kind of cast (without using `map()` or caching)

We should agree in the name, because as it is, it might be confused with `cast` (and users might think `cast_column` caches the result as `cast`)

CC: @lhoestq @patrickvonplaten @anton-l 

IMO cast and cast_column should have the exact same behavior, to make the experience simple for the user (no distinction between shallow or deep cast).

Maybe we should change `cast` to use `cast_column` on every column and make `cast_column` use `map` if and only if it's necessary. For Audio for example `map` is not needed.

We just need to do some tests to know which casts always need map and which ones don't. This implies either looking at the PyArrow source code (the documentation doesn't mention all these details) or playing with PyArrow to figure it out.

I guess for now we can just have the simplest `cast_column` which always uses map unless it's an Audio feature type.

Let me know what you think !

@lhoestq I totally agree: `cast` and `cast_column` should be analog to each other.

For the implementation, let me try something simpler than the one suggested by you...

@lhoestq what do you think of an approach like this 633ef09?

If it's OK, then we should implement passing parameters to `cast`.

@lhoestq maybe for now we could make a simple implementation and finish this PR. Then we could make a follow-up PR to deal specifically with the optimal implementation of `cast_column` and `cast`, as this issue is not specific to the Audio feature.

> @lhoestq what do you think of an approach like this 633ef09?

Yea that's good enough for the time being :)

I think the last thing we need to do is make sure that `cast_column` changes the fingerprint of the dataset. Feel free to use the `fingerprint_transform` decorator, as for `remove_columns`.

(note that cast currently doesn't use the decorator since it's based on `map` that already updates the fingerprint).

> 
> I think the last thing we need to do is make sure that `cast_column` changes the fingerprint of the dataset. Feel free to use the `fingerprint_transform` decorator, as for `remove_columns`.
> 
> (note that cast currently doesn't use the decorator since it's based on `map` that already updates the fingerprint).

@lhoestq note that `cast_column` may call `cast` in some cases, and the decorator would not be necessary for these cases...
- I did it by setting `inplace=False`, and updating fingerprint explicitly only when `cast` is not called.

I think current state of this PR could be included in our next release, as experimental feature, for stress testing it and try to find all potential issues. What do you think?

CC: @lhoestq @patrickvonplaten @anton-l 

Looks great! Ready to try it out on the transformers examples after the release :)

Think we are good to merge here no? :-)