Issue: [WIP] Add TEDLIUM dataset
State: closed
Author: sanchit-gandhi

Adds the TED-LIUM dataset https://www.tensorflow.org/datasets/catalog/tedlium#tedliumrelease3 

TODO:

- [x] Port `tedium.py` from TF datasets using `convert_dataset.sh` script
- [x] Make `load_dataset` work
- [ ] ~~Run `datasets-cli` command to generate `dataset_infos.json`~~
- [ ] ~~Create dummy data for continuous testing~~
- [ ] ~~Dummy data tests~~
- [ ] ~~Real data tests~~
- [ ] Create the metadata JSON
- [ ] Close PR and add directly to the Hub under LIUM org

Comments:
_The documentation is not available anymore as the PR was closed or merged._

```python
from datasets import load_dataset

ds = load_dataset('./datasets/tedlium', 'release1', cache_dir='/home/sanchitgandhi/cache')
```

```
Downloading and preparing dataset tedlium/release1 to /home/sanchitgandhi/cache/tedlium/release1/1.0.1/5a9fcb97b4b52d5a1c9dc7bde4b1d5994cd89c4a3425ea36c789bf6096fee4f0...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/sanchit_huggingface_co/datasets/src/datasets/load.py", line 1703, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/sanchit_huggingface_co/datasets/src/datasets/builder.py", line 605, in download_and_prepare
    self._download_and_prepare(
  File "/home/sanchit_huggingface_co/datasets/src/datasets/builder.py", line 1240, in _download_and_prepare
    raise MissingBeamOptions(
datasets.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). 
Example of usage: 
        `load_dataset('tedlium', 'release1', beam_runner='DirectRunner')`
```
Specifying the `beam_runner='DirectRunner'` works:
```python
from datasets import load_dataset

ds = load_dataset('./datasets/tedlium', 'release1', cache_dir='/home/sanchitgandhi/cache', beam_runner='DirectRunner')
```

Extra Python imports/Linux packages:
```
pip install pydub
sudo apt install ffmpeg
```

Script heavily inspired by the TF datasets script at: https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/audio/tedlium.py

The TF datasets script uses the module AudioSegment from the package `pydub` (https://github.com/jiaaro/pydub), which is used to to open the audio files (stored in .sph format):
https://github.com/huggingface/datasets/blob/61bf6123634bf6e7c7287cd6097909eb26118c58/datasets/tedlium/tedlium.py#L167-L170
This package requires the pip install of `pydub` and the system installation of `ffmpeg`: https://github.com/jiaaro/pydub#installation
Is it ok to use these packages? Or do we tend to avoid introducing additional dependencies?

The TF datasets script also uses `_build_pcollection`:
https://github.com/huggingface/datasets/blob/8afbbb6fe66b40d05574e2e72e65e974c72ae769/datasets/tedlium/tedlium.py#L200-L206
However, I was advised against using `beam` logic. Thus, I have reverted to generating the examples file-by-file: https://github.com/huggingface/datasets/blob/61bf6123634bf6e7c7287cd6097909eb26118c58/datasets/tedlium/tedlium.py#L112-L138

I am now able to generate examples by running the `load_dataset` command:
```python
from datasets import load_dataset

ds = load_dataset('./datasets/tedlium', 'release1', cache_dir='/home/sanchitgandhi/cache')
```

Here, generating examples is **extremely** slow: it takes ~1 second per example, so ~60k seconds for the train set (~16 hours). Is there a way of paralleling this to make it faster?

> This package requires the pip install of pydub and the system installation of ffmpeg: https://github.com/jiaaro/pydub#installation
Is it ok to use these packages? Or do we tend to avoid introducing additional dependencies?

It's ok, windows users will have have a bad time but I'm not sure we can do much about it.

> Here, generating examples is extremely slow: it takes ~1 second per example, so ~60k seconds for the train set (~16 hours). Is there a way of paralleling this to make it faster?

Not at the moment. For such cases we advise hosting the dataset ourselves in a processed format. The license doesn't allow this since the license is "NoDerivatives". Currently the only way to parallelize it is by keeping is as a beam dataset and let users pay Google Dataflow to process it (or use spark or whatever).

Thanks for your super speedy reply @lhoestq!

I’ve uploaded the script and README.md to the org here: https://huggingface.co/datasets/LIUM/tedlium
Is any modification of the script required to be able to use it from the Hub? When I run:
```python
from datasets import load_dataset

tedlium = load_dataset("LIUM/tedlium", "release1") # for Release 1
```
I get the following error:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [2], in <cell line: 1>()
----> 1 load_dataset("LIUM/tedlium", "release1")

File ~/datasets/src/datasets/load.py:1676, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1673 ignore_verifications = ignore_verifications or save_infos
   1675 # Create a dataset builder
-> 1676 builder_instance = load_dataset_builder(
   1677     path=path,
   1678     name=name,
   1679     data_dir=data_dir,
   1680     data_files=data_files,
   1681     cache_dir=cache_dir,
   1682     features=features,
   1683     download_config=download_config,
   1684     download_mode=download_mode,
   1685     revision=revision,
   1686     use_auth_token=use_auth_token,
   1687     **config_kwargs,
   1688 )
   1690 # Return iterable dataset in case of streaming
   1691 if streaming:

File ~/datasets/src/datasets/load.py:1502, in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, **config_kwargs)
   1500     download_config = download_config.copy() if download_config else DownloadConfig()
   1501     download_config.use_auth_token = use_auth_token
-> 1502 dataset_module = dataset_module_factory(
   1503     path,
   1504     revision=revision,
   1505     download_config=download_config,
   1506     download_mode=download_mode,
   1507     data_dir=data_dir,
   1508     data_files=data_files,
   1509 )
   1511 # Get dataset builder class from the processing script
   1512 builder_cls = import_main_class(dataset_module.module_path)

File ~/datasets/src/datasets/load.py:1254, in dataset_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1249             if isinstance(e1, FileNotFoundError):
   1250                 raise FileNotFoundError(
   1251                     f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "
   1252                     f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
   1253                 ) from None
-> 1254             raise e1 from None
   1255 else:
   1256     raise FileNotFoundError(
   1257         f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory."
   1258     )

File ~/datasets/src/datasets/load.py:1227, in dataset_module_factory(path, revision, download_config, download_mode, force_local_path, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1225         raise e
   1226 if filename in [sibling.rfilename for sibling in dataset_info.siblings]:
-> 1227     return HubDatasetModuleFactoryWithScript(
   1228         path,
   1229         revision=revision,
   1230         download_config=download_config,
   1231         download_mode=download_mode,
   1232         dynamic_modules_path=dynamic_modules_path,
   1233     ).get_module()
   1234 else:
   1235     return HubDatasetModuleFactoryWithoutScript(
   1236         path,
   1237         revision=revision,
   (...)
   1241         download_mode=download_mode,
   1242     ).get_module()

File ~/datasets/src/datasets/load.py:940, in HubDatasetModuleFactoryWithScript.get_module(self)
    938 def get_module(self) -> DatasetModule:
    939     # get script and other files
--> 940     local_path = self.download_loading_script()
    941     dataset_infos_path = self.download_dataset_infos_file()
    942     imports = get_imports(local_path)

File ~/datasets/src/datasets/load.py:918, in HubDatasetModuleFactoryWithScript.download_loading_script(self)
    917 def download_loading_script(self) -> str:
--> 918     file_path = hf_hub_url(path=self.name, name=self.name.split("/")[1] + ".py", revision=self.revision)
    919     download_config = self.download_config.copy()
    920     if download_config.download_desc is None:

TypeError: hf_hub_url() got an unexpected keyword argument 'name'
```

Note that I am able to load the dataset from the `datasets` repo with the following lines of code:
```python
from datasets import load_dataset

ds = load_dataset('./datasets/tedlium', 'release1', cache_dir='/home/sanchitgandhi/cache')
```

What version of `datasets` do you have ?
Updating `datasets` should fix the error ;)


> This package requires the pip install of pydub and the system installation of ffmpeg: https://github.com/jiaaro/pydub#installation
Is it ok to use these packages? Or do we tend to avoid introducing additional dependencies?

`soundfile`, which is a required audio dependency, should also work with `.sph` files, no?

> `soundfile`, which is a required audio dependency, should also work with `.sph` files, no?

Awesome, thanks for the pointer @mariosasko! Switched `pydub` to `soundfile`, and having specifying the `dtype` argument in `soundfile.read` as `np.int16`, the arrays match with those from `pydub` ✅

I also did some heavy optimising of the script with the processing of the `.stm` and `.sph` files - it now runs 2000x faster than before, so there probably isn't a need to upload the data to the Hub @lhoestq. The total processing time is just ~2mins now 🚀


TEDLIUM completed and uploaded to the HF Hub: https://huggingface.co/datasets/LIUM/tedlium

Awesome !