Sharding error with Multilingual LibriSpeech
Opened by Nithin-Holla (closed)

### Describe the bug

Loading the German Multilingual LibriSpeech dataset results in a RuntimeError regarding sharding with the following stacktrace:

```
Downloading and preparing dataset multilingual_librispeech/german to /home/nithin/datadrive/cache/huggingface/datasets/facebook___multilingual_librispeech/german/2.1.0/1904af50f57a5c370c9364cc337699cfe496d4e9edcae6648a96be23086362d0...
Downloading data files: 100%
3/3 [00:00<00:00, 107.23it/s]
Downloading data files: 100%
1/1 [00:00<00:00, 35.08it/s]
Downloading data files: 100%
6/6 [00:00<00:00, 303.36it/s]
Downloading data files: 100%
3/3 [00:00<00:00, 130.37it/s]
Downloading data files: 100%
1049/1049 [00:00<00:00, 4491.40it/s]
Downloading data files: 100%
37/37 [00:00<00:00, 1096.78it/s]
Downloading data files: 100%
40/40 [00:00<00:00, 1003.93it/s]
Extracting data files: 100%
3/3 [00:11<00:00, 2.62s/it]
Generating train split:
469942/0 [34:13<00:00, 273.21 examples/s]
Output exceeds the size limit. Open the full output data in a text editor
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-14-74fa6d092bdc> in <module>
----> 1 mls = load_dataset(MLS_DATASET, 
      2                    LANGUAGE,
      3                    cache_dir="~/datadrive/cache/huggingface/datasets",
      4                    ignore_verifications=True)

/anaconda/envs/py38_default/lib/python3.8/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, num_proc, **config_kwargs)
   1755 
   1756     # Download and prepare data
-> 1757     builder_instance.download_and_prepare(
   1758         download_config=download_config,
   1759         download_mode=download_mode,

/anaconda/envs/py38_default/lib/python3.8/site-packages/datasets/builder.py in download_and_prepare(self, output_dir, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
    858                         if num_proc is not None:
    859                             prepare_split_kwargs["num_proc"] = num_proc
--> 860                         self._download_and_prepare(
    861                             dl_manager=dl_manager,
    862                             verify_infos=verify_infos,

/anaconda/envs/py38_default/lib/python3.8/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_splits_kwargs)
   1609 
   1610     def _download_and_prepare(self, dl_manager, verify_infos, **prepare_splits_kwargs):
...
RuntimeError: Sharding is ambiguous for this dataset: we found several data sources lists of different lengths, and we don't know over which list we should parallelize:
    - key audio_archives has length 1049
    - key local_extracted_archive has length 1049
    - key limited_ids_paths has length 1
To fix this, check the 'gen_kwargs' and make sure to use lists only for data sources, and use tuples otherwise. In the end there should only be one single list, or several lists with the same length.
```

### Steps to reproduce the bug

Here is the code to reproduce it:

```python
from datasets import load_dataset

MLS_DATASET = "facebook/multilingual_librispeech"
LANGUAGE = "german"

mls = load_dataset(MLS_DATASET, 
                   LANGUAGE, 
                   cache_dir="~/datadrive/cache/huggingface/datasets", 
                   ignore_verifications=True)
```

### Expected behavior

The expected behaviour is that the dataset is successfully loaded.

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-5.4.0-1094-azure-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyArrow version: 10.0.1
- Pandas version: 1.2.4

Comments:
Thanks for reporting, @Nithin-Holla.

This is a known issue for multiple datasets and we are investigating it:
- See e.g.: https://huggingface.co/datasets/ami/discussions/3

Main issue:
- #5415

@albertvillanova Thanks! As a workaround for now, can I use the dataset in streaming mode?

Yes, @Nithin-Holla, in the meantime you can use this dataset in streaming mode.