GitHub issue #5259: datasets 2.7 introduces sharding error
State: closed | Created: 2022-11-17T15:36:52Z

### Describe the bug

dataset fails to load with runtime error

`RuntimeError: Sharding is ambiguous for this dataset: we found several data sources lists of different lengths, and we don't know over which list we should parallelize:
	- key audio_files has length 46
	- key data has length 0
To fix this, check the 'gen_kwargs' and make sure to use lists only for data sources, and use tuples otherwise. In the end there should only be one single list, or several lists with the same length.`

### Steps to reproduce the bug

With datasets[audio] 2.7 loaded, and logged into hugging face, 

`data = datasets.load_dataset('sil-ai/bloom-speech', 'bis', use_auth_token=True)`

creates the error.

Full stack trace:
```---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

[<ipython-input-7-8cb9ca0f79f0>](https://localhost:8080/#) in <module>
----> 1 data = datasets.load_dataset('sil-ai/bloom-speech', 'bis', use_auth_token=True)

5 frames

[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, num_proc, **config_kwargs)
   1745         try_from_hf_gcs=try_from_hf_gcs,
   1746         use_auth_token=use_auth_token,
-> 1747         num_proc=num_proc,
   1748     )
   1749 

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in download_and_prepare(self, output_dir, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
    824                             verify_infos=verify_infos,
    825                             **prepare_split_kwargs,
--> 826                             **download_and_prepare_kwargs,
    827                         )
    828                     # Sync info

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in _download_and_prepare(self, dl_manager, verify_infos, **prepare_splits_kwargs)
   1554     def _download_and_prepare(self, dl_manager, verify_infos, **prepare_splits_kwargs):
   1555         super()._download_and_prepare(
-> 1556             dl_manager, verify_infos, check_duplicate_keys=verify_infos, **prepare_splits_kwargs
   1557         )
   1558 

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    911             try:
    912                 # Prepare split will record examples associated to the split
--> 913                 self._prepare_split(split_generator, **prepare_split_kwargs)
    914             except OSError as e:
    915                 raise OSError(

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split(self, split_generator, check_duplicate_keys, file_format, num_proc, max_shard_size)
   1362         fpath = path_join(self._output_dir, fname)
   1363 
-> 1364         num_input_shards = _number_of_shards_in_gen_kwargs(split_generator.gen_kwargs)
   1365         if num_input_shards <= 1 and num_proc is not None:
   1366             logger.warning(

[/usr/local/lib/python3.7/dist-packages/datasets/utils/sharding.py](https://localhost:8080/#) in _number_of_shards_in_gen_kwargs(gen_kwargs)
     16                 + "\n".join(f"\t- key {key} has length {length}" for key, length in lists_lengths.items())
     17                 + "\nTo fix this, check the 'gen_kwargs' and make sure to use lists only for data sources, "
---> 18                 + "and use tuples otherwise. In the end there should only be one single list, or several lists with the same length."
     19             )
     20         )

RuntimeError: Sharding is ambiguous for this dataset: we found several data sources lists of different lengths, and we don't know over which list we should parallelize:
	- key audio_files has length 46
	- key data has length 0
To fix this, check the 'gen_kwargs' and make sure to use lists only for data sources, and use tuples otherwise. In the end there should only be one single list, or several lists with the same length.```

### Expected behavior

the dataset loads in datasets version 2.6.1 and should load with datasets 2.7

### Environment info

- `datasets` version: 2.7.0
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.15
- PyArrow version: 6.0.1
- Pandas version: 1.3.5