datasets 2.7 introduces sharding error
State: closed | Author: DCNemesis (NONE)

### Describe the bug

dataset fails to load with runtime error

`RuntimeError: Sharding is ambiguous for this dataset: we found several data sources lists of different lengths, and we don't know over which list we should parallelize:
	- key audio_files has length 46
	- key data has length 0
To fix this, check the 'gen_kwargs' and make sure to use lists only for data sources, and use tuples otherwise. In the end there should only be one single list, or several lists with the same length.`

### Steps to reproduce the bug

With datasets[audio] 2.7 loaded, and logged into hugging face, 

`data = datasets.load_dataset('sil-ai/bloom-speech', 'bis', use_auth_token=True)`

creates the error.

Full stack trace:
```---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

[<ipython-input-7-8cb9ca0f79f0>](https://localhost:8080/#) in <module>
----> 1 data = datasets.load_dataset('sil-ai/bloom-speech', 'bis', use_auth_token=True)

5 frames

[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, num_proc, **config_kwargs)
   1745         try_from_hf_gcs=try_from_hf_gcs,
   1746         use_auth_token=use_auth_token,
-> 1747         num_proc=num_proc,
   1748     )
   1749 

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in download_and_prepare(self, output_dir, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, num_proc, storage_options, **download_and_prepare_kwargs)
    824                             verify_infos=verify_infos,
    825                             **prepare_split_kwargs,
--> 826                             **download_and_prepare_kwargs,
    827                         )
    828                     # Sync info

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in _download_and_prepare(self, dl_manager, verify_infos, **prepare_splits_kwargs)
   1554     def _download_and_prepare(self, dl_manager, verify_infos, **prepare_splits_kwargs):
   1555         super()._download_and_prepare(
-> 1556             dl_manager, verify_infos, check_duplicate_keys=verify_infos, **prepare_splits_kwargs
   1557         )
   1558 

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    911             try:
    912                 # Prepare split will record examples associated to the split
--> 913                 self._prepare_split(split_generator, **prepare_split_kwargs)
    914             except OSError as e:
    915                 raise OSError(

[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split(self, split_generator, check_duplicate_keys, file_format, num_proc, max_shard_size)
   1362         fpath = path_join(self._output_dir, fname)
   1363 
-> 1364         num_input_shards = _number_of_shards_in_gen_kwargs(split_generator.gen_kwargs)
   1365         if num_input_shards <= 1 and num_proc is not None:
   1366             logger.warning(

[/usr/local/lib/python3.7/dist-packages/datasets/utils/sharding.py](https://localhost:8080/#) in _number_of_shards_in_gen_kwargs(gen_kwargs)
     16                 + "\n".join(f"\t- key {key} has length {length}" for key, length in lists_lengths.items())
     17                 + "\nTo fix this, check the 'gen_kwargs' and make sure to use lists only for data sources, "
---> 18                 + "and use tuples otherwise. In the end there should only be one single list, or several lists with the same length."
     19             )
     20         )

RuntimeError: Sharding is ambiguous for this dataset: we found several data sources lists of different lengths, and we don't know over which list we should parallelize:
	- key audio_files has length 46
	- key data has length 0
To fix this, check the 'gen_kwargs' and make sure to use lists only for data sources, and use tuples otherwise. In the end there should only be one single list, or several lists with the same length.```

### Expected behavior

the dataset loads in datasets version 2.6.1 and should load with datasets 2.7

### Environment info

- `datasets` version: 2.7.0
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.15
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

Comments: I notice a comment in the code says:
`Having lists of different sizes makes sharding ambigious, raise an error in this case until we decide how to define sharding without ambiguity for users` 
    
 ... which suggests this update was pushed knowing that it might break some things. But, it didn't seem to have a useful error message of an argument that could be passed to avoid the error.

Sorry for the inconvenience, I opened a PR in your repo to fix this: https://huggingface.co/datasets/sil-ai/bloom-speech/discussions/2

Basically we've always considered lists in `gen_kwargs` to be a shard list that we can split and pass into different workers to generate the dataset (e.g. if you pass `num_proc=` in `load_dataset()` to generate the dataset in parallel), but it was documented only recently

@lhoestq Thanks for the help. It looks like that took care of it.