Create dataset_infos.json with VALIDATION and TEST splits
Author: skalinin
State: closed
Labels: name: bug

The problem is described in that [issue](https://github.com/huggingface/datasets/issues/4895#issuecomment-1247975569). 

> When I try to create data_infos.json using datasets-cli test Peter.py --save_infos --all_configs I get an error:
> ValueError: Unknown split "test". Should be one of ['train'].
> 
> The data_infos.json is created perfectly fine when I use only one split - datasets.Split.TRAIN
> 
> You can find the code here: https://huggingface.co/datasets/sberbank-ai/Peter/tree/add_splits (add_splits branch)

I tried to clear the cache folder, than I got an another error. I run:

```
git clone https://huggingface.co/datasets/sberbank-ai/Peter
cd Peter
git checkout add_splits  # switch to a add_splits branch
rm dataset_infos.json  # remove local dataset_infos.json
rm -r ~/.cache/huggingface   # remove cached dataset_infos.json
datasets-cli test Peter.py --save_infos --all_configs  # trying to create new dataset_infos.json
```

The error message:
```
Using custom data configuration default
Testing builder 'default' (1/1)
Downloading and preparing dataset peter/default to /Users/kalinin/.cache/huggingface/datasets/peter/default/0.0.0/ef579519e140d6a40df2555996f26165f04c47557d7373709c8d7e7b4fd7465d...
Downloading data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 5160.63it/s]
Extracting data files:   0%|                                                                                                            | 0/4 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/usr/local/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/datasets/commands/datasets_cli.py", line 39, in main
    service.run()
  File "/usr/local/lib/python3.9/site-packages/datasets/commands/test.py", line 137, in run
    builder.download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 1227, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 771, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/Users/kalinin/.cache/huggingface/modules/datasets_modules/datasets/Peter/ef579519e140d6a40df2555996f26165f04c47557d7373709c8d7e7b4fd7465d/Peter.py", line 23, in _split_generators
    data_files = dl_manager.download_and_extract(_URLS)
  File "/usr/local/lib/python3.9/site-packages/datasets/download/download_manager.py", line 431, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/usr/local/lib/python3.9/site-packages/datasets/download/download_manager.py", line 403, in extract
    extracted_paths = map_nested(
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 393, in map_nested
    mapped = [
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 394, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/py_utils.py", line 330, in _single_map_nested
    return function(data_struct)
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 213, in cached_path
    output_path = ExtractManager(cache_dir=download_config.cache_dir).extract(
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/extract.py", line 46, in extract
    self.extractor.extract(input_path, output_path, extractor_format)
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/extract.py", line 263, in extract
    with FileLock(lock_path):
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/filelock.py", line 399, in __init__
    max_filename_length = os.statvfs(os.path.dirname(lock_file)).f_namemax
FileNotFoundError: [Errno 2] No such file or directory: ''
Exception ignored in: <function BaseFileLock.__del__ at 0x11caeec10>
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/filelock.py", line 328, in __del__
    self.release(force=True)
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/filelock.py", line 303, in release
    with self._thread_lock:
AttributeError: 'UnixFileLock' object has no attribute '_thread_lock'
Extracting data files:   0%|                                                                                                            | 0/4 [00:00<?, ?it/s]
```

Can you help me please?


## Environment info
- `datasets` version: 2.4.0
- Platform: macOS-12.5.1-x86_64-i386-64bit
- Python version: 3.9.5
- PyArrow version: 9.0.0
- Pandas version: 1.2.4


Comments: @mariosasko could you help me with this issue? we've started the discussion from [here](https://github.com/huggingface/datasets/issues/4895#issuecomment-1248227130)

Hi again! Can you please pass the directory name containing the dataset script instead of the script name to `datasets-cli test`?

Yes, it worked! thanks a lot