Unable to load wiki_auto_asset_turk from GEM
Opened by abhinavsethy (closed)

### Describe the bug

I am unable to load the wiki_auto_asset_turk dataset. I get a fatal error while trying to access wiki_auto_asset_turk and load it with datasets.load_dataset. The error (TypeError: expected str, bytes or os.PathLike object, not NoneType) is from filenames_for_dataset_split in a os.path.join call 


>>import datasets
>>print (datasets.__version__)
>>dataset = datasets.load_dataset("GEM/wiki_auto_asset_turk")

System output:

Generating train split: 100%|█| 483801/483801 [00:03<00:00, 127164.26 examples/s
Generating validation split: 100%|█| 20000/20000 [00:00<00:00, 116052.94 example
Generating test_asset split: 100%|██| 359/359 [00:00<00:00, 76155.93 examples/s]
Generating test_turk split: 100%|███| 359/359 [00:00<00:00, 87691.76 examples/s]
Traceback (most recent call last):
  File "/Users/abhinav.sethy/Code/openai_evals/evals/evals/grammarly_tasks/gem_sari.py", line 3, in <module>
    dataset = datasets.load_dataset("GEM/wiki_auto_asset_turk")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/load.py", line 2582, in load_dataset
    builder_instance.download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1005, in download_and_prepare
    self._download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1767, in _download_and_prepare
    super()._download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1100, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1565, in _prepare_split
    split_info = self.info.splits[split_generator.name]
                 ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/splits.py", line 532, in __getitem__
    instructions = make_file_instructions(
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/arrow_reader.py", line 121, in make_file_instructions
    info.name: filenames_for_dataset_split(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/naming.py", line 72, in filenames_for_dataset_split
    prefix = os.path.join(path, prefix)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 76, in join
TypeError: expected str, bytes or os.PathLike object, not NoneType

### Steps to reproduce the bug

import datasets
print (datasets.__version__)
dataset = datasets.load_dataset("GEM/wiki_auto_asset_turk")


### Expected behavior

Should be able to load the dataset without any issues

### Environment info

datasets version 2.18.0 (was able to reproduce bug with older versions 2.16 and 2.14 also)
Python 3.12.0

Comments:
Hi! I've opened a [PR](https://huggingface.co/datasets/GEM/wiki_auto_asset_turk/discussions/5) with a fix. While waiting for it to be merged, you can load the dataset from the PR branch with `datasets.load_dataset("GEM/wiki_auto_asset_turk", revision="refs/pr/5")`

Thanks Mario. Still getting the same issue though with the suggested fix

#cat gem_sari.py
import datasets
print (datasets.__version__)
dataset =datasets.load_dataset("GEM/wiki_auto_asset_turk", revision="refs/pr/5")

End up with 
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/load.py", line 2582, in load_dataset
    builder_instance.download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1005, in download_and_prepare
    self._download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1767, in _download_and_prepare
    super()._download_and_prepare(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1100, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/builder.py", line 1565, in _prepare_split
    split_info = self.info.splits[split_generator.name]
                 ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/splits.py", line 532, in __getitem__
    instructions = make_file_instructions(
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/arrow_reader.py", line 121, in make_file_instructions
    info.name: filenames_for_dataset_split(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/datasets/naming.py", line 72, in filenames_for_dataset_split
    prefix = os.path.join(path, prefix)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen posixpath>", line 76, in join
TypeError: expected str, bytes or os.PathLike object, not NoneType

Hmm, that's weird. Maybe try deleting the cache with `!rm -rf ~/.cache/huggingface/datasets` and then re-download.

Tried that a couple of time. It does download the data fresh but end up with same error. Is there a way to see if its using the right version ?

You can check the version with `python -c "import datasets; print(datasets.__version__)"`

the datasets version is 2.18. 

I wanted to see if the command datasets.load_dataset("GEM/wiki_auto_asset_turk", revision="refs/pr/5") is using the right revision (refs/pr/5).  





 

Still have this problem

The issue is fixed once the fixing PR has been merged and the dataset has been converted to Parquet.

If the problem persists on your side, you should update your `datasets` library:
```shell
pip install -U datasets
```
And if you have already the latest version of `datasets`, then you need to delete the old version of this dataset in your cache:
```shell
rm -fr ~/.cache/huggingface/datasets/GEM___wiki_auto_asset_turk
rm -fr ~/.cache/huggingface/modules/datasets_modules/datasets/GEM--wiki_auto_asset_turk
```