load_dataset can't load local dataset,Unable to find ...
Opened by ahf876828330 (closed)


Traceback (most recent call last):
  File "/home/gs603/ahf/pretrained/model.py", line 48, in <module>
    dataset = load_dataset("json",data_files="dataset/dataset_infos.json")
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/load.py", line 1675, in load_dataset
    **config_kwargs,
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/load.py", line 1496, in load_dataset_builder
    data_files=data_files,
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/load.py", line 1155, in dataset_module_factory
    download_mode=download_mode,
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/load.py", line 800, in get_module
    data_files = DataFilesDict.from_local_or_remote(patterns, use_auth_token=self.downnload_config.use_auth_token)
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/data_files.py", line 582, in from_local_or_remote
    if not isinstance(patterns_for_key, DataFilesList)
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/data_files.py", line 544, in from_local_or_remote
    data_files = resolve_patterns_locally_or_by_urls(base_path, patterns, allowed_extensions)
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/data_files.py", line 194, in resolve_patterns_locally_or_by_urls
    for path in _resolve_single_pattern_locally(base_path, pattern, allowed_extensions):
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/data_files.py", line 144, in _resolve_single_pattern_locally
    raise FileNotFoundError(error_msg)
FileNotFoundError: Unable to find '/home/gs603/ahf/pretrained/dataset/dataset_infos.json' at /home/gs603/ahf/pretrained


![image](https://user-images.githubusercontent.com/33253979/164413285-84ea65ac-9126-408f-9cd2-ce4751a5dd73.png)
![image](https://user-images.githubusercontent.com/33253979/164413338-4735142f-408b-41d9-ab87-8484de2be54f.png)

the code is in the model.py,why I can't use the load_dataset function to load my local dataset?

Comments:
Hi! :)

I believe that should work unless `dataset_infos.json` isn't actually a dataset. For Hugging Face datasets, there is usually a file named `dataset_infos.json` which contains metadata about the dataset (eg. the dataset citation, license, description, etc). Can you double-check that `dataset_infos.json` isn't just metadata please?

Hi @ahf876828330, 

As @stevhliu pointed out, the proper way to load a dataset is not trying to load its metadata file.

In your case, as the dataset script is local, you should better point to your local loading script:
```python
dataset = load_dataset("dataset/opus_books.py")
```

Please, feel free to re-open this issue if the previous code snippet does not work for you.

> Hi! :)
> 
> I believe that should work unless `dataset_infos.json` isn't actually a dataset. For Hugging Face datasets, there is usually a file named `dataset_infos.json` which contains metadata about the dataset (eg. the dataset citation, license, description, etc). Can you double-check that `dataset_infos.json` isn't just metadata please?

Yes，you are right!So if I have a metadata dataset local,How can I turn it to a dataset that can be used by the load_dataset() function？Are there some examples?

The metadata file isn't a dataset so you can't turn it into one. You should try @albertvillanova's code snippet above (now merged in the docs [here](https://huggingface.co/docs/datasets/master/en/loading#local-loading-script)), which uses your local loading script `opus_books.py` to:

1. Download the actual dataset. 
2. Once the dataset is downloaded, `load_dataset` will load it for you.