❓ Dataset loading script from Hugging Face Hub
Opened by pietrolesci (closed)

Hi there,

I am trying to add my custom `ag_news` with its own loading script on the Hugging Face datasets hub. In particular, I would like to test the addition of a second configuration to the existing `ag_news` dataset. Once it works in my hub, I plan to make a PR to the original dataset. However, in trying to do so I have encountered certain problems as detailed below.

Issues I have encountered:
- Without a loading script, the train and test files are loaded together into a unique `dataset.Dataset` -> so I wrote a loading script. Also, I need a loading script otherwise I cannot specify multiple configurations
- Once my loading script is working locally, I do not manage to make it work on the hub. In particular, I would like to be able to load the dataset like this
```python
load_dataset("pietrolesci/ag_news", name="my_configuration")
```

Apparently, the `load_dataset` is able to pick up the loading script from the hub and run it. However, it errors because it is unable to find the files. The structure of my hub repo is the following
```
ag_news.py
train.csv
test.csv
```
and the loading script I specify  `data_dir=Path(__file__).parent` and `data_files=DataFilesDict({"train": "train.csv", "test": "test.csv"})`. In the documentation I could not find info regarding loading a dataset from the hub using a loading script present on the hub.

Any suggestion is very much appreciated.

Best,
Pietro

Link to the hub repo: https://huggingface.co/datasets/pietrolesci/ag_news

BONUS: how can I make the data viewer work in this specific case? :)

Comments:
Hi ! In the next version of `datasets`, your train and test splits will be correctly separated (changes from #3027) if you create a dataset repository with only your CSV files.

Also it seems that you overwrite the `data_files` and `data_dir` arguments in your code, when you instantiate the AGNewsConfig objects. Those parameters are not necessary since you already know which files you want to load.

You can find an example on how to specify which file the dataset has to download in this [example script](https://huggingface.co/datasets/lhoestq/custom_squad/blob/main/custom_squad.py#L101-L107):
```python
_URLS = {
    "train": "train-v1.1.json",  # you can use a URL or a relative path from the python script to your file in the repository
    "dev": "dev-v1.1.json",
}
```
```python
    def _split_generators(self, dl_manager):
        downloaded_files = dl_manager.download_and_extract(_URLS)

        return [
            datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={"filepath": downloaded_files["train"]}),
            datasets.SplitGenerator(name=datasets.Split.VALIDATION, gen_kwargs={"filepath": downloaded_files["dev"]}),
        ]
```

Also I think the viewer will be updated when you fix the dataset script, let me know if it doesn't

Hi @lhoestq,

Thanks a lot for the super quick answer!

Your suggestion solves my issue. I am now able to load the dataset properly 🚀 
However, the dataviewer is not working yet.

Really, thanks a lot for your help and consideration!

Best,
Pietro

Great ! We'll take a look at the viewer to fix it

@lhoestq  I think I am having a related problem.
My call to load_dataset() looks like this:

```
    datasets = load_dataset(
        os.path.abspath(layoutlmft.data.datasets.xfun.__file__),
        f"xfun.{data_args.lang}",
        additional_langs=data_args.additional_langs,
        keep_in_memory=True,
    )

```

My _split_generation code is:

```
    def _split_generators(self, dl_manager):
        """Returns SplitGenerators."""

        downloaded_file = dl_manager.download_and_extract("https://guillaumejaume.github.io/FUNSD/dataset.zip")
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN, gen_kwargs={"filepath": f"{downloaded_file}/dataset/training_data/"}
            ),
            datasets.SplitGenerator(
                name=datasets.Split.TEST, gen_kwargs={"filepath": f"{downloaded_file}/dataset/testing_data/"}
            ),
        ]

```
However I get the error "TypeError: _generate_examples() got an unexpected keyword argument 'filepath'"
The path looks right and I see the data in the path so I think the only problem I have is that it doesn't like the key "filepath".  However, the documentation (example [here](https://huggingface.co/datasets/lhoestq/custom_squad/blob/main/custom_squad.py#L101-L107)) seems to show that this is the correct parameter. 

Here is the full stack trace:

```
Downloading and preparing dataset xfun/xfun.en (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /Users/caseygre/.cache/huggingface/datasets/xfun/xfun.en/0.0.0/96b8cb7c57f6f822f0ab37ae3be7b82d84ac57062e774c9361ccf0a4b9ef61cc...
Traceback (most recent call last):
  File "/Users/caseygre/PycharmProjects/aegis-ml-new/unilm/venv-LayoutLM/lib/python3.9/site-packages/datasets/builder.py", line 574, in download_and_prepare
    self._download_and_prepare(
  File "/Users/caseygre/PycharmProjects/aegis-ml-new/unilm/venv-LayoutLM/lib/python3.9/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/Users/caseygre/PycharmProjects/aegis-ml-new/unilm/venv-LayoutLM/lib/python3.9/site-packages/datasets/builder.py", line 975, in _prepare_split
    generator = self._generate_examples(**split_generator.gen_kwargs)
TypeError: _generate_examples() got an unexpected keyword argument 'filepath'
python-BaseException
```

Hi ! The `gen_kwargs` dictionary is passed to `_generate_examples`, so in your case it must be defined this way:
```python
def _generate_examples(self, filepath):
    ...
```

And here is an additional tip: you can use `os.path.join(downloaded_file, "dataset/testing_data")` instead of `f"downloaded_file}/dataset/testing_data/"` to get compatibility with Windows and streaming.

Indeed Windows uses a backslash separator, not a slash, and streaming uses chained URLs (like `zip://dataset/testing_data::https://https://guillaumejaume.github.io/FUNSD/dataset.zip` for example)

Thanks for you quick reply @lhoestq and so sorry for my very delayed response.
We have gotten around the error another way but I will try to duplicate this when I can.  We may have had "filepaths" instead of "filepath" in our def of _generate_examples() and not noticed the difference.  If I find a more useful answer for others I will add to this ticket so they know what the issue was.
Note: we do have our own _generate_examples() defined with the same def as Quentin has.  (But one version does have "filepaths".)


Fixed in the viewer: https://huggingface.co/datasets/pietrolesci/ag_news