SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: `SystemError 15` thrown in `Dataset.__del__` when using `Dataset.map()` with `num_proc>1`

Comment:
Hi @mariosasko, thank you for the hint, that helped me to move forward with that issue. 

I did a major refactoring of my project to disentangle my `LightningDataModule` and `Dataset`.  Just FYI, it looks like:

```python
class Builder():
   def __call__() -> DatasetDict:
      # load and preprocess the data
      return dataset

class DataModule(LightningDataModule):
    def prepare_data():
       self.builder()
    def setup():
       self.dataset = self.builder()
```

Unfortunately, the entanglement between `LightningDataModule` and `Dataset` was not the issue.

The culprit was `hydra` and a slight adjustment of the structure of my project solved this issue. The problematic project  structure was:

```
src/
 | - cli.py
 | - training/
      | -experiment.py

# code in experiment.py
def run_experiment(config):
     # preprocess data and run
     
# code in cli.py
@hydra.main(...)
def run(config):
    return run_experiment(config)
```

Moving `run()` from `clip.py` to `training.experiment.py` solved the issue with `SystemError 15`. No idea why. 

Even if the traceback was referring to `Dataset.__del__`, the problem does not seem to be primarily related to `datasets`, so I will close this issue. Thank you for your help!
Issue: `SystemError 15` thrown in `Dataset.__del__` when using `Dataset.map()` with `num_proc>1`

Comment:
Please allow me to revive this discussion, as I have an extremely similar issue. Instead of an error, my datasets functions simply aren't caching properly. My setup is almost the same as yours, with hydra to configure my experiment parameters.

@vlievin Could you confirm if your code correctly loads the cache? If so, do you have any public code that I can reference for comparison?

I will post a full example with hydra that illustrates this problem in a little bit, probably on another thread.
Issue: `SystemError 15` thrown in `Dataset.__del__` when using `Dataset.map()` with `num_proc>1`

Comment:
Hello @mariomeissner, very sorry for the late reply, I hope you have found a solution to your problem!

I don't have public code at the moment. I have not experienced any other issue with hydra, even if I don't understand why changing the location of the definition of `run()` fixed the problem. 

Overall, I don't have issue with caching anymore, even when 
1. using custom fingerprints using the argument `new_fingerprint 
2. when using `num_proc>1`
Issue: `SystemError 15` thrown in `Dataset.__del__` when using `Dataset.map()` with `num_proc>1`

Comment:
I solved my issue by turning the map callable into a class static method, like they do in `lightning-transformers`. Very strange...
Issue: `SystemError 15` thrown in `Dataset.__del__` when using `Dataset.map()` with `num_proc>1`

Comment:
I have this issue with datasets v2.5.2 with Python 3.8.10 on Ubuntu 20.04.4 LTS.  It does not occur when num_proc=1.  When num_proc>1, it intermittently occurs and will cause process to hang.  As previously mentioned, it occurs even when datasets have been previously cached.  I have tried wrapping logic in a static class as suggested with @mariomeissner with no improvement.
Issue: Raise exceptions instead of using assertions for control flow

Comment:
Adding the remaining tasks for this issue to help new code contributors.  
$ cd src/datasets && ack assert -lc 
- [x] commands/convert.py:1
- [x] arrow_reader.py:3
- [x] load.py:7
- [x] utils/py_utils.py:2
- [x] features/features.py:9
- [x] arrow_writer.py:7
- [x] search.py:6
- [x] table.py:1
- [x] metric.py:3
- [x] tasks/image_classification.py:1
- [x] arrow_dataset.py:17
- [x] fingerprint.py:6
- [x] io/json.py:1
- [x] io/csv.py:1
Issue: Raise exceptions instead of using assertions for control flow

Comment:
Hi all,
I am interested in taking up `fingerprint.py`, `search.py`, `arrow_writer.py` and `metric.py`. Will raise a PR soon!
Issue: OpenSLR/83 is empty

Comment:
Hi @tyrius02, thanks for reporting. I see you self-assigned this issue: are you working on this?
Issue: OpenSLR/83 is empty

Comment:
@albertvillanova Yes. Figured I introduced the broken config, I should fix it too.

I've got it working, but I'm struggling with one of the tests. I've started a PR so I/we can work through it.
Issue: bookcorpusopen no longer works

Comment:
I tried with the latest changes from #3280 on google colab and it worked fine :)
We'll do a new release soon, in the meantime you can use the updated version with:
```python
load_dataset("bookcorpusopen", revision="master")
```
Issue: Add raw data files to the Hub with GitHub LFS for canonical dataset

Comment:
Hi @zlucia, I would actually suggest hosting the dataset as a huggingface.co-hosted dataset.

The only difference with a "canonical"/legacy dataset is that it's nested under an organization (here `stanford` or `stanfordnlp` for instance – completely up to you) but then you can upload your data using git-lfs (unlike "canonical" datasets where we don't host the data)

Let me know if this fits your use case!

cc'ing @osanseviero @lhoestq and rest of the team 🤗
Issue: Add raw data files to the Hub with GitHub LFS for canonical dataset

Comment:
Hi @zlucia,

As @julien-c pointed out, the way to store/host raw data files in our Hub is by using what we call "community" datasets:
- either at your personal namespace: `load_dataset("zlucia/casehold")`
- or at an organization namespace: for example, if you create the organization `reglab`, then `load_dataset("reglab/casehold")`

Please note that "canonical" datasets do not normally store/host their raw data at our Hub, but in a third-party server. For "canonical" datasets, we just host the "loading script", that is, a Python script that downloads the raw data from a third-party server, creates the HuggingFace dataset from it and caches it locally.

In order to create an organization namespace in our Hub, please follow this link: https://huggingface.co/organizations/new

There are already many organizations at our Hub (complete list here: https://huggingface.co/organizations), such as:
- Stanford CRFM: https://huggingface.co/stanford-crfm
- Stanford NLP: https://huggingface.co/stanfordnlp
- Stanford CS329S: Machine Learning Systems Design: https://huggingface.co/stanford-cs329s

Also note that you in your organization namespace:
- you can add any number of members
- you can store both raw datasets and models, and those can be immediately accessed using `datasets` and `transformers`

Once you have created an organization, these are the steps to upload/host a raw dataset: 
- The no-code procedure: https://huggingface.co/docs/datasets/upload_dataset.html
- Using the command line (terminal): https://huggingface.co/docs/datasets/share.html#add-a-community-dataset

Please, feel free to ping me if you have any further questions or need help.
Issue: Add raw data files to the Hub with GitHub LFS for canonical dataset

Comment:
Ah I see, I think I was unclear whether there were benefits to uploading a canonical dataset vs. a community provided dataset. Thanks for clarifying. I'll see if we want to create an organization namespace and otherwise, will upload the dataset under my personal namespace.
Issue: `datasets-cli test` should work with datasets without scripts

Comment:
> It would be really useful to be able to run `datasets-cli test`for datasets that don't have scripts attached to them (whether the datasets are private or not).
> 
> I wasn't able to run the script for a private test dataset that I had created on the hub (https://huggingface.co/datasets/huggingface/DataMeasurementsTest/tree/main) -- although @lhoestq came to save the day!

why don't you try to share that info with people, so you can also save some days.
Issue: `datasets-cli test` should work with datasets without scripts

Comment:
Hi ! You can run the command if you download the repository
```
git clone https://huggingface.co/datasets/huggingface/DataMeasurementsTest
```
and run the command
```
datasets-cli test DataMeasurementsTest/DataMeasurementsTest.py
```

(though on my side it doesn't manage to download the data since the dataset is private ^^)
Issue: `datasets-cli test` should work with datasets without scripts

Comment:
> Hi ! You can run the command if you download the repository
> 
> ```
> git clone https://huggingface.co/datasets/huggingface/DataMeasurementsTest
> ```
> 
> and run the command
> 
> ```
> datasets-cli test DataMeasurementsTest/DataMeasurementsTest.py
> ```
> 
> (though on my side it doesn't manage to download the data since the dataset is private ^^)

Hi! Thanks for the info. 
git cannot find the repository. Do you know if they have depreciated these tests and created a new one?
Issue: `datasets-cli test` should work with datasets without scripts

Comment:
I think it's become private, but feel free to try with any other dataset like `lhoestq/test` for example at `https://huggingface.co/datasets/lhoestq/test`
Issue: `datasets-cli test` should work with datasets without scripts

Comment:
> I think it's become private, but feel free to try with any other dataset like `lhoestq/test` for example at `https://huggingface.co/datasets/lhoestq/test`

your example repo and this page `https://huggingface.co/docs/datasets/add_dataset.html` helped me to solve.. thanks a lot
Issue: Illegal instruction (core dumped) at datasets import

Comment:
It seems to be an issue with how conda-forge is building the binaries. It works on some machines, but not a machine with AMD Opteron 8384 processors.
Issue: Sacrebleu unexpected behaviour/requirement for data format

Comment:
Hi @BramVanroy!

Good question. This project relies on PyArrow (tables) to store data too big to fit in RAM. In the case of metrics, this means that the number of predictions and references has to match to form a table.

That's why your example throws an error even though it matches the schema:
```python
refs = [
    ['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
    ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'],
]  # len(refs) = 2

hyps = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.'] # len(hyps) = 3
```

Instead, it should be:
```python
refs = [
    ['The dog bit the man.', 'The dog had bit the man.'],
    ['It was not unexpected.', 'No one was surprised.'],
    ['The man bit him first.', 'The man had bitten the dog.'], 
]  # len(refs) = 3

hyps = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.'] # len(hyps) = 3
```

However, `sacreblue` works with the format that's described in your example, hence this part:
https://github.com/huggingface/datasets/blob/87c71b9c29a40958973004910f97e4892559dfed/metrics/sacrebleu/sacrebleu.py#L94-L99

Hope you get an idea!
Issue: Sacrebleu unexpected behaviour/requirement for data format

Comment:
Thanks, that makes sense. It is a bit unfortunate because it may be confusing to users since the input format is suddenly different than what they may expect from the underlying library/metric. But it is understandable due to how `datasets` works!
Issue: Streaming with num_workers != 0

Comment:
I can confirm that I was able to reproduce the bug. This seems odd given that #3423 reports duplicate data retrieval when `num_workers` and `streaming` are used together, which is obviously different from what is reported here.
Issue: Streaming with num_workers != 0

Comment:
Any update? A possible solution is to have multiple arrow files as shards, and handle them like what webdatasets does.
![image](https://user-images.githubusercontent.com/11533479/148176637-72746b2c-c122-47aa-bbfe-224b13ee9a71.png)

Pytorch's new dataset RFC is supporting sharding now, which may helps avoid duplicate data under streaming mode. (https://github.com/pytorch/pytorch/blob/master/torch/utils/data/datapipes/iter/grouping.py#L13)
Issue: Streaming with num_workers != 0

Comment:
Hi ! Thanks for the insights :) Note that in streaming mode there're usually no arrow files. The data are streamed from TAR, ZIP, text, etc. files directly from the web. Though for sharded datasets we can definitely adopt a similar strategy !
Issue: [when Image type will exist] provide a way to get the data as binary + filename

Comment:
@severo I'll keep that in mind.

You can track progress on the Image feature in #3163 (still in the early stage).
Issue: [when Image type will exist] provide a way to get the data as binary + filename

Comment:
Hi ! As discussed with @severo offline it looks like the dataset viewer already supports reading PIL images, so maybe the dataset viewer doesn't need to disable decoding after all
Issue: Provide a way to write a streamed dataset to the disk

Comment:
Yes, I agree this feature is much needed. We could do something similar to what TF does (https://www.tensorflow.org/api_docs/python/tf/data/Dataset#cache). 

Ideally, if the entire streamed dataset is consumed/cached, the generated cache should be reusable for the Arrow dataset.
Issue: Make inspect.get_dataset_config_names always return a non-empty list of configs

Comment:
Hi @severo, I guess this issue requests not only to be able to access the configuration name (by using `inspect.get_dataset_config_names`), but the configuration itself as well (I mean you use the name to get the configuration afterwards, maybe using `builder_cls.builder_configs`), is this right?
Issue: Make inspect.get_dataset_config_names always return a non-empty list of configs

Comment:
Yes, maybe the issue could be reformulated. As a user, I want to avoid having to manage special cases:
- I want to be able to get the names of a dataset's configs, and use them in the rest of the API (get the data, get the split names, etc).
- I don't want to have to manage datasets with named configs (`glue`) differently from datasets without named configs (`acronym_identification`, `Check/region_1`)
Issue: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/rouge/rouge.py

Comment:
Hi,

Did you try to run the code multiple times (GitHub URLs can be down sometimes for various reasons)? I can access `https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/rouge/rouge.py`, so this code is working without an error on my side. 

Additionally, can you please run the `datasets-cli env` command because it seems to me that you are using the `datasets` version different from `1.12.1`?
Issue: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/rouge/rouge.py

Comment:
Same issue when running `metric = datasets.load_metric("accuracy")`.
Error info is:
```
metric = datasets.load_metric("accuracy")
Traceback (most recent call last):

  File "<ipython-input-2-d25db38b26c5>", line 1, in <module>
    metric = datasets.load_metric("accuracy")

  File "D:\anaconda3\lib\site-packages\datasets\load.py", line 610, in load_metric
    module_path, _ = prepare_module(

  File "D:\anaconda3\lib\site-packages\datasets\load.py", line 330, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)

  File "D:\anaconda3\lib\site-packages\datasets\utils\file_utils.py", line 288, in cached_path
    output_path = get_from_cache(

  File "D:\anaconda3\lib\site-packages\datasets\utils\file_utils.py", line 605, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/accuracy/accuracy.py
```


 My `datasets-cli env` result is as follows:
- `datasets` version: 1.11.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyArrow version: 6.0.0

@yananchen1989 did you find a way to solve this?
Issue: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/rouge/rouge.py

Comment:
It seems to be able to solve this issue by adding the equivalent `accuracy.py` locally. 
change `metric = datasets.load_metric("accuracy")` to  `metric = datasets.load_metric(path = "./accuracy.py")`.
Copy `accuracy.py` from browser at [accuracy.py](https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/accuracy/accuracy.py)
Issue: datasets-cli: convertion of a tfds dataset to a huggingface one.

Comment:
Hi,

the MNIST dataset is already available on the Hub. You can use it as follows:
```python
import datasets
dataset_dict = datasets.load_dataset("mnist")
```

As for the conversion of TFDS datasets to HF datasets, we will be working on it in the coming months, so stay tuned.
Issue: "arabic_billion_words" dataset does not create the full dataset

Comment:
Thanks for reporting, @vitalyshalumov.

Apparently the script to parse the data has a bug, and does not generate the entire dataset.

I'm fixing it.
Issue: Segmentation fault when loading datasets from file

Comment:
Hi ! I created an issue on Arrow's JIRA after making a minimum reproducible example

https://issues.apache.org/jira/browse/ARROW-14439

```python
import io

import pyarrow.json as paj

batch = b'{"a": [], "b": 1}\n{"b": 1}'
block_size = 12

paj.read_json(
    io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
)
```

I don't see a way to workaround this properly now without hurting the performance of the JSON loader significantly though
Issue: Segmentation fault when loading datasets from file

Comment:
The issue has been fixed in pyarrow 6.0.0, please update pyarrow :)

The issue was due to missing fields in the JSON data of type list. Now it's working fine and missing list fields are replaced with empty lists
Issue: OSError with a custom dataset loading script

Comment:
Hi,

there is a difference in how the `data_dir` is zipped between the `classla/janes_tag` and the `classla/reldi_hr` dataset. After unzipping, for the former, the data files (`*.conllup`) are in the root directory (root -> data files), and for the latter, they are inside the `data` directory (root -> `data` -> data files).

This can be fixed by removing the `os.path.join` call in https://huggingface.co/datasets/classla/janes_tag/blob/main/janes_tag.py#L86

Let me know if this works for you.
Issue: OSError with a custom dataset loading script

Comment:
Hi Mario,

I had already tried that before, but it didn't work. I have now recreated the `classla/janes_tag`  zip file so that it also contains the `data` directory, but I am still getting the same error.
Issue: OSError with a custom dataset loading script

Comment:
Hi,

I just tried to download the `classla/janes_tag` dataset, and this time the zip file is extracted correctly. However, the script is now throwing the IndexError, probably due to a bug in the `_generate_examples`.

Let me know if you are still getting the same error.
Issue: OSError with a custom dataset loading script

Comment:
Hi, 

could you try to download the dataset with a different `cache_dir` like so:
```python
import datasets
dataset = datasets.load_dataset('classla/janes_tag', split='validation', cache_dir="path/to/different/cache/dir")
```
If this works, then most likely the cached extracted data is causing issues. This data is stored at `~/.cache/huggingface/datasets/downloads/extracted` and needs to be deleted, and then it should work (you can easily locate the directory with the path given in the `OSError` message). Additionally, I'd suggest you to update `datasets` to the newest version with:
```
pip install -U datasets
```
Issue: OSError with a custom dataset loading script

Comment:
Thank you, deleting the `~/.cache/huggingface/datasets/downloads/extracted` directory helped. However, I am still having problems.

There was indeed a bug in the script that was throwing an `IndexError`, which I have now corrected (added the condition to skip the lines starting with '# text') and it is working locally, but still throws an error when I try to load the dataset from HuggingFace. I literally copied and pasted the `_generate_examples` function and ran it on the `dev_all.conllup` file, which I even re-downloaded from the repository to be certain that the files are exactly the same. I also deleted everything again just in case, but it didn't help. The code works locally, but throws an `IndexError` when loading from `datasets.`
Issue: OSError with a custom dataset loading script

Comment:
Hi,

Did some investigation.

To fix the dataset script on the Hub, append the following labels to the `names` list of the `upos_tags` field:
```'INTJ NOUN', 'AUX PRON', 'PART ADV', 'PRON ADP', 'INTJ INTJ', 'VERB NOUN', 'NOUN AUX'```.

This step is required to avoid an error due to missing labels in the following step which is:
```python
load_dataset("classla/janes_tag", split="validation", download_mode="force_redownload")
```
This will generate and cache the dataset, so specifying `download_mode` will not be required anymore unless you update the script/data on the Hub.
Issue: Add OpenSLR 83 - Crowdsourced high-quality UK and Ireland English Dialect speech

Comment:
Ugh. The index files for SLR83 are CSV, not TSV. I need to add logic to process these index files.
Issue: load_from_disk in DatasetsDict/Dataset not working with PyArrowHDFS wrapper implementing fsspec.spec.AbstractFileSystem

Comment:
Hi ! Can you try again with pyarrow 6.0.0 ? I think it includes some changes regarding filesystems compatibility with fsspec.
Issue: load_from_disk in DatasetsDict/Dataset not working with PyArrowHDFS wrapper implementing fsspec.spec.AbstractFileSystem

Comment:
Hi @lhoestq! I ended up using `fsspec.implementations.arrow.HadoopFileSystem` which doesn't have the problem I described with pyarrow 5.0.0.

I'll try again with `PyArrowHDFS` once I update arrow to 6.0.0.

Thanks!
Issue: Loading Data from HDF files

Comment:
I would also like this support or something similar. Geospatial datasets come in netcdf which is derived from hdf5, or zarr. I've gotten zarr stores to work with datasets and streaming, but it takes awhile to convert the data to zarr if it's not stored in that natively.
Issue: Loading Data from HDF files

Comment:
@mariosasko , I would like to contribute on this "good second issue" . Is there anything in the works for this Issue or can I go ahead ?
Issue: Loading Data from HDF files

Comment:
Hi @VijayKalmath! As far as I know, nobody is working on it, so feel free to take over. Also, before you start, I suggest you comment `#self-assign` on this issue to assign it to yourself.
Issue: OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

Comment:
I am very unsure on why you tagged me here. I am not a maintainer of the Datasets library and have no idea how to help you.
Issue: OverflowError: There was an overflow in the <class 'pyarrow.lib.ListArray'>. Try to reduce writer_batch_size to have batches smaller than 2GB

Comment:
Ok got it, tensor full of NaNs, cf.

~\anaconda3\envs\xxx\lib\site-packages\datasets\arrow_writer.py in write_examples_on_file(self)
315 # This check fails with FloatArrays with nans, which is not what we want, so account for that:
Issue: AttributeError: module 'huggingface_hub.hf_api' has no attribute 'DatasetInfo'

Comment:
Hi @JTWang2000, thanks for reporting.

However, I cannot reproduce your reported bug:
```python
>>> from datasets import load_dataset

>>> dataset = load_dataset("sst", "default")
>>> dataset
DatasetDict({
    train: Dataset({
        features: ['sentence', 'label', 'tokens', 'tree'],
        num_rows: 8544
    })
    validation: Dataset({
        features: ['sentence', 'label', 'tokens', 'tree'],
        num_rows: 1101
    })
    test: Dataset({
        features: ['sentence', 'label', 'tokens', 'tree'],
        num_rows: 2210
    })
})
```

Maybe, the cause is that you have a quite old version of `huggingface_hub`. Could you please try to update it and confirm if the problem persists?
```
pip install -U huggingface_hub
```