SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Improve torch formatting performance

Comment:
@lhoestq @thomwolf @albertvillanova The new approach is definitely faster, dataloader now takes less than 3% cumulative time (pink rectangle two rectangles to the right of tensor.py backward invocation)

![Screen Shot 2021-06-16 at 3 05 06 PM](https://user-images.githubusercontent.com/458335/122224432-19de4700-ce82-11eb-982f-d45d4bcc1e41.png)

When we drill down into dataloader next invocation we get:

![Screen Shot 2021-06-16 at 3 09 56 PM](https://user-images.githubusercontent.com/458335/122224976-a1c45100-ce82-11eb-8d40-59194740d616.png)

And finally format_batch:

![Screen Shot 2021-06-16 at 3 11 07 PM](https://user-images.githubusercontent.com/458335/122225132-cae4e180-ce82-11eb-8a16-967ab7c1c2aa.png)


Not sure this could be further improved but this is definitely a decent step forward.
Issue: Improve torch formatting performance

Comment:
> ```python
> datasets @ git+ssh://git@github.com/huggingface/datasets.git@refs/pull/2505/head
> ```

@albertvillanova how would I replace datasets dependency in https://github.com/huggingface/transformers/blob/master/setup.py as the above approach is not working.
Issue: Improve torch formatting performance

Comment:
@vblagoje I tested my proposed approach before posting it here and it worked for me. 

Is it not working in your case because of the SSH protocol? In that case you could try the same approach but using HTTPS:
```
"datasets @ git+https://github.com/huggingface/datasets.git@refs/pull/2505/head",
```
Issue: Improve torch formatting performance

Comment:
@albertvillanova of course it works. Apologies. I needed to change datasets in all deps references , like [here](https://github.com/huggingface/transformers/blob/master/setup.py#L235) for example.
Issue: Improve torch formatting performance

Comment:
Is time spent casting an issue here? See https://github.com/huggingface/datasets/issues/4676 that Datasets can spend huge amounts of time repeatedly casting to Python objects.
Issue: Delete extracted files to save disk space

Comment:
My suggestion for this would be to have this enabled by default.

Plus I don't know if there should be a dedicated issue to that is another functionality. But I propose layered building rather than all at once. That is:

1. uncompress a handful of files via a generator enough to generate one arrow file
2. process arrow file 1
3. delete all the files that went in and aren't needed anymore.

rinse and repeat.

1. This way much less disc space will be required - e.g. on JZ we won't be running into inode limitation, also it'd help with the collaborative hub training project
2. The user doesn't need to go and manually clean up all the huge files that were left after pre-processing
3. It would already include deleting temp files this issue is talking about

I wonder if the new streaming API would be of help, except here the streaming would be into arrow files as the destination, rather than dataloaders.
Issue: Set download/extracted paths configurable

Comment:
For example to be able to send uncompressed and temp  build files to another volume/partition, so that the user gets the minimal disk usage on their primary setup - and ends up with just the downloaded compressed data + arrow files, but outsourcing the huge files and building to another partition. e.g. on JZ there is a special partition for fast data, but it's also volatile, so only temp files should go there.

Think of it as `TMPDIR` so we need the equivalent for `datasets`.
Issue: Create release script

Comment:
I've aligned the release script with Transformers in #6004, so I think this issue can be closed.
Issue: cache_dir parameter for load_from_disk ?

Comment:
Hi ! `load_from_disk` doesn't move the data. If you specify a local path to your mounted drive, then the dataset is going to be loaded directly from the arrow file in this directory. The cache files that result from `map` operations are also stored in the same directory by default.

However note than writing data to your google drive actually fills the VM's disk (see https://github.com/huggingface/datasets/issues/643)

Given that, I don't think that changing the cache directory changes anything.

Let me know what you think
Issue: cache_dir parameter for load_from_disk ?

Comment:
Thanks for your answer! I am a little surprised since I just want to read the dataset.

After debugging a bit, I noticed that the VM’s disk fills up when the tables (generator) are converted to a list:

https://github.com/huggingface/datasets/blob/5ba149773d23369617563d752aca922081277ec2/src/datasets/table.py#L850

If I try to iterate through the table’s generator e.g.: 

`length = sum(1 for x in tables)`

the VM’s disk fills up as well.

I’m running out of Ideas 😄
Issue: cache_dir parameter for load_from_disk ?

Comment:
Indeed reading the data shouldn't increase the VM's disk. Not sure what google colab does under the hood for that to happen
Issue: cache_dir parameter for load_from_disk ?

Comment:
Apparently, Colab uses a local cache of the data files read/written from Google Drive. See:
- https://github.com/googlecolab/colabtools/issues/2087#issuecomment-860818457
- https://github.com/googlecolab/colabtools/issues/1915#issuecomment-804234540
- https://github.com/googlecolab/colabtools/issues/2147#issuecomment-885052636
Issue: Fix automatic generation of Zenodo DOI

Comment:
I have received a reply from Zenodo support:
> We are currently investigating and fixing this issue related to GitHub releases. As soon as we have solved it we will reach back to you.
Issue: Fix automatic generation of Zenodo DOI

Comment:
Other repo maintainers had the same problem with Zenodo. 

There is an open issue on their GitHub repo: zenodo/zenodo#2181
Issue: Fix automatic generation of Zenodo DOI

Comment:
I have received the following request from Zenodo support:
> Could you send us the link to the repository as well as the release tag?

My reply:
> Sure, here it is:
> - Link to the repository: https://github.com/huggingface/datasets
> - Link to the repository at the release tag: https://github.com/huggingface/datasets/releases/tag/1.8.0
> - Release tag: 1.8.0
Issue: Crash when `num_proc` > dataset length for `map()` on a `datasets.Dataset`.

Comment:
Hi ! It looks like the issue comes from pyarrow. What version of pyarrow are you using ? How did you install it ?
Issue: Crash when `num_proc` > dataset length for `map()` on a `datasets.Dataset`.

Comment:
Thank you for the quick reply! I have `pyarrow==4.0.0`, and I am installing with `pip`. It's not one of my explicit dependencies, so I assume it came along with something else.
Issue: Crash when `num_proc` > dataset length for `map()` on a `datasets.Dataset`.

Comment:
Could you trying reinstalling pyarrow with pip ?
I'm not sure why it would check in your multicurtural-sc directory for source files.
Issue: Crash when `num_proc` > dataset length for `map()` on a `datasets.Dataset`.

Comment:
Sure! I tried reinstalling to get latest. pip was mad because it looks like Datasets currently wants <4.0.0 (which is interesting, because apparently I ended up with 4.0.0 already?), but I gave it a shot anyway:

```bash
$ pip install --upgrade --force-reinstall pyarrow
Collecting pyarrow
  Downloading pyarrow-4.0.1-cp39-cp39-manylinux2014_x86_64.whl (21.9 MB)
     |████████████████████████████████| 21.9 MB 23.8 MB/s
Collecting numpy>=1.16.6
  Using cached numpy-1.20.3-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.4 MB)
Installing collected packages: numpy, pyarrow
  Attempting uninstall: numpy
    Found existing installation: numpy 1.20.3
    Uninstalling numpy-1.20.3:
      Successfully uninstalled numpy-1.20.3
  Attempting uninstall: pyarrow
    Found existing installation: pyarrow 3.0.0
    Uninstalling pyarrow-3.0.0:
      Successfully uninstalled pyarrow-3.0.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 1.8.0 requires pyarrow<4.0.0,>=1.0.0, but you have pyarrow 4.0.1 which is incompatible.
Successfully installed numpy-1.20.3 pyarrow-4.0.1
```

Trying it, the same issue:

![image](https://user-images.githubusercontent.com/1170062/121730226-3f470b80-caa4-11eb-85a5-684c44c816da.png)

I tried installing `"pyarrow<4.0.0"`, which gave me 3.0.0. Running, still, same issue.

I agree it's weird that pyarrow is checking the source code directory for its files. (There is no `pyarrow/` directory there.) To me, that makes it seem like an issue with how pyarrow is called.

Out of curiosity, I tried running this with fewer workers to see when the error arises:

- 1: ✅
- 2: ✅
- 4: ✅
- 8: ✅
- 10: ✅
- 11: ❌  🤔
- 12: ❌
- 16: ❌
- 32: ❌

checking my datasets:

```python
>>> datasets
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 389290
    })
    validation.sc: Dataset({
        features: ['text'],
        num_rows: 10  # 🤔
    })
    validation.wvs: Dataset({
        features: ['text'],
        num_rows: 93928
    })
})
```

New hypothesis: crash if `num_proc` > length of a dataset? 😅

If so, this might be totally my fault, as the caller. Could be a docs fix, or maybe this library could do a check to limit `num_proc` for this case?
Issue: Crash when `num_proc` > dataset length for `map()` on a `datasets.Dataset`.

Comment:
Good catch ! Not sure why it could raise such a weird issue from pyarrow though
We should definitely reduce num_proc to the length of the dataset if needed and log a warning.
Issue: Crash when `num_proc` > dataset length for `map()` on a `datasets.Dataset`.

Comment:
This has been fixed in #2566, thanks @connor-mccarthy !
We'll make a new release soon that includes the fix ;)
Issue: Merge DatasetDict and Dataset

Comment:
Unless there is high demande I don't think we will end up implementing this. This is a lot of work with very few advantages
Issue: BLUE file not found

Comment:
Hi ! The `blue` metric doesn't exist, but the `bleu` metric does.
You can get the full list of metrics [here](https://github.com/huggingface/datasets/tree/master/metrics) or by running
```python
from datasets import list_metrics

print(list_metrics())
```
Issue: dataset adversarial_qa has no answers in the "test" set

Comment:
Hi ! I'm pretty sure that the answers are not made available for the test set on purpose because it is part of the DynaBench benchmark, for which you can submit your predictions on the website.
In any case we should mention this in the dataset card of this dataset.
Issue: dataset adversarial_qa has no answers in the "test" set

Comment:
Makes sense, but not intuitive for someone searching through the datasets.  Thanks for adding the note to clarify.
Issue: `yelp_polarity` is broken

Comment:
```
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/script_runner.py", line 332, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp-viewer/run.py", line 233, in <module>
    configs = get_confs(option)
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/caching.py", line 604, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/caching.py", line 588, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp-viewer/run.py", line 148, in get_confs
    builder_cls = nlp.load.import_main_class(module_path[0], dataset=True)
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/datasets/load.py", line 85, in import_main_class
    module = importlib.import_module(module_path)
File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/sasha/.cache/huggingface/modules/datasets_modules/datasets/yelp_polarity/a770787b2526bdcbfc29ac2d9beb8e820fbc15a03afd3ebc4fb9d8529de57544/yelp_polarity.py", line 36, in <module>
    from datasets.tasks import TextClassification
```
Issue: Sentence Boundaries missing in Dataset: xtreme / udpos

Comment:
Hi,

This is a known issue. More info on this issue can be found in #2061. If you are looking for an open-source contribution, there are step-by-step instructions in the linked issue that you can follow to fix it.
Issue: Some tests hang on Windows

Comment:
Hi ! That would be nice indeed to at least have a warning, since we don't handle the max path length limit.
Also if we could have an error instead of an infinite loop I'm sure windows users will appreciate that
Issue: Some tests hang on Windows

Comment:
Unfortunately, I know this problem very well... 😅 

I remember having proposed to throw an error instead of hanging in an infinite loop #2220: 60c7d1b6b71469599a27147a08100f594e7a3f84, 8c8ab60018b00463edf1eca500e434ff061546fc 
but @lhoestq told me:
> Note that the filelock module comes from this project that hasn't changed in years - while still being used by ten of thousands of projects:
https://github.com/benediktschmitt/py-filelock
> 
> Unless we have proper tests for this, I wouldn't recommend to change it

I opened an Issue requesting a warning/error at startup for that case: #2224
Issue: Some tests hang on Windows

Comment:
@albertvillanova Thanks for additional info on this issue.

Yes, I think the best option is to throw an error instead of suppressing it in a loop. I've considered 2 more options, but I don't really like them:
1. create a temporary file with a filename longer than 255 characters on import; if this fails, long paths are not enabled and raise a warning. I'm not sure about this approach because I don't like the idea of creating a temporary file on import for this purpose.
2. check if long paths are enabled with [this code](https://stackoverflow.com/a/46546731/14095927). As mentioned in the comment, this code relies on an undocumented function and Win10-specific.
Issue: DuplicatedKeysError on personal dataset

Comment:
Hi ! In your dataset script you must be yielding examples like
```python
for line in file:
    ...
    yield key, {...}
```

Since `datasets` 1.7.0 we enforce the keys to be unique.
However it looks like your examples generator creates duplicate keys: at least two examples have key 0.

You can fix that by making sure that your keys are unique.

For example if you use a counter to define the key of each example, make sure that your counter is not reset to 0 in during examples generation (between two open files for examples).

Let me know if you have other questions :)
Issue: Remove `extended` field from dataset tagger

Comment:
The tagger also doesn't insert the value for the `size_categories` field automatically, so this should be fixed too
Issue: Remove `extended` field from dataset tagger

Comment:
Thanks for reporting. Indeed the `extended` tag doesn't exist. Not sure why we had that in the tagger.
The repo of the tagger is here if someone wants to give this a try: https://github.com/huggingface/datasets-tagging
Otherwise I can probably fix it next week
Issue: Extend QuestionAnsweringExtractive template to handle nested columns

Comment:
this is also the case for the following datasets and configurations:

* `mlqa` with config `mlqa-translate-train.ar`
Issue: Extend QuestionAnsweringExtractive template to handle nested columns

Comment:
The current task API is somewhat deprecated (we plan to align it with `train eval index` at some point), so I think we can close this issue.
Issue: DuplicatedKeysError when trying to load adversarial_qa

Comment:
Thanks for reporting !
#2433 fixed the issue, thanks @mariosasko :)

We'll do a patch release soon of the library.
In the meantime, you can use the fixed version of adversarial_qa by adding `script_version="master"` in `load_dataset`
Issue: Saving Graph/Structured Data in Datasets

Comment:
It should probably work out of the box to save structured data. If you want to show an example we can help you.
Issue: Saving Graph/Structured Data in Datasets

Comment:
An example of a toy dataset is like:
```json
[
    {
        "name": "mike",
        "friends": [
            "tom",
            "lily"
        ],
        "articles": [
            {
                "title": "aaaaa",
                "reader": [
                    "tom",
                    "lucy"
                ]
            }
        ]
    },
    {
        "name": "tom",
        "friends": [
            "mike",
            "bbb"
        ],
        "articles": [
            {
                "title": "xxxxx",
                "reader": [
                    "tom",
                    "qqqq"
                ]
            }
        ]
    }
]
```
We can use the friendship relation to build a directional graph, and a user node can be represented using the articles written by himself. And the relationship between articles can be built when the article has read by the same user.
This dataset can be used to model the heterogeneous relationship between users and articles, and this graph can be used to build recommendation systems to recommend articles to the user, or potential friends to the user.
Issue: Saving Graph/Structured Data in Datasets

Comment:
Hi,

you can do the following to load this data into a `Dataset`:
```python
from datasets import Dataset
examples = [
    {
        "name": "mike",
        "friends": [
            "tom",
            "lily"
        ],
        "articles": [
            {
                "title": "aaaaa",
                "reader": [
                    "tom",
                    "lucy"
                ]
            }
        ]
    },
    {
        "name": "tom",
        "friends": [
            "mike",
            "bbb"
        ],
        "articles": [
            {
                "title": "xxxxx",
                "reader": [
                    "tom",
                    "qqqq"
                ]
            }
        ]
    }
]

keys = examples[0].keys()
values = [ex.values() for ex in examples]
dataset = Dataset.from_dict({k: list(v) for k, v in zip(keys, zip(*values))})
```

Let us know if this works for you.
Issue: Saving Graph/Structured Data in Datasets

Comment:
Thank you so much, and that works! I also have a question that if the dataset is very large, that cannot be loaded into the memory. How to create the Dataset?
Issue: Saving Graph/Structured Data in Datasets

Comment:
If your dataset doesn't fit in memory, store it in a local file and load it from there. Check out [this chapter](https://huggingface.co/docs/datasets/master/loading_datasets.html#from-local-files) in the docs for more info.
Issue: load_from_disk and save_to_disk are not compatible with each other

Comment:
Hi,

`load_dataset` returns an instance of `DatasetDict` if `split` is not specified, so instead of `Dataset.load_from_disk`, use `DatasetDict.load_from_disk` to load the dataset from disk.
Issue: load_from_disk and save_to_disk are not compatible with each other

Comment:
Though I see a stream of issues open by people lost between datasets and datasets dicts so maybe there is here something that could be better in terms of UX. Could be better error handling or something else smarter to even avoid said errors but maybe we should think about this. Reopening to use this issue as a discussion place but feel free to open a new open if you prefer @lhoestq @albertvillanova
Issue: load_from_disk and save_to_disk are not compatible with each other

Comment:
We should probably improve the error message indeed.

Also note that there exists a function `load_from_disk` that can load a Dataset or a DatasetDict. Under the hood it calls either `Dataset.load_from_disk` or `DatasetDict.load_from_disk`:


```python
from datasets import load_from_disk

dataset_dict = load_from_disk("path/to/dataset/dict")
single_dataset = load_from_disk("path/to/single/dataset")
```
Issue: Cached dataset not loaded

Comment:
It actually seems to happen all the time in above configuration:
* the function `filter_by_duration` correctly loads cached processed dataset
* the function `prepare_dataset` is always reexecuted

I end up solving the issue by saving to disk my dataset at the end but I'm still wondering if it's a bug or limitation here.
Issue: Cached dataset not loaded

Comment:
Hi ! The hash used for caching `map` results is the fingerprint of the resulting dataset. It is computed using three things:
- the old fingerprint of the dataset
- the hash of the function
- the hash of the other parameters passed to `map`

You can compute the hash of your function (or any python object) with
```python
from datasets.fingerprint import Hasher

my_func = lambda x: x + 1
print(Hasher.hash(my_func))
```

If `prepare_dataset` is always executed, maybe this is because your `processor` has a different hash each time you want to execute it.
Issue: Cached dataset not loaded

Comment:
> If `prepare_dataset` is always executed, maybe this is because your `processor` has a different hash each time you want to execute it.

Yes I think that was the issue.

For the hash of the function:
* does it consider just the name or the actual code of the function
* does it consider variables that are not passed explicitly as parameters to the functions (such as the processor here)
Issue: Cached dataset not loaded

Comment:
> does it consider just the name or the actual code of the function

It looks at the name and the actual code and all variables such as recursively. It uses `dill` to do so, which is based on `pickle`.
Basically the hash is computed using the pickle bytes of your function (computed using `dill` to support most python objects).

> does it consider variables that are not passed explicitly as parameters to the functions (such as the processor here)

Yes it does thanks to recursive pickling.
Issue: AttributeError: 'DatasetInfo' object has no attribute 'task_templates'

Comment:
Hi ! Can you try using a more up-to-date version ? We added the task_templates in `datasets` 1.7.0.

Ideally when you're working on new datasets, you should install and use the local version of your fork of `datasets`. Here I think you tried to run the 1.7.0 tests with the 1.6.2 code
Issue: .map() function got an unexpected keyword argument 'cache_file_name'

Comment:
Hi @cindyxinyiwang,
Did you try adding `.arrow` after `cache_file_name` argument? Here I think they're expecting something like that only for a cache file:
https://github.com/huggingface/datasets/blob/e08362256fb157c0b3038437fc0d7a0bbb50de5c/src/datasets/arrow_dataset.py#L1556-L1558
Issue: .map() function got an unexpected keyword argument 'cache_file_name'

Comment:
Hi ! `cache_file_name` is an argument of the `Dataset.map` method. Can you check that your `dataset` is indeed a `Dataset` object ?

If you loaded several splits, then it would actually be a `DatasetDict` (one dataset per split, in a dictionary).
In this case, since there are several datasets in the dict, the `DatasetDict.map` method requires a `cache_file_names` argument (with an 's'), so that you can provide one file name per split.
Issue: .map() function got an unexpected keyword argument 'cache_file_name'

Comment:
I think you are right. I used cache_file_names={data1: name1, data2: name2} and it works. Thank you!
Issue: Concatenate several datasets with removed columns is not working.

Comment:
Hi,

did you fill out the env info section manually or by copy-pasting the output of the `datasets-cli env` command?

This code should work without issues on 1.6.2 version (I'm working on master (1.6.2.dev0 version) and can't reproduce this error).