helmo/github-issues huggingface dataset:
wmt datasets fail to load
State: closed | Author: stas00 (CONTRIBUTOR)

on master:
```
python -c 'from datasets import load_dataset; load_dataset("wmt14", "de-en")'
Downloading and preparing dataset wmt14/de-en (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/stas/.cache/huggingface/datasets/wmt14/de-en/1.0.0/43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/load.py", line 740, in load_dataset
    builder_instance.download_and_prepare(
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/builder.py", line 578, in download_and_prepare
    self._download_and_prepare(
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/builder.py", line 634, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/stas/.cache/huggingface/modules/datasets_modules/datasets/wmt14/43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e/wmt_utils.py", line 760, in _split_generators
    extraction_map = dict(downloaded_files, **manual_files)
```

it worked fine recently. same problem if I try wmt16.

git bisect points to this commit from Feb 25 as the culprit https://github.com/huggingface/datasets/commit/792f1d9bb1c5361908f73e2ef7f0181b2be409fa

@albertvillanova 

Comments: @stas00 Mea culpa... May I fix this tomorrow morning?

yes, of course, I reverted to the version before that and it works ;)

but since a new release was just made you will probably need to make a hotfix.

and add the wmt to the tests?

Sure, I will implement a regression test!

@stas00 it is fixed. @lhoestq are you releasing the hot fix or would you prefer me to do it?

I'll do a patch release for this issue early tomorrow.

And yes we absolutly need tests for the wmt datasets: The missing tests for wmt are an artifact from the early development of the lib but now we have tools to generate automatically the dummy data used for tests :)

still facing the same issue or similar:
from datasets import load_dataset
wtm14_test = load_dataset('wmt14',"de-en",cache_dir='./datasets')

~.cache\huggingface\modules\datasets_modules\datasets\wmt14\43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e\wmt_utils.py in _split_generators(self, dl_manager)
758 # Extract manually downloaded files.
759 manual_files = dl_manager.extract(manual_paths_dict)
--> 760 extraction_map = dict(downloaded_files, **manual_files)
761
762 for language in self.config.language_pair:

TypeError: type object argument after ** must be a mapping, not list
Loading all answers from drop
State: closed | Author: KaijuML (CONTRIBUTOR)

Hello all,

I propose this change to the DROP loading script so that all answers are loaded no matter their type. Currently, only "span" answers are loaded, which excludes a significant amount of answers from drop (i.e. "number" and "date").

I updated the script with the version I use for my work. However, I couldn't find a way to verify that all is working when integrated with the datasets repo, since the `load_dataset` method seems to always download the script from github and not local files.

Note that 9 items from the train set have no answers, as well as 1 from the validation set. The script I propose simply do not load them.

Let me know if there is anything else I can do,
Clément

Comments: Nice thanks for the change !
This looks all good to me

Before we merge can you just update the dataset_infos.json file of drop ? You can do it by running
```
datasets-cli test ./datasets/drop --all_configs --save_infos --ignore_verifications
```

Done!
Add article_id and process test set template for semeval 2020 task 11…
State: closed | Author: hemildesai (CONTRIBUTOR)

… dataset

- `article_id` is needed to create the submission file for the task at https://propaganda.qcri.org/semeval2020-task11/
- The `technique classification` task provides the span indices in a template for the test set that is necessary to complete the task. This PR implements processing of that template for the dataset.

Comments: Thanks !
Now to fix the CI the only thing left is to add a dummy `test-task-tc-template.out` file inside the `dummy_data.zip` at `./datasets/sem_eval_2020_task_11/dummy/1.1.0`
It must contain the labels template for each dummy article of the test set included in `dummy_data.zip`

After that we should be good to merge this one :)

@lhoestq Made the changes! The failure now seems to be unrelated to the changes. Any idea what's going on?

This is a bug on master that we're investigating. You can ignore it
Adding ro sts dataset
State: closed | Author: lorinczb (CONTRIBUTOR)

Adding [RO-STS](https://github.com/dumitrescustefan/RO-STS) dataset

Comments: @lhoestq thank you very much for the quick review and useful comments! 

I have tried to address them all, and a few comments that you left for ro_sts I have applied to the ro_sts_parallel as well (in read-me: fixed source_datasets, links to homepage, repository, leaderboard, thanks to me message, in ro_sts_parallel.py changed to camel case as well). In the ro_sts_parallel I have changed the order on the languages, also in the example, as you said order doesn't matter, but just to have them listed in the readme in the same order.

I have commented above on why we would like to keep them as separate datasets, hope it makes sense.

If there is anything else I should change please let me know.

Thanks again!

@lhoestq I tried to adjust the ro_sts_parallel, locally when I run the tests they are passing, but somewhere it has the old name of rosts-parallel-ro-en which I am trying to change to ro_sts_parallel. I don't think I have left anything related to rosts-parallel-ro-en, but when the dataset_infos.json is regenerated it adds it. Could you please help me out, how can I fix this? Thanks in advance!

Great, thanks for all your help!
feat(docs): navigate with left/right arrow keys
State: closed | Author: ydcjeff (NONE)

Enables docs navigation with left/right arrow keys. It can be useful for the ones who navigate with keyboard a lot.
More info : https://github.com/sphinx-doc/sphinx/pull/2064

You can try here : https://29353-250213286-gh.circle-artifacts.com/0/docs/_build/html/index.html
Question: what gets stored in the datasets cache and why is it so huge?
State: closed | Author: ioana-blue (NONE)

I'm running several training jobs (around 10) with a relatively large dataset (3M samples). The datasets cache reached 178G and it seems really large. What is it stored in there and why is it so large? I don't think I noticed this problem before and seems to be related to the new version of the datasets library. Any insight? Thank you!

Comments: Echo'ing this observation: I have a few datasets in the neighborhood of 2GB CSVs uncompressed, and when I use something like `Dataset.save_to_disk()` it's ~18GB on disk.

If this is unexpected behavior, would be happy to help run debugging as needed.

Thanks @ioana-blue for pointing out this problem (and thanks also @justin-yan). You are right that current implementation of the datasets caching files take too much memory. We are definitely changing this and optimizing the defaults, so that the file sizes are considerably reduced. I will come back to you as soon as this is fixed.

Thank you! Also I noticed that the files don't seem to be cleaned after the jobs finish. Last night I had only 3 jobs running, but the cache was still at 180GB. 

And to clarify, it's not memory, it's disk space. Thank you!

Hi ! As Albert said they can sometimes take more space that expected but we'll fix that soon.

Also, to give more details about caching: computations on a dataset are cached by default so that you don't have to recompute them the next time you run them.

So by default the cache files stay on your disk when you job is finished (so that if you re-execute it, it will be reloaded from the cache).
Feel free to clear your cache after your job has finished, or disable caching using
```python
import datasets

datasets.set_caching_enabled(False)
```

Thanks for the tip, this is useful. 

Hi @ioana-blue, we have optimized Datasets' disk usage in the latest release v1.5.

Feel free to update your Datasets version
```shell
pip install -U datasets
```
and see if it better suits your needs.

Thank you!
Fix ArrowWriter closes stream at exit
State: closed | Author: albertvillanova (MEMBER)

Current implementation of ArrowWriter does not properly release the `stream` resource (by closing it) if its `finalize()` method is not called and/or an Exception is raised before/during the call to its `finalize()` method.

Therefore, ArrowWriter should be used as a context manager that properly closes its `stream` resource at exit.

Comments: Oh nice thanks for adding the context manager ! All the streams and RecordBatchWriter will be properly closed now. Hopefully this gives a better experience on windows on which it's super important to close stuff.

Not sure about the error, it looks like a process crashed silently.
Let me take a look

> Hopefully this gives a better experience on windows on which it's super important to close stuff.

Exactly! On Windows, you got:
> PermissionError: [WinError 32] The process cannot access the file because it is being used by another process

when trying to access the unclosed `stream` file, e.g. by `with incomplete_dir(self._cache_dir) as tmp_data_dir`: `shutil.rmtree(tmp_dir)`

The reason is: https://docs.python.org/3/library/os.html#os.remove

> On Windows, attempting to remove a file that is in use causes an exception to be raised; on Unix, the directory entry is removed but the storage allocated to the file is not made available until the original file is no longer in use.




The test passes on my windows. This was probably a circleCI issue. I re-ran the circleCI tests

NICE! It passed!

Maybe you can merge master into this branch and check the CI before merging ?

@lhoestq done! ;)

Thanks ! merging
Add Turkish News Category Dataset - 270K - Lite Version
State: closed | Author: yavuzKomecoglu (CONTRIBUTOR)

This PR adds the Turkish News Categories Dataset (270K - Lite Version) dataset which is a text classification dataset by me, @basakbuluz and @serdarakyol.
This dataset contains the same news from the current [interpress_news_category_tr dataset](https://huggingface.co/datasets/interpress_news_category_tr) but contains less information, OCR errors are reduced, can be easily separated, and can be divided into 10 classes ("kültürsanat", "ekonomi", "siyaset", "eğitim", "dünya", "spor", "teknoloji", "magazin", "sağlık", "gündem") were rearranged.

Comments: Thanks for the change, merging now !
Fix metrics collision in separate multiprocessed experiments
State: closed | Author: lhoestq (MEMBER)

As noticed in #1942 , there's a issue with locks if you run multiple separate evaluation experiments in a multiprocessed setup.

Indeed there is a time span in Metric._finalize() where the process 0 loses its lock before re-acquiring it. This is bad since the lock of the process 0 tells the other process that the corresponding cache file is available for writing/reading/deleting: we end up having one metric cache that collides with another one. This can raise FileNotFound errors when a metric tries to read the cache file and if the second conflicting metric deleted it.

To fix that I made sure that the lock file of the process 0 stays acquired from the cache file creation to the end of the metric computation. This way the other metrics can simply sample a new hashing name in order to avoid the collision.

Finally I added missing tests for separate experiments in distributed setup.

Comments: Since the failure was originally intermittent, there is no 100% telling that the problem is gone. 
But if my artificial race condition setup https://github.com/huggingface/datasets/issues/1942#issuecomment-787124529 is to be the litmus test then the problem has been fixed, as with this PR branch that particular race condition is taken care of correctly.

Thank you for taking care of this, @lhoestq  - locking can be very tricky to do right!
Can we parallelized the add_faiss_index process over dataset shards ?
State: closed | Author: shamanez (NONE)

I am thinking of making the  **add_faiss_index** process faster. What if we run the add_faiss_index process on separate dataset shards and then combine them before (dataset.concatenate) saving the faiss.index file ?

I feel theoretically this will reduce the accuracy of retrieval since it affects the indexing process.

@lhoestq


Comments: Hi !
As far as I know not all faiss indexes can be computed in parallel and then merged. 
For example [here](https://github.com/facebookresearch/faiss/wiki/Special-operations-on-indexes#splitting-and-merging-indexes) is is mentioned that only IndexIVF indexes can be merged.
Moreover faiss already works using multithreading to parallelize the workload over your different CPU cores. You can find more info [here](https://github.com/facebookresearch/faiss/wiki/Threads-and-asynchronous-calls#internal-threading)
So I feel like the gains we would get by implementing a parallel `add_faiss_index` would not be that important, but let me know what you think.


Actually, you are right. I also had the same idea. I am trying this in the context of end-ton-end retrieval training in RAG.  So far I have parallelized the embedding re-computation within the training loop by using datasets shards. 

Then I was thinking of can I calculate the indexes for each shard and combined them with **concatenate**  before I save.

@lhoestq  As you mentioned faiss is already using multiprocessing. I tried to do the add_index with faiss for a dataset object inside a RAY actor and the process became very slow... if fact it takes so much time. It is because a ray actor comes with a single CPU core unless we assign it more. I also tried assigning more cores but still running add_index in the main process is very fast.
bug in SNLI dataset 
State: closed | Author: dorost1234 (NONE)

Hi
There is label of -1 in train set of SNLI dataset, please find the code below:

```
import numpy as np 
import datasets 
data = datasets.load_dataset("snli")["train"]
labels = []
for d in data:
   labels.append(d["label"])
print(np.unique(labels))
```

and results:

`[-1  0  1  2]`

version of datasets used:
`datasets                  1.2.1                     <pip>
`

thanks for your help. @lhoestq 

Comments: Hi ! The labels -1 correspond to the examples without gold labels in the original snli dataset.
Feel free to remove these examples if you don't need them by using
```python
data = data.filter(lambda x: x["label"] != -1)
```
Fix unused arguments
State: closed | Author: mariosasko (COLLABORATOR)

Noticed some args in the codebase are not used, so managed to find all such occurrences with Pylance and fix them.

Comments: @lhoestq Re-added the arg. The ConnectionError in CI seems unrelated to this PR (the same test fails on master as well).

Thanks !
I'm re-running the CI, maybe this was an issue with circleCI

Looks all good now, merged :)
Allow stateful function in dataset.map
State: closed | Author: mariosasko (COLLABORATOR)

Removes the "test type" section in Dataset.map which would modify the state of the stateful function. Now, the return type of the map function is inferred after processing the first example.

Fixes #1940 

@lhoestq Not very happy with the usage of `nonlocal`. Would like to hear your opinion on this.

Comments: @lhoestq Added a test. If you can come up with a better stateful callable, I'm all ears 😄.  

Sorry I said earlier that it was good to have it inside the loop, my mistake !

@lhoestq Okay, did some refactoring and now the "cache" part comes before the for loop. Thanks for the guidance.

Think this is ready for the final review.
Bug in skip_rows argument of load_dataset function ?
State: closed | Author: LedaguenelArthur (NONE)

Hello everyone,

I'm quite new to Git so sorry in advance if I'm breaking some ground rules of issues posting... :/
I tried to use the load_dataset function, from Huggingface datasets library, on a csv file using the skip_rows argument described on Huggingface page to skip the first row containing column names

`test_dataset = load_dataset('csv', data_files=['test_wLabel.tsv'], delimiter='\t', column_names=["id", "sentence", "label"], skip_rows=1)`

But I got the following error message

`__init__() got an unexpected keyword argument 'skip_rows'`

Have I used the wrong argument ? Am I missing something or is this a bug ?

Thank you very much for your time,
Best regards,
Arthur

Comments: Hi,

try `skiprows` instead. This part is not properly documented in the docs it seems.

@lhoestq I'll fix this as part of a bigger PR that fixes typos in the docs.
XSum dataset download link broken
State: closed | Author: himat (NONE)

I did 
```
from datasets import load_dataset

dataset = load_dataset("xsum")
```

This returns
`ConnectionError: Couldn't reach http://bollin.inf.ed.ac.uk/public/direct/XSUM-EMNLP18-Summary-Data-Original.tar.gz`

Comments: Never mind, I ran it again and it worked this time. Strange.
[distributed env] potentially unsafe parallel execution
State: closed | Author: stas00 (CONTRIBUTOR)

```
metric = load_metric('glue', 'mrpc', num_process=num_process, process_id=rank)
```

presumes that there is only one set of parallel processes running - and will intermittently fail if you have multiple sets running as they will surely overwrite each other. Similar to https://github.com/huggingface/datasets/issues/1942 (but for a different reason).
That's why dist environments use some unique to a group identifier so that each group is dealt with separately. 

e.g. the env-way of pytorch dist syncing is done with a unique per set `MASTER_ADDRESS+MASTER_PORT`

So ideally this interface should ask for a shared secret to do the right thing.

I'm not reporting an immediate need, but am only flagging that this will hit someone down the road.

This problem can be remedied by adding a new optional `shared_secret` option, which can then be used to differentiate different groups of processes. and this secret should be part of the file lock name and the experiment.

Thank you

Comments: You can pass the same `experiment_id` for all the metrics of the same group, and use another `experiment_id` for the other groups.
Maybe we can add an environment variable that sets the default value for `experiment_id` ? What do you think ?

Ah, you're absolutely correct, @lhoestq - it's exactly the equivalent of the shared secret. Thank you!
add a new column 
State: closed | Author: dorost1234 (NONE)

Hi
I'd need to add a new column to the dataset, I was wondering how this can be done? thanks 
@lhoestq 

Comments: Hi
not sure how change the lable after creation, but this is an issue not dataset request. thanks 

Hi ! Currently you have to use `map` . You can see an example of how to do it in this comment: https://github.com/huggingface/datasets/issues/853#issuecomment-727872188

In the future we'll add support for a more native way of adding a new column ;)
Handle timeouts
State: closed | Author: lhoestq (MEMBER)

As noticed in https://github.com/huggingface/datasets/issues/1939, timeouts were not properly handled when loading a dataset.
This caused the connection to hang indefinitely when working in a firewalled environment cc @stas00 

I added a default timeout, and included an option to our offline environment for tests to be able to simulate both connection errors and timeout errors (previously it was simulating connection errors only).

Now networks calls don't hang indefinitely.
The default timeout is set to 10sec (we might reduce it).

Comments: I never said the calls were hanging indefinitely, what we need is quite different - in the firewalled env with a network, there should be no network calls or they should fail instantly.

To make this work I suppose on top of this PR we need:
1. `DATASETS_OFFLINE` env var to force set timeout to 0 globally (or to 0.0001 if 0 has a special meaning of no timeout)
2. `DATASETS_OFFLINE` should guard against failing network calls and not fail the program if it has all the data it needs locally.

Bottom line - if the logic wants to check online if the local file matches online dataset name, let it go wild, but it should fail instantly, recover and use the local file - if one is specified explicitly or cache if there is one. And only if neither was found only then assert.

I hope this makes sense and is doable.

I have started on the same approach for transformers https://github.com/huggingface/transformers/pull/10407

Thank you, @lhoestq 

Yes that was the first step to add DATASETS_OFFLINE :)

With this PR, if a request times out (which couldn't happen before because no time out was set), it falls back on the local files with no error.

As you said, setting the timeout to something like 1e-16 makes the requests fail instantly, which is one step forward. One last thing left is to disable request retries and everything will be instant !

Ah, fantastic. Thank you for elucidating that this PR is part of a bigger master plan!  

Merging this one, then I'll open a new PR for the `DATASETS_OFFLINE` env var :)
Add cross-platform support for datasets-cli
State: closed | Author: mariosasko (COLLABORATOR)

One thing I've noticed while going through the codebase is the usage of `scripts` in `setup.py`. This [answer](https://stackoverflow.com/a/28119736/14095927) on SO explains it nicely why it's better to use `entry_points` instead of `scripts`. To add cross-platform support to the CLI, this PR replaces `scripts` with `entry_points` in `setup.py` and moves datasets-cli to src/datasets/commands/datasets_cli.py. All *.md and *.rst files are updated accordingly. The same changes were made in the transformers repo to add cross-platform ([link to PR](https://github.com/huggingface/transformers/pull/4131)).

Comments: @mariosasko This is kinda cool!
Enable Fast Filtering using Arrow Dataset
State: open | Author: gchhablani (CONTRIBUTOR)

Hi @lhoestq,

As mentioned in Issue #1796, I would love to work on enabling fast filtering/mapping. Can you please share the expectations? It would be great if you could point me to the relevant methods/files involved. Or the docs or maybe an overview of `arrow_dataset.py`. I only ask this because I am having trouble getting started ;-;

Any help would be appreciated.

Thanks,
Gunjan

Comments: Hi @gchhablani :)
Thanks for proposing your help !

I'll be doing a refactor of some parts related to filtering in the scope of https://github.com/huggingface/datasets/issues/1877
So I would first wait for this refactor to be done before working on the filtering. In particular because I plan to make things simpler to manipulate.

Your feedback on this refactor would also be appreciated since it also aims at making the core code more accessible (basically my goal is that no one's ever "having troubles getting started" ^^)

This will be available in a few days, I will be able to give you more details at that time if you don't mind waiting a bit !

Sure! I don't mind waiting. I'll check the refactor and try to understand what you're trying to do :)
Update documentation with not in place transforms and update DatasetDict
State: closed | Author: lhoestq (MEMBER)

In #1883 were added the not in-place transforms `flatten`, `remove_columns`, `rename_column` and `cast`.

I added them to the documentation and added a paragraph on how to use them

You can preview the documentation [here](https://28862-250213286-gh.circle-artifacts.com/0/docs/_build/html/processing.html#renaming-removing-casting-and-flattening-columns)

I also added these methods to the DatasetDict class.