AniketTec/github-issues huggingface dataset:
Title: wmt datasets fail to load
State: closed

on master:
```
python -c 'from datasets import load_dataset; load_dataset("wmt14", "de-en")'
Downloading and preparing dataset wmt14/de-en (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/stas/.cache/huggingface/datasets/wmt14/de-en/1.0.0/43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/load.py", line 740, in load_dataset
    builder_instance.download_and_prepare(
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/builder.py", line 578, in download_and_prepare
    self._download_and_prepare(
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/builder.py", line 634, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/stas/.cache/huggingface/modules/datasets_modules/datasets/wmt14/43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e/wmt_utils.py", line 760, in _split_generators
    extraction_map = dict(downloaded_files, **manual_files)
```

it worked fine recently. same problem if I try wmt16.

git bisect points to this commit from Feb 25 as the culprit https://github.com/huggingface/datasets/commit/792f1d9bb1c5361908f73e2ef7f0181b2be409fa

@albertvillanova 

Comments:
Title: Loading all answers from drop
State: closed

Hello all,

I propose this change to the DROP loading script so that all answers are loaded no matter their type. Currently, only "span" answers are loaded, which excludes a significant amount of answers from drop (i.e. "number" and "date").

I updated the script with the version I use for my work. However, I couldn't find a way to verify that all is working when integrated with the datasets repo, since the `load_dataset` method seems to always download the script from github and not local files.

Note that 9 items from the train set have no answers, as well as 1 from the validation set. The script I propose simply do not load them.

Let me know if there is anything else I can do,
Clément

Comments:
Title: Add article_id and process test set template for semeval 2020 task 11…
State: closed

… dataset

- `article_id` is needed to create the submission file for the task at https://propaganda.qcri.org/semeval2020-task11/
- The `technique classification` task provides the span indices in a template for the test set that is necessary to complete the task. This PR implements processing of that template for the dataset.

Comments:
Title: ModuleNotFoundError: No module named 'apache_beam' for wikipedia datasets 
State: open

Hi
I am trying to run run_mlm.py code [1] of huggingface with following "wikipedia"/ "20200501.aa"  dataset:

`python run_mlm.py     --model_name_or_path bert-base-multilingual-cased --dataset_name wikipedia     --dataset_config_name 20200501.aa     --do_train     --do_eval     --output_dir /tmp/test-mlm --max_seq_length 256
`

I am getting this error, but as per documentation, huggingface dataset provide processed version of this dataset and users can load it without requiring setup extra settings for apache-beam. could you help me please to load this dataset? 
Do you think I can run run_ml.py with this dataset? or anyway I could subsample and train the model? I greatly appreciate providing the processed version of all languages for this dataset, which allow the user to use them without setting up apache-beam,. thanks 

I really appreciate your help.
@lhoestq 

thanks.

[1] https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py

error I get: 

```
>>> import datasets 
>>> datasets.load_dataset("wikipedia", "20200501.aa")
Downloading and preparing dataset wikipedia/20200501.aa (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /dara/temp/cache_home_2/datasets/wikipedia/20200501.aa/1.0.0/4021357e28509391eab2f8300d9b689e7e8f3a877ebb3d354b01577d497ebc63...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/dara/temp/libs/anaconda3/envs/codes/lib/python3.7/site-packages/datasets-1.3.0-py3.7.egg/datasets/load.py", line 746, in load_dataset
    use_auth_token=use_auth_token,
  File "/dara/temp/libs/anaconda3/envs/codes/lib/python3.7/site-packages/datasets-1.3.0-py3.7.egg/datasets/builder.py", line 573, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/dara/temp/libs/anaconda3/envs/codes/lib/python3.7/site-packages/datasets-1.3.0-py3.7.egg/datasets/builder.py", line 1099, in _download_and_prepare
    import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'

```

Comments:
Title: Question: what gets stored in the datasets cache and why is it so huge?
State: closed

I'm running several training jobs (around 10) with a relatively large dataset (3M samples). The datasets cache reached 178G and it seems really large. What is it stored in there and why is it so large? I don't think I noticed this problem before and seems to be related to the new version of the datasets library. Any insight? Thank you!

Comments:
Title: Add Turkish News Category Dataset - 270K - Lite Version
State: closed

This PR adds the Turkish News Categories Dataset (270K - Lite Version) dataset which is a text classification dataset by me, @basakbuluz and @serdarakyol.
This dataset contains the same news from the current [interpress_news_category_tr dataset](https://huggingface.co/datasets/interpress_news_category_tr) but contains less information, OCR errors are reduced, can be easily separated, and can be divided into 10 classes ("kültürsanat", "ekonomi", "siyaset", "eğitim", "dünya", "spor", "teknoloji", "magazin", "sağlık", "gündem") were rearranged.

Comments:
Title: Fix metrics collision in separate multiprocessed experiments
State: closed

As noticed in #1942 , there's a issue with locks if you run multiple separate evaluation experiments in a multiprocessed setup.

Indeed there is a time span in Metric._finalize() where the process 0 loses its lock before re-acquiring it. This is bad since the lock of the process 0 tells the other process that the corresponding cache file is available for writing/reading/deleting: we end up having one metric cache that collides with another one. This can raise FileNotFound errors when a metric tries to read the cache file and if the second conflicting metric deleted it.

To fix that I made sure that the lock file of the process 0 stays acquired from the cache file creation to the end of the metric computation. This way the other metrics can simply sample a new hashing name in order to avoid the collision.

Finally I added missing tests for separate experiments in distributed setup.

Comments:
Title: Can we parallelized the add_faiss_index process over dataset shards ?
State: closed

I am thinking of making the  **add_faiss_index** process faster. What if we run the add_faiss_index process on separate dataset shards and then combine them before (dataset.concatenate) saving the faiss.index file ?

I feel theoretically this will reduce the accuracy of retrieval since it affects the indexing process.

@lhoestq


Comments:
Title: bug in SNLI dataset 
State: closed

Hi
There is label of -1 in train set of SNLI dataset, please find the code below:

```
import numpy as np 
import datasets 
data = datasets.load_dataset("snli")["train"]
labels = []
for d in data:
   labels.append(d["label"])
print(np.unique(labels))
```

and results:

`[-1  0  1  2]`

version of datasets used:
`datasets                  1.2.1                     <pip>
`

thanks for your help. @lhoestq 

Comments:
Title: Bug in skip_rows argument of load_dataset function ?
State: closed

Hello everyone,

I'm quite new to Git so sorry in advance if I'm breaking some ground rules of issues posting... :/
I tried to use the load_dataset function, from Huggingface datasets library, on a csv file using the skip_rows argument described on Huggingface page to skip the first row containing column names

`test_dataset = load_dataset('csv', data_files=['test_wLabel.tsv'], delimiter='\t', column_names=["id", "sentence", "label"], skip_rows=1)`

But I got the following error message

`__init__() got an unexpected keyword argument 'skip_rows'`

Have I used the wrong argument ? Am I missing something or is this a bug ?

Thank you very much for your time,
Best regards,
Arthur

Comments:
Title: [distributed env] potentially unsafe parallel execution
State: closed

```
metric = load_metric('glue', 'mrpc', num_process=num_process, process_id=rank)
```

presumes that there is only one set of parallel processes running - and will intermittently fail if you have multiple sets running as they will surely overwrite each other. Similar to https://github.com/huggingface/datasets/issues/1942 (but for a different reason).
That's why dist environments use some unique to a group identifier so that each group is dealt with separately. 

e.g. the env-way of pytorch dist syncing is done with a unique per set `MASTER_ADDRESS+MASTER_PORT`

So ideally this interface should ask for a shared secret to do the right thing.

I'm not reporting an immediate need, but am only flagging that this will hit someone down the road.

This problem can be remedied by adding a new optional `shared_secret` option, which can then be used to differentiate different groups of processes. and this secret should be part of the file lock name and the experiment.

Thank you

Comments:
Title: Handle timeouts
State: closed

As noticed in https://github.com/huggingface/datasets/issues/1939, timeouts were not properly handled when loading a dataset.
This caused the connection to hang indefinitely when working in a firewalled environment cc @stas00 

I added a default timeout, and included an option to our offline environment for tests to be able to simulate both connection errors and timeout errors (previously it was simulating connection errors only).

Now networks calls don't hang indefinitely.
The default timeout is set to 10sec (we might reduce it).

Comments:
Title: Add cross-platform support for datasets-cli
State: closed

One thing I've noticed while going through the codebase is the usage of `scripts` in `setup.py`. This [answer](https://stackoverflow.com/a/28119736/14095927) on SO explains it nicely why it's better to use `entry_points` instead of `scripts`. To add cross-platform support to the CLI, this PR replaces `scripts` with `entry_points` in `setup.py` and moves datasets-cli to src/datasets/commands/datasets_cli.py. All *.md and *.rst files are updated accordingly. The same changes were made in the transformers repo to add cross-platform ([link to PR](https://github.com/huggingface/transformers/pull/4131)).

Comments:
Title: Enable Fast Filtering using Arrow Dataset
State: open

Hi @lhoestq,

As mentioned in Issue #1796, I would love to work on enabling fast filtering/mapping. Can you please share the expectations? It would be great if you could point me to the relevant methods/files involved. Or the docs or maybe an overview of `arrow_dataset.py`. I only ask this because I am having trouble getting started ;-;

Any help would be appreciated.

Thanks,
Gunjan

Comments:
Title: dataset loading logger level
State: closed

on master I get this with `--dataset_name wmt16 --dataset_config ro-en`:

```
WARNING:datasets.arrow_dataset:Loading cached processed dataset at /home/stas/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/9dc00622c30446e99c4c63d12a484ea4fb653f2f37c867d6edcec839d7eae50f/cache-2e01bead8cf42e26.arrow
WARNING:datasets.arrow_dataset:Loading cached processed dataset at /home/stas/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/9dc00622c30446e99c4c63d12a484ea4fb653f2f37c867d6edcec839d7eae50f/cache-ac3bebaf4f91f776.arrow
WARNING:datasets.arrow_dataset:Loading cached processed dataset at /home/stas/.cache/huggingface/datasets/wmt16/ro-en/1.0.0/9dc00622c30446e99c4c63d12a484ea4fb653f2f37c867d6edcec839d7eae50f/cache-810c3e61259d73a9.arrow
```

why are those WARNINGs? Should be INFO, no?

warnings should only be used when a user needs to pay attention to something, this is just informative - I'd even say it should be DEBUG, but definitely not WARNING.

Thank you.


Comments:
Title: Update documentation with not in place transforms and update DatasetDict
State: closed

In #1883 were added the not in-place transforms `flatten`, `remove_columns`, `rename_column` and `cast`.

I added them to the documentation and added a paragraph on how to use them

You can preview the documentation [here](https://28862-250213286-gh.circle-artifacts.com/0/docs/_build/html/processing.html#renaming-removing-casting-and-flattening-columns)

I also added these methods to the DatasetDict class.

Comments:
Title: AttributeError: 'DatasetDict' object has no attribute 'concatenate_datasets'
State: closed

Hi
I am trying to concatenate a list of huggingface datastes as:

` train_dataset = datasets.concatenate_datasets(train_datasets)
`
Here is the `train_datasets` when I print:

```
[Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 120361
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 2670
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 6944
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 38140
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 173711
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 1655
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 4274
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 2019
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 2109
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 11963
})]
```

I am getting the following error:

`AttributeError: 'DatasetDict' object has no attribute 'concatenate_datasets'
`

I was wondering if you could help me with this issue, thanks a lot 

Comments:
Title: Add Turkish News Category Dataset (270K - Lite Version)
State: closed

This PR adds the Turkish News Categories Dataset (270K - Lite Version) dataset which is a text classification dataset by me,  @basakbuluz and @serdarakyol. 
This dataset contains the same news from the current [interpress_news_category_tr dataset](https://huggingface.co/datasets/interpress_news_category_tr) but contains less information, OCR errors are reduced, can be easily separated, and can be divided into 10 classes ("kültürsanat", "ekonomi", "siyaset", "eğitim", "dünya", "spor", "teknoloji", "magazin", "sağlık", "gündem") were rearranged.

@SBrandeis @lhoestq, can you please review this PR?


Comments:
Title: [experiment] missing default_experiment-1-0.arrow
State: closed

the original report was pretty bad and incomplete - my apologies!

Please see the complete version here: https://github.com/huggingface/datasets/issues/1942#issuecomment-786336481

------------

As mentioned here https://github.com/huggingface/datasets/issues/1939 metrics don't get cached, looking at my local `~/.cache/huggingface/metrics` - there are many `*.arrow.lock` files but zero metrics files.

w/o the network I get:
```
FileNotFoundError: [Errno 2] No such file or directory: '~/.cache/huggingface/metrics/sacrebleu/default/default_experiment-1-0.arrow
```
there is just `~/.cache/huggingface/metrics/sacrebleu/default/default_experiment-1-0.arrow.lock`

I did run the same `run_seq2seq.py` script on the instance with network and it worked just fine, but only the lock file was left behind.

this is with master.

Thank you.

Comments:
Title: Side effect when filtering data due to `does_function_return_dict` call in `Dataset.map()`
State: closed
Labels: description: New feature or request
name: enhancement

Hi there!

In my codebase I have a function to filter rows in a dataset, selecting only a certain number of examples per class. The function passes a extra argument to maintain a counter of the number of dataset rows/examples already selected per each class, which are the ones I want to keep in the end:

```python
      def fill_train_examples_per_class(example, per_class_limit: int, counter: collections.Counter):
          label = int(example['label'])
          current_counter = counter.get(label, 0)
          if current_counter < per_class_limit:
              counter[label] = current_counter + 1
              return True
          return False
```

At some point I invoke it through the `Dataset.filter()` method in the `arrow_dataset.py` module like this:

```python
...
kwargs = {"per_class_limit": train_examples_per_class_limit, "counter": Counter()}
datasets['train'] = datasets['train'].filter(fill_train_examples_per_class,  num_proc=1, fn_kwargs=kwargs)
...
```

The problem is that, passing a stateful container (the counter,) provokes a side effect in the new filtered dataset obtained. This is due to the fact that at some point in `filter()`, the `map()`'s function `does_function_return_dict` is invoked in  line [1290](https://github.com/huggingface/datasets/blob/96578adface7e4bc1f3e8bafbac920d72ca1ca60/src/datasets/arrow_dataset.py#L1290). 

When this occurs, the state of the counter is initially modified by the effects of the function call on the 1 or 2 rows selected in lines 1288 and 1289 of the same file (which are marked as `test_inputs` & `test_indices` respectively in lines 1288 and 1289. This happens out of the control of the user (which for example can't reset the state of the counter before continuing the execution,) provoking in the end an undesired side effect in the results obtained. 

In my case, the resulting dataset -despite of the counter results are ok- lacks an instance of the classes 0 and 1 (which happen to be the classes of the first two examples of my dataset.) The rest of the classes I have in my dataset, contain the right number of examples as they were not affected by the effects of `does_function_return_dict` call.

I've debugged my code extensively and made a workaround myself hardcoding the necessary stuff (basically putting `update_data=True` in line 1290,) and then I obtain the results I expected without the side effect.

Is there a way to avoid that call to `does_function_return_dict` in map()'s line 1290 ? (e.g. extracting the required information that `does_function_return_dict` returns without making the testing calls to the user function on dataset rows 0 & 1) 

Thanks in advance,

Francisco Perez-Sorrosal



Comments:
Title: [firewalled env] OFFLINE mode
State: closed

This issue comes from a need to be able to run `datasets` in a firewalled env, which currently makes the software hang until it times out, as it's unable to complete the network calls.

I propose the following approach to solving this problem, using the example of `run_seq2seq.py` as a sample program. There are 2 possible ways to going about it.

## 1. Manual

manually prepare data and metrics files, that is transfer to the firewalled instance the dataset and the metrics and run:

```
DATASETS_OFFLINE=1 run_seq2seq.py  --train_file xyz.csv --validation_file xyz.csv ...
```

`datasets` must not make any network calls and if there is a logic to do that and something is missing it should assert that this or that action requires network and therefore it can't proceed.

## 2. Automatic

In some clouds one can prepare a datastorage ahead of time with a normal networked environment but which doesn't have gpus and then one switches to the gpu instance which is firewalled, but it can access all the cached data. This is the ideal situation, since in this scenario we don't have to do anything manually, but simply run the same application twice:

1. on the non-firewalled instance:
```
run_seq2seq.py  --dataset_name wmt16 --dataset_config ro-en ...
```

which should download and cached everything.

2. and then immediately after on the firewalled instance, which shares the same filesystem
```
DATASETS_OFFLINE=1 run_seq2seq.py  --dataset_name wmt16 --dataset_config ro-en ...
```

and the metrics and datasets should be cached by the invocation number 1 and any network calls be skipped and if the logic is missing data it should assert and not try to fetch any data from online.

## Common Issues

1. for example currently `datasets` tries to look up online datasets if the files contain json or csv, despite the paths already provided

```
     if dataset and path in _PACKAGED_DATASETS_MODULES:
```

2. it has an issue with metrics. e.g. I had to manually copy `rouge/rouge.py` from the `datasets` repo to the current dir - or it was hanging.

I had to comment out `head_hf_s3(...)` calls to make things work. So all those `try: head_hf_s3(...)` shouldn't be tried with `DATASETS_OFFLINE=1`

Here is the corresponding issue for `transformers`: https://github.com/huggingface/transformers/issues/10379

Thanks.

Comments:
Title: CommonGen dataset page shows an error OSError: [Errno 28] No space left on device
State: closed
Labels: name: nlp-viewer

The page of the CommonGen data https://huggingface.co/datasets/viewer/?dataset=common_gen  shows 
![image](https://user-images.githubusercontent.com/10104354/108959311-1865e600-7629-11eb-868c-cf4cb27034ea.png)


Comments:
Title: [WIP] Adding Support for Reading Pandas Category
State: closed

@lhoestq - continuing our conversation from https://github.com/huggingface/datasets/issues/1906#issuecomment-784247014

The goal of this PR is to support `Dataset.from_pandas(df)` where the dataframe contains a Category.

Just the 4 line change below actually does seem to work:

```
>>> from datasets import Dataset
>>> import pandas as pd
>>> df = pd.DataFrame(pd.Series(["a", "b", "c", "a"], dtype="category"))
>>> ds = Dataset.from_pandas(df)
>>> ds.to_pandas()
   0
0  a
1  b
2  c
3  a
>>> ds.to_pandas().dtypes
0    category
dtype: object
```

save_to_disk, etc. all seem to work as well.  The main things that are theoretically "incorrect" if we leave this are:

```
>>> ds.features.type
StructType(struct<0: int64>)
```
there are a decent number of references to this property in the library, but I can't find anything that seems to actually break as a result of this being int64 vs. dictionary?  I think the gist of my question is: a) do we *need* to change the dtype of Classlabel and have get_nested_type return a pyarrow.DictionaryType instead of int64? and b) do you *want* it to change?  The biggest challenge I see to implementing this correctly is that the data will need to be passed in along with the pyarrow schema when instantiating the Classlabel (I *think* this is unavoidable, since the type itself doesn't contain the actual label values) which could be a fairly intrusive change - e.g. `from_arrow_schema`'s interface would need to change to include optional arrow data?  Once we start going down this path of modifying the public interfaces I am admittedly feeling a little bit outside of my comfort zone

Additionally I think `int2str`, `str2int`, and `encode_example` probably won't work - but I can't find any usages of them in the library itself.

Comments: