Navigation version breaking

Hi,

when navigating docs (Chrome, Ubuntu) (e.g. on this page: https://huggingface.co/docs/datasets/loading_metrics.html#using-a-custom-metric-script) the version control dropdown has the wrong string displayed as the current version:

![image](https://user-images.githubusercontent.com/3007947/102632187-02cad080-414f-11eb-813b-28f3c8d80def.png)

**Edit:** this actually happens _only_ if you open a link to a concrete subsection.

IMO, the best way to fix this without getting too deep into the intricacies of retrieving version numbers from the URL would be to change [this](https://github.com/huggingface/datasets/blob/master/docs/source/_static/js/custom.js#L112) line to:
```
let label = (version in versionMapping) ? version : stableVersion
```
which delegates the check to the (already maintained) keys of the version mapping dictionary & should be more robust. There's a similar ternary expression [here](https://github.com/huggingface/datasets/blob/master/docs/source/_static/js/custom.js#L97) which should also fail in this case.

I'd also suggest swapping this [block](https://github.com/huggingface/datasets/blob/master/docs/source/_static/js/custom.js#L80-L90) to `string.contains(version) for version in versionMapping` which might be more robust. I'd add a PR myself but I'm by no means competent in JS :)

I also have a side question wrt. docs versioning: I'm trying to make docs for a project which are versioned alike to your dropdown versioning. I was wondering how do you handle storage of multiple doc versions on your server? Do you update what `https://huggingface.co/docs/datasets` points to for every stable release & manually create new folders for each released version?
So far I'm building & publishing (scping) the docs to the server with a github action which works well for a single version, but would ideally need to reorder the public files triggered on a new release.


Add tests for the download functions ?

enhancement

New feature or request

AFAIK the download functions in `DownloadManager` are not tested yet. It could be good to add some to ensure behavior is as expected.


Add retries to HTTP requests

New feature or request

## What does this PR do ?

Adding retries to HTTP GET & HEAD requests, when they fail with a `ConnectTimeout` exception.

The "canonical" way to do this is to use [urllib's Retry class](https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.Retry) and wrap it in a [HttpAdapter](https://requests.readthedocs.io/en/master/api/#requests.adapters.HTTPAdapter). Seems a bit overkill to me, plus it forces us to use the `requests.Session` object. I prefer this simpler implementation. I'm open to remarks and suggestions @lhoestq @yjernite

Fixes #1102


second update of id_newspapers_2018

The feature "url" is currently set wrongly to data["date"], this PR fix it to data["url"].
I added also an additional POC.


second update of the id_newspapers_2018

I close this PR, since it based on 1 week old repo. And I will create a new one


question

Further information is requested

Hi @david-waterworth!

As indicated in the error message, `load_dataset("csv")` returns a `DatasetDict` object, which is mapping of `str` to `Dataset` objects. I believe in this case the behavior is to return a `train` split with all the data.
`train_test_split` is a method of the `Dataset` object, so you will need to do something like this:
```python
dataset_dict = load_dataset(`'csv', data_files='data.txt')
dataset = dataset_dict['split name, eg train']
dataset.train_test_split(test_size=0.1)
```

Please let me know if this helps. 🙂

Thanks, that's working - the same issue also tripped me up with training.

I also agree https://github.com/huggingface/datasets/issues/767 would be a useful addition.

> ```python
> dataset_dict = load_dataset(`'csv', data_files='data.txt')
> dataset = dataset_dict['split name, eg train']
> dataset.train_test_split(test_size=0.1)
> ```

I am getting error like
KeyError: 'split name, eg train'
Could you please tell me how to solve this?

dataset = load_dataset('csv', data_files=['files/datasets/dataset.csv'])
dataset = dataset['train']
dataset = dataset.train_test_split(test_size=0.1)

The following code fails with "'DatasetDict' object has no attribute 'train_test_split'" - am I doing something wrong?
```
from datasets import load_dataset
dataset = load_dataset('csv', data_files='data.txt')
dataset = dataset.train_test_split(test_size=0.1)
```

> AttributeError: 'DatasetDict' object has no attribute 'train_test_split'


add Korean Sarcasm Dataset


made suggested changes in fake-news-english


adding hate-speech-and-offensive-language

made suggested changes and opened PR https://github.com/huggingface/datasets/pull/1628


made suggested changes to hate-speech-and-offensive-language


Logiqa en

I'm getting an error when I try to create the dummy data:
```python
aclifton@pop-os:~/data/hf_datasets_sprint/datasets$ python datasets-cli dummy_data ./datasets/logiqa_en/ --auto_generate
2021-01-07 10:50:12.024791: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2021-01-07 10:50:12.024814: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Using custom data configuration default
Couldn't generate dummy file 'datasets/dummy/1.1.0/dummy_data/master.zip/LogiQA-dataset-master/README.md'. Ignore that if this file is not useful for dummy data.
Dummy data generation done but dummy data test failed since splits ['train', 'test', 'validation'] have 0 examples for config 'default''.
Automatic dummy data generation failed for some configs of './datasets/logiqa_en/'
```

Hi ! Sorry for the delay

To fix your issue for the dummy data you must increase the number of lines that will be kept to generate the dummy files. By default it's 5, and as you need at least 8 lines here to yield one example you must increase this.

You can increase the number of lines to 32 for example by doing
```
datasets-cli dummy_data ./datasets/logica_en --auto_generate --n_lines 32
```

Also it looks like there are changes about other datasets in this PR (imppres). Can you fix that ? You may need to create another branch and another PR.

To fix the branch issue, I went ahead and made a backup of the dataset then deleted my local copy of my fork of `datasets`. I then followed the [detailed guide](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md) from the beginning to reclone the fork and start a new branch.

However, when it came time to create the dummy data I got the following error:
```python
aclifton@pop-os:~/data/hf_datasets_sprint/datasets$ datasets-cli dummy_data ./datasets/logiqa_en --auto_generate --n_lines 32
2021-02-03 11:23:23.145885: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2021-02-03 11:23:23.145914: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Using custom data configuration default
Couldn't generate dummy file 'datasets/logiqa_en/dummy/1.1.0/dummy_data/master.zip/LogiQA-dataset-master/README.md'. Ignore that if this file is not useful for dummy data.
Traceback (most recent call last):
  File "/home/aclifton/anaconda3/bin/datasets-cli", line 36, in <module>
    service.run()
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/commands/dummy_data.py", line 317, in run
    keep_uncompressed=self._keep_uncompressed,
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/commands/dummy_data.py", line 355, in _autogenerate_dummy_data
    dataset_builder._prepare_split(split_generator)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/builder.py", line 905, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/features.py", line 799, in encode_example
    return encode_nested_example(self, example)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/features.py", line 710, in encode_nested_example
    (k, encode_nested_example(sub_schema, sub_obj)) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/features.py", line 710, in <genexpr>
    (k, encode_nested_example(sub_schema, sub_obj)) for k, (sub_schema, sub_obj) in utils.zip_dict(schema, obj)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/features.py", line 737, in encode_nested_example
    return schema.encode_example(obj)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/features.py", line 522, in encode_example
    example_data = self.str2int(example_data)
  File "/home/aclifton/anaconda3/lib/python3.7/site-packages/datasets/features.py", line 481, in str2int
    output.append(self._str2int[str(value)])
KeyError: "Some Cantonese don't like chili, so some southerners don't like chili."
```

Hi ! The error happens when the script is verifying that the generated dummy data work fine with the dataset script.
Apparently it fails because the text `"Some Cantonese don't like chili, so some southerners don't like chili."` was given in a field that is a ClassLabel feature (probably the `answer` field), while it actually expects "a", "b", "c" or "d". Can you fix the script so that it returns the expected labels for this field instead of the text ?

Also it would be awesome to rename this field `answerKey` instead of `answer` to have the same column names as the other multiple-choice-QA datasets in the library :)

Ok getting closer! I got the dummy data to work. However I am now getting the following error:
```python
aclifton@pop-os:~/data/hf_datasets_sprint/datasets$ RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_logiqa_en
===================================================================== test session starts ======================================================================
platform linux -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1
rootdir: /home/aclifton/data/hf_datasets_sprint/datasets
plugins: astropy-header-0.1.2, xdist-2.1.0, doctestplus-0.5.0, forked-1.3.0, hypothesis-5.5.4, arraydiff-0.3, remotedata-0.3.2, openfiles-0.4.0
collected 0 items / 1 error

============================================================================ ERRORS ============================================================================
________________________________________________________ ERROR collecting tests/test_dataset_common.py _________________________________________________________
ImportError while importing test module '/home/aclifton/data/hf_datasets_sprint/datasets/tests/test_dataset_common.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_dataset_common.py:42: in <module>
    from datasets.packaged_modules import _PACKAGED_DATASETS_MODULES
E   ModuleNotFoundError: No module named 'datasets.packaged_modules'
----------------------------------------------------------------------- Captured stderr ------------------------------------------------------------------------
2021-02-10 11:06:14.345510: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2021-02-10 11:06:14.345551: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
======================================================================= warnings summary =======================================================================
/home/aclifton/anaconda3/lib/python3.7/site-packages/tensorflow/python/autograph/utils/testing.py:21
  /home/aclifton/anaconda3/lib/python3.7/site-packages/tensorflow/python/autograph/utils/testing.py:21: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

/home/aclifton/anaconda3/lib/python3.7/site-packages/apache_beam/typehints/typehints.py:693
  /home/aclifton/anaconda3/lib/python3.7/site-packages/apache_beam/typehints/typehints.py:693: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    if not isinstance(type_params, collections.Iterable):

/home/aclifton/anaconda3/lib/python3.7/site-packages/apache_beam/typehints/typehints.py:532
  /home/aclifton/anaconda3/lib/python3.7/site-packages/apache_beam/typehints/typehints.py:532: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    if not isinstance(type_params, (collections.Sequence, set)):

/home/aclifton/anaconda3/lib/python3.7/site-packages/elasticsearch/compat.py:38
  /home/aclifton/anaconda3/lib/python3.7/site-packages/elasticsearch/compat.py:38: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import Mapping

-- Docs: https://docs.pytest.org/en/latest/warnings.html
================================================================= 4 warnings, 1 error in 2.74s =================================================================
ERROR: not found: /home/aclifton/data/hf_datasets_sprint/datasets/tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_logiqa_en
(no name '/home/aclifton/data/hf_datasets_sprint/datasets/tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_logiqa_en' in any of [<Module test_dataset_common.py>])

```

Hi ! It looks like the version of `datasets` that is installed in your environment doesn't match the version of `datasets` you're using for the tests. Can you try uninstalling datasets and reinstall it again ?
```
pip uninstall datasets -y
pip install -e .
```

Closer still!
```python
aclifton@pop-os:~/data/hf_datasets_sprint/datasets$ git commit
[logiqa_en 2664fe7f] fixed several issues with logiqa_en.
 4 files changed, 324 insertions(+)
 create mode 100644 datasets/logiqa_en/README.md
 create mode 100644 datasets/logiqa_en/dataset_infos.json
 create mode 100644 datasets/logiqa_en/dummy/1.1.0/dummy_data.zip
 create mode 100644 datasets/logiqa_en/logiqa_en.py
aclifton@pop-os:~/data/hf_datasets_sprint/datasets$ git fetch upstream
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (1/1), 590 bytes | 590.00 KiB/s, done.
From https://github.com/huggingface/datasets
   6e114a0c..318b09eb  master     -> upstream/master
aclifton@pop-os:~/data/hf_datasets_sprint/datasets$ git rebase upstream/master
error: cannot rebase: You have unstaged changes.
error: Please commit or stash them.
aclifton@pop-os:~/data/hf_datasets_sprint/datasets$ git push -u origin logiqa_en
Username for 'https://github.com': aclifton314
Password for 'https://aclifton314@github.com':
To https://github.com/aclifton314/datasets
 ! [rejected]          logiqa_en -> logiqa_en (non-fast-forward)
error: failed to push some refs to 'https://github.com/aclifton314/datasets'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
```

logiqa in english.


connection error

This happen quite often when they are too many concurrent requests to github.

i can understand it’s a bit cumbersome to handle on the user side. Maybe we should try a few times in the lib (eg with timeout) before failing, what do you think @lhoestq ?

Yes currently there's no retry afaik. We should add retries

Retries were added in #1603 :)
It will be available in the next release

Hi @lhoestq thank you for the modification, I will use`script_version="master"` for now  :), to my experience, also setting timeout to a larger number like 3*60 which I normally use helps a lot on this.

Hi
I am hitting to this error, thanks

```
> Traceback (most recent call last):
  File "finetune_t5_trainer.py", line 379, in <module>
    main()
  File "finetune_t5_trainer.py", line 208, in main
    if training_args.do_eval or training_args.evaluation_strategy != EvaluationStrategy.NO
  File "finetune_t5_trainer.py", line 207, in <dictcomp>
    for task in data_args.eval_tasks}
  File "/workdir/seq2seq/data/tasks.py", line 70, in get_dataset
    dataset = self.load_dataset(split=split)
  File "/workdir/seq2seq/data/tasks.py", line 66, in load_dataset
    return datasets.load_dataset(self.task.name, split=split, script_version="master")
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 589, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 267, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py", line 487, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/master/datasets/boolq/boolq.py
el/0 I1217 01:11:33.898849 354161 main shadow.py:210 Current job status: FINISHED
```


Access to key in DatasetDict map

enhancement

New feature or request

Indeed that would be cool

Also FYI right now the easiest way to do this is
```python
dataset_dict["train"] = dataset_dict["train"].map(my_transform_for_the_train_set)
dataset_dict["test"] = dataset_dict["test"].map(my_transform_for_the_test_set)
```

It is possible that we want to do different things in the `map` function (and possibly other functions too) of a `DatasetDict`, depending on the key. I understand that `DatasetDict.map` is a really thin wrapper of `Dataset.map`, so it is easy to directly implement this functionality in the client code. Still, it'd be nice if there can be a flag, similar to `with_indices`, that allows the callable to know the key inside `DatasetDict`.


Using datasets.Metric with Trainer()

dataset request

Requesting to add a new dataset

We are indeed working on the integration with `Trainer` :)

## Using datasets.Metric with Trainer()

Hi team, I was quite surprised in the [Metric documentation](https://huggingface.co/docs/datasets/using_metrics.html) I don't see how it can be used with `Trainer()`. That would be the most intuitive use case instead of having to iterate the batches and add predictions and references to the metric, then compute the metric manually. Ideally, any pre-built metrics can be added to `compute_metrics` argument of `Trainer()` and they will be calculated at an interval specified by `TrainingArguments.evaluation_strategy`.

Is this option available but just not mentioned in the documentation or it's not possible at the moment? I notice in the [Transformer | Training and fine-tuning](https://huggingface.co/transformers/training.html) tutorial, you are using custom scripts to calculate the accuracy, P/R/F, which are already in the pre-built metrics.


IWSLT-17 Link Broken

duplicate

This issue or pull request already exists

dataset bug

A bug in a dataset script provided in the library

Sorry, this is a duplicate of #1287. Not sure why it didn't come up when I searched `iwslt` in the issues list.

```
FileNotFoundError: Couldn't find file at https://wit3.fbk.eu/archive/2017-01-trnmted//texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.tgz
```


Add helper to resolve namespace collision

I was thinking about using something like [importlib](https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly) to over-ride the collision.

**Reason requested**: I use the [following template](https://github.com/jramapuram/ml_base/) repo where I house all my datasets as a submodule.

Alternatively huggingface could consider some submodule type structure like:

`import huggingface.datasets`
`import huggingface.transformers`

`datasets` is a very common module in ML and should be an end-user decision and not scope all of python ¯\_(ツ)_/¯

That's a interesting option indeed. We'll think about it.

It also wasn't initially obvious to me that the samples which contain `import datasets` were in fact importing a huggingface library (in fact all the huggingface imports are very generic - transformers, tokenizers, datasets...)

Many projects use a module called `datasets`, however this is incompatible with huggingface datasets. It would be great if there if there was some helper or similar function to resolve such a common conflict.


Update doc2dial.py

Thanks for adding the `doc2dial_rc` config :)

It looks like you're missing the dummy data for this config though. Could you add them please ?
Also to fix the CI you'll need to format the code with `make style`

Added data loader for machine reading comprehension tasks proposed in the Doc2Dial EMNLP 2020 paper.


Modified hind encorp

description added, unnecessary comments removed from .py and readme.md reformated
@lhoestq  for #1584


Add nq_open question answering dataset

this is pr is a copy of #1506 due to messed up git history in that pr.


added irc disentangle dataset

@lhoestq sorry, this was the only way I was able to fix the pull request

@lhoestq Thank you for the feedback. I wondering whether I should be passing an 'id' field in the dictionary since the 'connections' reference the 'id' of the linked messages. This 'id' would just be the same as the id_ that is in the yielded tuple.

Yes indeed it would be cool to have the ids in the dictionary. This way the dataset can be shuffled and all without losing information about the connections. Can you add it if you don't mind ?

Thanks :) could you also add the ids in the dictionary since they're useful for the connection links ?

Thanks !
Also it looks like the dummy_data.zip were regenerated and are now back to being too big (300KB each).
Can you reduce their sizes ? You can actually just revert to the ones you had before the last commit

added irc disentanglement dataset


FileNotFoundError for `amazon_polarity`

Hi @phtephanx , the `amazon_polarity` dataset has not been released yet. It will be available in the coming soon v2of `datasets` :)

You can still access it now if you want, but you will need to install datasets via the master branch:
`pip install git+https://github.com/huggingface/datasets.git@master`

Version: `datasets==v1.1.3`

### Reproduction
```python
from datasets import load_dataset
data = load_dataset("amazon_polarity")
```
crashes with
```bash
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/amazon_polarity/amazon_polarity.py
```
and
```bash
FileNotFoundError: Couldn't find file at https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/amazon_polarity/amazon_polarity.py
```
and
```bash
FileNotFoundError: Couldn't find file locally at amazon_polarity/amazon_polarity.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/amazon_polarity/amazon_polarity.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/amazon_polarity/amazon_polarity.py
```


Load hind encorp

reformated well documented, yaml tags added, code


Update metrics docstrings.

#1478 Correcting the argument descriptions for metrics.

Let me know if there's any issues.


Adding wiki lingua dataset as new branch

Adding the dataset as new branch as advised here: #1470


Installing datasets and transformers in a tensorflow docker image throws Permission Error on 'import transformers'

Thanks for reporting !
You can override the directory in which cache file are stored using for example
```
ENV HF_HOME="/root/cache/hf_cache_home"
```

This way both `transformers` and `datasets` will use this directory instead of the default `.cache`

Great, thanks. I didn't see documentation about than ENV variable, looks like an obvious solution.

> Thanks for reporting !
> You can override the directory in which cache file are stored using for example
>
> ```
> ENV HF_HOME="/root/cache/hf_cache_home"
> ```
>
> This way both `transformers` and `datasets` will use this directory instead of the default `.cache`

can we disable caching directly?

Hi ! Unfortunately no since we need this directory to load datasets.
When you load a dataset, it downloads the raw data files in the cache directory inside <cache_dir>/downloads. Then it builds the dataset and saves it as arrow data inside <cache_dir>/<dataset_name>.

However you can specify the directory of your choice, and it can be a temporary directory if you want to clean everything up at one point.

I'm closing this to keep issues a bit cleaner

I am using a docker container, based on latest tensorflow-gpu image, to run transformers and datasets (4.0.1 and 1.1.3 respectively - Dockerfile attached below). Importing transformers throws a Permission Error to access `/.cache`:

```
$ docker run --gpus=all --rm -it -u $(id -u):$(id -g) -v $(pwd)/data:/root/data -v $(pwd):/root -v $(pwd)/models/:/root/models -v $(pwd)/saved_models/:/root/saved_models -e "HOST_HOSTNAME=$(hostname)" hf-error:latest /bin/bash

________                               _______________
___  __/__________________________________  ____/__  /________      __
__  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
_  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ /
/_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/

You are running this container as user with ID 1000 and group 1000,
which should map to the ID and group for your user on the Docker host. Great!

tf-docker /root > python
Python 3.6.9 (default, Oct  8 2020, 12:12:24)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import transformers
2020-12-15 23:53:21.165827: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/transformers/__init__.py", line 22, in <module>
    from .integrations import (  # isort:skip
  File "/usr/local/lib/python3.6/dist-packages/transformers/integrations.py", line 5, in <module>
    from .trainer_utils import EvaluationStrategy
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer_utils.py", line 25, in <module>
    from .file_utils import is_tf_available, is_torch_available, is_torch_tpu_available
  File "/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py", line 88, in <module>
    import datasets  # noqa: F401
  File "/usr/local/lib/python3.6/dist-packages/datasets/__init__.py", line 26, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py", line 40, in <module>
    from .arrow_reader import ArrowReader
  File "/usr/local/lib/python3.6/dist-packages/datasets/arrow_reader.py", line 31, in <module>
    from .utils import cached_path, logging
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/__init__.py", line 20, in <module>
    from .download_manager import DownloadManager, GenerateMode
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/download_manager.py", line 25, in <module>
    from .file_utils import HF_DATASETS_CACHE, cached_path, get_from_cache, hash_url_to_filename
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py", line 118, in <module>
    os.makedirs(HF_MODULES_CACHE, exist_ok=True)
  File "/usr/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/.cache'
```
I've pinned the problem to `RUN pip install datasets`, and by commenting it you can actually import transformers correctly. Another workaround I've found is creating the directory and giving permissions to it directly on the Dockerfile.

```
FROM tensorflow/tensorflow:latest-gpu-jupyter
WORKDIR /root

EXPOSE 80
EXPOSE 8888
EXPOSE 6006

ENV SHELL /bin/bash
ENV PATH="/root/.local/bin:${PATH}"

ENV CUDA_CACHE_PATH="/root/cache/cuda"
ENV CUDA_CACHE_MAXSIZE="4294967296"

ENV TFHUB_CACHE_DIR="/root/cache/tfhub"

RUN pip install --upgrade pip

RUN apt update -y && apt upgrade -y

RUN pip install transformers

#Installing datasets will throw the error, try commenting and rebuilding
RUN pip install datasets

#Another workaround is creating the directory and give permissions explicitly
#RUN mkdir /.cache
#RUN chmod 777 /.cache
```


made suggested changes in diplomacy_detection.py


Adding CLIMATE-FEVER dataset

I `git rebase`ed my branch to `upstream/master` as suggested in point 7 of <https://huggingface.co/docs/datasets/share_dataset.html> and subsequently used `git pull` to be able to push to my remote branch. However, I think this messed up the history.

Please let me know if I should create a clean new PR with my changes.

Update: I also fixed the dataset name in the Dataset Card.

Dear @SBrandeis , @lhoestq . I am not sure how to fix the PR with respect to the additional files that are currently included in the commits. Could you provide me with an example? Otherwise I would be happy to close/re-open another PR. Please let me know if anything is missing for the review.

Hi @tdiggelm, thanks for the contribution! This dataset is really awesome.
I believe creating a new branch from master and opening a new PR with your changes is the simplest option since no review has been done yet. Feel free to ping us when it's done.

> Hi @tdiggelm, thanks for the contribution! This dataset is really awesome.
> I believe creating a new branch from master and opening a new PR with your changes is the simplest option since no review has been done yet. Feel free to ping us when it's done.

Thank you very much for your quick reply! Will do ASAP and ping you when done.

This PR request the addition of the CLIMATE-FEVER dataset:
A dataset adopting the FEVER methodology that consists of 1,535 real-world claims regarding climate-change collected on the internet. Each claim is accompanied by five manually annotated evidence sentences retrieved from the English Wikipedia that support, refute or do not give enough information to validate the claim totalling in 7,675 claim-evidence pairs. The dataset features challenging claims that relate multiple facets and disputed cases of claims where both supporting and refuting evidence are present.

More information can be found at:
- Homepage: <http://climatefever.ai>
- Paper: <https://arxiv.org/abs/2012.00614>


update multiwozv22 checksums

a file was updated on the GitHub repo for the dataset


Add comet metric

I also thought a bit about the fact that "sources" can't be added to the batch.. but changing that would require a lot more changes. And I agree that the idea of adding them as part of the references is not ideal. Conceptually they are not references.

I would keep it like this for now.. And in the future, work on a more consistent batch interface.

Hey! I decided to add our new Crosslingual Optimized Metric for Evaluation of Translation (COMET) to the list of the available metrics.

COMET was [presented at EMNLP20](https://www.aclweb.org/anthology/2020.emnlp-main.213/) and it is the highest performing metric, so far, on the WMT19 benchmark.

We also participated in the [WMT20 Metrics shared task ](http://www.statmt.org/wmt20/pdf/2020.wmt-1.101.pdf) where once again COMET was validated as a top-performing metric.

I hope that this metric will help researcher's and industry workers to better validate their MT systems in the future 🤗 !

Cheers,
Ricardo


Remove the contributors section

sourcerer is down


Hind_Encorp all done

@rahul-art  kindly run the following from the datasets folder

```
make style
flake8 datasets

```

@skyprince999 I did that before it says all done

I did that again it gives the same output all done and then  I synchronized my changes with this branch

@lhoestq i did all the changes you suggested but at the time of load_dataset it is giving me error
`**`datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=76591256, num_examples=1, dataset_name='hind_encorp'), 'recorded': SplitInfo(name='train', num_bytes=78945714, num_examples=273885, dataset_name='hind_encorp')}]`**`

I cloned the branch and it seems to work fine at my end. try to clear the cache -

```
rm -rf  /home/ubuntu/.cache/huggingface/datasets/
rm -rf  /home/ubuntu/.cache/huggingface/modules/datasets_modules//datasets/
```
But the dataset has only one record. Is that correct?
![image](https://user-images.githubusercontent.com/9033954/102331376-c7929b00-3fb0-11eb-8a6c-81b2cf47bc2a.png)

> @lhoestq i did all the changes you suggested but at the time of load_dataset it is giving me error
> `**`datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=76591256, num_examples=1, dataset_name='hind_encorp'), 'recorded': SplitInfo(name='train', num_bytes=78945714, num_examples=273885, dataset_name='hind_encorp')}]`**`

You can ignore this error by adding `ignore_verifications=True` to `load_dataset`.

This error is raised because you're loading a dataset that you've already loaded once in the past. Therefore the library does some verifications to make sure it's generated the same way.

However since you've done changes in the dataset script you should ignore these verifications.

You can regenerate the dataset_infos.json with
```
datasets-cli test ./datasets/hindi_encorp --save_infos --all_configs --ignore_verifications
```

> I cloned the branch and it seems to work fine at my end. try to clear the cache -
>
> ```
> rm -rf  /home/ubuntu/.cache/huggingface/datasets/
> rm -rf  /home/ubuntu/.cache/huggingface/modules/datasets_modules//datasets/
> ```
>
> But the dataset has only one record. Is that correct?
> ![image](https://user-images.githubusercontent.com/9033954/102331376-c7929b00-3fb0-11eb-8a6c-81b2cf47bc2a.png)

Yes the current parsing is wrong, I've already given @rahul-art some suggestions and it looks like it works way better now (num_examples=273885).

Thanks for fixing the parsing @rahul-art !
Feel free to commit and push your changes once it's ready :)

i ran the command you provided datasets-cli test ./datasets/hindi_encorp --save_infos --all_configs --ignore_verifications
but now its giving this error @lhoestq

FileNotFoundError: Couldn't find file locally at ./datasets/hindi_encorp/hindi_encorp.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/./datasets/hindi_encorp/hindi_encorp.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/./datasets/hindi_encorp/hindi_encorp.py.
If the dataset was added recently, you may need to to pass script_version="master" to find the loading script on the master branch.

whoops I meant `hind_encorp` instead of `hindi_encorp` sorry

@lhoestq  all changes have done successfully in this PR #1584

Ok thanks ! closing this one in favor of #1584


Diplomacy detection 3


adding dataset for diplomacy detection-2


add Gnad10 dataset

reference [PR#1317](https://github.com/huggingface/datasets/pull/1317)


Fixing the KILT tasks to match our current standards

This introduces a few changes to the Knowledge Intensive Learning task benchmark to bring it more in line with our current datasets, including adding the (minimal) dataset card and having one config per sub-task


Documentation for loading CSV datasets misleads the user into thinking setting `quote_char' to False will disable quoting.

There are two problems here:
  i) `quote_char' is misspelled, must be `quotechar'
  ii) the documentation should mention `quoting'


added un_ga dataset

Hi :hugs:, This is a PR for [United nations general assembly resolutions: A six-language parallel corpus](http://opus.nlpl.eu/UN.php) dataset.
With suggested changes in #1330


Added the dataset clickbait_news_bg

Hi @tsvm  Great work!
Since you have raised a clean PR could you close the earlier one -   #1445 ?

> Hi @tsvm Great work!
> Since you have raised a clean PR could you close the earlier one - #1445 ?

Done.

There was a problem with my [previous PR 1445](https://github.com/huggingface/datasets/pull/1445) after rebasing, so I'm copying the dataset code into a new branch and submitting a new PR.


[wording] Update Readme.md

Make the features of the library clearer.


Add Microsoft Research Sequential Question Answering (SQA) Dataset

I proposed something a few weeks ago in #898 (un-merged) but I think that the way that @mattbui added the dataset in the present PR is smarter and simpler should replace my PR #898.

(Narrator voice: *And it was around that time that Thomas realized that the community was now a lot smarter than him and he should hand-over the library he had started with @lhoestq to the community and stop pretending he knew everything about it.*)

For more information: https://msropendata.com/datasets/b25190ed-0f59-47b1-9211-5962858142c2


Create README.md

@ManuelFay thanks you so much for adding a dataset card, this is such a cool contribution!

This looks like it uses an old template for the card we've moved things around a bit and we have an app you should be using to get the tags and the structure of the Data Fields paragraph :) Would you mind moving your text to the newer format (we're also asking contributors to keep the full template structure, even if some sections still have [More Information Needed] for the time being)

Here's the link to the instructions:
https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

Out of curiosity, what was your landing point for filling out the card? Did you follow the "Update on Github" when navigating the datasets? Trying to make the instructions as clear as possible :)

@yjernite

Perfect, I'll follow the instructions when I have a bit more time tomorrow ! I was actually browsing the new contributions after the dataset sprint and realized most of the "old" datasets were not tagged, so I just copied and pasted the readme from another dataset and was not aware there was precise instructions... Will fix !

BTW, amazing job with the retriBert work, I used the contrastive + in-batch negative quite a bit for various projects. Probably neither the time nor place to talk about that but I was curious as to why, in your original work, you prefered using a simple projection in the last layer to differentiate the question vs answer embedding, rather than allowing for bias in the dense layer or even just to fine-tune 2 different embedders for question + answer ?

Cool! Looking forward to the next version!

Quick answer for retriBERT is that I expected a simple projection to generalize better and more importantly only having to store the gradients for the proj means training with larger batches :) If you want to keep chatting about it, feel free to send me an email!

Hi @ManuelFay !
If you're still interested in completing the FQuAD dataset card, note that we've generated one that is pre-filled.
Therefore feel free to complete it with the content you already have in your README.md.
This would be awesome ! And thanks again for your contribution :)

Yo @lhoestq , just not sure about the tag table at the top, I used @yjernite eli5 template so hope it's okay ! Also want to signal the streamlit app for dataset tagging has a weird behavior with the size categories when filling in the form.

Thanks to you guys for doing that and sorry about the time it took, i completely forgot about it !


added saudinewsnet

Hi @abdulelahsm - This is an interesting dataset! But there are multiple issues with the PR. Some of them are listed below:
-  default builder config is not defined. There should be atleast one builder config
- URL is incorrectly constructed so the data files are not being downloaded
- dataset_info.json file was not created

Please have a look at some existing merged datasets to get a reference on building the data loader. If you are still stuck, reach out.

@skyprince999 I totally agree. Thx for the feedback!

Hi @abdulelahsm ! Thanks for adding this one :)
you don't actually have to add builder configurations if you don't need them. It's fine as it is now

And as @skyprince999 noticed, the current URLs don't work. to download files.
You can use this one for example for the first batch instead:
https://github.com/parallelfold/SaudiNewsNet/raw/master/dataset/2015-07-21.zip

Feel free to ping me if you have questions or if you're ready for a review :)

@lhoestq Hey, I tried using the first batch instead, the data was downloaded but I got this error, not sure why it can't find the path?

for content, I ran ``` "./datasets/saudinewsnet/test.py"```

which is a local test I'm running for the dataset, it contains the following code

```
from datasets import load_dataset

data = load_dataset("./datasets/saudinewsnet", split= "train")

print(data)

print(data[1])
```

this is the error I got

```
2020-12-18 21:45:39.403908: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-12-18 21:45:39.403953: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Using custom data configuration default
Downloading and preparing dataset saudi_news_net/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/mesfas/.cache/huggingface/datasets/saudi_news_net/default/0.0.0/62ece5ef0a991415352d4b1efac681d75b5b3404064fd4f6a1d659499dab18f4...
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.42M/3.42M [00:03<00:00, 1.03MB/s]
Traceback (most recent call last):
  File "/home/mesfas/opensource/datasets/src/datasets/builder.py", line 604, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/mesfas/opensource/datasets/src/datasets/builder.py", line 902, in _prepare_split
    for key, record in utils.tqdm(
  File "/home/mesfas/environments/ar_res_reviews/lib/python3.8/site-packages/tqdm/std.py", line 1133, in __iter__
    for obj in iterable:
  File "/home/mesfas/.cache/huggingface/modules/datasets_modules/datasets/saudinewsnet/62ece5ef0a991415352d4b1efac681d75b5b3404064fd4f6a1d659499dab18f4/saudinewsnet.py", line 108, in _generate_examples
    with open(filepath, encoding="utf-8").read() as f:
IsADirectoryError: [Errno 21] Is a directory: '/home/mesfas/.cache/huggingface/datasets/downloads/extracted/314fd983aa07d3dada9429911a805270c3285f48759d3584a1343c2d86260765'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./datasets/saudinewsnet/test.py", line 3, in <module>
    data = load_dataset("./datasets/saudinewsnet", split= "train")
  File "/home/mesfas/opensource/datasets/src/datasets/load.py", line 607, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/mesfas/opensource/datasets/src/datasets/builder.py", line 526, in download_and_prepare
    self._download_and_prepare(
  File "/home/mesfas/opensource/datasets/src/datasets/builder.py", line 606, in _download_and_prepare
    raise OSError(
OSError: Cannot find data file.
Original error:
[Errno 21] Is a directory: '/home/mesfas/.cache/huggingface/datasets/downloads/extracted/314fd983aa07d3dada9429911a805270c3285f48759d3584a1343c2d86260765'
```

this is the split code

```
    def _split_generators(self, dl_manager):
        """Returns SplitGenerators."""
        my_urls = _URL
        datadir = dl_manager.download_and_extract(my_urls)
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN,
                # These kwargs will be passed to _generate_examples
                gen_kwargs={
                    "filepath": datadir,
                    "split": "train"
                },
            ),
        ]
```
and this is how I'm generating the examples

```
    def _generate_examples(self, filepath, split):

        #logging.info("generating examples from = %s", filepath)
        with open(filepath, encoding="utf-8") as f:
            articles = json.load(f)
            for article in articles:
                title = article.get("title", "").strip()
                source = article.get("source", "").strip()
                date = article.get("date_extracted", "").strip()
                link = article.get("url", "").strip()
                author = article.get("author", "").strip()
                content = article.get("content", "").strip()

                yield id_, {
                    "title": title,
                    "source": source,
                    "date": date,
                    "link": link,
                    "author": author,
                    "content": content
                }
```

What's `_URL` ?

It looks like you are downloading an archive.
Therefore you may need to get to the file path using `filepath = os.path.join(datadir, "actual_file_name_inside_the_downloaded_archive")`

@lhoestq you were 100% right. Thank you. All fixed

@lhoestq  added the remaining 17 batches and modified the readme.md to reflect that + resolved the camel case comment

I'm having issues in creating the dummy data. I'm still investigating how to fix it. I'll close the PR if I couldn't find a solution


adding tmu-gfm-dataset

@lhoestq Thank you for your code review! I think I could do the necessary corrections. Could you please check it again when you have time?

Adding TMU-GFM-Dataset for Grammatical Error Correction.

https://github.com/tmu-nlp/TMU-GFM-Dataset

A dataset for GEC metrics with manual evaluations of grammaticality, fluency, and meaning preservation for system outputs.
More detail about the creation of the dataset can be found in [Yoshimura et al. (2020)](https://www.aclweb.org/anthology/2020.coling-main.573.pdf).


Add dataset COrpus of Urdu News TExt Reuse (COUNTER).

Just a small revision from simon's review: 20KB for the dummy_data.zip is fine, you can keep them this way.

Also the CI is failing because of an error `tests/test_file_utils.py::TempSeedTest::test_tensorflow` that is not related to your dataset and is fixed on master. You can ignore it


Lama

Let me know why the pyarrow test is failing. For one of the config "trex", I had to load an initial datafile for a dictionary which is used to augment the rest of the datasets. In the dummy data, the dictionary file was truncated so I had to fudge that. I'm not sure if that is the issue.

@ontocord it just needs a rerun and it will be good to go.

@ontocord contributor can’t rerun it, the maintainers will rerun it, it may take lil bit of time as there are so many PRs left to be reviewed and merged

@lhoestq not sure why it is failing. i've made all modifications.

This the LAMA dataset for probing facts and common sense from language models.

See https://github.com/facebookresearch/LAMA for more details.


Adding the BrWaC dataset

Adding the BrWaC dataset, a large corpus of Portuguese language texts


adding dataset card information to CONTRIBUTING.md

Added a documentation line and link to the full sprint guide in the "How to add a dataset" section, and a section on how to contribute to the dataset card of an existing dataset.

And a thank you note at the end :hugs:


Adding Igbo NER data

Thanks for the PR @purvimisal.

Few comments below.

Hi, @lhoestq Thank you for the review. I have made all the changes. PTAL!

the CI error is not related to your dataset, merging

This PR adds the Igbo NER dataset.
Data: https://github.com/IgnatiusEzeani/IGBONLP/tree/master/ig_ner


HindEncorp again commited

Yes this has the right files!!!

I'll close the previous one then :)

Now to pass the tests, you will need to:
- `make style` and run `flake8 datasets` from your repository root directory
- fix the dummy data

Did you generate the dummy data with the auto-generation tool (see the guide) or manually?

manually with the tool, it is not able to create

Cool, in that case you need to pay special attention to the directory structure given to you by the tool, most failures are because the files are in the wrong directory or at the wrong level :)

Also, make sure that the tests pass locally before pushing to the branch, it should help you get the structure right ;)

yes I have give proper directory structure datasets/hind_encorp/dummy/0.0.0/dummy_data.zip  but in my dummy_data.zip only 1 file hind_encorp.plaintext is present because the dataset I got has only 1 file with both English and Hindi languages on 1 file itself may be this is causing issue

Looks like the name of the file is the issue here: you have a file called `hindencorp05.plaintext`, but it should be called `hindencorp05.plaintext.gz%3Fsequence%3D3%26isAllowed%3Dy`. You just have to rename it to pass the test:
```
cd datasets/hind_encorp/dummy/0.0.0
rm -rf dummy_data
unzip dummy_data.zip
rm dummy_data.zip
mv dummy_data/hindencorp05.plaintext "dummy_data/hindencorp05.plaintext.gz%3Fsequence%3D3%26isAllowed%3Dy"
zip -r dummy_data.zip dummy_data
```

Then **once you pass the tests locally** you just have to remember to `make style` and `flake8 datasets` to pass the style checks, and you should be good to go :hugs:

For reference, here are the instructions given by the tool:
```
$ python datasets-cli dummy_data datasets/hind_encorp/
2020-12-14 13:16:26.824828: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-12-14 13:16:26.824846: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Using custom data configuration default

==============================DUMMY DATA INSTRUCTIONS==============================
- In order to create the dummy data for , please go into the folder 'datasets/hind_encorp/dummy/0.0.0' with `cd datasets/hind_encorp/dummy/0.0.0` .

- Please create the following dummy data files 'dummy_data/hindencorp05.plaintext.gz%3Fsequence%3D3%26isAllowed%3Dy' from the folder 'datasets/hind_encorp/dummy/0.0.0'

- For each of the splits 'train', make sure that one or more of the dummy data files provide at least one example

- If the method `_generate_examples(...)` includes multiple `open()` statements, you might have to create other files in addition to 'dummy_data/hindencorp05.plaintext.gz%3Fsequence%3D3%26isAllowed%3Dy'. In this case please refer to the `_generate_examples(...)` method

-After all dummy data files are created, they should be zipped recursively to 'dummy_data.zip' with the command `zip -r dummy_data.zip dummy_data/`

-You can now delete the folder 'dummy_data' with the command `rm -r dummy_data`

- To get the folder 'dummy_data' back for further changes to the dummy data, simply unzip dummy_data.zip with the command `unzip dummy_data.zip`

- Make sure you have created the file 'dummy_data.zip' in 'datasets/hind_encorp/dummy/0.0.0'
===================================================================================
```

all test passed locally.   my  new  PR #1575


add bswac


Added Opus TedTalks

@lhoestq I saw some common changes you made on the other PR's (Similar Opus Datasets). I fixed those changes here. Can you please review it once ?
Thanks.

Dataset : http://opus.nlpl.eu/TedTalks.php


Opus CAPES added

Hi @rkc007 , thanks for the contribution.
Unfortunately, the CAPES dataset has already been added here: #1307
I'm closing the PR

Dataset : http://opus.nlpl.eu/CAPES.php


added air_dialogue

UPDATE2 (3797ce5): Updated for multi-configs

UPDATE (7018082): manually created the dummy_datasets. All tests were cleared locally. Pushed it to origin/master

DRAFT VERSION (57fdb20):  (_no longer draft_)
Uploaded the air_dialogue database.
dummy_data creation was failing in local, since the original downloaded file has some nested folders. Pushing it since the tests with real data was cleared. Will re-check & update via manually creating some dummy_data


Added OPUS ParaCrawl

@rkc007 @lhoestq just noticed a dataset named para_crawl has been added a long time ago: #91.

They're not exactly the same so it's ok to have both.

Especially the `para_crawl` that already exists only uses the text from the ParaCrawl release 4.

Could you regenerate the dataset_infos.json @rkc007 please ?
It looks like it has some issues due to the dataset class name change

@SBrandeis Thank you for suggesting changes. I made the changes you suggested.

@lhoestq I generated `dataset_infos.json` again. I ran both tests(Dummy & Real data) and it passed. Can you please review it again?

Dataset : http://opus.nlpl.eu/ParaCrawl.php


Monero

Hi @iliemihai  - you need to add the Readme file! Otherwise seems good.
Also don't forget to run `make style` & `flake8 datasets` locally, from the datasets folder

@skyprince999 I will add the README.d for it. Thank you :D

Biomedical Romanian dataset :)


Add offensive langauge dravidian dataset


Generics kb new branch

Datasets need manual downloads. Have thus created dummy data as well. But pytest on real and dummy data are failing.
I have completed the readme , tags and other required things. I need to create the metadata json once tests get successful.
Opening a PR while working with Yacine Jernite to resolve my pytest issues.


Fix `🤗Datasets` - `tfds` differences link + a few aesthetics


Adding PolEval2019 Machine Translation Task dataset

**NOTE:**

- Train and Dev: Manually downloaded (auto download is repeatedly giving `ConnectionError` for one of the files), Test: Auto Download
- Dummy test is passing
- The json file has been created with hard-coded paths for the manual downloads _(hardcoding has been removed from the final uploaded script)_
- datasets-cli is still **failing** . It is not picking the right directory for the config. For instance, my folder structure is as below:
 ```
            ~/Downloads/Data/
             |--- English-to-Polish
                   |--- (corresponding files)
             |--- Russian-Polish
                   |--- (corresponding files)
```

When ru-pl is selected, ideally it has to search in Russian-Polish folder, but it is searching in '/Downloads/Data/' folder and hence getting a FileNotFound error.

The command run is
`python datasets-cli test datasets/poleval2019_mt/ --save_infos --all_configs --data_dir ~/Downloads/Data/
`

Hi !
Thanks for the changes :)

The only error left is the dummy data. Since we changed for standard downloads instead of manual downloads its structure changed. Fortunately you can auto-generate the dummy data with this command:

```
datasets-cli dummy_data ./datasets/poleval2019_mt --auto_generate --match_text_files "*"
```

Can you regenerate the dummy data using this command please ?

Thank you for the help @lhoestq !! I was generating the dummy dataset in a wrong way! That _--match_text_files "*"_ did the trick! Now all the tests have passed! :-)

Facing an error with pytest in training. Dummy data is passing.
README has to be updated.


Add persian ner dataset

HI @SBrandeis. Thanks for all the comments - very helpful. I realised that the tests had failed and had been trying to figure out what was causing them to do so. All the tests pass when I run the load_real_dataset test however when I run `RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_persian_ner` I get the below error. One thing to note is that the automated dummy data file generation failed when I tried to run it so I manually created the dummy data and ensured that the last line in the file was an empty line as per your comments. Would appreciate your thoughts on what might be causing this:

```
__________________________________________________ LocalDatasetTest.test_load_dataset_all_configs_persian_ner __________________________________________________

self = <tests.test_dataset_common.LocalDatasetTest testMethod=test_load_dataset_all_configs_persian_ner>, dataset_name = 'persian_ner'

    @slow
    def test_load_dataset_all_configs(self, dataset_name):
        configs = self.dataset_tester.load_all_configs(dataset_name, is_local=True)
>       self.dataset_tester.check_load_dataset(dataset_name, configs, is_local=True)

tests/test_dataset_common.py:237:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_dataset_common.py:198: in check_load_dataset
    self.parent.assertTrue(len(dataset[split]) > 0)
E   AssertionError: False is not true
--------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
Downloading and preparing dataset persian_ner/fold1 (download: 1.00 MiB, generated: 1.00 MiB, post-processed: Unknown size, total: 2.00 MiB) to /var/folders/nk/yp5_m5c95cnc0cm_vbd7h7g80000gn/T/tmpzh495aac/persian_ner/fold1/1.1.0...
Dataset persian_ner downloaded and prepared to /var/folders/nk/yp5_m5c95cnc0cm_vbd7h7g80000gn/T/tmpzh495aac/persian_ner/fold1/1.1.0. Subsequent calls will reuse this data.
--------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------

======================================================================= warnings summary =======================================================================
env/lib/python3.7/site-packages/tensorflow/python/autograph/utils/testing.py:21
  /Users/karimfoda/Documents/STUDIES/PYTHON/DATASETS/env/lib/python3.7/site-packages/tensorflow/python/autograph/utils/testing.py:21: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

env/lib/python3.7/site-packages/apache_beam/typehints/typehints.py:693
  /Users/karimfoda/Documents/STUDIES/PYTHON/DATASETS/env/lib/python3.7/site-packages/apache_beam/typehints/typehints.py:693: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    if not isinstance(type_params, collections.Iterable):

env/lib/python3.7/site-packages/apache_beam/typehints/typehints.py:532
  /Users/karimfoda/Documents/STUDIES/PYTHON/DATASETS/env/lib/python3.7/site-packages/apache_beam/typehints/typehints.py:532: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    if not isinstance(type_params, (collections.Sequence, set)):

env/lib/python3.7/site-packages/elasticsearch/compat.py:38
  /Users/karimfoda/Documents/STUDIES/PYTHON/DATASETS/env/lib/python3.7/site-packages/elasticsearch/compat.py:38: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    from collections import Mapping

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=================================================================== short test summary info ====================================================================
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_persian_ner - AssertionError: False is not true
```

Thanks @SBrandeis. It turns out the error was because I had to manually increase the n_lines variable to get the dummy data generation to cover at least one example. Should all be working okay now.

Great, thanks!
It looks good to me, I'll let @lhoestq take over

Adding the following dataset:

https://github.com/HaniehP/PersianNER


add hrwac


Added Wiki Summary Dataset

Maybe an issue with CircleCI, let me try to make them run

The CI error `tests/test_file_utils.py::TempSeedTest::test_tensorflow` is not related to this dataset and is fixed on master, you can ignore it

Now the delimiter of the csv reader is fixed, thanks :)

I just added a comment suggesting to try using actual URLS instead of a manual download if possible.
This would make things more convenient for the users. Can you try using the `dl_manager` to download the train/dev/test csv files instead of requiring manual download ?

Also pinging @m3hrdadfi , since I just noticed that there's already a dataset script that was created 3 weeks ago for this dataset here: https://github.com/m3hrdadfi/wiki-summary/tree/master/datasets/wiki_summary_persian

@lhoestq I am getting this error while generating the dummy data
![Screenshot (181)](https://user-images.githubusercontent.com/33005287/102628819-50a40080-4170-11eb-9e96-efce74b45ff4.png)

Can you try by adding the flag `--match_text_files "*"` ?

@lhoestq pytest on dummy data passed, but on real data raising this issue
![Screenshot (196)](https://user-images.githubusercontent.com/33005287/102630784-fa848c80-4172-11eb-9f7e-e5a58dcf7abe.png)
how to resolve it

I see ! This is because the library did some verification to make sure it downloads the same files as in the first time you ran the `datasets-cli test` command with `--save_infos`. Since we're now downloading files, the verification fails.

To fix that you just need to regenerate the dataset_infos.json file:
```
datasets-cli test ./datasets/wiki_summary --save_infos --all_configs --ignore_verifications
```

@lhoestq I have modified everything and It worked fine, dont know why it is not passing the tests

Awesome thank you !

The CI error `tests/test_file_utils.py::TempSeedTest::test_tensorflow` is not related to your dataset and is fixed on master.
You can ignore it :)

The dataset script is all good now ! The dummy data and the dataset_infos.json file are good too :)

@lhoestq yay, thanks for helping me out ,

Wiki Summary: Dataset extracted from Persian Wikipedia into the form of articles and highlights.
Link: https://github.com/m3hrdadfi/wiki-summary


adding HindEncorp

@lhoestq I have created a new PR by reforking and creating a new branch

@rahul-art unfortunately this didn't quite work, here's how you can try again:
- `git checkout master` to go back to the main branch
- `git pull upstream master` to make it up to date
- `git checkout -b add_hind_encorp` to create a new branch

Then add the dataset script, `README.md`, `dummy_data.zip`, and `dataset_infos.json` to the tracked files for the branch with `git add` (please add all of these files individually, NOT the whole directory as we don't want the other data files)
Then after you have passed the style checks and the local tests, do:
- `git commit . -m initial_commit`
- `git push --set-upstream origin add_hind_encorp`

Then you can go to this branch on the WebApp and open a new PR

adding Hindi Wikipedia corpus


fix typo readme


connection issue while downloading data

could you tell me how I can avoid download, by pre-downloading the data first, put them in a folder so the code does not try to redownload? could you tell me the path to put the downloaded data, and how to do it? thanks
@lhoestq

Does your instance have an internet connection ?

If you don't have an internet connection you'll need to have the dataset on the instance disk.
To do so first download the dataset on another machine using `load_dataset` and then you can save it in a folder using `my_dataset.save_to_disk("path/to/folder")`. Once the folder is copied on your instance you can reload the dataset with `datasets.load_from_disk("path/to/folder")`

Hi
I am running my codes on google cloud, and I am getting this error resulting in the failure of the codes when trying to download the data, could you assist me to solve this? also as a temporary solution, could you tell me how I can increase the number of retries and timeout to at least let the models run for now. thanks

```
Traceback (most recent call last):
  File "finetune_t5_trainer.py", line 361, in <module>
    main()
  File "finetune_t5_trainer.py", line 269, in main
    add_prefix=False if training_args.train_adapters else True)
  File "/workdir/seq2seq/data/tasks.py", line 70, in get_dataset
    dataset = self.load_dataset(split=split)
  File "/workdir/seq2seq/data/tasks.py", line 306, in load_dataset
    return datasets.load_dataset('glue', 'cola', split=split)
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 589, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 263, in prepare_module
    head_hf_s3(path, filename=name, dataset=dataset)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py", line 200, in head_hf_s3
    return http_head(hf_bucket_url(identifier=identifier, filename=filename, use_cdn=use_cdn, dataset=dataset))
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py", line 403, in http_head
    url, proxies=proxies, headers=headers, cookies=cookies, allow_redirects=allow_redirects, timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 104, in head
    return request('head', url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 504, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /datasets.huggingface.co/datasets/datasets/glue/glue.py (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f47db511e80>, 'Connection to s3.amazonaws.com timed out. (connect timeout=10)'))
```


added TTC4900: A Benchmark Data for Turkish Text Categorization dataset

@lhoestq, can you help with creating dummy_data?

> Hi @yavuzKomecoglu did you manage to build the dummy data ?

Hi, sorry for the return. I've created dummy_data.zip manually.

> Nice thank you !
>
> Before we merge can you fill the two sections of the dataset card I suggested ?
> And also remove one remaining print statement

I updated your suggestions. Thank you very much for your support.

I think you accidentally pushed the readme of another dataset (name_to_nation).
I removed it so you have to `git pull`

Because of that I guess your changes about the ttc4900 was not included.
Feel free to ping me once they're added

> I think you accidentally pushed the readme of another dataset (name_to_nation).
> I removed it so you have to `git pull`
>
> Because of that I guess your changes about the ttc4900 was not included.
> Feel free to ping me once they're added

I did `git pull` and updated readme **ttc4900**.

This PR adds the TTC4900 dataset which is a Turkish Text Categorization dataset by me and @basakbuluz.

Homepage: [https://www.kaggle.com/savasy/ttc4900](https://www.kaggle.com/savasy/ttc4900)
Point of Contact: [Savaş Yıldırım](mailto:savasy@gmail.com) / @savasy


Added Wiki Asp dataset

> Awesome thank you !
>
> I just left one comment.
>
> Also it looks like the dummy_data.zip files are quite big (around 500KB each)
> Can you try to reduce their sizes please ? Ideally they should be <20KB each
>
> To do so feel free to take a look inside them and in the jsonl files only keep 1 or 2 samples instead of 5 and also remove big chunks of text to only keep a few passages.

Thanks, I have updated the dummy data to keep each domain <20/30KB.

> > Awesome thank you !
> > I just left one comment.
> > Also it looks like the dummy_data.zip files are quite big (around 500KB each)
> > Can you try to reduce their sizes please ? Ideally they should be <20KB each
> > To do so feel free to take a look inside them and in the jsonl files only keep 1 or 2 samples instead of 5 and also remove big chunks of text to only keep a few passages.
>
> Thanks, I have updated the dummy data to keep each domain <20/30KB.

Looks like this branch has other commits. I will open a new PR with suggested changes.

Hello,

I have added Wiki Asp dataset. Please review the PR.


tweets_hate_speech_detection

Hi @lhoestq  I have added this new dataset for tweet's hate speech detection.

Please if u could review it.

Thank you

Hi @darshan-gandhi have you add a chance to take a look at my suggestions ?

Feel free to ping me when you're ready for the final review


added ohsumed

UPDATE2: PR passed all tests. Now waiting for review.

UPDATE: pushed  a new version. cross fingers that it should complete all the tests! :)
               If it passes all tests then it's not a draft version.

This is a draft version


Add Hippocorpus Dataset

> Before we merge can you try to reduce the size of the dummy_data.zip file ?
>
> To do so feel free to only keep a few lines of the csv files ans also remove unnecessary chunks of texts (for example keep only the first sentences of a story).

Hi @lhoestq, I have reduced the size of the dummy_data.zip file by making the necessary changes you had suggested.


Adding Igbo monolingual dataset

@lhoestq Thank you for the review. I have made all the changes you mentioned. PTAL!

This PR adds the Igbo Monolingual dataset.
Data: https://github.com/IgnatiusEzeani/IGBONLP/tree/master/ig_monoling
Paper: https://arxiv.org/abs/2004.00648


adding dataset for diplomacy detection

Requested changes made and new PR submitted here: https://github.com/huggingface/datasets/pull/1580


add id_panl_bppt, a parallel corpus for en-id

Hi @lhoestq, thanks for the review. I will have a look and update it accordingly.

Strange error message :-)

```
>           tf_context = tf.python.context.context()  # eager mode context
E           AttributeError: module 'tensorflow' has no attribute 'python'
```

Parallel Text Corpora for English - Indonesian


adding hate-speech-and-offensive-language

made suggested changes and a new PR created here : https://github.com/huggingface/datasets/pull/1597


add indonlu benchmark datasets

The IndoNLU benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems for the Indonesian language. There are 12 datasets in IndoNLU.

This is a new clean PR from [#1322](https://github.com/huggingface/datasets/pull/1322)


Ro sent

Hi @iliemihai, it looks like this PR holds changes from your previous PR #1493 .
Would you mind removing them from the branch please ?

@SBrandeis I am sorry. Yes I will remove them. Thank you :D

Hi @lhoestq @SBrandeis @iliemihai

Is this still in progress or can I take over this one?

Thanks,
Gunjan

Hi,
While trying to add this dataset, I found some potential issues.
The homepage mentioned is : https://github.com/katakonst/sentiment-analysis-tensorflow/tree/master/datasets/ro/, where the dataset is different from the URLs: https://raw.githubusercontent.com/dumitrescustefan/Romanian-Transformers/examples/examples/sentiment_analysis/ro/train.csv. It is unclear which dataset is "correct". I checked the total examples (train+test) in both places and they do not match.

We should use the data from dumitrescustefan and set the homepage to his repo IMO, since he's first author of the paper of the dataset.

Hi @lhoestq,

Cool, I'll get working on it.

Thanks

Closing in favor of #2011
Thanks again for adding it !

Movies reviews dataset for Romanian language.


initial commit for Common Crawl Domain Names


Add : Conv AI 2 (Messed up original PR)

@lhoestq Sorry I messed up the previous 2 PR's -> https://github.com/huggingface/datasets/pull/1462 -> https://github.com/huggingface/datasets/pull/1383. So created a new one. Also, everything is fixed in this PR. Can you please review it ?
Thanks in advance.


added Hebrew thisworld corpus

added corpus from https://thisworld.online/ , https://github.com/thisworld1/thisworld.online


Adding a second branch for Atomic to fix git errors

Adding the Atomic common sense dataset.
See https://homes.cs.washington.edu/~msap/atomic/


ADD: swahili dataset for language modeling

Add a corpus for Swahili language modelling. All tests passed locally. README updated with all information available.


Add eHealth Knowledge Discovery dataset

Thank you very much for your review @lewtun !

I've updated the script metadata, created the README and fixed the two details you commented.

Ready for another review! 🤗

I've updated the task tag as we discussed and also added a couple of lines of code to make sure I include all the available examples.

Thank you again!

This Spanish dataset can be used to mine knowledge from unstructured health texts.

In particular, for:
- Entity recognition
- Relation extraction


Add semeval 2020 task 11

@SBrandeis : Thanks for the feedback! Just updated to use context manager for the `open`s and removed the placeholder text from the `README`!

Great, thanks @ZacharySBrown !
Failing tests seem to be unrelated to your changes, merging the current master branch into yours should fix them.

Adding in propaganda detection task (task 11) from Sem Eval 2020


Atomic

I had to create a new PR to fix git errors. See: https://github.com/huggingface/datasets/pull/1525

I'm closing this PR.

This is the ATOMIC common sense dataset. More info can be found here:

* README.md still to be created.


ru_reviews dataset adding

Hi @lhoestq

I have added the readme as well

Please do have a look at it when suitable

Chatted with @darshan-gandhi on Slack about parsing examples into a separate text and sentiment field

RuReviews: An Automatically Annotated Sentiment Analysis Dataset for Product Reviews in Russian


Initial commit for AQuaMuSe

@yjernite Thank you for your help, generating the dummy data 🤗 Having that all the tests have passed 👍🏻

There is an issue in generation of dummy data. Tests on real data have passed locally.


Add twi text

Add Twi texts


Kd conv smangrul

Hii please follow me
merging since the CI is fixed on master


adding wrbsc

@lhoestq thanks for the comments! Should be fixed in the latest commit, I assume the CI errors are unrelated.


Add yoruba text

Adding Yoruba text C3


how to get all the options of a property in datasets

question

Further information is requested

In a dataset, labels correspond to the `ClassLabel` feature that has the `names` property that returns string represenation of the integer classes (or `num_classes` to get the number of different classes).

I think the `features` attribute of the dataset object is what you are looking for:
```
>>> dataset.features
{'sentence1': Value(dtype='string', id=None),
 'sentence2': Value(dtype='string', id=None),
 'label': ClassLabel(num_classes=2, names=['not_equivalent', 'equivalent'], names_file=None, id=None),
 'idx': Value(dtype='int32', id=None)
}
>>> dataset.features["label"].names
['not_equivalent', 'equivalent']
```

For reference: https://huggingface.co/docs/datasets/exploring.html

Hi
could you tell me how I can get all unique options of a property of dataset?
for instance in case of boolq, if the user wants to know which unique labels it has, is there a way to access unique labels without getting all training data lables and then forming a set i mean? thanks


app_reviews_by_users

Hi @lhoestq

I have added the readme file as well, please if you could check it once

Thank you

Software Applications User Reviews


Add Hippocorpus Dataset


poleval cyberbullying


Add Dataset for (qa_srl)Question-Answer Driven Semantic Role Labeling

- Added tags, Readme file
- Added code changes


Added dataset Makhzan

The only CI error comes from
```
FAILED tests/test_file_utils.py::TempSeedTest::test_tensorflow
```

which is not related to your PR and is fixed on master.

You can ignore it

@lhoestq I've made the changes. Please review and merge.

I have a similar PR https://github.com/huggingface/datasets/pull/1562 for another dataset. I'll incorporate your comment about sorting and reducing dummy dataset size there.

The CI raises an error `FAILED tests/test_file_utils.py::TempSeedTest::test_tensorflow` but it's not related to this dataset.
This issue is fixed on master

You did all the work ;) thanks

merging since the CI is fixed on master

Need help with the dummy data.


Fix namedsplit docs

Hii please follow me
Thanks @mariosasko!

Fixes a broken link and `DatasetInfoMixin.split`'s docstring.


Add SelQA Dataset

The CI error `FAILED tests/test_file_utils.py::TempSeedTest::test_tensorflow` is not related with this dataset and is fixed on master. You can ignore it

Add the SelQA Dataset, a new benchmark for selection-based question answering tasks
Repo: https://github.com/emorynlp/selqa/
Paper: https://arxiv.org/pdf/1606.08513.pdf


Add nq_open question answering dataset

@SBrandeis  thanks for the review, I applied your suggested changes, but CI is failing now not sure about the error.

Many thanks @Nilanshrajput  !
The failing tests on CI are not related to your changes, merging master on your branch should fix them :)
If you're interested in what causes the CI to fail, checkout [this commit](https://github.com/huggingface/datasets/commit/9a0f1e20ca1e783cb14c1ab1cc2f54b0b5b201e8)

Hello @Nilanshrajput, your PR includes changes from other branches too now (485 files changed)
Would you mind creating another branch from master with your changes and opening a new PR?

@SBrandeis sorry i messed up the git history, #1587 I opened this new pr!

Added nq_open Open-domain question answering dataset.

The NQ-Open task is currently being used to evaluate submissions to the EfficientQA competition, which is part of the NeurIPS 2020 competition track.


add ilist dataset

This PR will add Indo-Aryan Language Identification Shared Task Dataset.


Add SentiWS dataset for pos-tagging and sentiment-scoring (German)

Hi @lhoestq @yjernite, requesting you to review this for any changes needed. Thanks! :)


Adding COVID QA dataset in Chinese and English from UC SanDiego

Changed the pre-processing based on the comments raised in [PR-1482](https://github.com/huggingface/datasets/pull/1482).The below command is passing in my local environment:

`python datasets-cli test datasets/covid_qa_ucsd/ --save_infos --all_configs --data_dir ~/Downloads/Medical-Dialogue-Dataset/CovidDailogue/`


Add Senti_Lex Dataset

Better will be if you close this PR and make a fresh PR

Feel free to ping me if you also have questions about the dummy data

also it looks like this PR includes changes about dummy_data.zip files in the ./datasets//un_pc folder. Can you remove them ?

Thanks for all the advice @lhoestq. I've implemented the changes you kindly highlighted and have made sure the scripts pass all the test. I've also marked this as ready for review as I believe it's in a good place to be merged now.

Great suggestion I fixed the dummy data and the file paths.

TODO:
Fix feature format issue
Create dataset_info.json file
Run pytests
Make Style


Adds XED dataset


adding polsum

@lhoestq thanks for the comments! Should be fixed in the latest commit, I assume the CI errors are unrelated.


update the dataset id_newspapers_2018

Hi, I need to update the link to the dataset. The link in the previous PR was to a small test dataset. Thanks


add stereoset

StereoSet is a dataset that measures stereotype bias in language models. StereoSet consists of 17,000 sentences that measures model preferences across gender, race, religion, and profession.


adding fake-news-english-5

made suggested changes and created a PR here: https://github.com/huggingface/datasets/pull/1598


Add Multi-Dimensional Gender Bias classification data

https://parl.ai/projects/md_gender/

Mostly has the ABOUT dimension since the others are inferred from other datasets in most cases.

I tried to keep the dummy data small but one of the configs has 140 splits ( > 56KB data)


Opus DGT added

Dataset : http://opus.nlpl.eu/DGT.php


Added Opus Wikipedia

Dataset : http://opus.nlpl.eu/Wikipedia.php


Added RONEC dataset.

Thanks for the PR @iliemihai .

Few comments -

Can you run -
`python datasets-cli dummy_data ./datasets/ronec --auto_generate` to generate dummy data.

Also, before committing files run :
`make style`
`flake8 datasets`
then you can add and commit files.

> Thanks for the PR @iliemihai .
>
> Few comments -
>
> Can you run -
> `python datasets-cli dummy_data ./datasets/ronec --auto_generate` to generate dummy data.
>
> Also, before committing files run :
> `make style`
> `flake8 datasets`
> then you can add and commit files.

Sorry, forgot to generate dummy data. I will do it now :D

Awesome, good job @iliemihai !
I think the PR is ready to merge.
@lhoestq would you mind double-checking this ?

Had to regenerate the dummy data since I just found out they were empty files


OPUS UBUNTU dataset

Dataset : http://opus.nlpl.eu/Ubuntu.php


added opus GNOME data

Dataset : http://opus.nlpl.eu/GNOME.php


ADD: opus_rf dataset for translation

Passed all local tests. Hopefully passes all Circle CI tests too. Tried to keep the commit history clean.


Fake news english 4

Thanks for the PR @MisbahKhan789 !

Few comments to help you along (I'm NOT a maintainer, just offering help to unblock the process) :-
 - Could you re-run `make style` and fix the errors related to code quality specific to your dataset in the `datasets/fake_news_english` folder?
(These seem to show errors that need manual fixes on running `flake8 datasets/fake_news_english`)
- Please run the local tests and check if they pass
`RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_<your-dataset-name>`
 `RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<your-dataset-name>`
(If not, you may have to regenerate the dummy data and the `dataset_infos.json` files)

> Thanks for the PR @MisbahKhan789 !
>
> Few comments to help you along (I'm NOT a maintainer, just offering help to unblock the process) :-
>
> * Could you re-run `make style` and fix the errors related to code quality specific to your dataset in the `datasets/fake_news_english` folder?
>   (These seem to show errors that need manual fixes on running `flake8 datasets/fake_news_english`)
> * Please run the local tests and check if they pass
>   `RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_<your-dataset-name>`
>   `RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<your-dataset-name>`
>   (If not, you may have to regenerate the dummy data and the `dataset_infos.json` files)

Hey Bharat, thanks for the reply. I actually submitted a new PR with the changes that are required :)


Adding NELL

hi @lhoestq, I wanted to push another change to this branch b/c I found a bug in the parsing. I need to swap arg1 and arg2. I tried to git push -u origin nell but it didn't work. So I tried to do git push --force -u origin nell which seems to work, but nothing is happening to this branch. I think this is because it's closed. Do I need to open another PR?

The change should be below in _generate_examples:
                    best_arg1 = row[9]
                    best_arg2 = row[8]

Hi @ontocord !

Yup the easiest thing to do is to open a new PR with a title like "Fix NELL dataset argument order". If it's a simple fix we can look at it pretty fast :)

NELL is a knowledge base and knowledge graph along with sentences used to create the KB. See http://rtw.ml.cmu.edu/rtw/ for more details.


added conv_ai_3 dataset

@lhoestq Thank you for suggesting changes. I fixed all the changes you suggested. Can you please review it again?

@lhoestq Thank you for reviewing and suggesting changes. I made the requested changes. Can you please review it again?

Thanks @lhoestq for reviewing it again. I made the required changes. Can you please have a look ?

Dataset : https://github.com/aliannejadi/ClariQ/


hate speech 18 dataset

The error `tests/test_file_utils.py::TempSeedTest::test_tensorflow` just appeared because of tensorflow's update.
Once it's fixed on master we'll be free to merge this one

It's fixed on master now :)

merging this once

This is again a PR instead of #1339, because something went wrong there.


Re-added wiki_movies dataset due to previous PR having changes from m…

…any other unassociated files.


Add peer-read dataset

> Cool thank you !
>
> I left a few comments

Thank you @lhoestq  addressed your comments. Haven't changed the code but I see that tests are failing now. Do I need to rebase or something?

The CI error is not related to your dataset and is fixed on master.
You can ignore it


Added Times of India News Headlines Dataset

@lhoestq @abhishekkrthakur what happened here ?

@tanmoyio please have patience. @lhoestq has to look at 150+ PRs and it may take time. The PR looks good to me but we wait for his confirmation :) 🤗

Dataset name: Times of India News Headlines
link: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/DPQMQH


Adding medical database chinese and english

Let me know it that helps !
Also feel free to ping me if you have other questions or if I can help you.

Now I am getting an Assertion Error!
![image](https://user-images.githubusercontent.com/16264631/101943915-f5bf5600-3c11-11eb-84e5-045bbc472162.png)

All tests have passed. However, PyTest is still failing with the `AssertionError` as before. Also _datasets_info.json_ actually does not seem to provide much info. Please review and let me know what has to be improved.

Thanks!

[PR-1503](https://github.com/huggingface/datasets/pull/1503) on the COVID dialog dataset from the same University has similar features. I kept it separate because it is only on COVID qa. Also consists of only single files per language. Kindly let me know if it has to be added as two more configurations to this existing dataset itself. If it has to be added, then can I still freeze the file names like in PR-1503?

Error in creating dummy dataset


Fix ADD_NEW_DATASET to avoid rebasing once pushed


Adding the Mac-Morpho dataset

Adding the Mac-Morpho dataset, a Portuguese language dataset for Part-of-speech tagging tasks


Add narrativeQA

@lhoestq this is now only failing some random windows test (it appears to be somewhere in wnut_17)

This is a connection error, you can ignore it :)
The level of activity on the lib is quite overwhelming, it stresses a bit the CI ^^

Redo of #1368 #309 #499

In redoing the dummy data a few times, I ended up adding a load of files to git. Hopefully this should work.


Inconsistent argument names.

Also for the `Accuracy` metric the `accuracy_score` method should have its args in the opposite order so `accuracy_score(predictions, references,,,)`.

Thanks for pointing this out ! 🕵🏻
Predictions and references should indeed be swapped in the docstring.
However, the call to `accuracy_score` should not be changed, it [signature](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score) being:
```
sklearn.metrics.accuracy_score(y_true, y_pred, *, normalize=True, sample_weight=None)
```

Feel free to open a PR if you want to fix this :)

Just find it a wee bit odd that in the transformers library `predictions` are those made by the model:
https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_utils.py#L51-L61

While in many datasets metrics they are the ground truth labels:
https://github.com/huggingface/datasets/blob/c3f53792a744ede18d748a1133b6597fdd2d8d18/metrics/accuracy/accuracy.py#L31-L40

Do you think predictions & references should be swapped? I'd be willing to do some refactoring here if you agree.


Jigsaw toxicity pred

Managed to mess up my original pull request, opening a fresh one incorporating the changes suggested by @lhoestq.


Add Spanish Billion Words Corpus

Add an unannotated Spanish corpus of nearly 1.5 billion words, compiled from different resources from the web.


Fix XML iterparse in opus_dogc dataset

I forgot to add `elem.clear()` to clear the element from memory.


Create JSON dummy data without loading all dataset in memory

See #1442.

The statement `json.load()` loads **all the file content in memory**.

In order to avoid this, file content should be parsed **iteratively**, by using the library `ijson` e.g.

I have refactorized the code into a function `_create_json_dummy_data` and I have added some tests.


add srwac

Connection error failed. Need rerun
merging since the CI is fixed on master


add Srwac


Adding the HAREM dataset

Thanks for the changes !

Sorry if I wasn't clear about the suggestion of adding the `raw` dataset as well.
By `raw` I meant the dataset with its original features, i.e. not tokenized to follow the conll format for NER.
The `raw` dataset has data fields `doc_text`, `doc_id` and `entities`.

Alright, @lhoestq, now I understand your suggestion, but the JSON files the script downloads aren't actually the "raw" HAREM dataset, the [real raw version](https://www.linguateca.pt/primeiroHAREM/harem/ColeccaoDouradaHAREM.zip) of it is on XML format that needs a lot of preprocessing. Those JSON are just pre-processed versions of the dataset.

I can make the config of the raw version of the pre-processed dataset, but I'll leave a comment on the dataset summary about those details.

Oh I see ! Sorry I thought there really were the raw (i.e. original/unprocessed) files

The original xml format doesn't seem very practical to use :/
We can ignore that and only keep the default and selective configs then.
Can you revert to the old configs ?

Thanks again for adding this dataset !

Hi @lhoestq, I've reverted the changes and put more details on README about the dataset.

Adding the HAREM dataset, a Portuguese language dataset for NER tasks


Add wiki lingua dataset

it’s failing because of `RemoteDatasetTest.test_load_dataset_orange_sum`
which i think is not the dataset you are doing a PR for. Try rebasing with:
```
git fetch upstream
git rebase upstream/master
git push -u -f origin your_branch
```

> it’s failing because of `RemoteDatasetTest.test_load_dataset_orange_sum`
> which i think is not the dataset you are doing a PR for. Try rebasing with:
>
> ```
> git fetch upstream
> git rebase upstream/master
> git push -u -f origin your_branch
> ```

Thanks, my branch seems to be up to date.
```Current branch add-wiki-lingua-dataset is up to date.```

Also where do the google drive urls come from ?

looks like this PR includes changes about many other files than the ones for wiki_lingua.

Can you create another branch and another PR ?
(or you can try to fix this branch with rebase and push force if you're familiar with it)

Thanks for fixing the dummy data and removing the glob call :)

> looks like this PR includes changes about many other files than the ones for wiki_lingua.
>
> Can you create another branch and another PR ?
> (or you can try to fix this branch with rebase and push force if you're familiar with it)

Easier to create a new branch and submit, I have submitted a new PR #1582

Hello @lhoestq ,

I am opening a fresh pull request as advised in my original PR https://github.com/huggingface/datasets/pull/1308

Thanks


ADD: Wino_bias dataset

Updated PR to counter messed up history of previous one (https://github.com/huggingface/datasets/pull/1235) due to rebase.
Removed manual downloading of dataset.


add Indonesian newspapers (id_newspapers_2018)

Looks like there's a `Path` issue on windows. Could you try switching to
`glob.glob(os.path.join(article_dir, "*.json"))`

> Looks like there's a `Path` issue on windows. Could you try switching to
> `glob.glob(os.path.join(article_dir, "*.json"))`

Thanks, I replaced it with glob. Let's see if it solves the issue. Anyway, the main directory has a space, could it make the issue on windows? the test on linux don't have this problem.

It seems glob doesn't help also. Btw, one of the failing test tried to connect aws which failed:
```
C:\tools\miniconda3\lib\site-packages\urllib3\connection.py:160:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

address = ('s3.amazonaws.com', 443), timeout = 10, source_address = None
socket_options = [(6, 1, 1)]

```
Why did it try to connect to aws? I don't use it.

It seems that the circleci make a test for whole datasets repository, that means if only one of the dataset in the official repository has a download issue, this will also affect the test of a new dataset like mine, isn't it?
I changed the url to my newspaper dataset which contains only few simple json files and simple directory structure. But it still failed. And it failed not only on windows test. This is one of the error message:
```
-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_chr_en
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_chr_en
===== 4 failed, 2667 passed, 2052 skipped, 4 warnings in 432.05s (0:07:12) =====

Exited with code exit status 1
CircleCI received exit code 1
```
The test failed on twitter dataset even my dataset has nothing to do with twitter?

Hi, thanks for merging the dataset. I create a new PR (#1499) since I need to update the link to the dataset.

The dataset contains around 500K articles (136M of words) from 7 Indonesian newspapers. The size of uncompressed 500K json files (newspapers-json.tgz) is around 2.2GB.


adding snow_simplified_japanese_corpus

merging since the CI is fixed on master
Thank you for the updates and merging!

Adding simplified Japanese corpus "SNOW T15" and "SNOW T23".
They contain original Japanese, simplified Japanese, and original English (the original text is gotten from en-ja translation corpus). Hence, it can be used not only for Japanese simplification but also for en-ja translation.

- http://www.jnlp.org/SNOW/T15
- http://www.jnlp.org/SNOW/T23


Add Turkish News Category Dataset (270K).Updates were made for review…

@SBrandeis, What exactly is it that makes the tests fail? Can you help me please?

These errors
```
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_chr_en
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_chr_en
```
appeared on master 3 hours ago and are now fixed.
(it was due to today's update of `xlrd` that broke two datasets)

You can ignore them, they're not related to your dataset

> These errors
>
> ```
> =========================== short test summary info ============================
> FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ajgt_twitter_ar
> FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_chr_en
> FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_ajgt_twitter_ar
> FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_chr_en
> ```
>
> appeared on master 3 hours ago and are now fixed.
> (it was due to today's update of `xlrd` that broke two datasets)
>
> You can ignore them, they're not related to your dataset

**If it is not a problem caused by us, we have already completed the review notes. You can then check it out and confirm?**

This PR adds the **Turkish News Categories Dataset (270K)** dataset which is a text classification dataset by me and @yavuzKomecoglu. Turkish news dataset consisting of **273601 news in 17 categories**, compiled from printed media and news websites between 2010 and 2017 by the [Interpress](https://www.interpress.com/) media monitoring company.

**Note**: Resubmitted as a clean version of the previous Pull Request(#1419).  @SBrandeis @lhoestq


Add clean menyo20k data

New Clean PR for menyo20k_mt


Reddit jokes

@lhoestq would you please rerun the test,

I re-started the test.

@lhoestq let's hold off on merging for now though, having a conversation on Slack about some of the offensive content in the dataset and how/whether we want to present it.

196k Reddit Jokes dataset
Dataset link- https://raw.githubusercontent.com/taivop/joke-dataset/master/reddit_jokes.json


Adding enriched_web_nlg features + handling xml bugs

This PR adds features of the enriched_web_nlg dataset that were not present yet (most notably sorted rdf triplet sets), and deals with some xml issues that led to returning no data in cases where surgery could be performed to salvage it.


Added conv ai 2 (Again)

Looking perfect to me, need to rerun the tests

Thanks, @tanmoyio.
How do I rerun the tests? Should I change something or push a new commit?

@rkc007 you don't need to rerun it, @lhoestq @yjernite will rerun it, as there are huge number of PRs in the queue it might take lil bit of time.

Thank you @abhishekkrthakur. Can you please rerun it again? It seems something was broken in CI during the previous test.

@lhoestq Sorry for the mess. I don't know why this keeps on happening. I tried step by step process of updating the PR but seems something is wrong. This happened for 2nd time with the same PR. Apologies for that.

New PR -> https://github.com/huggingface/datasets/pull/1527
Also, I fixed everything in the new PR.

The original PR -> https://github.com/huggingface/datasets/pull/1383

Reason for creating again -

The reason I had to create the PR again was due to the master rebasing issue. After rebasing the changes, all the previous commits got added to the branch.