AniketTec/github-issues huggingface dataset:
Title: Adding english plaintext jokes dataset
State: closed

This PR adds a dataset of 200k English plaintext Jokes from three sources: Reddit, Stupidstuff, and Wocka.
Link: https://github.com/taivop/joke-dataset 

This is my second PR. 
First was: [#1269 ](https://github.com/huggingface/datasets/pull/1269)

Comments:
Title: Add FreebaseQA dataset
State: closed

This PR adds the FreebaseQA dataset: A Trivia-type QA Data Set over the Freebase Knowledge Graph

Repo: https://github.com/kelvin-jiang/FreebaseQA

Paper: https://www.aclweb.org/anthology/N19-1028.pdf


## TODO: create dummy data

Error encountered when running `python datasets-cli dummy_data datasets/freebase_qa --auto_generate`
```
    f"Couldn't parse columns {list(json_data.keys())}. "
ValueError: Couldn't parse columns ['Dataset', 'Version', 'Questions']. Maybe specify which json field must be used to read the data with --json_field <my_field>.
```


Comments:
Title: Ar cov19
State: closed

Adding ArCOV-19 dataset. ArCOV-19 is an Arabic COVID-19 Twitter dataset that covers the period from 27th of January till 30th of April 2020. ArCOV-19 is the first publicly-available Arabic Twitter dataset covering COVID-19 pandemic that includes over 1M tweets alongside the propagation networks of the most-popular subset of them (i.e., most-retweeted and-liked). The propagation networks include both retweets and conversational threads (i.e., threads of replies). ArCOV-19 is designed to enable research under several domains including natural language processing, information retrieval, and social computing, among others. 

Comments:
Title: Add Turkish News Category Dataset (270K)
State: closed

This PR adds the Turkish News Categories Dataset (270K) dataset which is a text classification dataset by me and @yavuzKomecoglu.  Turkish news dataset consisting of **273601 news** in **17 categories**, compiled from printed media and news websites between 2010 and 2017 by the [Interpress](https://www.interpress.com/) media monitoring company.



Comments:
Title: Add OffComBR
State: closed

Add [OffComBR](https://github.com/rogersdepelle/OffComBR) from [Offensive Comments in the Brazilian Web: a dataset and baseline results](https://sol.sbc.org.br/index.php/brasnam/article/view/3260/3222) paper.

But I'm having a hard time generating dummy data since the original dataset extion is `.arff` and the [_create_dummy_data function](https://github.com/huggingface/datasets/blob/a4aeaf911240057286a01bff1b1d75a89aedd57b/src/datasets/commands/dummy_data.py#L185) doesn't allow it.

Comments:
Title: Add script_version suggestion when dataset/metric not found
State: closed

Adds a helpful prompt to the error message when a dataset/metric is not found, suggesting the user might need to pass `script_version="master"` if the dataset was added recently. The whole error looks like:

> Couldn't find file locally at blah/blah.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1/metrics/blah/blah.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/met
rics/blah/blah.py.
If the dataset was added recently, you may need to to pass script_version="master" to find the loading script on the master branch.

Comments:
Title: add best2009
State: closed

`best2009` is a Thai word-tokenization dataset from encyclopedia, novels, news and articles by [NECTEC](https://www.nectec.or.th/) (148,995/2,252 lines of train/test). It was created for [BEST 2010: Word Tokenization Competition](https://thailang.nectec.or.th/archive/indexa290.html?q=node/10). The test set answers are not provided publicly.

Comments:
Title: Use passed --cache_dir for modules cache
State: open

When passed `--cache_dir` arg:
```shell
python datasets-cli test datasets/<my-dataset-folder> --save_infos --all_configs --cache_dir <my-cache-dir>
```
it is not used for caching the modules, which are cached in the default location at `.cache/huggingface/modules`.

With this fix, the modules will be cached at `<my-cache-dir>/modules`.

Comments:
Title: Add Mkqa dataset
State: closed

# MKQA: Multilingual Knowledge Questions & Answers Dataset
Adding the [MKQA](https://github.com/apple/ml-mkqa) dataset as part of the sprint 🎉

There is no official data splits so I added just a `train` split.
 
differently from the original:
- answer:type field is a ClassLabel (I thought it might be possible to train on this as a label for categorizing questions)
- answer:entity field has a default value of empty string '' (since this key is not available for all in original)
- answer:alias has default value of []

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)


Comments:
Title: add wisesight1000
State: closed

`wisesight1000` contains Thai social media texts randomly drawn from the full `wisesight-sentiment`, tokenized by human annotators. Out of the labels `neg` (negative), `neu` (neutral), `pos` (positive), `q` (question), 250 samples each. Some texts are removed because they look like spam.Because these samples are representative of real world content, we believe having these annotaed samples will allow the community to robustly evaluate tokenization algorithms.

Comments:
Title: Add spider dataset
State: closed

This PR adds the Spider dataset, a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 Yale students. The goal of the Spider challenge is to develop natural language interfaces to cross-domain databases.

Dataset website: https://yale-lily.github.io/spider
Paper link: https://www.aclweb.org/anthology/D18-1425/

Comments:
Title: Add spanish billion words
State: closed

Add an unannotated corpus of the Spanish language of nearly 1.5 billion words, compiled from different resources from the web.

The dataset needs 10 GB (download: 1.89 GiB, generated: 8.34 GiB, post-processed: Unknown size, total: 10.22 GiB), the test using dummy data pass but my laptop isn't able to run it on the real data (I left it running for over 8 hours and it didn't finish).

Comments:
Title: Added the NewsPH Raw dataset and corresponding dataset card
State: closed

This PR adds the original NewsPH dataset which is used to autogenerate the NewsPH-NLI dataset. Reopened a new PR as the previous one had problems.

Paper: https://arxiv.org/abs/2010.11574
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

Comments:
Title: ❓ Sharing ElasticSearch indexed dataset 
State: open
Labels: description: Requesting to add a new dataset
name: dataset request

Hi there,

First of all, thank you very much for this amazing library. Datasets have become my preferred data structure for basically everything I am currently doing.

**Question:** I'm working with a dataset and I have an elasticsearch container running at localhost:9200. I added an elasticsearch index and I was wondering

- how can I know where it has been saved?

- how can I share the indexed dataset with others?

I tried to dig into the docs, but could not find anything about that.

Thank you very much for your help.

Best,
Pietro

Edit: apologies for the wrong label

Comments:
Title: Add CC-News dataset of English language articles
State: closed

Adds [CC-News](https://commoncrawl.org/2016/10/news-dataset-available/) dataset. It contains 708241 English language news articles. Although each article has a language field these tags are not reliable. I've used Spacy language detection [pipeline](https://spacy.io/universe/project/spacy-langdetect) to confirm that the article language is indeed English. 

The prepared dataset is temporarily hosted on my private Google Storage [bucket](https://storage.googleapis.com/hf_datasets/cc_news.tar.gz). We can move it to HF storage and update this PR before merging. 

Comments:
Title: Added the WikiText-TL39 dataset and corresponding card
State: closed

This PR adds the WikiText-TL-39 Filipino Language Modeling dataset. Restarted a new pull request since there were problems with the earlier one.

Paper: https://arxiv.org/abs/1907.00409
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

Comments:
Title: ethos first commit
State: closed

Ethos passed all the tests except from this one: 
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<your-dataset-name>

with this error: 
E               OSError: Cannot find data file. 
E               Original error:
E               [Errno 2] No such file or directory: 

Comments:
Title: Allow GitHub releases as dataset source
State: closed

# Summary

Providing a GitHub release URL to `DownloadManager.download()` currently throws a `ConnectionError: Couldn't reach [DOWNLOAD_URL]`. This PR fixes this problem by adding an exception for GitHub releases in `datasets.utils.file_utils.get_from_cache()`.

# Reproduce

```
import datasets
url = 'http://github.com/benjaminvdb/DBRD/releases/download/v3.0/DBRD_v3.tgz'
result = datasets.utils.file_utils.get_from_cache(url)

# Returns: ConnectionError: Couldn't reach http://github.com/benjaminvdb/DBRD/releases/download/v3.0/DBRD_v3.tgz
```

# Cause

GitHub releases returns a HTTP status 403 (FOUND), indicating that the request is being redirected (to AWS S3, in this case). `get_from_cache()` checks whether the status is 200 (OK) or if it is part of two exceptions (Google Drive or Firebase), otherwise the mentioned error is thrown.

# Solution

Just like the exceptions for Google Drive and Firebase, add a condition for GitHub releases URLs that return the HTTP status 403. If this is the case, continue normally.

Comments:
Title: Add HateSpeech Corpus for Polish
State: closed

This PR adds a HateSpeech Corpus for Polish, containing offensive language examples.

- **Homepage:** http://zil.ipipan.waw.pl/HateSpeech
- **Paper:** http://www.qualitativesociologyreview.org/PL/Volume38/PSJ_13_2_Troszynski_Wawer.pdf

Comments:
Title: Add OffensEval-TR 2020 Dataset
State: closed

This PR adds the OffensEval-TR 2020 dataset which is a Turkish offensive language corpus by me and @basakbuluz. The corpus consist of randomly sampled tweets and annotated in a similar way to [OffensEval](https://sites.google.com/site/offensevalsharedtask/) and [GermEval](https://projects.fzai.h-da.de/iggsa/).

- **Homepage:** [offensive-turkish](https://coltekin.github.io/offensive-turkish/)
- **Paper:** [A Corpus of Turkish Offensive Language on Social Media](https://coltekin.github.io/offensive-turkish/troff.pdf)
- **Point of Contact:** [Çağrı Çöltekin](ccoltekin@sfs.uni-tuebingen.de)

Comments:
Title: Add Wiki Lingua Dataset
State: closed

Hello,

This is my first PR. 

I have added Wiki Lingua Dataset along with dataset card to the best of my knowledge.
There was one hiccup though. I was unable to create dummy data because the data is in pkl format.
From the document, I see that:
```At the moment it supports data files in the following format: txt, csv, tsv, jsonl, json, xml```



Comments:
Title: add W&I + LOCNESS dataset (BEA-2019 workshop shared task on GEC)
State: closed

- **Name:** W&I + LOCNESS dataset (from the BEA-2019 workshop shared task on GEC)
- **Description:** https://www.cl.cam.ac.uk/research/nl/bea2019st/#data
- **Paper:** https://www.aclweb.org/anthology/W19-4406/
- **Motivation:** This is a recent dataset (actually two in one) for grammatical error correction and is used for benchmarking in this field of NLP.

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


Comments:
Title: can't load "german_legal_entity_recognition" dataset
State: closed

FileNotFoundError: Couldn't find file locally at german_legal_entity_recognition/german_legal_entity_recognition.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py


Comments:
Title: imdb dataset cannot be downloaded
State: closed

hi
please find error below getting imdb train spli:
thanks

`
datasets.load_dataset>>> datasets.load_dataset("imdb", split="train")`


errors


```
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
Downloading and preparing dataset imdb/plain_text (download: 80.23 MiB, generated: 127.06 MiB, post-processed: Unknown size, total: 207.28 MiB) to /idiap/temp/rkarimi/cache_home_1/datasets/imdb/plain_text/1.0.0/90099cb476936b753383ba2ae6ab2eae419b2e87f71cd5189cb9c8e5814d12a3...
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets/downloads
Traceback (most recent call last):        
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 558, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 73, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='unsupervised', num_bytes=67125548, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=7486451, num_examples=5628, dataset_name='imdb')}]


```

Comments:
Title: 'iwslt2017-ro-nl', cannot be downloaded 
State: closed
Labels: description: A bug in a dataset script provided in the library
name: dataset bug

Hi
I am trying 
`>>> datasets.load_dataset("iwslt2017", 'iwslt2017-ro-nl', split="train")`

getting this error thank you for your help
```
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
Downloading and preparing dataset iwsl_t217/iwslt2017-ro-nl (download: 314.07 MiB, generated: 39.92 MiB, post-processed: Unknown size, total: 354.00 MiB) to /idiap/temp/rkarimi/cache_home_1/datasets/iwsl_t217/iwslt2017-ro-nl/1.0.0/cca6935a0851a8ceac1202a62c958738bdfa23c57a51bc52ac1c5ebd2aa172cd...
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets/downloads
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/iwslt2017/cca6935a0851a8ceac1202a62c958738bdfa23c57a51bc52ac1c5ebd2aa172cd/iwslt2017.py", line 118, in _split_generators
    dl_dir = dl_manager.download_and_extract(MULTI_URL)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 254, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 179, in download
    num_proc=download_config.num_proc,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 216, in map_nested
    return function(data_struct)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 477, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://wit3.fbk.eu/archive/2017-01-trnmted//texts/DeEnItNlRo/DeEnItNlRo/DeEnItNlRo-DeEnItNlRo.tgz

```

Comments:
Title: [libprotobuf FATAL /sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:1505] CHECK failed: (index) >= (0):  terminate called after throwing an instance of 'google::protobuf::FatalException'   what():  CHECK failed: (index) >= (0):  Aborted
State: closed

Hi
I am getting this error when evaluating on wmt16-ro-en using finetune_trainer.py of huggingface repo. thank for your help

{'epoch': 20.0}                                                                                                                                             
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:16<00:00,  1.22it/s]
12/08/2020 10:41:19 - INFO - seq2seq.trainers.trainer -   Saving model checkpoint to outputs/experiment/joint/finetune/lr-2e-5
12/08/2020 10:41:24 - INFO - __main__ -   {'wmt16-en-ro': Dataset(features: {'src_texts': Value(dtype='string', id=None), 'task': Value(dtype='string', id=None), 'tgt_texts': Value(dtype='string', id=None)}, num_rows: 1998), 'qnli': Dataset(features: {'src_texts': Value(dtype='string', id=None), 'task': Value(dtype='string', id=None), 'tgt_texts': Value(dtype='string', id=None)}, num_rows: 5462), 'scitail': Dataset(features: {'src_texts': Value(dtype='string', id=None), 'task': Value(dtype='string', id=None), 'tgt_texts': Value(dtype='string', id=None)}, num_rows: 1303)}
12/08/2020 10:41:24 - INFO - __main__ -   *** Evaluate ***
12/08/2020 10:41:24 - INFO - seq2seq.utils.utils -   using task specific params for wmt16-en-ro: {'max_length': 300, 'num_beams': 4}
12/08/2020 10:41:24 - INFO - seq2seq.trainers.trainer -   ***** Running Evaluation *****
12/08/2020 10:41:24 - INFO - seq2seq.trainers.trainer -     Num examples = 1998
12/08/2020 10:41:24 - INFO - seq2seq.trainers.trainer -     Batch size = 64
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:37<00:00,  1.19s/it][libprotobuf FATAL /sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:1505] CHECK failed: (index) >= (0): 
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: (index) >= (0): 
Aborted


Comments:
Title: boolq does not work 
State: closed

Hi
I am getting this error when trying to load boolq, thanks for your help

ts_boolq_default_0.1.0_2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11.lock
Traceback (most recent call last):
  File "finetune_t5_trainer.py", line 274, in <module>
    main()
  File "finetune_t5_trainer.py", line 147, in main
    for task in data_args.tasks]
  File "finetune_t5_trainer.py", line 147, in <listcomp>
    for task in data_args.tasks]
  File "/remote/idiap.svm/user.active/rkarimi/dev/ruse/seq2seq/tasks/tasks.py", line 58, in get_dataset
    dataset = self.load_dataset(split=split)
  File "/remote/idiap.svm/user.active/rkarimi/dev/ruse/seq2seq/tasks/tasks.py", line 54, in load_dataset
    return datasets.load_dataset(self.task.name, split=split)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/boolq/2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11/boolq.py", line 74, in _split_generators
    downloaded_files = dl_manager.download_custom(urls_to_download, tf.io.gfile.copy)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 149, in download_custom
    custom_download(url, path)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/tensorflow/python/lib/io/file_io.py", line 516, in copy_v2
    compat.path_to_bytes(src), compat.path_to_bytes(dst), overwrite)
tensorflow.python.framework.errors_impl.AlreadyExistsError: file already exists



Comments:
Title: Add dutch book review dataset
State: closed

- Name: Dutch Book Review Dataset (DBRD)
- Description: The DBRD (pronounced dee-bird) dataset contains over 110k book reviews along with associated binary sentiment polarity labels and is intended as a benchmark for sentiment classification in Dutch.
- Paper: https://arxiv.org/abs/1910.00896
- Data: https://github.com/benjaminvdb/DBRD
- Motivation: A large (real-life) dataset of Dutch book reviews and sentiment polarity (positive/negative), based on the associated rating.

Checks
- [x] Create the dataset script /datasets/dbrd/dbrd.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _info(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.

Comments:
Title: add thaiqa_squad
State: closed

Example format is a little different from SQuAD since `thaiqa` always have one answer per question so I added a check to convert answers to lists if they are not already one to future-proof additional questions that might have multiple answers.

`thaiqa_squad` is an open-domain, extractive question answering dataset (4,000 questions in `train` and 74 questions in `dev`) in [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) format, originally created by [NECTEC](https://www.nectec.or.th/en/) from Wikipedia articles and adapted to [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) format by [PyThaiNLP](https://github.com/PyThaiNLP/).

Comments: