jamesLeeeeeee/datasets-github-issues huggingface dataset:
connection issue while downloading data
Author: rabeehkarimimahabadi | Pull request: no | State: closed

Hi
I am running my codes on google cloud, and I am getting this error resulting in the failure of the codes when trying to download the data, could you assist me to solve this? also as a temporary solution, could you tell me how I can increase the number of retries and timeout to at least let the models run for now. thanks 

```
Traceback (most recent call last):
  File "finetune_t5_trainer.py", line 361, in <module>
    main()
  File "finetune_t5_trainer.py", line 269, in main
    add_prefix=False if training_args.train_adapters else True)
  File "/workdir/seq2seq/data/tasks.py", line 70, in get_dataset
    dataset = self.load_dataset(split=split)
  File "/workdir/seq2seq/data/tasks.py", line 306, in load_dataset
    return datasets.load_dataset('glue', 'cola', split=split)
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 589, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 263, in prepare_module
    head_hf_s3(path, filename=name, dataset=dataset)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py", line 200, in head_hf_s3
    return http_head(hf_bucket_url(identifier=identifier, filename=filename, use_cdn=use_cdn, dataset=dataset))
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py", line 403, in http_head
    url, proxies=proxies, headers=headers, cookies=cookies, allow_redirects=allow_redirects, timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 104, in head
    return request('head', url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 504, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /datasets.huggingface.co/datasets/datasets/glue/glue.py (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f47db511e80>, 'Connection to s3.amazonaws.com timed out. (connect timeout=10)'))
```
added TTC4900: A Benchmark Data for Turkish Text Categorization dataset
Author: yavuzKomecoglu | Pull request: yes | State: closed

This PR adds the TTC4900 dataset which is a Turkish Text Categorization dataset by me and @basakbuluz. 

Homepage: [https://www.kaggle.com/savasy/ttc4900](https://www.kaggle.com/savasy/ttc4900)
Point of Contact: [Savaş Yıldırım](mailto:savasy@gmail.com) / @savasy
Add : Conv AI 2 (Messed up original PR)
Author: rkc007 | Pull request: yes | State: closed

@lhoestq Sorry I messed up the previous 2 PR's -> https://github.com/huggingface/datasets/pull/1462 -> https://github.com/huggingface/datasets/pull/1383. So created a new one. Also, everything is fixed in this PR. Can you please review it ?
Thanks in advance.
Inconsistent argument names.
Author: Fraser-Greenlee | Pull request: no | State: closed

Just find it a wee bit odd that in the transformers library `predictions` are those made by the model:
https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_utils.py#L51-L61

While in many datasets metrics they are the ground truth labels:
https://github.com/huggingface/datasets/blob/c3f53792a744ede18d748a1133b6597fdd2d8d18/metrics/accuracy/accuracy.py#L31-L40

Do you think predictions & references should be swapped? I'd be willing to do some refactoring here if you agree.
Create JSON dummy data without loading all dataset in memory
Author: albertvillanova | Pull request: yes | State: open

See #1442.

The statement `json.load()` loads **all the file content in memory**.

In order to avoid this, file content should be parsed **iteratively**, by using the library `ijson` e.g.

I have refactorized the code into a function `_create_json_dummy_data` and I have added some tests.
adding snow_simplified_japanese_corpus
Author: forest1988 | Pull request: yes | State: closed

Adding simplified Japanese corpus "SNOW T15" and "SNOW T23".
They contain original Japanese, simplified Japanese, and original English (the original text is gotten from en-ja translation corpus). Hence, it can be used not only for Japanese simplification but also for en-ja translation.

- http://www.jnlp.org/SNOW/T15
- http://www.jnlp.org/SNOW/T23
Add Turkish News Category Dataset (270K).Updates were made for review…
Author: basakbuluz | Pull request: yes | State: closed

This PR adds the **Turkish News Categories Dataset (270K)** dataset which is a text classification dataset by me and @yavuzKomecoglu. Turkish news dataset consisting of **273601 news in 17 categories**, compiled from printed media and news websites between 2010 and 2017 by the [Interpress](https://www.interpress.com/) media monitoring company.

**Note**: Resubmitted as a clean version of the previous Pull Request(#1419).  @SBrandeis @lhoestq
Add id_nergrit_corpus
Author: cahya-wirawan | Pull request: yes | State: closed

Nergrit Corpus is a dataset collection of Indonesian Named Entity Recognition, Statement Extraction, and Sentiment Analysis. 
Recently my PR for id_nergrit_ner has been accepted and merged to the main branch. The id_nergrit_ner has only one dataset (NER), and this new PR renamed the dataset from id_nergrit_ner to id_nergrit_corpus and added 2 other remaining datasets (Statement Extraction, and Sentiment Analysis.)
SNLI dataset contains labels with value -1
Author: aarnetalman | Pull request: no | State: closed

```
import datasets
nli_data = datasets.load_dataset("snli")
train_data = nli_data['train']
train_labels = train_data['label']
label_set = set(train_labels)
print(label_set)
```

**Output:**
`{0, 1, 2, -1}`
add W&I + LOCNESS dataset (BEA-2019 workshop shared task on GEC) [PROPER]
Author: aseifert | Pull request: yes | State: closed

- **Name:** W&I + LOCNESS dataset (from the BEA-2019 workshop shared task on GEC)
- **Description:** https://www.cl.cam.ac.uk/research/nl/bea2019st/#data
- **Paper:** https://www.aclweb.org/anthology/W19-4406/
- **Motivation:** This is a recent dataset (actually two in one) for grammatical error correction and is used for benchmarking in this field of NLP.

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.
add thai_toxicity_tweet
Author: cstorm125 | Pull request: yes | State: closed

Thai Toxicity Tweet Corpus contains 3,300 tweets (506 tweets with texts missing) annotated by humans with guidelines including a 44-word dictionary. The author obtained 2,027 and 1,273 toxic and non-toxic tweets, respectively; these were labeled by three annotators. The result of corpus analysis indicates that tweets that include toxic words are not always toxic. Further, it is more likely that a tweet is toxic, if it contains toxic words indicating their original meaning. Moreover, disagreements in annotation are primarily because of sarcasm, unclear existing target, and word sense ambiguity.

Notes from data cleaner: The data is included into [huggingface/datasets](https://www.github.com/huggingface/datasets) in Dec 2020. By this time, 506 of the tweets are not available publicly anymore. We denote these by `TWEET_NOT_FOUND` in `tweet_text`.
Processing can be found at [this PR](https://github.com/tmu-nlp/ThaiToxicityTweetCorpus/pull/1).
FileNotFound remotly, can't load a dataset
Author: sadakmed | Pull request: no | State: closed

```py
!pip install datasets
import datasets as ds

corpus = ds.load_dataset('large_spanish_corpus')
```
gives the error

> FileNotFoundError: Couldn't find file locally at large_spanish_corpus/large_spanish_corpus.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/large_spanish_corpus/large_spanish_corpus.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/large_spanish_corpus/large_spanish_corpus.py

not just `large_spanish_corpus`,  `zest` too, but `squad` is available. 

this was using colab and localy
Create XML dummy data without loading all dataset in memory
Author: albertvillanova | Pull request: yes | State: closed

While I was adding one XML dataset, I noticed that all the dataset was loaded in memory during the dummy data generation process (using nearly all my laptop RAM).

Looking at the code, I have found that the origin is the use of `ET.parse()`. This method loads **all the file content in memory**.

In order to fix this, I have refactorized the code and use `ET.iterparse()` instead, which **parses the file content incrementally**.

I have also implemented a test.
Adding english plaintext jokes dataset
Author: purvimisal | Pull request: yes | State: closed

This PR adds a dataset of 200k English plaintext Jokes from three sources: Reddit, Stupidstuff, and Wocka.
Link: https://github.com/taivop/joke-dataset 

This is my second PR. 
First was: [#1269 ](https://github.com/huggingface/datasets/pull/1269)
Add FreebaseQA dataset
Author: anaerobeth | Pull request: yes | State: closed

This PR adds the FreebaseQA dataset: A Trivia-type QA Data Set over the Freebase Knowledge Graph

Repo: https://github.com/kelvin-jiang/FreebaseQA

Paper: https://www.aclweb.org/anthology/N19-1028.pdf


## TODO: create dummy data

Error encountered when running `python datasets-cli dummy_data datasets/freebase_qa --auto_generate`
```
    f"Couldn't parse columns {list(json_data.keys())}. "
ValueError: Couldn't parse columns ['Dataset', 'Version', 'Questions']. Maybe specify which json field must be used to read the data with --json_field <my_field>.
```
Ar cov19
Author: Fatima-Haouari | Pull request: yes | State: closed

Adding ArCOV-19 dataset. ArCOV-19 is an Arabic COVID-19 Twitter dataset that covers the period from 27th of January till 30th of April 2020. ArCOV-19 is the first publicly-available Arabic Twitter dataset covering COVID-19 pandemic that includes over 1M tweets alongside the propagation networks of the most-popular subset of them (i.e., most-retweeted and-liked). The propagation networks include both retweets and conversational threads (i.e., threads of replies). ArCOV-19 is designed to enable research under several domains including natural language processing, information retrieval, and social computing, among others.
Add Turkish News Category Dataset (270K)
Author: basakbuluz | Pull request: yes | State: closed

This PR adds the Turkish News Categories Dataset (270K) dataset which is a text classification dataset by me and @yavuzKomecoglu.  Turkish news dataset consisting of **273601 news** in **17 categories**, compiled from printed media and news websites between 2010 and 2017 by the [Interpress](https://www.interpress.com/) media monitoring company.
Add OffComBR
Author: hugoabonizio | Pull request: yes | State: closed

Add [OffComBR](https://github.com/rogersdepelle/OffComBR) from [Offensive Comments in the Brazilian Web: a dataset and baseline results](https://sol.sbc.org.br/index.php/brasnam/article/view/3260/3222) paper.

But I'm having a hard time generating dummy data since the original dataset extion is `.arff` and the [_create_dummy_data function](https://github.com/huggingface/datasets/blob/a4aeaf911240057286a01bff1b1d75a89aedd57b/src/datasets/commands/dummy_data.py#L185) doesn't allow it.
Add script_version suggestion when dataset/metric not found
Author: joeddav | Pull request: yes | State: closed

Adds a helpful prompt to the error message when a dataset/metric is not found, suggesting the user might need to pass `script_version="master"` if the dataset was added recently. The whole error looks like:

> Couldn't find file locally at blah/blah.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1/metrics/blah/blah.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/met
rics/blah/blah.py.
If the dataset was added recently, you may need to to pass script_version="master" to find the loading script on the master branch.
add best2009
Author: cstorm125 | Pull request: yes | State: closed

`best2009` is a Thai word-tokenization dataset from encyclopedia, novels, news and articles by [NECTEC](https://www.nectec.or.th/) (148,995/2,252 lines of train/test). It was created for [BEST 2010: Word Tokenization Competition](https://thailang.nectec.or.th/archive/indexa290.html?q=node/10). The test set answers are not provided publicly.
Use passed --cache_dir for modules cache
Author: albertvillanova | Pull request: yes | State: open

When passed `--cache_dir` arg:
```shell
python datasets-cli test datasets/<my-dataset-folder> --save_infos --all_configs --cache_dir <my-cache-dir>
```
it is not used for caching the modules, which are cached in the default location at `.cache/huggingface/modules`.

With this fix, the modules will be cached at `<my-cache-dir>/modules`.
Add Mkqa dataset
Author: cceyda | Pull request: yes | State: closed

# MKQA: Multilingual Knowledge Questions & Answers Dataset
Adding the [MKQA](https://github.com/apple/ml-mkqa) dataset as part of the sprint 🎉

There is no official data splits so I added just a `train` split.
 
differently from the original:
- answer:type field is a ClassLabel (I thought it might be possible to train on this as a label for categorizing questions)
- answer:entity field has a default value of empty string '' (since this key is not available for all in original)
- answer:alias has default value of []

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)
add wisesight1000
Author: cstorm125 | Pull request: yes | State: closed

`wisesight1000` contains Thai social media texts randomly drawn from the full `wisesight-sentiment`, tokenized by human annotators. Out of the labels `neg` (negative), `neu` (neutral), `pos` (positive), `q` (question), 250 samples each. Some texts are removed because they look like spam.Because these samples are representative of real world content, we believe having these annotaed samples will allow the community to robustly evaluate tokenization algorithms.
Add spider dataset
Author: olinguyen | Pull request: yes | State: closed

This PR adds the Spider dataset, a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 Yale students. The goal of the Spider challenge is to develop natural language interfaces to cross-domain databases.

Dataset website: https://yale-lily.github.io/spider
Paper link: https://www.aclweb.org/anthology/D18-1425/
Add spanish billion words
Author: mariagrandury | Pull request: yes | State: closed

Add an unannotated corpus of the Spanish language of nearly 1.5 billion words, compiled from different resources from the web.

The dataset needs 10 GB (download: 1.89 GiB, generated: 8.34 GiB, post-processed: Unknown size, total: 10.22 GiB), the test using dummy data pass but my laptop isn't able to run it on the real data (I left it running for over 8 hours and it didn't finish).
Added the NewsPH Raw dataset and corresponding dataset card
Author: jcblaisecruz02 | Pull request: yes | State: closed

This PR adds the original NewsPH dataset which is used to autogenerate the NewsPH-NLI dataset. Reopened a new PR as the previous one had problems.

Paper: https://arxiv.org/abs/2010.11574
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks
❓ Sharing ElasticSearch indexed dataset 
Author: pietrolesci | Pull request: no | State: open

Hi there,

First of all, thank you very much for this amazing library. Datasets have become my preferred data structure for basically everything I am currently doing.

**Question:** I'm working with a dataset and I have an elasticsearch container running at localhost:9200. I added an elasticsearch index and I was wondering

- how can I know where it has been saved?

- how can I share the indexed dataset with others?

I tried to dig into the docs, but could not find anything about that.

Thank you very much for your help.

Best,
Pietro

Edit: apologies for the wrong label
Add CC-News dataset of English language articles
Author: vblagoje | Pull request: yes | State: closed

Adds [CC-News](https://commoncrawl.org/2016/10/news-dataset-available/) dataset. It contains 708241 English language news articles. Although each article has a language field these tags are not reliable. I've used Spacy language detection [pipeline](https://spacy.io/universe/project/spacy-langdetect) to confirm that the article language is indeed English. 

The prepared dataset is temporarily hosted on my private Google Storage [bucket](https://storage.googleapis.com/hf_datasets/cc_news.tar.gz). We can move it to HF storage and update this PR before merging.
Added the WikiText-TL39 dataset and corresponding card
Author: jcblaisecruz02 | Pull request: yes | State: closed

This PR adds the WikiText-TL-39 Filipino Language Modeling dataset. Restarted a new pull request since there were problems with the earlier one.

Paper: https://arxiv.org/abs/1907.00409
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks
ethos first commit
Author: iamollas | Pull request: yes | State: closed

Ethos passed all the tests except from this one: 
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<your-dataset-name>

with this error: 
E               OSError: Cannot find data file. 
E               Original error:
E               [Errno 2] No such file or directory:
Allow GitHub releases as dataset source
Author: benjaminvdb | Pull request: yes | State: closed

# Summary

Providing a GitHub release URL to `DownloadManager.download()` currently throws a `ConnectionError: Couldn't reach [DOWNLOAD_URL]`. This PR fixes this problem by adding an exception for GitHub releases in `datasets.utils.file_utils.get_from_cache()`.

# Reproduce

```
import datasets
url = 'http://github.com/benjaminvdb/DBRD/releases/download/v3.0/DBRD_v3.tgz'
result = datasets.utils.file_utils.get_from_cache(url)

# Returns: ConnectionError: Couldn't reach http://github.com/benjaminvdb/DBRD/releases/download/v3.0/DBRD_v3.tgz
```

# Cause

GitHub releases returns a HTTP status 403 (FOUND), indicating that the request is being redirected (to AWS S3, in this case). `get_from_cache()` checks whether the status is 200 (OK) or if it is part of two exceptions (Google Drive or Firebase), otherwise the mentioned error is thrown.

# Solution

Just like the exceptions for Google Drive and Firebase, add a condition for GitHub releases URLs that return the HTTP status 403. If this is the case, continue normally.
Add HateSpeech Corpus for Polish
Author: kacperlukawski | Pull request: yes | State: closed

This PR adds a HateSpeech Corpus for Polish, containing offensive language examples.

- **Homepage:** http://zil.ipipan.waw.pl/HateSpeech
- **Paper:** http://www.qualitativesociologyreview.org/PL/Volume38/PSJ_13_2_Troszynski_Wawer.pdf
Add OffensEval-TR 2020 Dataset
Author: yavuzKomecoglu | Pull request: yes | State: closed

This PR adds the OffensEval-TR 2020 dataset which is a Turkish offensive language corpus by me and @basakbuluz. The corpus consist of randomly sampled tweets and annotated in a similar way to [OffensEval](https://sites.google.com/site/offensevalsharedtask/) and [GermEval](https://projects.fzai.h-da.de/iggsa/).

- **Homepage:** [offensive-turkish](https://coltekin.github.io/offensive-turkish/)
- **Paper:** [A Corpus of Turkish Offensive Language on Social Media](https://coltekin.github.io/offensive-turkish/troff.pdf)
- **Point of Contact:** [Çağrı Çöltekin](ccoltekin@sfs.uni-tuebingen.de)
Add Wiki Lingua Dataset
Author: katnoria | Pull request: yes | State: closed

Hello,

This is my first PR. 

I have added Wiki Lingua Dataset along with dataset card to the best of my knowledge.
There was one hiccup though. I was unable to create dummy data because the data is in pkl format.
From the document, I see that:
```At the moment it supports data files in the following format: txt, csv, tsv, jsonl, json, xml```
add W&I + LOCNESS dataset (BEA-2019 workshop shared task on GEC)
Author: aseifert | Pull request: yes | State: closed

- **Name:** W&I + LOCNESS dataset (from the BEA-2019 workshop shared task on GEC)
- **Description:** https://www.cl.cam.ac.uk/research/nl/bea2019st/#data
- **Paper:** https://www.aclweb.org/anthology/W19-4406/
- **Motivation:** This is a recent dataset (actually two in one) for grammatical error correction and is used for benchmarking in this field of NLP.

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.
can't load "german_legal_entity_recognition" dataset
Author: nataly-obr | Pull request: no | State: closed

FileNotFoundError: Couldn't find file locally at german_legal_entity_recognition/german_legal_entity_recognition.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py