helmo/github-issues huggingface dataset:
Added the NewsPH Raw dataset and corresponding dataset card
State: closed | Author: jcblaisecruz02 (NONE)

This PR adds the original NewsPH dataset which is used to autogenerate the NewsPH-NLI dataset. Reopened a new PR as the previous one had problems.

Paper: https://arxiv.org/abs/2010.11574
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks
❓ Sharing ElasticSearch indexed dataset 
State: open | Author: pietrolesci (NONE)
Labels: name: dataset request

Hi there,

First of all, thank you very much for this amazing library. Datasets have become my preferred data structure for basically everything I am currently doing.

**Question:** I'm working with a dataset and I have an elasticsearch container running at localhost:9200. I added an elasticsearch index and I was wondering

- how can I know where it has been saved?

- how can I share the indexed dataset with others?

I tried to dig into the docs, but could not find anything about that.

Thank you very much for your help.

Best,
Pietro

Edit: apologies for the wrong label

Comments: Hello @pietrolesci , I am not sure to understand what you are trying to do here.

If you're looking for ways to save a dataset on disk, you can you the `save_to_disk` method:
```python
>>> import datasets
>>> loaded_dataset = datasets.load("dataset_name")
>>> loaded_dataset.save_to_disk("/path/on/your/disk")
```

The saved dataset can later be retrieved using:
```python
>>> loaded_dataset = datasets.Dataset.load_from_disk("/path/on/your/disk")
```

Also, I'd recommend posting your question directly in the issue section of the [elasticsearch repo](https://github.com/elastic/elasticsearch)

Hi @SBrandeis,

Thanks a lot for picking up my request. 

Maybe I can clarify my use-case with a bit of context. Say I have the IMDb dataset. I create an ES index on it. Now I can save and reload the dataset from disk normally. Once I reload the dataset, it is easy to retrieve the ES index on my machine. I was wondering: is there a way I can share the (now) indexed version of the IMDb dataset with my colleagues without requiring them to re-index it?

Thanks a lot in advance for your consideration.

Best,

Pietro

Thanks for the clarification.

I am not familiar with ElasticSearch, but if I understand well you're trying to migrate your data along with the ES index.
My advice would be to check out ES documentation, for instance, this might help you: https://www.elastic.co/guide/en/cloud/current/ec-migrate-data.html

Let me know if it helps
Add CC-News dataset of English language articles
State: closed | Author: vblagoje (CONTRIBUTOR)

Adds [CC-News](https://commoncrawl.org/2016/10/news-dataset-available/) dataset. It contains 708241 English language news articles. Although each article has a language field these tags are not reliable. I've used Spacy language detection [pipeline](https://spacy.io/universe/project/spacy-langdetect) to confirm that the article language is indeed English. 

The prepared dataset is temporarily hosted on my private Google Storage [bucket](https://storage.googleapis.com/hf_datasets/cc_news.tar.gz). We can move it to HF storage and update this PR before merging. 

Comments: @vblagoje nice work, please add the README.md file and it would be ready

@lhoestq @tanmoyio @yjernite please have a look at the dataset card. Don't forget that the dataset is still hosted on my private gs bucket and should eventually be moved to the HF bucket

I will move the files soon and ping you when it's done and with the new URLs :) 

Hi !

I just moved the file to a HF bucket. It's available at https://storage.googleapis.com/huggingface-nlp/datasets/cc_news/cc_news.tar.gz

Sorry for the delay ^^'

@lhoestq no worries, updated PR with the new URL and rebased to master
added dutch_social
State: closed | Author: skyprince999 (CONTRIBUTOR)

The Dutch social media tweets dataset. Which has a total of more than 210k tweets in dutch language. These tweets have been machine annotated with sentiment scores (`label` feature) and `industry` and `hisco_codes`

It can be used for sentiment analysis, multi-label classification and entity tagging

Comments: @lhoestq 
Updated the `dummy_data.zip `(<10kb)I had to reduce it to just a few samples. 
Train-Test-Dev  (20-5-5 samples) 

But the push also added changes from other PRs (probably because of a rebase!) So the files changed tab shows 466 files were changed!  


Thanks ! The dummy data are all good now :) 

Looks like this PR includes changes to many other files than the ones for dutch_social now.

Can you create another branch and another PR please ?

> 
> Can you create another branch and another PR please ?
@lhoestq 

I did a rebase. Now it doesn't include the other files. Does that help? 



Yes thanks !
Added the WikiText-TL39 dataset and corresponding card
State: closed | Author: jcblaisecruz02 (NONE)

This PR adds the WikiText-TL-39 Filipino Language Modeling dataset. Restarted a new pull request since there were problems with the earlier one.

Paper: https://arxiv.org/abs/1907.00409
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks
adding wili-2018 language identification dataset
State: closed | Author: Shubhambindal2017 (CONTRIBUTOR)


Comments: @lhoestq Not sure what happened, I just changed the py file but it is showing some TensorFlow error now.

You can ignore it.
It's caused by the Tensorflow update that happened 30min ago. They added breaking changes.
I'm working on a fix on the master branch right now


oh okay, btw I have made the required change for reading the CSV, I think it should be fine now, please take a look at it when you have some time.

merging since the CI is fixed on master
ethos first commit
State: closed | Author: iamollas (CONTRIBUTOR)

Ethos passed all the tests except from this one: 
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<your-dataset-name>

with this error: 
E               OSError: Cannot find data file. 
E               Original error:
E               [Errno 2] No such file or directory: 

Comments: > Nice thanks !
> 
> I left a few comments
> 
> Also it looks like this PR includes changes about other files than the ones for ethos
> 
> Can you create another branch and another PR please ?

@lhoestq Should I close this PR? The new one is the: #1453

You can create another PR and close this one if you don't mind

> You can create another PR and close this one if you don't mind

Perfect! You should see the #1453 PR for the fixed version! Thanks
add 10k German News Article Dataset
State: closed | Author: stevhliu (MEMBER)


Comments: You can just create another branch from master on your fork and create another PR:

first update your master branch
```
git checkout master
git fetch upstream
git rebase upstream/master
git push
```

then create a new branch
```
git checkout -b my-new-branch-name
```

Then you can add, commit and push the gnad10 files and open a new PR

closing in favor of #1572
Allow GitHub releases as dataset source
State: closed | Author: benjaminvdb (CONTRIBUTOR)

# Summary

Providing a GitHub release URL to `DownloadManager.download()` currently throws a `ConnectionError: Couldn't reach [DOWNLOAD_URL]`. This PR fixes this problem by adding an exception for GitHub releases in `datasets.utils.file_utils.get_from_cache()`.

# Reproduce

```
import datasets
url = 'http://github.com/benjaminvdb/DBRD/releases/download/v3.0/DBRD_v3.tgz'
result = datasets.utils.file_utils.get_from_cache(url)

# Returns: ConnectionError: Couldn't reach http://github.com/benjaminvdb/DBRD/releases/download/v3.0/DBRD_v3.tgz
```

# Cause

GitHub releases returns a HTTP status 403 (FOUND), indicating that the request is being redirected (to AWS S3, in this case). `get_from_cache()` checks whether the status is 200 (OK) or if it is part of two exceptions (Google Drive or Firebase), otherwise the mentioned error is thrown.

# Solution

Just like the exceptions for Google Drive and Firebase, add a condition for GitHub releases URLs that return the HTTP status 403. If this is the case, continue normally.
Add snips built in intents 2016 12
State: closed | Author: bduvenhage (CONTRIBUTOR)

This PR proposes to add the Snips.ai built in intents dataset. The first configuration added is for the intent labels only, but the dataset includes entity slots that may in future be added as alternate configurations.

Comments: It is not clear how to automatically add the dummy data if the source data is a more complex json format. Should I manually take a fraction of the source data and include it as dummy data?


Added a fraction of the real data as dummy data.

merging since the CI is fixed on master
Add HateSpeech Corpus for Polish
State: closed | Author: kacperlukawski (CONTRIBUTOR)

This PR adds a HateSpeech Corpus for Polish, containing offensive language examples.

- **Homepage:** http://zil.ipipan.waw.pl/HateSpeech
- **Paper:** http://www.qualitativesociologyreview.org/PL/Volume38/PSJ_13_2_Troszynski_Wawer.pdf

Comments: @lhoestq Do you think using the ClassLabel is correct if we don't know the meaning of them?

Once we find out the meanings we can still add them to the dataset card

Feel free to ping me when the PR is ready for the final review
Add OffensEval-TR 2020 Dataset
State: closed | Author: yavuzKomecoglu (CONTRIBUTOR)

This PR adds the OffensEval-TR 2020 dataset which is a Turkish offensive language corpus by me and @basakbuluz. The corpus consist of randomly sampled tweets and annotated in a similar way to [OffensEval](https://sites.google.com/site/offensevalsharedtask/) and [GermEval](https://projects.fzai.h-da.de/iggsa/).

- **Homepage:** [offensive-turkish](https://coltekin.github.io/offensive-turkish/)
- **Paper:** [A Corpus of Turkish Offensive Language on Social Media](https://coltekin.github.io/offensive-turkish/troff.pdf)
- **Point of Contact:** [Çağrı Çöltekin](ccoltekin@sfs.uni-tuebingen.de)

Comments: @lhoestq, can you please review this PR? 

> Awesome thank you !

Thanks for the small fixes @lhoestq 

@coltekin, we have added the data set that you created an article that says "Turkish Attack Language Community in Social Media", HuggingFace dataset update sprint for you. We added Sprint quickly for a short time. I hope you welcome it too. The dataset is accessible at https://huggingface.co/datasets/offenseval2020_tr. 

Thank you for the heads up. I am not familiar with the terminology above (no idea what a sprint is), but I am happy that you found the data useful. Please feel free to distribute/use it as you see fit.

The OffensEval version you included in your data set has only binary labels. There is also a version [here](https://coltekin.github.io/offensive-turkish/troff-v1.0.tsv.gz) which also includes fine-grained labels similar to the OffensEval English data set - Just in case it would be of interest.

If you have questions about the data set, or need more information please let me know.
Add SAMSum Corpus dataset
State: closed | Author: changjonathanc (CONTRIBUTOR)

Did not spent much time writing README, might update later.

Copied description and some stuff from tensorflow_datasets
https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/summarization/samsum.py

Comments: also to fix the check_code_quality CI you have to remove the imports of the unused `csv` and `os`

@lhoestq Thanks for the review! I have done what you asked, README is also updated. 🤗 
The CI fails because of the added dependency. I have never used circleCI before, so I am curious how will you solve that?

I just added `py7zr` to our test dependencies

merging since the CI is fixed on master

Thanks! 🤗
Add Wiki Lingua Dataset
State: closed | Author: katnoria (CONTRIBUTOR)

Hello,

This is my first PR. 

I have added Wiki Lingua Dataset along with dataset card to the best of my knowledge.
There was one hiccup though. I was unable to create dummy data because the data is in pkl format.
From the document, I see that:
```At the moment it supports data files in the following format: txt, csv, tsv, jsonl, json, xml```



Comments: I am done adding the dataset. Requesting to review and advise.

looks like this PR has changes about many other files than the ones for WIki Lingua 

Can you create another branch and another PR please ?

Any reason to have english as the default config over the other languages ?

> looks like this PR has changes about many other files than the ones for WIki Lingua
> 
> Can you create another branch and another PR please ?

Ok, I will create another branch and submit a fresh PR.

> Any reason to have english as the default config over the other languages ?

The data for all other languages has a direct reference to English article. Thus, I kept English as default.

closing in favor of #1470
add W&I + LOCNESS dataset (BEA-2019 workshop shared task on GEC)
State: closed | Author: aseifert (CONTRIBUTOR)

- **Name:** W&I + LOCNESS dataset (from the BEA-2019 workshop shared task on GEC)
- **Description:** https://www.cl.cam.ac.uk/research/nl/bea2019st/#data
- **Paper:** https://www.aclweb.org/anthology/W19-4406/
- **Motivation:** This is a recent dataset (actually two in one) for grammatical error correction and is used for benchmarking in this field of NLP.

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


Comments: I created a clean PR where I also incorporated the suggested changes here: https://github.com/huggingface/datasets/pull/1449
can't load "german_legal_entity_recognition" dataset
State: closed | Author: nataly-obr (NONE)

FileNotFoundError: Couldn't find file locally at german_legal_entity_recognition/german_legal_entity_recognition.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py


Comments: Please if you could tell me more about the error? 

1. Please check the directory you've been working on
2. Check for any typos

> Please if you could tell me more about the error?
> 
> 1. Please check the directory you've been working on
> 2. Check for any typos

Error happens during the execution of this line:
dataset = load_dataset("german_legal_entity_recognition")

Also, when I try to open mentioned links via Opera I have errors "404: Not Found" and "This XML file does not appear to have any style information associated with it. The document tree is shown below." respectively.

Hello @nataly-obr, the `german_legal_entity_recognition` dataset has not yet been released (it is part of the coming soon v2 release).

You can still access it now if you want, but you will need to install `datasets` via the master branch:
`pip install git+https://github.com/huggingface/datasets.git@master`

Please let me know if it solves the issue :)
The Snips Built In Intents 2016 dataset.
State: closed | Author: bduvenhage (CONTRIBUTOR)

This PR proposes to add the Snips.ai built in intents dataset. The first configuration added is for the intent labels only, but the dataset includes entity slots that may in future be added as alternate configurations.

Comments: It is not clear how to automatically add the dummy data if the source data is a more complex json format. Should I manually take a fraction of the source data and include it as dummy data?

Will tag the dataset and update the dataset card.
Add dutch book review dataset
State: closed | Author: benjaminvdb (CONTRIBUTOR)

- Name: Dutch Book Review Dataset (DBRD)
- Description: The DBRD (pronounced dee-bird) dataset contains over 110k book reviews along with associated binary sentiment polarity labels and is intended as a benchmark for sentiment classification in Dutch.
- Paper: https://arxiv.org/abs/1910.00896
- Data: https://github.com/benjaminvdb/DBRD
- Motivation: A large (real-life) dataset of Dutch book reviews and sentiment polarity (positive/negative), based on the associated rating.

Checks
- [x] Create the dataset script /datasets/dbrd/dbrd.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _info(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.

Comments: > Really cool thanks !
> 
> I left some (minor) comments

Thank you for your comments! 👍 I went ahead and improved the dataset card using your suggestions and some tweaks of my own. I hope you like it! 😄
add thaiqa_squad
State: closed | Author: cstorm125 (CONTRIBUTOR)

Example format is a little different from SQuAD since `thaiqa` always have one answer per question so I added a check to convert answers to lists if they are not already one to future-proof additional questions that might have multiple answers.

`thaiqa_squad` is an open-domain, extractive question answering dataset (4,000 questions in `train` and 74 questions in `dev`) in [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) format, originally created by [NECTEC](https://www.nectec.or.th/en/) from Wikipedia articles and adapted to [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) format by [PyThaiNLP](https://github.com/PyThaiNLP/).
Craigslist bargains
State: closed | Author: ZacharySBrown (CONTRIBUTOR)

`craigslist_bargains` dataset from [here](https://worksheets.codalab.org/worksheets/0x453913e76b65495d8b9730d41c7e0a0c/)

Comments: Seeing this in the CircleCI builds, this is what I was originally getting before I started messing around with the download URLS to try to fix this:

`FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpwvji917g/extracted/d6185140afb24ad8fee67392100a478269cba286b0d88915a137fdf88872de14/dummy_data/train__VARIABLE_MISUSE__SStuB.txt-00001-of-00300'`

Could this be because of the files in my `dummy_data.zip`? I had to manually create it, and it looked like the test was looking for the following files, so I created the `.zip` with this structure:

```
Archive:  dummy_data.zip
   creating: dummy_data/
  inflating: dummy_data/blobtest     
  inflating: dummy_data/parsed.jsontrain  
  inflating: dummy_data/parsed.jsonvalidation 
```

Going to close this out and link to a new (cleaner) PR
add One Million Posts Corpus
State: closed | Author: aseifert (CONTRIBUTOR)

- **Name:** One Million Posts Corpus
- **Description:** The “One Million Posts” corpus is an annotated data set consisting of user comments posted to an Austrian newspaper website (in German language).
- **Paper:** https://dl.acm.org/doi/10.1145/3077136.3080711
- **Data:** https://github.com/OFAI/million-post-corpus
- **Motivation:** Big German (real-life) dataset containing different annotations around forum moderation with expert annotations.

### Checkbox

- [X] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [X] Fill the `_DESCRIPTION` and `_CITATION` variables
- [X] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [X] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [X] Generate the metadata file `dataset_infos.json` for all configurations
- [X] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [X] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [X] Both tests for the real data and the dummy data pass.


Comments: merging since the CI is fixed on master
oclar-dataset
State: closed | Author: alaameloh (CONTRIBUTOR)

Opinion Corpus for Lebanese Arabic Reviews (OCLAR) corpus is utilizable for Arabic sentiment classification on reviews, including hotels, restaurants, shops, and others. : [homepage](http://archive.ics.uci.edu/ml/datasets/Opinion+Corpus+for+Lebanese+Arabic+Reviews+%28OCLAR%29#)

Comments: merging since the CI is fixed on master
Created wiki_movies dataset.
State: closed | Author: aclifton314 (CONTRIBUTOR)

First PR (ever). Hopefully this movies dataset is useful to others!

Comments: looks like your PR includes changes about many other files than the ones for wiki_movies

Can you create another branch and another PR please ?

I'm happy to. What's the best way to do that (sorry, I'm new to PRs etc.)?

Sure !

First please save your new dataset files somewhere.
Then you can do in this order:
```
git checkout master
git fetch upstream
git rebase upstream/master
git push
git checkout -b my-new-branch-name
```
This will create a new branch from the updated master branch.
Then you can re-add your files and commit + push them

Once it's done you should be able to create a new PR using your new branch :) 

Done!

closing in favor of #1485
add DFKI SmartData Corpus
State: closed | Author: aseifert (CONTRIBUTOR)

- **Name:** DFKI SmartData Corpus
- **Description:** DFKI SmartData Corpus is a dataset of 2598 German-language documents which has been annotated with fine-grained geo-entities, such as streets, stops and routes, as well as standard named entity types.
- **Paper:** https://www.dfki.de/fileadmin/user_upload/import/9427_lrec_smartdata_corpus.pdf
- **Data:** https://github.com/DFKI-NLP/smartdata-corpus
- **Motivation:** Contains fine-grained NER labels for German.

### Checkbox

- [X] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [X] Fill the `_DESCRIPTION` and `_CITATION` variables
- [X] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [X] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [X] Generate the metadata file `dataset_infos.json` for all configurations
- [X] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [X] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [X] Both tests for the real data and the dummy data pass.
Adding OneStopEnglish corpus dataset
State: closed | Author: purvimisal (CONTRIBUTOR)

This PR adds OneStopEnglish Corpus containing texts classified into reading levels (elementary, intermediate, advance) for automatic readability assessment and text simplification. 

Link to the paper: https://www.aclweb.org/anthology/W18-0535.pdf 

Comments: Hi @lhoestq, thanks for the review.
I have made all the changes, PTAL! :)