noamaanMulla-03/datasets-issues huggingface dataset:
adding hind_encorp dataset

adding Hindi_Encorp05 dataset


help needed in dummy data

extension of the file is .plaintext so dummy data generation is failing


you can add the `--match_text_file "*.plaintext"` flag when generating the dummy data

also it looks like the PR is empty, is this expected ?

yes it is expected because I  made all my changes in  PR  #1186  then I again run code and open PR #1188 to see if this time test passes or not only so there is no code change from #1186 to #1188  
i tried  --match_text_file "*.plaintext" this time it is also not generating dummy data don't know why

well this PR includes no code change at all, can you make sure you added your changes in this one ?

feel free to ping me when you have added the files so I can take a look and help you with the dummy data

how to do that i dont know did i have to open new PR


 actually all my changes are visible in  #1186 but  don't know how to show same changes here

these are a the which i did in #1186 and same in #1188 
![1](https://user-images.githubusercontent.com/56379013/101646577-b4864500-3a5d-11eb-8a5a-91b1b441040a.png)
![2](https://user-images.githubusercontent.com/56379013/101646965-32e2e700-3a5e-11eb-94d9-276e602c6ded.png)
![4](https://user-images.githubusercontent.com/56379013/101646989-38d8c800-3a5e-11eb-92bb-d9c4cb2c3595.png)
![5](https://user-images.githubusercontent.com/56379013/101647017-41c99980-3a5e-11eb-87cf-5268e79df19d.png)
![6](https://user-images.githubusercontent.com/56379013/101647038-48581100-3a5e-11eb-8d05-f67834fcaa7b.png)

![8](https://user-images.githubusercontent.com/56379013/101647080-55750000-3a5e-11eb-8455-8936a35b35c2.png)
![9](https://user-images.githubusercontent.com/56379013/101647084-55750000-3a5e-11eb-988e-ae87f0b252a0.png)
![10](https://user-images.githubusercontent.com/56379013/101647182-6f164780-3a5e-11eb-8af3-f0b0186483c9.png)
![11](https://user-images.githubusercontent.com/56379013/101647230-7c333680-3a5e-11eb-9aeb-2b4ce65965e0.png)
![13](https://user-images.githubusercontent.com/56379013/101647257-848b7180-3a5e-11eb-871c-2fd77b047320.png)
![14](https://user-images.githubusercontent.com/56379013/101647268-89502580-3a5e-11eb-9e2a-b9f7ff1fc95e.png)
these same codes are in both #1186 and #1188 so because it is already present from PR #1186 because of that it is showing zeor code change in #1188 because it is already present from #1186 how i can show or highlight those changes











well for me https://github.com/huggingface/datasets/pull/1188/files is blank

This PR tries to merge the master branch of you fork into this repo, however I can't find changes with your files inside your master branch.

Maybe you can fork again the repo and try to create another PR ?

@lhoestq  i  opened a new pr #1438 but this time it fails many circl ci tests

Closing this one since a new PR was created
add One Million Posts Corpus

- **Name:** One Million Posts Corpus
- **Description:** The “One Million Posts” corpus is an annotated data set consisting of user comments posted to an Austrian newspaper website (in German language).
- **Paper:** https://dl.acm.org/doi/10.1145/3077136.3080711
- **Data:** https://github.com/OFAI/million-post-corpus
- **Motivation:** Big German (real-life) dataset containing different annotations around forum moderation with expert annotations.

### Checkbox

- [X] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [X] Fill the `_DESCRIPTION` and `_CITATION` variables
- [X] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [X] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [X] Generate the metadata file `dataset_infos.json` for all configurations
- [X] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [X] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [X] Both tests for the real data and the dummy data pass.



merging since the CI is fixed on master
add Indonesian newspapers (id_newspapers_2018)

The dataset contains around 500K articles (136M of words) from 7 Indonesian newspapers. The size of uncompressed 500K json files (newspapers-json.tgz) is around 2.2GB.


Looks like there's a `Path` issue on windows. Could you try switching to
`glob.glob(os.path.join(article_dir, "*.json"))`

> Looks like there's a `Path` issue on windows. Could you try switching to
> `glob.glob(os.path.join(article_dir, "*.json"))`

Thanks, I replaced it with glob. Let's see if it solves the issue. Anyway, the main directory has a space, could it make the issue on windows? the test on linux don't have this problem.

It seems glob doesn't help also. Btw, one of the failing test tried to connect aws which failed:
```
C:\tools\miniconda3\lib\site-packages\urllib3\connection.py:160: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

address = ('s3.amazonaws.com', 443), timeout = 10, source_address = None
socket_options = [(6, 1, 1)]

```
Why did it try to connect to aws? I don't use it.

It seems that the circleci make a test for whole datasets repository, that means if only one of the dataset in the official repository has a download issue, this will also affect the test of a new dataset like mine, isn't it?
I changed the url to my newspaper dataset which contains only few simple json files and simple directory structure. But it still failed. And it failed not only on windows test. This is one of the error message:
```
-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_chr_en
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_chr_en
===== 4 failed, 2667 passed, 2052 skipped, 4 warnings in 432.05s (0:07:12) =====

Exited with code exit status 1
CircleCI received exit code 1
```
The test failed on twitter dataset even my dataset has nothing to do with twitter? 

merging since the CI is fixed on master

Hi, thanks for merging the dataset. I create a new PR (#1499) since I need to update the link to the dataset.
Add OffensEval-TR 2020 Dataset

This PR adds the OffensEval-TR 2020 dataset which is a Turkish offensive language corpus by me and @basakbuluz. The corpus consist of randomly sampled tweets and annotated in a similar way to [OffensEval](https://sites.google.com/site/offensevalsharedtask/) and [GermEval](https://projects.fzai.h-da.de/iggsa/).

- **Homepage:** [offensive-turkish](https://coltekin.github.io/offensive-turkish/)
- **Paper:** [A Corpus of Turkish Offensive Language on Social Media](https://coltekin.github.io/offensive-turkish/troff.pdf)
- **Point of Contact:** [Çağrı Çöltekin](ccoltekin@sfs.uni-tuebingen.de)


@lhoestq, can you please review this PR? 

> Awesome thank you !

Thanks for the small fixes @lhoestq 

@coltekin, we have added the data set that you created an article that says "Turkish Attack Language Community in Social Media", HuggingFace dataset update sprint for you. We added Sprint quickly for a short time. I hope you welcome it too. The dataset is accessible at https://huggingface.co/datasets/offenseval2020_tr. 

Thank you for the heads up. I am not familiar with the terminology above (no idea what a sprint is), but I am happy that you found the data useful. Please feel free to distribute/use it as you see fit.

The OffensEval version you included in your data set has only binary labels. There is also a version [here](https://coltekin.github.io/offensive-turkish/troff-v1.0.tsv.gz) which also includes fine-grained labels similar to the OffensEval English data set - Just in case it would be of interest.

If you have questions about the data set, or need more information please let me know.
hate_speech_18 initial commit



> Nice thanks !
> 
> Can you rename the dataset folder and the dataset script name `hate_speech18` instead of `hate_speech_18` to follow the snake case convention we're using ?
> 
> Also it looks like the dummy_data.zip file is quite big (almost 4MB).
> Can you try to reduce its size ?
> 
> To do so feel free to take a look inside it and remove all the unnecessary files or chunks of texts. The idea is to only keep a few examples

Done, thanks! 

Re-opened in https://github.com/huggingface/datasets/pull/1486
Fake news english 4



Thanks for the PR @MisbahKhan789 !

Few comments to help you along (I'm NOT a maintainer, just offering help to unblock the process) :-
 - Could you re-run `make style` and fix the errors related to code quality specific to your dataset in the `datasets/fake_news_english` folder?
(These seem to show errors that need manual fixes on running `flake8 datasets/fake_news_english`)
- Please run the local tests and check if they pass
`RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_<your-dataset-name>`
 `RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<your-dataset-name>`
(If not, you may have to regenerate the dummy data and the `dataset_infos.json` files)



Hii please follow me

> Thanks for the PR @MisbahKhan789 !
> 
> Few comments to help you along (I'm NOT a maintainer, just offering help to unblock the process) :-
> 
> * Could you re-run `make style` and fix the errors related to code quality specific to your dataset in the `datasets/fake_news_english` folder?
>   (These seem to show errors that need manual fixes on running `flake8 datasets/fake_news_english`)
> * Please run the local tests and check if they pass
>   `RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_<your-dataset-name>`
>   `RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<your-dataset-name>`
>   (If not, you may have to regenerate the dummy data and the `dataset_infos.json` files)

Hey Bharat, thanks for the reply. I actually submitted a new PR with the changes that are required :)
terminate called after throwing an instance of 'google::protobuf::FatalException'

Hi
I am using the dataset "iwslt2017-en-nl", and after downloading it I am getting this error when trying to evaluate it on T5-base with seq2seq_trainer.py in the huggingface repo could you assist me please? thanks 


100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 63/63 [02:47<00:00,  2.18s/it][libprotobuf FATAL /sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:1505] CHECK failed: (index) >= (0): 
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: (index) >= (0): 
run_t5_base_eval.sh: line 19:  5795 Aborted 


Loading the iwslt2017-en-nl config of iwslt2017 works fine on my side. 
Maybe you can open an issue on transformers as well ? And also add more details about your environment (OS, python version, version of transformers and datasets etc.)

closing now, figured out this is because the max length of decoder was set smaller than the input_dimensions. thanks
Added conv ai 2 (Again)

The original PR -> https://github.com/huggingface/datasets/pull/1383

Reason for creating again - 

The reason I had to create the PR again was due to the master rebasing issue. After rebasing the changes, all the previous commits got added to the branch. 


Looking perfect to me, need to rerun the tests


Thanks, @tanmoyio. 
How do I rerun the tests? Should I change something or push a new commit?

@rkc007 you don't need to rerun it, @lhoestq @yjernite will rerun it, as there are huge number of PRs in the queue it might take lil bit of time. 

ive just re-run the tests

Thank you @abhishekkrthakur. Can you please rerun it again? It seems something was broken in CI during the previous test.

@lhoestq Sorry for the mess. I don't know why this keeps on happening. I tried step by step process of updating the PR but seems something is wrong. This happened for 2nd time with the same PR. Apologies for that. 

New PR -> https://github.com/huggingface/datasets/pull/1527
Also, I fixed everything in the new PR.
Adding english plaintext jokes dataset

This PR adds a dataset of 200k English plaintext Jokes from three sources: Reddit, Stupidstuff, and Wocka.
Link: https://github.com/taivop/joke-dataset 

This is my second PR. 
First was: [#1269 ](https://github.com/huggingface/datasets/pull/1269)


Hi @purvimisal, thanks for your contributions!

This jokes dataset has come up before, and after a conversation with the initial submitter, we decided not to add it then. Humor is important, but looking at the actual data points in this set raises several concerns :) 

The main issue is the Reddit part of the dataset which has most of the examples. A cursory look at the data shows a large number of highly offensive jokes that reproduce some pretty harmful biases (the second one from the top is a Holocaust joke). 

The other two sources have similar issues (especially the "Blond Jokes") to a slightly lesser extent.

While such datasets can be useful in the right context, there is a real concern that people using the library might miss some of this context (however much we outline it), and end up unwittingly training models that rely on some pretty racist and sexist associations.

We would recommend skipping this dataset altogether.

If you feel really strongly about having a joke dataset, then we would ask that you:
- remove the Reddit part of the dataset altogether
- write an in-depth description of the social biases present in the remaining data

Let us know which of the two you decide! And if you want recommendations on other datasets to add, hit us up on Slack 🤗 

Hi @yjernite, thanks so much. I should've totally thought about this earlier. The harmful biases make so much sense. I should've consulted before making a PR. 
I will be closing this one and skipping this dataset altogether. 
Thanks again
adding covid-tweets-japanese (again)

I had mistaken use git rebase, I was so hurried to fix it. However, I didn't fully consider the use of git reset , so I unintendedly stopped PR (#1367) altogether. Sorry about that.
I'll make a new PR.


README.md is not created yet. I'll add it soon.

Thank you for your detailed code review! It's so helpful.
I'll reflect them to the code in 24 hours.

You may have told me in Slack (I cannot find the conversation log though I've looked through threads), but I'm sorry it seems I'm still misunderstanding how to get YAML from the tagger.
I'm now asking on Slack if I am looking at the tagger the wrong way.

One more thing I'd like to ask.
Should I make changes by myself, or can I use the "Commit suggestion" feature?
I'm new to this feature and I don't know how the rules work in this repository, so I'd like to ask just in case.

Thank you very much for merging!
Add twi text c3

Added Twi texts for training embeddings and language models based on the paper https://www.aclweb.org/anthology/2020.lrec-1.335/


looks like this PR includes changes about other datasets

Can you only include the changes related to twi text c3 please ?

Hi @lhoestq , I have removed the unnecessary files. Can you please confirm?

You might need to either find a way to go back to the commit before it changes 389 files or create a new branch.

okay, I have created another branch, see the latest pull https://github.com/huggingface/datasets/pull/1518  @cstorm125 

Hii please follow me

Closing this one in favor of #1518
Add yoruba text c3

Added Yoruba texts for training embeddings and language models based on the paper https://www.aclweb.org/anthology/2020.lrec-1.335/


looks like this PR includes changes about other datasets


Thanks for the review. I'm a bit confused how to remove the files. Every time I add a new branch name using the following commands:

git fetch upstream
git rebase upstream/master
git checkout -b a-descriptive-name-for-my-changes

and push to the origin, this issue occurs

Can you try to create the branch from the master branch of your fork ?

first update your master branch:
```
git checkout master
git fetch upstream
git rebase upstream/master
git push
```

then create a new one:
```
git checkout -b my-new-branch
```

I think you were still having the files because you were creating the new branch from a branch in which you've committed the files, instead of creating the new branch from the master branch

Got it, will correct that. Thanks

@lhoestq , I have removed the unnecessary files. Looks like I still have one error. How do I resolve this?

> @lhoestq , I have removed the unnecessary files. Looks like I still have one error. How do I resolve this?

I think it's connection error on piqa dataset. Can you try triggering the test again? I usually resolve similar issues with:
```
git fetch upstream
git rebase upstream/master
git push -u -f origin your_branch_name
```

thank you  @cstorm125 

I have created another pull request for this https://github.com/huggingface/datasets/pull/1515 @cstorm125  @lhoestq 

Hii please follow me

merging since the CI is fixed on master

Great, thanks a lot
Adding TaPaCo Dataset with README.md



We want to keep the repo as light as possible so that it doesn't take ages to clone, that's why we ask for small dummy data files (especially when there are many of them). Let me know if you have questions or if we can help you on this

Hello @lhoestq , made the changes as you suggested and pushed, please review. By default, the dummy data was generated the way it was by the dummy data auto generate command. Thank you.
adding HindEncorp

adding Hindi Wikipedia corpus


@lhoestq I have created a new PR by reforking and creating a new branch 

@rahul-art unfortunately this didn't quite work, here's how you can try again:
- `git checkout master` to go back to the main branch
- `git pull upstream master` to make it up to date
- `git checkout -b add_hind_encorp` to create a new branch

Then add the dataset script, `README.md`, `dummy_data.zip`, and `dataset_infos.json` to the tracked files for the branch with `git add` (please add all of these files individually, NOT the whole directory as we don't want the other data files)
Then after you have passed the style checks and the local tests, do:
- `git commit . -m initial_commit`
- `git push --set-upstream origin add_hind_encorp`

Then you can go to this branch on the WebApp and open a new PR

@yjernite #1557 created new PR
Add snips built in intents 2016 12

This PR proposes to add the Snips.ai built in intents dataset. The first configuration added is for the intent labels only, but the dataset includes entity slots that may in future be added as alternate configurations.


It is not clear how to automatically add the dummy data if the source data is a more complex json format. Should I manually take a fraction of the source data and include it as dummy data?


Added a fraction of the real data as dummy data.

merging since the CI is fixed on master
Add Wiki Lingua Dataset

Hello,

This is my first PR. 

I have added Wiki Lingua Dataset along with dataset card to the best of my knowledge.
There was one hiccup though. I was unable to create dummy data because the data is in pkl format.
From the document, I see that:
```At the moment it supports data files in the following format: txt, csv, tsv, jsonl, json, xml```




I am done adding the dataset. Requesting to review and advise.

looks like this PR has changes about many other files than the ones for WIki Lingua 

Can you create another branch and another PR please ?

Any reason to have english as the default config over the other languages ?

> looks like this PR has changes about many other files than the ones for WIki Lingua
> 
> Can you create another branch and another PR please ?

Ok, I will create another branch and submit a fresh PR.

> Any reason to have english as the default config over the other languages ?

The data for all other languages has a direct reference to English article. Thus, I kept English as default.

closing in favor of #1470
add best2009

`best2009` is a Thai word-tokenization dataset from encyclopedia, novels, news and articles by [NECTEC](https://www.nectec.or.th/) (148,995/2,252 lines of train/test). It was created for [BEST 2010: Word Tokenization Competition](https://thailang.nectec.or.th/archive/indexa290.html?q=node/10). The test set answers are not provided publicly.
Add SAMSum Corpus dataset

Did not spent much time writing README, might update later.

Copied description and some stuff from tensorflow_datasets
https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/summarization/samsum.py


also to fix the check_code_quality CI you have to remove the imports of the unused `csv` and `os`

@lhoestq Thanks for the review! I have done what you asked, README is also updated. 🤗 
The CI fails because of the added dependency. I have never used circleCI before, so I am curious how will you solve that?

I just added `py7zr` to our test dependencies

merging since the CI is fixed on master

Thanks! 🤗
Created wiki_movies dataset.

First PR (ever). Hopefully this movies dataset is useful to others!


looks like your PR includes changes about many other files than the ones for wiki_movies

Can you create another branch and another PR please ?

I'm happy to. What's the best way to do that (sorry, I'm new to PRs etc.)?

Sure !

First please save your new dataset files somewhere.
Then you can do in this order:
```
git checkout master
git fetch upstream
git rebase upstream/master
git push
git checkout -b my-new-branch-name
```
This will create a new branch from the updated master branch.
Then you can re-add your files and commit + push them

Once it's done you should be able to create a new PR using your new branch :) 

Done!

closing in favor of #1485
Add doc2dial dataset

### Doc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset v0.9

Once complete this will add the [Doc2dial](https://doc2dial.github.io/data.html) dataset from the generic data sets list.


It not always practical to use nested `Sequence`. If you have troubles with sequence you can use lists instead. 

For example
```python

features=datasets.Features(
    {
        "dial_id": datasets.Value("string"),
        "doc_id": datasets.Value("string"),
        "domain": datasets.Value("string"),
        "turns": [
            {
                "turn_id": datasets.Value("int32"),
                "role": datasets.Value("string"),
                "da": datasets.Value("string"),
                "reference": [
                    {
                        "keys" : datasets.Value("string"),
                        "values": datasets.Value("string"),   
                    }

                ],
                "utterance": datasets.Value("string"),
            }
        ],
    }
),
```

this way `turns` will be a list of dict, and the "reference" key of `turns` will be a list of dict as well

No problem thanks for all your help getting this to the final stages! Added .gitignore, removed .lock and applied the changes you asked for.
added un_ga dataset

Hi :hugs:, This is a PR for [United nations general assembly resolutions: A six-language parallel corpus](http://opus.nlpl.eu/UN.php) dataset


Looks like this PR includes changes about many other files than the ones for un_ga

Can you create another branch an another PR please ?

@lhoestq, Thank you for suggestions. I have made the changes and raised the new PR https://github.com/huggingface/datasets/pull/1569.