noamaanMulla-03/datasets-issues huggingface dataset:
Add the DKTC dataset (Extension of #3564)

New pull request of #3564. (for DKTC)


Labels: name: dataset contribution

I reflect your comment! @lhoestq 

Wait, the format of the data just changed, so I'll take it into consideration and commit it.

I update the code according to the dataset structure change.

Thanks ! I think the dummy data are not valid yet - the dummy train.csv file only contains a partial example (the quotes `"` start but never end).

> Thanks ! I think the dummy data are not valid yet - the dummy train.csv file only contains a partial example (the quotes `"` start but never end).

Hi! @lhoestq There is a problem.  
<img src="https://user-images.githubusercontent.com/42150335/149804142-3800e635-f5a0-44d9-9694-0c2b0c05f16b.png" width=500>
  
As shown in the picture above, the conversation is divided into "\n" in the "conversion" column.  
That's why there's an error in the file path that only saved only five lines like below. 

```
'idx', 'class', 'conversation'
'0', '협박 대화', '"지금 너 스스로를 죽여달라고 애원하는 것인가?'
아닙니다. 죄송합니다.'
죽을 거면 혼자 죽지 우리까지 사건에 휘말리게 해? 진짜 죽여버리고 싶게.'
정말 잘못했습니다.
```
   
In fact, these five lines are all one line.  
  



Hi ! I see, in this case ca you make sure that the dummy data has a full sample ?

Feel free to open the dummy train.csv in the dummy_data.zip file and add the missing lines

Sorry, I'm late to check! I'll send it to you soon!

Thanks for your contribution, @sooftware. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there, under this organization namespace: https://huggingface.co/tunib

Please, feel free to tell us if you need some help.

Close this PR. Thanks!
Add the KMWP dataset (extension of #3564)

New pull request of #3564 (Add the KMWP dataset)

Labels: name: dataset contribution

Sorry, I'm late to check! I'll send it to you soon!

Thanks for your contribution, @sooftware. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there, under this organization namespace: https://huggingface.co/tunib

Please, feel free to tell us if you need some help.

Close this PR. Thanks!
Fix bugs in msr_sqa dataset

The last version has many problems,

1) Errors in table load-in. Split by a single comma instead of using pandas is wrong.
2) id reduplicated in _generate_examples function.
3) Missing information of history questions which make it hard to use.

I fix it refer to https://github.com/HKUNLP/UnifiedSKG. And we test it to perform normally.

Labels: name: dataset contribution

It shows below when I run test:

FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_msr_sqa - ValueError: Unknown split "validation". Should be one of ['train', 'test'].

It make no sense for me😂. 


@albertvillanova Does this PR has some additional fixes compared to https://github.com/huggingface/datasets/pull/3771 or we can close it?

@mariosasko besides the fix of the DuplicatedKeysError, this PR:
- changes the reading of one of the files: use pandas instead of splitting by comma
- changes the splits: modifying train and adding validation
- adds some extra logic in the processing of the data: adding a new field "question_and_history"

We should decide whether validating these additional changes.
- for example, if we accept as pertinent the addition of the field "question_and_history", this should be added as feature to the info, and the matadata should be regenerated...

Hi guys, anything we can do to fix that bug👀? @mariosasko @albertvillanova @lhoestq 

_The documentation is not available anymore as the PR was closed or merged._
Fix license/citation information of squadshifts dataset card

This PR fixes the license/citation information of squadshifts dataset card, once the dataset owners have responded to our request for information:
- https://github.com/modestyachts/squadshifts-website/issues/1

Additionally, we have updated the mention in their website to our `datasets` library (they were referring old name `nlp`):
- https://github.com/modestyachts/squadshifts-website/pull/2#event-7500953009

Labels: name: dataset contribution

_The documentation is not available anymore as the PR was closed or merged._
The speechocean762 dataset

[speechocean762](https://www.openslr.org/101/) is a non-native English corpus for pronunciation scoring tasks. It is free for both commercial and non-commercial use.

I believe it will be easier to use if it could be available on Hugging Face.

Labels: name: dataset contribution

CircleCL reported two errors, but I didn't find the reason. The error message:
```
_________________ ERROR collecting tests/test_dataset_cards.py _________________
tests/test_dataset_cards.py:53: in <module>
    @pytest.mark.parametrize("dataset_name", get_changed_datasets(repo_path))
tests/test_dataset_cards.py:35: in get_changed_datasets
    diff_output = check_output(["git", "diff", "--name-only", "origin/master...HEAD"], cwd=repo_path)
../.pyenv/versions/3.6.15/lib/python3.6/subprocess.py:356: in check_output
    **kwargs).stdout
../.pyenv/versions/3.6.15/lib/python3.6/subprocess.py:438: in run
    output=stdout, stderr=stderr)
E   subprocess.CalledProcessError: Command '['git', 'diff', '--name-only', 'origin/master...HEAD']' returned non-zero exit status 128.

=========================== short test summary info ============================
ERROR tests/test_dataset_cards.py - subprocess.CalledProcessError: Command '[...
ERROR tests/test_dataset_cards.py - subprocess.CalledProcessError: Command '[...
= 4011 passed, 2357 skipped, 2 xfailed, 1 xpassed, 116 warnings, 2 errors in 284.32s (0:04:44) =

Exited with code exit status 1
```
I'm not sure if it was caused by this PR ...

I ran `tests/test_dataset_cards.py` in my local environment, and it passed:
```
(venv)$ pytest tests/test_dataset_cards.py
============================== test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/zhangjunbo/src/datasets
plugins: forked-1.4.0, datadir-1.3.1, xdist-2.5.0
collected 1531 items

tests/test_dataset_cards.py ..... [100%]
======================= 766 passed, 765 skipped in 2.55s ========================
```


@sanchit-gandhi could you also maybe take a quick look? :-)

Thanks for your contribution, @jimbozhang. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.

> Thanks for your contribution, @jimbozhang. Are you still interested in adding this dataset?
> 
> We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets
> 
> We would suggest you create this dataset there. Please, feel free to tell us if you need some help.

Yes, I just planned to finish this dataset these days, and this suggestion is just in time! Thanks a lot!
I will create this dataset to Hugging Face Hub soon, maybe this week.
Adding STSb-TR dataset

Semantic Textual Similarity benchmark Turkish (STSb-TR) dataset introduced in our paper [Semantic Similarity Based Evaluation for Abstractive News Summarization](https://aclanthology.org/2021.gem-1.3.pdf) added.

Labels: name: dataset contribution

Thanks for your contribution, @figenfikri.

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Add CRASS dataset

Added crass dataset

Labels: name: dataset contribution

Hi Albert,
Thank you for your comments.
I hope I have uploaded my local git repo to include the dummy files and style reworkings.
Added YAML in Readme as well.

Please check again.

Hope it works now :)

Thanks for your contribution, @apergo-ai. 

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. It's OK for you? Please, feel free to tell us if you need some help.
Adding mMARCO dataset

Adding mMARCO (v1.1) to HF datasets.

Labels: name: dataset contribution

Hi @albertvillanova we've made a major overhaul of the loading script including all configurations we're making available. Could you please review it again?

@albertvillanova :ping_pong: 

Thanks @lhbonifacio for adding this dataset.
Hi there, i got an error about mmarco:
ConnectionError: Couldn't reach 'unicamp-dl/mmarco' on the Hub (ConnectionError)
code:
`from datasets import list_datasets, load_dataset
dataset = load_dataset('unicamp-dl/mmarco', language='portuguese')`

Any help will be appreciated!

Hi @catqaq, we updated the loading script. Now you can load the datasets with:

```python
dataset = load_dataset('unicamp-dl/mmarco', 'portuguese')
```

You can check the list of supported languages and usage examples in [this link](https://huggingface.co/datasets/unicamp-dl/mmarco). Feel free to contact us if you have any issues.




> 



> Hi @catqaq, we updated the loading script. Now you can load the datasets with:
> 
> ```python
> dataset = load_dataset('unicamp-dl/mmarco', 'portuguese')
> ```
> 
> You can check the list of supported languages and usage examples in [this link](https://huggingface.co/datasets/unicamp-dl/mmarco). Feel free to contact us if you have any issues.

Thanks for your quick updates. So, how can i get the fixed version, install from the source? It seems that the merging is blocked.

@catqaq you can load mMARCO using the namespace `unicamp-dl/mmarco` while this PR remains under review.

Thanks for your contribution, @lhbonifacio and  @hugoabonizio. And sorry for the late response.

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

As you already created this dataset under your organization namespace (https://huggingface.co/datasets/unicamp-dl/mmarco), I think we can safely close this PR.

We would suggest you complete your dataset card with the YAML tags, to make it searchable and discoverable.

Please, feel free to tell us if you need some help.
Add ETHICS dataset

This PR adds the ETHICS dataset, including all 5 sub-datasets.
From https://arxiv.org/abs/2008.02275

Labels: name: dataset contribution

Thanks for your contribution, @ssss1029. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Sunbird AI Ugandan low resource language dataset

Multi-way parallel text corpus of 5 key Ugandan languages for the task of machine translation. 

Labels: name: dataset contribution

Hi @ak3ra , have you had a chance to take my comments into account ?

Let me know if you have questions or if I can help :)

@lhoestq  Working on this, thanks for the detailed review :) 

Hi ! Cool thanks :)
Feel free to merge master into your branch to fix the CI issues

Let me know if you have questions or if I can help

Thanks for your contribution, @ak3ra. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Adds APPS dataset to the hub [WIP]

A loading script for [APPS dataset](https://github.com/hendrycks/apps) 

Labels: name: dataset contribution

Thanks for your contribution, @arampacha. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Add XFUN dataset

This PR adds XFUN dataset.

Home page and repository: https://github.com/doc-analysis/XFUND

Source code: https://github.com/microsoft/unilm/blob/master/layoutlmft/layoutlmft/data/datasets/xfun.py

Labels: name: dataset contribution

_The documentation is not available anymore as the PR was closed or merged._

Not sure how to generate dummy data.

The downloaded file structure is 

- document file paths
    - (a json file containing all documents info, document images folder)
    - (a json file containing all documents info, document images folder)
    - ...

Hey @mariosasko, thanks for the review. I'm not sure how to suggest these changes to the owner @ranpox, and I did spend some time to write the model card and hope to get it on the official repo. Is that possible?

Since the author is not responding, maybe we can go ahead with this PR ?

Go for it!

On Tue, Apr 12, 2022 at 10:24 AM Quentin Lhoest ***@***.***>
wrote:

> Since the author is not responding, maybe we can go ahead with this PR ?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/3974#issuecomment-1096797650>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ATFNL66EVUFWS3P2FOAS7SLVEWBP3ANCNFSM5RFH3MXA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


@qqaatw Do you plan to finish this PR? I can give you some pointers and help you with the code if needed.

@mariosasko Yes, I'll apply all of the suggestions when I have some time.

Thanks for your contribution, @qqaatw.

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you propose this changes there to the original repo. Please, feel free to tell us if you need some help.
Add Rico Dataset

Hi there!

I'm wanting to add the Rico datasets for software engineering type data to y'alls awesome library. However, as I have started coding, I've ran into a few hiccups so I thought it best to open the PR early to get a bit of discussion on how the Rico datasets should be added to the `datasets` lib.

1) There are 7 different datasets under Rico and so I was wondering, should I make a folder for each or should I put them as different configurations of a single dataset?
You can see the datasets available for Rico here: http://interactionmining.org/rico

2) As of right now, I have a semi working version of the first dataset which has pairs of screenshots and hierarchies from android applications. However, these screenshots are very large (1440, 2560, 3) and there are 66,000 of them so I am not able to perform the processing that  the `datasets` lib does after downloading and extracting the dataset since I run out of memory very fast. Is there a way to have `datasets` lib not put everything into memory while it is processing the dataset?

2.1) If there is not a way, would it be better to just return the path to the screenshots instead of the actual image?

3) The hierarchies are JSON objects and looking through the documentation of `datasets`, I didn't see any feature that I could use for this type of data. So, for now I just have it being read in as a string, is this okay or should I be doing it differently?

4) One of the Rico datasets is a bunch of animations (GIFs), is there a `datasets` feature that I can put this type of data into or should I just return the path as a string?

I appreciate any and all help I can get for this PR, I think the Rico datasets will be an awesome addition to the library :nerd_face: !

Labels: name: dataset contribution

Hi ! Thanks for adding this dataset :)

Regarding your questions:
1. We can have them as different configuration of the `rico` dataset
2. Yes please use the path to the image and not open the image directly, so that we can let users open the image one at at time during training if they want to for example. In the future we'll have an Image feature type that will decode the encoded image data on the fly when accessing the examples.
3. Feel free to keep the hierarchies as strings if they don't follow a fixed format
4. You can just return the path



Thanks for your contribution, @ncoop57. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Adding myPOS2017 dataset

myPOS Corpus (Myanmar Part-of-Speech Corpus) for Myanmar language NLP Research and Developments

Labels: name: dataset contribution

I've updated the code and Readme to reflect your comments.
Thank you very much,

looks like this PR includes changes about many other files than the ones for myPOS2017

Could you open another branch and another PR please ?
(or fix this branch)

Hi @hungluumfc ! Have you had a chance to fix this PR so that it only includes the changes for `mypos` ? 

Feel free to ping me if you have questions or if I can help :) 

Thanks for your contribution, @hungluumfc. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Monero

Biomedical Romanian dataset :)

Labels: name: dataset contribution

Hi @iliemihai  - you need to add the Readme file! Otherwise seems good. 
Also don't forget to run `make style` & `flake8 datasets` locally, from the datasets folder

@skyprince999 I will add the README.d for it. Thank you :D 

Thanks for your contribution, @iliemihai. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
ru_reviews dataset adding

RuReviews: An Automatically Annotated Sentiment Analysis Dataset for Product Reviews in Russian

Labels: name: dataset contribution

Hi @lhoestq  

I have added the readme as well 

Please do have a look at it when suitable 

Chatted with @darshan-gandhi on Slack about parsing examples into a separate text and sentiment field

Thanks for your contribution, @darshan-gandhi. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Add Indosum dataset


Labels: name: dataset contribution

Hi @prasastoadi have you had a chance to take a look at my suggestions ?

Feel free to ping ;e if you have questions or when you're ready for a review

Thanks for your contribution, @prasastoadi. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Add german common crawl dataset

Adding a subpart of the Common Crawl which was extracted with this repo https://github.com/facebookresearch/cc_net and additionally filtered for duplicates 

Labels: name: dataset contribution

Hi @Phil1108 !
Have you had a chance to take a look at my suggestions ?
Feel free to ping me if you have questions or if you're ready for a review

Thanks again for adding this dataset, this one is very useful !

> 
> 
> Hi @Phil1108 !
> Have you had a chance to take a look at my suggestions ?
> Feel free to ping me if you have questions or if you're ready for a review
> 
> Thanks again for adding this dataset, this one is very useful !

Hello @lhoestq , 

okay so should we go on with only the 2 parts or wait until the hosting of all 200 GB is cleared at HF? 
If you want to merge it with the 2 parts only at first und upgrade later to all, sure I'll include the minor changes in the next few days and update it here

I think we can wait a bit to find a solution for the hosting and update the PR to include all the files when it's done.
In the meantime you can update the PR to take into account the suggestions regarding the dataset card, the parallel downloads and the use of eval.

A safer alternative to eval is `ast.literal_eval`, it should do the job

Thanks for your contribution, @Phil1108. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
adding marathi-wiki dataset

Adding marathi-wiki-articles dataset. 

Labels: name: dataset contribution

Can you make it a draft PR until you've added the dataset please ? @ekdnam 

Done

Thanks for your contribution, @ekdnam. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Add OpenPI Dataset

Add the OpenPI Dataset by AI2 (AllenAI)

Labels: name: dataset contribution

Hi @Bharat123rox ! It looks like some of the dummy data is broken or missing. Did you auto-generate it? Does the local test pass for you?

@yjernite requesting you to have a look as to why the tests are failing only on Windows, there seems to be a backslash error somewhere, could it be the result of `os.path.join` and what should be the fix for this?

This is the `black` output locally:
```
(datasets_env) datasets (openpi) > black --check --line-length 119 --target-version py36 datasets/openpi/
All done! ✨ 🍰 ✨
1 file would be left unchanged.
```

Can you check your version of black (should be `20.8b1`) and run `make style again`? (And don't forget to rebase before pushing ;) )

The other test was a time-out error so should be good on the next commit

Thanks @yjernite the CI tests finally passed!!

Hi @Bharat123rox did you manage to join the different config into one using the IDs ?

Feel free to ping me when you're ready for the next review :) 

> Hi @Bharat123rox did you manage to join the different config into one using the IDs ?
> 
> Feel free to ping me when you're ready for the next review :) 

Not yet @lhoestq still working on this! Meanwhile please review #1507 where I added the SelQA dataset :)

Ok ! Let me review SelQA then :) 
Thanks for your help !

Apologies for the very late response. Here is the openpi dataset file with a single file per partition after merging `id_answers, answers.jsonl, question.jsonl , question_metadata.jsonl`

https://github.com/allenai/openpi-dataset/blob/main/data/gold-v1.1/dev.jsonl

Nice thank you @nikett !

Hi @Bharat123rox , when you get a chance, please feel free to use the dataset from the repo ( [Link](https://github.com/allenai/openpi-dataset/blob/main/data/gold-v1.1/dev.jsonl) ) . Please let me know if any file is missing! Thank you 

Hi @Bharat123rox are you working on this? 

@nikett Sorry I'm no longer working on this as I'm out of time for it, please feel free to raise a new PR for this



We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest to create this dataset there. Please, feel free to tell us if you need some help.