helmo/github-issues huggingface dataset:
Add CHiME4 dataset
State: open | Author: patrickvonplaten (CONTRIBUTOR)
Labels: name: dataset request

name: speech

## Adding a Dataset
- **Name:** Chime4
- **Description:** Chime4 is a dataset for automatic speech recognition. It is especially useful for evaluating models in a noisy environment and for multi-channel ASR
- **Paper:** Dataset comes from a channel: http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/ . Results paper: 
- **Data:** http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/download.html
- **Motivation:** So far there are very little datasets for speech in `datasets`. Only `lbirispeech_asr` so far.

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments: @patrickvonplaten not sure whether it is still needed, but willing to tackle this issue

Hey @patrickvonplaten, I have managed to download the zip on [here]( http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/download.html) and successfully uploaded all the files on a hugging face dataset: 

https://huggingface.co/datasets/ksbai123/Chime4

However I am getting this error when trying to use the dataset viewer:

![Screenshot 2023-12-27 at 18 40 59](https://github.com/huggingface/datasets/assets/35923560/a5a9ed3d-8dbd-41c4-b83a-4e80728b1450)

Can you take a look and let me know if I have missed any files please

@patrickvonplaten ?

Hi @KossaiSbai,

Thanks for your contribution.

As the issue is not strictly related to the `datasets` library, but to the specific implementation of the CHiME4 dataset, I have opened an issue in the Discussion tab of the dataset: https://huggingface.co/datasets/ksbai123/Chime4/discussions/2
Let's continue the discussion there!

Hi, please help me understand this dataset. I am trying to run ESPnet chime4 recepie. I downloaded the dataset (https://www.chimechallenge.org/challenges/chime4/download) and it doesn't look like the official site (https://www.chimechallenge.org/challenges/chime4/data#Audio):

<img width="220" alt="Image" src="https://github.com/user-attachments/assets/91d2d727-bc65-4c45-ad87-b643fc26a40e" />

There is no 
├── audio
├── transcriptions
└── WSJ0
only 
├── annotations

same in your huggingface card. 
How can I access the missing data?
Add OSCAR dataset card
State: closed | Author: pjox (CONTRIBUTOR)

I added more information and completed the dataset card for OSCAR which was started by @lhoestq in his previous [PR](https://github.com/huggingface/datasets/pull/1824).

Comments: @lhoestq Thanks for the suggestions! I agree with all of them. Should I accept them one by one or can I accept them all at once? When I try to load the whole diff GitHub is complaining and it does no render them well (probably my browser?) 😅 

I just merged the tables as suggested 😄 . However I noticed something weird, the train sizes are identical for both the original and deduplicated files ... This is not normal, in general the original files are almost twice as big as the deduplicated ones 🤔 

Good catch @pjox ! I just checked and this is because the scripts doesn't handle having several blank lines in a row.
Blank lines introduced by deduplication are currently not ignored so we end up with the same number of examples in the dataset as the original version (but with empty examples...)
I fixed that in this [commit](https://github.com/huggingface/datasets/commit/837a152e4724adc5308e2c4481908c00a8d93383). I'm re-running the metadata generation for deduplicated configs.

I got the new sizes today, will update the dataset_infos.json and the dataset card tomorrow

> I got the new sizes today, will update the dataset_infos.json and the dataset card tomorrow

great, I just wanted to report that I got error message "NonMatchingSplitsSizesError" when I tried to load one of the oscar dataset.

Hi @cahya-wirawan, which configuration of oscar do you have this issue with ?

Ok I see you're having this issue because I haven't updated the sizes yet ! I'm opening a PR

I just checked and indeed there's an issue with the `deduplicated` configurations since the commit I mentioned above.
I'm fixing this by using the new sizes I got yesterday :) 


I just updated the size in the table @pjox it should be good now :) 
I also updated the sizes in the dataset_infos.json in https://github.com/huggingface/datasets/pull/1868 (merged)

Thanks @lhoestq for fixing the issue, it works now

Thank you so much @lhoestq !
Add Tweet Eval Dataset
State: closed | Author: gchhablani (CONTRIBUTOR)

Closes Draft PR #1407. 

Notes:
1. I have excluded `mapping.txt` from the dataset at it only contained the name mappings, which are already present in the ClassLabels.
2. I have also exluded the textual names for the emojis mentioned in the [mapping](https://github.com/cardiffnlp/tweeteval/blob/main/datasets/emoji/mapping.txt).
3. I do not understand @abhishekkrthakur's example generator on #1407. Maybe he was trying to build up on code from some other dataset.

Requesting @lhoestq  to review.
Regarding On-the-fly Data Loading
State: closed | Author: gchhablani (CONTRIBUTOR)

Hi,

I was wondering if it is possible to load images/texts as a batch during the training process, without loading the entire dataset on the RAM at any given point.

Thanks,
Gunjan

Comments: Possible duplicate

#1776 https://github.com/huggingface/datasets/issues/

really looking PR for this feature

Hi @acul3 

Issue #1776 talks about  doing on-the-fly data pre-processing, which I think is solved in the next release as mentioned in the issue #1825. I also look forward to using this feature, though :)

I wanted to ask about on-the-fly data loading from the cache (before pre-processing).

Hi ! Currently when you load a dataset via `load_dataset` for example, then the dataset is memory-mapped from an Arrow file on disk. Therefore there's almost no RAM usage even if your dataset contains TB of data.
Usually at training time only one batch of data at a time is loaded in memory.

Does that answer your question or were you thinking about something else ?

Hi @lhoestq,

I apologize for the late response. This answers my question. Thanks a lot.
Add OSCAR dataset card
State: closed | Author: lhoestq (MEMBER)

I started adding the dataset card for OSCAR !

For now it's just basic info for all the different configurations in `Dataset Structure`.
In particular the Data Splits section tells how may samples there are for each config. The Data Instances section show an example for each config, and it also shows the size in MB. Since the Data Instances section is very long the user has to click to expand the info. I was able to generate it thanks to the tools made by  @madlag and @yjernite :D

Cc @pjox could you help me with the other sections ? (Dataset Description, Dataset Creation, Considerations for Using the Data, Additional Information)


Comments: Hi @lhoestq! When are you planning to release the version with this dataset?

BTW: What a huge README file :astonished:

Next week !

Closing in favor of #1833
Provide better exception message when one of many files results in an exception
State: closed | Author: david-waterworth (NONE)

I find when I process many files, i.e.

```
train_files = glob.glob('rain*.csv')
validation_files = glob.glob(validation*.csv')
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files))
```

I sometimes encounter an error due to one of the files being misformed (i.e. no data, or a comma in a field that isn't quoted, etc).

For example, this is the tail of an exception which I suspect is due to a stray comma.

>   File "pandas/_libs/parsers.pyx", line 756, in pandas._libs.parsers.TextReader.read
>   File "pandas/_libs/parsers.pyx", line 783, in pandas._libs.parsers.TextReader._read_low_memory
>   File "pandas/_libs/parsers.pyx", line 827, in pandas._libs.parsers.TextReader._read_rows
>   File "pandas/_libs/parsers.pyx", line 814, in pandas._libs.parsers.TextReader._tokenize_rows
>   File "pandas/_libs/parsers.pyx", line 1951, in pandas._libs.parsers.raise_parser_error
> pandas.errors.ParserError: Error tokenizing data. C error: Expected 2 fields in line 559, saw 3

It would be nice if the exception trace contained the name of the file being processed (I have 250 separate files!)

Comments: Hi!

Thank you for reporting this issue. I agree that the information about the exception should be more clear and explicit.

I could take on this issue.

On the meantime, as you can see from the exception stack trace, HF Datasets uses pandas to read the CSV files. You can pass arguments to `pandas.read_csv` by passing additional keyword arguments to `load_dataset`. For example, you may find useful this argument:
- `error_bad_lines` : bool, default True
  Lines with too many fields (e.g. a csv line with too many commas) will by default cause an exception to be raised, and no DataFrame will be returned. If False, then these “bad lines” will be dropped from the DataFrame that is returned.

You could try:
```python
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files), error_bad_lines=False)
```
Add metrics usage examples and tests
State: closed | Author: lhoestq (MEMBER)

All metrics finally have usage examples and proper fast + slow tests :)

I added examples of usage for every metric, and I use doctest to make sure they all work as expected.

For "slow" metrics such as bert_score or bleurt which require to download + run a transformer model, the download + forward pass are only done in the slow test.
In the fast test on the other hand, the download + forward pass are monkey patched.

Metrics that need to be installed from github are not added to setup.py because it prevents uploading the `datasets` package to pypi.
An additional-test-requirements.txt file is used instead. This file also include `comet` in order to not have to resolve its *impossible* dependencies.

Also `comet` is not tested on windows because one of its dependencies (fairseq) can't be installed in the CI for some reason.
pyarrow.lib.ArrowInvalid: Column 1 named input_ids expected length 599 but got length 1500
State: closed | Author: LuCeHe (NONE)

I am trying to preprocess any dataset in this package with GPT-2 tokenizer, so I need to structure the datasets as long sequences of text without padding. I've been following a couple of your tutorials and here you can find the script that is failing right at the end

https://github.com/LuCeHe/GenericTools/blob/master/KerasTools/lm_preprocessing.py

In the last iteration of the last dset.map, it gives the error that I copied in the title. Another issue that I have, if I leave the batch_size set as 1000 in the last .map, I'm afraid it's going to lose most text, so I'm considering setting both writer_batch_size and batch_size to 300 K, but I'm not sure it's the best way to go.

Can you help me?
Thanks!

Comments: Hi !
The error you have is due to the `input_ids` column not having the same number of examples as the other columns.
Indeed you're concatenating the `input_ids` at this line:

https://github.com/LuCeHe/GenericTools/blob/431835d8e13ec24dceb5ee4dc4ae58f0e873b091/KerasTools/lm_preprocessing.py#L134

However the other columns are kept unchanged, and therefore you end up with an `input_ids` column with 599 elements while the others columns like `attention_mask` have 1500.

To fix that you can instead concatenate them all using
```python
concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
```

Also you may need to drop the "text" column before applying `group_texts` since strings can't be concatenated with lists. You can drop it at the tokenization step:
```python
dset = dset.map(
    tokenize_function,
    batched=True,
    remove_columns=["text"]
)
```

You saved my life.
Support future datasets
State: closed | Author: lhoestq (MEMBER)

If a dataset is available at the version of the local installation of `datasets` (e.g. 1.2.0), then loading this dataset means loading the script at this version.

However when trying to load a dataset that is only available on master, currently users have to specify `script_version="master"` in `load_dataset` to make it work.

However we could automatically get the dataset from master instead in this case.

I added this feature in this PR.
I also added a warning if a dataset is not available at the version of the local installation of `datasets` but is loaded from master:
```python
>>> load_dataset("silicone", "dyda_da")
Couldn't find file locally at silicone/silicone.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.2.0/datasets/silicone/silicone.py.
The file was picked from the master branch on github instead at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/silicone/silicone.py.
Downloading and preparing dataset silicone/dyda_da (download: 8.46 MiB, generated: 9.39 MiB, post-processed: Unknown size, total: 17.86 MiB) to /Users/quentinlhoest/.cache/huggingface/datasets/silicone/dyda_da/1.0.0/d41d8c0b73c6df035b1369c45774418f0051163ea689b5502b8bda783adf6342...
...
```
Unable to add Multi-label Datasets
State: closed | Author: gchhablani (CONTRIBUTOR)

I am trying to add [CIFAR-100](https://www.cs.toronto.edu/~kriz/cifar.html) dataset. The dataset contains two labels per image - `fine label` and `coarse label`. Using just one label in supervised keys as 
`supervised_keys=("img", "fine_label")` raises no issue. But trying `supervised_keys=("img", "fine_label","coarse_label")` leads to this error : 

```python
Traceback (most recent call last):
  File "test_script.py", line 2, in <module>
    d = load_dataset('./datasets/cifar100')
  File "~/datasets/src/datasets/load.py", line 668, in load_dataset
    **config_kwargs,
  File "~/datasets/src/datasets/builder.py", line 896, in __init__
    super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
  File "~/datasets/src/datasets/builder.py", line 247, in __init__
    info.update(self._info())
  File "~/.cache/huggingface/modules/datasets_modules/datasets/cifar100/61d2489b2d4a4abc34201432541b7380984ec714e290817d9a1ee318e4b74e0f/cifar100.py", line 79, in _info
    citation=_CITATION,
  File "<string>", line 19, in __init__
  File "~/datasets/src/datasets/info.py", line 136, in __post_init__
    self.supervised_keys = SupervisedKeysData(*self.supervised_keys)
TypeError: __init__() takes from 1 to 3 positional arguments but 4 were given
```
Is there a way I can fix this?

Also, what does adding `supervised_keys` do? Is it necessary? How would I specify `supervised_keys` for a multi-input, multi-label dataset?

Thanks,
Gunjan

Comments: Thanks for adding this dataset! As far as I know `supervised_keys` is mostly a holdover from TFDS, but isn't really used, so feel free to drop it (@lhoestq  or @thomwolf correct me if I'm wrong). It definitely shouldn't be blocking :) 

I can confirm that it comes from TFDS and is not used at the moment.

Thanks @yjernite @lhoestq 

The template for new dataset makes it slightly confusing. I suppose the comment suggesting its update can be removed.

Closing this issue since it was answered.
Add Hateful Memes Dataset
State: open | Author: gchhablani (CONTRIBUTOR)
Labels: name: dataset request

name: vision

## Add Hateful Memes Dataset
- **Name:** Hateful Memes
- **Description:** [https://ai.facebook.com/blog/hateful-memes-challenge-and-data-set]( https://ai.facebook.com/blog/hateful-memes-challenge-and-data-set)
- **Paper:** [https://arxiv.org/pdf/2005.04790.pdf](https://arxiv.org/pdf/2005.04790.pdf)
- **Data:** [This link](https://drivendata-competition-fb-hateful-memes-data.s3.amazonaws.com/XjiOc5ycDBRRNwbhRlgH.zip?AWSAccessKeyId=AKIARVBOBDCY4MWEDJKS&Signature=DaUuGgZWUgDHzEPPbyJ2PhSJ56Q%3D&Expires=1612816874)
- **Motivation:** Including multi-modal datasets to 🤗 datasets.

I will be adding this dataset. It requires the user to sign an agreement on DrivenData. So, it will be used with a manual download.

The issue with this dataset is that the images are of different sizes. The image datasets added so far (CIFAR-10 and MNIST) have a uniform shape throughout.
So something like 
```python
 datasets.Array2D(shape=(28, 28), dtype="uint8")
```
won't work for the images. How would I add image features then? I checked `datasets/features.py` but couldn't figure out the appropriate class for this. I'm assuming I would want to avoid re-sizing at all since we want the user to be able to access the original images.

Also, in case I want to load only a subset of the data, since the actual data is around 8.8GB, how would that be possible?

Thanks,
Gunjan

Comments: I am not sure, but would `datasets.Sequence(datasets.Sequence(datasets.Sequence(datasets.Value("int")))` work?

Also, I found the information for loading only subsets of the data [here](https://github.com/huggingface/datasets/blob/master/docs/source/splits.rst).

Hi @lhoestq,

Request you to check this once.

Thanks,
Gunjan

Hi @gchhablani since Array2D doesn't support images of different sizes, I would suggest to store in the dataset the paths to the image file instead of the image data. This has the advantage of not decompressing the data (images are often compressed using jpeg, png etc.). Users can still apply `.map` to load the images if they want to. Though it would en up being Sequences features.

In the future we'll add support for ragged tensors for this case and update the relevant dataset with this feature.
Add FreebaseQA dataset
State: closed | Author: gchhablani (CONTRIBUTOR)

Adding FreebaseQA dataset suggested in PR #1435 with minor edits. Also closes that PR.
Requesting @lhoestq to review.

Comments: Hi ! It looks like this PR contains changes about other datasets than freebase_qa such as DuoRC.

Can you remove these changes please ?

Hi @lhoestq,

I think this happened because of rebasing. I'm unable to remove the duorc commit from the branch.  GEM, Arabic sarcasm datasets are also there. I can't see any merge conflicts, however. Before commiting I always rebase (shouldn't have done that).
Can you explain what is to be done? Should I create a clean PR?

Hi @gchhablani 
I think you can simply create another branch and another PR.

If I understand correctly the github diff is messed up because you rebased instead of merge.
Rebasing is supposed to be used only before pushing the branch the first time, or github messes up the diff.
If you want to include changes from master on a branch that is already push you need to use git merge.

Thanks @lhoestq.

I understand the issue now. I missed the instructions on the template. Sorry for bothering you unnecessarily, I'm pretty new to contributing on GitHub. I'll make a fresh PR.


No problem, I'm not a big fan of this weird behavior tbh.
Thanks for making a new PR

@lhoestq Haha, well, it's not as weird as not reading the [instructions](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#open-a-pull-request-on-the-main-huggingface-repo-and-share-your-work).
Also, I'm enjoying adding new datasets so it's all cool :)
writing Datasets in a human readable format
State: closed | Author: ghost (NONE)
Labels: name: enhancement

name: question

Hi
I see there is a save_to_disk function to save data, but this is not human readable format, is there a way I could save a Dataset object in a human readable  format to a file like json? thanks @lhoestq 

Comments: AFAIK, there is currently no built-in method on the `Dataset` object to do this.
However, a workaround is to directly use the Arrow table backing the dataset, **but it implies loading the whole dataset in memory** (correct me if I'm mistaken @lhoestq).

You can convert the Arrow table to a pandas dataframe to save the data as csv as follows:
```python
arrow_table = dataset.data
dataframe = arrow_table.to_pandas()
dataframe.to_csv("/path/to/file.csv")
```

Similarly, you can convert the dataset to a Python dict and save it as JSON:
```python
import json
arrow_table = dataset.data
py_dict = arrow_table.to_pydict()
with open("/path/to/file.json", "w+") as f:
    json.dump(py_dict, f)
```

Indeed this works as long as you have enough memory.
It would be amazing to have export options like csv, json etc. !

It should be doable to implement something that iterates through the dataset batch by batch to write to csv for example.
There is already an `export` method but currently the only export type that is supported is `tfrecords`.

Hi! `datasets` now supports `Dataset.to_csv` and `Dataset.to_json` for saving data in a human readable format.
Adding an aggregated dataset for the GEM benchmark
State: closed | Author: yjernite (MEMBER)

This dataset gathers modified versions of several other conditional text generation datasets which together make up the shared task for the Generation Evaluation and Metrics workshop (think GLUE for text generation)

The changes from the original datasets are detailed in the Dataset Cards on the GEM website, which are linked to in this dataset card.

cc @sebastianGehrmann


Comments: Nice !
add github of contributors
State: closed | Author: thevasudevgupta (CONTRIBUTOR)

This PR will add contributors GitHub id at the end of every dataset cards.

Comments: @lhoestq Can you confirm if this format is fine? I will update cards based on your feedback.

On HuggingFace side we also have a mapping of hf user => github user (GitHub info used to be required when signing up until not long ago – cc @gary149 @beurkinger) so we can also add a link to HF profile

All the dataset cards have been updated with GitHub ids :)
[GEM] Updated the source link of the data to update correct tokenized version.
State: closed | Author: mounicam (CONTRIBUTOR)


Comments: @mounicam we'll keep the original version in the Turk dataset proper, and use the updated file in the GEM aggregated dataset which I'll add later today

@lhoestq do not merge, I'll close when I've submitted the GEM dataset PR :) 

Closed by https://github.com/huggingface/datasets/pull/1807
Add DuoRC Dataset
State: closed | Author: gchhablani (CONTRIBUTOR)

Hi,

DuoRC SelfRC is one type of the [DuoRC Dataset](https://duorc.github.io/). DuoRC SelfRC is a crowdsourced Abstractive/Extractive Question-Answering dataset based on Wikipedia movie plots. It contains examples that may have answers in the movie plot, synthesized answers which are not present in the movie plot, or no answers.  I have also added ParaphraseRC - the other type of DuoRC dataset where questions are based on Wikipedia movie plots and answers are based on corresponding IMDb movie plots.

Paper :  [https://arxiv.org/abs/1804.07927](https://arxiv.org/abs/1804.07927)

I want to add this to 🤗 datasets to make it more accessible to the community. I have added  all the details that I could find. Please let me know if anything else is needed from my end.

Thanks,
Gunjan


Comments: Thanks for approving @lhoestq!
Will apply these changes for the other datasets I've added too.
Update: SWDA - Fixed code to use all metadata features. Added comments and cleaned c…
State: closed | Author: gmihaila (CONTRIBUTOR)

This is a dataset I currently use my research and I realized some features are not being returned.

Previous code was not using all available metadata and was kind of messy

I fixed code to use all metadata and made some modification to be more efficient and better formatted.


Please let me know if I need to make any changes.

Comments: @yjernite Pushed all the changes you recommended. Thank you for your help!
Connection error
State: closed | Author: smile0925 (NONE)

Hi
I am hitting to the error, help me and thanks.

`train_data = datasets.load_dataset("xsum", split="train")`
`ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.0.2/datasets/xsum/xsum.py`

Comments: Hi ! For future references let me add a link to our discussion here : https://github.com/huggingface/datasets/issues/759#issuecomment-770684693

Let me know if you manage to fix your proxy issue or if we can do something on our end to help you :)