konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
Add LJ Speech dataset

This PR adds the LJ Speech dataset (https://keithito.com/LJ-Speech-Dataset/)
As requested by #1841 
The ASR format is based on #1767 

There are a couple of quirks that should be addressed:
- I tagged this dataset as `other-other-automatic-speech-recognition` and `other-other-text-to-speech` (as classified by paperswithcode). Since the number of speech datasets is about to grow, maybe these categories should be added to the main list? 
- Similarly to #1767 this dataset uses only a single dummy sample to reduce the zip size (`wav`s are quite heavy). Is there a plan to allow LFS or S3 usage for dummy data in the repo?
- The dataset is distributed under the Public Domain license, which is not used anywhere else in the repo, AFAIK. Do you think Public Domain is worth adding to the tagger app as well?

Pinging @patrickvonplaten to review

Hey @anton-l,

Thanks a lot for the very clean integration!

1) I think we should now start having "automatic-speech-recognition" as a label in the dataset tagger (@yjernite is it easy to add?). But we can surely add this dataset with the tag you've added and then later change the label to `asr` 

2) That's perfect! Yeah good question - we're currently thinking about a better design with @lhoestq 

3) Again tagging @yjernite & @lhoestq here - guess we should add this license though!

Thanks @anton-l for adding this one :)
About the points you mentioned:
1. Sure as soon as we've updated the tag sets in https://github.com/huggingface/datasets-tagging/blob/main/task_set.json, we can update the tags in this dataset card and also in the other audio dataset card.
2. For now we just try to have them as small as possible but we may switch to S3/LFS at one point indeed
3. If it's not part of the license set at https://github.com/huggingface/datasets-tagging/blob/main/license_set.json we can add it to this license set

For now it's ok to have the other-* tags but we'll update them very soon

Let's merge this one and then we'll update the tags for the audio datasets. We'll probably also add something like this:
```
type:
- text
- audio
```

Thank you so much for adding this one, good job !
Implement Dataset add_item

Implement `Dataset.add_item`.

Close #1854.

Thanks @lhoestq for your remarks. Yes, I agree there are still many issues to be tackled... This PR is just a starting point, so that we can discuss how Dataset should be generalized.

Sure ! I opened an issue #1877 so we can discuss this specific aspect :)

I am going to implement this consolidation step in #2151.

Sounds good !

I retake this PR once the consolidation step is already implemented by #2151.
Updated OPUS Open Subtitles Dataset with metadata information

Close #1844 

Problems:
- I ran `python datasets-cli test datasets/open_subtitles --save_infos --all_configs`, hence the change in `dataset_infos.json`, but it appears that the metadata features have not been added for all pairs. Any idea why that might be?
- Possibly related to the above, I tried doing `pip uninstall datasets && pip install -e ".[dev]"` after the changes, and loading the dataset via `load_dataset("open_subtitles", lang1='hi', lang2='it')` to check if the update worked, but the loaded dataset did not contain the metadata fields (neither in the features nor doing `next(iter(dataset['train']))`). What step(s) did I miss?

Questions:
- Is it ok to have a `classmethod` in there? I have not seen any in the few other datasets I have checked. I could make it a local method of the `_generate_examples` method, but I'd rather not duplicate the logic...

Hi !
About the problems you mentioned:
- Saving the infos is only done for the configurations inside the BUILDER_CONFIGS. Otherwise you would need to run the scripts on ALL language pairs, which is not what we want.
- Moreover when you're on your branch, please specify the path to your local version of the dataset script, like "./datasets/open_subtitles". Otherwise the dataset is loaded from the master branch on github.
Hope that clarifies things a bit

And of course feel free to add methods or classmethods to your builder.


Great! Thank you :)
I'll close the issue as well.
Add Winogender Schemas

## Adding a Dataset
- **Name:** Winogender Schemas
- **Description:** Winogender Schemas (inspired by Winograd Schemas) are minimal pairs of sentences that differ only by the gender of one pronoun in the sentence, designed to test for the presence of gender bias in automated coreference resolution systems.
- **Paper:** https://arxiv.org/abs/1804.09301
- **Data:** https://github.com/rudinger/winogender-schemas (see data directory)
- **Motivation:** Testing gender bias in automated coreference resolution systems, improve coreference resolution in general.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Nevermind, this one is already available on the hub under the name `'wino_bias'`: https://huggingface.co/datasets/wino_bias
Add WikiCREM

## Adding a Dataset
- **Name:** WikiCREM
- **Description:** A large unsupervised corpus for coreference resolution.
- **Paper:** https://arxiv.org/abs/1905.06290
- **Github repo:**: https://github.com/vid-koci/bert-commonsense
- **Data:** https://ora.ox.ac.uk/objects/uuid:c83e94bb-7584-41a1-aef9-85b0e764d9e3
- **Motivation:** Coreference resolution, common sense reasoning

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Hi @NielsRogge I would like to work on this dataset.

Thanks!

Hi @udapy, are you working on this?
Add loading from the Datasets Hub + add relative paths in download manager

With the new Datasets Hub on huggingface.co it's now possible to have a dataset repo with your own script and data.
For example: https://huggingface.co/datasets/lhoestq/custom_squad/tree/main contains one script and two json files.

You can load it using
```python
from datasets import load_dataset

d = load_dataset("lhoestq/custom_squad")
```

To be able to use the data files that live right next to the dataset script on the repo in the hub, I added relative paths support for the DownloadManager. For example in the repo mentioned above, there are two json files that can be downloaded via
```python
_URLS = {
    "train": "train-v1.1.json",
    "dev": "dev-v1.1.json",
}
downloaded_files = dl_manager.download_and_extract(_URLS)
```

To make it work, I set the `base_path` of the DownloadManager to be the parent path of the dataset script (which comes from either a local path or a remote url).

I also had to add the auth header of the requests to huggingface.co for private datasets repos. The token is fetched from [huggingface_hub](https://github.com/huggingface/huggingface_hub).

I just added the steps to share a dataset on the datasets hub. It's highly inspired by the steps to share a model in the `transformers` doc.

Moreover once the new huggingface_hub is released we can update the version in the setup.py. We also need to update the command to create a dataset repo in the documentation

I added a few more tests with the "lhoestq/test" dataset I added on the hub and it works fine :) 

Here is the PR adding support for datasets repos in `huggingface_hub`: https://github.com/huggingface/huggingface_hub/pull/14
Error "in void don't know how to serialize this type of index" when saving index to disk when device=0 (GPU)

Error serializing faiss index.  Error as follows:

`Error in void faiss::write_index(const faiss::Index*, faiss::IOWriter*) at /home/conda/feedstock_root/build_artifacts/faiss-split_1612472484670/work/faiss/impl/index_write.cpp:453: don't know how to serialize this type of index`


Note:

`torch.cuda.is_available()` reports:

```
Cuda is available
cuda:0

```

Adding index, device=0 for GPU.

`dataset.add_faiss_index(column='embeddings', index_name='idx_embeddings', device=0)`

However, during a quick debug, self.faiss_index has no attr "device" when checked in` search.py, method save`, so fails to transform gpu index to cpu index.  If I add index without device, index is saved OK.


```
def save(self, file: str):
        """Serialize the FaissIndex on disk"""
        import faiss  # noqa: F811

        if (
            hasattr(self.faiss_index, "device")
            and self.faiss_index.device is not None
            and self.faiss_index.device > -1
        ):
            index = faiss.index_gpu_to_cpu(self.faiss_index)
        else:
            index = self.faiss_index
        faiss.write_index(index, file)
```


Hi @corticalstack ! Thanks for reporting. Indeed in the recent versions of Faiss we must use `getDevice` to check if the index in on GPU.

I'm opening a PR

I fixed this issue. It should work fine now.
Feel free to try it out by installing `datasets` from source.
Otherwise you can wait for the next release of `datasets` (in a few days)

Thanks for such a quick fix and merge to master, pip installed git master, tested all OK
Unable to upload "community provided" dataset - 400 Client Error

Hi,
i'm trying to a upload a dataset as described [here](https://huggingface.co/docs/datasets/v1.2.0/share_dataset.html#sharing-a-community-provided-dataset). This is what happens:

``` 
$ datasets-cli login
$ datasets-cli upload_dataset my_dataset
About to upload file /path/to/my_dataset/dataset_infos.json to S3 under filename my_dataset/dataset_infos.json and namespace username
About to upload file /path/to/my_dataset/my_dataset.py to S3 under filename my_dataset/my_dataset.py and namespace username
Proceed? [Y/n] Y
Uploading... This might take a while if files are large
400 Client Error: Bad Request for url: https://huggingface.co/api/datasets/presign
huggingface.co migrated to a new model hosting system.
You need to upgrade to transformers v3.5+ to upload new models.
More info at https://discuss.hugginface.co or https://twitter.com/julien_c. Thank you! 
```
I'm using the latest releases of datasets and transformers.

Hi ! We're in the process of switching the community datasets to git repos, exactly like what we're doing for models.
You can find an example here:
https://huggingface.co/datasets/lhoestq/custom_squad/tree/main

We'll update the CLI in the coming days and do a new release :)

Also cc @julien-c maybe we can make improve the error message ?
Feature Request: Dataset.add_item

I'm trying to integrate `huggingface/datasets` functionality into `fairseq`, which requires (afaict) being able to build a dataset through an `add_item` method, such as https://github.com/pytorch/fairseq/blob/master/fairseq/data/indexed_dataset.py#L318, as opposed to loading all the text into arrow, and then `dataset.map(binarizer)`.
Is this possible at the moment? Is there an example? I'm happy to use raw `pa.Table` but not sure whether it will support uneven length entries.

### Desired API

```python
import numpy as np
tokenized: List[np.NDArray[np.int64]] = [np.array([4,4,2]), np.array([8,6,5,5,2]), np.array([3,3,31,5])

def build_dataset_from_tokenized(tokenized: List[np.NDArray[int]]) -> Dataset:
   """FIXME"""
   dataset = EmptyDataset()
   for t in tokenized: dataset.append(t)
   return dataset
ds = build_dataset_from_tokenized(tokenized)
assert (ds[0] == np.array([4,4,2])).all()
```

### What I tried
grep, google for "add one entry at a time", "datasets.append"

### Current Code
This code achieves the same result but doesn't fit into the `add_item` abstraction.

```python
    dataset = load_dataset('text', data_files={'train': 'train.txt'})
    tokenizer = RobertaTokenizerFast.from_pretrained('roberta-base', max_length=4096)
    def tokenize_function(examples):
        ids = tokenizer(examples['text'], return_attention_mask=False)['input_ids']
        return {'input_ids': [x[1:] for x in ids]}
    ds = dataset.map(tokenize_function, batched=True, num_proc=4, remove_columns=['text'], load_from_cache_file=not overwrite_cache)
	print(ds['train'][0]) => np array
```

Thanks in advance!

Hi @sshleifer.

I am not sure of understanding the need of the `add_item` approach...

By just reading your "Desired API" section, I would say you could (nearly) get it with a 1-column Dataset:
```python
data = {"input_ids": [np.array([4,4,2]), np.array([8,6,5,5,2]), np.array([3,3,31,5])]}
ds = Dataset.from_dict(data)
assert (ds["input_ids"][0] == np.array([4,4,2])).all()
```

Hi @sshleifer :) 

We don't have methods like `Dataset.add_batch` or `Dataset.add_entry/add_item` yet.
But that's something we'll add pretty soon. Would an API that looks roughly like this help ? Do you have suggestions ?
```python
import numpy as np
from datasets import Dataset

tokenized = [np.array([4,4,2]), np.array([8,6,5,5,2]), np.array([3,3,31,5])

# API suggestion (not available yet)
d = Dataset()
for input_ids in tokenized:
    d.add_item({"input_ids": input_ids})

print(d[0]["input_ids"])
# [4, 4, 2]
```

Currently you can define a dataset with what @albertvillanova suggest, or via a generator using dataset builders. It's also possible to [concatenate datasets](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=concatenate#datasets.concatenate_datasets).

Your API looks perfect @lhoestq, thanks!
Add cord 19 dataset

Initial version only reading the metadata in CSV.

### Checklist:
- [x] Create the dataset script /datasets/my_dataset/my_dataset.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _infos(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template and at least fill the tags
- [x] Both tests for the real data and the dummy data pass.

### Extras:
- [x] add more metadata
- [x] add full text
- [x] add pre-computed document embedding

Cleaned-up version of previous PR: https://github.com/huggingface/datasets/pull/1129

@lhoestq FYI

Before merging I might tweak a little bit the dummy data to avoid having to check if the `document_parses` and `embeddings` directories exist or not. I'll do that later today

Looks all good now ! Thanks a lot @ggdupont :)
Merging
Add TIMIT

## Adding a Dataset
- **Name:** *TIMIT*
- **Description:** *The TIMIT corpus of read speech has been designed to provide speech data for the acquisition of acoustic-phonetic knowledge and for the development and evaluation of automatic speech recognition systems*

- **Paper:** *Homepage*: http://groups.inf.ed.ac.uk/ami/corpus/ / *Wikipedia*: https://en.wikipedia.org/wiki/TIMIT
- **Data:** *https://deepai.org/dataset/timit*
- **Motivation:** Important speech dataset


If interested in tackling this issue, feel free to tag @patrickvonplaten


Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


@patrickvonplaten Could you please help me with how the output text has to be represented in the data? TIMIT has Words, Phonemes and texts. Also has lot on info on the speaker and the dialect. Could you please help me? An example of how to arrange it would be super helpful!



Hey @vrindaprabhu - sure I'll help you :-) Could you open a first PR for TIMIT where you copy-paste more or less the `librispeech_asr` script: https://github.com/huggingface/datasets/blob/28be129db862ec89a87ac9349c64df6b6118aff4/datasets/librispeech_asr/librispeech_asr.py#L93 (obviously replacing all the naming and links correctly...) and then you can list all possible outputs in the features dict: https://github.com/huggingface/datasets/blob/28be129db862ec89a87ac9349c64df6b6118aff4/datasets/librispeech_asr/librispeech_asr.py#L104 (words, phonemes should probably be of kind `datasets.Sequence(datasets.Value("string"))` and texts I think should be of type `"text": datasets.Value("string")`.

When you've opened a first PR, I think it'll be much easier for us to take a look together :-) 

I am sorry! I created the PR [#1903](https://github.com/huggingface/datasets/pull/1903#). Requesting your comments! CircleCI tests are failing, will address them along with your comments!
[Metrics] Add word error metric metric

This PR adds the word error rate metric to datasets. 
WER: https://en.wikipedia.org/wiki/Word_error_rate
for speech recognition. WER is the main metric used in ASR. 

`jiwer` seems to be a solid library (see https://github.com/asteroid-team/asteroid/pull/329#discussion_r525158939)

Feel free to merge once the CI is all green ;)
Make DownloadManager downloaded/extracted paths accessible

Make accessible the file paths downloaded/extracted by DownloadManager.

Close #1831.

The approach:
- I set these paths as DownloadManager attributes: these are DownloadManager's concerns
- To access to these from DatasetBuilder, I set the DownloadManager instance as DatasetBuilder attribute: object composition

First I was thinking of the dict, which makes sense for .download, mapping URL to downloaded path. However does this make sense for .extract, mapping the downloaded path to the extracted path? I ask this because the user did not chose the downloaded path, so this is completely unknown for them...

There could be several situations:
- download a file with no extraction
- download a file and extract it
- download a file, extract it and then inside the output folder extract some more files
- extract a local file (for datasets with data that are manually downloaded for example)
- extract a local file, and then inside the output folder extract some more files

So I think it's ok to have `downloaded_paths` as a dict url -> downloaded_path and `extracted_paths` as a dict local_path -> extracted_path.

OK. I am refactoring this. I have opened #1879, as an intermediate step...
Enable logging propagation and remove logging handler

We used to have logging propagation disabled because of this issue: https://github.com/tensorflow/tensorflow/issues/26691
But since it's now fixed we should re-enable it. This is important to keep the default logging behavior for users, and propagation is also needed for pytest fixtures as asked in #1826 

I also removed the handler that was added since, according to the logging [documentation](https://docs.python.org/3/howto/logging.html#configuring-logging-for-a-library):
> It is strongly advised that you do not add any handlers other than NullHandler to your library’s loggers. This is because the configuration of handlers is the prerogative of the application developer who uses your library. The application developer knows their target audience and what handlers are most appropriate for their application: if you add handlers ‘under the hood’, you might well interfere with their ability to carry out unit tests and deliver logs which suit their requirements.

It could have been useful if we wanted to have a custom formatter for the logging but I think it's more important to keep the logging as default to not interfere with the users' logging management.

Therefore I also removed the two methods `datasets.logging.enable_default_handler` and `datasets.logging.disable_default_handler`.

cc @albertvillanova this should let you use capsys/caplog in pytest
cc @LysandreJik @sgugger if you want to do the same in `transformers`

Thank you @lhoestq. This logging configuration makes more sense to me.

Once propagation is allowed, the end-user can customize logging behavior and add custom handlers to the proper top logger in the hierarchy.

And I also agree with following the best practices and removing any custom handlers:
- it is the end user who has to implement any custom handlers
- indeed, the previous logging problem with TensorFlow was due to the fact that absl did not follow best practices and had implemented a custom handler

Our errors/warnings will be displayed anyway, even if we do not implement any custom handler. Since Python 3.2, logging has a built-in "default" handler (logging.lastResort) with the expected default behavior (sending error/warning messages to sys.stderr), which is used only if the end user has not configured any custom handler.
Add AMI Corpus

## Adding a Dataset
- **Name:** *AMI*
- **Description:** *The AMI Meeting Corpus is a multi-modal data set consisting of 100 hours of meeting recordings. For a gentle introduction to the corpus, see the corpus overview. To access the data, follow the directions given there. Around two-thirds of the data has been elicited using a scenario in which the participants play different roles in a design team, taking a design project from kick-off to completion over the course of a day. The rest consists of naturally occurring meetings in a range of domains. Detailed information can be found in the documentation section.*

- **Paper:** *Homepage*: http://groups.inf.ed.ac.uk/ami/corpus/
- **Data:** *http://groups.inf.ed.ac.uk/ami/download/* - Select all cases in 1) and select "Individual Headsets" & "Microphone array" for 2)
- **Motivation:** Important speech dataset


If interested in tackling this issue, feel free to tag @patrickvonplaten


Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Available here: ~https://huggingface.co/datasets/ami~ https://huggingface.co/datasets/edinburghcstr/ami

@mariosasko actually the "official" AMI dataset can be found here: https://huggingface.co/datasets/edinburghcstr/ami -> the old one under `datasets/ami` doesn't work and should be deleted. 

The new one was tested by fine-tuning a Wav2Vec2 model on it + we uploaded all the processed audio directly into it

@patrickvonplaten Thanks for correcting me! I've updated the link.
Add ljspeech

## Adding a Dataset
- **Name:** *ljspeech*
- **Description:** *This is a public domain speech dataset consisting of 13,100 short audio clips of a single speaker reading passages from 7 non-fiction books. A transcription is provided for each clip. Clips vary in length from 1 to 10 seconds and have a total length of approximately 24 hours.

The texts were published between 1884 and 1964, and are in the public domain. The audio was recorded in 2016-17 by the LibriVox project and is also in the public domain.)*
- **Paper:** *Homepage*: https://keithito.com/LJ-Speech-Dataset/
- **Data:** *https://keithito.com/LJ-Speech-Dataset/*
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/ljspeech
If interested in tackling this issue, feel free to tag @patrickvonplaten


Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Add common voice

## Adding a Dataset
- **Name:** *common voice*
- **Description:** *Mozilla Common Voice Dataset*
- **Paper:** Homepage: https://voice.mozilla.org/en/datasets
- **Data:** https://voice.mozilla.org/en/datasets
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/common_voice
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


I have started working on adding this dataset.

Hey @BirgerMoell - awesome that you started working on Common Voice. Common Voice is a bit special since, there is no direct download link to download the data. In these cases we usually consider two options:

1) Find a hacky solution to extract the download link somehow from the XLM tree of the website 
2) If this doesn't work we force the user to download the data himself and add a `"data_dir"` as an input parameter. E.g. you can take a look at how it is done for [this](https://github.com/huggingface/datasets/blob/66f2a7eece98d2778bd22bb5034cb7c2376032d4/datasets/arxiv_dataset/arxiv_dataset.py#L66) 

Also the documentation here: https://huggingface.co/docs/datasets/add_dataset.html?highlight=data_dir#downloading-data-files-and-organizing-splits (especially the "note") might be helpful.

Let me know if you have any other questions

I added a Work in Progress pull request (hope that is ok). I've made a card for the dataset and filled out the common_voice.py file with information about the datset (not completely).

I didn't manage to get the tagging tool working locally on my machine but will look into that later.

Left to do.

- Tag the dataset
- Add missing information and update common_voice.py

https://github.com/huggingface/datasets/pull/1886

Awesome! I left a longer comment on the PR :-)

I saw that this current datasets package holds common voice version 6.1, how to add the new version 7.0 that is already available?

Will me merged next week - we're working on it :-)

Common voice still appears to be a 6.1. Is the plan still to upgrade to 7.0?

We actually already have the code and everything ready to add Common Voice 7.0 to `datasets` but are still waiting for the common voice authors to give us the green light :-) 

Also gently pinging @phirework and @milupo here

Common Voice 7.0 is available here now: https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0

For anyone else stumbling upon this thread, the 8.0 version is also available now: https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0
Add Voxforge

## Adding a Dataset
- **Name:** *voxforge* 
- **Description:** *VoxForge is a language classification dataset. It consists of user submitted audio clips submitted to the website. In this release, data from 6 languages is collected - English, Spanish, French, German, Russian, and Italian. Since the website is constantly updated, and for the sake of reproducibility, this release contains only recordings submitted prior to 2020-01-01. The samples are splitted between train, validation and testing so that samples from each speaker belongs to exactly one split.*
- **Paper:** *Homepage*: http://www.voxforge.org/
- **Data:** *http://www.voxforge.org/home/downloads*
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/voxforge
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Add tedlium

## Adding a Dataset
- **Name:** *tedlium*
- **Description:** *The TED-LIUM 1-3 corpus is English-language TED talks, with transcriptions, sampled at 16kHz. It contains about 118 hours of speech.*
- **Paper:** Homepage: http://www.openslr.org/7/, https://lium.univ-lemans.fr/en/ted-lium2/ &, https://www.openslr.org/51/
- **Data:** http://www.openslr.org/7/
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/tedlium
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Hi @patrickvonplaten 
I can have a look to this dataset later since I am trying to add the OpenSLR dataset https://github.com/huggingface/datasets/pull/2173
Hopefully I have enough space since the compressed file is 21GB. The release 3 is even bigger: 54GB :-0

Resolved via https://github.com/huggingface/datasets/pull/4309
Add VCTK

## Adding a Dataset
- **Name:** *VCTK*
- **Description:** *This CSTR VCTK Corpus includes speech data uttered by 110 English speakers with various accents. Each speaker reads out about 400 sentences, which were selected from a newspaper, the rainbow passage and an elicitation paragraph used for the speech accent archive.*
- **Paper:** Homepage: https://datashare.ed.ac.uk/handle/10283/3443
- **Data:** https://datashare.ed.ac.uk/handle/10283/3443
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/vctk

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


@patrickvonplaten I'd like to take this, if nobody has already done it. I have added datasets before through the datasets sprint, but I feel rusty on the details, so I'll look at the guide as well as similar audio PRs (#1878 in particular comes to mind). If there is any detail I should be aware of please, let me know! Otherwise, I'll try to write up a PR in the coming days.

That sounds great @jaketae - let me know if you need any help i.e. feel free to ping me on a first PR :-)
test.json has been removed from the limit dataset repo (breaks dataset)

https://github.com/huggingface/datasets/blob/16042b233dbff2a7585110134e969204c69322c3/datasets/limit/limit.py#L51

The URL is not valid anymore since test.json has been removed in master for some reason. Directly referencing the last commit works:

`https://raw.githubusercontent.com/ilmgut/limit_dataset/0707d3989cd8848f0f11527c77dcf168fefd2b23/data`

Thanks for the heads up ! I'm opening a PR to fix that
Add CHiME4 dataset

## Adding a Dataset
- **Name:** Chime4
- **Description:** Chime4 is a dataset for automatic speech recognition. It is especially useful for evaluating models in a noisy environment and for multi-channel ASR
- **Paper:** Dataset comes from a channel: http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/ . Results paper: 
- **Data:** http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/download.html
- **Motivation:** So far there are very little datasets for speech in `datasets`. Only `lbirispeech_asr` so far.

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


@patrickvonplaten not sure whether it is still needed, but willing to tackle this issue

Hey @patrickvonplaten, I have managed to download the zip on [here]( http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/download.html) and successfully uploaded all the files on a hugging face dataset: 

https://huggingface.co/datasets/ksbai123/Chime4

However I am getting this error when trying to use the dataset viewer:

![Screenshot 2023-12-27 at 18 40 59](https://github.com/huggingface/datasets/assets/35923560/a5a9ed3d-8dbd-41c4-b83a-4e80728b1450)

Can you take a look and let me know if I have missed any files please

@patrickvonplaten ?

Hi @KossaiSbai,

Thanks for your contribution.

As the issue is not strictly related to the `datasets` library, but to the specific implementation of the CHiME4 dataset, I have opened an issue in the Discussion tab of the dataset: https://huggingface.co/datasets/ksbai123/Chime4/discussions/2
Let's continue the discussion there!

Hi, please help me understand this dataset. I am trying to run ESPnet chime4 recepie. I downloaded the dataset (https://www.chimechallenge.org/challenges/chime4/download) and it doesn't look like the official site (https://www.chimechallenge.org/challenges/chime4/data#Audio):

<img width="220" alt="Image" src="https://github.com/user-attachments/assets/91d2d727-bc65-4c45-ad87-b643fc26a40e" />

There is no 
├── audio
├── transcriptions
└── WSJ0
only 
├── annotations

same in your huggingface card. 
How can I access the missing data?
Add OSCAR dataset card

I added more information and completed the dataset card for OSCAR which was started by @lhoestq in his previous [PR](https://github.com/huggingface/datasets/pull/1824).

@lhoestq Thanks for the suggestions! I agree with all of them. Should I accept them one by one or can I accept them all at once? When I try to load the whole diff GitHub is complaining and it does no render them well (probably my browser?) 😅 

I just merged the tables as suggested 😄 . However I noticed something weird, the train sizes are identical for both the original and deduplicated files ... This is not normal, in general the original files are almost twice as big as the deduplicated ones 🤔 

Good catch @pjox ! I just checked and this is because the scripts doesn't handle having several blank lines in a row.
Blank lines introduced by deduplication are currently not ignored so we end up with the same number of examples in the dataset as the original version (but with empty examples...)
I fixed that in this [commit](https://github.com/huggingface/datasets/commit/837a152e4724adc5308e2c4481908c00a8d93383). I'm re-running the metadata generation for deduplicated configs.

I got the new sizes today, will update the dataset_infos.json and the dataset card tomorrow

> I got the new sizes today, will update the dataset_infos.json and the dataset card tomorrow

great, I just wanted to report that I got error message "NonMatchingSplitsSizesError" when I tried to load one of the oscar dataset.

Hi @cahya-wirawan, which configuration of oscar do you have this issue with ?

Ok I see you're having this issue because I haven't updated the sizes yet ! I'm opening a PR

I just checked and indeed there's an issue with the `deduplicated` configurations since the commit I mentioned above.
I'm fixing this by using the new sizes I got yesterday :) 


I just updated the size in the table @pjox it should be good now :) 
I also updated the sizes in the dataset_infos.json in https://github.com/huggingface/datasets/pull/1868 (merged)

Thanks @lhoestq for fixing the issue, it works now

Thank you so much @lhoestq !
Regarding On-the-fly Data Loading

Hi,

I was wondering if it is possible to load images/texts as a batch during the training process, without loading the entire dataset on the RAM at any given point.

Thanks,
Gunjan

Possible duplicate

#1776 https://github.com/huggingface/datasets/issues/

really looking PR for this feature

Hi @acul3 

Issue #1776 talks about  doing on-the-fly data pre-processing, which I think is solved in the next release as mentioned in the issue #1825. I also look forward to using this feature, though :)

I wanted to ask about on-the-fly data loading from the cache (before pre-processing).

Hi ! Currently when you load a dataset via `load_dataset` for example, then the dataset is memory-mapped from an Arrow file on disk. Therefore there's almost no RAM usage even if your dataset contains TB of data.
Usually at training time only one batch of data at a time is loaded in memory.

Does that answer your question or were you thinking about something else ?

Hi @lhoestq,

I apologize for the late response. This answers my question. Thanks a lot.
Add OSCAR dataset card

I started adding the dataset card for OSCAR !

For now it's just basic info for all the different configurations in `Dataset Structure`.
In particular the Data Splits section tells how may samples there are for each config. The Data Instances section show an example for each config, and it also shows the size in MB. Since the Data Instances section is very long the user has to click to expand the info. I was able to generate it thanks to the tools made by  @madlag and @yjernite :D

Cc @pjox could you help me with the other sections ? (Dataset Description, Dataset Creation, Considerations for Using the Data, Additional Information)


Hi @lhoestq! When are you planning to release the version with this dataset?

BTW: What a huge README file :astonished:

Next week !

Closing in favor of #1833
Provide better exception message when one of many files results in an exception

I find when I process many files, i.e.

```
train_files = glob.glob('rain*.csv')
validation_files = glob.glob(validation*.csv')
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files))
```

I sometimes encounter an error due to one of the files being misformed (i.e. no data, or a comma in a field that isn't quoted, etc).

For example, this is the tail of an exception which I suspect is due to a stray comma.

>   File "pandas/_libs/parsers.pyx", line 756, in pandas._libs.parsers.TextReader.read
>   File "pandas/_libs/parsers.pyx", line 783, in pandas._libs.parsers.TextReader._read_low_memory
>   File "pandas/_libs/parsers.pyx", line 827, in pandas._libs.parsers.TextReader._read_rows
>   File "pandas/_libs/parsers.pyx", line 814, in pandas._libs.parsers.TextReader._tokenize_rows
>   File "pandas/_libs/parsers.pyx", line 1951, in pandas._libs.parsers.raise_parser_error
> pandas.errors.ParserError: Error tokenizing data. C error: Expected 2 fields in line 559, saw 3

It would be nice if the exception trace contained the name of the file being processed (I have 250 separate files!)

Hi!

Thank you for reporting this issue. I agree that the information about the exception should be more clear and explicit.

I could take on this issue.

On the meantime, as you can see from the exception stack trace, HF Datasets uses pandas to read the CSV files. You can pass arguments to `pandas.read_csv` by passing additional keyword arguments to `load_dataset`. For example, you may find useful this argument:
- `error_bad_lines` : bool, default True
  Lines with too many fields (e.g. a csv line with too many commas) will by default cause an exception to be raised, and no DataFrame will be returned. If False, then these “bad lines” will be dropped from the DataFrame that is returned.

You could try:
```python
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files), error_bad_lines=False)
```
pyarrow.lib.ArrowInvalid: Column 1 named input_ids expected length 599 but got length 1500

I am trying to preprocess any dataset in this package with GPT-2 tokenizer, so I need to structure the datasets as long sequences of text without padding. I've been following a couple of your tutorials and here you can find the script that is failing right at the end

https://github.com/LuCeHe/GenericTools/blob/master/KerasTools/lm_preprocessing.py

In the last iteration of the last dset.map, it gives the error that I copied in the title. Another issue that I have, if I leave the batch_size set as 1000 in the last .map, I'm afraid it's going to lose most text, so I'm considering setting both writer_batch_size and batch_size to 300 K, but I'm not sure it's the best way to go.

Can you help me?
Thanks!

Hi !
The error you have is due to the `input_ids` column not having the same number of examples as the other columns.
Indeed you're concatenating the `input_ids` at this line:

https://github.com/LuCeHe/GenericTools/blob/431835d8e13ec24dceb5ee4dc4ae58f0e873b091/KerasTools/lm_preprocessing.py#L134

However the other columns are kept unchanged, and therefore you end up with an `input_ids` column with 599 elements while the others columns like `attention_mask` have 1500.

To fix that you can instead concatenate them all using
```python
concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
```

Also you may need to drop the "text" column before applying `group_texts` since strings can't be concatenated with lists. You can drop it at the tokenization step:
```python
dset = dset.map(
    tokenize_function,
    batched=True,
    remove_columns=["text"]
)
```

You saved my life.
Unable to add Multi-label Datasets

I am trying to add [CIFAR-100](https://www.cs.toronto.edu/~kriz/cifar.html) dataset. The dataset contains two labels per image - `fine label` and `coarse label`. Using just one label in supervised keys as 
`supervised_keys=("img", "fine_label")` raises no issue. But trying `supervised_keys=("img", "fine_label","coarse_label")` leads to this error : 

```python
Traceback (most recent call last):
  File "test_script.py", line 2, in <module>
    d = load_dataset('./datasets/cifar100')
  File "~/datasets/src/datasets/load.py", line 668, in load_dataset
    **config_kwargs,
  File "~/datasets/src/datasets/builder.py", line 896, in __init__
    super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
  File "~/datasets/src/datasets/builder.py", line 247, in __init__
    info.update(self._info())
  File "~/.cache/huggingface/modules/datasets_modules/datasets/cifar100/61d2489b2d4a4abc34201432541b7380984ec714e290817d9a1ee318e4b74e0f/cifar100.py", line 79, in _info
    citation=_CITATION,
  File "<string>", line 19, in __init__
  File "~/datasets/src/datasets/info.py", line 136, in __post_init__
    self.supervised_keys = SupervisedKeysData(*self.supervised_keys)
TypeError: __init__() takes from 1 to 3 positional arguments but 4 were given
```
Is there a way I can fix this?

Also, what does adding `supervised_keys` do? Is it necessary? How would I specify `supervised_keys` for a multi-input, multi-label dataset?

Thanks,
Gunjan

Thanks for adding this dataset! As far as I know `supervised_keys` is mostly a holdover from TFDS, but isn't really used, so feel free to drop it (@lhoestq  or @thomwolf correct me if I'm wrong). It definitely shouldn't be blocking :) 

I can confirm that it comes from TFDS and is not used at the moment.

Thanks @yjernite @lhoestq 

The template for new dataset makes it slightly confusing. I suppose the comment suggesting its update can be removed.

Closing this issue since it was answered.
Add Hateful Memes Dataset

## Add Hateful Memes Dataset
- **Name:** Hateful Memes
- **Description:** [https://ai.facebook.com/blog/hateful-memes-challenge-and-data-set]( https://ai.facebook.com/blog/hateful-memes-challenge-and-data-set)
- **Paper:** [https://arxiv.org/pdf/2005.04790.pdf](https://arxiv.org/pdf/2005.04790.pdf)
- **Data:** [This link](https://drivendata-competition-fb-hateful-memes-data.s3.amazonaws.com/XjiOc5ycDBRRNwbhRlgH.zip?AWSAccessKeyId=AKIARVBOBDCY4MWEDJKS&Signature=DaUuGgZWUgDHzEPPbyJ2PhSJ56Q%3D&Expires=1612816874)
- **Motivation:** Including multi-modal datasets to 🤗 datasets.

I will be adding this dataset. It requires the user to sign an agreement on DrivenData. So, it will be used with a manual download.

The issue with this dataset is that the images are of different sizes. The image datasets added so far (CIFAR-10 and MNIST) have a uniform shape throughout.
So something like 
```python
 datasets.Array2D(shape=(28, 28), dtype="uint8")
```
won't work for the images. How would I add image features then? I checked `datasets/features.py` but couldn't figure out the appropriate class for this. I'm assuming I would want to avoid re-sizing at all since we want the user to be able to access the original images.

Also, in case I want to load only a subset of the data, since the actual data is around 8.8GB, how would that be possible?

Thanks,
Gunjan

I am not sure, but would `datasets.Sequence(datasets.Sequence(datasets.Sequence(datasets.Value("int")))` work?

Also, I found the information for loading only subsets of the data [here](https://github.com/huggingface/datasets/blob/master/docs/source/splits.rst).

Hi @lhoestq,

Request you to check this once.

Thanks,
Gunjan

Hi @gchhablani since Array2D doesn't support images of different sizes, I would suggest to store in the dataset the paths to the image file instead of the image data. This has the advantage of not decompressing the data (images are often compressed using jpeg, png etc.). Users can still apply `.map` to load the images if they want to. Though it would en up being Sequences features.

In the future we'll add support for ragged tensors for this case and update the relevant dataset with this feature.
Add FreebaseQA dataset

Adding FreebaseQA dataset suggested in PR #1435 with minor edits. Also closes that PR.
Requesting @lhoestq to review.

Hi ! It looks like this PR contains changes about other datasets than freebase_qa such as DuoRC.

Can you remove these changes please ?

Hi @lhoestq,

I think this happened because of rebasing. I'm unable to remove the duorc commit from the branch.  GEM, Arabic sarcasm datasets are also there. I can't see any merge conflicts, however. Before commiting I always rebase (shouldn't have done that).
Can you explain what is to be done? Should I create a clean PR?

Hi @gchhablani 
I think you can simply create another branch and another PR.

If I understand correctly the github diff is messed up because you rebased instead of merge.
Rebasing is supposed to be used only before pushing the branch the first time, or github messes up the diff.
If you want to include changes from master on a branch that is already push you need to use git merge.

Thanks @lhoestq.

I understand the issue now. I missed the instructions on the template. Sorry for bothering you unnecessarily, I'm pretty new to contributing on GitHub. I'll make a fresh PR.


No problem, I'm not a big fan of this weird behavior tbh.
Thanks for making a new PR

@lhoestq Haha, well, it's not as weird as not reading the [instructions](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#open-a-pull-request-on-the-main-huggingface-repo-and-share-your-work).
Also, I'm enjoying adding new datasets so it's all cool :)
writing Datasets in a human readable format

Hi
I see there is a save_to_disk function to save data, but this is not human readable format, is there a way I could save a Dataset object in a human readable  format to a file like json? thanks @lhoestq 

AFAIK, there is currently no built-in method on the `Dataset` object to do this.
However, a workaround is to directly use the Arrow table backing the dataset, **but it implies loading the whole dataset in memory** (correct me if I'm mistaken @lhoestq).

You can convert the Arrow table to a pandas dataframe to save the data as csv as follows:
```python
arrow_table = dataset.data
dataframe = arrow_table.to_pandas()
dataframe.to_csv("/path/to/file.csv")
```

Similarly, you can convert the dataset to a Python dict and save it as JSON:
```python
import json
arrow_table = dataset.data
py_dict = arrow_table.to_pydict()
with open("/path/to/file.json", "w+") as f:
    json.dump(py_dict, f)
```

Indeed this works as long as you have enough memory.
It would be amazing to have export options like csv, json etc. !

It should be doable to implement something that iterates through the dataset batch by batch to write to csv for example.
There is already an `export` method but currently the only export type that is supported is `tfrecords`.

Hi! `datasets` now supports `Dataset.to_csv` and `Dataset.to_json` for saving data in a human readable format.
add github of contributors

This PR will add contributors GitHub id at the end of every dataset cards.

@lhoestq Can you confirm if this format is fine? I will update cards based on your feedback.

On HuggingFace side we also have a mapping of hf user => github user (GitHub info used to be required when signing up until not long ago – cc @gary149 @beurkinger) so we can also add a link to HF profile

All the dataset cards have been updated with GitHub ids :)
[GEM] Updated the source link of the data to update correct tokenized version.


@mounicam we'll keep the original version in the Turk dataset proper, and use the updated file in the GEM aggregated dataset which I'll add later today

@lhoestq do not merge, I'll close when I've submitted the GEM dataset PR :) 

Closed by https://github.com/huggingface/datasets/pull/1807
Add DuoRC Dataset

Hi,

DuoRC SelfRC is one type of the [DuoRC Dataset](https://duorc.github.io/). DuoRC SelfRC is a crowdsourced Abstractive/Extractive Question-Answering dataset based on Wikipedia movie plots. It contains examples that may have answers in the movie plot, synthesized answers which are not present in the movie plot, or no answers.  I have also added ParaphraseRC - the other type of DuoRC dataset where questions are based on Wikipedia movie plots and answers are based on corresponding IMDb movie plots.

Paper :  [https://arxiv.org/abs/1804.07927](https://arxiv.org/abs/1804.07927)

I want to add this to 🤗 datasets to make it more accessible to the community. I have added  all the details that I could find. Please let me know if anything else is needed from my end.

Thanks,
Gunjan


Thanks for approving @lhoestq!
Will apply these changes for the other datasets I've added too.