kardwalker/my_dataset huggingface dataset:
GitHub issue #1865: Updated OPUS Open Subtitles Dataset with metadata information
State: closed | Created: 2021-02-11T13:26:26Z

Close #1844 

Problems:
- I ran `python datasets-cli test datasets/open_subtitles --save_infos --all_configs`, hence the change in `dataset_infos.json`, but it appears that the metadata features have not been added for all pairs. Any idea why that might be?
- Possibly related to the above, I tried doing `pip uninstall datasets && pip install -e ".[dev]"` after the changes, and loading the dataset via `load_dataset("open_subtitles", lang1='hi', lang2='it')` to check if the update worked, but the loaded dataset did not contain the metadata fields (neither in the features nor doing `next(iter(dataset['train']))`). What step(s) did I miss?

Questions:
- Is it ok to have a `classmethod` in there? I have not seen any in the few other datasets I have checked. I could make it a local method of the `_generate_examples` method, but I'd rather not duplicate the logic...
GitHub issue #1864: Add Winogender Schemas
State: closed | Created: 2021-02-11T08:18:38Z

## Adding a Dataset
- **Name:** Winogender Schemas
- **Description:** Winogender Schemas (inspired by Winograd Schemas) are minimal pairs of sentences that differ only by the gender of one pronoun in the sentence, designed to test for the presence of gender bias in automated coreference resolution systems.
- **Paper:** https://arxiv.org/abs/1804.09301
- **Data:** https://github.com/rudinger/winogender-schemas (see data directory)
- **Motivation:** Testing gender bias in automated coreference resolution systems, improve coreference resolution in general.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #1863: Add WikiCREM
State: open | Created: 2021-02-11T08:16:00Z

## Adding a Dataset
- **Name:** WikiCREM
- **Description:** A large unsupervised corpus for coreference resolution.
- **Paper:** https://arxiv.org/abs/1905.06290
- **Github repo:**: https://github.com/vid-koci/bert-commonsense
- **Data:** https://ora.ox.ac.uk/objects/uuid:c83e94bb-7584-41a1-aef9-85b0e764d9e3
- **Motivation:** Coreference resolution, common sense reasoning

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #1861: Fix Limit url
State: closed | Created: 2021-02-10T15:44:56Z

The test.json file of the Literal-Motion-in-Text (LiMiT) dataset was removed recently on the master branch of the repo at https://github.com/ilmgut/limit_dataset

This PR uses the previous commit sha to download the file instead, as suggested by @Paethon

Close #1836
GitHub issue #1860: Add loading from the Datasets Hub + add relative paths in download manager
State: closed | Created: 2021-02-10T13:24:11Z

With the new Datasets Hub on huggingface.co it's now possible to have a dataset repo with your own script and data.
For example: https://huggingface.co/datasets/lhoestq/custom_squad/tree/main contains one script and two json files.

You can load it using
```python
from datasets import load_dataset

d = load_dataset("lhoestq/custom_squad")
```

To be able to use the data files that live right next to the dataset script on the repo in the hub, I added relative paths support for the DownloadManager. For example in the repo mentioned above, there are two json files that can be downloaded via
```python
_URLS = {
    "train": "train-v1.1.json",
    "dev": "dev-v1.1.json",
}
downloaded_files = dl_manager.download_and_extract(_URLS)
```

To make it work, I set the `base_path` of the DownloadManager to be the parent path of the dataset script (which comes from either a local path or a remote url).

I also had to add the auth header of the requests to huggingface.co for private datasets repos. The token is fetched from [huggingface_hub](https://github.com/huggingface/huggingface_hub).
GitHub issue #1859: Error "in void don't know how to serialize this type of index" when saving index to disk when device=0 (GPU)
State: closed | Created: 2021-02-10T12:41:00Z

Error serializing faiss index.  Error as follows:

`Error in void faiss::write_index(const faiss::Index*, faiss::IOWriter*) at /home/conda/feedstock_root/build_artifacts/faiss-split_1612472484670/work/faiss/impl/index_write.cpp:453: don't know how to serialize this type of index`


Note:

`torch.cuda.is_available()` reports:

```
Cuda is available
cuda:0

```

Adding index, device=0 for GPU.

`dataset.add_faiss_index(column='embeddings', index_name='idx_embeddings', device=0)`

However, during a quick debug, self.faiss_index has no attr "device" when checked in` search.py, method save`, so fails to transform gpu index to cpu index.  If I add index without device, index is saved OK.


```
def save(self, file: str):
        """Serialize the FaissIndex on disk"""
        import faiss  # noqa: F811

        if (
            hasattr(self.faiss_index, "device")
            and self.faiss_index.device is not None
            and self.faiss_index.device > -1
        ):
            index = faiss.index_gpu_to_cpu(self.faiss_index)
        else:
            index = self.faiss_index
        faiss.write_index(index, file)
```
GitHub issue #1857: Unable to upload "community provided" dataset - 400 Client Error
State: closed | Created: 2021-02-10T10:39:01Z

Hi,
i'm trying to a upload a dataset as described [here](https://huggingface.co/docs/datasets/v1.2.0/share_dataset.html#sharing-a-community-provided-dataset). This is what happens:

``` 
$ datasets-cli login
$ datasets-cli upload_dataset my_dataset
About to upload file /path/to/my_dataset/dataset_infos.json to S3 under filename my_dataset/dataset_infos.json and namespace username
About to upload file /path/to/my_dataset/my_dataset.py to S3 under filename my_dataset/my_dataset.py and namespace username
Proceed? [Y/n] Y
Uploading... This might take a while if files are large
400 Client Error: Bad Request for url: https://huggingface.co/api/datasets/presign
huggingface.co migrated to a new model hosting system.
You need to upgrade to transformers v3.5+ to upload new models.
More info at https://discuss.hugginface.co or https://twitter.com/julien_c. Thank you! 
```
I'm using the latest releases of datasets and transformers.
GitHub issue #1856: load_dataset("amazon_polarity") NonMatchingChecksumError
State: closed | Created: 2021-02-10T10:00:56Z

Hi, it seems that loading the amazon_polarity dataset gives a NonMatchingChecksumError.

To reproduce:
```
load_dataset("amazon_polarity")
```
This will give the following error:
```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-3-8559a03fe0f8> in <module>()
----> 1 dataset = load_dataset("amazon_polarity")

3 frames
/usr/local/lib/python3.6/dist-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     37     if len(bad_urls) > 0:
     38         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 39         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     40     logger.info("All the checksums matched successfully" + for_verification_name)
     41 

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?id=0Bz8a_Dbh9QhbaW12WVVZS2drcnM&export=download']
```
GitHub issue #1854: Feature Request: Dataset.add_item
State: closed | Created: 2021-02-10T06:06:00Z

I'm trying to integrate `huggingface/datasets` functionality into `fairseq`, which requires (afaict) being able to build a dataset through an `add_item` method, such as https://github.com/pytorch/fairseq/blob/master/fairseq/data/indexed_dataset.py#L318, as opposed to loading all the text into arrow, and then `dataset.map(binarizer)`.
Is this possible at the moment? Is there an example? I'm happy to use raw `pa.Table` but not sure whether it will support uneven length entries.

### Desired API

```python
import numpy as np
tokenized: List[np.NDArray[np.int64]] = [np.array([4,4,2]), np.array([8,6,5,5,2]), np.array([3,3,31,5])

def build_dataset_from_tokenized(tokenized: List[np.NDArray[int]]) -> Dataset:
   """FIXME"""
   dataset = EmptyDataset()
   for t in tokenized: dataset.append(t)
   return dataset
ds = build_dataset_from_tokenized(tokenized)
assert (ds[0] == np.array([4,4,2])).all()
```

### What I tried
grep, google for "add one entry at a time", "datasets.append"

### Current Code
This code achieves the same result but doesn't fit into the `add_item` abstraction.

```python
    dataset = load_dataset('text', data_files={'train': 'train.txt'})
    tokenizer = RobertaTokenizerFast.from_pretrained('roberta-base', max_length=4096)
    def tokenize_function(examples):
        ids = tokenizer(examples['text'], return_attention_mask=False)['input_ids']
        return {'input_ids': [x[1:] for x in ids]}
    ds = dataset.map(tokenize_function, batched=True, num_proc=4, remove_columns=['text'], load_from_cache_file=not overwrite_cache)
	print(ds['train'][0]) => np array
```

Thanks in advance!
GitHub issue #1850: Add cord 19 dataset
State: closed | Created: 2021-02-09T10:22:08Z

Initial version only reading the metadata in CSV.

### Checklist:
- [x] Create the dataset script /datasets/my_dataset/my_dataset.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _infos(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template and at least fill the tags
- [x] Both tests for the real data and the dummy data pass.

### Extras:
- [x] add more metadata
- [x] add full text
- [x] add pre-computed document embedding
GitHub issue #1849: Add TIMIT
State: closed | Created: 2021-02-09T07:29:41Z

## Adding a Dataset
- **Name:** *TIMIT*
- **Description:** *The TIMIT corpus of read speech has been designed to provide speech data for the acquisition of acoustic-phonetic knowledge and for the development and evaluation of automatic speech recognition systems*

- **Paper:** *Homepage*: http://groups.inf.ed.ac.uk/ami/corpus/ / *Wikipedia*: https://en.wikipedia.org/wiki/TIMIT
- **Data:** *https://deepai.org/dataset/timit*
- **Motivation:** Important speech dataset


If interested in tackling this issue, feel free to tag @patrickvonplaten


Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #1847: [Metrics] Add word error metric metric
State: closed | Created: 2021-02-08T18:41:15Z

This PR adds the word error rate metric to datasets. 
WER: https://en.wikipedia.org/wiki/Word_error_rate
for speech recognition. WER is the main metric used in ASR. 

`jiwer` seems to be a solid library (see https://github.com/asteroid-team/asteroid/pull/329#discussion_r525158939)
GitHub issue #1846: Make DownloadManager downloaded/extracted paths accessible
State: closed | Created: 2021-02-08T18:14:42Z

Make accessible the file paths downloaded/extracted by DownloadManager.

Close #1831.

The approach:
- I set these paths as DownloadManager attributes: these are DownloadManager's concerns
- To access to these from DatasetBuilder, I set the DownloadManager instance as DatasetBuilder attribute: object composition
GitHub issue #1845: Enable logging propagation and remove logging handler
State: closed | Created: 2021-02-08T16:22:13Z

We used to have logging propagation disabled because of this issue: https://github.com/tensorflow/tensorflow/issues/26691
But since it's now fixed we should re-enable it. This is important to keep the default logging behavior for users, and propagation is also needed for pytest fixtures as asked in #1826 

I also removed the handler that was added since, according to the logging [documentation](https://docs.python.org/3/howto/logging.html#configuring-logging-for-a-library):
> It is strongly advised that you do not add any handlers other than NullHandler to your library’s loggers. This is because the configuration of handlers is the prerogative of the application developer who uses your library. The application developer knows their target audience and what handlers are most appropriate for their application: if you add handlers ‘under the hood’, you might well interfere with their ability to carry out unit tests and deliver logs which suit their requirements.

It could have been useful if we wanted to have a custom formatter for the logging but I think it's more important to keep the logging as default to not interfere with the users' logging management.

Therefore I also removed the two methods `datasets.logging.enable_default_handler` and `datasets.logging.disable_default_handler`.

cc @albertvillanova this should let you use capsys/caplog in pytest
cc @LysandreJik @sgugger if you want to do the same in `transformers`
GitHub issue #1844: Update Open Subtitles corpus with original sentence IDs
State: closed | Created: 2021-02-08T13:55:13Z

Hi! It would be great if you could add the original sentence ids to [Open Subtitles](https://huggingface.co/datasets/open_subtitles).

I can think of two reasons: first, it's possible to gather sentences for an entire document (the original ids contain media id, subtitle file id and sentence id), therefore somewhat allowing for document-level machine translation (and other document-level stuff which could be cool to have); second, it's possible to have parallel sentences in multiple languages, as they share the same ids across bitexts.

I think I should tag @abhishekkrthakur as he's the one who added it in the first place.

Thanks!
GitHub issue #1843: MustC Speech Translation
State: open | Created: 2021-02-08T13:27:45Z

## Adding a Dataset
- **Name:** *IWSLT19*
- **Description:** *The Speech Translation Task addresses the translation of English audio into German and Portuguese text.*
- **Hompage:** *https://sites.google.com/view/iwslt-evaluation-2019/speech-translation*
- **Data:** *https://sites.google.com/view/iwslt-evaluation-2019/speech-translation* - all data under "Allowed Training Data" and "Development and Evalutaion Data for TED/How2"
- **Motivation:** Important speech dataset

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #1842: Add AMI Corpus
State: closed | Created: 2021-02-08T13:25:00Z

## Adding a Dataset
- **Name:** *AMI*
- **Description:** *The AMI Meeting Corpus is a multi-modal data set consisting of 100 hours of meeting recordings. For a gentle introduction to the corpus, see the corpus overview. To access the data, follow the directions given there. Around two-thirds of the data has been elicited using a scenario in which the participants play different roles in a design team, taking a design project from kick-off to completion over the course of a day. The rest consists of naturally occurring meetings in a range of domains. Detailed information can be found in the documentation section.*

- **Paper:** *Homepage*: http://groups.inf.ed.ac.uk/ami/corpus/
- **Data:** *http://groups.inf.ed.ac.uk/ami/download/* - Select all cases in 1) and select "Individual Headsets" & "Microphone array" for 2)
- **Motivation:** Important speech dataset


If interested in tackling this issue, feel free to tag @patrickvonplaten


Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #1841: Add ljspeech
State: closed | Created: 2021-02-08T13:22:26Z

## Adding a Dataset
- **Name:** *ljspeech*
- **Description:** *This is a public domain speech dataset consisting of 13,100 short audio clips of a single speaker reading passages from 7 non-fiction books. A transcription is provided for each clip. Clips vary in length from 1 to 10 seconds and have a total length of approximately 24 hours.

The texts were published between 1884 and 1964, and are in the public domain. The audio was recorded in 2016-17 by the LibriVox project and is also in the public domain.)*
- **Paper:** *Homepage*: https://keithito.com/LJ-Speech-Dataset/
- **Data:** *https://keithito.com/LJ-Speech-Dataset/*
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/ljspeech
If interested in tackling this issue, feel free to tag @patrickvonplaten


Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #1840: Add common voice
State: closed | Created: 2021-02-08T13:21:05Z

## Adding a Dataset
- **Name:** *common voice*
- **Description:** *Mozilla Common Voice Dataset*
- **Paper:** Homepage: https://voice.mozilla.org/en/datasets
- **Data:** https://voice.mozilla.org/en/datasets
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/common_voice
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #1839: Add Voxforge
State: open | Created: 2021-02-08T13:19:56Z

## Adding a Dataset
- **Name:** *voxforge* 
- **Description:** *VoxForge is a language classification dataset. It consists of user submitted audio clips submitted to the website. In this release, data from 6 languages is collected - English, Spanish, French, German, Russian, and Italian. Since the website is constantly updated, and for the sake of reproducibility, this release contains only recordings submitted prior to 2020-01-01. The samples are splitted between train, validation and testing so that samples from each speaker belongs to exactly one split.*
- **Paper:** *Homepage*: http://www.voxforge.org/
- **Data:** *http://www.voxforge.org/home/downloads*
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/voxforge
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #1838: Add tedlium
State: closed | Created: 2021-02-08T13:17:52Z

## Adding a Dataset
- **Name:** *tedlium*
- **Description:** *The TED-LIUM 1-3 corpus is English-language TED talks, with transcriptions, sampled at 16kHz. It contains about 118 hours of speech.*
- **Paper:** Homepage: http://www.openslr.org/7/, https://lium.univ-lemans.fr/en/ted-lium2/ &, https://www.openslr.org/51/
- **Data:** http://www.openslr.org/7/
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/tedlium
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #1837: Add VCTK
State: closed | Created: 2021-02-08T13:15:28Z

## Adding a Dataset
- **Name:** *VCTK*
- **Description:** *This CSTR VCTK Corpus includes speech data uttered by 110 English speakers with various accents. Each speaker reads out about 400 sentences, which were selected from a newspaper, the rainbow passage and an elicitation paragraph used for the speech accent archive.*
- **Paper:** Homepage: https://datashare.ed.ac.uk/handle/10283/3443
- **Data:** https://datashare.ed.ac.uk/handle/10283/3443
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/vctk

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #1836: test.json has been removed from the limit dataset repo (breaks dataset)
State: closed | Created: 2021-02-08T12:45:53Z

https://github.com/huggingface/datasets/blob/16042b233dbff2a7585110134e969204c69322c3/datasets/limit/limit.py#L51

The URL is not valid anymore since test.json has been removed in master for some reason. Directly referencing the last commit works:

`https://raw.githubusercontent.com/ilmgut/limit_dataset/0707d3989cd8848f0f11527c77dcf168fefd2b23/data`
GitHub issue #1835: Add CHiME4 dataset
State: open | Created: 2021-02-08T12:36:38Z

## Adding a Dataset
- **Name:** Chime4
- **Description:** Chime4 is a dataset for automatic speech recognition. It is especially useful for evaluating models in a noisy environment and for multi-channel ASR
- **Paper:** Dataset comes from a channel: http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/ . Results paper: 
- **Data:** http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/download.html
- **Motivation:** So far there are very little datasets for speech in `datasets`. Only `lbirispeech_asr` so far.

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #1832: Looks like nokogumbo is up-to-date now, so this is no longer needed.
State: closed | Created: 2021-02-07T06:52:07Z

Looks like nokogumbo is up-to-date now, so this is no longer needed.

__Originally posted by @dependabot in https://github.com/discourse/discourse/pull/11373#issuecomment-738993432__
GitHub issue #1831: Some question about raw dataset download info in the project .
State: closed | Created: 2021-02-07T05:33:36Z

Hi , i review the code in 
https://github.com/huggingface/datasets/blob/master/datasets/conll2003/conll2003.py
in the _split_generators function is the truly logic of download raw datasets with dl_manager
and use Conll2003 cls by use import_main_class in load_dataset function
My question is that , with this logic it seems that i can not have the raw dataset download location
in variable in downloaded_files in _split_generators.
If someone also want use huggingface datasets as raw dataset downloader,
how can he retrieve the raw dataset download path from attributes in 
datasets.dataset_dict.DatasetDict ?
GitHub issue #1830: using map on loaded Tokenizer 10x - 100x slower than default Tokenizer?
State: open | Created: 2021-02-06T21:00:26Z

This could total relate to me misunderstanding particular call functions, but I added words to a GPT2Tokenizer, and saved it to disk (note I'm only showing snippets but I can share more) and the map function ran much slower: 

````
def save_tokenizer(original_tokenizer,text,path="simpledata/tokenizer"):
    words_unique = set(text.split(" "))
    for i in words_unique:
        original_tokenizer.add_tokens(i)
    original_tokenizer.save_pretrained(path)

tokenizer2 = GPT2Tokenizer.from_pretrained(os.path.join(experiment_path,experiment_name,"tokenizer_squad"))

train_set_baby=Dataset.from_dict({"text":[train_set["text"][0][0:50]]})
````

I then applied the dataset map function on a fairly small set of text:

```
%%time
train_set_baby = train_set_baby.map(lambda d:tokenizer2(d["text"]),batched=True)

```


The run time for train_set_baby.map was 6 seconds, and the batch itself was 2.6 seconds

**100% 1/1 [00:02<00:00, 2.60s/ba] CPU times: user 5.96 s, sys: 36 ms, total: 5.99 s Wall time: 5.99 s**

In comparison using (even after adding additional tokens): 
`
tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")`

```
%%time
train_set_baby = train_set_baby.map(lambda d:tokenizer2(d["text"]),batched=True)

```
The time is 
**100% 1/1 [00:00<00:00, 34.09ba/s] CPU times: user 68.1 ms, sys: 16 µs, total: 68.1 ms Wall time: 62.9 ms**

It seems this might relate to the tokenizer save or load function, however, the issue appears to come up when I apply the loaded tokenizer  to the map function. 

I should also add that playing around with the amount of words I add to the tokenizer before I save it to disk and load it into memory  appears to impact the time it takes to run the map function.
GitHub issue #1829: Add Tweet Eval Dataset
State: closed | Created: 2021-02-06T12:36:25Z

Closes Draft PR #1407. 

Notes:
1. I have excluded `mapping.txt` from the dataset at it only contained the name mappings, which are already present in the ClassLabels.
2. I have also exluded the textual names for the emojis mentioned in the [mapping](https://github.com/cardiffnlp/tweeteval/blob/main/datasets/emoji/mapping.txt).
3. I do not understand @abhishekkrthakur's example generator on #1407. Maybe he was trying to build up on code from some other dataset.

Requesting @lhoestq  to review.
GitHub issue #1828: Add CelebA Dataset
State: closed | Created: 2021-02-05T20:20:55Z

Trying to add CelebA Dataset. 
Need help with testing. Loading examples takes a lot of time so I am unable to generate the `dataset_infos.json` and unable to test. Also, need help with creating `dummy_data.zip`.

Additionally, trying to load a few examples using `load_dataset('./datasets/celeb_a',split='train[10:20]')` still loads all the examples (doesn't stop at 10).
GitHub issue #1825: Datasets library not suitable for huge text datasets.
State: closed | Created: 2021-02-05T11:06:50Z

Hi,

I'm trying to use datasets library to load a 187GB dataset of pure text, with the intention of building a Language Model. The problem is that from the 187GB it goes to some TB when processed by Datasets. First of all, I think the pre-tokenizing step (with tokenizer.map()) is not really thought for datasets this big, but for fine-tuning datasets, as this process alone takes so much time, usually in expensive machines (due to the need of tpus - gpus) which is not being used for training. It would possibly be more efficient in such cases to tokenize each batch at training time (receive batch - tokenize batch - train with batch), so that the whole time the machine is up it's being used for training. 
Moreover, the pyarrow objects created from a 187 GB datasets are huge, I mean, we always receive OOM, or No Space left on device errors when only 10-12% of the dataset has been processed, and only that part occupies 2.1TB in disk, which is so many times the disk  usage of the pure text (and this doesn't make sense, as tokenized texts should be lighter than pure texts).

Any suggestions??
GitHub issue #1824: Add OSCAR dataset card
State: closed | Created: 2021-02-05T10:30:26Z

I started adding the dataset card for OSCAR !

For now it's just basic info for all the different configurations in `Dataset Structure`.
In particular the Data Splits section tells how may samples there are for each config. The Data Instances section show an example for each config, and it also shows the size in MB. Since the Data Instances section is very long the user has to click to expand the info. I was able to generate it thanks to the tools made by  @madlag and @yjernite :D

Cc @pjox could you help me with the other sections ? (Dataset Description, Dataset Creation, Considerations for Using the Data, Additional Information)
GitHub issue #1823: Add FewRel Dataset
State: closed | Created: 2021-02-05T10:22:03Z

Hi,

This PR closes this [Card](https://github.com/huggingface/datasets/projects/1#card-53285184) and Issue #1757.

I wasn't sure how to add `pid2name` along with the dataset so I added it as a separate configuration. For each (head, tail, tokens) triplet, I have created one example. I have added the dictionary key as `"relation"` in the dataset. Additionally, for `pubmed_unsupervised`, I kept `"relation":""` in the dictionary.

Please recommend better alternatives, if any.

Thanks,
Gunjan
GitHub issue #1821: Provide better exception message when one of many files results in an exception
State: closed | Created: 2021-02-05T00:49:03Z

I find when I process many files, i.e.

```
train_files = glob.glob('rain*.csv')
validation_files = glob.glob(validation*.csv')
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files))
```

I sometimes encounter an error due to one of the files being misformed (i.e. no data, or a comma in a field that isn't quoted, etc).

For example, this is the tail of an exception which I suspect is due to a stray comma.

>   File "pandas/_libs/parsers.pyx", line 756, in pandas._libs.parsers.TextReader.read
>   File "pandas/_libs/parsers.pyx", line 783, in pandas._libs.parsers.TextReader._read_low_memory
>   File "pandas/_libs/parsers.pyx", line 827, in pandas._libs.parsers.TextReader._read_rows
>   File "pandas/_libs/parsers.pyx", line 814, in pandas._libs.parsers.TextReader._tokenize_rows
>   File "pandas/_libs/parsers.pyx", line 1951, in pandas._libs.parsers.raise_parser_error
> pandas.errors.ParserError: Error tokenizing data. C error: Expected 2 fields in line 559, saw 3

It would be nice if the exception trace contained the name of the file being processed (I have 250 separate files!)
GitHub issue #1820: Add metrics usage examples and tests
State: closed | Created: 2021-02-04T18:23:50Z

All metrics finally have usage examples and proper fast + slow tests :)

I added examples of usage for every metric, and I use doctest to make sure they all work as expected.

For "slow" metrics such as bert_score or bleurt which require to download + run a transformer model, the download + forward pass are only done in the slow test.
In the fast test on the other hand, the download + forward pass are monkey patched.

Metrics that need to be installed from github are not added to setup.py because it prevents uploading the `datasets` package to pypi.
An additional-test-requirements.txt file is used instead. This file also include `comet` in order to not have to resolve its *impossible* dependencies.

Also `comet` is not tested on windows because one of its dependencies (fairseq) can't be installed in the CI for some reason.
GitHub issue #1817: pyarrow.lib.ArrowInvalid: Column 1 named input_ids expected length 599 but got length 1500
State: closed | Created: 2021-02-04T02:30:23Z

I am trying to preprocess any dataset in this package with GPT-2 tokenizer, so I need to structure the datasets as long sequences of text without padding. I've been following a couple of your tutorials and here you can find the script that is failing right at the end

https://github.com/LuCeHe/GenericTools/blob/master/KerasTools/lm_preprocessing.py

In the last iteration of the last dset.map, it gives the error that I copied in the title. Another issue that I have, if I leave the batch_size set as 1000 in the last .map, I'm afraid it's going to lose most text, so I'm considering setting both writer_batch_size and batch_size to 300 K, but I'm not sure it's the best way to go.

Can you help me?
Thanks!
GitHub issue #1815: Add CCAligned Multilingual Dataset
State: closed | Created: 2021-02-03T18:59:52Z

Hello,

I'm trying to add [CCAligned Multilingual Dataset](http://www.statmt.org/cc-aligned/). This has the potential to close #1756.

This dataset has two types - Document-Pairs, and Sentence-Pairs.

The datasets are huge, so I won't be able to test all of them. At the same time, a user might only want to download one particular language and not all. To provide this feature, `load_dataset`'s `**config_kwargs` should allow some random keyword args, in this case -`language_code`. This will be needed before the dataset is downloaded and extracted.

I'm expecting the usage to be something like - 
`load_dataset('ccaligned_multilingual','documents',language_code='en_XX-af_ZA')`. Ofcourse, at a later stage we can provide just two character language codes. This also has an issue where one language has multiple files (`my_MM` and `my_MM_zaw` on the link), but before that the required functionality must be added to `load_dataset`.

It would be great if someone could either tell me an alternative way to do this, or point me to where changes need to be made, if any, apart from the `BuilderConfig` definition. 

Additionally, I believe the tests will also have to be modified if this change is made, since it would not be possible to test for any random keyword arguments. 

A decent way to go about this would be to provide all the options in a list/dictionary for `language_code` and use that to test the arguments. In essence, this is similar to the pre-trained checkpoint dictionary as `transformers`. That means writing dataset specific tests, or adding something new to dataset generation script to make it easier for everyone to add keyword arguments without having to worry about the tests.

Thanks,
Gunjan

Requesting @lhoestq / @yjernite to review.
GitHub issue #1813: Support future datasets
State: closed | Created: 2021-02-03T15:26:49Z

If a dataset is available at the version of the local installation of `datasets` (e.g. 1.2.0), then loading this dataset means loading the script at this version.

However when trying to load a dataset that is only available on master, currently users have to specify `script_version="master"` in `load_dataset` to make it work.

However we could automatically get the dataset from master instead in this case.

I added this feature in this PR.
I also added a warning if a dataset is not available at the version of the local installation of `datasets` but is loaded from master:
```python
>>> load_dataset("silicone", "dyda_da")
Couldn't find file locally at silicone/silicone.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.2.0/datasets/silicone/silicone.py.
The file was picked from the master branch on github instead at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/silicone/silicone.py.
Downloading and preparing dataset silicone/dyda_da (download: 8.46 MiB, generated: 9.39 MiB, post-processed: Unknown size, total: 17.86 MiB) to /Users/quentinlhoest/.cache/huggingface/datasets/silicone/dyda_da/1.0.0/d41d8c0b73c6df035b1369c45774418f0051163ea689b5502b8bda783adf6342...
...
```
GitHub issue #1811: Unable to add Multi-label Datasets
State: closed | Created: 2021-02-02T11:50:56Z

I am trying to add [CIFAR-100](https://www.cs.toronto.edu/~kriz/cifar.html) dataset. The dataset contains two labels per image - `fine label` and `coarse label`. Using just one label in supervised keys as 
`supervised_keys=("img", "fine_label")` raises no issue. But trying `supervised_keys=("img", "fine_label","coarse_label")` leads to this error : 

```python
Traceback (most recent call last):
  File "test_script.py", line 2, in <module>
    d = load_dataset('./datasets/cifar100')
  File "~/datasets/src/datasets/load.py", line 668, in load_dataset
    **config_kwargs,
  File "~/datasets/src/datasets/builder.py", line 896, in __init__
    super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
  File "~/datasets/src/datasets/builder.py", line 247, in __init__
    info.update(self._info())
  File "~/.cache/huggingface/modules/datasets_modules/datasets/cifar100/61d2489b2d4a4abc34201432541b7380984ec714e290817d9a1ee318e4b74e0f/cifar100.py", line 79, in _info
    citation=_CITATION,
  File "<string>", line 19, in __init__
  File "~/datasets/src/datasets/info.py", line 136, in __post_init__
    self.supervised_keys = SupervisedKeysData(*self.supervised_keys)
TypeError: __init__() takes from 1 to 3 positional arguments but 4 were given
```
Is there a way I can fix this?

Also, what does adding `supervised_keys` do? Is it necessary? How would I specify `supervised_keys` for a multi-input, multi-label dataset?

Thanks,
Gunjan
GitHub issue #1810: Add Hateful Memes Dataset
State: open | Created: 2021-02-02T10:53:59Z

## Add Hateful Memes Dataset
- **Name:** Hateful Memes
- **Description:** [https://ai.facebook.com/blog/hateful-memes-challenge-and-data-set]( https://ai.facebook.com/blog/hateful-memes-challenge-and-data-set)
- **Paper:** [https://arxiv.org/pdf/2005.04790.pdf](https://arxiv.org/pdf/2005.04790.pdf)
- **Data:** [This link](https://drivendata-competition-fb-hateful-memes-data.s3.amazonaws.com/XjiOc5ycDBRRNwbhRlgH.zip?AWSAccessKeyId=AKIARVBOBDCY4MWEDJKS&Signature=DaUuGgZWUgDHzEPPbyJ2PhSJ56Q%3D&Expires=1612816874)
- **Motivation:** Including multi-modal datasets to 🤗 datasets.

I will be adding this dataset. It requires the user to sign an agreement on DrivenData. So, it will be used with a manual download.

The issue with this dataset is that the images are of different sizes. The image datasets added so far (CIFAR-10 and MNIST) have a uniform shape throughout.
So something like 
```python
 datasets.Array2D(shape=(28, 28), dtype="uint8")
```
won't work for the images. How would I add image features then? I checked `datasets/features.py` but couldn't figure out the appropriate class for this. I'm assuming I would want to avoid re-sizing at all since we want the user to be able to access the original images.

Also, in case I want to load only a subset of the data, since the actual data is around 8.8GB, how would that be possible?

Thanks,
Gunjan
GitHub issue #1807: Adding an aggregated dataset for the GEM benchmark
State: closed | Created: 2021-02-02T00:39:53Z

This dataset gathers modified versions of several other conditional text generation datasets which together make up the shared task for the Generation Evaluation and Metrics workshop (think GLUE for text generation)

The changes from the original datasets are detailed in the Dataset Cards on the GEM website, which are linked to in this dataset card.

cc @sebastianGehrmann
GitHub issue #1803: Querying examples from big datasets is slower than small datasets
State: closed | Created: 2021-02-01T11:08:23Z

After some experiments with bookcorpus I noticed that querying examples from big datasets is slower than small datasets.
For example
```python
from datasets import load_dataset

b1 = load_dataset("bookcorpus", split="train[:1%]")
b50 = load_dataset("bookcorpus", split="train[:50%]")
b100 = load_dataset("bookcorpus", split="train[:100%]")

%timeit _ = b1[-1]                                                                     
# 12.2 µs ± 70.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit _ = b50[-1]                                                                    
# 92.5 µs ± 1.24 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

%timeit _ = b100[-1]                                                                      
# 177 µs ± 3.13 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

```

It looks like the time to fetch the example increases with the size of the dataset.

This is maybe due to the use of the Arrow streaming format to store the data on disk. I guess pyarrow needs to iterate through the file as a stream to find the queried sample.

Maybe switching to the Arrow IPC file format could help fixing this issue.

Indeed according to the [documentation](https://arrow.apache.org/docs/format/Columnar.html?highlight=arrow1#ipc-file-format), it's identical to the streaming format except that it contains the memory offsets of each sample, which could fix the issue:
> We define a “file format” supporting random access that is build with the stream format. The file starts and ends with a magic string ARROW1 (plus padding). What follows in the file is identical to the stream format. At the end of the file, we write a footer containing a redundant copy of the schema (which is a part of the streaming format) plus memory offsets and sizes for each of the data blocks in the file. This enables random access any record batch in the file. See File.fbs for the precise details of the file footer.

cc @gaceladri since it can help speed up your training when this one is fixed.
GitHub issue #1800: Add DuoRC Dataset
State: closed | Created: 2021-01-31T20:01:59Z

Hi,

DuoRC SelfRC is one type of the [DuoRC Dataset](https://duorc.github.io/). DuoRC SelfRC is a crowdsourced Abstractive/Extractive Question-Answering dataset based on Wikipedia movie plots. It contains examples that may have answers in the movie plot, synthesized answers which are not present in the movie plot, or no answers.  I have also added ParaphraseRC - the other type of DuoRC dataset where questions are based on Wikipedia movie plots and answers are based on corresponding IMDb movie plots.

Paper :  [https://arxiv.org/abs/1804.07927](https://arxiv.org/abs/1804.07927)

I want to add this to 🤗 datasets to make it more accessible to the community. I have added  all the details that I could find. Please let me know if anything else is needed from my end.

Thanks,
Gunjan
GitHub issue #1799: Update: SWDA - Fixed code to use all metadata features. Added comments and cleaned c…
State: closed | Created: 2021-01-31T19:18:55Z

This is a dataset I currently use my research and I realized some features are not being returned.

Previous code was not using all available metadata and was kind of messy

I fixed code to use all metadata and made some modification to be more efficient and better formatted.


Please let me know if I need to make any changes.
GitHub issue #1797: Connection error
State: closed | Created: 2021-01-30T07:32:45Z

Hi
I am hitting to the error, help me and thanks.

`train_data = datasets.load_dataset("xsum", split="train")`
`ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.0.2/datasets/xsum/xsum.py`
GitHub issue #1796: Filter on dataset too much slowww
State: open | Created: 2021-01-30T04:09:19Z

I have a dataset with 50M rows.
For pre-processing, I need to tokenize this and filter rows with the large sequence.

My tokenization took roughly 12mins. I used `map()` with batch size 1024 and multi-process with 96 processes.

When I applied the `filter()` function it is taking too much time. I need to filter sequences based on a boolean column.
Below are the variants I tried.
1. filter() with batch size 1024, single process (takes roughly 3 hr)
2. filter() with batch size 1024, 96 processes (takes 5-6 hrs ¯\\\_(ツ)\_/¯)
3. filter() with loading all data in memory, only a single boolean column (never ends).

Can someone please help?

Below is a sample code for small dataset.

```
from datasets import load_dataset
dataset = load_dataset('glue', 'mrpc', split='train')
dataset = dataset.map(lambda x: {'flag': random.randint(0,1)==1})

def _amplify(data):
        return data

dataset = dataset.filter(_amplify, batch_size=1024, keep_in_memory=False, input_columns=['flag'])
```
GitHub issue #1795: Custom formatting for lazy map + arrow data extraction refactor
State: closed | Created: 2021-01-29T16:35:53Z

Hi !

This PR refactors the way data are extracted from pyarrow tables to extend it to the use of custom formatting functions.

While the internal storage of the dataset is always the Apache Arrow format, by setting a specific format on a dataset, you can cast the output of `datasets.Dataset.__getitem__` in NumPy/pandas/PyTorch/TensorFlow, on-the-fly.

A specific format can be activated with `datasets.Dataset.set_format`. For example: `dataset.set_format(type='torch', columns=['label'])`.

### What's new:

You can now also define your own formatting function that is applied on-the-fly. To do so you can pass your formatting function in the `transform` parameter of `datasets.Dataset.set_format`, and keep `type` to `None`.
A formatting function is a callable that takes a batch (as a dict, formatted as python) as input and returns a batch.

Here is an example to tokenize and pad tokens on-the-fly when accessing the samples:

```python
from datasets import load_dataset
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
def encode(batch):
    return tokenizer(batch["sentence1"], padding="longest", truncation=True, max_length=512, return_tensors="pt")

dataset = load_dataset("glue", "mrpc", split="train")
dataset.set_format(transform=encode)
dataset.format
# {'type': 'custom', 'format_kwargs': {'transform': <function __main__.encode(batch)>}, 'columns': ['idx', 'label', 'sentence1', 'sentence2'], 'output_all_columns': False}
dataset[:2]
# {'input_ids': tensor([[  101,  2572,  3217, ... 102]]), 'token_type_ids': tensor([[0, 0, 0, ... 0]]), 'attention_mask': tensor([[1, 1, 1, ... 1]])}
```

Let me know what you think of this API !
We can still change it if we want to.

Especially @sgugger since this may be useful when using `datasets` to train models.

EDIT: this was changed to `dataset.set_transform(encode)`

-------------------

Note:

I had to refactor the way data are extracted and formatted from pyarrow tables and I made it more robust and flexible. In particular I modularized it to be able to unit-test it properly. This was very helpful since I detected some bugs in the previous implementation and was able to fix them.

Some bugs I found and fixed:
- certain slices/ranges were not supported because negative ids were passed to pyarrow
- formatting as numpy/torch/tensorflow a column would make it lose its precision information (for example a column as `Value("float32")`) would be returned as a tensor of float64 (default behavior for numpy)
- on windows  integers formatted as numpy/torch/tensorflow were not always int64 tensors by default but were int32 

The unit tests for those are now really extensive :)
GitHub issue #1791: Small fix with corrected logging of train vectors
State: closed | Created: 2021-01-29T14:26:06Z

Now you can set `train_size` to the whole dataset size via `train_size = -1` and login writes not `Training the index with the first -1 vectors` but (for example) `Training the index with the first 16123 vectors`. And maybe more than dataset length. Logging will be correct
GitHub issue #1790: ModuleNotFoundError: No module named 'apache_beam', when specific languages.
State: open | Created: 2021-01-29T08:17:24Z

```py
import datasets
wiki = datasets.load_dataset('wikipedia', '20200501.ja', cache_dir='./datasets')
```
then `ModuleNotFoundError: No module named 'apache_beam'` happend.

The error doesn't appear when it's '20200501.en'.
I don't know Apache Beam, but according to #498 it isn't necessary when it's saved to local. is it correct?
GitHub issue #1786: How to use split dataset 
State: closed | Created: 2021-01-27T21:37:47Z

![Capture1](https://user-images.githubusercontent.com/78090287/106057436-cb6a1f00-6111-11eb-8c9c-3658065b1fdf.PNG)

Hey,
I want to split the lambada dataset into corpus, test, train and valid txt files (like penn treebank) but I am not able to achieve this. What I am doing is, executing the lambada.py file in my project but its not giving desired results. Any help will be appreciated!