Kokolipa/github-issues huggingface dataset:
Issue: Error when exploring `arabic_speech_corpus`
State: closed
Labels: name: bug

name: nlp-viewer

name: speech

Navigate to https://huggingface.co/datasets/viewer/?dataset=arabic_speech_corpus

Error:
```
ImportError: To be able to use this dataset, you need to install the following dependencies['soundfile'] using 'pip install soundfile' for instance'
Traceback:
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/script_runner.py", line 332, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp-viewer/run.py", line 233, in <module>
    configs = get_confs(option)
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/caching.py", line 604, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/caching.py", line 588, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp-viewer/run.py", line 145, in get_confs
    module_path = nlp.load.prepare_module(path, dataset=True
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/datasets/load.py", line 342, in prepare_module
    f"To be able to use this {module_type}, you need to install the following dependencies"
```

Thanks for reporting! We'll fix that as soon as possible

Actually soundfile is not a dependency of this dataset.
The error comes from a bug that was fixed in this commit: https://github.com/huggingface/datasets/pull/1767/commits/c304e63629f4453367de2fd42883a78768055532
Basically the library used to consider the `import soundfile` in the docstring as a dependency, while it's just here as a code example.

Updating the viewer to the latest version of `datasets` should fix this issue


Hi! The viewer at https://huggingface.co/datasets/arabic_speech_corpus works fine. Closing.
Issue: How to load a dataset with load_from disk and save it again after doing transformations without changing the original? 
State: closed

I am using the latest datasets library.  In my work, I first use **load_from_disk** to load a data set that contains 3.8Gb information. Then during my training process, I update that dataset object and add new elements and save it in a different place.  

When I save the dataset with **save_to_disk**, the original dataset which is already in the disk also gets updated. I do not want to update it.  How to prevent from this?


Hi ! That looks like a bug, can you provide some code so that we can reproduce ?
It's not supposed to update the original dataset

Hi, I experimented with RAG. 

Actually, you can run the [use_own_knowldge_dataset.py](https://github.com/shamanez/transformers/blob/rag-end-to-end-retrieval/examples/research_projects/rag/use_own_knowledge_dataset.py#L80). In the 80 you can save the dataset object to the disk with save_to_disk. Then in order to compute the embeddings in this use **load_from_disk**. 

Then finally save it. You can see the original dataset object (CSV after splitting also will be changed)

One more thing- when I save the dataset object with **save_to_disk** it name the arrow file with cache.... rather than using dataset. arrow.  Can you add a variable that we can feed a name to save_to_disk function?

@lhoestq   I also found that cache in tmp directory gets updated after transformations. This is really problematic when using datasets interactively. Let's say we use the shards function to a dataset loaded with csv, atm when we do transformations to shards and combine them it updates the original csv cache. 

I plan to update the save_to_disk method in #2025 so I can make sure the new save_to_disk doesn't corrupt your cache files.
But from your last message it looks like save_to_disk isn't the root cause right ?

ok, one more thing. When we use save_to_disk there are two files other than .arrow. dataset_info.json and state.json. Sometimes most of the fields in the dataset_infor.json are null, especially when saving dataset objects. Anyways I think load_from_disk uses the arrow files mentioned in state.json right? 

> Anyways I think load_from_disk uses the arrow files mentioned in state.json right?

Yes exactly

Perfect.  For now, I am loading the dataset from CSV in my interactive process and will wait until you make the PR!
Issue: Readme.md is misleading about kinds of datasets?
State: closed

Hi!

At the README.MD, you say: "efficient data pre-processing: simple, fast and reproducible data pre-processing for the above public datasets as well as your own local datasets in CSV/JSON/text. "

But here:
https://github.com/huggingface/datasets/blob/master/templates/new_dataset_script.py#L82-L117

You mention other kinds of datasets, with images and so on. I'm confused. 

Is it possible to use it to store, say, imagenet locally? 

Hi ! Yes it's possible to use image data. There are already a few of them available (MNIST, CIFAR..)
Issue: wmt datasets fail to load
State: closed

~\.cache\huggingface\modules\datasets_modules\datasets\wmt14\43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e\wmt_utils.py in _split_generators(self, dl_manager)
    758         # Extract manually downloaded files.
    759         manual_files = dl_manager.extract(manual_paths_dict)
--> 760         extraction_map = dict(downloaded_files, **manual_files)
    761 
    762         for language in self.config.language_pair:

TypeError: type object argument after ** must be a mapping, not list

caching issue, seems to work again..
Issue: The size of CoNLL-2003 is not consistant with the official release.
State: closed

Thanks for the dataset sharing! But when I use conll-2003, I meet some questions.
The statistics of conll-2003 in this repo is  : 
\#train 14041  \#dev 3250 \#test 3453
While the official statistics is:
\#train 14987 \#dev 3466 \#test 3684
Wish for your reply~

Hi,

if you inspect the raw data, you can find there are 946 occurrences of `-DOCSTART- -X- -X- O` in the train split and `14041 + 946 = 14987`, which is exactly the number of sentences the authors report. `-DOCSTART-` is a special line that acts as a boundary between two different documents and is filtered out in our implementation.

@lhoestq What do you think about including these lines? ([Link](https://github.com/flairNLP/flair/issues/1097) to a similar issue in the flairNLP repo)

We should mention in the Conll2003 dataset card that these lines have been removed indeed.

If some users are interested in using these lines (maybe to recombine documents ?) then we can add a parameter to the conll2003 dataset to include them.

But IMO the default config should stay the current one (without the `-DOCSTART-` stuff), so that you can directly train NER models without additional preprocessing. Let me know what you think

@lhoestq Yes, I agree adding a small note should be sufficient.

Currently, NLTK's `ConllCorpusReader` ignores the `-DOCSTART-` lines so I think it's ok if we do the same. If there is an interest in the future to use these lines, then we can include them.

I added a mention of this in conll2003's dataset card:
https://github.com/huggingface/datasets/blob/fc9796920da88486c3b97690969aabf03d6b4088/datasets/conll2003/README.md#conll2003

Edit: just saw your PR @mariosasko (noticed it too late ^^)
Let me take a look at it :)
Issue: wmt datasets fail to load
State: closed

on master:
```
python -c 'from datasets import load_dataset; load_dataset("wmt14", "de-en")'
Downloading and preparing dataset wmt14/de-en (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/stas/.cache/huggingface/datasets/wmt14/de-en/1.0.0/43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/load.py", line 740, in load_dataset
    builder_instance.download_and_prepare(
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/builder.py", line 578, in download_and_prepare
    self._download_and_prepare(
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/builder.py", line 634, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/stas/.cache/huggingface/modules/datasets_modules/datasets/wmt14/43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e/wmt_utils.py", line 760, in _split_generators
    extraction_map = dict(downloaded_files, **manual_files)
```

it worked fine recently. same problem if I try wmt16.

git bisect points to this commit from Feb 25 as the culprit https://github.com/huggingface/datasets/commit/792f1d9bb1c5361908f73e2ef7f0181b2be409fa

@albertvillanova 

@stas00 Mea culpa... May I fix this tomorrow morning?

yes, of course, I reverted to the version before that and it works ;)

but since a new release was just made you will probably need to make a hotfix.

and add the wmt to the tests?

Sure, I will implement a regression test!

@stas00 it is fixed. @lhoestq are you releasing the hot fix or would you prefer me to do it?

I'll do a patch release for this issue early tomorrow.

And yes we absolutly need tests for the wmt datasets: The missing tests for wmt are an artifact from the early development of the lib but now we have tools to generate automatically the dummy data used for tests :)

still facing the same issue or similar:
from datasets import load_dataset
wtm14_test = load_dataset('wmt14',"de-en",cache_dir='./datasets')

~.cache\huggingface\modules\datasets_modules\datasets\wmt14\43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e\wmt_utils.py in _split_generators(self, dl_manager)
758 # Extract manually downloaded files.
759 manual_files = dl_manager.extract(manual_paths_dict)
--> 760 extraction_map = dict(downloaded_files, **manual_files)
761
762 for language in self.config.language_pair:

TypeError: type object argument after ** must be a mapping, not list
Issue: Question: what gets stored in the datasets cache and why is it so huge?
State: closed

I'm running several training jobs (around 10) with a relatively large dataset (3M samples). The datasets cache reached 178G and it seems really large. What is it stored in there and why is it so large? I don't think I noticed this problem before and seems to be related to the new version of the datasets library. Any insight? Thank you!

Echo'ing this observation: I have a few datasets in the neighborhood of 2GB CSVs uncompressed, and when I use something like `Dataset.save_to_disk()` it's ~18GB on disk.

If this is unexpected behavior, would be happy to help run debugging as needed.

Thanks @ioana-blue for pointing out this problem (and thanks also @justin-yan). You are right that current implementation of the datasets caching files take too much memory. We are definitely changing this and optimizing the defaults, so that the file sizes are considerably reduced. I will come back to you as soon as this is fixed.

Thank you! Also I noticed that the files don't seem to be cleaned after the jobs finish. Last night I had only 3 jobs running, but the cache was still at 180GB. 

And to clarify, it's not memory, it's disk space. Thank you!

Hi ! As Albert said they can sometimes take more space that expected but we'll fix that soon.

Also, to give more details about caching: computations on a dataset are cached by default so that you don't have to recompute them the next time you run them.

So by default the cache files stay on your disk when you job is finished (so that if you re-execute it, it will be reloaded from the cache).
Feel free to clear your cache after your job has finished, or disable caching using
```python
import datasets

datasets.set_caching_enabled(False)
```

Thanks for the tip, this is useful. 

Hi @ioana-blue, we have optimized Datasets' disk usage in the latest release v1.5.

Feel free to update your Datasets version
```shell
pip install -U datasets
```
and see if it better suits your needs.

Thank you!
Issue: Can we parallelized the add_faiss_index process over dataset shards ?
State: closed

I am thinking of making the  **add_faiss_index** process faster. What if we run the add_faiss_index process on separate dataset shards and then combine them before (dataset.concatenate) saving the faiss.index file ?

I feel theoretically this will reduce the accuracy of retrieval since it affects the indexing process.

@lhoestq


Hi !
As far as I know not all faiss indexes can be computed in parallel and then merged. 
For example [here](https://github.com/facebookresearch/faiss/wiki/Special-operations-on-indexes#splitting-and-merging-indexes) is is mentioned that only IndexIVF indexes can be merged.
Moreover faiss already works using multithreading to parallelize the workload over your different CPU cores. You can find more info [here](https://github.com/facebookresearch/faiss/wiki/Threads-and-asynchronous-calls#internal-threading)
So I feel like the gains we would get by implementing a parallel `add_faiss_index` would not be that important, but let me know what you think.


Actually, you are right. I also had the same idea. I am trying this in the context of end-ton-end retrieval training in RAG.  So far I have parallelized the embedding re-computation within the training loop by using datasets shards. 

Then I was thinking of can I calculate the indexes for each shard and combined them with **concatenate**  before I save.

@lhoestq  As you mentioned faiss is already using multiprocessing. I tried to do the add_index with faiss for a dataset object inside a RAY actor and the process became very slow... if fact it takes so much time. It is because a ray actor comes with a single CPU core unless we assign it more. I also tried assigning more cores but still running add_index in the main process is very fast.
Issue: bug in SNLI dataset 
State: closed

Hi
There is label of -1 in train set of SNLI dataset, please find the code below:

```
import numpy as np 
import datasets 
data = datasets.load_dataset("snli")["train"]
labels = []
for d in data:
   labels.append(d["label"])
print(np.unique(labels))
```

and results:

`[-1  0  1  2]`

version of datasets used:
`datasets                  1.2.1                     <pip>
`

thanks for your help. @lhoestq 

Hi ! The labels -1 correspond to the examples without gold labels in the original snli dataset.
Feel free to remove these examples if you don't need them by using
```python
data = data.filter(lambda x: x["label"] != -1)
```
Issue: Bug in skip_rows argument of load_dataset function ?
State: closed

Hello everyone,

I'm quite new to Git so sorry in advance if I'm breaking some ground rules of issues posting... :/
I tried to use the load_dataset function, from Huggingface datasets library, on a csv file using the skip_rows argument described on Huggingface page to skip the first row containing column names

`test_dataset = load_dataset('csv', data_files=['test_wLabel.tsv'], delimiter='\t', column_names=["id", "sentence", "label"], skip_rows=1)`

But I got the following error message

`__init__() got an unexpected keyword argument 'skip_rows'`

Have I used the wrong argument ? Am I missing something or is this a bug ?

Thank you very much for your time,
Best regards,
Arthur

Hi,

try `skiprows` instead. This part is not properly documented in the docs it seems.

@lhoestq I'll fix this as part of a bigger PR that fixes typos in the docs.
Issue: XSum dataset download link broken
State: closed

I did 
```
from datasets import load_dataset

dataset = load_dataset("xsum")
```

This returns
`ConnectionError: Couldn't reach http://bollin.inf.ed.ac.uk/public/direct/XSUM-EMNLP18-Summary-Data-Original.tar.gz`

Never mind, I ran it again and it worked this time. Strange.
Issue: [distributed env] potentially unsafe parallel execution
State: closed

```
metric = load_metric('glue', 'mrpc', num_process=num_process, process_id=rank)
```

presumes that there is only one set of parallel processes running - and will intermittently fail if you have multiple sets running as they will surely overwrite each other. Similar to https://github.com/huggingface/datasets/issues/1942 (but for a different reason).
That's why dist environments use some unique to a group identifier so that each group is dealt with separately. 

e.g. the env-way of pytorch dist syncing is done with a unique per set `MASTER_ADDRESS+MASTER_PORT`

So ideally this interface should ask for a shared secret to do the right thing.

I'm not reporting an immediate need, but am only flagging that this will hit someone down the road.

This problem can be remedied by adding a new optional `shared_secret` option, which can then be used to differentiate different groups of processes. and this secret should be part of the file lock name and the experiment.

Thank you

You can pass the same `experiment_id` for all the metrics of the same group, and use another `experiment_id` for the other groups.
Maybe we can add an environment variable that sets the default value for `experiment_id` ? What do you think ?

Ah, you're absolutely correct, @lhoestq - it's exactly the equivalent of the shared secret. Thank you!
Issue: add a new column 
State: closed

Hi
I'd need to add a new column to the dataset, I was wondering how this can be done? thanks 
@lhoestq 

Hi
not sure how change the lable after creation, but this is an issue not dataset request. thanks 

Hi ! Currently you have to use `map` . You can see an example of how to do it in this comment: https://github.com/huggingface/datasets/issues/853#issuecomment-727872188

In the future we'll add support for a more native way of adding a new column ;)
Issue: Enable Fast Filtering using Arrow Dataset
State: open

Hi @lhoestq,

As mentioned in Issue #1796, I would love to work on enabling fast filtering/mapping. Can you please share the expectations? It would be great if you could point me to the relevant methods/files involved. Or the docs or maybe an overview of `arrow_dataset.py`. I only ask this because I am having trouble getting started ;-;

Any help would be appreciated.

Thanks,
Gunjan

Hi @gchhablani :)
Thanks for proposing your help !

I'll be doing a refactor of some parts related to filtering in the scope of https://github.com/huggingface/datasets/issues/1877
So I would first wait for this refactor to be done before working on the filtering. In particular because I plan to make things simpler to manipulate.

Your feedback on this refactor would also be appreciated since it also aims at making the core code more accessible (basically my goal is that no one's ever "having troubles getting started" ^^)

This will be available in a few days, I will be able to give you more details at that time if you don't mind waiting a bit !

Sure! I don't mind waiting. I'll check the refactor and try to understand what you're trying to do :)
Issue: AttributeError: 'DatasetDict' object has no attribute 'concatenate_datasets'
State: closed

Hi
I am trying to concatenate a list of huggingface datastes as:

` train_dataset = datasets.concatenate_datasets(train_datasets)
`
Here is the `train_datasets` when I print:

```
[Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 120361
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 2670
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 6944
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 38140
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 173711
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 1655
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 4274
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 2019
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 2109
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 11963
})]
```

I am getting the following error:

`AttributeError: 'DatasetDict' object has no attribute 'concatenate_datasets'
`

I was wondering if you could help me with this issue, thanks a lot 

sorry my mistake, datasets were overwritten closing now, thanks a lot
Issue: CommonGen dataset page shows an error OSError: [Errno 28] No space left on device
State: closed
Labels: name: nlp-viewer

The page of the CommonGen data https://huggingface.co/datasets/viewer/?dataset=common_gen  shows 
![image](https://user-images.githubusercontent.com/10104354/108959311-1865e600-7629-11eb-868c-cf4cb27034ea.png)


Facing the same issue for [Squad](https://huggingface.co/datasets/viewer/?dataset=squad) and [TriviaQA](https://huggingface.co/datasets/viewer/?dataset=trivia_qa) datasets as well.

We just fixed the issue, thanks for reporting !
Issue: Add Stanford Sentiment Treebank (SST)
State: closed
Labels: name: dataset request

I am going to add SST:

- **Name:** The Stanford Sentiment Treebank
- **Description:** The first corpus with fully labeled parse trees that allows for a complete analysis of the compositional effects of sentiment in language
- **Paper:** [Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)
- **Data:** https://nlp.stanford.edu/sentiment/index.html
- **Motivation:** Already requested in #353, SST is a popular dataset for Sentiment Classification

What's the difference with the [_SST-2_](https://huggingface.co/datasets/viewer/?dataset=glue&config=sst2) dataset included in GLUE? Essentially, SST-2 is a version of SST where:
- the labels were mapped from real numbers in [0.0, 1.0] to a binary label: {0, 1}
- the labels of the *sub-sentences* were included only in the training set
- the labels in the test set are obfuscated

So there is a lot more information in the original SST. The tricky bit is, the data is scattered into many text files and, for one in particular, I couldn't find the original encoding ([*but I'm not the only one*](https://groups.google.com/g/word2vec-toolkit/c/QIUjLw6RqFk/m/_iEeyt428wkJ) 🎵). The only solution I found was to manually replace all the è, ë, ç and so on into an `utf-8` copy of the text file. I uploaded the result in my Dropbox and I am using that as the main repo for the dataset.

Also, the _sub-sentences_ are built at run-time from the information encoded in several text files, so generating the examples is a bit more cumbersome than usual. Luckily, the dataset is not enormous.

I plan to divide the dataset in 2 configs: one with just whole sentences with their labels, the other with sentences _and their sub-sentences_ with their labels. Each config will be split in train, validation and test. Hopefully this makes sense, we may discuss it in the PR I'm going to submit.





Dataset added in release [1.5.0](https://github.com/huggingface/datasets/releases/tag/1.5.0), I think I can close this.
Issue: Anonymous Dataset Addition (i.e Anonymous PR?)
State: closed

Hello,
Thanks a lot for your librairy.
We plan to submit a paper on OpenReview using the Anonymous setting. Is it possible to add a new dataset without breaking the anonimity, with a link to the paper ? 
Cheers 
@eusip

Hi !
I guess you can add a dataset without the fields that must be kept anonymous, and then update those when the anonymity period is over.
You can also make the PR from an anonymous org.
Pinging @yjernite just to make sure it's ok

Hello,
I would prefer to do the reverse: adding a link to an anonymous paper without the people names/institution in the PR. Would it be conceivable ?
Cheers


Sure, I think it's ok on our side

Yup, sounds good!
Issue: How to update the "wino_bias" dataset
State: open

Hi all,

Thanks for the efforts to collect all the datasets! But I think there is a problem with the wino_bias dataset. The current link is not correct. How can I update that?

Thanks!

Hi @JieyuZhao !

You can edit the dataset card of wino_bias to update the URL via a Pull Request. This would be really appreciated :)

The dataset card is the README.md file you can find at https://github.com/huggingface/datasets/tree/master/datasets/wino_bias
Also the homepage url is also mentioned in the wino_bias.py so feel free to update it there as well.

You can create a Pull Request directly from the github interface by editing the files you want and submit a PR, or from a local clone of the repository.

Thanks for noticing !
Issue: Failure to save with save_to_disk
State: closed

When I try to save a dataset locally using the `save_to_disk` method I get the error:

```bash
FileNotFoundError: [Errno 2] No such file or directory: '/content/squad/train/squad-train.arrow'
```

To replicate:

1. Install `datasets` from master
2. Run this code:

    ```python
    from datasets import load_dataset
    squad = load_dataset("squad")   # or any other dataset
    squad.save_to_disk("squad")     # error here
    ```

The problem is that the method is not creating a directory with the name `dataset_path` for saving the dataset in (i.e. it's not creating the *train* and *validation* directories in this case). After creating the directory the problem resolves.
I'll open a PR soon doing that and linking this issue.


Hi thanks for reporting and for proposing a fix :)

I just merged a fix, feel free to try it from the master branch !

Closing since this has been fixed by #1923
Issue: UnicodeDecodeError: windows 10 machine
State: closed

Windows 10
Php 3.6.8

when running

```
import datasets

oscar_am = datasets.load_dataset("oscar", "unshuffled_deduplicated_am")
print(oscar_am["train"][0])
```
I get the following error

```
file "C:\PYTHON\3.6.8\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 58: character maps to <undefined>
```

upgraded to php 3.9.2 and it works!
Issue: DBPedia14 Dataset Checksum bug?
State: closed

Hi there!!!

I've been using successfully the DBPedia dataset (https://huggingface.co/datasets/dbpedia_14) with my codebase in the last couple of weeks, but in the last couple of days now I get this error:

```
Traceback (most recent call last):
  File "./conditional_classification/basic_pipeline.py", line 178, in <module>
    main()
  File "./conditional_classification/basic_pipeline.py", line 128, in main
    corpus.load_data(limit_train_examples_per_class=args.data_args.train_examples_per_class,
  File "/home/fp/dev/conditional_classification/conditional_classification/datasets_base.py", line 83, in load_data
    datasets = load_dataset(self.name, split=dataset_split)
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/load.py", line 609, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/builder.py", line 526, in download_and_prepare
    self._download_and_prepare(
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/builder.py", line 586, in _download_and_prepare
    verify_checksums(
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 39, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbQ2Vic1kxMmZZQ1k']
```

I've seen this has happened before in other datasets as reported in #537.

I've tried clearing my cache and call again `load_dataset` but still is not working. My same codebase is successfully downloading and using other datasets (e.g. AGNews) without any problem, so I guess something has happened specifically to the DBPedia dataset in the last few days. 

Can you please check if there's a problem with the checksums? 

Or this is related to any other stuff? I've seen that the path in the cache for the dataset is `/home/fp/.cache/huggingface/datasets/d_bpedia14/dbpedia_14/2.0.0/a70413e39e7a716afd0e90c9e53cb053691f56f9ef5fe317bd07f2c368e8e897...` and includes `d_bpedia14` instead maybe of `dbpedia_14`. Was this maybe a bug introduced recently?

Thanks!

Hi ! :)

This looks like the same issue as https://github.com/huggingface/datasets/issues/1856 
Basically google drive has quota issues that makes it inconvenient for downloading files.

If the quota of a file is exceeded, you have to wait 24h for the quota to reset (which is painful).

The error says that the checksum of the downloaded file doesn't match because google drive returns a text file with the "Quota Exceeded" error instead of the actual data file.

Thanks @lhoestq! Yes, it seems back to normal after a couple of days.
Issue: ALT dataset has repeating instances in all splits
State: closed
Labels: name: dataset bug

The [ALT](https://huggingface.co/datasets/alt) dataset has all the same instances within each split :/
Seemed like a great dataset for some experiments I wanted to carry out, especially since its medium-sized, and has all splits.

Would be great if this could be fixed :)

Added a snapshot of the contents from `explore-datset` feature, for quick reference.

![image](https://user-images.githubusercontent.com/33179372/108206321-442a2d00-714c-11eb-882f-b4b6e708ef9c.png)


Thanks for reporting. This looks like a very bad issue. I'm looking into it

I just merged a fix, we'll do a patch release soon. Thanks again for reporting, and sorry for the inconvenience.
In the meantime you can load `ALT` using `datasets` from the master branch

Thanks!!! works perfectly in the bleading edge master version

Closed by #1899
Issue: request to mirror wmt datasets, as they are really slow to download
State: closed

Would it be possible to mirror the wmt data files under hf? Some of them take hours to download and not because of the local speed. They are all quite small datasets, just extremely slow to download.

Thank you!

Yes that would be awesome. Not only the download speeds are awful, but also some files are missing.
We list all the URLs in the datasets/wmt19/wmt_utils.py so we can make a script to download them all and host on S3.
Also I think most of the materials are under the CC BY-NC-SA 3.0 license (must double check) so it should be possible to redistribute the data with no issues.

cc @patrickvonplaten who knows more about the wmt scripts

Yeah, the scripts are pretty ugly! A big refactor would make sense here...and I also remember that the datasets were veeery slow to download

I'm downloading them.
I'm starting with the ones hosted on http://data.statmt.org which are the slowest ones

@lhoestq better to use our new git-based system than just raw S3, no? (that way we have built-in CDN etc.)

Closing since the urls were changed to mirror urls in #1912 

Hi there! What about mirroring other datasets like [CCAligned](http://www.statmt.org/cc-aligned/) as well? All of them are really slow to download...
Issue: Add Winogender Schemas
State: closed
Labels: name: dataset request

## Adding a Dataset
- **Name:** Winogender Schemas
- **Description:** Winogender Schemas (inspired by Winograd Schemas) are minimal pairs of sentences that differ only by the gender of one pronoun in the sentence, designed to test for the presence of gender bias in automated coreference resolution systems.
- **Paper:** https://arxiv.org/abs/1804.09301
- **Data:** https://github.com/rudinger/winogender-schemas (see data directory)
- **Motivation:** Testing gender bias in automated coreference resolution systems, improve coreference resolution in general.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Nevermind, this one is already available on the hub under the name `'wino_bias'`: https://huggingface.co/datasets/wino_bias
Issue: Add WikiCREM
State: open
Labels: name: dataset request

## Adding a Dataset
- **Name:** WikiCREM
- **Description:** A large unsupervised corpus for coreference resolution.
- **Paper:** https://arxiv.org/abs/1905.06290
- **Github repo:**: https://github.com/vid-koci/bert-commonsense
- **Data:** https://ora.ox.ac.uk/objects/uuid:c83e94bb-7584-41a1-aef9-85b0e764d9e3
- **Motivation:** Coreference resolution, common sense reasoning

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Hi @NielsRogge I would like to work on this dataset.

Thanks!

Hi @udapy, are you working on this?
Issue: Error "in void don't know how to serialize this type of index" when saving index to disk when device=0 (GPU)
State: closed

Error serializing faiss index.  Error as follows:

`Error in void faiss::write_index(const faiss::Index*, faiss::IOWriter*) at /home/conda/feedstock_root/build_artifacts/faiss-split_1612472484670/work/faiss/impl/index_write.cpp:453: don't know how to serialize this type of index`


Note:

`torch.cuda.is_available()` reports:

```
Cuda is available
cuda:0

```

Adding index, device=0 for GPU.

`dataset.add_faiss_index(column='embeddings', index_name='idx_embeddings', device=0)`

However, during a quick debug, self.faiss_index has no attr "device" when checked in` search.py, method save`, so fails to transform gpu index to cpu index.  If I add index without device, index is saved OK.


```
def save(self, file: str):
        """Serialize the FaissIndex on disk"""
        import faiss  # noqa: F811

        if (
            hasattr(self.faiss_index, "device")
            and self.faiss_index.device is not None
            and self.faiss_index.device > -1
        ):
            index = faiss.index_gpu_to_cpu(self.faiss_index)
        else:
            index = self.faiss_index
        faiss.write_index(index, file)
```


Hi @corticalstack ! Thanks for reporting. Indeed in the recent versions of Faiss we must use `getDevice` to check if the index in on GPU.

I'm opening a PR

I fixed this issue. It should work fine now.
Feel free to try it out by installing `datasets` from source.
Otherwise you can wait for the next release of `datasets` (in a few days)

Thanks for such a quick fix and merge to master, pip installed git master, tested all OK
Issue: Unable to upload "community provided" dataset - 400 Client Error
State: closed

Hi,
i'm trying to a upload a dataset as described [here](https://huggingface.co/docs/datasets/v1.2.0/share_dataset.html#sharing-a-community-provided-dataset). This is what happens:

``` 
$ datasets-cli login
$ datasets-cli upload_dataset my_dataset
About to upload file /path/to/my_dataset/dataset_infos.json to S3 under filename my_dataset/dataset_infos.json and namespace username
About to upload file /path/to/my_dataset/my_dataset.py to S3 under filename my_dataset/my_dataset.py and namespace username
Proceed? [Y/n] Y
Uploading... This might take a while if files are large
400 Client Error: Bad Request for url: https://huggingface.co/api/datasets/presign
huggingface.co migrated to a new model hosting system.
You need to upgrade to transformers v3.5+ to upload new models.
More info at https://discuss.hugginface.co or https://twitter.com/julien_c. Thank you! 
```
I'm using the latest releases of datasets and transformers.

Hi ! We're in the process of switching the community datasets to git repos, exactly like what we're doing for models.
You can find an example here:
https://huggingface.co/datasets/lhoestq/custom_squad/tree/main

We'll update the CLI in the coming days and do a new release :)

Also cc @julien-c maybe we can make improve the error message ?
Issue: Feature Request: Dataset.add_item
State: closed
Labels: name: enhancement

I'm trying to integrate `huggingface/datasets` functionality into `fairseq`, which requires (afaict) being able to build a dataset through an `add_item` method, such as https://github.com/pytorch/fairseq/blob/master/fairseq/data/indexed_dataset.py#L318, as opposed to loading all the text into arrow, and then `dataset.map(binarizer)`.
Is this possible at the moment? Is there an example? I'm happy to use raw `pa.Table` but not sure whether it will support uneven length entries.

### Desired API

```python
import numpy as np
tokenized: List[np.NDArray[np.int64]] = [np.array([4,4,2]), np.array([8,6,5,5,2]), np.array([3,3,31,5])

def build_dataset_from_tokenized(tokenized: List[np.NDArray[int]]) -> Dataset:
   """FIXME"""
   dataset = EmptyDataset()
   for t in tokenized: dataset.append(t)
   return dataset
ds = build_dataset_from_tokenized(tokenized)
assert (ds[0] == np.array([4,4,2])).all()
```

### What I tried
grep, google for "add one entry at a time", "datasets.append"

### Current Code
This code achieves the same result but doesn't fit into the `add_item` abstraction.

```python
    dataset = load_dataset('text', data_files={'train': 'train.txt'})
    tokenizer = RobertaTokenizerFast.from_pretrained('roberta-base', max_length=4096)
    def tokenize_function(examples):
        ids = tokenizer(examples['text'], return_attention_mask=False)['input_ids']
        return {'input_ids': [x[1:] for x in ids]}
    ds = dataset.map(tokenize_function, batched=True, num_proc=4, remove_columns=['text'], load_from_cache_file=not overwrite_cache)
	print(ds['train'][0]) => np array
```

Thanks in advance!

Hi @sshleifer.

I am not sure of understanding the need of the `add_item` approach...

By just reading your "Desired API" section, I would say you could (nearly) get it with a 1-column Dataset:
```python
data = {"input_ids": [np.array([4,4,2]), np.array([8,6,5,5,2]), np.array([3,3,31,5])]}
ds = Dataset.from_dict(data)
assert (ds["input_ids"][0] == np.array([4,4,2])).all()
```

Hi @sshleifer :) 

We don't have methods like `Dataset.add_batch` or `Dataset.add_entry/add_item` yet.
But that's something we'll add pretty soon. Would an API that looks roughly like this help ? Do you have suggestions ?
```python
import numpy as np
from datasets import Dataset

tokenized = [np.array([4,4,2]), np.array([8,6,5,5,2]), np.array([3,3,31,5])

# API suggestion (not available yet)
d = Dataset()
for input_ids in tokenized:
    d.add_item({"input_ids": input_ids})

print(d[0]["input_ids"])
# [4, 4, 2]
```

Currently you can define a dataset with what @albertvillanova suggest, or via a generator using dataset builders. It's also possible to [concatenate datasets](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=concatenate#datasets.concatenate_datasets).

Your API looks perfect @lhoestq, thanks!
Issue: Add TIMIT
State: closed
Labels: name: dataset request

name: speech

## Adding a Dataset
- **Name:** *TIMIT*
- **Description:** *The TIMIT corpus of read speech has been designed to provide speech data for the acquisition of acoustic-phonetic knowledge and for the development and evaluation of automatic speech recognition systems*

- **Paper:** *Homepage*: http://groups.inf.ed.ac.uk/ami/corpus/ / *Wikipedia*: https://en.wikipedia.org/wiki/TIMIT
- **Data:** *https://deepai.org/dataset/timit*
- **Motivation:** Important speech dataset


If interested in tackling this issue, feel free to tag @patrickvonplaten


Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


@patrickvonplaten Could you please help me with how the output text has to be represented in the data? TIMIT has Words, Phonemes and texts. Also has lot on info on the speaker and the dialect. Could you please help me? An example of how to arrange it would be super helpful!



Hey @vrindaprabhu - sure I'll help you :-) Could you open a first PR for TIMIT where you copy-paste more or less the `librispeech_asr` script: https://github.com/huggingface/datasets/blob/28be129db862ec89a87ac9349c64df6b6118aff4/datasets/librispeech_asr/librispeech_asr.py#L93 (obviously replacing all the naming and links correctly...) and then you can list all possible outputs in the features dict: https://github.com/huggingface/datasets/blob/28be129db862ec89a87ac9349c64df6b6118aff4/datasets/librispeech_asr/librispeech_asr.py#L104 (words, phonemes should probably be of kind `datasets.Sequence(datasets.Value("string"))` and texts I think should be of type `"text": datasets.Value("string")`.

When you've opened a first PR, I think it'll be much easier for us to take a look together :-) 

I am sorry! I created the PR [#1903](https://github.com/huggingface/datasets/pull/1903#). Requesting your comments! CircleCI tests are failing, will address them along with your comments!
Issue: Add AMI Corpus
State: closed
Labels: name: dataset request

name: speech

## Adding a Dataset
- **Name:** *AMI*
- **Description:** *The AMI Meeting Corpus is a multi-modal data set consisting of 100 hours of meeting recordings. For a gentle introduction to the corpus, see the corpus overview. To access the data, follow the directions given there. Around two-thirds of the data has been elicited using a scenario in which the participants play different roles in a design team, taking a design project from kick-off to completion over the course of a day. The rest consists of naturally occurring meetings in a range of domains. Detailed information can be found in the documentation section.*

- **Paper:** *Homepage*: http://groups.inf.ed.ac.uk/ami/corpus/
- **Data:** *http://groups.inf.ed.ac.uk/ami/download/* - Select all cases in 1) and select "Individual Headsets" & "Microphone array" for 2)
- **Motivation:** Important speech dataset


If interested in tackling this issue, feel free to tag @patrickvonplaten


Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Available here: ~https://huggingface.co/datasets/ami~ https://huggingface.co/datasets/edinburghcstr/ami

@mariosasko actually the "official" AMI dataset can be found here: https://huggingface.co/datasets/edinburghcstr/ami -> the old one under `datasets/ami` doesn't work and should be deleted. 

The new one was tested by fine-tuning a Wav2Vec2 model on it + we uploaded all the processed audio directly into it

@patrickvonplaten Thanks for correcting me! I've updated the link.
Issue: Add ljspeech
State: closed
Labels: name: dataset request

name: speech

## Adding a Dataset
- **Name:** *ljspeech*
- **Description:** *This is a public domain speech dataset consisting of 13,100 short audio clips of a single speaker reading passages from 7 non-fiction books. A transcription is provided for each clip. Clips vary in length from 1 to 10 seconds and have a total length of approximately 24 hours.

The texts were published between 1884 and 1964, and are in the public domain. The audio was recorded in 2016-17 by the LibriVox project and is also in the public domain.)*
- **Paper:** *Homepage*: https://keithito.com/LJ-Speech-Dataset/
- **Data:** *https://keithito.com/LJ-Speech-Dataset/*
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/ljspeech
If interested in tackling this issue, feel free to tag @patrickvonplaten


Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Issue: Add common voice
State: closed
Labels: name: dataset request

name: speech

## Adding a Dataset
- **Name:** *common voice*
- **Description:** *Mozilla Common Voice Dataset*
- **Paper:** Homepage: https://voice.mozilla.org/en/datasets
- **Data:** https://voice.mozilla.org/en/datasets
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/common_voice
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


I have started working on adding this dataset.

Hey @BirgerMoell - awesome that you started working on Common Voice. Common Voice is a bit special since, there is no direct download link to download the data. In these cases we usually consider two options:

1) Find a hacky solution to extract the download link somehow from the XLM tree of the website 
2) If this doesn't work we force the user to download the data himself and add a `"data_dir"` as an input parameter. E.g. you can take a look at how it is done for [this](https://github.com/huggingface/datasets/blob/66f2a7eece98d2778bd22bb5034cb7c2376032d4/datasets/arxiv_dataset/arxiv_dataset.py#L66) 

Also the documentation here: https://huggingface.co/docs/datasets/add_dataset.html?highlight=data_dir#downloading-data-files-and-organizing-splits (especially the "note") might be helpful.

Let me know if you have any other questions

I added a Work in Progress pull request (hope that is ok). I've made a card for the dataset and filled out the common_voice.py file with information about the datset (not completely).

I didn't manage to get the tagging tool working locally on my machine but will look into that later.

Left to do.

- Tag the dataset
- Add missing information and update common_voice.py

https://github.com/huggingface/datasets/pull/1886

Awesome! I left a longer comment on the PR :-)

I saw that this current datasets package holds common voice version 6.1, how to add the new version 7.0 that is already available?

Will me merged next week - we're working on it :-)

Common voice still appears to be a 6.1. Is the plan still to upgrade to 7.0?

We actually already have the code and everything ready to add Common Voice 7.0 to `datasets` but are still waiting for the common voice authors to give us the green light :-) 

Also gently pinging @phirework and @milupo here

Common Voice 7.0 is available here now: https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0

For anyone else stumbling upon this thread, the 8.0 version is also available now: https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0
Issue: Add Voxforge
State: open
Labels: name: dataset request

name: speech

## Adding a Dataset
- **Name:** *voxforge* 
- **Description:** *VoxForge is a language classification dataset. It consists of user submitted audio clips submitted to the website. In this release, data from 6 languages is collected - English, Spanish, French, German, Russian, and Italian. Since the website is constantly updated, and for the sake of reproducibility, this release contains only recordings submitted prior to 2020-01-01. The samples are splitted between train, validation and testing so that samples from each speaker belongs to exactly one split.*
- **Paper:** *Homepage*: http://www.voxforge.org/
- **Data:** *http://www.voxforge.org/home/downloads*
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/voxforge
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).