helmo/github-issues huggingface dataset:
DBPedia14 Dataset Checksum bug?
State: closed | Author: francisco-perez-sorrosal (CONTRIBUTOR)

Hi there!!!

I've been using successfully the DBPedia dataset (https://huggingface.co/datasets/dbpedia_14) with my codebase in the last couple of weeks, but in the last couple of days now I get this error:

```
Traceback (most recent call last):
  File "./conditional_classification/basic_pipeline.py", line 178, in <module>
    main()
  File "./conditional_classification/basic_pipeline.py", line 128, in main
    corpus.load_data(limit_train_examples_per_class=args.data_args.train_examples_per_class,
  File "/home/fp/dev/conditional_classification/conditional_classification/datasets_base.py", line 83, in load_data
    datasets = load_dataset(self.name, split=dataset_split)
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/load.py", line 609, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/builder.py", line 526, in download_and_prepare
    self._download_and_prepare(
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/builder.py", line 586, in _download_and_prepare
    verify_checksums(
  File "/home/fp/anaconda3/envs/conditional/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 39, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbQ2Vic1kxMmZZQ1k']
```

I've seen this has happened before in other datasets as reported in #537.

I've tried clearing my cache and call again `load_dataset` but still is not working. My same codebase is successfully downloading and using other datasets (e.g. AGNews) without any problem, so I guess something has happened specifically to the DBPedia dataset in the last few days. 

Can you please check if there's a problem with the checksums? 

Or this is related to any other stuff? I've seen that the path in the cache for the dataset is `/home/fp/.cache/huggingface/datasets/d_bpedia14/dbpedia_14/2.0.0/a70413e39e7a716afd0e90c9e53cb053691f56f9ef5fe317bd07f2c368e8e897...` and includes `d_bpedia14` instead maybe of `dbpedia_14`. Was this maybe a bug introduced recently?

Thanks!

Comments: Hi ! :)

This looks like the same issue as https://github.com/huggingface/datasets/issues/1856 
Basically google drive has quota issues that makes it inconvenient for downloading files.

If the quota of a file is exceeded, you have to wait 24h for the quota to reset (which is painful).

The error says that the checksum of the downloaded file doesn't match because google drive returns a text file with the "Quota Exceeded" error instead of the actual data file.

Thanks @lhoestq! Yes, it seems back to normal after a couple of days.
Standardizing datasets.dtypes
State: closed | Author: justin-yan (CONTRIBUTOR)

This PR was further branched off of jdy-str-to-pyarrow-parsing, so it depends on https://github.com/huggingface/datasets/pull/1900 going first for the diff to be up-to-date (I'm not sure if there's a way for me to use jdy-str-to-pyarrow-parsing as a base branch while having it appear in the pull requests here).

This moves away from `str(pyarrow.DataType)` as the method of choice for creating dtypes, favoring an explicit mapping to a list of supported Value dtypes.

I believe in practice this should be backward compatible, since anyone previously using Value() would only have been able to use dtypes that had an identically named pyarrow factory function, which are all explicitly supported here.

Comments: Also - I took a stab at updating the docs, but I'm not sure how to actually check the outputs to see if it's formatted properly.
Fix to_pandas for boolean ArrayXD
State: closed | Author: lhoestq (MEMBER)

As noticed in #1887 the conversion of a dataset with a boolean ArrayXD feature types fails because of the underlying ListArray conversion to numpy requires `zero_copy_only=False`.

zero copy is available for all primitive types except booleans
see https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.to_numpy
and https://issues.apache.org/jira/browse/ARROW-2871?jql=text%20~%20%22boolean%20to_numpy%22

cc @SBrandeis 

Comments: Thanks!
Initial commit for the addition of TIMIT dataset
State: closed | Author: vrindaprabhu (CONTRIBUTOR)

Below points needs to be addressed:

- Creation of dummy dataset is failing
- Need to check on the data representation
- License is not creative commons. Copyright: Portions © 1993 Trustees of the University of Pennsylvania

Also the links (_except the download_) point to the ami corpus! ;-)

@patrickvonplaten  Requesting your comments, will be happy to address them!

Comments: @patrickvonplaten could you please review and help me close this PR?

@lhoestq  Thank you so much for your comments and for patiently reviewing the code. Have _hopefully_ included all the suggested changes. Let me know if any more changes are required.

Sorry the code had lots of silly errors from my side!:' Will be more careful from next time! :)
Issue #1895: Bugfix for string_to_arrow timestamp[ns] support
State: closed | Author: justin-yan (CONTRIBUTOR)

Should resolve https://github.com/huggingface/datasets/issues/1895

The main part of this PR adds additional parsing in `string_to_arrow` to convert the timestamp dtypes that result from `str(pa_type)` back into the pa.DataType TimestampType.

While adding unit-testing, I noticed that support for the double/float types also don't invert correctly, so I added them, which I believe would hypothetically make this section of `Value` redundant:

```
    def __post_init__(self):
        if self.dtype == "double":  # fix inferred type
            self.dtype = "float64"
        if self.dtype == "float":  # fix inferred type
            self.dtype = "float32"
```

However, since I think Value.dtype is part of the public interface, removing that would result in a backward-incompatible change, so I didn't muck with that.

The rest of the PR consists of docstrings that I added while developing locally so I could keep track of which functions were supposed to be inverses of each other, and thought I'd include them initially in case you want to keep them around, but I'm happy to delete or remove any of them at your request!

Comments: OK! Thank you for the review - I will follow up with a separate PR for the comments here (https://github.com/huggingface/datasets/pull/1900#discussion_r578319725)!
ALT dataset has repeating instances in all splits
State: closed | Author: 10-zin (NONE)
Labels: name: dataset bug

The [ALT](https://huggingface.co/datasets/alt) dataset has all the same instances within each split :/
Seemed like a great dataset for some experiments I wanted to carry out, especially since its medium-sized, and has all splits.

Would be great if this could be fixed :)

Added a snapshot of the contents from `explore-datset` feature, for quick reference.

![image](https://user-images.githubusercontent.com/33179372/108206321-442a2d00-714c-11eb-882f-b4b6e708ef9c.png)


Comments: Thanks for reporting. This looks like a very bad issue. I'm looking into it

I just merged a fix, we'll do a patch release soon. Thanks again for reporting, and sorry for the inconvenience.
In the meantime you can load `ALT` using `datasets` from the master branch

Thanks!!! works perfectly in the bleading edge master version

Closed by #1899
Fix PandasArrayExtensionArray conversion to native type
State: closed | Author: lhoestq (MEMBER)

To make the conversion to csv work in #1887 , we need  PandasArrayExtensionArray used for multidimensional numpy arrays to be converted to pandas native types.
However previously pandas.core.internals.ExtensionBlock.to_native_types would fail with an PandasExtensionArray because
1. the PandasExtensionArray.isna method was wrong
2. the conversion of a PandasExtensionArray to a numpy array with dtype=object was returning a multidimensional array while pandas excepts a 1D array in this case (more info [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.api.extensions.ExtensionArray.html#pandas.api.extensions.ExtensionArray))

I fixed these two issues and now the conversion to native types works, and so is the export to csv.
cc @SBrandeis
request to mirror wmt datasets, as they are really slow to download
State: closed | Author: stas00 (CONTRIBUTOR)

Would it be possible to mirror the wmt data files under hf? Some of them take hours to download and not because of the local speed. They are all quite small datasets, just extremely slow to download.

Thank you!

Comments: Yes that would be awesome. Not only the download speeds are awful, but also some files are missing.
We list all the URLs in the datasets/wmt19/wmt_utils.py so we can make a script to download them all and host on S3.
Also I think most of the materials are under the CC BY-NC-SA 3.0 license (must double check) so it should be possible to redistribute the data with no issues.

cc @patrickvonplaten who knows more about the wmt scripts

Yeah, the scripts are pretty ugly! A big refactor would make sense here...and I also remember that the datasets were veeery slow to download

I'm downloading them.
I'm starting with the ones hosted on http://data.statmt.org which are the slowest ones

@lhoestq better to use our new git-based system than just raw S3, no? (that way we have built-in CDN etc.)

Closing since the urls were changed to mirror urls in #1912 

Hi there! What about mirroring other datasets like [CCAligned](http://www.statmt.org/cc-aligned/) as well? All of them are really slow to download...
Implement to_csv for Dataset
State: closed | Author: SBrandeis (CONTRIBUTOR)

cc @thomwolf 

`to_csv` supports passing either a file path or a *binary* file object
The writing is batched to avoid loading the whole table in memory

Comments: @lhoestq I stumbled upon an interesting failure when adding tests for CSV serialization of `ArrayXD` features (see the failing unit tests in the CI)

It's due to the fact that booleans cannot be converted from arrow format to numpy without copy: https://arrow.apache.org/docs/python/generated/pyarrow.Array.html#pyarrow.Array.to_numpy

Good catch ! I must be able to fix that one by allowing copies for this kind of arrays.
This is the kind of surprise you get sometimes when playing with arrow x)

Raising this error for booleans was introduced in https://issues.apache.org/jira/browse/ARROW-2871?jql=text%20~%20%22boolean%20to_numpy%22 without much explanations unfortunately.
So "no copy" only works for primitive types - except booleans.
This is confirmed in the source code at https://github.com/wesm/arrow/blob/c07b9b48cf3e0bbbab493992a492ae47e5b04cad/python/pyarrow/array.pxi#L621

I'm opening a PR to allow copies for booleans...

I just merged the fix for boolean ArrayXD, feel free to merge from master to see if it fixes the ci :)

@lhoestq unfirtunately, arrays of strings (or any other non-primitive type) require a copy too

A list of primitive types can be found here: https://github.com/wesm/arrow/blob/c07b9b48cf3e0bbbab493992a492ae47e5b04cad/python/pyarrow/types.pxi#L821

pyarrow provides a `is_primitive` function to check whether a type is primitive , I used it to set `zero_copy_only`

Also, `PandasArrayExtensionArray.isna` was using `numpy.isnan` which fails for arrays of strings. I replaced it with `pandas.isna`. Let me know what you think! :)
Add not-in-place implementations for several dataset transforms
State: closed | Author: SBrandeis (CONTRIBUTOR)

Should we deprecate in-place versions of such methods?

Comments: @lhoestq I am not sure how to test `dictionary_encode_column` (in-place version was not tested before)

I can take a look at dictionary_encode_column tomorrow.
Although it's likely that it doesn't work then. It was added at the beginning of the lib and never tested nor used afaik.

Now let's update the documentation to use the new methods x)
Create Remote Manager
State: open | Author: albertvillanova (MEMBER)

Refactoring to separate the concern of remote (HTTP/FTP requests) management.

Comments: @lhoestq I have refactorized the logic. Instead of the previous hierarchy call (local temp file opening -> remote call -> use again temp local file logic but from within the remote caller scope), now it is flattened. Schematically:
```python
with src.open() as src_file, dst.open() as dst_file:
    src_file.fetch(dst_file)
```

I have created `RemotePath` (analogue to Path) with method `.open()` that returns `FtpFile`/`HttpFile` (analogue to file-like).

Now I am going to implement `RemotePath.exists()` method (analogue to the Path's method) to check if remote resource is accessible, using `Ftp/Http.head()`.

Quick update on this one:
we discussed offline with @albertvillanova on this PR and I think using `fsspec` can help a lot, since it already implements many parts of the abstraction we need to have nice download tools for both http and ftp (and others !)
`list_datasets()` returns a list of strings, not objects
State: closed | Author: pminervini (CONTRIBUTOR)

Here and there in the docs there is still stuff like this:

```python
>>> datasets_list = list_datasets()
>>> print(', '.join(dataset.id for dataset in datasets_list))
```

However, my understanding is that `list_datasets()` returns a list of strings rather than a list of objects.
Replace flatten_nested
State: closed | Author: albertvillanova (MEMBER)

Replace `flatten_nested` with `NestedDataStructure.flatten`.

This is a first step towards having all NestedDataStructure logic as a separated concern, independent of the caller/user of the data structure.

Eventually, all checks (whether the underlying data is list, dict, etc.) will be only inside this class.

I have also generalized the flattening, and now it handles multiple levels of nesting.

Comments: Hi @lhoestq. If you agree to merge this, I will start separating the logic for NestedDataStructure.map ;)
Add LJ Speech dataset
State: closed | Author: anton-l (MEMBER)

This PR adds the LJ Speech dataset (https://keithito.com/LJ-Speech-Dataset/)
As requested by #1841 
The ASR format is based on #1767 

There are a couple of quirks that should be addressed:
- I tagged this dataset as `other-other-automatic-speech-recognition` and `other-other-text-to-speech` (as classified by paperswithcode). Since the number of speech datasets is about to grow, maybe these categories should be added to the main list? 
- Similarly to #1767 this dataset uses only a single dummy sample to reduce the zip size (`wav`s are quite heavy). Is there a plan to allow LFS or S3 usage for dummy data in the repo?
- The dataset is distributed under the Public Domain license, which is not used anywhere else in the repo, AFAIK. Do you think Public Domain is worth adding to the tagger app as well?

Pinging @patrickvonplaten to review

Comments: Hey @anton-l,

Thanks a lot for the very clean integration!

1) I think we should now start having "automatic-speech-recognition" as a label in the dataset tagger (@yjernite is it easy to add?). But we can surely add this dataset with the tag you've added and then later change the label to `asr` 

2) That's perfect! Yeah good question - we're currently thinking about a better design with @lhoestq 

3) Again tagging @yjernite & @lhoestq here - guess we should add this license though!

Thanks @anton-l for adding this one :)
About the points you mentioned:
1. Sure as soon as we've updated the tag sets in https://github.com/huggingface/datasets-tagging/blob/main/task_set.json, we can update the tags in this dataset card and also in the other audio dataset card.
2. For now we just try to have them as small as possible but we may switch to S3/LFS at one point indeed
3. If it's not part of the license set at https://github.com/huggingface/datasets-tagging/blob/main/license_set.json we can add it to this license set

For now it's ok to have the other-* tags but we'll update them very soon

Let's merge this one and then we'll update the tags for the audio datasets. We'll probably also add something like this:
```
type:
- text
- audio
```

Thank you so much for adding this one, good job !
add iapp_wiki_qa_squad
State: closed | Author: cstorm125 (CONTRIBUTOR)

`iapp_wiki_qa_squad` is an extractive question answering dataset from Thai Wikipedia articles.
It is adapted from [the original iapp-wiki-qa-dataset](https://github.com/iapp-technology/iapp-wiki-qa-dataset)
to [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) format, resulting in
5761/742/739 questions from 1529/191/192 articles.
Implement Dataset add_item
State: closed | Author: albertvillanova (MEMBER)
Labels: name: enhancement

Implement `Dataset.add_item`.

Close #1854.

Comments: Thanks @lhoestq for your remarks. Yes, I agree there are still many issues to be tackled... This PR is just a starting point, so that we can discuss how Dataset should be generalized.

Sure ! I opened an issue #1877 so we can discuss this specific aspect :)

I am going to implement this consolidation step in #2151.

Sounds good !

I retake this PR once the consolidation step is already implemented by #2151.
Update oscar sizes
State: closed | Author: lhoestq (MEMBER)

This commit https://github.com/huggingface/datasets/commit/837a152e4724adc5308e2c4481908c00a8d93383 removed empty lines from the oscar deduplicated datasets. This PR updates the size of each deduplicated dataset to fix possible `NonMatchingSplitsSizesError` errors. cc @cahya-wirawan
Updated OPUS Open Subtitles Dataset with metadata information
State: closed | Author: Valahaar (CONTRIBUTOR)

Close #1844 

Problems:
- I ran `python datasets-cli test datasets/open_subtitles --save_infos --all_configs`, hence the change in `dataset_infos.json`, but it appears that the metadata features have not been added for all pairs. Any idea why that might be?
- Possibly related to the above, I tried doing `pip uninstall datasets && pip install -e ".[dev]"` after the changes, and loading the dataset via `load_dataset("open_subtitles", lang1='hi', lang2='it')` to check if the update worked, but the loaded dataset did not contain the metadata fields (neither in the features nor doing `next(iter(dataset['train']))`). What step(s) did I miss?

Questions:
- Is it ok to have a `classmethod` in there? I have not seen any in the few other datasets I have checked. I could make it a local method of the `_generate_examples` method, but I'd rather not duplicate the logic...

Comments: Hi !
About the problems you mentioned:
- Saving the infos is only done for the configurations inside the BUILDER_CONFIGS. Otherwise you would need to run the scripts on ALL language pairs, which is not what we want.
- Moreover when you're on your branch, please specify the path to your local version of the dataset script, like "./datasets/open_subtitles". Otherwise the dataset is loaded from the master branch on github.
Hope that clarifies things a bit

And of course feel free to add methods or classmethods to your builder.


Great! Thank you :)
I'll close the issue as well.
Add Winogender Schemas
State: closed | Author: NielsRogge (CONTRIBUTOR)
Labels: name: dataset request

## Adding a Dataset
- **Name:** Winogender Schemas
- **Description:** Winogender Schemas (inspired by Winograd Schemas) are minimal pairs of sentences that differ only by the gender of one pronoun in the sentence, designed to test for the presence of gender bias in automated coreference resolution systems.
- **Paper:** https://arxiv.org/abs/1804.09301
- **Data:** https://github.com/rudinger/winogender-schemas (see data directory)
- **Motivation:** Testing gender bias in automated coreference resolution systems, improve coreference resolution in general.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments: Nevermind, this one is already available on the hub under the name `'wino_bias'`: https://huggingface.co/datasets/wino_bias
Add WikiCREM
State: open | Author: NielsRogge (CONTRIBUTOR)
Labels: name: dataset request

## Adding a Dataset
- **Name:** WikiCREM
- **Description:** A large unsupervised corpus for coreference resolution.
- **Paper:** https://arxiv.org/abs/1905.06290
- **Github repo:**: https://github.com/vid-koci/bert-commonsense
- **Data:** https://ora.ox.ac.uk/objects/uuid:c83e94bb-7584-41a1-aef9-85b0e764d9e3
- **Motivation:** Coreference resolution, common sense reasoning

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments: Hi @NielsRogge I would like to work on this dataset.

Thanks!

Hi @udapy, are you working on this?
Add loading from the Datasets Hub + add relative paths in download manager
State: closed | Author: lhoestq (MEMBER)

With the new Datasets Hub on huggingface.co it's now possible to have a dataset repo with your own script and data.
For example: https://huggingface.co/datasets/lhoestq/custom_squad/tree/main contains one script and two json files.

You can load it using
```python
from datasets import load_dataset

d = load_dataset("lhoestq/custom_squad")
```

To be able to use the data files that live right next to the dataset script on the repo in the hub, I added relative paths support for the DownloadManager. For example in the repo mentioned above, there are two json files that can be downloaded via
```python
_URLS = {
    "train": "train-v1.1.json",
    "dev": "dev-v1.1.json",
}
downloaded_files = dl_manager.download_and_extract(_URLS)
```

To make it work, I set the `base_path` of the DownloadManager to be the parent path of the dataset script (which comes from either a local path or a remote url).

I also had to add the auth header of the requests to huggingface.co for private datasets repos. The token is fetched from [huggingface_hub](https://github.com/huggingface/huggingface_hub).

Comments: I just added the steps to share a dataset on the datasets hub. It's highly inspired by the steps to share a model in the `transformers` doc.

Moreover once the new huggingface_hub is released we can update the version in the setup.py. We also need to update the command to create a dataset repo in the documentation

I added a few more tests with the "lhoestq/test" dataset I added on the hub and it works fine :) 

Here is the PR adding support for datasets repos in `huggingface_hub`: https://github.com/huggingface/huggingface_hub/pull/14