jamesLeeeeeee/datasets-github-issues huggingface dataset:
SciFact dataset - minor changes
Author: dwadden | Pull request: no | State: closed

Hi,

SciFact dataset creator here. First of all, thanks for adding the dataset to Huggingface, much appreciated!

I'd like to make a few minor changes, including the citation information and the `_URL` from which to download the dataset. Can I submit a PR for this?

It also looks like the dataset is being downloaded directly from Huggingface's Google cloud account rather than via the `_URL` in [scifact.py](https://github.com/huggingface/datasets/blob/master/datasets/scifact/scifact.py). Can you help me update the version on gcloud?

Thanks,

Dave
IsADirectoryError when trying to download C4
Author: fredriko | Pull request: no | State: closed

**TLDR**:

I fail to download C4 and see a stacktrace originating in `IsADirectoryError` as an explanation for failure.

How can the problem be fixed? 

**VERBOSE**:

I use Python version 3.7 and have the following dependencies listed in my project:

```
datasets==1.2.0
apache-beam==2.26.0
```

When running the following code, where `/data/huggingface/unpacked/` contains a single unzipped `wet.paths` file manually downloaded as per the instructions for C4:

```
from datasets import load_dataset

load_dataset("c4", "en", data_dir="/data/huggingface/unpacked", beam_runner='DirectRunner')
```

I get the following stacktrace:

```
/Users/fredriko/venv/misc/bin/python /Users/fredriko/source/misc/main.py
Downloading and preparing dataset c4/en (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /Users/fredriko/.cache/huggingface/datasets/c4/en/2.3.0/8304cf264cc42bdebcb13fca4b9cb36368a96f557d36f9dc969bebbe2568b283...
Traceback (most recent call last):
  File "/Users/fredriko/source/misc/main.py", line 3, in <module>
    load_dataset("c4", "en", data_dir="/data/huggingface/unpacked", beam_runner='DirectRunner')
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/load.py", line 612, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/builder.py", line 527, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/builder.py", line 1066, in _download_and_prepare
    pipeline=pipeline,
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/builder.py", line 582, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/Users/fredriko/.cache/huggingface/modules/datasets_modules/datasets/c4/8304cf264cc42bdebcb13fca4b9cb36368a96f557d36f9dc969bebbe2568b283/c4.py", line 190, in _split_generators
    file_paths = dl_manager.download_and_extract(files_to_download)
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 258, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 189, in download
    self._record_sizes_checksums(url_or_urls, downloaded_path_or_paths)
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 117, in _record_sizes_checksums
    self._recorded_sizes_checksums[str(url)] = get_size_checksum_dict(path)
  File "/Users/fredriko/venv/misc/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 80, in get_size_checksum_dict
    with open(path, "rb") as f:
IsADirectoryError: [Errno 21] Is a directory: '/'

Process finished with exit code 1
```
Databases
Author: JimmyJim1 | Pull request: no | State: closed

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
<html dir="ltr" lang="en" class="focus-outline-visible"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Author: Louiejay54 | Pull request: no | State: closed

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Added generated READMEs for datasets that were missing one.
Author: madlag | Pull request: yes | State: closed

This is it: we worked on a generator with Yacine @yjernite , and we generated dataset cards for all missing ones (161), with all the information we could gather from datasets repository, and using dummy_data to generate examples when possible.

Code is available here for the moment: https://github.com/madlag/datasets_readme_generator .
We will move it to a Hugging Face repository and to https://huggingface.co/datasets/card-creator/ later.
Improvements regarding caching and fingerprinting
Author: lhoestq | Pull request: yes | State: closed

This PR adds these features:
- Enable/disable caching
    If disabled, the library will no longer reload cached datasets files when applying transforms to the datasets.
    It is equivalent to setting `load_from_cache` to `False` in dataset transforms.
```python
from datasets import set_caching_enabled

set_caching_enabled(False)
```
- Allow unpicklable functions in `map`
    If an unpicklable function is used, then it's not possible to hash it to update the dataset fingerprint that is used to name cache files. To workaround that, a random fingerprint is generated instead and a warning is raised.
```python
logger.warning(
    f"Transform {transform} couldn't be hashed properly, a random hash was used instead. "
    "Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. "
    "If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything."
)
```

and also (open to discussion, EDIT: actually NOT included):
- Enable/disable fingerprinting
    Fingerprinting allows to have one deterministic fingerprint per dataset state.
    A dataset fingerprint is updated after each transform.
    Re-running the same transforms on a dataset in a different session results in the same fingerprint.
    Disabling the fingerprinting mechanism makes all the fingerprints random.
    Since the caching mechanism uses fingerprints to name the cache files, then cache file names will be different.
    Therefore disabling fingerprinting will prevent the caching mechanism from reloading datasets files that have already been computed.
    Disabling fingerprinting may speed up the lib for users that don't care about this feature and don't want to use caching.
```python
from datasets import set_fingerprinting_enabled

set_fingerprinting_enabled(False)
```

Other details:
- I renamed the `fingerprint` decorator to `fingerprint_transform` since the name was clearly not explicit. This decorator is used on dataset transform functions to allow them to update fingerprints.
- I added some `ignore_kwargs` when decorating transforms with `fingerprint_transform`, to make the fingerprint update not sensible to kwargs like `load_from_cache` or `cache_file_name`.

Todo: tests for set_fingerprinting_enabled + documentation for all the above features
Some datasets miss dataset_infos.json or dummy_data.zip
Author: madlag | Pull request: no | State: closed

While working on dataset REAME generation script at https://github.com/madlag/datasets_readme_generator , I noticed that some datasets miss a dataset_infos.json : 

```
c4
lm1b
reclor
wikihow
```

And some does not have a dummy_data.zip : 

```
kor_nli
math_dataset
mlqa
ms_marco
newsgroup
qa4mre
qangaroo
reddit_tifu
super_glue
trivia_qa
web_of_science
wmt14
wmt15
wmt16
wmt17
wmt18
wmt19
xtreme
```

But it seems that some of those last do have a "dummy" directory .
Update DBRD dataset card and download URL
Author: benjaminvdb | Pull request: yes | State: closed

I've added the Dutch Bood Review Dataset (DBRD) during the recent sprint. This pull request makes two minor changes:

1.  I'm changing the download URL from Google Drive to the dataset's GitHub release package. This is now possible because of PR #1316.
2. I've updated the dataset card.

Cheers! 😄
Add OSCAR
Author: lhoestq | Pull request: yes | State: closed

Continuation of #348 
The files have been moved to S3 and only the unshuffled version is available.
Both original and deduplicated versions of each language are available.

Example of usage:
```python
from datasets import load_dataset

oscar_dedup_en = load_dataset("oscar", "unshuffled_deduplicated_en", split="train")
oscar_orig_fr = load_dataset("oscar", "unshuffled_original_fr", split="train")
```

cc @pjox @jonatasgrosman 

-------------

To make the metadata generation work in parallel I did a few changes in the `datasets-cli test` command to add the `num_proc` and `proc_rank` arguments. This way you can run multiple processes for the metadata computation.

```
datasets-cli test ./datasets/oscar --save_infos --all_configs --num_proc 4 --proc_rank 0 --clear_cache --cache_dir tmp0
```

-------------

ToDo: add the dummy_data
Updated HuggingFace Datasets README (fix typos)
Author: 8bitmp3 | Pull request: yes | State: closed

Awesome work on 🤗 Datasets. I found a couple of small typos in the README. Hope this helps.



![](https://emojipedia-us.s3.dualstack.us-west-1.amazonaws.com/thumbs/160/google/56/hugging-face_1f917.png)
Fast start up
Author: lhoestq | Pull request: yes | State: closed

Currently if optional dependencies such as tensorflow, torch, apache_beam, faiss and elasticsearch are installed, then it takes a long time to do `import datasets` since it imports all of these heavy dependencies.

To make a fast start up for `datasets` I changed that so that they are not imported when `datasets` is being imported. On my side it changed the import time of `datasets` from 5sec to 0.5sec, which is enjoyable.

To be able to check if optional dependencies are available without importing them I'm using `importlib_metadata`, which is part of the standard lib in python>=3.8 and was backported. The difference with `importlib` is that it also enables to get the versions of the libraries without importing them.
I added this dependency in `setup.py`.
Fix ade_corpus_v2 config names
Author: lhoestq | Pull request: yes | State: closed

There are currently some typos in the config names of the `ade_corpus_v2` dataset, I fixed them:

- Ade_corpos_v2_classificaion -> Ade_corpus_v2_classification
- Ade_corpos_v2_drug_ade_relation -> Ade_corpus_v2_drug_ade_relation
- Ade_corpos_v2_drug_dosage_relation -> Ade_corpus_v2_drug_dosage_relation
Question: Shouldn't .info be a part of DatasetDict?
Author: KennethEnevoldsen | Pull request: no | State: open

Currently, only `Dataset` contains the .info or .features, but as many datasets contains standard splits (train, test) and thus the underlying information is the same (or at least should be) across the datasets. 

For instance:
```
>>> ds = datasets.load_dataset("conll2002", "es")
>>> ds.info
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'DatasetDict' object has no attribute 'info'
```

I could imagine that this wouldn't work for datasets dicts which hold entirely different datasets (multimodal datasets), but it seems odd that splits of the same dataset is treated the same as what is essentially different datasets. 

Intuitively it would also make sense that if a dataset is supplied via. the load_dataset that is have a common .info which covers the entire dataset.

It is entirely possible that I am missing another perspective
Dataset Error: DaNE contains empty samples at the end
Author: KennethEnevoldsen | Pull request: no | State: closed

The dataset DaNE, contains empty samples at the end. It is naturally easy to remove using a filter but should probably not be there, to begin with as it can cause errors.

```python
>>> import datasets
[...]
>>> dataset = datasets.load_dataset("dane")
[...]
>>> dataset["test"][-1]
{'dep_ids': [], 'dep_labels': [], 'lemmas': [], 'morph_tags': [], 'ner_tags': [], 'pos_tags': [], 'sent_id': '', 'text': '', 'tok_ids': [], 'tokens': []}
>>> dataset["train"][-1]
{'dep_ids': [], 'dep_labels': [], 'lemmas': [], 'morph_tags': [], 'ner_tags': [], 'pos_tags': [], 'sent_id': '', 'text': '', 'tok_ids': [], 'tokens': []}
```

Best,
Kenneth
Update README.md of covid-tweets-japanese
Author: forest1988 | Pull request: yes | State: closed

Update README.md of covid-tweets-japanese added by PR https://github.com/huggingface/datasets/pull/1367 and https://github.com/huggingface/datasets/pull/1402.

- Update "Data Splits" to be more precise that no information is provided for now.
  - old: [More Information Needed]
  - new: No information about data splits is provided for now.

- The automatic generation of links seemed not working properly, so I added a space before and after the URL to make the links work correctly.
Don't use xlrd for xlsx files
Author: lhoestq | Pull request: yes | State: closed

Since the latest release of `xlrd` (2.0), the support for xlsx files stopped.
Therefore we needed to use something else.
A good alternative is `openpyxl` which has also an integration with pandas si we can still call `pd.read_excel`.

I left the unused import of `openpyxl` in the dataset scripts to show users that this is a required dependency to use the scripts.

I tested the different datasets using `datasets-cli test` and the tests are successful (no missing examples).
Dataset "dane" missing
Author: KennethEnevoldsen | Pull request: no | State: closed

the `dane` dataset appear to be missing in the latest version (1.1.3).

```python
>>> import datasets
>>> datasets.__version__
'1.1.3'
>>> "dane" in datasets.list_datasets()
True
```

As we can see it should be present, but doesn't seem to be findable when using `load_dataset`.

```python
>>> datasets.load_dataset("dane")
Traceback (most recent call last):
  File "/home/kenneth/.Envs/EDP/lib/python3.8/site-packages/datasets/load.py", line 267, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/kenneth/.Envs/EDP/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 300, in cached_path
    output_path = get_from_cache(
  File "/home/kenneth/.Envs/EDP/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 486, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/dane/dane.py

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kenneth/.Envs/EDP/lib/python3.8/site-packages/datasets/load.py", line 278, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/kenneth/.Envs/EDP/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 300, in cached_path
    output_path = get_from_cache(
  File "/home/kenneth/.Envs/EDP/lib/python3.8/site-packages/datasets/utils/file_utils.py", line 486, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/dane/dane.py

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kenneth/.Envs/EDP/lib/python3.8/site-packages/datasets/load.py", line 588, in load_dataset
    module_path, hash = prepare_module(
  File "/home/kenneth/.Envs/EDP/lib/python3.8/site-packages/datasets/load.py", line 280, in prepare_module
    raise FileNotFoundError(
FileNotFoundError: Couldn't find file locally at dane/dane.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/dane/dane.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/dane/dane.py
```

This issue might be relevant to @ophelielacroix from the Alexandra Institut whom created the data.
added TurkishProductReviews dataset
Author: basakbuluz | Pull request: yes | State: closed

This PR added **Turkish Product Reviews Dataset contains 235.165 product reviews collected online. There are 220.284 positive, 14881 negative reviews**.

- **Repository:** [turkish-text-data](https://github.com/fthbrmnby/turkish-text-data)
- **Point of Contact:** Fatih Barmanbay - @fthbrmnby
Can't import cc100 dataset
Author: alighofrani95 | Pull request: no | State: closed

There is some issue to import cc100 dataset.

```
from datasets import load_dataset
dataset = load_dataset("cc100")
```

FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/cc100/cc100.py

During handling of the above exception, another exception occurred:

FileNotFoundError                         Traceback (most recent call last)
FileNotFoundError: Couldn't find file at https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/cc100/cc100.py

During handling of the above exception, another exception occurred:

FileNotFoundError                         Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, **download_kwargs)
    280                 raise FileNotFoundError(
    281                     "Couldn't find file locally at {}, or remotely at {} or {}".format(
--> 282                         combined_path, github_file_path, file_path
    283                     )
    284                 )

FileNotFoundError: Couldn't find file locally at cc100/cc100.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/cc100/cc100.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/cc100/cc100.py
Switchboard Dialog Act Corpus added under `datasets/swda`
Author: gmihaila | Pull request: yes | State: closed

Switchboard Dialog Act Corpus

Intro:
The Switchboard Dialog Act Corpus (SwDA) extends the Switchboard-1 Telephone Speech Corpus, Release 2,
with turn/utterance-level dialog-act tags. The tags summarize syntactic, semantic, and pragmatic information
about the associated turn. The SwDA project was undertaken at UC Boulder in the late 1990s.

Details:
[homepage](http://compprag.christopherpotts.net/swda.html)
[repo](https://github.com/NathanDuran/Switchboard-Corpus/raw/master/swda_data/)

I believe this is an important dataset to have since there is no dataset related to dialogue act added.

I didn't find any formatting for pull request. I hope all this information is enough.

For any support please contact me.
Switchboard Dialog Act Corpus added under `datasets/swda`
Author: gmihaila | Pull request: yes | State: closed

Pleased to announced that I added my first dataset **Switchboard Dialog Act Corpus**.


I think this is an important datasets to be added since it is the only one related to dialogue act classification. 

Hope the pull request is ok. Wasn't able to see any special formatting for the pull request form.


The Switchboard Dialog Act Corpus (SwDA) extends the Switchboard-1 Telephone Speech Corpus, Release 2,
with turn/utterance-level dialog-act tags. The tags summarize syntactic, semantic, and pragmatic information
about the associated turn. The SwDA project was undertaken at UC Boulder in the late 1990s.


[webpage](http://compprag.christopherpotts.net/swda.html)

[repo](https://github.com/NathanDuran/Switchboard-Corpus/raw/master/swda_data/)

Please contact me for any support!

All tests passed and followed all steps in the contribution guide!
new version of Ted Talks IWSLT (WIT3)
Author: skyprince999 | Pull request: yes | State: closed

In the previous iteration #1608 I had used language pairs. Which created 21,582 configs (109*108) !!! 

Now, TED talks in _each language_ is a separate config. So it's more cleaner with _just 109 configs_  (one for each language). Dummy files were created manually. 

Locally  I was able to clear the `python datasets-cli test datasets/......` . Which created the `dataset_info.json`  file . The test for the dummy files was also cleared. However couldn't figure out how to specify the local data folder for the real dataset


**Note: that this requires manual download of the dataset.** 
**Note2: The high number of _Files changed (112)_ is because of the large number of dummy files/configs!**
Add the 800GB Pile dataset?
Author: lewtun | Pull request: no | State: closed

## Adding a Dataset
- **Name:** The Pile
- **Description:** The Pile is a 825 GiB diverse, open source language modelling data set that consists of 22 smaller, high-quality datasets combined together. See [here](https://twitter.com/nabla_theta/status/1345130408170541056?s=20) for the Twitter announcement
- **Paper:** https://pile.eleuther.ai/paper.pdf
- **Data:** https://pile.eleuther.ai/
- **Motivation:** Enables hardcore  (GPT-3 scale!) language modelling

## Remarks
Given the extreme size of this dataset, I'm not sure how feasible this will be to include in `datasets` 🤯  . I'm also unsure how many `datasets` users are pretraining LMs, so the usage of this dataset may not warrant the effort to integrate it.