josteece/github-issues huggingface dataset:
Optimize ZIP format inference
Opened by albertvillanova (closed)

**Is your feature request related to a problem? Please describe.**
When hundreds of ZIP files are present in a dataset, format inference takes too long.

See: https://github.com/bigscience-workshop/data_tooling/issues/232#issuecomment-986685497

**Describe the solution you'd like**
Iterate over a maximum number of files.

CC: @lhoestq 


Comments:
Cannot import name 'maybe_sync'
Opened by KMFODA (closed)

## Describe the bug
Cannot seem to import datasets when running run_summarizer.py script on a VM set up on ovhcloud

## Steps to reproduce the bug
```python
from datasets import load_dataset
```

## Expected results
No error

## Actual results
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.7/site-packages/datasets/__init__.py", line 34, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 48, in <module>
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_writer.py", line 27, in <module>
    from .features import (
  File "/opt/conda/lib/python3.7/site-packages/datasets/features/__init__.py", line 2, in <module>
    from .audio import Audio
  File "/opt/conda/lib/python3.7/site-packages/datasets/features/audio.py", line 8, in <module>
    from ..utils.streaming_download_manager import xopen
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/streaming_download_manager.py", line 16, in <module>
    from ..filesystems import COMPRESSION_FILESYSTEMS
  File "/opt/conda/lib/python3.7/site-packages/datasets/filesystems/__init__.py", line 13, in <module>
    from .s3filesystem import S3FileSystem  # noqa: F401
  File "/opt/conda/lib/python3.7/site-packages/datasets/filesystems/s3filesystem.py", line 1, in <module>
    import s3fs
  File "/opt/conda/lib/python3.7/site-packages/s3fs/__init__.py", line 1, in <module>
    from .core import S3FileSystem, S3File
  File "/opt/conda/lib/python3.7/site-packages/s3fs/core.py", line 11, in <module>
    from fsspec.asyn import AsyncFileSystem, sync, sync_wrapper, maybe_sync
ImportError: cannot import name 'maybe_sync' from 'fsspec.asyn' (/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.0
- Platform: OVH Cloud Tesla V100 Machine
- Python version: 3.7.9
- PyArrow version: 6.0.1


Comments:
Hi ! Can you try updating `fsspec` ? The minimum version is `2021.05.0`

hey @lhoestq. I'm using `fsspec-2021.11.1` but still getting that error.

Maybe this discussion can help:

https://github.com/fsspec/filesystem_spec/issues/597#issuecomment-958646964

Thanks @lhoestq. Downgrading `fsspec and s3fs` to `2021.10` fixed this issue!
Add Wikimedia pre-processed datasets
Opened by albertvillanova (closed)

## Adding a Dataset
- **Name:** Add pre-processed data to:
  - *wikimedia/wikipedia*: https://huggingface.co/datasets/wikimedia/wikipedia
  - *wikimedia/wikisource*: https://huggingface.co/datasets/wikimedia/wikisource
- **Description:** Add pre-processed data to the Hub for all languages
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** This will be very useful for the NLP community, as the pre-processing has a high cost for lot of researchers (both in computation and in knowledge)

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

CC: @geohci, @yjernite

Comments:
As we are planning to stop using Apache Beam (our `datasets.BeamBasedBuilder`) for the generation of some datasets (including [Wikipedia](https://huggingface.co/datasets/wikipedia/blob/main/wikipedia.py)), I have been working on [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) to:
- Port the Wikipedia generation script to use `datasets.GeneratorBasedBuilder` instead and place it under the "script" branch: https://huggingface.co/datasets/wikimedia/wikipedia/tree/script
- Improve the efficiency of the code and make it highly parellizable. See:
  - [Parallelize dataset generation over multistreams](https://huggingface.co/datasets/wikimedia/wikipedia/commit/610c55864586dbdad7ac5a13c21a367bb000a1d3)
  - [Parallelize data downloading](https://huggingface.co/datasets/wikimedia/wikipedia/commit/b35d406bd9e81f08c68e7bf95d130d2f506dfe77)

  With these improvements, I can generate the English Wikipedia in 5h using 16 processors in a machine without needing a huge amount of RAM (the machine had 32 GB, but I think less can be used as well):
  ```python
  ds = load_dataset("wikimedia/wikipedia", revision="script", date="20231101", language="en", host="https://mirror.accum.se/mirror/wikimedia.org/dumps", split="train", num_proc=16)
  ```
- Pre-process all Wikipedia languages for the latest 2023-11-01 dump and make them available to the entire community for easy use:
  ```python
  ds = load_dataset("wikimedia/wikipedia", "20231101.en", split="train", num_proc=16)
  ```
CC: @geohci
Improve Wikipedia loading script
Opened by albertvillanova (closed)

As reported by @geohci, the "wikipedia" processing/loading script could be improved by some additional small suggested processing functions:
- _extract_content(filepath):
  - Replace .startswith("#redirect") with more structured approach: if elem.find(f"./{namespace}redirect") is None: continue
- _parse_and_clean_wikicode(raw_content, parser):
  - Remove rm_template from cleaning -- this is redundant with .strip_code() from mwparserformhell
  - Build a language-specific list of namespace prefixes to filter out per below get_namespace_prefixes
  - Optional: strip prefixes like categories -- e.g., Category:Towns in Tianjin becomes Towns in Tianjin
  - Optional: strip magic words





Comments:
Thanks! See https://public.paws.wmcloud.org/User:Isaac_(WMF)/HuggingFace%20Wikipedia%20Processing.ipynb for more implementation details / some data around the overhead induced by adding the extra preprocessing steps (stripping link prefixes and magic words)

Closed by:
- #3435
Add Wikisource dataset
Opened by albertvillanova (closed)

## Adding a Dataset
- **Name:** *wikisource*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** Additional high quality textual data, besides Wikipedia.

Add loading script as "canonical" dataset (as it is the case for ""wikipedia").

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

CC: @geohci, @yjernite 

Comments:
See notebook by @geohci: https://public.paws.wmcloud.org/User:Isaac_(WMF)/HuggingFace%20Wikisource%20Processing.ipynb

See: https://huggingface.co/datasets/wikimedia/wikisource
Install Audio dependencies to support audio decoding
Opened by albertvillanova (closed)

## Dataset viewer issue for '*openslr*', '*projecte-aina/parlament_parla*'

**Link:** *https://huggingface.co/datasets/openslr*
**Link:** *https://huggingface.co/datasets/projecte-aina/parlament_parla*

Error:
```
Status code:   400
Exception:     ImportError
Message:       To support decoding audio files, please install 'librosa'.
```

Am I the one who added this dataset ? Yes-No
- openslr: No
- projecte-aina/parlament_parla: Yes


Comments:
https://huggingface.co/datasets/projecte-aina/parlament_parla -> works (but we still have to show an audio player)

https://huggingface.co/datasets/openslr -> another issue: `Message:       [Errno 2] No such file or directory: '/home/hf/datasets-preview-backend/zip:/asr_javanese/data/00/00004fe6aa.flac'`

Done

https://huggingface.co/datasets/projecte-aina/parlament_parla/viewer/clean/train works

<img width="1535" alt="Capture d’écran 2022-04-12 à 13 58 47" src="https://user-images.githubusercontent.com/1676121/162957855-cb3d9e2e-4b61-488c-99ca-8065cd8fe377.png">


But https://huggingface.co/datasets/openslr/viewer does not work

<img width="678" alt="Capture d’écran 2022-04-12 à 13 59 46" src="https://user-images.githubusercontent.com/1676121/162958013-e31ef2ae-f886-47b7-9f27-664ed3d4b5a1.png">

Same issue as #4126:

```
Status code:   400
Exception:     TypeError
Message:       __init__() got an unexpected keyword argument 'audio_column'
```

Fixed:
<img width="1561" alt="Capture d’écran 2022-04-25 à 18 11 51" src="https://user-images.githubusercontent.com/1676121/165129813-018ece9e-8b20-4544-844d-4e88148e738f.png">
Preserve all feature types when saving a dataset on the Hub with `push_to_hub`
Opened by mariosasko (closed)

Currently, if one of the dataset features is of type `ClassLabel`, saving the dataset with `push_to_hub` and reloading the dataset with `load_dataset` will return the feature of type `Value`. To fix this, we should do something similar to `save_to_disk` (which correctly preserves the types) and not only push the parquet files in `push_to_hub`, but also the dataset `info` (stored in a JSON file).

Comments:
According to this [comment in the forum](https://discuss.huggingface.co/t/save-datasetdict-to-huggingface-hub/12075/8?u=lhoestq), using `push_to_hub` on a dataset with `ClassLabel` can also make the feature simply disappear when it's reloaded !

Maybe we can also fix https://github.com/huggingface/datasets/issues/3035 while working on this because, as pointed out in my initial post, `save_to_disk` also saves the `dataset_info.json`  file.
Common Voice Belarusian Dataset
Opened by wiedymi (open)

## Adding a Dataset
- **Name:** *Common Voice Belarusian Dataset*
- **Description:** *[commonvoice.mozilla.org/be](https://commonvoice.mozilla.org/be)*
- **Data:** *[commonvoice.mozilla.org/be/datasets](https://commonvoice.mozilla.org/be/datasets)*
- **Motivation:** *It has more than 7GB of data, so it will be great to have it in this package so anyone can try to train something for Belarusian language.*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
Dataset viewer issue for `dansbecker/hackernews_hiring_posts`
Opened by severo (closed)

## Dataset viewer issue for `dansbecker/hackernews_hiring_posts`

**Link:** https://huggingface.co/datasets/dansbecker/hackernews_hiring_posts

*short description of the issue*

Dataset preview not showing for uploaded DatasetDict. See https://discuss.huggingface.co/t/dataset-preview-not-showing-for-uploaded-datasetdict/12603

Am I the one who added this dataset ?

No -> @dansbecker

Comments:
This issue was fixed by me calling `all_datasets.push_to_hub("hackernews_hiring_posts")`.

The previous problems were from calling `all_datasets.save_to_disk` and then pushing with `my_repo.git_add` and `my_repo.push_to_hub`.
method to select columns
Opened by changjonathanc (closed)

**Is your feature request related to a problem? Please describe.**

* There is currently no way to select some columns of a dataset. In pandas, one can use `df[['col1', 'col2']]` to select columns, but in `datasets`, it results in error.

**Describe the solution you'd like**

* A new method that can be used to create a new dataset with only a list of specified columns.

**Describe alternatives you've considered**

`.remove_columns(self, columns: Union[str, List[str]], inverse: bool = False)`
Or 
`.select(self,  indices: Iterable = None, columns: List[str] = None)`





Comments:
duplicate of #2655
Loading dataset throws "KeyError: 'Field "builder_name" does not exist in table schema'"
Opened by R4ZZ3 (closed)

## Describe the bug
I have prepared dataset to datasets and now I am trying to load it back Finnish-NLP/voxpopuli_fi
I get "KeyError: 'Field "builder_name" does not exist in table schema'"

My dataset folder and files should be like @patrickvonplaten  has here https://huggingface.co/datasets/flax-community/german-common-voice-processed

How my voxpopuli dataset looks like:
![image](https://user-images.githubusercontent.com/25264037/144895598-b7d9ae91-b04a-4046-9f06-b71ff0824d13.png)

Part of the processing (path column is the absolute path to audio files)
```
def add_audio_column(example):
    example['audio'] = example['path']
    return example

voxpopuli = voxpopuli.map(add_audio_column)
voxpopuli.cast_column("audio", Audio())
voxpopuli["audio"] <-- to my knowledge this does load the local files and prepares those arrays
voxpopuli = voxpopuli.cast_column("audio", Audio(sampling_rate=16_000)) resampling 16kHz
```

I have then saved it to disk_
`voxpopuli.save_to_disk('/asr_disk/datasets_processed_new/voxpopuli')`

and made folder structure same as @patrickvonplaten 
I also get same error while trying to load_dataset from his repo:
![image](https://user-images.githubusercontent.com/25264037/144895872-e9b8f326-cf2b-46cf-9417-606a0ce14077.png)


## Steps to reproduce the bug
```python
dataset = load_dataset("Finnish-NLP/voxpopuli_fi")
```

## Expected results
Dataset is loaded correctly and looks like in the first picture

## Actual results
Loading throws keyError:
KeyError: 'Field "builder_name" does not exist in table schema'


Resources I have been trying to follow:
https://huggingface.co/docs/datasets/audio_process.html 
https://huggingface.co/docs/datasets/share_dataset.html

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.2.dev0
- Platform: Ubuntu 20.04.2 LTS
- Python version: 3.8.12
- PyArrow version: 6.0.1


Comments:
Got solved it with push_to_hub, closing
Add EDGAR
Opened by philschmid (open)

## Adding a Dataset
- **Name:** EDGAR Database
- **Description:** https://www.sec.gov/edgar/about EDGAR, the Electronic Data Gathering, Analysis, and Retrieval system, is the primary system for companies and others submitting documents under the Securities Act of 1933, the Securities Exchange Act of 1934, the Trust Indenture Act of 1939, and the Investment Company Act of 1940. Containing millions of company and individual filings, EDGAR benefits investors, corporations, and the U.S. economy overall by increasing the efficiency, transparency, and fairness of the securities markets. The system processes about 3,000 filings per day, serves up 3,000 terabytes of data to the public annually, and accommodates 40,000 new filers per year on average. EDGAR® and EDGARLink® are registered trademarks of the SEC.
- **Data:** https://www.sec.gov/os/accessing-edgar-data
- **Motivation:** Enabling and improving FSI (Financial Services Industry) datasets to increase ease of use 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
cc @juliensimon 

Datasets are not tracked in this repository anymore. But you can make your own dataset in the huggingface hub
Fix flaky test of the temporary directory used by load_from_disk
Opened by lhoestq (closed)

The test is flaky, here is an example of random CI failure:
https://github.com/huggingface/datasets/commit/73ed6615b4b3eb74d5311684f7b9e05cdb76c989

I fixed that by not checking the content of the random part of the temporary directory name

Comments:
CI failed because of a server error - merging
Create Language Modeling task
Opened by albertvillanova (closed)

Create Language Modeling task to be able to specify the input "text" column in a dataset.

This can be useful for datasets which are not exclusively used for language modeling and have more than one column:
- for text classification datasets (with columns "review" and "rating", for example), the Language Modeling task can be used to specify the "text" column ("review" in this case).

TODO:
- [ ] Add the LanguageModeling task to all dataset scripts which can be used for language modeling

Comments:
#3337 Add typing overloads to Dataset.__getitem__ for mypy
Opened by Dref360 (closed)

Add typing overloads to Dataset.__getitem__ for mypy

Fixes #3337 

**Iterable**
Iterable from `collections` cannot have a type, so you can't do `Iterable[int]` for example. `typing` has a Generic version that builds upon the one from `collections`.

**Flake8**
I had to add `# noqa: F811`, this is a bug from Flake8.

datasets uses flake8==3.7.9 which released in October 2019 if I update flake8 (4.0.1), I no longer get these errors, but I did not want to make the update without your approval. (It also triggers other errors like no args in f-strings.)

Comments:
Locally the `make quality` passes with the same dependencies. I would suggest upgrading flake8. (I can take care of it in another PR)
cc @lhoestq 

Thank you for fixing flake8! I think we are ready to merge then.
[Quick poll] Give your opinion on the future of the Hugging Face Open Source ecosystem!
Opened by LysandreJik (closed)

Thanks to all of you, `datasets` will pass 11.5k stars :star2: this week!

If you have a couple of minutes and want to participate in shaping the future of the ecosystem, please share your thoughts: 

[**hf.co/oss-survey**](https://hf.co/oss-survey)

(please reply in the above feedback form rather than to this thread)

Thank you all on behalf of the HuggingFace team! 🤗

Comments:
COCO 🥥 on the 🤗 Hub?
Opened by merveenoyan (closed)

This is a draft PR since I ran into few small problems. 
I referred to this TFDS code: https://github.com/tensorflow/datasets/blob/2538a08c184d53b37bfcf52cc21dd382572a88f4/tensorflow_datasets/object_detection/coco.py
cc: @mariosasko 

Comments:
@mariosasko I fixed couple of bugs

TO-DO: 
- [x] Add unlabeled 2017 splits, train and validation splits of 2015
- [x] Add Class Labels as list instead

@mariosasko added fine & coarse grained labels, will fix the bugs (currently getting set up with VM, my internet is too slow to run the tests and download the data 🥲)

migrated to here https://github.com/huggingface/datasets/tree/coco
Support streaming zipped dataset repo by passing only repo name
Opened by albertvillanova (closed)

Proposed solution:
- I have added the method `iter_files` to DownloadManager and StreamingDownloadManager
- I use this in modules: "csv", "json", "text"
- I test for CSV/JSONL/TXT zipped (and non-zipped) files, both in streaming and non-streaming modes

Fix #3373.

Comments:
I just tested and I think this only opens one file ? If there are several files in the ZIP, only the first one is opened. To open several files from a ZIP, one has to call `open` several times.

What about updating the CSV loader to make it `download_and_extract` zip files, and open each extracted file ?

I have implemented the  glob of ZIP files in the packaged modules:
- csv
- json
- text

Also for streaming and non-streaming.

In https://github.com/huggingface/datasets/pull/3375/commits/c10275fe36085601cb7bdb9daee9a8f1fc734f48, there were 3 failing tests, only on Linux:
```
=========================== short test summary info ============================
FAILED tests/test_streaming_download_manager.py::test_streaming_dl_manager_get_extraction_protocol[https://drive.google.com/uc?export=download&id=1k92sUfpHxKq8PXWRr7Y5aNHXwOCNUmqh-zip]
FAILED tests/test_streaming_download_manager.py::test_streaming_gg_drive - Fi...
FAILED tests/test_streaming_download_manager.py::test_streaming_gg_drive_zipped
= 3 failed, 3553 passed, 2950 skipped, 2 xfailed, 1 xpassed, 125 warnings in 192.79s (0:03:12) =
```

After re-running the CI in https://github.com/huggingface/datasets/pull/3375/commits/57bfe1f342cd3c59d2510b992d5f06a0761eb147, there was only 1 failing test:
- On Linux:
```
=========================== short test summary info ============================
FAILED tests/test_streaming_download_manager.py::test_streaming_gg_drive_zipped
= 1 failed, 3555 passed, 2950 skipped, 2 xfailed, 1 xpassed, 125 warnings in 199.76s (0:03:19) =
```
- On Windows:
```
=========================== short test summary info ===========================
FAILED tests/test_load.py::test_load_dataset_builder_for_community_dataset_without_script
= 1 failed, 3551 passed, 2954 skipped, 2 xfailed, 1 xpassed, 121 warnings in 478.58s (0:07:58) =
```

The test `tests/test_streaming_download_manager.py::test_streaming_gg_drive_zipped` passes locally.

I guess the issue is caused by those tests and has nothing to do with this PR.

@lhoestq my final proposed solution:
- I have added the method `iter_files` to DownloadManager and StreamingDownloadManager
- I use this in modules: "csv", "json", "text"
- I test for CSV/JSONL/TXT zipped (and non-zipped) files, both in streaming and non-streaming modes

> Note that at one point we might consider switching to using `iter_archive` for ZIP files in the json/text/csv loaders since it should be faster.

As far as the functionality is kept... ;)
NonMatchingChecksumError for the CLUE:cluewsc2020, chid, c3 and tnews
Opened by Namco0816 (closed)

Hi, it seems like there are updates in cluewsc2020, chid, c3 and tnews, since i could not load them due to the checksum error.

Comments:
Seems like the issue still exists,:
`Downloading and preparing dataset clue/chid (download: 127.15 MiB, generated: 259.71 MiB, post-processed: Unknown size, total: 386.86 MiB) to /mnt/cache/tanhaochen/.cache/huggingface/datasets/clue/chid/1.0.0/e55b490cb7809dcd8db31b9a87119f2e2ec87cdc060da8a9ac070b070ca3e379...
Traceback (most recent call last):
  File "/mnt/cache/tanhaochen/PromptCLUE/test_datasets.py", line 3, in <module>
    cluewsc2020 = datasets.load_dataset("clue","chid")
  File "/mnt/cache/tanhaochen/dependencies/datasets/src/datasets/load.py", line 1667, in load_dataset
    builder_instance.download_and_prepare(
  File "/mnt/cache/tanhaochen/dependencies/datasets/src/datasets/builder.py", line 593, in download_and_prepare
    self._download_and_prepare(
  File "/mnt/cache/tanhaochen/dependencies/datasets/src/datasets/builder.py", line 663, in _download_and_prepare
    verify_checksums(
  File "/mnt/cache/tanhaochen/dependencies/datasets/src/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://storage.googleapis.com/cluebenchmark/tasks/chid_public.zip']
`

Hi,

the fix hasn't been merged yet (it should be merged early next week).
Support streaming zipped CSV dataset repo by passing only repo name
Opened by albertvillanova (closed)

Given a community 🤗 dataset repository containing only a zipped CSV file (only raw data, no loading script), I would like to load it in streaming mode without passing `data_files`:
```
ds_name = "bigscience-catalogue-data/vietnamese_poetry_from_fsoft_ai_lab"
ds = load_dataset(ds_name, split="train", streaming=True, use_auth_token=True)
item = next(iter(ds))
```

Currently, it gives a `FileNotFoundError` because there is no glob (no "\*" after "zip://": "zip://*") in the passed URL:
```
'zip://::https://huggingface.co/datasets/bigscience-catalogue-data/vietnamese_poetry_from_fsoft_ai_lab/resolve/e5d45f1bd9a8a798cc14f0a45ebc1ce91907c792/poems_dataset.zip'
```


Comments: