SurAyush/github_issues huggingface dataset:
Issue: Error about load_metric
State: closed
Author: jiacheng-ye

## Describe the bug
  File "/opt/conda/lib/python3.8/site-packages/datasets/load.py", line 1371, in load_metric
    metric = metric_cls(
TypeError: 'NoneType' object is not callable

## Steps to reproduce the bug
```python
metric = load_metric("glue", "sst2")
```


## Environment info
- `datasets` version: 1.16.1
- Platform: Linux-4.15.0-161-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyArrow version: 6.0.1


Comments:
Hi ! I wasn't able to reproduce your error.

Can you try to clear your cache at `~/.cache/huggingface/modules` and try again ?
Issue: Adding mMARCO dataset
State: closed
Author: lhbonifacio

Adding mMARCO (v1.1) to HF datasets.

Comments:
Hi @albertvillanova we've made a major overhaul of the loading script including all configurations we're making available. Could you please review it again?

@albertvillanova :ping_pong: 

Thanks @lhbonifacio for adding this dataset.
Hi there, i got an error about mmarco:
ConnectionError: Couldn't reach 'unicamp-dl/mmarco' on the Hub (ConnectionError)
code:
`from datasets import list_datasets, load_dataset
dataset = load_dataset('unicamp-dl/mmarco', language='portuguese')`

Any help will be appreciated!

Hi @catqaq, we updated the loading script. Now you can load the datasets with:

```python
dataset = load_dataset('unicamp-dl/mmarco', 'portuguese')
```

You can check the list of supported languages and usage examples in [this link](https://huggingface.co/datasets/unicamp-dl/mmarco). Feel free to contact us if you have any issues.




> 



> Hi @catqaq, we updated the loading script. Now you can load the datasets with:
> 
> ```python
> dataset = load_dataset('unicamp-dl/mmarco', 'portuguese')
> ```
> 
> You can check the list of supported languages and usage examples in [this link](https://huggingface.co/datasets/unicamp-dl/mmarco). Feel free to contact us if you have any issues.

Thanks for your quick updates. So, how can i get the fixed version, install from the source? It seems that the merging is blocked.

@catqaq you can load mMARCO using the namespace `unicamp-dl/mmarco` while this PR remains under review.

Thanks for your contribution, @lhbonifacio and  @hugoabonizio. And sorry for the late response.

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

As you already created this dataset under your organization namespace (https://huggingface.co/datasets/unicamp-dl/mmarco), I think we can safely close this PR.

We would suggest you complete your dataset card with the YAML tags, to make it searchable and discoverable.

Please, feel free to tell us if you need some help.
Issue: Add eli5_category dataset
State: closed
Author: jingshenSN2

This pull request adds a categorized Long-form question answering dataset `ELI5_Category`. It's a new variant of the [ELI5](https://huggingface.co/datasets/eli5) dataset that uses the Reddit tags to alleviate the training/validation overlapping in the origin ELI5 dataset.

A [report](https://celeritasml.netlify.app/posts/2021-12-01-eli5c/)(Section 2) on this dataset. 

Comments:
> Thanks a lot for adding this dataset ! Good job with the dataset card and the dataset scripts - they're really good :)
> 
> I just added minor changes

Thanks for fixing typos!
Issue: Fix type of bridge field in QED
State: closed
Author: mariosasko

Use `Value("string")` instead of `Value("bool")` for the feature type of the `"bridge"` field in the QED dataset. If the value is `False`, set to `None`.

The following paragraph in the QED repo explains the purpose of this field:
>Each annotation in referential_equalities is a pair of spans, the question_reference and the sentence_reference, corresponding to an entity mention in the question and the selected_sentence respectively. As described in the paper, sentence_references can be "bridged in", in which case they do not correspond with any actual span in the selected_sentence. Hence, sentence_reference spans contain an additional field, bridge, which is a prepositional phrase when a reference is bridged, and is False otherwise. Prepositional phrases serve to link bridged references to an anchoring phrase in the selected_sentence. In the case a sentence_reference is bridged, the start and end, as well as the span string, map to such an anchoring phrase in the selected_sentence.

Fix #3346

cc @VictorSanh 

Comments:
Issue: disaster_response_messages unavailable
State: closed
Author: sacdallago

## Dataset viewer issue for '* disaster_response_messages*'

**Link:** https://huggingface.co/datasets/disaster_response_messages

Dataset unavailable. Link dead: https://datasets.appen.com/appen_datasets/disaster_response_data/disaster_response_messages_training.csv

Am I the one who added this dataset ?No


Comments:
Hi, thanks for reporting! This is a duplicate of https://github.com/huggingface/datasets/issues/3240. We are working on a fix.
Issue: Non-deterministic tests: CI tests randomly fail
State: closed
Author: albertvillanova

## Describe the bug
Some CI tests fail randomly.

1. In https://github.com/huggingface/datasets/pull/3375/commits/c10275fe36085601cb7bdb9daee9a8f1fc734f48, there were 3 failing tests, only on Linux:
   ```
   =========================== short test summary info ============================
   FAILED tests/test_streaming_download_manager.py::test_streaming_dl_manager_get_extraction_protocol[https://drive.google.com/uc?export=download&id=1k92sUfpHxKq8PXWRr7Y5aNHXwOCNUmqh-zip]
   FAILED tests/test_streaming_download_manager.py::test_streaming_gg_drive - Fi...
   FAILED tests/test_streaming_download_manager.py::test_streaming_gg_drive_zipped
   = 3 failed, 3553 passed, 2950 skipped, 2 xfailed, 1 xpassed, 125 warnings in 192.79s (0:03:12) =
   ```

2. After re-running the CI (without any change in the code) in https://github.com/huggingface/datasets/pull/3375/commits/57bfe1f342cd3c59d2510b992d5f06a0761eb147, there was only 1 failing test (one on Linux and a different one on Windows):
   - On Linux:
      ```
      =========================== short test summary info ============================
      FAILED tests/test_streaming_download_manager.py::test_streaming_gg_drive_zipped
      = 1 failed, 3555 passed, 2950 skipped, 2 xfailed, 1 xpassed, 125 warnings in 199.76s (0:03:19) =
      ```
   - On Windows:
      ```
      =========================== short test summary info ===========================
      FAILED tests/test_load.py::test_load_dataset_builder_for_community_dataset_without_script
      = 1 failed, 3551 passed, 2954 skipped, 2 xfailed, 1 xpassed, 121 warnings in 478.58s (0:07:58) =
      ```

   The test `tests/test_streaming_download_manager.py::test_streaming_gg_drive_zipped` passes locally.

3. After re-running again the CI (without any change in the code) in https://github.com/huggingface/datasets/pull/3375/commits/39f32f2119cf91b86867216bb5c356c586503c6a, ALL the tests passed.



Comments:
I think it might come from two different issues:
1. Google Drive is an unreliable host, mainly because of quota limitations
2. the staging environment can sometimes raise some errors

For Google Drive tests we could set up some retries with backup URLs if necessary I guess.
For staging on the other hand, I guess we can investigate what causes this and discuss with the back-end team

Closed by:
- #3982
Issue: Fix flaky test again for s3 serialization
State: closed
Author: lhoestq

Following https://github.com/huggingface/datasets/pull/3388 that wasn't enough (see CI error [here](https://app.circleci.com/pipelines/github/huggingface/datasets/9080/workflows/b971fb27-ff20-4220-9416-c19acdfdf6f4/jobs/55985))

Comments:
Issue: [chinese wwm] load_datasets behavior not as expected when using run_mlm_wwm.py script
State: open
Author: hyusterr

## Describe the bug

Model I am using (Bert, XLNet ...): bert-base-chinese

The problem arises when using:
* [https://github.com/huggingface/transformers/blob/master/examples/research_projects/mlm_wwm/run_mlm_wwm.py] the official example scripts: `rum_mlm_wwm.py`

The tasks I am working on is: pretraining whole word masking with my own dataset and ref.json file
I tried follow the run_mlm_wwm.py procedure to do whole word masking on pretraining task. my file is in .txt form, where one line represents one sample, with `9,264,784` chinese lines in total. the ref.json file is also contains 9,264,784 lines of whole word masking reference data for my chinese corpus. but when I try to adapt the run_mlm_wwm.py script, it shows that somehow after 
`datasets["train"] = load_dataset(...`
`len(datasets["train"])` returns `9,265,365`
then, after `tokenized_datasets = datasets.map(...`
`len(tokenized_datasets["train"])` returns `9,265,279`
I'm really confused and tried to trace code by myself but can't know what happened after a week trial.

I want to know what happened in the `load_dataset()` function and `datasets.map` here and how did I get more lines of data than I input. so I'm here to ask.
## To reproduce
Sorry that I can't provide my data here since it did not belong to me. but I'm sure I remove the blank lines.

## Expected behavior
I expect the code run as it should. but the AssertionError in line 167 keeps raise as the line of reference json and datasets['train'] differs.

Thanks for your patient reading!

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.29
- Python version: 3.8.10 
- PyArrow version: 3.0.0


Comments:
@LysandreJik not so sure who to @
Could you help?

Hi @hyusterr, I believe it is @wlhgtc from https://github.com/huggingface/transformers/pull/9887
Issue: Pass new_fingerprint in multiprocessing
State: closed
Author: lhoestq

Following https://github.com/huggingface/datasets/pull/3045

Currently one can pass `new_fingerprint` to `.map()` to use a custom fingerprint instead of the one computed by hashing the map transform. However it's ignored if `num_proc>1`.

In this PR I fixed that by passing `new_fingerprint` to `._map_single()` when `num_proc>1`.
More specifically, `new_fingerprint` with a suffix based on the process `rank` is passed, so that each process has a different `new_fingerprint`

cc @TevenLeScao @vlievin

Comments:
@lhoestq Hi~, does this support that `datasets.map(func, batched=True, batch_size, num_proc>1, new_fingerprint="func_v1")` even if `func` can't pickle. I also notice that you said "Unfortunately you need picklable mapping functions to make multiprocessing work :confused: Also feel free to open an issue or send me a dm if you are in a situation where the caching fails. I can help you with that :slight_smile:" in [here](https://discuss.huggingface.co/t/how-to-deal-with-unpickable-objects-in-map/1547/8). So, I want to ask that is there a way for users to use multiprocessing in `datasets.map` when the `func` can't pickle? 
Thanks in advance!

Yea you need your function to be picklable for multiprocessing, otherwise the main process is not able to pass your function to the child processes.
Issue: Typo in Dataset viewer error message
State: closed
Author: lewtun

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

When creating an empty dataset repo, the Dataset Preview provides a helpful message that no files were found. There is a tiny typo in that message: "ressource" should be "resource"

![Screen Shot 2021-12-09 at 15 31 31](https://user-images.githubusercontent.com/26859204/145415725-9cd728f0-c2c8-4b4e-a8e1-4f4d7841c94a.png)


Am I the one who added this dataset ? 

N/A


Comments:
Fixed, thanks
<img width="661" alt="Capture d’écran 2021-12-22 à 12 02 30" src="https://user-images.githubusercontent.com/1676121/147082881-cf700e8d-0511-4431-b214-d6cf8137db10.png">
Issue: Use max number of data files to infer module
State: closed
Author: albertvillanova

When inferring the module for datasets without script, set a maximum number of iterations over data files.

This PR fixes the issue of taking too long when hundred of data files present.

Please, feel free to agree on both numbers:
```
# Datasets without script
DATA_FILES_MAX_NUMBER = 10
ARCHIVED_DATA_FILES_MAX_NUMBER = 5
```

Fix #3404.

Comments:
Cool thanks :) Feel free to merge if it's all good for you
Issue: ZIP format inference does not work when files located in a dir inside the archive
State: closed
Author: albertvillanova

## Describe the bug
When a zipped file contains archived files within a directory, the function `infer_module_for_data_files_in_archives` does not work.

It only works for files located in the root directory of the ZIP file.

## Steps to reproduce the bug
```python
infer_module_for_data_files_in_archives(["path/to/zip/file.zip"], False)
```

Comments:
Issue: Optimize ZIP format inference
State: closed
Author: albertvillanova

**Is your feature request related to a problem? Please describe.**
When hundreds of ZIP files are present in a dataset, format inference takes too long.

See: https://github.com/bigscience-workshop/data_tooling/issues/232#issuecomment-986685497

**Describe the solution you'd like**
Iterate over a maximum number of files.

CC: @lhoestq 


Comments:
Issue: Cannot import name 'maybe_sync'
State: closed
Author: KMFODA

## Describe the bug
Cannot seem to import datasets when running run_summarizer.py script on a VM set up on ovhcloud

## Steps to reproduce the bug
```python
from datasets import load_dataset
```

## Expected results
No error

## Actual results
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.7/site-packages/datasets/__init__.py", line 34, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 48, in <module>
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_writer.py", line 27, in <module>
    from .features import (
  File "/opt/conda/lib/python3.7/site-packages/datasets/features/__init__.py", line 2, in <module>
    from .audio import Audio
  File "/opt/conda/lib/python3.7/site-packages/datasets/features/audio.py", line 8, in <module>
    from ..utils.streaming_download_manager import xopen
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/streaming_download_manager.py", line 16, in <module>
    from ..filesystems import COMPRESSION_FILESYSTEMS
  File "/opt/conda/lib/python3.7/site-packages/datasets/filesystems/__init__.py", line 13, in <module>
    from .s3filesystem import S3FileSystem  # noqa: F401
  File "/opt/conda/lib/python3.7/site-packages/datasets/filesystems/s3filesystem.py", line 1, in <module>
    import s3fs
  File "/opt/conda/lib/python3.7/site-packages/s3fs/__init__.py", line 1, in <module>
    from .core import S3FileSystem, S3File
  File "/opt/conda/lib/python3.7/site-packages/s3fs/core.py", line 11, in <module>
    from fsspec.asyn import AsyncFileSystem, sync, sync_wrapper, maybe_sync
ImportError: cannot import name 'maybe_sync' from 'fsspec.asyn' (/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.0
- Platform: OVH Cloud Tesla V100 Machine
- Python version: 3.7.9
- PyArrow version: 6.0.1


Comments:
Hi ! Can you try updating `fsspec` ? The minimum version is `2021.05.0`

hey @lhoestq. I'm using `fsspec-2021.11.1` but still getting that error.

Maybe this discussion can help:

https://github.com/fsspec/filesystem_spec/issues/597#issuecomment-958646964

Thanks @lhoestq. Downgrading `fsspec and s3fs` to `2021.10` fixed this issue!
Issue: Add Wikimedia pre-processed datasets
State: closed
Author: albertvillanova

## Adding a Dataset
- **Name:** Add pre-processed data to:
  - *wikimedia/wikipedia*: https://huggingface.co/datasets/wikimedia/wikipedia
  - *wikimedia/wikisource*: https://huggingface.co/datasets/wikimedia/wikisource
- **Description:** Add pre-processed data to the Hub for all languages
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** This will be very useful for the NLP community, as the pre-processing has a high cost for lot of researchers (both in computation and in knowledge)

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

CC: @geohci, @yjernite

Comments:
As we are planning to stop using Apache Beam (our `datasets.BeamBasedBuilder`) for the generation of some datasets (including [Wikipedia](https://huggingface.co/datasets/wikipedia/blob/main/wikipedia.py)), I have been working on [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) to:
- Port the Wikipedia generation script to use `datasets.GeneratorBasedBuilder` instead and place it under the "script" branch: https://huggingface.co/datasets/wikimedia/wikipedia/tree/script
- Improve the efficiency of the code and make it highly parellizable. See:
  - [Parallelize dataset generation over multistreams](https://huggingface.co/datasets/wikimedia/wikipedia/commit/610c55864586dbdad7ac5a13c21a367bb000a1d3)
  - [Parallelize data downloading](https://huggingface.co/datasets/wikimedia/wikipedia/commit/b35d406bd9e81f08c68e7bf95d130d2f506dfe77)

  With these improvements, I can generate the English Wikipedia in 5h using 16 processors in a machine without needing a huge amount of RAM (the machine had 32 GB, but I think less can be used as well):
  ```python
  ds = load_dataset("wikimedia/wikipedia", revision="script", date="20231101", language="en", host="https://mirror.accum.se/mirror/wikimedia.org/dumps", split="train", num_proc=16)
  ```
- Pre-process all Wikipedia languages for the latest 2023-11-01 dump and make them available to the entire community for easy use:
  ```python
  ds = load_dataset("wikimedia/wikipedia", "20231101.en", split="train", num_proc=16)
  ```
CC: @geohci
Issue: Improve Wikipedia loading script
State: closed
Author: albertvillanova

As reported by @geohci, the "wikipedia" processing/loading script could be improved by some additional small suggested processing functions:
- _extract_content(filepath):
  - Replace .startswith("#redirect") with more structured approach: if elem.find(f"./{namespace}redirect") is None: continue
- _parse_and_clean_wikicode(raw_content, parser):
  - Remove rm_template from cleaning -- this is redundant with .strip_code() from mwparserformhell
  - Build a language-specific list of namespace prefixes to filter out per below get_namespace_prefixes
  - Optional: strip prefixes like categories -- e.g., Category:Towns in Tianjin becomes Towns in Tianjin
  - Optional: strip magic words





Comments:
Thanks! See https://public.paws.wmcloud.org/User:Isaac_(WMF)/HuggingFace%20Wikipedia%20Processing.ipynb for more implementation details / some data around the overhead induced by adding the extra preprocessing steps (stripping link prefixes and magic words)

Closed by:
- #3435
Issue: Add Wikisource dataset
State: closed
Author: albertvillanova

## Adding a Dataset
- **Name:** *wikisource*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** Additional high quality textual data, besides Wikipedia.

Add loading script as "canonical" dataset (as it is the case for ""wikipedia").

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

CC: @geohci, @yjernite 

Comments:
See notebook by @geohci: https://public.paws.wmcloud.org/User:Isaac_(WMF)/HuggingFace%20Wikisource%20Processing.ipynb

See: https://huggingface.co/datasets/wikimedia/wikisource
Issue: Install Audio dependencies to support audio decoding
State: closed
Author: albertvillanova

## Dataset viewer issue for '*openslr*', '*projecte-aina/parlament_parla*'

**Link:** *https://huggingface.co/datasets/openslr*
**Link:** *https://huggingface.co/datasets/projecte-aina/parlament_parla*

Error:
```
Status code:   400
Exception:     ImportError
Message:       To support decoding audio files, please install 'librosa'.
```

Am I the one who added this dataset ? Yes-No
- openslr: No
- projecte-aina/parlament_parla: Yes


Comments:
https://huggingface.co/datasets/projecte-aina/parlament_parla -> works (but we still have to show an audio player)

https://huggingface.co/datasets/openslr -> another issue: `Message:       [Errno 2] No such file or directory: '/home/hf/datasets-preview-backend/zip:/asr_javanese/data/00/00004fe6aa.flac'`

Done

https://huggingface.co/datasets/projecte-aina/parlament_parla/viewer/clean/train works

<img width="1535" alt="Capture d’écran 2022-04-12 à 13 58 47" src="https://user-images.githubusercontent.com/1676121/162957855-cb3d9e2e-4b61-488c-99ca-8065cd8fe377.png">


But https://huggingface.co/datasets/openslr/viewer does not work

<img width="678" alt="Capture d’écran 2022-04-12 à 13 59 46" src="https://user-images.githubusercontent.com/1676121/162958013-e31ef2ae-f886-47b7-9f27-664ed3d4b5a1.png">

Same issue as #4126:

```
Status code:   400
Exception:     TypeError
Message:       __init__() got an unexpected keyword argument 'audio_column'
```

Fixed:
<img width="1561" alt="Capture d’écran 2022-04-25 à 18 11 51" src="https://user-images.githubusercontent.com/1676121/165129813-018ece9e-8b20-4544-844d-4e88148e738f.png">
Issue: Preserve all feature types when saving a dataset on the Hub with `push_to_hub`
State: closed
Author: mariosasko

Currently, if one of the dataset features is of type `ClassLabel`, saving the dataset with `push_to_hub` and reloading the dataset with `load_dataset` will return the feature of type `Value`. To fix this, we should do something similar to `save_to_disk` (which correctly preserves the types) and not only push the parquet files in `push_to_hub`, but also the dataset `info` (stored in a JSON file).

Comments:
According to this [comment in the forum](https://discuss.huggingface.co/t/save-datasetdict-to-huggingface-hub/12075/8?u=lhoestq), using `push_to_hub` on a dataset with `ClassLabel` can also make the feature simply disappear when it's reloaded !

Maybe we can also fix https://github.com/huggingface/datasets/issues/3035 while working on this because, as pointed out in my initial post, `save_to_disk` also saves the `dataset_info.json`  file.
Issue: Common Voice Belarusian Dataset
State: open
Author: wiedymi

## Adding a Dataset
- **Name:** *Common Voice Belarusian Dataset*
- **Description:** *[commonvoice.mozilla.org/be](https://commonvoice.mozilla.org/be)*
- **Data:** *[commonvoice.mozilla.org/be/datasets](https://commonvoice.mozilla.org/be/datasets)*
- **Motivation:** *It has more than 7GB of data, so it will be great to have it in this package so anyone can try to train something for Belarusian language.*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
Issue: Dataset viewer issue for `dansbecker/hackernews_hiring_posts`
State: closed
Author: severo

## Dataset viewer issue for `dansbecker/hackernews_hiring_posts`

**Link:** https://huggingface.co/datasets/dansbecker/hackernews_hiring_posts

*short description of the issue*

Dataset preview not showing for uploaded DatasetDict. See https://discuss.huggingface.co/t/dataset-preview-not-showing-for-uploaded-datasetdict/12603

Am I the one who added this dataset ?

No -> @dansbecker

Comments:
This issue was fixed by me calling `all_datasets.push_to_hub("hackernews_hiring_posts")`.

The previous problems were from calling `all_datasets.save_to_disk` and then pushing with `my_repo.git_add` and `my_repo.push_to_hub`.