noamaanMulla-03/datasets-issues huggingface dataset:
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.

Hi, when i try to run prepare_dataset function in [fine tuning ASR tutorial 4](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_tuning_Wav2Vec2_for_English_ASR.ipynb) , i got this error.
I got this error
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
There is no other logs available, so i have no clue what is the cause of it.
```

def prepare_dataset(batch):
    audio = batch["path"]
    # batched output is "un-batched"
    batch["input_values"] = processor(audio["array"], sampling_rate=audio["sampling_rate"]).input_values[0]
    batch["input_length"] = len(batch["input_values"])
    with processor.as_target_processor():
        batch["labels"] = processor(batch["text"]).input_ids
    return batch

data = data.map(prepare_dataset, remove_columns=data.column_names["train"],
                      num_proc=4)
```


Specify the actual results or traceback.
There is no traceback except
`Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.`
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-5.15.0-43-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.4.3


Labels: name: bug

Fixed by installing either resampy<3 or resampy>=4
No module named 'fsspec.archive'

## Describe the bug
Cannot import datasets after installation.

## Steps to reproduce the bug
```shell
$ python
Python 3.9.7 (default, Sep 16 2021, 13:09:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import datasets
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/__init__.py", line 34, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 61, in <module>
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/arrow_writer.py", line 28, in <module>
    from .features import (
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/features/__init__.py", line 2, in <module>
    from .audio import Audio
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/features/audio.py", line 7, in <module>
    from ..utils.streaming_download_manager import xopen
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 18, in <module>
    from ..filesystems import COMPRESSION_FILESYSTEMS
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/filesystems/__init__.py", line 6, in <module>
    from . import compression
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/filesystems/compression.py", line 5, in <module>
    from fsspec.archive import AbstractArchiveFileSystem
ModuleNotFoundError: No module named 'fsspec.archive'
```


Labels: name: bug
Is it possible to pass multiple links to a split in load script?

**Is your feature request related to a problem? Please describe.**
I wanted to use a python loading script in hugging face datasets that use different sources of text (it's somehow a compilation of multiple datasets + my own dataset) based on how `load_dataset` [works](https://huggingface.co/docs/datasets/loading) I assumed I could do something like bellow in my loading script:

```python
...
_URL = "MY_DATASET_URL/resolve/main/data/"
_URLS = {
    "train": [
        "FIRST_URL_TO.txt",
        _URL + "train-00000-of-00001-676bfebbc8742592.parquet"
    ]
}
...
```
but when loading the dataset it raises the following error:
```python
File ~/.local/lib/python3.8/site-packages/datasets/builder.py:704, in DatasetBuilder.download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    702         logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    703 if not downloaded_from_gcs:
--> 704     self._download_and_prepare(
    705         dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
...
    668     if isinstance(a, str):
    669         # Force-cast str subclasses to str (issue #21127)
    670         parts.append(str(a))

TypeError: expected str, bytes or os.PathLike object, not list
```

**Describe the solution you'd like**
I believe since it's possible for `load_dataset` to get list of URLs instead of just a URL for `train` split it can be possible here too.

**Describe alternatives you've considered**
An alternative solution would be to download all needed datasets locally and `push_to_hub` them all, but since the datasets I'm talking about are huge it's not among my options. 

**Additional context**
I think loading `text` beside the `parquet` is completely a different issue but I believe I can figure it out by proposing a config for my dataset to load each entry of `_URLS['train']` separately either by `load_dataset("text", ...` or `load_dataset("parquet", ...`.


Labels: name: enhancement
Bug in function validate_type for Python >= 3.9

## Describe the bug
The function `validate_type` assumes that the type `typing.Optional[str]` is automatically transformed to `typing.Union[str, NoneType]`.
```python
In [4]: typing.Optional[str]
Out[4]: typing.Union[str, NoneType]
```

However, this is not the case for Python 3.9:
```python
In [3]: typing.Optional[str]
Out[3]: typing.Optional[str]
```


Labels: name: bug
Fix bug in function validate_type for Python >= 3.9

Fix `validate_type` function, so that it uses `get_origin` instead. This makes the function forward compatible.

This fixes #4811 because:
```python
In [4]: typing.Optional[str]
Out[4]: typing.Optional[str]

In [5]: get_origin(typing.Optional[str])
Out[5]: typing.Union
```

Fix #4811.


_The documentation is not available anymore as the PR was closed or merged._
Fix test of _get_extraction_protocol for TAR files

While working in another PR, I discovered an xpass test (a test that is supposed to xfail but nevertheless passes) when testing `_get_extraction_protocol`: https://github.com/huggingface/datasets/runs/7818845285?check_suite_focus=true
```
XPASS tests/test_streaming_download_manager.py::test_streaming_dl_manager_get_extraction_protocol_throws[https://foo.bar/train.tar] 
```

This PR:
- refactors the test so that it tests the raise of the exceptions instead of xfailing
- fixes the test for TAR files: it does not raise an exception, but returns "tar"
- fixes some tests wrongly named: exchange `test_streaming_dl_manager_get_extraction_protocol` with `test_streaming_dl_manager_get_extraction_protocol_gg_drive`


_The documentation is not available anymore as the PR was closed or merged._
Update documentation card of miam dataset

Hi !
Paper has been published at EMNLP.


_The documentation is not available anymore as the PR was closed or merged._

Ahahah :D not sur how i broke something by updating the README :D 

Thanks for the fix @PierreColombo. 

Once a README is modified, our CI runs tests on it, requiring additional quality fixes, so that all READMEs are progressively improved and have some minimal tags/sections/information.

For this specific README file, the additional quality requirements of the CI are: https://github.com/huggingface/datasets/runs/7819924428?check_suite_focus=true
```
E           The following issues were found for the README at `/home/runner/work/datasets/datasets/datasets/miam/README.md`:
E           -	Section `Additional Information` is missing subsection: `Dataset Curators`.
E           -	Section `Additional Information` is missing subsection: `Contributions`.
E           -	`Additional Information` has an extra subsection: `Benchmark Curators`. Skipping further validation checks for this subsection as expected structure is unknown.
```

Thanks a lot Albert :)))
No preprocessed wikipedia is working on huggingface/datasets

## Describe the bug
20220301 wikipedia dump has been deprecated, so now there is no working wikipedia dump on huggingface

https://huggingface.co/datasets/wikipedia

https://dumps.wikimedia.org/enwiki/




Labels: name: bug

Thanks for reporting @aninrusimha.

Please, note that the preprocessed datasets are still available, as described in the dataset card, e.g.: https://huggingface.co/datasets/wikipedia
```python
ds = load_dataset("wikipedia", "20220301.en")
``` 

This is working now, but I was getting an error a few days ago when running an existing script. Unfortunately I did not do a proper bug report, but for some reason I was unable to load the dataset due to a request being made to the wikimedia website. However, its working now. Thanks for the reply!
can't install using conda on Windows 10

## Describe the bug
I wanted to install using conda or Anaconda navigator. That didn't work, so I had to install using pip.

## Steps to reproduce the bug
conda install -c huggingface -c conda-forge datasets

## Expected results
Should have indicated successful installation.

## Actual results
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
... took forever, so I cancelled it with ctrl-c

## Environment info
- `datasets` version: 2.4.0  # after installing with pip
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.9.12
- PyArrow version: 9.0.0
- Pandas version: 1.4.2
- conda version: 4.13.0

conda info

     active environment : base
    active env location : G:\anaconda2022
            shell level : 1
       user config file : C:\Users\michael\.condarc
 populated config files : C:\Users\michael\.condarc
          conda version : 4.13.0
    conda-build version : 3.21.8
         python version : 3.9.12.final.0
       virtual packages : __cuda=11.1=0
                          __win=0=0
                          __archspec=1=x86_64
       base environment : G:\anaconda2022  (writable)
      conda av data dir : G:\anaconda2022\etc\conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/pytorch/win-64
                          https://conda.anaconda.org/pytorch/noarch
                          https://conda.anaconda.org/huggingface/win-64
                          https://conda.anaconda.org/huggingface/noarch
                          https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/anaconda-fusion/win-64
                          https://conda.anaconda.org/anaconda-fusion/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : G:\anaconda2022\pkgs
                          C:\Users\michael\.conda\pkgs
                          C:\Users\michael\AppData\Local\conda\conda\pkgs
       envs directories : G:\anaconda2022\envs
                          C:\Users\michael\.conda\envs
                          C:\Users\michael\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.13.0 requests/2.27.1 CPython/3.9.12 Windows/10 Windows/10.0.19044
          administrator : False
             netrc file : None
           offline mode : False


Labels: name: bug
Pass new_fingerprint in multiprocessing

Following https://github.com/huggingface/datasets/pull/3045

Currently one can pass `new_fingerprint` to `.map()` to use a custom fingerprint instead of the one computed by hashing the map transform. However it's ignored if `num_proc>1`.

In this PR I fixed that by passing `new_fingerprint` to `._map_single()` when `num_proc>1`.
More specifically, `new_fingerprint` with a suffix based on the process `rank` is passed, so that each process has a different `new_fingerprint`

cc @TevenLeScao @vlievin


@lhoestq Hi~, does this support that `datasets.map(func, batched=True, batch_size, num_proc>1, new_fingerprint="func_v1")` even if `func` can't pickle. I also notice that you said "Unfortunately you need picklable mapping functions to make multiprocessing work :confused: Also feel free to open an issue or send me a dm if you are in a situation where the caching fails. I can help you with that :slight_smile:" in [here](https://discuss.huggingface.co/t/how-to-deal-with-unpickable-objects-in-map/1547/8). So, I want to ask that is there a way for users to use multiprocessing in `datasets.map` when the `func` can't pickle? 
Thanks in advance!

Yea you need your function to be picklable for multiprocessing, otherwise the main process is not able to pass your function to the child processes.
Include entity positions as feature in ReCoRD

https://huggingface.co/datasets/super_glue/viewer/record/validation

TLDR: We need to record entity positions, which are included in the source data but excluded by the loading script, to enable efficient and effective training for ReCoRD.

Currently, the loading script ignores the entity positions ("entity_start", "entity_end") and only records entity text. This might be because the training method of the official baseline is to make n training instance from a datapoint by replacing \"\@ placeholder\" in query with each entity individually.

But it increases the already heavy computation by multiple folds. So DeBERTa uses a method that take entity embeddings by their positions in the passage, and thus makes one training instance from one data point. It is way more efficient and proved effective for the ReCoRD task.

Can anybody help me with the dataset card rendering error? Maybe @lhoestq ?


_The documentation is not available anymore as the PR was closed or merged._

Thanks for the reply @lhoestq !

I have sucessed on `datasets-cli test ./datasets/super_glue --name record --save_infos`,
But as you can see, the check ran into `FAILED tests/test_dataset_cards.py::test_changed_dataset_card[super_glue] - V...`.
How can we solve it?

That would be neat! Let me implement it.
TFDS wiki_dialog dataset to Huggingface dataset

## Adding a Dataset
- **Name:** *Wiki_dialog*
- **Description: https://github.com/google-research/dialog-inpainting#:~:text=JSON%20object%2C%20for-,example,-%3A
- **Paper: https://arxiv.org/abs/2205.09073
- **Data: https://github.com/google-research/dialog-inpainting
- **Motivation:** *Research and Development on biggest corpus of dialog data*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).


Labels: name: dataset request

@albertvillanova any help ? The linked dataset is in beam format which is similar to wikipedia dataset in huggingface that you  scripted..

Nvm, I was able to port it to huggingface datasets, will upload to the hub soon

https://huggingface.co/datasets/djaym7/wiki_dialog

Thanks for the addition, @djaym7.
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648

## Describe the bug
Following the example in CodeParrot, I receive an array size limitation error when deduplicating larger datasets.

## Steps to reproduce the bug
```python
dataset_name = "the_pile"
ds = load_dataset(dataset_name, split="train")
ds = ds.map(preprocess, num_proc=num_workers)
uniques = set(ds.unique("hash"))
```
Gists for minimum reproducible example:
https://gist.github.com/conceptofmind/c5804428ea1bd89767815f9cd5f02d9a
https://gist.github.com/conceptofmind/feafb07e236f28d79c2d4b28ffbdb6e2

## Expected results
Chunking and writing out a deduplicated dataset. 

## Actual results
```
return dataset._data.column(column).unique().to_pylist()
File "pyarrow/table.pxi", line 394, in pyarrow.lib.ChunkedArray.unique
File "pyarrow/_compute.pyx", line 531, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 330, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 124, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648
```

Labels: name: bug

Thanks for reporting @conceptofmind.

Could you please give details about your environment? 
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:
```

Hi @albertvillanova ,

Here is the environment information:
```
- `datasets` version: 2.3.2
- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.27
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.4.2
```
Thanks,

Enrico

I think this issue is solved here https://discuss.huggingface.co/t/minhash-deduplication/19992/12?u=loubnabnl, this only happens for very large datasets we will update it in CodeParrot code

Hi @loubnabnl,

Yes, the issue is solved in the discussion thread.

I will close this issue.

Thank you again for all of your help.

Enrico

Thanks @loubnabnl for pointing out the solution to this issue.
Dataset Viewer issue for super_glue

### Link

https://huggingface.co/datasets/super_glue

### Description

can't view super_glue dataset on the web page

### Owner

_No response_

Labels: name: dataset-viewer

Thanks for reporting @wzsxxa.

However the "super_glue" dataset is rendered properly by the Dataset preview: https://huggingface.co/datasets/super_glue
adding mafand to datasets

I'm addding the MAFAND dataset by Masakhane based on the paper/repository below:

Paper: https://aclanthology.org/2022.naacl-main.223/
Code: https://github.com/masakhane-io/lafand-mt

Please, help merge this

Everything works except for creating dummy data file

Labels: name: wontfix

_The documentation is not available anymore as the PR was closed or merged._

Hi @dadelani, thanks for your awesome contribution!!! :heart: 

However, now we are using the Hub to add new datasets, instead of this GitHub repo. 

You could share this dataset under your Hub organization namespace: [Masakhane NLP](https://huggingface.co/masakhane). This way the dataset will be accessible using:
```python
ds = load_dataset("masakhane/mafand")
```

You have the procedure documented in our online docs: 
- [Create a dataset loading script](https://huggingface.co/docs/datasets/dataset_script)
- [Share](https://huggingface.co/docs/datasets/share)

Moreover, datasets shared on the Hub no longer need the dummy data files.

Please, feel free to ping me if you need any further guidance/support.

thank you for the comment. I have moved it to the Hub https://huggingface.co/datasets/masakhane/mafand

Great job, @dadelani!!

Please, note that in the README.md file, the YAML tags should be preceded and followed by three dashes `---`, so that they are properly parsed. See, e.g.: https://raw.githubusercontent.com/huggingface/datasets/main/templates/README.md

Also you could replace the line:
```
# Dataset Card for [Needs More Information]
```
with
```
# Dataset Card for MAFAND-MT
```

Great, thank you for the feedback. I have fixed both issues.
Dataset Viewer issue for MoritzLaurer/multilingual_nli

### Link

_No response_

### Description

I've just uploaded a new dataset to the hub and the viewer does not work for some reason, see here: https://huggingface.co/datasets/MoritzLaurer/multilingual_nli

It displays the error: 
```
Status code:   400
Exception:     Status400Error
Message:       The dataset does not exist.
```

Weirdly enough the dataviewer works for an earlier version of the same dataset. The only difference is that it is smaller, but I'm not aware of other changes I have made: https://huggingface.co/datasets/MoritzLaurer/multilingual_nli_test

Do you know why the dataviewer is not working?

### Owner

_No response_

Labels: name: dataset-viewer

Thanks for reporting  @MoritzLaurer.

Currently, the dataset preview is working properly: https://huggingface.co/datasets/MoritzLaurer/multilingual_nli

Please note that when a dataset is modified, it might take some time until the preview is completely updated.

@severo might it be worth adding a clearer error message, something like "The preview is updating, please retry later"?

Thanks for your response. You are right, its now working well. I had waited for 30 min or so and refreshed several times and thought there was some other error. Yeah, a different error message sounds like a good idea to avoid confusion. 

I'm closing this issue then.

> @severo might it be worth adding a clearer error message, something like "The preview is updating, please retry later"?

Yes, it's a known issue, and we're about to ship a better version
Issue with fine classes in trec dataset

## Describe the bug
According to their paper, the TREC dataset contains 2 kinds of classes:
- 6 coarse classes: TREC-6
- 50 fine classes: TREC-50

However, our implementation only has 47 (instead of 50) fine classes. The reason for this is that we only considered the last segment of the label, which is repeated for several coarse classes:
- We have one `desc` fine label instead of 2:
  - `DESC:desc`
  - `HUM:desc`
- We have one `other` fine label instead of 3:
  - `ENTY:other`
  - `LOC:other`
  - `NUM:other`

From their paper:
> We define a two-layered taxonomy, which represents a natural semantic classification for typical answers in the TREC task. The hierarchy contains 6 coarse classes and 50 fine classes,

> Each coarse class contains a non-overlapping set of fine classes.



Labels: name: bug
Fix fine classes in trec dataset

This PR:
- replaces the fine labels, so that there are 50 instead of 47
- once more labels are added, all they (fine and coarse) have been re-ordered, so that they align with the order in: https://cogcomp.seas.upenn.edu/Data/QA/QC/definition.html
- the feature names have been fixed: `fine_label` instead of `label-fine`
  - to sneak-case (underscores instead of hyphens)
  - words have been reordered

Fix #4790.


_The documentation is not available anymore as the PR was closed or merged._