SurAyush/github_issues huggingface dataset:
Issue: VoxPopuli
State: closed
Author: VictorSanh

## Adding a Dataset
- **Name:** VoxPopuli
- **Description:** A Large-Scale Multilingual Speech Corpus
- **Paper:** https://arxiv.org/pdf/2101.00390.pdf
- **Data:** https://github.com/facebookresearch/voxpopuli
- **Motivation:** one of the largest (if not the largest) multilingual speech corpus: 400K hours of multilingual unlabeled speech + 17k hours of labeled speech

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

👀 @kahne @Molugan


Comments:
duplicate of https://github.com/huggingface/datasets/issues/2300
Issue: Patch all module attributes in its namespace
State: closed
Author: albertvillanova

When patching module attributes, only those defined in its `__all__` variable were considered by default (only falling back to `__dict__` if `__all__` was None).

However those are only a subset of all the module attributes in its namespace (`__dict__` variable).

This PR fixes the problem of modules that have non-None `__all__` variable, but try to access an attribute present in `__dict__` (and not in `__all__`).

For example, `pandas` has attribute `__version__` only present in `__dict__`.
- Before version 1.4, pandas `__all__` was None, thus all attributes in `__dict__` were patched
- From version 1.4, pandas `__all__` is not None, thus attributes in `__dict__` not present in `__all__` are ignored

Fix #3724.

CC: @severo @lvwerra 

Comments:
Issue: Bug while streaming CSV dataset with pandas 1.4
State: closed
Author: albertvillanova

## Describe the bug
If we upgrade to pandas `1.4`, the patching of the pandas module is no longer working
```
AttributeError: '_PatchedModuleObj' object has no attribute '__version__'
```

## Steps to reproduce the bug
```
pip install pandas==1.4
```
```python
from datasets import load_dataset
ds = load_dataset("lvwerra/red-wine", split="train", streaming=True)
item = next(iter(ds))
item
```



Comments:
Issue: Fix flatten of complex feature types
State: closed
Author: mariosasko

Fix `flatten` for the following feature types: Image/Audio, Translation, and  TranslationVariableLanguages.

Inspired by `cast`/`table_cast`, I've introduced a `table_flatten` function to handle the Image/Audio types.

CC: @SBrandeis  

Fix #3686.


Comments:
Apparently the merge brought back some tests that use `flatten_()` that we removed recently

_The documentation is not available anymore as the PR was closed or merged._
Issue: Multi-GPU support for `FaissIndex`
State: closed
Author: rentruewang

Per #3716 , current implementation does not take into consideration that `faiss` can run on multiple GPUs.

In this commit, I provided multi-GPU support for `FaissIndex` by modifying the device management in `IndexableMixin.add_faiss_index` and `FaissIndex.load`.

Now users are able to pass in
1. a positive integer (as usual) to use 1 GPU
2. a negative integer `-1` to use all GPUs
3. a list of integers e.g. `[0, 1]` to run only on those GPUs
4. Of course, passing in nothing still runs on CPU.

This closes: #3716 

Comments:
Any love?

Hi, any update?

@albertvillanova Sorry for bothering you again, quick follow up: is there anything else you want me to add / modify?

Hi @rentruewang , we updated the documentation on `master`, could you merge `master` into your branch please ?

@lhoestq I've merge `huggingface/datasets/master` into this PR. Please review. Thanks! 🤗

Edit: Umm... I was experimenting with what renaming a branch would do to a pull request. Please ignore the `closed this PR` down below. 🤗
Issue: wrong condition in `Features ClassLabel encode_example`
State: closed
Author: Tudyx

## Describe the bug

The `encode_example` function in *features.py* seems to have a wrong condition.

```python
if not -1 <= example_data < self.num_classes:
    raise ValueError(f"Class label {example_data:d} greater than configured num_classes {self.num_classes}")
```

## Expected results

The `not - 1` condition change the result of the condition. For instance, if  `example_data`  equals 4 and ` self.num_classes` equals 4 too, `example_data < self.num_classes` will give `False` as expected  . But if i add the  `not - 1`  condition, `not -1 <= example_data < self.num_classes` will give `True` and raise an exception.

## Environment info

- `datasets` version: 1.18.3
- Python version: 3.8.10
- PyArrow version: 7.00


Comments:
Hi @Tudyx, 

Please note that in Python, the boolean NOT operator (`not`) has lower precedence than comparison operators (`<=`, `<`), thus the expression you mention is equivalent to:
```python
 not (-1 <= example_data < self.num_classes)
```

Also note that as expected, the exception is raised if:
- `example_data < -1`
- or `example_data >= self.num_classes`

The raise of the exception is expected when `example_data` equals 4 and `self.num_classes` equals 4 too.
Issue: `FaissIndex` to support multiple GPU and `custom_index`
State: closed
Author: rentruewang

**Is your feature request related to a problem? Please describe.**
Currently, because `device` is of the type `int | None`, to leverage `faiss-gpu`'s multi-gpu support, you need to create a `custom_index`. However, if using a `custom_index` created by e.g. `faiss.index_cpu_to_all_gpus`, then `FaissIndex.save` does not work properly because it checks the device id (which is an int, so no multiple GPUs).

**Describe the solution you'd like**
I would like `FaissIndex` to support multiple GPUs, by passing in a list to `add_faiss_index`.

**Describe alternatives you've considered**
Alternatively, I would like it to at least provide a warning cause it wasn't the behavior that I expected.

**Additional context**
Relavent source code here:
 https://github.com/huggingface/datasets/blob/6ed6ac9448311930557810383d2cfd4fe6aae269/src/datasets/search.py#L340-L349

Device management needs changing to support multiple GPUs, probably by `isinstance` calls.

I can provide a PR if you like :)

Thanks for reading!


Comments:
Hi @rentruewang, thansk for reporting and for your PR!!! We should definitely support this. 

@albertvillanova Great! :)
Issue: Fix bugs in msr_sqa dataset
State: closed
Author: Timothyxxx

The last version has many problems,

1) Errors in table load-in. Split by a single comma instead of using pandas is wrong.
2) id reduplicated in _generate_examples function.
3) Missing information of history questions which make it hard to use.

I fix it refer to https://github.com/HKUNLP/UnifiedSKG. And we test it to perform normally.

Comments:
It shows below when I run test:

FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_msr_sqa - ValueError: Unknown split "validation". Should be one of ['train', 'test'].

It make no sense for me😂. 


@albertvillanova Does this PR has some additional fixes compared to https://github.com/huggingface/datasets/pull/3771 or we can close it?

@mariosasko besides the fix of the DuplicatedKeysError, this PR:
- changes the reading of one of the files: use pandas instead of splitting by comma
- changes the splits: modifying train and adding validation
- adds some extra logic in the processing of the data: adding a new field "question_and_history"

We should decide whether validating these additional changes.
- for example, if we accept as pertinent the addition of the field "question_and_history", this should be added as feature to the info, and the matadata should be regenerated...

Hi guys, anything we can do to fix that bug👀? @mariosasko @albertvillanova @lhoestq 

_The documentation is not available anymore as the PR was closed or merged._
Issue: tatoeba_mt: File not found error and key error
State: closed
Author: jorgtied

## Dataset viewer issue for 'tatoeba_mt'

**Link:** https://huggingface.co/datasets/Helsinki-NLP/tatoeba_mt

My data loader script does not seem to work.
The files are part of the local repository but cannot be found. An example where it should work is the subset for "afr-eng".

Another problem is that I do not have validation data for all subsets and I don't know how to properly check whether validation exists in the configuration before I try to download it. An example is the subset for "afr-deu".

Am I the one who added this dataset ? Yes


Comments:
Looks like I solved my problems ...
Issue: Rm sphinx doc
State: closed
Author: mishig25

Checklist 
- [x] Update circle ci yaml
- [x] Delete sphinx static & python files in docs dir
- [x] Update readme in docs dir
- [ ] Update docs config in setup.py

Comments:
Thanks for pushing this :)
One minor comment regarding the PR itself - I noticed that some changes are coming from the upstream master, this might be due to a rebase. Would be nice if this PR doesn't include them for readabily, feel free to open a new one if necessary

Closing in favour https://github.com/huggingface/datasets/pull/3741
Issue: Set base path to hub url for canonical datasets
State: closed
Author: lhoestq

This should allow canonical datasets to use relative paths to download data files from the Hub

cc @polinaeterna this will be useful if we have audio datasets that are canonical and for which you'd like to host data files

Comments:
If we agree to have data files in a dedicated directory "data/" then we should be fine. You're right we should not try to edit a dataset script from the repository directly, but from github, in order to avoid conflicts
Issue: `.select`: unexpected behavior with `indices`
State: closed
Author: gabegma

## Describe the bug
The `.select` method will not throw when sending `indices` bigger than the dataset length; `indices` will be wrapped instead. This behavior is not documented anywhere, and is not intuitive. 

## Steps to reproduce the bug
```python
from datasets import Dataset
ds = Dataset.from_dict({"text": ["d", "e", "f"], "label": [4, 5, 6]})
res1 = ds.select([1, 2, 3])['text']
res2 = ds.select([1000])['text']
```

## Expected results
Both results should throw an `Error`.

## Actual results
`res1` will give `['e', 'f', 'd']`
`res2` will give `['e']`

## Environment info
Bug found from this environment:
- `datasets` version: 1.16.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.7
- PyArrow version: 6.0.1

It was also replicated on `master`.


Comments:
Hi! Currently, we compute the final index as `index % len(dset)`. I agree this behavior is somewhat unexpected and that it would be more appropriate to raise an error instead (this is what `df.iloc` in Pandas does, for instance).

@albertvillanova @lhoestq wdyt?

I agree. I think `index % len(dset)` was used to support negative indices.

I think this needs to be fixed in `datasets.formatting.formatting._check_valid_index_key` if I'm not mistaken
Issue: Unable to load dataset 'big_patent'
State: closed
Author: ankitk2109

## Describe the bug
Unable to load the "big_patent" dataset

## Steps to reproduce the bug
```python
load_dataset('big_patent', 'd', 'validation')
```

## Expected results
Download big_patents' validation split from the 'd' subset

## Getting an error saying:
{FileNotFoundError}Local file ..\huggingface\datasets\downloads\6159313604f4f2c01e7d1cac52139343b6c07f73f6de348d09be6213478455c5\bigPatentData\train.tar.gz doesn't exist

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:1.18.3
- Platform: Windows
- Python version:3.8
- PyArrow version:7.0.0


Comments:
Hi @ankitk2109,

Have you tried passing the split name with the keyword `split=`? See e.g. an example in our Quick Start docs: https://huggingface.co/docs/datasets/quickstart.html#load-the-dataset-and-model
```python
 ds = load_dataset("big_patent", "d", split="validation")

Hi @albertvillanova,

Thanks for your response.

Yes, I tried the `split='validation'` as well. But getting the same issue. 

I'm sorry, but I can't reproduce your problem:
```python
In [5]: ds = load_dataset("big_patent", "d", split="validation")
Downloading and preparing dataset big_patent/d (download: 6.01 GiB, generated: 169.61 MiB, post-processed: Unknown size, total: 6.17 GiB) to .../.cache/big_patent/d/1.0.0/bdefa7c0b39fba8bba1c6331b70b738e30d63c8ad4567f983ce315a5fef6131c...
Downloading data: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6.45G/6.45G [27:36<00:00, 3.89MB/s]
Extracting data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [03:18<00:00, 66.08s/it]
Dataset big_patent downloaded and prepared to .../.cache/big_patent/d/1.0.0/bdefa7c0b39fba8bba1c6331b70b738e30d63c8ad4567f983ce315a5fef6131c. Subsequent calls will reuse this data.    

In [6]: ds
Out[6]: 
Dataset({
    features: ['description', 'abstract'],
    num_rows: 565
})


Maybe you had a connection issue while downloading the file and this was corrupted?
Our cache system uses the file you downloaded first time.
If so, you could try forcing redownload of the file with:
```python
ds = load_dataset("big_patent", "d", split="validation", download_mode="force_redownload")

I am able to download the dataset with ``` download_mode="force_redownload"```. As you mentioned it was an issue with the cached version which was failed earlier due to a network issue. I am closing the issue now, once again thank you.
Issue: ImportError: To be able to use this metric, you need to install the following dependencies['seqeval'] using 'pip install seqeval' for instance'
State: closed
Author: zhangyifei1

hi :
I want to use the seqeval indicator because of direct load_ When metric ('seqeval '), it will prompt that the network connection fails. So I downloaded the seqeval Py to load locally. Loading code: metric = load_ metric(path='mymetric/seqeval/seqeval.py')

But tips:

Traceback (most recent call last):
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers/examples/pytorch/token-classification/run_ner.py", line 604, in <module>
    main()
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers/examples/pytorch/token-classification/run_ner.py", line 481, in main
    metric = load_metric(path='mymetric/seqeval/seqeval.py')
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers_venv_0209/lib/python3.7/site-packages/datasets/load.py", line 610, in load_metric
    dataset=False,
  File "/home/ubuntu/Python3.6_project/zyf_project/transformers_venv_0209/lib/python3.7/site-packages/datasets/load.py", line 450, in prepare_module
    f"To be able to use this {module_type}, you need to install the following dependencies"
ImportError: To be able to use this metric, you need to install the following dependencies['seqeval'] using 'pip install seqeval' for instance'


**What should I do? Please help me, thank you**



Comments:
![图片](https://user-images.githubusercontent.com/28425091/153547502-6bb0938d-788b-4857-b946-c3cf08fefce4.png)
My datasets version

![图片](https://user-images.githubusercontent.com/28425091/153547587-f4677166-af9b-44a0-95ad-b6dba873978a.png)


Hi! Some of our metrics require additional dependencies to work. In your case, simply installing the `seqeval` package with `pip install seqeval` should resolve the issue.

> Hi! Some of our metrics require additional dependencies to work. In your case, simply installing the `seqeval` package with `pip install seqeval` should resolve the issue.
I installed seqeval, but still reported the same error. That's too bad.


> > Hi! Some of our metrics require additional dependencies to work. In your case, simply installing the `seqeval` package with `pip install seqeval` should resolve the issue.
> > I installed seqeval, but still reported the same error. That's too bad.

Same issue here. What should I do to fix this error? Please help! Thank you.

I tried to install **seqeval** package through anaconda instead of pip:
`conda install -c conda-forge seqeval`
It worked for me!

I can run it through the following steps:
![image](https://user-images.githubusercontent.com/69563759/159264511-1e252a4e-c8c8-44ab-b7bc-b4aac609bd9e.png)
Thank you for answering for me!

just change the file name seqeval.py to myseqeval.py

Metrics are deprecated in `datasets` and `evaluate` should be used instead: https://github.com/huggingface/evaluate
Issue: Unable to load a dataset
State: closed
Author: PaulchauvinAI

## Describe the bug
Unable to load a dataset from Huggingface that I have just saved.


## Steps to reproduce the bug
On Google colab
`! pip install datasets `
`from datasets import load_dataset`
`my_path = "wiki_dataset"`
`dataset = load_dataset('wikipedia', "20200501.fr")`
`dataset.save_to_disk(my_path)`
`dataset = load_dataset(my_path)`


## Expected results
Loading the dataset

## Actual results
ValueError: Couldn't cast
_data_files: list<item: struct<filename: string>>
  child 0, item: struct<filename: string>
      child 0, filename: string
_fingerprint: string
_format_columns: null
_format_kwargs: struct<>
_format_type: null
_indexes: struct<>
_output_all_columns: bool
_split: string
to
{'builder_name': Value(dtype='string', id=None), 'citation': Value(dtype='string', id=None), 'config_name': Value(dtype='string', id=None), 'dataset_size': Value(dtype='int64', id=None), 'description': Value(dtype='string', id=None), 'download_checksums': {}, 'download_size': Value(dtype='int64', id=None), 'features': {'title': {'dtype': Value(dtype='string', id=None), 'id': Value(dtype='null', id=None), '_type': Value(dtype='string', id=None)}, 'text': {'dtype': Value(dtype='string', id=None), 'id': Value(dtype='null', id=None), '_type': Value(dtype='string', id=None)}}, 'homepage': Value(dtype='string', id=None), 'license': Value(dtype='string', id=None), 'post_processed': Value(dtype='null', id=None), 'post_processing_size': Value(dtype='null', id=None), 'size_in_bytes': Value(dtype='int64', id=None), 'splits': {'train': {'name': Value(dtype='string', id=None), 'num_bytes': Value(dtype='int64', id=None), 'num_examples': Value(dtype='int64', id=None), 'dataset_name': Value(dtype='string', id=None)}}, 'supervised_keys': Value(dtype='null', id=None), 'task_templates': Value(dtype='null', id=None), 'version': {'version_str': Value(dtype='string', id=None), 'description': Value(dtype='string', id=None), 'major': Value(dtype='int64', id=None), 'minor': Value(dtype='int64', id=None), 'patch': Value(dtype='int64', id=None)}}
because column names don't match

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1



Comments:
Hi! `load_dataset` is intended to be used to load a canonical dataset (`wikipedia`), a packaged dataset (`csv`, `json`, ...) or a dataset hosted on the Hub. For local datasets saved with `save_to_disk("path/to/dataset")`, use `load_from_disk("path/to/dataset")`.

Maybe we should raise an informative error message in this case...

How should we load a locally self-gathered dataset then?
Issue: Add dev-only config to Natural Questions dataset
State: closed
Author: albertvillanova

As suggested by @lhoestq and @thomwolf, a new config has been added to Natural Questions dataset, so that only dev split can be downloaded. 

Fix #413.

Comments:
Great thanks ! I think we can fix the CI by copying the NQ folder on gcs to 0.0.3. Does that sound good ?

I've copied the 0.0.2 folder content to 0.0.3, as suggested.

I'm updating the dataset card...
Issue: Add finetune-data CodeFill 
State: closed
Author: rgismondi


Comments:
Thanks for your contribution, @rgismondi. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Issue: Add code-fill datasets for pretraining/finetuning/evaluating
State: closed
Author: rgismondi


Comments:
Hi ! Thanks for adding this dataset :)

It looks like your PR contains many changes in files that are unrelated to your changes, I think it might come from running `make style` with an outdated version of `black`. Could you try opening a new PR that only contains your additions ? (or force push to this PR)
Issue: Fix ClassLabel to/from dict when passed names_file
State: closed
Author: albertvillanova

Currently, `names_file` is a field of the data class `ClassLabel`, thus appearing when transforming it to dict (when saving infos). Afterwards, when trying to read it from infos, it conflicts with the other field `names`.

This PR, removes `names_file` as a field of the data class `ClassLabel`.
- it is only used at instantiation to generate the `labels` field

Fix #3631.

Comments:
Issue: Update data URL in pubmed dataset
State: closed
Author: albertvillanova

Fix #3655.

Comments:
- I updated the previous dummy data: I just had to rename the file and its directory
  - the dummy data zip contains only a single file: `pubmed22n0001.xml.gz`

Then I discover it fails: https://app.circleci.com/pipelines/github/huggingface/datasets/9800/workflows/173a4433-8feb-4fc6-ab9e-59762084e3e1/jobs/60437
```
No such file or directory: '.../dummy_data/pubmed22n0002.xml.gz'
```
- it needs dummy data for all the 1114 files: 
  `_URLs = [f"ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed22n{i:04d}.xml.gz" for i in range(1, 1115)]`
- this confirms me that it never passed the test: these dummy data files were not present before my PR
- therefore, is it really useful the data test if we just ignore it when it does not pass?

In relation with JSON metadata, I was generating the file for `pubmed` (see above) in a GCP instance: after running during ~12h without finishing, I decided to stop the process.

Hi ! Yes I remembered we hardcoded an exception for this one:
https://github.com/huggingface/datasets/blob/36db39c75179a0a491c69a4491f7ae7e4615e66f/src/datasets/utils/mock_download_manager.py#L174-L176

The exception was used to only require one dummy data file, feel free to update it if you want