noamaanMulla-03/datasets-issues huggingface dataset:
Fix iter_archive getting reset

The `DownloadManager.iter_archive` method currently returns an iterator - which is  **empty** once you iter over it once. This means you can't pass the same archive iterator to several splits.

To fix that, I changed the ouput of `DownloadManager.iter_archive` to be an iterable that you can iterate over several times, instead of a one-time-use iterator.

The `StreamingDownloadManager.iter_archive` already returns an appropriate iterable, and the code added in this PR is inspired from the one in `streaming_download_manager.py`
Add URL and ID fields to Wikipedia dataset

This PR adds the URL field, so that we conform to proper attribution, required by their license: provide credit to the authors by including a hyperlink (where possible) or URL to the page or pages you are re-using.

About the conversion from title to URL, I found that apart from replacing blanks with underscores, some other special character must also be percent-encoded (e.g. `"` to `%22`): https://meta.wikimedia.org/wiki/Help:URL

Therefore, I have finally used `urllib.parse.quote` function. This additionally percent-encodes non-ASCII characters, but Wikimedia docs say these are equivalent:
> For the other characters either the code or the character can be used in internal and external links, they are equivalent. The system does a conversion when needed.
> [[%C3%80_propos_de_M%C3%A9ta]]
> is rendered as [À_propos_de_Méta](https://meta.wikimedia.org/wiki/%C3%80_propos_de_M%C3%A9ta), almost like [À propos de Méta](https://meta.wikimedia.org/wiki/%C3%80_propos_de_M%C3%A9ta), which leads to this page on Meta with in the address bar the URL
> [http://meta.wikipedia.org/wiki/%C3%80_propos_de_M%C3%A9ta](https://meta.wikipedia.org/wiki/%C3%80_propos_de_M%C3%A9ta)
> while [http://meta.wikipedia.org/wiki/À_propos_de_Méta](https://meta.wikipedia.org/wiki/%C3%80_propos_de_M%C3%A9ta) leads to the same. 

Fix #3398.
CC: @geohci


Do you think we have a dedicated branch for all the changes we want to do to wikipedia ? Then once everything looks good + we have preprocessed the main languages, we can merge it on the `master` branch

Yes, @lhoestq, I agree with you.

I have just created the dedicated branch [`update-wikipedia`](https://github.com/huggingface/datasets/tree/update-wikipedia). We can merge every PR (once validated) to that branch; once all changes are merged to that branch, we could create the preprocessed datasets and then merge the branch to master. 

@lhoestq I guess you approve this PR?
`pubmed_qa` checksum mismatch

## Describe the bug
Loading [`pubmed_qa`](https://huggingface.co/datasets/pubmed_qa) results in a mismatched checksum error.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
import datasets
try:
    datasets.load_dataset("pubmed_qa", "pqa_labeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_unlabeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_artificial")
except Exception as e:
    print(e)
```

## Expected results
Successful download.

## Actual results
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/datasets/load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1RsGLINVce-0GsDkCLDuLZmoLuzfmoCuQ', 'https://drive.google.com/uc?export=download&id=15v1x6aQDlZymaHGP7cZJZZYFfeJt2NdS']

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: macOS
- Python version: 3.8.1
- PyArrow version: 3.0.0


Labels: name: bug

name: duplicate

Hi @jon-tow, thanks for reporting.

This issue was already reported and its root cause is a change in the Google Drive service. See:
- #3786 

We have already fixed it. See:
- #3787 

We are planning to make a patch release today.

In the meantime, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```
Simplify Common Voice code

In #3736 we introduced one method to generate examples when streaming, that is different from the one when not streaming.

In this PR I propose a new implementation which is simpler: it only has one function, based on `iter_archive`. And you still have access to local audio files when loading the dataset in non-streaming mode.

cc @patrickvonplaten @polinaeterna @anton-l @albertvillanova since this will become the template for many audio datasets to come.

This change can also trivially be applied to the other audio datasets that already exist. Using this line, you can get access to local files in non-streaming mode:
```python
local_extracted_archive = dl_manager.extract(archive_path) if not dl_manager.is_streaming else None
```


I think the script looks pretty clean and readable now! cool!
Fix Google Drive URL to avoid Virus scan warning

This PR fixes, in the datasets library instead of in every specific dataset, the issue of downloading the Virus scan warning page instead of the actual data file for Google Drive URLs.

Fix #3786, fix #3784.


Thanks for this @albertvillanova!

Once this PR merged into master and until our next `datasets` library release, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```

Thanks, that solved a bunch of problems we had downstream!
cf. https://github.com/ElementAI/picard/issues/61
Integrate Milvus (pymilvus) library

Milvus is a popular open-source vector database. We should add a new vector index to support this project.

Labels: name: enhancement

Hi @mariosasko，Just search randomly and I found this issue~ I'm the tech lead of Milvus and we are looking forward to integrate milvus together with huggingface datasets.

Any suggestion on how we could start?


Feel free to assign to me and we probably need some guide on it

@mariosasko any updates my man?


Hi! For starters, I suggest you take a look at this file: https://github.com/huggingface/datasets/blob/master/src/datasets/search.py, which contains all the code for Faiss/ElasticSearch support. We could set up a Slack channel for additional guidance. Let me know what you prefer.

> Hi! For starters, I suggest you take a look at this file: https://github.com/huggingface/datasets/blob/master/src/datasets/search.py, which contains all the code for Faiss/ElasticSearch support. We could set up a Slack channel for additional guidance. Let me know what you prefer.

Sure, we take a look and do some research
Pin responses to fix CI for Windows

Temporarily fix CI for Windows by pinning `responses`. 

See: https://app.circleci.com/pipelines/github/huggingface/datasets/10292/workflows/83de4a55-bff7-43ec-96f7-0c335af5c050/jobs/63355

Fix: #3839


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3840). All of your documentation changes will be reflected on that endpoint.
`FaissIndex` to support multiple GPU and `custom_index`

**Is your feature request related to a problem? Please describe.**
Currently, because `device` is of the type `int | None`, to leverage `faiss-gpu`'s multi-gpu support, you need to create a `custom_index`. However, if using a `custom_index` created by e.g. `faiss.index_cpu_to_all_gpus`, then `FaissIndex.save` does not work properly because it checks the device id (which is an int, so no multiple GPUs).

**Describe the solution you'd like**
I would like `FaissIndex` to support multiple GPUs, by passing in a list to `add_faiss_index`.

**Describe alternatives you've considered**
Alternatively, I would like it to at least provide a warning cause it wasn't the behavior that I expected.

**Additional context**
Relavent source code here:
 https://github.com/huggingface/datasets/blob/6ed6ac9448311930557810383d2cfd4fe6aae269/src/datasets/search.py#L340-L349

Device management needs changing to support multiple GPUs, probably by `isinstance` calls.

I can provide a PR if you like :)

Thanks for reading!


Labels: name: enhancement

Hi @rentruewang, thansk for reporting and for your PR!!! We should definitely support this. 

@albertvillanova Great! :)
Multi-GPU support for `FaissIndex`

Per #3716 , current implementation does not take into consideration that `faiss` can run on multiple GPUs.

In this commit, I provided multi-GPU support for `FaissIndex` by modifying the device management in `IndexableMixin.add_faiss_index` and `FaissIndex.load`.

Now users are able to pass in
1. a positive integer (as usual) to use 1 GPU
2. a negative integer `-1` to use all GPUs
3. a list of integers e.g. `[0, 1]` to run only on those GPUs
4. Of course, passing in nothing still runs on CPU.

This closes: #3716 


Any love?

Hi, any update?

@albertvillanova Sorry for bothering you again, quick follow up: is there anything else you want me to add / modify?

Hi @rentruewang , we updated the documentation on `master`, could you merge `master` into your branch please ?

@lhoestq I've merge `huggingface/datasets/master` into this PR. Please review. Thanks! 🤗

Edit: Umm... I was experimenting with what renaming a branch would do to a pull request. Please ignore the `closed this PR` down below. 🤗
[Breaking] Align `map` when streaming: update instead of overwrite + add missing parameters

Currently the datasets in streaming mode and in non-streaming mode have two distinct API for `map` processing.

In this PR I'm aligning the two by changing `map` in streamign mode. This includes a **major breaking change** and will require a major release of the library: **Datasets 2.0**

In particular, `Dataset.map` adds new columns (with dict.update) BUT `IterableDataset.map` used to discard previous columns (it overwrites the dict). In this PR I'm chaning the `IterableDataset.map` to behave the same way as `Dataset.map`: it will update the examples instead of overwriting them.

I'm also adding those missing parameters to streaming `map`: with_indices, input_columns, remove_columns

### TODO
- [x] tests
- [x] docs

Related to https://github.com/huggingface/datasets/issues/3444


Right ! Will add it in another PR :)
Add rmse and mae metrics.

This PR adds RMSE - Root Mean Squared Error and MAE - Mean Absolute Error to the metrics API.
Both implementations are based on usage of sciket-learn.

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)

Any suggestions and changes required will be helpful.



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3844). All of your documentation changes will be reflected on that endpoint.

@dnaveenr This PR is in pretty good shape, so feel free to reopen it.
Align IterableDataset.shuffle with Dataset.shuffle

From #3444 , Dataset.shuffle can have the same API than IterableDataset.shuffle (i.e. in streaming mode).

Currently you can pass an optional seed to both if you want, BUT currently IterableDataset.shuffle always requires a buffer_size, used for approximate shuffling. I propose using a reasonable default value (maybe 1000) instead.

In this PR, I set the default `buffer_size` value to 1,000, and I reorder the `IterableDataset.shuffle` arguments to match `Dataset.shuffle`, i.e. making `seed` the first argument.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3842). All of your documentation changes will be reflected on that endpoint.

We should also add `generator` as a param to `shuffle` to fully align the APIs, no?

I added the `generator` argument.

I had to make a few other adjustments to make it work. In particular when you call `set_epoch()` on a streaming dataset, it updates the underlying random generator by using a new effective seed. The effective seed is generated using the previous generator and the epoch number.
Logo float left

<img width="1000" alt="Screenshot 2022-03-07 at 09 35 29" src="https://user-images.githubusercontent.com/11827707/156996422-339ba43e-932b-4849-babf-9321cb99c922.png">



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3836). All of your documentation changes will be reflected on that endpoint.

Weird, the logo doesn't seem to be floating on my side (using Chrome) at https://huggingface.co/docs/datasets/master/en/index

https://huggingface.co/docs/datasets/index

The needed css change from moon-landing just got deployed
The Pile's _FEATURE spec seems to be incorrect

## Describe the bug
If you look at https://huggingface.co/datasets/the_pile/blob/main/the_pile.py:

For "all"
*  the pile_set_name is never set for data
* there's actually an id field inside of "meta" 

For subcorpora pubmed_central and hacker_news:
* the meta is specified to be a string, but it's actually a dict with an id field inside.

## Steps to reproduce the bug


## Expected results
Feature spec should match the data I'd think?

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:


Labels: name: bug

Hi @dlwh, thanks for reporting.

Please note, that the source data files for "all" config are different from the other configurations.

The "all" config contains the official Pile data files, from https://mystic.the-eye.eu/public/AI/pile/
All data examples contain a "meta" dict with a single "pile_set_name" key:
```python
In [1]: from datasets import load_dataset
        ds = load_dataset("the_pile", "all", split="train", streaming=True)
        item = next(iter(ds))
Downloading builder script: 9.09kB [00:00, 4.42MB/s]

In [3]: item["meta"]
Out[3]: {'pile_set_name': 'Pile-CC'}
```

On the other hand, all the other subset configs data files come from the Pile preliminary components directory: https://mystic.the-eye.eu/public/AI/pile_preliminary_components/
For theses components, the "meta" field may have different keys depending on the subset: "id", "language", "pmid",... Because of that, if we had kept the `dict` data format for the "meta" field, we would have an error when trying to concatenate different subsets, whose "meta" keys are not identical. In order to avoid that, the "meta" field is cast to `str` in all these cases, so that there is no incompatibility in their "meta" data type when concatenating.

You can check, for example, that for "pubmed_central" the "meta" field is cast to `str`:
```python
In [4]: from datasets import load_dataset
        ds = load_dataset("the_pile", "pubmed_central", split="train", streaming=True)
        item = next(iter(ds))

In [5]: item["meta"]
Out[5]: "{'id': 'PMC6071596'}"
```

Feel free to reopen this issue if you have further questions.
Add tqdm arguments

In this PR, tqdm arguments can be passed to the map() function and such, in order to be more flexible.


Hi ! Thanks this will be very useful :)

It looks like there are some changes in the github diff that are not related to your contribution, can you try fixing this by merging `master` into your PR, or create a new PR from an updated version of `master` ?

I have already solved the conflict on this latest version. This is my first time sending PR, if there's anything I need to adjust just let me know~

Thanks, most changes are gone :)
It still seems to include changes though - do you mind try creating a new branch from upstream/master and create a new PR please ?

Yeah sure, I'll try to send a new PR today!

Please forward to [#3850](https://github.com/huggingface/datasets/pull/3850)

Thanks ! Closing this one in favor of https://github.com/huggingface/datasets/pull/3850/files
Add mIoU metric

This PR adds the mean Intersection-over-Union metric to the library, useful for tasks like semantic segmentation.

It is entirely based on mmseg's [implementation](https://github.com/open-mmlab/mmsegmentation/blob/master/mmseg/core/evaluation/metrics.py).

I've removed any PyTorch dependency, and rely on Numpy only.


Hmm the doctest failed again - maybe the full result needs to be on one single line

cc @lhoestq for the final review

Cool ! Feel free to merge if it's all good for you
when using to_tf_dataset with shuffle is true, not all completed batches are made

## Describe the bug
when converting a dataset to tf_dataset by using to_tf_dataset with shuffle true, the remainder is not converted to one batch

## Steps to reproduce the bug
this is the sample code below
https://colab.research.google.com/drive/1_oRXWsR38ElO1EYF9ayFoCU7Ou1AAej4?usp=sharing


## Expected results
regardless of shuffle is true or not, 67 rows dataset should be 5 batches when batch size is 16.

## Actual results
4 batches

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1


Labels: name: bug

Maybe @Rocketknight1 can help here

Hi @greenned, this is expected behaviour for `to_tf_dataset`. By default, we drop the smaller 'remainder' batch during training (i.e. when `shuffle=True`). If you really want to keep that batch, you can set `drop_remainder=False` when calling `to_tf_dataset()`.

@Rocketknight1 Oh, thank you. I didn't get **drop_remainder** Have a nice day!

No problem!
Fix push_to_hub with null images

This code currently raises an error because of the null image:
```python
import datasets

dataset_dict = { 'name': ['image001.jpg', 'image002.jpg'], 'image': ['cat.jpg', None] }
features = datasets.Features({
    'name': datasets.Value('string'),
    'image': datasets.Image(),
})
dataset = datasets.Dataset.from_dict(dataset_dict, features)
dataset.push_to_hub("username/dataset") # this line produces an error: 'NoneType' object is not subscriptable
```
I fixed this in this PR

TODO:
- [x] add a test


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3856). All of your documentation changes will be reflected on that endpoint.
Redundant add dataset information and dead link.

> Alternatively, you can follow the steps to [add a dataset](https://huggingface.co/docs/datasets/add_dataset.html) and [share a dataset](https://huggingface.co/docs/datasets/share_dataset.html) in the documentation.

The "add a dataset link" gives 404 Error, and the share_dataset link has changed. I feel this information is redundant/deprecated now since we have a more detailed guide for "How to add a dataset?".


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3852). All of your documentation changes will be reflected on that endpoint.
Bring back imgs so that forsk dont get broken



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3866). All of your documentation changes will be reflected on that endpoint.

I think we just need to keep `datasets_logo_name.jpg` and `course_banner.png` because they appear in the README.md of the forks of `datasets`. The other images can be removed

Force pushed those two imgs only
Small doc fixes



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3860). All of your documentation changes will be reflected on that endpoint.

There are still some `.. code-block:: python` (e.g. see [this](https://huggingface.co/docs/datasets/master/en/package_reference/main_classes#datasets.Dataset.align_labels_with_mapping)) directives in our codebase, so maybe we can remove those as well as part of this PR.
Making the Hub the place for datasets in Portuguese 

Let's make Hugging Face Datasets the central hub for datasets in Portuguese :)

**Motivation**. Datasets are currently quite scattered and an open-source central point such as the Hugging Face Hub would be ideal to support the growth of the Portuguese speaking community.

What are some datasets in Portuguese worth integrating into the Hugging Face hub?

Special thanks to @augusnunes for his collaboration on identifying the first ones:

-  [NILC - USP](http://www.nilc.icmc.usp.br/nilc/index.php/tools-and-resources).

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

cc @osanseviero




Labels: name: dataset request

Hi @omarespejel! I think the philosophy for `datasets` issues is to create concrete issues with proposals to add a specific, individual dataset rather than umbrella issues for things such as datasets for a language, since we could end up with hundreds of issues (one per language). I see NILC - USP has many datasets, I would suggest to either create an issue for their datasets, or even better, we are trying to push to upload datasets as community datasets instead of adding them to the core library as guided in https://huggingface.co/docs/datasets/share. That would have the additional benefit that the dataset would live under the NILC organization.

@lhoestq correct me if I'm wrong please 😄
add pandas to env command

Pandas is a required packages and used quite a bit. I don't see any downside with adding its version to the `datasets-cli env` command.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3871). All of your documentation changes will be reflected on that endpoint.

Think failures are unrelated - feel free to merge whenever you want :-)
Ignore duplicate keys if `ignore_verifications=True`

Currently, it's impossible to generate a dataset if some keys from `_generate_examples` are duplicated. This PR allows skipping the check for duplicate keys if `ignore_verifications` is set to `True`.  


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3868). All of your documentation changes will be reflected on that endpoint.

Cool thanks ! Could you add a test please ?
Align metadata to DCAT/DCAT-AP

**Is your feature request related to a problem? Please describe.**
Align to DCAT metadata to describe datasets

**Describe the solution you'd like**
Reuse terms and structure from DCAT in the metadata file, ideally generate a json-ld file dcat compliant

**Describe alternatives you've considered**


**Additional context**
DCAT is a W3C standard extended in Europe with DCAT-AP, an example is data.europa.eu publishing datasets metadata in DCAT-AP


Labels: name: enhancement
Update code blocks

Following https://github.com/huggingface/datasets/pull/3860#issuecomment-1061756712 and https://github.com/huggingface/datasets/pull/3690 we need to update the code blocks to use markdown instead of sphinx


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3863). All of your documentation changes will be reflected on that endpoint.