helmo/github-issues huggingface dataset:
Making the Hub the place for datasets in Portuguese 
State: open | Author: omarespejel (NONE)
Labels: name: dataset request

Let's make Hugging Face Datasets the central hub for datasets in Portuguese :)

**Motivation**. Datasets are currently quite scattered and an open-source central point such as the Hugging Face Hub would be ideal to support the growth of the Portuguese speaking community.

What are some datasets in Portuguese worth integrating into the Hugging Face hub?

Special thanks to @augusnunes for his collaboration on identifying the first ones:

-  [NILC - USP](http://www.nilc.icmc.usp.br/nilc/index.php/tools-and-resources).

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

cc @osanseviero




Comments: Hi @omarespejel! I think the philosophy for `datasets` issues is to create concrete issues with proposals to add a specific, individual dataset rather than umbrella issues for things such as datasets for a language, since we could end up with hundreds of issues (one per language). I see NILC - USP has many datasets, I would suggest to either create an issue for their datasets, or even better, we are trying to push to upload datasets as community datasets instead of adding them to the core library as guided in https://huggingface.co/docs/datasets/share. That would have the additional benefit that the dataset would live under the NILC organization.

@lhoestq correct me if I'm wrong please 😄
Ignore duplicate keys if `ignore_verifications=True`
State: closed | Author: mariosasko (COLLABORATOR)

Currently, it's impossible to generate a dataset if some keys from `_generate_examples` are duplicated. This PR allows skipping the check for duplicate keys if `ignore_verifications` is set to `True`.  

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3868). All of your documentation changes will be reflected on that endpoint.

Cool thanks ! Could you add a test please ?
Update for the rename doc-builder -> hf-doc-utils
State: closed | Author: sgugger (CONTRIBUTOR)

This PR adapts the job to the upcoming change of name of `doc-builder`.

Comments: why utils? it's a builder no?

~~@julien-c there was a vote 🙂 https://huggingface.slack.com/archives/C021H1P1HKR/p1646405136644739~~

oh I see you already commeented in the thread as well

Thanks ! It looks all good to me (provided `hf-doc-utils` is the name we keep in the end). I'm fine with this name, and `hf-doc-builder` is also fine IMHO

ok, this is definitely not a hill I'll die on =) @mishig25 @sgugger
Bring back imgs so that forsk dont get broken
State: closed | Author: mishig25 (NONE)


Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3866). All of your documentation changes will be reflected on that endpoint.

I think we just need to keep `datasets_logo_name.jpg` and `course_banner.png` because they appear in the README.md of the forks of `datasets`. The other images can be removed

Force pushed those two imgs only
Update code blocks
State: closed | Author: lhoestq (MEMBER)

Following https://github.com/huggingface/datasets/pull/3860#issuecomment-1061756712 and https://github.com/huggingface/datasets/pull/3690 we need to update the code blocks to use markdown instead of sphinx

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3863). All of your documentation changes will be reflected on that endpoint.
Manipulate columns on IterableDataset (rename columns, cast, etc.)
State: closed | Author: lhoestq (MEMBER)

I added:
- add_column
- cast
- rename_column
- rename_columns

related to https://github.com/huggingface/datasets/issues/3444

TODO:
- [x] docs
- [x] tests

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3862). All of your documentation changes will be reflected on that endpoint.

> IIUC we check if columns are present/not present directly in the yielded examples and not in info.features because info.features can be None (after map, for instance)?

Yes exactly

> We should develop a solution that ensures info.features is never None. For example, one approach would be to infer them from examples in map and make them promotable from Value("null") to a specific type, in case of None values.

I agree this would be useful. Though inferring the type requires to start streaming some data, which takes a few seconds (compared to being instantaneous right now).

Let's discuss this in a new issue maybe ?
big_patent cased version
State: closed | Author: slvcsl (NONE)
Labels: name: dataset request

Hi! I am interested in working with the big_patent dataset.

In Tensorflow, there are a number of versions of the dataset:

- 1.0.0 : lower cased tokenized words
- 2.0.0 : Update to use cased raw strings
- 2.1.2 (default): Fix update to cased raw strings.

The version in the huggingface `datasets` library is the 1.0.0. I would be very interested in using the 2.1.2 cased version (used more, recently, for example in the Pegasus paper), but it does not seem to be supported (I tried using the `revision` parameter in `load_datasets`). Is there a way to already load it, or would it be possible to add that version? 

Comments: To follow up on this: the cased and uncased versions actually contain different content, and the cased one is easier since it contains a Summary of the Invention in the input.

See the paper describing the issue here:
https://aclanthology.org/2022.gem-1.34/

Thanks for proposing the addition of the cased version of this dataset and for pinging again recently.

I have just merged a PR that adds the cased version: https://huggingface.co/datasets/big_patent/discussions/3

The cased version (2.1.2) is the default one:
```python
ds = load_dataset("big_patent", "all")
```

To use the 1.0.0 version (lower cased tokenized words), pass both parameters `codes` and `version`:
```python
ds = load_dataset("big_patent", codes="all", version="1.0.0")
```

Closed by: https://huggingface.co/datasets/big_patent/discussions/3
Small doc fixes
State: closed | Author: mishig25 (NONE)


Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3860). All of your documentation changes will be reflected on that endpoint.

There are still some `.. code-block:: python` (e.g. see [this](https://huggingface.co/docs/datasets/master/en/package_reference/main_classes#datasets.Dataset.align_labels_with_mapping)) directives in our codebase, so maybe we can remove those as well as part of this PR.
Order of dataset changes due to glob.glob.
State: open | Author: patrickvonplaten (CONTRIBUTOR)
Labels: name: generic discussion

## Describe the bug

After discussion with @lhoestq, just want to mention here that `glob.glob(...)` should always be used in combination with `sorted(...)` to make sure the list of files returned by `glob.glob(...)` doesn't change depending on the OS system. 

There are currently multiple datasets that use `glob.glob()` without making use of `sorted(...)` even the streaming download manager (if I'm not mistaken): 
https://github.com/huggingface/datasets/blob/c14bfeb4af89da14f870de5ddaa584b08aa08eeb/src/datasets/utils/streaming_download_manager.py#L483

Comments: I agree using `glob.glob` alone is bad practice because it's not deterministic. Using `sorted` is a nice solution.

Note that the `xglob` function you are referring to in the `streaming_download_manager.py` code just extends `glob.glob` for URLs - we don't change its behavior. That's why it has no `sorted()`
Fix push_to_hub with null images
State: closed | Author: lhoestq (MEMBER)

This code currently raises an error because of the null image:
```python
import datasets

dataset_dict = { 'name': ['image001.jpg', 'image002.jpg'], 'image': ['cat.jpg', None] }
features = datasets.Features({
    'name': datasets.Value('string'),
    'image': datasets.Image(),
})
dataset = datasets.Dataset.from_dict(dataset_dict, features)
dataset.push_to_hub("username/dataset") # this line produces an error: 'NoneType' object is not subscriptable
```
I fixed this in this PR

TODO:
- [x] add a test

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3856). All of your documentation changes will be reflected on that endpoint.
Bad error message when loading private dataset
State: closed | Author: patrickvonplaten (CONTRIBUTOR)
Labels: name: bug

## Describe the bug

A pretty common behavior of an interaction between the Hub and datasets is the following.
An organization adds a dataset in private mode and wants to load it afterward.


```python
from transformers import load_dataset

ds = load_dataset("NewT5/dummy_data", "dummy")
```

This command then fails with:

```bash
FileNotFoundError: Couldn't find a dataset script at /home/patrick/NewT5/dummy_data/dummy_data.py or any data file in the same directory. Couldn't find 'NewT5/dummy_data' on the Hugging Face Hub either: FileNotFoundError: Dataset 'NewT5/dummy_data' doesn't exist on the Hub
```

**even though** the user has access to the website `NewT5/dummy_data` since she/he is part of the org. 

We need to improve the error message here similar to how @sgugger, @LysandreJik and @julien-c have done it for transformers IMO.

## Steps to reproduce the bug

E.g. execute the following code to see the different error messages between `transformes` and `datasets`.

1. Transformers
```python
from transformers import BertModel

BertModel.from_pretrained("NewT5/dummy_model")
```

The error message is clearer here - it gives:

```
OSError: patrickvonplaten/gpt2-xl is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo with `use_auth_token` or log in with `huggingface-cli login` and pass `use_auth_token=True`.
```

Let's maybe do the same for datasets? The PR was introduced to `transformers` here:
https://github.com/huggingface/transformers/pull/15261

## Expected results

Better error message

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.4.dev0
- Platform: Linux-5.15.15-76051515-generic-x86_64-with-glibc2.34
- Python version: 3.9.7
- PyArrow version: 6.0.1



Comments: We raise the error “ FileNotFoundError: can’t find the dataset” mainly to follow best practice in security (otherwise users could be able to guess what private repositories users/orgs may have)

We can indeed reformulate this and add the "If this is a private repository,..." part !

Resolved via https://github.com/huggingface/datasets/pull/4536
load only England English dataset from common voice english dataset
State: closed | Author: amanjaiswal777 (NONE)
Labels: name: question

training_data = load_dataset("common_voice", "en",split='train[:250]+validation[:250]')
testing_data = load_dataset("common_voice", "en", split="test[:200]")

I'm trying to load only 8% of the English common voice data with accent == "England English." Can somebody assist me with this?

**Typical Voice Accent Proportions:**

- 24% United States English 
- 8% England English 
- 5% India and South Asia (India, Pakistan, Sri Lanka) 
- 3% Australian English 
- 3% Canadian English 
- 2% Scottish English 
- 1% Irish English 
- 1% Southern African (South Africa, Zimbabwe, Namibia) 
- 1% New Zealand English

Can we replicate this for Age as well?

**Age proportions of the common voice:-**

- 24% 19 - 29 
- 14% 30 - 39 
- 10% 40 - 49 
- 6% < 19 
- 4% 50 - 59 
- 4% 60 - 69 
- 1% 70 – 79 

Comments: Hi @amanjaiswal777,

First note that the dataset you are trying to load is deprecated: it was the Common Voice dataset release as of Dec 2020.

Currently, Common Voice dataset releases are directly hosted on the Hub, under the Mozilla Foundation organization: https://huggingface.co/mozilla-foundation

For example, to get their latest Common Voice relase (8.0):
- Go to the dataset page and request access permission (Mozilla Foundation requires this for people willing to use their datasets): https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0
- Looking at the dataset card, you can check that data instances have, among other fields, the ones you are interested in: "accent", "age",... 
- Then you can load their "en" language dataset as usual, besides passing your authentication token (more info on auth token here: https://huggingface.co/docs/hub/security)
  ```python
  from datasets import load_dataset
  ds_en = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token=True)
  ```
- Finally, you can filter only the data instances you are interested in (more info on `filter` here: https://huggingface.co/docs/datasets/process#select-and-filter):
  ```python
  ds_england_en = ds_en.filter(lambda item: item["accent"] == "England English")
  ```

Feel free to reopen this issue if you need further assistance.

Hey @albertvillanova trying the same approach as you with the common_voice_16_1 dataset. What I'm trying to do is to filter the valencian accent in the catalan subset. Gave me this error and I have everything it asks for decoding mp3:
![image](https://github.com/huggingface/datasets/assets/96977715/7ec02483-e728-4358-9372-ba74ec1b7fd4)

![image](https://github.com/huggingface/datasets/assets/96977715/c10fcf23-a141-4dba-a88d-89e293acfe67)
add ontonotes_conll dataset
State: closed | Author: richarddwang (CONTRIBUTOR)

# Introduction of the dataset
OntoNotes v5.0 is the final version of OntoNotes corpus, and is a large-scale, multi-genre,
multilingual corpus manually annotated with syntactic, semantic and discourse information.

This dataset is the version of OntoNotes v5.0 extended and used in the CoNLL-2012 shared task
, includes v4 train/dev and v9 test data for English/Chinese/Arabic and corrected version v12 train/dev/test data (English only).

This dataset is widely used in name entity recognition, coreference resolution, and semantic role labeling.

In dataset loading script, I modify and use the code of [AllenNLP/Ontonotes](https://docs.allennlp.org/models/main/models/common/ontonotes/#ontonotes) to read the special conll files while don't get extra package dependency. 

# Some workarounds I did

1. task ids
I add tasks that I can't find anywhere `semantic-role-labeling`, `lemmatization`, and `word-sense-disambiguation` to the task category `structure-prediction`, because they are related to "syntax". I feel there is another good name for the task category since some tasks mentioned aren't related to structure, but I have no good idea.

2. `dl_manage.extract`
Since we'll get another zip after unzip the downloaded zip data, I have to use `dl_manager.extract` directly inside `_generate_examples`. But when testing dummy data, `dl_manager.extract` do nothing. So I make a conditional such that it  manually  extract data when testing dummy data.

# Help
Don't know how to fix the building doc error.

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3853). All of your documentation changes will be reflected on that endpoint.

The CI fail is unrelated to this dataset, merging :)
Redundant add dataset information and dead link.
State: closed | Author: dnaveenr (CONTRIBUTOR)

> Alternatively, you can follow the steps to [add a dataset](https://huggingface.co/docs/datasets/add_dataset.html) and [share a dataset](https://huggingface.co/docs/datasets/share_dataset.html) in the documentation.

The "add a dataset link" gives 404 Error, and the share_dataset link has changed. I feel this information is redundant/deprecated now since we have a more detailed guide for "How to add a dataset?".

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3852). All of your documentation changes will be reflected on that endpoint.
Add "Adversarial GLUE" dataset to datasets library
State: closed | Author: jxmorris12 (CONTRIBUTOR)

Adds the Adversarial GLUE dataset: https://adversarialglue.github.io/

```python
>>> import datasets
>>> >>> datasets.load_dataset('adv_glue')
Using the latest cached version of the module from /home/jxm3/.cache/huggingface/modules/datasets_modules/datasets/adv_glue/26709a83facad2830d72d4419dd179c0be092f4ad3303ad0ebe815d0cdba5cb4 (last modified on Mon Mar  7 19:19:48 2022) since it couldn't be found locally at adv_glue., or remotely on the Hugging Face Hub.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jxm3/random/datasets/src/datasets/load.py", line 1657, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/jxm3/random/datasets/src/datasets/load.py", line 1510, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/home/jxm3/random/datasets/src/datasets/builder.py", line 1021, in __init__
    super().__init__(*args, **kwargs)
  File "/home/jxm3/random/datasets/src/datasets/builder.py", line 258, in __init__
    self.config, self.config_id = self._create_builder_config(
  File "/home/jxm3/random/datasets/src/datasets/builder.py", line 337, in _create_builder_config
    raise ValueError(
ValueError: Config name is missing.
Please pick one among the available configs: ['adv_sst2', 'adv_qqp', 'adv_mnli', 'adv_mnli_mismatched', 'adv_qnli', 'adv_rte']
Example of usage:
	`load_dataset('adv_glue', 'adv_sst2')`
>>> datasets.load_dataset('adv_glue', 'adv_sst2')['validation'][0]
Reusing dataset adv_glue (/home/jxm3/.cache/huggingface/datasets/adv_glue/adv_sst2/1.0.0/3719a903f606f2c96654d87b421bc01114c37084057cdccae65cd7bc24b10933)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 604.11it/s]
{'sentence': "it 's an uneven treat that bores fun at the democratic exercise while also examining its significance for those who take part .", 'label': 1, 'idx': 0}
```

Comments: _The documentation is not available anymore as the PR was closed or merged._

@lhoestq can you review when you have some time?

Hi @lhoestq -- thanks so much for your review! I just added the stuff you requested to the README.md, including an example from the dataset, the table of contents, and lots of section headers with "More Information Needed" below. Let me know if there's anything else I need to do!

Feel free to also merge `master` into your branch to get the latest updates for the tests ;)

thanks @lhoestq - just made all the updates you requested!
Update faiss device docstring
State: closed | Author: lhoestq (MEMBER)

Following https://github.com/huggingface/datasets/pull/3721 I updated the docstring of the `device` argument of the FAISS related methods of `Dataset`

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3846). All of your documentation changes will be reflected on that endpoint.
add RMSE and MAE metrics.
State: closed | Author: dnaveenr (CONTRIBUTOR)

This PR adds RMSE - Root Mean Squared Error and MAE - Mean Absolute Error to the metrics API.
Both implementations are based on usage of sciket-learn.

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)

Please suggest any changes if required. Thank you.

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3845). All of your documentation changes will be reflected on that endpoint.

@mariosasko I've reopened it here. Please suggest any changes if required. Thank you.

Thanks for suggestions. :) I have added update the KWARGS_DESCRIPTION for the missing params and also changed RMSE to MSE.
While testing, I noticed that when the input is a list of lists, we get an error :
`TypeError: float() argument must be a string or a number, not 'list'`
Could you suggest the datasets.Value() attribute to support both list of floats and list of lists containing floats ?


Just add a new config to cover that case. You can do this by replacing the current `features` dict with:
```python
features=datasets.Features(
    {
        "predictions": datasets.Sequence(datasets.Value("float")),
        "references": datasets.Sequence(datasets.Value("float")),
    }
    if self.config_name == "multioutput"
    else {
        "predictions": datasets.Value("float"),
        "references": datasets.Value("float"),
     }
),
```
Feel free to suggest a better name for the config than `multioutput`

Also, could you please move the changes to a new branch and open a PR from there (for the 3rd time 😄) because the diff shows changes from unrelated PRs (maybe due to rebasing?).

Thanks for the input, I have added new config to support multi-dimensional lists and updated the examples as well.

Sure. Will do that and open a new PR for these changes.
Add rmse and mae metrics.
State: closed | Author: dnaveenr (CONTRIBUTOR)

This PR adds RMSE - Root Mean Squared Error and MAE - Mean Absolute Error to the metrics API.
Both implementations are based on usage of sciket-learn.

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)

Any suggestions and changes required will be helpful.


Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3844). All of your documentation changes will be reflected on that endpoint.

@dnaveenr This PR is in pretty good shape, so feel free to reopen it.
Fix Google Drive URL to avoid Virus scan warning in streaming mode
State: closed | Author: mariosasko (COLLABORATOR)

The streaming version of https://github.com/huggingface/datasets/pull/3787.

Fix #3835 

CC: @albertvillanova 

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3843). All of your documentation changes will be reflected on that endpoint.

Cool ! Looks like it breaks `test_streaming_gg_drive_gzipped` for some reason...
Align IterableDataset.shuffle with Dataset.shuffle
State: closed | Author: lhoestq (MEMBER)

From #3444 , Dataset.shuffle can have the same API than IterableDataset.shuffle (i.e. in streaming mode).

Currently you can pass an optional seed to both if you want, BUT currently IterableDataset.shuffle always requires a buffer_size, used for approximate shuffling. I propose using a reasonable default value (maybe 1000) instead.

In this PR, I set the default `buffer_size` value to 1,000, and I reorder the `IterableDataset.shuffle` arguments to match `Dataset.shuffle`, i.e. making `seed` the first argument.

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3842). All of your documentation changes will be reflected on that endpoint.

We should also add `generator` as a param to `shuffle` to fully align the APIs, no?

I added the `generator` argument.

I had to make a few other adjustments to make it work. In particular when you call `set_epoch()` on a streaming dataset, it updates the underlying random generator by using a new effective seed. The effective seed is generated using the previous generator and the epoch number.
Pin responses to fix CI for Windows
State: closed | Author: albertvillanova (MEMBER)

Temporarily fix CI for Windows by pinning `responses`. 

See: https://app.circleci.com/pipelines/github/huggingface/datasets/10292/workflows/83de4a55-bff7-43ec-96f7-0c335af5c050/jobs/63355

Fix: #3839

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3840). All of your documentation changes will be reflected on that endpoint.