Minimartzz/github-issues huggingface dataset:
[docs] make dummy data creation optional
State: closed

Related to #3507 : dummy data for datasets created on the Hugging Face Hub are optional.

We can discuss later to make them optional for datasets in this repository as well

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3894). All of your documentation changes will be reflected on that endpoint.

The dev doc build rendering doesn't seem to be updated with my last commit for some reason

Merging it anyway since I'd like to share this page with users 🙃
Add default branch for doc building
State: closed

Since other libraries use `main` as their default branch and it's now the standard default, you have to specify a different name in the doc config if you're using `master` like datasets (`doc-builder` tries to guess it, but in the job, we have weird checkout of merge commits so it doesn't always manage to get it right).

This PR makes sure it will always use master for the dev doc (until you decide to switchto main)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3893). All of your documentation changes will be reflected on that endpoint.

Yes! And when we discovered on the Transformers side that this check fails on the GitHub actions, we added a config attribute to have a default. Setting in Transformers fixed the issue of the doc being deployed to main, so porting the fix here too :-)
Fix CLI test checksums
State: closed

Previous PR:
- #3796 

introduced a side effect: `datasets-cli test` generates `dataset_infos.json` with `None` checksum values.

See:
- #3805 

This PR introduces a way for `datasets-cli test` to force to record infos, even if `verify_infos=False`

Close #3848.

CC: @craffel 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3892). All of your documentation changes will be reflected on that endpoint.

Feel free to merge if it's good for you :)

I've added a test @lhoestq. Once all green, I'll merge. 

Last failing tests do not have nothing to do with this PR.
Fix race condition in doc build
State: closed

Following https://github.com/huggingface/datasets/runs/5499386744 it seems that race conditions that create issues when updating the doc. I took the same approach as in `transformers` to fix race conditions

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3891). All of your documentation changes will be reflected on that endpoint.
Update beans download urls
State: closed

Replace the old URLs with the Hub [URLs](https://huggingface.co/datasets/beans/tree/main/data).

Also reported by @stevhliu. 

Fix #3889 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3890). All of your documentation changes will be reflected on that endpoint.

@albertvillanova Thanks for investigating and fixing that issue. I regenerated the `dataset_infos.json` file.
Cannot load beans dataset (Couldn't reach the dataset)
State: closed
Labels: description: A bug in a dataset script provided in the library
name: dataset bug

## Describe the bug
The beans dataset is unavailable to download.

## Steps to reproduce the bug
```python
from datasets import load_dataset
ds = load_dataset('beans')
```

## Expected results
The dataset would be downloaded with no issue.

## Actual results

```
ConnectionError: Couldn't reach https://storage.googleapis.com/ibeans/train.zip (error 403)
```

[It looks like the billing of this project has been disabled because it is associated with a delinquent account.](https://storage.googleapis.com/ibeans/train.zip )

## Environment info
Google Colab


Comments:
Hi ! A pull request is open to fix the dataset, we'll release a patch soon with a new release of `datasets` :)
IterableDataset columns and feature types
State: open
Labels: description: Generic discussion on the library
name: generic discussion

name: streaming

Right now, an IterableDataset (e.g. when streaming a dataset) doesn't require to know the list of columns it contains, nor their types: `my_iterable_dataset.features` may be `None`

However it's often interesting to know the column types and types. This helps knowing what's inside your dataset without having to manually check a few examples, and this is useful to prepare a processing pipeline or to train models.

Here are a few cases that lead to `features` being `None`:

1. when loading a dataset with `load_dataset` on CSV, JSON Lines, etc. files: type inference is only done when iterating over the dataset
2. when calling `map`, because we don't know in advance what's the output of the user's function passed to `map`
3. when calling `rename_columns`, `remove_columns`, etc. because they rely on `map`

Things we can consider, for each point above:

1.a infer the type automatically from the first samples on the dataset using prefetching, when the dataset builder doesn't provide the `features`
2.a allow the user to specify the `features` as an argument to `map` (this would be consistent with the non-streaming API)
2.b prefetch the first output value to infer the type
3.a don't rely on `map` directly and reuse the previous `features` and rename/remove the corresponding ones

The thing is that prefetching can take a few seconds, while the operations above are instantaneous since no data are downloaded. Therefore I'm not sure whether this solution may be worth it. Maybe prefetching could also be done when explicitly asked by the user

cc @mariosasko @albertvillanova 

Comments:
#self-assign

@alvarobartt I've assigned you the issue since I'm not actively working on it.

Cool thanks @mariosasko I'll try to fix it in the upcoming days, thanks!

@lhoestq so in order to address what’s not completed in this issue, do you think it makes sense to add a param `features` to `IterableDataset.map` so that the output features right after the `map` are defined there? 

Yes that would be ideal IMO, thanks again for the help :)

@lhoestq cool then if you agree I can work on that! I’ll also update the docs accordingly once done, thanks!

I've already started with a PR as a draft @lhoestq, should we also try to look for a way to explicitly request pre-fetching right after a map operation is applied, so that the features are inferred if the user says explicitly so? Thanks!

> should we also try to look for a way to explicitly request pre-fetching right after a map operation is applied, so that the features are inferred if the user says explicitly so?

Right now one can use `ds = ds._resolve_features()` do to so. It can be used after `map` or `load_dataset` if the features are not known. Maybe we can make this method public ?
ImageFolder improvements
State: closed

This PR adds the following improvements to the `imagefolder` dataset:
* skip the extract step for image files (as discussed in https://github.com/huggingface/datasets/pull/2830#discussion_r816817919)
* option to drop labels by setting `drop_labels=True` (useful for image pretraining cc @NielsRogge). This is faster than loading a dataset and removing the `label` column because we don't need to iterate over the files to infer class labels.  

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3887). All of your documentation changes will be reflected on that endpoint.
Retry HfApi call inside push_to_hub when 504 error
State: closed

Ass suggested by @lhoestq in #3872, this PR:
- Implements a retry function
- Retries HfApi call inside `push_to_hub` when 504 error. To be agreed:
  - max_retries = 2 (at 0.5 and 1 seconds)

Fix #3872.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3886). All of your documentation changes will be reflected on that endpoint.

I made it more robust by increasing the wait time, and I also added some logs when a request is retried. Let me know if it's ok for you

At the end you did not set the agreed max value of 60s. 

Moreover, with the new numbers, there is a slight contradiction: although you set max_retries=5, we will only make 4 retries at most because of the combined values of `base_wait_time` and `max_wait_time`.

Yea I thought that in total we could wait 1min, but if we have a max_wait_time of 20sec between each request it's fine IMO

> Moreover, with the new numbers, there is a slight contradiction: although you set max_retries=5, we will only make 4 retries at most because of the combined values of base_wait_time and max_wait_time.

What makes you think this ? If the exponential wait time becomes bigger than `max_wait_time` then it still does the retry, but after a wait time of `max_wait_time`

Sorry, I meant 4 retries **with exponential backoff**; the fifth one is with constant backoff.

OK, and one question: do you think that the retries do not affect the time the server needs to be operational again and able to process the request? I guess that if does not affect, then the cause are other users' requests, or others; not our specific request.

Just to be sure: 
- Then 20s at most between consecutive requests do not impact the server.
- And we expect after a total of 5 retries (within a total 50s of wait time + request processing/uploading time), the server should be able to come back to normality.

> do you think that the retries do not affect the time the server needs for being able to process the request (I guess in this case the cause are other users' requests, or other causes; not our specific request).

Yes I don't think the retries would affect the server, I think the cause of the 504 errors is elsewhere

> Just to be sure:
>
> Then 20s at most between consecutive requests do not impact the server.
> And we expect after a total of 5 retries (within a total 50s of wait time + request processing/uploading time), the server should be able to come back to normality.

Yes I think it's fine for now, we can still adapt this later if needed

Will be curious to see the impact of this in terms of upload reliability! Don't forget to let us know when you have more data. cc @huggingface/moon-landing-back
The metric Meteor doesn't work for nltk ==3.6.4
State: closed
Labels: description: Something isn't working
name: bug

## Describe the bug
Using the metric Meteor with nltk == 3.6.4 gives a TypeError:
TypeError: descriptor 'lower' for 'str' objects doesn't apply to a 'list' object

## Steps to reproduce the bug
```python
import datasets
metric = datasets.load_metric("meteor")
predictions = ["hello world"]
references = ["hello world"]
metric.compute(predictions=predictions, references=references)
```

## Expected results
TypeError: descriptor 'lower' for 'str' objects doesn't apply to a 'list' object

I think this TypeError exists because input sentences are tokenized into lists of tokens and the str.lower() is applied to this list of tokens.

## Actual results
No error but a meteor score

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: linux
- Python version: 3.8.12
- PyArrow version: 7.0.0


Comments:
Hi @zhaowei-wang98, thanks for reporting.

We are fixing it...
Image process doc
State: closed
Labels: description: Improvements or additions to documentation
name: documentation

This PR is a first draft of how to process image data. It adds:

- Load an image dataset with `image` and `path` (adds tip about `decode=False` param to access the path and bytes, thanks to @mariosasko).
- Load an image using the `ImageFolder` builder. I know there is an [example](https://huggingface.co/docs/datasets/master/en/loading#image-folders) of this already, but I also wanted to add it here so users don't miss it. This doc seems important for centralizing all of the image-related things so far. Datasets has grown so quickly 🚀  now that I think maybe splitting up the How-to guides by modality may be better since working with vision/audio data is slightly different from what users have seen up until now. This way we can continue to scale the docs to better accommodate vision/audio things.
- Add a data augmentation with `set_transform`. There is only 1 example here so far, but we can certainly add more. 

Todo:
- [x] Couldn't figure out why my augmentation function works with `set_transform` but not `map` 🥲. Working with @mariosasko on this!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3882). All of your documentation changes will be reflected on that endpoint.
Change the framework switches to the new syntax
State: closed

This PR updates the syntax of the framework-specific code samples. With this new syntax, you'll be able to:

- have paragraphs of text be framework-specific instead of just code samples
- have support for Flax code samples if you want.

This should be merged after https://github.com/huggingface/doc-builder/pull/63 and https://github.com/huggingface/doc-builder/pull/130

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3880). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3880). All of your documentation changes will be reflected on that endpoint.
Update cats_vs_dogs size
State: closed

It seems like 12 new examples have been added to the `cats_vs_dogs`. This PR updates the size in the card and the info file to avoid a verification error (reported by @stevhliu). 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3878). All of your documentation changes will be reflected on that endpoint.

Maybe `NonMatchingSplitsSizesError` errors should also tell the user to try using a more recent version of the dataset to get the fixes ?

@lhoestq Good idea. Will open a new PR to improve the error messages of NonMatchingSplitsSizesError, NonMatchingChecksumsError, ...

It seems there is still a problem. I am using datasets version 2.5.1. 
I just typed `ds = load_dataset("cats_vs_dogs")` and get the error below.

```
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=3893603, num_examples=23422, dataset_name='cats_vs_dogs'), 'recorded': SplitInfo(name='train', num_bytes=3891612, num_examples=23410, dataset_name='cats_vs_dogs')}]
```
It looks like the dataset still only has 23,410 examples....


Thanks for reporting, I opened https://github.com/huggingface/datasets/pull/5047
Align metadata to DCAT/DCAT-AP
State: open
Labels: description: New feature or request
name: enhancement

**Is your feature request related to a problem? Please describe.**
Align to DCAT metadata to describe datasets

**Describe the solution you'd like**
Reuse terms and structure from DCAT in the metadata file, ideally generate a json-ld file dcat compliant

**Describe alternatives you've considered**


**Additional context**
DCAT is a W3C standard extended in Europe with DCAT-AP, an example is data.europa.eu publishing datasets metadata in DCAT-AP


Comments:
Fix download_mode in dataset_module_factory
State: closed

Fix `download_mode` value set in `dataset_module_factory`.

Before the fix, it was set to `bool` (default to `False`).

Also set properly its default value in all public functions.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3876). All of your documentation changes will be reflected on that endpoint.
Module namespace cleanup for v2.0
State: closed

This is an attempt to make the user-facing `datasets`' submodule namespace cleaner:

In particular, this PR does the following:
* removes the unused `zip_nested` and `flatten_nest_dict` and their accompanying tests
* removes `pyarrow` from the top-level namespace
* properly uses `__all__` and the `from  <module> import *` syntax to avoid importing the `<module>`'s submodules
* cleans up the `utils` namespace
* moves the `temp_seed` context manage from `datasets/utils/file_utils.py` to `datasets/utils/py_utils.py`

Comments:
will it solve https://github.com/huggingface/datasets-preview-backend/blob/4c542a74244045929615640ccbba5a902c344c5a/pyproject.toml#L85-L89?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3875). All of your documentation changes will be reflected on that endpoint.

@severo No, this PR doesn't fix that issue in the current state. We can fix it by adding `__all__` to `datasets/__init__.py` and `datasets/formatting/__init__.py`. However, this would require updating `__all__` for each new function/class definition, which could become cumbersome, and we can't do this dynamically because `mypy` is a static type checker.

@lhoestq @albertvillanova WDYT?

Feel free to merge this one if it's good for you :)
add MSE and MAE metrics - V2
State: closed

Created a new pull request to resolve unrelated changes in PR caused due to rebasing.
Ref Older PR :  [#3845](https://github.com/huggingface/datasets/pull/3845) 

Feature request here : Add support for continuous metrics (RMSE, MAE) [#3608](https://github.com/huggingface/datasets/issues/3608)

Comments:
@mariosasko New PR here. I'm not sure how to add you as a co-author here.  Also I see flake8 tests are failing, any inputs on how to resolve this ?
Also, let me know if any other changes are required. Thank you.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3874). All of your documentation changes will be reflected on that endpoint.

Great. Thank you.

Thanks so much for this 🙏 💯
Create SQuAD metric README.md
State: closed

Proposal for a metrics card structure (with an example based on the SQuAD metric).
@thomwolf @lhoestq @douwekiela @lewtun -- feel free to comment on structure or content (it's an initial draft, so I realize there's stuff missing!).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3873). All of your documentation changes will be reflected on that endpoint.

Oh one last thing I almost forgot, I think I would add a section "Examples" with examples of inputs and outputs and in particular: an example giving maximal values, an examples giving minimal values and maybe a standard examples from SQuAD. What do you think?
HTTP error 504 Server Error: Gateway Time-out
State: closed

I am trying to push a large dataset(450000+) records with the help of `push_to_hub()`
While pushing, it gives some error like this.

```
Traceback (most recent call last):
  File "data_split_speech.py", line 159, in <module>
    data_new_2.push_to_hub("user-name/dataset-name",private=True)
  File "/opt/conda/lib/python3.8/site-packages/datasets/dataset_dict.py", line 951, in push_to_hub
    repo_id, split, uploaded_size, dataset_nbytes = self[split]._push_parquet_shards_to_hub(
  File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3556, in _push_parquet_shards_to_hub
    api.upload_file(
  File "/opt/conda/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 1017, in upload_file
    raise err
  File "/opt/conda/lib/python3.8/site-packages/huggingface_hub/hf_api.py", line 1008, in upload_file
    r.raise_for_status()
  File "/opt/conda/lib/python3.8/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/api/datasets/user-name/dataset-name/upload/main/data/train2-00041-of-00064.parquet

```
Can anyone help me to resolve this issue.


Comments:
is pushing directly with git (and git-lfs) an option for you?

I have installed git-lfs and doing this push with that


yes but is there any way you could try pushing with `git` command line directly instead of `push_to_hub`?

Okay. I didnt saved the dataset to my local machine. So, I processed the dataset and pushed it directly to the hub. I think I should try saving those dataset to my local machine by `save_to_disk` and then push it with git command line

cc @lhoestq @albertvillanova @LysandreJik because maybe I'm giving dumb advice here 😅 

`push_to_hub` is the preferred way of uploading a dataset to the Hub, which can then be reloaded with `load_dataset`. Feel free to try again and see if the server is working as expected now. Maybe we can add a retry mechanism in the meantime to workaround 504 errors.

Regarding `save_to_disk`, this must only be used for local serialization (because it's uncompressed and compatible with memory-mapping). If you upload a dataset saved with `save_to_disk` to the Hub, then to reload it you will have to download/clone the repository locally by yourself and use `load_from_disk`.
add pandas to env command
State: closed

Pandas is a required packages and used quite a bit. I don't see any downside with adding its version to the `datasets-cli env` command.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3871). All of your documentation changes will be reflected on that endpoint.

Think failures are unrelated - feel free to merge whenever you want :-)
Add wikitablequestions dataset
State: closed


Comments:
@lhoestq Would you mind reviewing it when you're available? Thanks!


> Awesome thanks for adding this dataset ! :) The dataset script and dataset cards look pretty good
> 
> It looks like your `dummy_data.zip` files are quite big though (>1MB each), do you think we can reduce their sizes ? This way this git repository doesn't become too big

I have manually reduced the `dummy_data.zip` and its current size is about 54KB. Hope it is fine for you!

@lhoestq I think the dataset is ready to merge now. Any follow-up question is welcome :-D

> Thanks ! It looks all good now :)

Awesome! Thanks for your quick response!
Making the Hub the place for datasets in Portuguese 
State: open
Labels: description: Requesting to add a new dataset
name: dataset request

Let's make Hugging Face Datasets the central hub for datasets in Portuguese :)

**Motivation**. Datasets are currently quite scattered and an open-source central point such as the Hugging Face Hub would be ideal to support the growth of the Portuguese speaking community.

What are some datasets in Portuguese worth integrating into the Hugging Face hub?

Special thanks to @augusnunes for his collaboration on identifying the first ones:

-  [NILC - USP](http://www.nilc.icmc.usp.br/nilc/index.php/tools-and-resources).

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

cc @osanseviero




Comments:
Hi @omarespejel! I think the philosophy for `datasets` issues is to create concrete issues with proposals to add a specific, individual dataset rather than umbrella issues for things such as datasets for a language, since we could end up with hundreds of issues (one per language). I see NILC - USP has many datasets, I would suggest to either create an issue for their datasets, or even better, we are trying to push to upload datasets as community datasets instead of adding them to the core library as guided in https://huggingface.co/docs/datasets/share. That would have the additional benefit that the dataset would live under the NILC organization.

@lhoestq correct me if I'm wrong please 😄
Ignore duplicate keys if `ignore_verifications=True`
State: closed

Currently, it's impossible to generate a dataset if some keys from `_generate_examples` are duplicated. This PR allows skipping the check for duplicate keys if `ignore_verifications` is set to `True`.  

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3868). All of your documentation changes will be reflected on that endpoint.

Cool thanks ! Could you add a test please ?