noamaanMulla-03/datasets-issues huggingface dataset:
Consistent metric outputs

To automate the use of metrics, they should return consistent outputs.
In particular I'm working on adding a conversion of metrics to keras metrics.
To achieve this we need two things:
- have each metric return dictionaries of string -> floats since each keras metrics should return one float
- define in the metric info the different fields of the output dictionary

In this PR I'm adding these two features.
I also fixed a few bugs in some metrics

#867 needs to be merged first

Labels: name: transfer-to-evaluate

I keep this PR in stand-by for next week's datasets sprint. If the next release is 2.0.0 then we can include it given that it's breaking for many metrics

Metrics are deprecated in `datasets` and `evaluate` should be used instead: https://github.com/huggingface/evaluate
Adding Enriched WebNLG dataset

This pull requests adds the `en` and `de` versions of the [Enriched WebNLG](https://github.com/ThiagoCF05/webnlg) dataset


Nice :) 

could you add the tags and also remove all the dummy data files that are not zipped ? The diff currently shows 800 files changes xD

Aaaaand it's rebase time - the new one is at #1264 !

closing this one since a new PR was created
Add Tweet Eval Dataset



Hi @lhoestq,

Seeing that it has been almost two months to this draft, I'm willing to take this forward if you and @abhishekkrthakur don't mind. :)

Hi @gchhablani !
Sure if @abhishekkrthakur doesn't mind
Thanks for your help :)

Please feel free :) 

Hi @lhoestq, @abhishekkrthakur 

I believe this can be closed. Merged in #1829.
Add GLUE Compat (compatible with transformers<3.5.0)

Link to our discussion on Slack (HF internal)
https://huggingface.slack.com/archives/C014N4749J9/p1609668119337400

The next step is to add a compatible option in the new `run_glue.py`

I duplicated `glue` and made the following changes:
1. Change the name to `glue_compat`.
2. Change the label assignments for MNLI and AX.


Maybe it would be simpler to just overwrite the order of the label classes of the `glue` dataset ?
```python
mnli = load_dataset("glue", "mnli", label_classes=["contradiction", "entailment", "neutral"])
```

Sounds good. Will close the issue if that works.
Add support for multiple dynamic dimensions and to_pandas conversion for dynamic arrays

Add support for multiple dynamic dimensions (e.g. `(None, None, 3)` for arbitrary sized images) and `to_pandas()` conversion for dynamic arrays.

TODOs:
* [ ] Cleaner code
* [ ] Formatting issues (if NumPy doesn't allow broadcasting even though dtype is np.object)
* [ ] Fix some issues with zero-dim tensors 
* [ ] Tests
Fix push to hub to allow individual split push

# Description of the issue

If one decides to push a split on a datasets repo, he uploads the dataset and overrides the config. However previous config splits end up being lost despite still having the dataset necessary.

The new flow is the following:
 - query the old config from the repo
 - update into a new config (add/overwrite new split for example)
 - push the new config

# Side fix

 - `repo_id` in HfFileSystem was wrongly typed.
 - I've added `indent=2` as it becomes much easier to read now.




This has been addressed in https://github.com/huggingface/datasets/pull/4415. Closing.
Update for the rename doc-builder -> hf-doc-utils

This PR adapts the job to the upcoming change of name of `doc-builder`.


why utils? it's a builder no?

~~@julien-c there was a vote 🙂 https://huggingface.slack.com/archives/C021H1P1HKR/p1646405136644739~~

oh I see you already commeented in the thread as well

Thanks ! It looks all good to me (provided `hf-doc-utils` is the name we keep in the end). I'm fine with this name, and `hf-doc-builder` is also fine IMHO

ok, this is definitely not a hill I'll die on =) @mishig25 @sgugger
Deterministic split order in DatasetDict.map

The order in which the splits are processed by `map` is not deterministic in `DatasetDict.map`. This can cause caching issues when the processing function is stateful and sensible to the order in which examples are processed

Close https://github.com/huggingface/datasets/issues/3847


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3913). All of your documentation changes will be reflected on that endpoint.

I'm surprised this is needed because the order of the `dict` keys is deterministic as of Python 3.6 (documented in 3.7). Is there a reproducer for this behavior? I wouldn't make this change unless it's absolutely needed because `sorted` modifies the initial order of the keys.

Indeed this doesn't fix the issue apparently. Actually this is probably because the tokenizer used to process the second split is in a state that has been modified by the first split.

Therefore after reloading the first split from the cache, then the second split can't be reloaded since the tokenizer hasn't seen the first split (and therefore is considered a different tokenizer).
Fix google drive streaming for small files

Google drive did another change recently, following #3787 #3843 .

In particular Google Drive now returns 403 for GET requests with `confirm=t` when a files doesn't have a virus warning message. I fixed this by passing `confirm=t` if and only if when there is one (i.e. when status code is 200 for HEAD)


_The documentation is not available anymore as the PR was closed or merged._

Actually the CI fails because of this
![image](https://user-images.githubusercontent.com/42851186/159281771-78e611b1-6b04-4a87-8324-b6ba2d8c6a6a.png)

It looks like we can't have a proper way to test google drive in the CI right now. Though it seems to work locally if you're not banned. I think I'll just disable those tests for now

this fix will not be included?

No we can't do anything except stop using google drive when possible
Fix building of documentation

Documentation building is failing:
- https://github.com/huggingface/datasets/runs/5716300989?check_suite_focus=true

```
ValueError: There was an error when converting ../datasets/docs/source/package_reference/main_classes.mdx to the MDX format.
Unable to find datasets.filesystems.S3FileSystem in datasets. Make sure the path to that object is correct.
```

Fix #4037.


_The documentation is not available anymore as the PR was closed or merged._

Superseded by huggingface/doc-builder@31fe6c8bc7225810e281c2f6c6cd32f38828c504
Change `from_csv` default arguments

Passing `sep=None` to pandas's `read_csv` lets pandas guess the CSV file's separator

This PR allows users to use this pandas's feature by passing `sep=None` to `Dataset.from_csv`:

```python
Dataset.from_csv(
    ...,
    sep=None
)
```


This is not the default in pandas right ?
We try to align our CSV loader with the pandas API.

Moreover according to their documentation, the python parser is used when sep is None, which might not be the fastest one.

Maybe users could just specify `sep=None` themselves ?
In this case we should add some documentation about this
Fix TestCommand to move dataset_infos instead of copying

Why do we copy instead of moving the file?

CC: @lhoestq @lvwerra 


All the datasets that are loaded normally with `load_dataset`, if `dataset_infos.json` exists, have this file in the importable directory. So it's fine if we copy the file instead of moving it but it's not a big deal.

Any reason to prefer moving it rather than copying it ?

@lvwerra reported than when generating the `dataset_infos.json` for multiple dataset directories containing only JSONL files, subsequent `dataset_infos.json` files contained all previous directories as configs:
- First generate metadata for dataset in dir `dir1`: dataset_infos.json contains one config for `dir1`
- Then generate metadata for dataset in dir `dir2`: dataset_infos.json contains 2 configs, for `dir1` and `dir2`

The reason is that all dataset_infos.json files are first created in the same dir (the one containing the json builder) and then **copied** to the user dir.

Subsequent calls of TestCommand don't replace the dataset_infos.json already present in the dir of the json builder, but append to it.

MAYBE: we should just move for this use case, and copy for the other use cases? See this use case here:
- #3680

@lhoestq aren't you mentioning the case in the else clause?
```python
else:  # in case of a remote dataset
    dataset_dir = None
```

In that case `dataset_infos.json` is not copied: `dataset_dir = None`

When using the JSON loader, calling `get_imported_module_dir()` returns a path inside the pip installed packages, so we shouldn't write files in it anyway, and the dataset_infos.json file should be written directly in the user's directory instead (some users don't have write access to the pip installed packages for example).

Maybe the packaged modules like `json` should override `_save_infos` to save them in the user's directory instead of next to the builder's script. What do you think ?

Anyway as a hotfix we can just add an exception for the `json` builder for now, if the issue has to be fixed soon

I'm closing this PR.
[DO NOT MERGE] Test doc-builder with skipped installation feature

This PR is just for testing that we can build PR docs with changes made on the [`skip-install-for-real`](https://github.com/huggingface/doc-builder/tree/skip-install-for-real) branch of `doc-builder`.


_The documentation is not available anymore as the PR was closed or merged._

Fix in https://github.com/huggingface/doc-builder/pull/162 works as expected (docs build), closing this
Document librispeech configs

Added an example of how to load one config or the other


I think the main purpose of #4179 was how to be able to load both configs into one, so should we maybe add this part of the code: https://github.com/huggingface/datasets/issues/4179#issuecomment-1102383717 

to the doc? 

Actually @lhoestq would this work given that they have different split names: https://huggingface.co/datasets/librispeech_asr#data-splits ? 

This doc extension does not explain why I can't simply load the whole dataset. Or what workaround I need to get the whole dataset, which is what people usually want for Librispeech.

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq, I can add a `"all"` config to Librispeech have the datasets already cached somewhere 

I'm closing this PR then, feel free to continue the discussion in https://github.com/huggingface/datasets/issues/4179
Add remove_columns=True

This should fix all the issue we have with in place operations in mapping functions. This is crucial as where we do some weird things like:
```
def apply(batch):
    batch_size = len(batch["id"])
    batch["text"] = ["potato" for _ range(batch_size)]
    return {}

# Columns are: {"id": int}
dset.map(apply, batched=True, remove_columns="text") # crashes because `text` is not in the original columns
dset.map(apply, batched=True) # mapped datasets has `text` column
```

In this PR we suggest to have `remove_columns=True` so that we ignore the input completely, and just use the output to generate mapped dataset. This means that inplace operations won't have any effects anymore.


_The documentation is not available anymore as the PR was closed or merged._

Any reason why we can't just do `[inputs.copy()]` in this line for in-place operations to not have effects anymore:
https://github.com/huggingface/datasets/blob/bf432011ff9155a5bc16c03956bc63e514baf80d/src/datasets/arrow_dataset.py#L2232.

(in the `batched` case, we can also copy the inputs' values (list objects) to ignore in-place modifications to the inputs' columns)

I think `remove_columns=True` has no meaning, so I'm not a fan of this change.

@mariosasko copy does have a cost associated with it ... and plus you'll have to consider `deepcopy` Imagine columnds that are list of list of list of list .... Though I have to agree that `remove_columns=True` doesn't make sense (but, IMO, neither does it in its current use-case as it should refer to `input_columns`) 

Okay closing this PR for the following reasons:
 - `remove_columns=True` was expected to keep the `.update`-like operator for `.map`. I initially thought it would be a good way to ignore function side effects and only keep output of that function (cf. PR description).
 - expected `remove_columns=True` is a bad API according to @mariosasko and introduces unecessary changes for little gain (strictly equivalent to `remove_columns=dset.column_names`)
Remove dots in config names

20+ datasets have dots in their config names. However it causes issues with the YAML tags of the dataset cards since we can't have dots in YAML keys.

This is related to https://github.com/huggingface/datasets/pull/2362 (internal https://github.com/huggingface/moon-landing/issues/946).

Also removing the dots in the config names would allow us to merge https://github.com/huggingface/datasets/pull/4302 which removes a hack that replaces all the dots by underscores in the YAML tags.

I also added a test in the CI that checks that all the YAML tags to make sure that:
- they can be parsed using a YAML parser
- they contain only valid YAML tags like `languages` or `task_ids`
- they contain valid config names (no invalid characters `<>:/\|?*.`)


_The documentation is not available anymore as the PR was closed or merged._

Closing in favor of https://github.com/huggingface/datasets/pull/4367
Add binary classification task IDs

As a precursor to aligning the task IDs in `datasets` and AutoTrain, we need a way to distinguish binary vs multiclass vs multilabel classification.

This PR adds binary classification to the task IDs to enable this.

Related AutoTrain issue: https://github.com/huggingface/autonlp-backend/issues/597

cc @abhishekkrthakur @SBrandeis 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4584). All of your documentation changes will be reflected on that endpoint.

> Awesome thanks ! Can you add it to https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts first please ? This is where we define the cross libraries tasks taxonomy ;)

Thanks for the tip! Done in https://github.com/huggingface/hub-docs/pull/217

I don't think we need to update this file anymore. We should remove it IMO, and simply update the dataset [tagging app](https://huggingface.co/spaces/huggingface/datasets-tagging)

I'm closing this PR.
Remove all paperswithcode_id: null

On the Hub there is a validation error on the `paperswithcode_id` tag when the value is `null`:

<img width="686" alt="image" src="https://user-images.githubusercontent.com/42851186/177151825-93d341c5-25bd-41ab-96c2-c0b516d51c68.png">

We've been using `null` to specify that we checked on pwc but the dataset doesn't exist there.

To have the validation working again we can simply remove all the `paperswithcode_id: null`.

cc @julien-c 


_The documentation is not available anymore as the PR was closed or merged._

> We've been using `null` to specify that we checked on pwc but the dataset doesn't exist there.

@lhoestq maybe it's better to accept it on the Hub side then? Let me know if you want us to do it Hub-side

Yup it's maybe better to support it on the Hub side then indeed, thanks ! Closing this one
Add test-datasets CI job

To avoid having too many conflicts in the datasets and metrics dependencies I split the CI into test and test-catalog

test does the test of the core of the `datasets` lib, while test-catalog tests the datasets scripts and metrics scripts

This also makes `pip install -e .[dev]` much smaller for developers

WDYT @albertvillanova ?


_The documentation is not available anymore as the PR was closed or merged._

Closing this one since the dataset scripts will be removed in https://github.com/huggingface/datasets/pull/4974
[GH->HF] Load datasets from the Hub

Currently datasets with no namespace (e.g. squad, glue) are loaded from github.

In this PR I changed this logic to use the Hugging Face Hub instead.

This is the first step in removing all the dataset scripts in this repository

related to discussions in https://github.com/huggingface/datasets/pull/4059 (I should have continued from this PR actually)


_The documentation is not available anymore as the PR was closed or merged._

Duplicate of:
- #4059
Return a split Dataset in load_dataset

...instead of a DatasetDict.

```python
# now supported
ds = load_dataset("squad")
ds[0]  
for example in ds:
    pass

# still works
ds["train"]
ds["validation"]

# new
ds.splits  # Dict[str, Dataset] | None

# soon to be supported (not in this PR)
ds = load_dataset("dataset_with_no_splits")
ds[0]
for example in ds:
    pass
```

I implemented `Dataset.__getitem__` and `IterableDataset.__getitem__` to be able to get a split from a dataset.
The splits are defined by the `ds.info.splits` dictionary.

Therefore a dataset is a table that optionally has some splits defined in the dataset info. And a split dataset is the concatenation of all its splits.

I made as little breaking changes as possible. Notable breaking changes:
- `load_dataset("potato").keys() / .items() / .values() /` don't work anymore, since we don't return a dict
- same for `for split_name in load_dataset("potato")`, since we now iterate on the examples
- ..

TODO:
- [x] Update push_to_hub
- [x] Update save_to_disk/load_from_disk
- [ ] check for other breaking changes
- [ ] fix existing tests
- [ ] add new tests
- [ ] docs

This is related to https://github.com/huggingface/datasets/issues/5189, to extend `load_dataset` to return datasets without splits


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5301). All of your documentation changes will be reflected on that endpoint.

Just noticed that now we have to deal with indexed & split datasets. The remaining tests are failing because one should be able to get an indexed dataset when accessing the split of a dataset made of indexed splits (right now the index is just trashed)
Update tasks.json

Context:

* https://github.com/huggingface/datasets/issues/5255#issuecomment-1339107195

Cc: @osanseviero 


_The documentation is not available anymore as the PR was closed or merged._

I think the only place where we need to add it is here https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts

And I think we can remove tasks.json completely from this repo

Isn't tasks.json used anymore in this repo?

> I think the only place where we need to add it is here https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts
> 
> And I think we can remove tasks.json completely from this repo

What about the warning I mentioned in https://github.com/huggingface/datasets/issues/5255#issuecomment-1339013527? Also, the depth estimation entry is already present in https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts. 

The update is based on what I received in the output of the export job (c.f. https://github.com/huggingface/datasets/issues/5255#issuecomment-1339107195). 

Edit: Oh, are you referring to the dataset card of NYU Depth V2?

Yes, my suggestion was for the dataset card: you got the error message because you tried to set `depth-estimation` in `class_ids` instead of `class_categories`.

> What about the warning I mentioned in https://github.com/huggingface/datasets/issues/5255#issuecomment-1339013527? Also, the depth estimation entry is already present in https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts.

if you place it in `task_categories` you should be good :)

yes i would suggest rm'ing tasks.json here for clarity

Closing it. 

It's not clear if we can remove it btw, since old versions of `evaluate` rely on it (see https://github.com/huggingface/evaluate/pull/309)

cc @lvwerra 

Actually it can be removed without incidence in old versions of evaluate since we kept an hardcoded `known_task_ids` that is marked "DEPRECATED"
IterableDataset formatting in numpy/torch/tf/jax

This code now returns a numpy array:
```python
from datasets import load_dataset

ds = load_dataset("imagenet-1k", split="train", streaming=True).with_format("np")
print(next(iter(ds))["image"])
```

It also works with "arrow", "pandas", "torch", "tf" and "jax"

### Implementation details:

I'm using the existing code to format an Arrow Table to the right output format for simplicity.
Therefore it's probbaly not the most optimized approach.

For example to output PyTorch tensors it does this for every example:

python data -> arrow table -> numpy extracted data -> pytorch formatted data

### Releasing this feature

Even though I consider this as a bug/inconsistency, this change is a breaking change.
And I'm sure some users were relying on the torch iterable dataset to return PIL Image and used data collators to convert to pytorch.

So I guess this is `datasets` 3.0 ?

### TODO

- [x] merge https://github.com/huggingface/datasets/pull/5072
- [ ] docs
- [ ] tests

Close https://github.com/huggingface/datasets/issues/5083


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5084). All of your documentation changes will be reflected on that endpoint.

Actually I'm not happy with this implementation. It always require the iterable dataset to have definite `features`, which removes a lot of flexibility. So I think we need an actual formatting from python objects, not from arrow data.

Closing this one since it has too many conflicts and still require some work - it will be easier to open a new PR
Remove redundant local path information in audio/image datasets

Remove the redundant path information in the audio/image dataset as discussed in https://github.com/huggingface/datasets/pull/3430#issuecomment-994734828

TODOs:
* [ ] merge https://github.com/huggingface/datasets/pull/3430
* [ ] merge https://github.com/huggingface/datasets/pull/3364
* [ ] re-generate the info files of the updated audio datasets

cc:  @patrickvonplaten @anton-l @nateraw (I expect this to break the audio/vision examples in Transformers; after this change you'll be able to access underlying paths as follows `dset = dset.cast_column("audio", Audio(..., decode=False)); path = dset[0]["audio"]`)


Labels: name: dataset contribution

Cool, I'm in favor of this PR. Our official examples in speech already make use of `"audio"` so no need to change anything there. It would be great if we could prominently feature how one can get the audio path without decoding in the docs.

@patrickvonplaten Yes, I agree.

cc @stevhliu we should add an example where decoding is disabled (to read paths) to  [this section](https://github.com/huggingface/datasets/blob/master/docs/source/audio_process.rst#audio-datasets) in the docs and remove the mentions of `path`/`file` (if we merge this PR).

Thanks @mariosasko. Are you planning to finish this before we remove the dataset scripts from GitHub?
Support PIL Image objects in `add_item`/`add_column`

Fix #4796 

PS: We should also improve the type inference in `OptimizedTypeSequence` to make it possible to also infer the complex types (only `Image` currently) in nested arrays (e.g.  `[[pil_image], [pil_image, pil_image]]` or `[{"img": pil_image}`]), but I plan to address this in a separate PR.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4828). All of your documentation changes will be reflected on that endpoint.

Hey @mariosasko could we please merge this? I'm still getting the original error at #4796 .

Are you planning to continue working on this?