helmo/github-issues huggingface dataset:
Revisit `enable/disable_` toggle function prefix
State: closed | Author: jaketae (CONTRIBUTOR)
Labels: name: enhancement

As discussed in https://github.com/huggingface/transformers/pull/15167, we should revisit the `enable/disable_` toggle function prefix, potentially in favor of `set_enabled_`. Concretely, this translates to

- De-deprecating `disable_progress_bar()`
- Adding `enable_progress_bar()`
- On the caching side, adding `enable_caching` and `disable_caching`

Additional decisions have to be made with regards to the existing `set_enabled_X` functions; that is, whether to keep them as is or deprecate them in favor of the aforementioned functions. 

cc @mariosasko @lhoestq
Datasets streaming + map doesn't work for `Audio`
State: closed | Author: patrickvonplaten (CONTRIBUTOR)
Labels: name: bug

name: duplicate

## Describe the bug
When using audio datasets in streaming mode, applying a `map(...)` before iterating leads to an error as the key `array` does not exist anymore.

## Steps to reproduce the bug

```python
from datasets import load_dataset

ds = load_dataset("common_voice", "en", streaming=True, split="train")


def map_fn(batch):
    print("audio keys", batch["audio"].keys())
    batch["audio"] = batch["audio"]["array"][:100]
    return batch


ds = ds.map(map_fn) 

sample = next(iter(ds))
```

I think the audio is somehow decoded before `.map(...)` is actually called.

## Expected results

IMO, the above code snippet should work.

## Actual results

```bash
audio keys dict_keys(['path', 'bytes'])
Traceback (most recent call last):
  File "./run_audio.py", line 15, in <module>
    sample = next(iter(ds))
  File "/home/patrick/python_bin/datasets/iterable_dataset.py", line 341, in __iter__
    for key, example in self._iter():
  File "/home/patrick/python_bin/datasets/iterable_dataset.py", line 338, in _iter
    yield from ex_iterable
  File "/home/patrick/python_bin/datasets/iterable_dataset.py", line 192, in __iter__
    yield key, self.function(example)
  File "./run_audio.py", line 9, in map_fn
    batch["input"] = batch["audio"]["array"][:100]
KeyError: 'array'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.17.1.dev0
- Platform: Linux-5.3.0-64-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 6.0.1


Comments: This seems related to https://github.com/huggingface/datasets/issues/3505.
Add The Medical Segmentation Decathlon Dataset
State: open | Author: omarespejel (NONE)
Labels: name: dataset request

name: vision

## Adding a Dataset
- **Name:** *The Medical Segmentation Decathlon Dataset*
- **Description:** The underlying data set was designed to explore the axis of difficulties typically encountered when dealing with medical images, such as small data sets, unbalanced labels, multi-site data, and small objects. 
- **Paper:** [link to the dataset paper if available](https://arxiv.org/abs/2106.05735)
- **Data:** http://medicaldecathlon.com/
- **Motivation:** Hugging Face seeks to democratize ML for society. One of the growing niches within ML is the ML + Medicine community. Key data sets will help increase the supply of HF resources for starting an initial community.

(cc @osanseviero @abidlabs )

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments: Hello! I have recently been involved with a medical image segmentation project myself and was going through the `The Medical Segmentation Decathlon Dataset` as well. 
I haven't yet had experience adding datasets to this repository yet but would love to get started. Should I take this issue?
If yes, I've got two questions -
1. There are 10 different datasets available, so are all datasets to be added in a single PR, or one at a time? 
2. Since it's a competition, masks for the test-set are not available. How is that to be tackled? Sorry if it's a silly question, I have recently started exploring `datasets`.

Hi! Sure, feel free to take this issue. You can self-assign the issue by commenting `#self-assign`.

To answer your questions:
1. It makes the most sense to add each one as a separate config, so one dataset script with 10 configs in a single PR.
2. Just set masks in the test set to `None`.

Note that the images/masks in this dataset are in NIfTI format, which our `Image` feature currently doesn't support, so I think it's best to yield the paths to the images/masks in the script and add a preprocessing section to the card where we explain how to load/process the images/masks with `nibabel` (I can help with that).   



> Note that the images/masks in this dataset are in NIfTI format, which our `Image` feature currently doesn't support, so I think it's best to yield the paths to the images/masks in the script and add a preprocessing section to the card where we explain how to load/process the images/masks with `nibabel` (I can help with that).

Gotcha, thanks. Will start working on the issue and let you know in case of any doubt.

#self-assign

This is great! There is a first model on the HUb that uses this dataset! https://huggingface.co/MONAI/example_spleen_segmentation
label information get lost after parquet serialization
State: closed | Author: Tudyx (NONE)
Labels: name: bug

## Describe the bug
In *dataset_info.json* file, information about the label get lost after the dataset serialization.

## Steps to reproduce the bug
```python
from datasets import load_dataset

# normal save
dataset = load_dataset('glue', 'sst2', split='train')
dataset.save_to_disk("normal_save")


# save after parquet serialization
dataset.to_parquet("glue-sst2-train.parquet")
dataset = load_dataset("parquet", data_files='glue-sst2-train.parquet')
dataset.save_to_disk("save_after_parquet")
```

## Expected results
I expected to keep label information in *dataset_info.json* file even after parquet serialization

## Actual results
In the normal serialization i got
```json
"label": {
      "num_classes": 2,
      "names": [
        "negative",
        "positive"
      ],
      "names_file": null,
      "id": null,
      "_type": "ClassLabel"
    },
```
And after parquet serialization i got
```json
"label": {
      "dtype": "int64",
      "id": null,
      "_type": "Value"
    },
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.0
- Platform: ubuntu 20.04
- Python version: 3.8.10
- PyArrow version: 6.0.1


Comments: Hi ! We did a release of `datasets` today that may fix this issue. Can you try updating `datasets` and trying again ?

EDIT: the issue is still there actually

I think we can fix that by storing the Features in the parquet schema metadata, and then reload them when loading the parquet file

This info is stored in the Parquet schema metadata as of https://github.com/huggingface/datasets/pull/5516
Add The Mexican Emotional Speech Database (MESD)
State: open | Author: omarespejel (NONE)
Labels: name: dataset request

name: speech

## Adding a Dataset
- **Name:** *The Mexican Emotional Speech Database (MESD)*
- **Description:** *Contains 864 voice recordings with six different prosodies: anger, disgust, fear, happiness, neutral, and sadness. Furthermore, three voice categories are included: female adult, male adult, and child. *
- **Paper:** *[Paper](https://ieeexplore.ieee.org/abstract/document/9629934/authors#authors)*
- **Data:** *[link to the Github repository or current dataset location](https://data.mendeley.com/datasets/cy34mh68j9/3)*
- **Motivation:** *Would add Spanish speech data to the HF datasets :) *

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Add Mauve metric
State: closed | Author: jthickstun (CONTRIBUTOR)

Add support for the [Mauve](https://github.com/krishnap25/mauve) metric introduced in this [paper](https://arxiv.org/pdf/2102.01454.pdf) (Neurips, 2021).

Comments: Hi ! The CI was failing because `mauve-text` wasn't installed. I added it to the CI setup :)

I also did some minor changes to the script itself, especially to remove `**kwargs` and explicitly mentioned all the supported arguments (this way if someone does a typo with some parameters they get an error)
ConnectionError in IndicGLUE dataset
State: closed | Author: sahoodib (NONE)
Labels: name: dataset bug

While I am trying to load IndicGLUE dataset (https://huggingface.co/datasets/indic_glue) it is giving me with the error:
```
ConnectionError: Couldn't reach https://storage.googleapis.com/ai4bharat-public-indic-nlp-corpora/evaluations/wikiann-ner.tar.gz (error 403)

Comments: @sahoodib, thanks for reporting.

Indeed, none of the data links appearing in the IndicGLUE website are working, e.g.: https://storage.googleapis.com/ai4bharat-public-indic-nlp-corpora/evaluations/soham-articles.tar.gz
```
<Error>
<Code>UserProjectAccountProblem</Code>
<Message>User project billing account not in good standing.</Message>
<Details>
The billing account for the owning project is disabled in state delinquent
</Details>
</Error>
```

We have contacted the data owners to inform them about their issue and ask them if they plan to fix it.

Yesterday I resent a reminder email with more AI4Bharat-related people in the loop.

I also opened an issue in their repos:
- https://github.com/AI4Bharat/indicnlp_corpus/issues/14
- https://github.com/AI4Bharat/ai4bharat.org/issues/71

We have received a reply from the authors reporting they have updated the URLs of their data files and opened a PR. See:
- #4978
Add the KMWP dataset (extension of #3564)
State: closed | Author: sooftware (NONE)
Labels: name: dataset contribution

New pull request of #3564 (Add the KMWP dataset)

Comments: Sorry, I'm late to check! I'll send it to you soon!

Thanks for your contribution, @sooftware. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there, under this organization namespace: https://huggingface.co/tunib

Please, feel free to tell us if you need some help.

Close this PR. Thanks!
Add the DKTC dataset (Extension of #3564)
State: closed | Author: sooftware (NONE)
Labels: name: dataset contribution

New pull request of #3564. (for DKTC)


Comments: I reflect your comment! @lhoestq 

Wait, the format of the data just changed, so I'll take it into consideration and commit it.

I update the code according to the dataset structure change.

Thanks ! I think the dummy data are not valid yet - the dummy train.csv file only contains a partial example (the quotes `"` start but never end).

> Thanks ! I think the dummy data are not valid yet - the dummy train.csv file only contains a partial example (the quotes `"` start but never end).

Hi! @lhoestq There is a problem.  
<img src="https://user-images.githubusercontent.com/42150335/149804142-3800e635-f5a0-44d9-9694-0c2b0c05f16b.png" width=500>
  
As shown in the picture above, the conversation is divided into "\n" in the "conversion" column.  
That's why there's an error in the file path that only saved only five lines like below. 

```
'idx', 'class', 'conversation'
'0', '협박 대화', '"지금 너 스스로를 죽여달라고 애원하는 것인가?'
아닙니다. 죄송합니다.'
죽을 거면 혼자 죽지 우리까지 사건에 휘말리게 해? 진짜 죽여버리고 싶게.'
정말 잘못했습니다.
```
   
In fact, these five lines are all one line.  
  



Hi ! I see, in this case ca you make sure that the dummy data has a full sample ?

Feel free to open the dummy train.csv in the dummy_data.zip file and add the missing lines

Sorry, I'm late to check! I'll send it to you soon!

Thanks for your contribution, @sooftware. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there, under this organization namespace: https://huggingface.co/tunib

Please, feel free to tell us if you need some help.

Close this PR. Thanks!
Fix push to hub to allow individual split push
State: closed | Author: thomasw21 (CONTRIBUTOR)

# Description of the issue

If one decides to push a split on a datasets repo, he uploads the dataset and overrides the config. However previous config splits end up being lost despite still having the dataset necessary.

The new flow is the following:
 - query the old config from the repo
 - update into a new config (add/overwrite new split for example)
 - push the new config

# Side fix

 - `repo_id` in HfFileSystem was wrongly typed.
 - I've added `indent=2` as it becomes much easier to read now.



Comments: This has been addressed in https://github.com/huggingface/datasets/pull/4415. Closing.
Add parameter `preserve_index` to `from_pandas`
State: closed | Author: Sorrow321 (CONTRIBUTOR)

Added optional parameter, so that user can get rid of useless index preserving. [Issue](https://github.com/huggingface/datasets/issues/3563)

Comments: > 

I did `make style` and it affected over 500 files

```
All done! ✨ 🍰 ✨
575 files reformatted, 372 files left unchanged.
isort tests src benchmarks datasets/**/*.py metri
```

(result)
![image](https://user-images.githubusercontent.com/20703486/149166681-2f9d1bc4-116a-4f53-ad42-e54e3b8bd605.png)


Nvm I was using wrong black version
Add the KMWP & DKTC dataset.
State: closed | Author: sooftware (NONE)

 Add the DKTC dataset.
- https://github.com/tunib-ai/DKTC

Comments: I reflect your review. cc. @lhoestq 

Ah sorry, I missed KMWP comment, wait.

I request 2 new pull requests. #3569 #3570
Dataset.from_pandas preserves useless index
State: closed | Author: Sorrow321 (CONTRIBUTOR)
Labels: name: bug

## Describe the bug
Let's say that you want to create a Dataset object from pandas dataframe. Most likely you will write something like this:

```
import pandas as pd
from datasets import Dataset


df = pd.read_csv('some_dataset.csv')
# Some DataFrame preprocessing code...
dataset = Dataset.from_pandas(df)
```
If your preprocessing code contain indexing operations like this:
```
df = df[df.col1 == some_value]
```
then your df.index can be changed from (default) ```RangeIndex(start=0, stop=16590, step=1)``` to something like this ```Int64Index([    0,     1,     2,     3,     4,     5,     6,     7,     8,
                9,
            ...
            83979, 83980, 83981, 83982, 83983, 83984, 83985, 83986, 83987,
            83988],
           dtype='int64', length=16590)```

In this case, PyArrow (by default) will preserve this non-standard index. In the result, your dataset object will have the extra field that you likely don't want to have: '__index_level_0__'.

You can easily fix this by just adding extra argument ```preserve_index=False``` to call of ```InMemoryTable.from_pandas``` in ```arrow_dataset.py```.

If you approve that this isn't desirable behavior, I can make a PR fixing that.

## Environment info
- `datasets` version: 1.16.1
- Platform: Linux-5.11.0-44-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 6.0.1



Comments: Hi! That makes sense. Sure, feel free to open a PR! Just a small suggestion: let's make `preserve_index` a parameter of `Dataset.from_pandas` (which we then pass to `InMemoryTable.from_pandas`) with `None` as a default value to not have this as a breaking change.
Cannot load ‘bookcorpusopen’
State: closed | Author: HUIYINXUE (NONE)
Labels: name: bug

name: dataset bug

## Describe the bug
Cannot load 'bookcorpusopen'


## Steps to reproduce the bug
```python
dataset = load_dataset('bookcorpusopen')
```
or
```python
dataset = load_dataset('bookcorpusopen',script_version='master')
```

## Actual results
ConnectionError: Couldn't reach https://the-eye.eu/public/AI/pile_preliminary_components/books1.tar.gz

## Environment info
- `datasets` version: 1.9.0
- Platform: Linux version 3.10.0-1160.45.1.el7.x86_64
- Python version:  3.6.13
- PyArrow version: 6.0.1


Comments: The host of this copy of the dataset (https://the-eye.eu) is down and has been down for a good amount of time ([potentially months](https://www.reddit.com/r/Roms/comments/q82s15/theeye_downdied/))

Finding this dataset is a little esoteric, as the original authors took down the official BookCorpus dataset some time ago.

There are community-created versions of BookCorpus, such as the files hosted in the link below.
https://battle.shawwn.com/sdb/bookcorpus/

And more discussion here:
https://github.com/soskek/bookcorpus

Do we want to remove this dataset entirely?  There's a fair argument for this, given that the official BookCorpus dataset was taken down by the authors.   If not, perhaps can open a PR with the link to the community-created tar above and updated dataset description.

Hi! The `bookcorpusopen` dataset is not working for the same reason as explained in this comment: https://github.com/huggingface/datasets/issues/3504#issuecomment-1004564980

Hi @HUIYINXUE, it should work now that the data owners created a mirror server with all data, and we updated the URL in our library.
Run pyupgrade for Python 3.6+
State: closed | Author: bryant1410 (CONTRIBUTOR)

Run the command:

```bash
pyupgrade $(find . -name "*.py" -type f) --py36-plus
```

Which mainly avoids unnecessary lists creations and also removes unnecessary code for Python 3.6+.

It was originally part of #3489.

Tip for reviewing faster: use the CLI (`git diff`) and scroll.

Comments: Hi ! Thanks for the change :)
Could it be possible to only run it for the code in `src/` ? We try to not change the code in the `datasets/` directory too often since it refreshes the users cache when they upgrade `datasets`.

> Hi ! Thanks for the change :)
> Could it be possible to only run it for the code in `src/` ? We try to not change the code in the `datasets/` directory too often since it refreshes the users cache when they upgrade `datasets`.

I reverted the changes in `datasets/` instead of changing only `src/`. Does it sound good?

I just resolved some conflicts with the master branch. If the CI is green we can merge :)
Integrate Milvus (pymilvus) library
State: open | Author: mariosasko (COLLABORATOR)
Labels: name: enhancement

Milvus is a popular open-source vector database. We should add a new vector index to support this project.

Comments: Hi @mariosasko，Just search randomly and I found this issue~ I'm the tech lead of Milvus and we are looking forward to integrate milvus together with huggingface datasets.

Any suggestion on how we could start?


Feel free to assign to me and we probably need some guide on it

@mariosasko any updates my man?


Hi! For starters, I suggest you take a look at this file: https://github.com/huggingface/datasets/blob/master/src/datasets/search.py, which contains all the code for Faiss/ElasticSearch support. We could set up a Slack channel for additional guidance. Let me know what you prefer.

> Hi! For starters, I suggest you take a look at this file: https://github.com/huggingface/datasets/blob/master/src/datasets/search.py, which contains all the code for Faiss/ElasticSearch support. We could set up a Slack channel for additional guidance. Let me know what you prefer.

Sure, we take a look and do some research
Fix bug in `ImageClassifcation` task template
State: closed | Author: mariosasko (COLLABORATOR)

Fixes a bug in the `ImageClassification` task template which requires specifying class labels twice in dataset scripts. Additionally, this PR refactors the API around the classification task templates for cleaner `labels` handling.

CC: @lewtun @nateraw

Comments: The CI failures are unrelated to the changes in this PR.

> The CI failures are unrelated to the changes in this PR.

It seems that some of the failures are due to the tests on the dataset cards (e.g. CIFAR, MNIST, FASHION_MNIST). Perhaps it's worth addressing those in this PR to avoid confusing downstream developers who branch off `master` and suddenly have a failing CI?

@lewtun We only run these tests against the modified datasets on the PR branch, so this will not lead to errors after merging.
DuplicatedKeysError when loading tweet_qa dataset
State: closed | Author: LeonieWeissweiler (NONE)
Labels: name: bug

When loading the tweet_qa dataset with `load_dataset('tweet_qa')`, the following error occurs: 

`DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 2a167f9e016ba338e1813fed275a6a1e
Keys should be unique and deterministic in nature
`
Might be related to issues #2433 and #2333

- `datasets` version: 1.17.0
- Python version: 3.8.5


Comments: Hi, we've just merged the PR with the fix. The fixed version of the dataset can be downloaded as follows:
```python
import datasets
dset = datasets.load_dataset("tweet_qa", revision="master")
```
set_format("np") no longer works for Image data
State: closed | Author: cgarciae (NONE)
Labels: name: bug

## Describe the bug
`dataset.set_format("np")` no longer works for image data, previously you could load the MNIST like this:

```python
dataset = load_dataset("mnist")
dataset.set_format("np")
X_train = dataset["train"]["image"][..., None] # <== No longer a numpy array
```
but now it doesn't work, `set_format("np")` seems to have no effect and the dataset just returns a list/array of PIL images instead of numpy arrays as requested.


Comments: A quick fix for now is doing this:

```python
X_train = np.stack(dataset["train"]["image"])[..., None]

This error also propagates to jax and is even trickier to fix, since `.with_format(type='jax')` will use numpy conversion internally (and fail). For a three line failure:

```python
dataset = datasets.load_dataset("mnist")
dataset.set_format("jax")
X_train = dataset["train"]["image"]
```

Hi! We've recently introduced a new Image feature that yields PIL Images (and caches transforms on them) instead of arrays.

However, this feature requires a custom transform to yield np arrays directly:
```python
ddict = datasets.load_dataset("mnist")

def pil_image_to_array(batch):
     return {"image": [np.array(img) for img in batch["image"]]}  # or jnp.array(img) for Jax

ddict.set_transform(pil_image_to_array, columns="image", output_all_columns=True)
```

[Docs](https://huggingface.co/docs/datasets/master/process.html#format-transform) on `set_transform`.

Also, the approach proposed by @cgarciae is not the best because it loads the entire column in memory.

@albertvillanova @lhoestq WDYT? The Audio and the Image feature currently don't support the TF/Jax/PT Formatters, but for the Numpy Formatter maybe it makes more sense to return np arrays (and not a dict in the case of the Audio feature or a PIL Image object in the case of the Image feature).

Yes I agree it should return arrays and not a PIL image (and possible an array instead of a dict for audio data).
I'm currently finishing some code refactoring of the image and audio and opening a PR today. Maybe we can look into that after the refactoring

This has been fixed in https://github.com/huggingface/datasets/pull/5072, which is included in the latest release of `datasets`.
Add the KMWP & DKTC dataset.
State: closed | Author: sooftware (NONE)

Add the KMWP & DKTC dataset.
  
Additional notes:
- Both datasets will be released on January 10 through the GitHub link below.
  - https://github.com/tunib-ai/DKTC
  - https://github.com/tunib-ai/KMWP
- So it doesn't work as a link at the moment, but the code will work soon (after it is released on January 10).