ahmedfarazsyk/datasets-github-issues huggingface dataset:
Add The Medical Segmentation Decathlon Dataset
Author: omarespejel
State: open
Labels: name: dataset request

name: vision

## Adding a Dataset
- **Name:** *The Medical Segmentation Decathlon Dataset*
- **Description:** The underlying data set was designed to explore the axis of difficulties typically encountered when dealing with medical images, such as small data sets, unbalanced labels, multi-site data, and small objects. 
- **Paper:** [link to the dataset paper if available](https://arxiv.org/abs/2106.05735)
- **Data:** http://medicaldecathlon.com/
- **Motivation:** Hugging Face seeks to democratize ML for society. One of the growing niches within ML is the ML + Medicine community. Key data sets will help increase the supply of HF resources for starting an initial community.

(cc @osanseviero @abidlabs )

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments: Hello! I have recently been involved with a medical image segmentation project myself and was going through the `The Medical Segmentation Decathlon Dataset` as well. 
I haven't yet had experience adding datasets to this repository yet but would love to get started. Should I take this issue?
If yes, I've got two questions -
1. There are 10 different datasets available, so are all datasets to be added in a single PR, or one at a time? 
2. Since it's a competition, masks for the test-set are not available. How is that to be tackled? Sorry if it's a silly question, I have recently started exploring `datasets`.

Hi! Sure, feel free to take this issue. You can self-assign the issue by commenting `#self-assign`.

To answer your questions:
1. It makes the most sense to add each one as a separate config, so one dataset script with 10 configs in a single PR.
2. Just set masks in the test set to `None`.

Note that the images/masks in this dataset are in NIfTI format, which our `Image` feature currently doesn't support, so I think it's best to yield the paths to the images/masks in the script and add a preprocessing section to the card where we explain how to load/process the images/masks with `nibabel` (I can help with that).   



> Note that the images/masks in this dataset are in NIfTI format, which our `Image` feature currently doesn't support, so I think it's best to yield the paths to the images/masks in the script and add a preprocessing section to the card where we explain how to load/process the images/masks with `nibabel` (I can help with that).

Gotcha, thanks. Will start working on the issue and let you know in case of any doubt.

#self-assign

This is great! There is a first model on the HUb that uses this dataset! https://huggingface.co/MONAI/example_spleen_segmentation
label information get lost after parquet serialization
Author: Tudyx
State: closed
Labels: name: bug

## Describe the bug
In *dataset_info.json* file, information about the label get lost after the dataset serialization.

## Steps to reproduce the bug
```python
from datasets import load_dataset

# normal save
dataset = load_dataset('glue', 'sst2', split='train')
dataset.save_to_disk("normal_save")


# save after parquet serialization
dataset.to_parquet("glue-sst2-train.parquet")
dataset = load_dataset("parquet", data_files='glue-sst2-train.parquet')
dataset.save_to_disk("save_after_parquet")
```

## Expected results
I expected to keep label information in *dataset_info.json* file even after parquet serialization

## Actual results
In the normal serialization i got
```json
"label": {
      "num_classes": 2,
      "names": [
        "negative",
        "positive"
      ],
      "names_file": null,
      "id": null,
      "_type": "ClassLabel"
    },
```
And after parquet serialization i got
```json
"label": {
      "dtype": "int64",
      "id": null,
      "_type": "Value"
    },
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.0
- Platform: ubuntu 20.04
- Python version: 3.8.10
- PyArrow version: 6.0.1


Comments: Hi ! We did a release of `datasets` today that may fix this issue. Can you try updating `datasets` and trying again ?

EDIT: the issue is still there actually

I think we can fix that by storing the Features in the parquet schema metadata, and then reload them when loading the parquet file

This info is stored in the Parquet schema metadata as of https://github.com/huggingface/datasets/pull/5516
Add The Mexican Emotional Speech Database (MESD)
Author: omarespejel
State: open
Labels: name: dataset request

name: speech

## Adding a Dataset
- **Name:** *The Mexican Emotional Speech Database (MESD)*
- **Description:** *Contains 864 voice recordings with six different prosodies: anger, disgust, fear, happiness, neutral, and sadness. Furthermore, three voice categories are included: female adult, male adult, and child. *
- **Paper:** *[Paper](https://ieeexplore.ieee.org/abstract/document/9629934/authors#authors)*
- **Data:** *[link to the Github repository or current dataset location](https://data.mendeley.com/datasets/cy34mh68j9/3)*
- **Motivation:** *Would add Spanish speech data to the HF datasets :) *

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
ConnectionError in IndicGLUE dataset
Author: sahoodib
State: closed
Labels: name: dataset bug

While I am trying to load IndicGLUE dataset (https://huggingface.co/datasets/indic_glue) it is giving me with the error:
```
ConnectionError: Couldn't reach https://storage.googleapis.com/ai4bharat-public-indic-nlp-corpora/evaluations/wikiann-ner.tar.gz (error 403)

Comments: @sahoodib, thanks for reporting.

Indeed, none of the data links appearing in the IndicGLUE website are working, e.g.: https://storage.googleapis.com/ai4bharat-public-indic-nlp-corpora/evaluations/soham-articles.tar.gz
```
<Error>
<Code>UserProjectAccountProblem</Code>
<Message>User project billing account not in good standing.</Message>
<Details>
The billing account for the owning project is disabled in state delinquent
</Details>
</Error>
```

We have contacted the data owners to inform them about their issue and ask them if they plan to fix it.

Yesterday I resent a reminder email with more AI4Bharat-related people in the loop.

I also opened an issue in their repos:
- https://github.com/AI4Bharat/indicnlp_corpus/issues/14
- https://github.com/AI4Bharat/ai4bharat.org/issues/71

We have received a reply from the authors reporting they have updated the URLs of their data files and opened a PR. See:
- #4978
Dataset.from_pandas preserves useless index
Author: Sorrow321
State: closed
Labels: name: bug

## Describe the bug
Let's say that you want to create a Dataset object from pandas dataframe. Most likely you will write something like this:

```
import pandas as pd
from datasets import Dataset


df = pd.read_csv('some_dataset.csv')
# Some DataFrame preprocessing code...
dataset = Dataset.from_pandas(df)
```
If your preprocessing code contain indexing operations like this:
```
df = df[df.col1 == some_value]
```
then your df.index can be changed from (default) ```RangeIndex(start=0, stop=16590, step=1)``` to something like this ```Int64Index([    0,     1,     2,     3,     4,     5,     6,     7,     8,
                9,
            ...
            83979, 83980, 83981, 83982, 83983, 83984, 83985, 83986, 83987,
            83988],
           dtype='int64', length=16590)```

In this case, PyArrow (by default) will preserve this non-standard index. In the result, your dataset object will have the extra field that you likely don't want to have: '__index_level_0__'.

You can easily fix this by just adding extra argument ```preserve_index=False``` to call of ```InMemoryTable.from_pandas``` in ```arrow_dataset.py```.

If you approve that this isn't desirable behavior, I can make a PR fixing that.

## Environment info
- `datasets` version: 1.16.1
- Platform: Linux-5.11.0-44-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 6.0.1



Comments: Hi! That makes sense. Sure, feel free to open a PR! Just a small suggestion: let's make `preserve_index` a parameter of `Dataset.from_pandas` (which we then pass to `InMemoryTable.from_pandas`) with `None` as a default value to not have this as a breaking change.
Cannot load ‘bookcorpusopen’
Author: HUIYINXUE
State: closed
Labels: name: bug

name: dataset bug

## Describe the bug
Cannot load 'bookcorpusopen'


## Steps to reproduce the bug
```python
dataset = load_dataset('bookcorpusopen')
```
or
```python
dataset = load_dataset('bookcorpusopen',script_version='master')
```

## Actual results
ConnectionError: Couldn't reach https://the-eye.eu/public/AI/pile_preliminary_components/books1.tar.gz

## Environment info
- `datasets` version: 1.9.0
- Platform: Linux version 3.10.0-1160.45.1.el7.x86_64
- Python version:  3.6.13
- PyArrow version: 6.0.1


Comments: The host of this copy of the dataset (https://the-eye.eu) is down and has been down for a good amount of time ([potentially months](https://www.reddit.com/r/Roms/comments/q82s15/theeye_downdied/))

Finding this dataset is a little esoteric, as the original authors took down the official BookCorpus dataset some time ago.

There are community-created versions of BookCorpus, such as the files hosted in the link below.
https://battle.shawwn.com/sdb/bookcorpus/

And more discussion here:
https://github.com/soskek/bookcorpus

Do we want to remove this dataset entirely?  There's a fair argument for this, given that the official BookCorpus dataset was taken down by the authors.   If not, perhaps can open a PR with the link to the community-created tar above and updated dataset description.

Hi! The `bookcorpusopen` dataset is not working for the same reason as explained in this comment: https://github.com/huggingface/datasets/issues/3504#issuecomment-1004564980

Hi @HUIYINXUE, it should work now that the data owners created a mirror server with all data, and we updated the URL in our library.
Integrate Milvus (pymilvus) library
Author: mariosasko
State: open
Labels: name: enhancement

Milvus is a popular open-source vector database. We should add a new vector index to support this project.

Comments: Hi @mariosasko，Just search randomly and I found this issue~ I'm the tech lead of Milvus and we are looking forward to integrate milvus together with huggingface datasets.

Any suggestion on how we could start?


Feel free to assign to me and we probably need some guide on it

@mariosasko any updates my man?


Hi! For starters, I suggest you take a look at this file: https://github.com/huggingface/datasets/blob/master/src/datasets/search.py, which contains all the code for Faiss/ElasticSearch support. We could set up a Slack channel for additional guidance. Let me know what you prefer.

> Hi! For starters, I suggest you take a look at this file: https://github.com/huggingface/datasets/blob/master/src/datasets/search.py, which contains all the code for Faiss/ElasticSearch support. We could set up a Slack channel for additional guidance. Let me know what you prefer.

Sure, we take a look and do some research
DuplicatedKeysError when loading tweet_qa dataset
Author: LeonieWeissweiler
State: closed
Labels: name: bug

When loading the tweet_qa dataset with `load_dataset('tweet_qa')`, the following error occurs: 

`DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 2a167f9e016ba338e1813fed275a6a1e
Keys should be unique and deterministic in nature
`
Might be related to issues #2433 and #2333

- `datasets` version: 1.17.0
- Python version: 3.8.5


Comments: Hi, we've just merged the PR with the fix. The fixed version of the dataset can be downloaded as follows:
```python
import datasets
dset = datasets.load_dataset("tweet_qa", revision="master")
```
set_format("np") no longer works for Image data
Author: cgarciae
State: closed
Labels: name: bug

## Describe the bug
`dataset.set_format("np")` no longer works for image data, previously you could load the MNIST like this:

```python
dataset = load_dataset("mnist")
dataset.set_format("np")
X_train = dataset["train"]["image"][..., None] # <== No longer a numpy array
```
but now it doesn't work, `set_format("np")` seems to have no effect and the dataset just returns a list/array of PIL images instead of numpy arrays as requested.


Comments: A quick fix for now is doing this:

```python
X_train = np.stack(dataset["train"]["image"])[..., None]

This error also propagates to jax and is even trickier to fix, since `.with_format(type='jax')` will use numpy conversion internally (and fail). For a three line failure:

```python
dataset = datasets.load_dataset("mnist")
dataset.set_format("jax")
X_train = dataset["train"]["image"]
```

Hi! We've recently introduced a new Image feature that yields PIL Images (and caches transforms on them) instead of arrays.

However, this feature requires a custom transform to yield np arrays directly:
```python
ddict = datasets.load_dataset("mnist")

def pil_image_to_array(batch):
     return {"image": [np.array(img) for img in batch["image"]]}  # or jnp.array(img) for Jax

ddict.set_transform(pil_image_to_array, columns="image", output_all_columns=True)
```

[Docs](https://huggingface.co/docs/datasets/master/process.html#format-transform) on `set_transform`.

Also, the approach proposed by @cgarciae is not the best because it loads the entire column in memory.

@albertvillanova @lhoestq WDYT? The Audio and the Image feature currently don't support the TF/Jax/PT Formatters, but for the Numpy Formatter maybe it makes more sense to return np arrays (and not a dict in the case of the Audio feature or a PIL Image object in the case of the Image feature).

Yes I agree it should return arrays and not a PIL image (and possible an array instead of a dict for audio data).
I'm currently finishing some code refactoring of the image and audio and opening a PR today. Maybe we can look into that after the refactoring

This has been fixed in https://github.com/huggingface/datasets/pull/5072, which is included in the latest release of `datasets`.
Bug in `openbookqa` dataset
Author: lucadiliello
State: closed
Labels: name: bug

name: dataset bug

## Describe the bug
Dataset entries contains a typo.

## Steps to reproduce the bug

```python
>>> from datasets import load_dataset
>>> obqa = load_dataset('openbookqa', 'main')
>>> obqa['train'][0]
```

## Expected results
```python
{'id': '7-980', 'question_stem': 'The sun is responsible for', 'choices': {'text': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting'], 'label': ['A', 'B', 'C', 'D']}, 'answerKey': 'D'}
```

## Actual results
```python
{'id': '7-980', 'question_stem': 'The sun is responsible for', 'choices': {'text': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting'], 'label': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting']}, 'answerKey': 'D'}
```

The bug is present in all configs and all splits.

## Environment info
- `datasets` version: 1.17.0
- Platform: Linux-5.4.0-1057-aws-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyArrow version: 4.0.1


Comments: Closed by:
- #4259
Specify the feature types of a dataset on the Hub without needing a dataset script
Author: lhoestq
State: closed
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
Currently if I upload a CSV with paths to audio files, the column type is string instead of Audio.

**Describe the solution you'd like**
I'd like to be able to specify the types of the column, so that when loading the dataset I directly get the features types I want.

The feature types could read from the `dataset_infos.json` for example.

**Describe alternatives you've considered**
Create a dataset script to specify the features, but that seems complicated for a simple thing.

cc @abidlabs 

Comments: After looking into this, discovered that this is already supported if the `dataset_infos.json` file is configured correctly! Here is a working example: https://huggingface.co/datasets/abidlabs/test-audio-13

This should be probably be documented, though.
Ability to split a dataset in multiple files.
Author: Dref360
State: open
Labels: name: enhancement

Hello,

**Is your feature request related to a problem? Please describe.**

My use case is that I have one writer that adds columns and multiple workers reading the same `Dataset`. Each worker should have access to columns added by the writer when they reload the dataset.

I understand that we shouldn't overwrite an arrow file as this could cause Segfault and so on. Before 1.16, I was able to overwrite the dataset and that would work most of the time with some retries.

**Describe the solution you'd like**

I was thinking that if we could append `Dataset._data_files`, when the workers reload the Dataset, they would get the new columns.


**Describe alternatives you've considered**
I currently need to
1. Save multiple "versions" of the dataset and load the latest.
2. Try working with cache files to get the latest columns.

**Additional context**

I think this would be a great addition to HFDataset as Parquet supports multi-files input out of the box!

I can make a PR myself with some pointers as needed :)
Allow loading community metrics from the hub, just like datasets
Author: eladsegal
State: closed
Labels: name: enhancement

name: generic discussion

**Is your feature request related to a problem? Please describe.**
Currently, I can load a metric implemented by me by providing the local path to the file in `load_metric`.
However, there is no option to do it with the metric uploaded to the hub.
This means that if I want to allow other users to use it, they must download it first which makes the usage less smooth.

**Describe the solution you'd like**
Load metrics from the hub just like datasets are loaded.
In order to not break stuff, the convention can be to put the metric file in a "metrics" folder in the hub.



Comments: Hi ! Thanks for your message :) This is a great idea indeed. We haven't started working on this yet though. For now I guess you can host your metric on the Hub (either with your model or your dataset) and use `hf_hub_download` to download it (docs [here](https://github.com/huggingface/huggingface_hub/blob/main/docs/hub/how-to-downstream.md#cached_download))

This is a great solution in the meantime, thanks!

Here's the code I used, in case it can be of help to someone else:
```python
import os, shutil
from huggingface_hub import hf_hub_download
def download_metric(repo_id, file_path):
    # repo_id: for models "username/model_name", for datasets "datasets/username/model_name"
    local_metric_path = hf_hub_download(repo_id=repo_id, filename=file_path)
    updated_local_metric_path = (os.path.dirname(local_metric_path) + os.path.basename(local_metric_path).replace(".", "_") + ".py")
    shutil.copy(local_metric_path, updated_local_metric_path)
    return updated_local_metric_path

metric = load_metric(download_metric(REPO_ID, FILE_PATH))
```

Solved with https://github.com/huggingface/evaluate 🤗 

Yay!! cc @lvwerra @sashavor @douwekiela 

Please share your feedback @eladsegal =)
Support 7-zip compressed data files
Author: albertvillanova
State: open
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**

We should support 7-zip compressed data files:
- [x] in `extract`:
  - #4672 
- [ ] in `iter_archive`: for streaming mode

both in streaming and non-streaming modes.


Comments: This should also resolve: https://github.com/huggingface/datasets/issues/3185.
How to convert  torch.utils.data.Dataset to datasets.arrow_dataset.Dataset?
Author: CindyTing
State: open
Labels: name: enhancement

Hi,
I use torch.utils.data.Dataset to define my own data, but I need to use the 'map' function of datasets.arrow_dataset.Dataset later, so I hope to convert torch.utils.data.Dataset to datasets.arrow_dataset.Dataset.

Here is an example.
```
from torch.utils.data import Dataset
from datasets.arrow_dataset import Dataset as HFDataset

class ADataset(Dataset):
    def __init__(self, data): 
        super().__init__()
        self.data = data
    def __getitem__(self, index):
        return self.data[index]
    def __len__(self):
        return self.len
class MDataset():
    def __init__(self, tokenizer: AutoTokenizer, data_args, training_args):
        self.train_dataset = ADataset(data_args)
        self.tokenizer = tokenizer
        self.data_args = data_args

        self.train_dataset = self.train_dataset.map(
            self.process_function,
            batched=True,
            remove_columns=column_names,
            load_from_cache_file=True,
            desc="Running tokenizer on train dataset",
             )
    def process_function(self, examples):
        sentences = [" ".join(sample[0][3]) for sample in examples]

        tokenized = self.tokenizer(
            sentences,
            max_length=self.max_seq_len,
            padding=self.padding, 
            truncation=True)
```
But it would raise an ERROR, AttributeError: 'ADataset' object has no attribute 'map'.

so how to convert  torch.utils.data.Dataset to datasets.arrow_dataset.Dataset?

Thanks in advance!
Task search function on hub not working correctly
Author: patrickvonplaten
State: open
Labels: name: bug

When I want to look at all datasets of the category: `speech-processing`  *i.e.* https://huggingface.co/datasets?task_categories=task_categories:speech-processing&sort=downloads , then the following dataset doesn't show up for some reason:

- https://huggingface.co/datasets/speech_commands

even thought it's task tags seem correct: 
https://raw.githubusercontent.com/huggingface/datasets/master/datasets/speech_commands/README.md

Comments: known issue due to https://github.com/huggingface/datasets/pull/2362 (and [internal](https://github.com/huggingface/moon-landing/issues/946)) , will be solved soon

hmm actually i have no recollection of why I said that

Because it has dots in some YAML keys, it can't be parsed and indexed by the back-end
Give clearer instructions to add the YAML tags
Author: albertvillanova
State: closed
Labels: name: bug

## Describe the bug
As reported by @julien-c, many community datasets contain the line `YAML tags:` at the top of the YAML section in the header of the README file.  See e.g.: https://huggingface.co/datasets/bigscience/P3/commit/a03bea08cf4d58f268b469593069af6aeb15de32

Maybe we should give clearer instruction/hints in the README template.
wmt19 is broken (zh-en)
Author: AjayP13
State: closed
Labels: name: bug

name: dataset bug

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("wmt19", 'zh-en')
```

## Expected results
The dataset should download.

## Actual results
`ConnectionError: Couldn't reach ftp://cwmt-wmt:cwmt-wmt@datasets.nju.edu.cn/parallel/casia2015.zip`

## Environment info
- `datasets` version: 1.15.1
- Platform: Linux
- Python version: 3.8 


Comments: This issue is not reproducible.