konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
Fixes to pubmed dataset download function

Pubmed has updated its settings for 2022 and thus existing download script does not work. 

Hi ! I think we can simply add a new configuration for the 2022 data instead of replacing them.
You can add the new configuration here:
```python
    BUILDER_CONFIGS = [
        datasets.BuilderConfig(name="2021", description="The 2021 annual record", version=datasets.Version("1.0.0")),
        datasets.BuilderConfig(name="2022", description="The 2022 annual record", version=datasets.Version("1.0.0")),
    ]
```

And we can have the URLs for these two versions this way:
```python
_URLs = {
    "2021": f"ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed21n{i:04d}.xml.gz" for i in range(1, 1063)],
    "2022": f"ftp://ftp.ncbi.nlm.nih.gov/pubmed/baseline/pubmed22n{i:04d}.xml.gz" for i in range(1, 1114)]
}
```
and depending on the configuration name (you can get it with `self.config.name`) we can pick the URLs of 2021 or the ones of 2022 and pass them to the `dl_manager` in `_split_generators`

Feel free to ping me if you have questions or if I can help !

Hi @spacemanidol, thanks for your contribution.

The update of the PubMed dataset URL (besides the update of the corresponding metadata and the dummy data) was already merged to master branch in this other PR:
- #3692 

I'm closing this PR then.

@lhoestq please take into account that 2021 data is no longer accessible: every year PubMed releases the baseline data (containing all previous data until that year) and from that on, they release daily updates.  

> @lhoestq please take into account that 2021 data is no longer accessible: every year PubMed releases the baseline data (containing all previous data until that year) and from that on, they release daily updates.

Oh ok I didn't know, thanks
Add support for continuous metrics (RMSE, MAE)

**Is your feature request related to a problem? Please describe.**

I am uploading our dataset and models for the "Constructing interval measures" method we've developed, which uses item response theory to convert multiple discrete labels into a continuous spectrum for hate speech. Once we have this outcome our NLP models conduct regression rather than classification, so binary metrics are not relevant. The only continuous metrics available at https://huggingface.co/metrics are pearson & spearman correlation, which don't ensure that the prediction is on the same scale as the outcome.

**Describe the solution you'd like**
I would like to be able to tag our models on the Hub with the following metrics:
- RMSE
- MAE

**Describe alternatives you've considered**

I don't know if there are any alternatives.

**Additional context**
Our preprint is available here: https://arxiv.org/abs/2009.10277 . We are making it available for use in Jigsaw's Toxic Severity Rating Kaggle competition: https://www.kaggle.com/c/jigsaw-toxic-severity-rating/overview . I have our first model uploaded to the Hub at https://huggingface.co/ucberkeley-dlab/hate-measure-roberta-large

Thanks,
Chris


Hey @ck37 

You can always use a custom metric as explained [in this guide from HF](https://huggingface.co/docs/datasets/master/loading_metrics.html#using-a-custom-metric-script).

If this issue needs to be contributed to (for enhancing the metric API) I think [this link](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html) would be helpful for the `MAE` metric.

You can use a local metric script just by providing its path instead of the usual shortcut name 

#self-assign I have starting working on this issue to enhance the metric API.
Adding Turkic X-WMT evaluation set for machine translation

This dataset is a human-translated evaluation set for MT crowdsourced and provided by the [Turkic Interlingua ](turkic-interlingua.org) community. It contains eval sets for 8 Turkic languages covering 88 language directions. Languages being covered are:

Azerbaijani (az)
Bashkir (ba)
English (en)
Karakalpak (kaa)
Kazakh (kk)
Kirghiz (ky)
Russian (ru)
Turkish (tr)
Sakha (sah)
Uzbek (uz)

More info about the corpus is here: [https://github.com/turkic-interlingua/til-mt/tree/master/xwmt](https://github.com/turkic-interlingua/til-mt/tree/master/xwmt)

A paper describing the test set is here: [https://arxiv.org/abs/2109.04593](https://arxiv.org/abs/2109.04593)


hi! Thank you for all the comments! I believe I addressed them all. Let me know if there is anything else

Hi there! I was wondering if there is anything else to change before this can be merged

@lhoestq Hi! Just a gentle reminder about the steps to merge this one! 

Thanks for the heads up ! I think I fixed the last issue with the YAML tags

The CI failure is unrelated to this PR and fixed on master, let's merge :)

Thanks a lot !
Dataset Viewer not showing Previews for Private Datasets

## Dataset viewer issue for 'abidlabs/test-audio-13'

It seems that the dataset viewer does not show previews for `private` datasets, even for the user who's private dataset it is. See [1] for example. If I change the visibility to public, then it does show, but it would be useful to have the viewer even for private datasets.

![image](https://user-images.githubusercontent.com/1778297/150200515-93ff1545-11fd-4793-be64-6bed3cd895e2.png)

**Link:** 

[1] https://huggingface.co/datasets/abidlabs/test-audio-13

**Am I the one who added this dataset?** 

Yes


Sure, it's on the roadmap.

Closing in favor of https://github.com/huggingface/datasets-server/issues/39.
Add British Library books dataset

This pull request adds a dataset of text from digitised (primarily 19th Century) books from the British Library. This collection has previously been used for training language models, e.g. https://github.com/dbmdz/clef-hipe/blob/main/hlms.md. It would be nice to make this dataset more accessible for others to use through datasets. 

This is still a WIP but I wanted to get some initial feedback in particular; I wanted to check:
- I am handling the use of `iter_archive` correctly - I intend to ensure that `dl_manager.download` gets the complete list of URLs to download upfront, so the progress bar knows how much is left to download and then to pass through the `gen_kwargs` a list of downloaded zip archives wrapped in `iter_archive`. I am unsure if there is a more elegant approach for this?
- the number of configs: I have aimed to keep this limited - there are a lot of URLs covering the entire dataset, but I have tried to base the configs on what I believe the majority of people will want to they are not presented with too many options - I am happy to hear suggestions for changing this

If there are other glaring omissions or mistakes, I'd be happy to hear them. If this approach seems sensible in general, I will finish all the remaining TODOs, generate dummy_data, etc. 
 

Thanks for all the help and suggestions

> Since the dataset has a very specific structure it might not be that easy so feel free to ping me if you have questions or if I can help !

I did get a little stuck here! So far I have created directories for each config i.e:

`datasets/datasets/blbooks/dummy/1700_1799/1.0.2/dummy_data.zip` 

I have then added two examples of the `jsonl.gz` files that are in the underlying dataset to each dummy_data directory.This fails the test using local files. 

Since 

```python
def _generate_examples(self, data_dirs):
```

takes as input `data_dirs` which is a list of `iter_dirs` do I need to put the dummy files inside another directory? i.e. 

`datasets/datasets/blbooks/dummy/1700_1799/1.0.2/dummy_data/1700/00.jsonl.gz` 


 

I think I managed to create the dummy data :)

I think everything is good now, if you don't have other changes to do, please mark your PR as "ready for review" and ping me!

> I think I managed to create the dummy data :)

Thanks so much for that!

> I think everything is good now, if you don't have other changes to do, please mark your PR as "ready for review" and ping me!

Think it is ready to merge from my end @lhoestq. 

The CI failure on windows is unrelated to your PR and fixed on `master`, we can ignore it
Update url for conll2003

Following https://github.com/huggingface/datasets/issues/3582 I'm changing the download URL of the conll2003 data files, since the previous host doesn't have the authorization to redistribute the data

Hi. lhoestq 

![image](https://user-images.githubusercontent.com/21982975/150345097-154f2b1a-bb12-47af-bddf-40eec0a0dadb.png)
What is the solution for it?
you can see it is still doesn't work here.
https://colab.research.google.com/drive/1l52FGWuSaOaGYchit4CbmtUSuzNDx_Ok?usp=sharing
Thank you.


For now you can specify `load_dataset(..., revision="master")` to use the fix on `master`.

We'll also do a new release of `datasets` tomorrow I think
Readme info not being parsed to show on Dataset card page

## Describe the bug
The info contained in the README.md file is not being shown in the dataset main page. Basic info and table of contents are properly formatted in the README.

## Steps to reproduce the bug

# Sample code to reproduce the bug
The README file is this one: https://huggingface.co/datasets/softcatala/Tilde-MODEL-Catalan/blob/main/README.md

## Expected results
README info should appear in the Dataset card page.

## Actual results
Nothing is shown. However, labels are parsed and shown successfully.


i suspect a markdown parsing error, @severo do you want to take a quick look at it when you have some time?

# Problem
The issue seems to coming from the front matter of the README
```---
annotations_creators:
- no-annotation
language_creators:
- machine-generated
languages:
- 'ca'
- 'de'
licenses:
- cc-by-4.0
multilinguality:
- translation
pretty_name: Catalan-German aligned corpora to train NMT systems.
size_categories:
- "1M<n<10M"            
source_datasets:
- extended|tilde_model
task_categories:
- machine-translation
task_ids:
- machine-translation
---
``` 
# Solution
The fix is to correctly style the README as explained [here](https://huggingface.co/docs/datasets/v1.12.0/dataset_card.html). I have also correctly parsed the font matter as shown below:
```
---
annotations_creators: []
language_creators: [machine-generated]
languages: ['ca', 'de']
licenses: []
multilinguality:
- multilingual
pretty_name: 'Catalan-German aligned corpora to train NMT systems.'
size_categories: 
- 1M<n<10M
source_datasets: ['extended|tilde_model']
task_categories: ['machine-translation']
task_ids: ['machine-translation']
---
```
You can find the README for a sample dataset [here](https://huggingface.co/datasets/ritwikraha/Test)

Thank you. It finally worked implementing your changes and leaving a white line between title and text in the description.

Thanks, if this solves your issue, can you please close it?
ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode: /content

## Bug
The install of streaming dataset is giving following error.

## Steps to reproduce the bug
```python
! git clone https://github.com/huggingface/datasets.git
! cd datasets
! pip install -e ".[streaming]"
```

## Actual results
Cloning into 'datasets'...
remote: Enumerating objects: 50816, done.
remote: Counting objects: 100% (2356/2356), done.
remote: Compressing objects: 100% (1606/1606), done.
remote: Total 50816 (delta 834), reused 1741 (delta 525), pack-reused 48460
Receiving objects: 100% (50816/50816), 72.47 MiB | 27.68 MiB/s, done.
Resolving deltas: 100% (22541/22541), done.
Checking out files: 100% (6722/6722), done.
ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode: /content



Hi! The `cd` command in Jupyer/Colab needs to start with `%`, so this should work:
```
!git clone https://github.com/huggingface/datasets.git
%cd datasets
!pip install -e ".[streaming]"
```

thanks @mariosasko i had the same mistake and your solution is what was needed
Add ImageNet toy datasets from fastai

Adds the ImageNet toy datasets from FastAI: Imagenette, Imagewoof and Imagewang.

TODOs:
* [ ] add dummy data
* [ ] add dataset card
* [ ] generate `dataset_info.json`

Thanks for your contribution, @mariosasko. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
fix multiple language downloading in mC4

If we try to access multiple languages of the [mC4 dataset](https://github.com/huggingface/datasets/tree/master/datasets/mc4), it will throw an error. For example, if we do 
```python
mc4_subset_two_langs = load_dataset("mc4", languages=["st", "su"])
```
we got
```
FileNotFoundError: Couldn't find file at https://huggingface.co/datasets/allenai/c4/resolve/1ddc917116b730e1859edef32896ec5c16be51d0/multilingual/c4-st+su.tfrecord-00000-of-00002.json.gz
```
Now it should work. Check it (from the root dir of a project): 
```python
mc4_subset_two_langs = load_dataset("./datasets/mc4/", languages=["st", "su"])
```

The CI failure is unrelated to your PR and fixed on master, merging :)
Add support for time, date, duration, and decimal dtypes

Add support for the pyarrow time (maps to `datetime.time` in python), date (maps to `datetime.time` in python), duration (maps to `datetime.timedelta` in python), and decimal (maps to `decimal.decimal` in python) dtypes. This should be helpful when writing scripts for time-series datasets. 

Is there a dataset which uses these four datatypes for tests purposes?


@severo Not yet. I'll let you know if that changes.
No module named 'fsspec.archive'

## Describe the bug
Cannot import datasets after installation.

## Steps to reproduce the bug
```shell
$ python
Python 3.9.7 (default, Sep 16 2021, 13:09:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import datasets
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/__init__.py", line 34, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 61, in <module>
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/arrow_writer.py", line 28, in <module>
    from .features import (
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/features/__init__.py", line 2, in <module>
    from .audio import Audio
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/features/audio.py", line 7, in <module>
    from ..utils.streaming_download_manager import xopen
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/utils/streaming_download_manager.py", line 18, in <module>
    from ..filesystems import COMPRESSION_FILESYSTEMS
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/filesystems/__init__.py", line 6, in <module>
    from . import compression
  File "/home/shuchen/miniconda3/envs/hf/lib/python3.9/site-packages/datasets/filesystems/compression.py", line 5, in <module>
    from fsspec.archive import AbstractArchiveFileSystem
ModuleNotFoundError: No module named 'fsspec.archive'
```
Revisit `enable/disable_` toggle function prefix

As discussed in https://github.com/huggingface/transformers/pull/15167, we should revisit the `enable/disable_` toggle function prefix, potentially in favor of `set_enabled_`. Concretely, this translates to

- De-deprecating `disable_progress_bar()`
- Adding `enable_progress_bar()`
- On the caching side, adding `enable_caching` and `disable_caching`

Additional decisions have to be made with regards to the existing `set_enabled_X` functions; that is, whether to keep them as is or deprecate them in favor of the aforementioned functions. 

cc @mariosasko @lhoestq
Datasets streaming + map doesn't work for `Audio`

## Describe the bug
When using audio datasets in streaming mode, applying a `map(...)` before iterating leads to an error as the key `array` does not exist anymore.

## Steps to reproduce the bug

```python
from datasets import load_dataset

ds = load_dataset("common_voice", "en", streaming=True, split="train")


def map_fn(batch):
    print("audio keys", batch["audio"].keys())
    batch["audio"] = batch["audio"]["array"][:100]
    return batch


ds = ds.map(map_fn) 

sample = next(iter(ds))
```

I think the audio is somehow decoded before `.map(...)` is actually called.

## Expected results

IMO, the above code snippet should work.

## Actual results

```bash
audio keys dict_keys(['path', 'bytes'])
Traceback (most recent call last):
  File "./run_audio.py", line 15, in <module>
    sample = next(iter(ds))
  File "/home/patrick/python_bin/datasets/iterable_dataset.py", line 341, in __iter__
    for key, example in self._iter():
  File "/home/patrick/python_bin/datasets/iterable_dataset.py", line 338, in _iter
    yield from ex_iterable
  File "/home/patrick/python_bin/datasets/iterable_dataset.py", line 192, in __iter__
    yield key, self.function(example)
  File "./run_audio.py", line 9, in map_fn
    batch["input"] = batch["audio"]["array"][:100]
KeyError: 'array'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.17.1.dev0
- Platform: Linux-5.3.0-64-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 6.0.1


This seems related to https://github.com/huggingface/datasets/issues/3505.
Add The Medical Segmentation Decathlon Dataset

## Adding a Dataset
- **Name:** *The Medical Segmentation Decathlon Dataset*
- **Description:** The underlying data set was designed to explore the axis of difficulties typically encountered when dealing with medical images, such as small data sets, unbalanced labels, multi-site data, and small objects. 
- **Paper:** [link to the dataset paper if available](https://arxiv.org/abs/2106.05735)
- **Data:** http://medicaldecathlon.com/
- **Motivation:** Hugging Face seeks to democratize ML for society. One of the growing niches within ML is the ML + Medicine community. Key data sets will help increase the supply of HF resources for starting an initial community.

(cc @osanseviero @abidlabs )

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Hello! I have recently been involved with a medical image segmentation project myself and was going through the `The Medical Segmentation Decathlon Dataset` as well. 
I haven't yet had experience adding datasets to this repository yet but would love to get started. Should I take this issue?
If yes, I've got two questions -
1. There are 10 different datasets available, so are all datasets to be added in a single PR, or one at a time? 
2. Since it's a competition, masks for the test-set are not available. How is that to be tackled? Sorry if it's a silly question, I have recently started exploring `datasets`.

Hi! Sure, feel free to take this issue. You can self-assign the issue by commenting `#self-assign`.

To answer your questions:
1. It makes the most sense to add each one as a separate config, so one dataset script with 10 configs in a single PR.
2. Just set masks in the test set to `None`.

Note that the images/masks in this dataset are in NIfTI format, which our `Image` feature currently doesn't support, so I think it's best to yield the paths to the images/masks in the script and add a preprocessing section to the card where we explain how to load/process the images/masks with `nibabel` (I can help with that).   



> Note that the images/masks in this dataset are in NIfTI format, which our `Image` feature currently doesn't support, so I think it's best to yield the paths to the images/masks in the script and add a preprocessing section to the card where we explain how to load/process the images/masks with `nibabel` (I can help with that).

Gotcha, thanks. Will start working on the issue and let you know in case of any doubt.

#self-assign

This is great! There is a first model on the HUb that uses this dataset! https://huggingface.co/MONAI/example_spleen_segmentation
label information get lost after parquet serialization

## Describe the bug
In *dataset_info.json* file, information about the label get lost after the dataset serialization.

## Steps to reproduce the bug
```python
from datasets import load_dataset

# normal save
dataset = load_dataset('glue', 'sst2', split='train')
dataset.save_to_disk("normal_save")


# save after parquet serialization
dataset.to_parquet("glue-sst2-train.parquet")
dataset = load_dataset("parquet", data_files='glue-sst2-train.parquet')
dataset.save_to_disk("save_after_parquet")
```

## Expected results
I expected to keep label information in *dataset_info.json* file even after parquet serialization

## Actual results
In the normal serialization i got
```json
"label": {
      "num_classes": 2,
      "names": [
        "negative",
        "positive"
      ],
      "names_file": null,
      "id": null,
      "_type": "ClassLabel"
    },
```
And after parquet serialization i got
```json
"label": {
      "dtype": "int64",
      "id": null,
      "_type": "Value"
    },
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.0
- Platform: ubuntu 20.04
- Python version: 3.8.10
- PyArrow version: 6.0.1


Hi ! We did a release of `datasets` today that may fix this issue. Can you try updating `datasets` and trying again ?

EDIT: the issue is still there actually

I think we can fix that by storing the Features in the parquet schema metadata, and then reload them when loading the parquet file

This info is stored in the Parquet schema metadata as of https://github.com/huggingface/datasets/pull/5516
Add The Mexican Emotional Speech Database (MESD)

## Adding a Dataset
- **Name:** *The Mexican Emotional Speech Database (MESD)*
- **Description:** *Contains 864 voice recordings with six different prosodies: anger, disgust, fear, happiness, neutral, and sadness. Furthermore, three voice categories are included: female adult, male adult, and child. *
- **Paper:** *[Paper](https://ieeexplore.ieee.org/abstract/document/9629934/authors#authors)*
- **Data:** *[link to the Github repository or current dataset location](https://data.mendeley.com/datasets/cy34mh68j9/3)*
- **Motivation:** *Would add Spanish speech data to the HF datasets :) *

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Add Mauve metric

Add support for the [Mauve](https://github.com/krishnap25/mauve) metric introduced in this [paper](https://arxiv.org/pdf/2102.01454.pdf) (Neurips, 2021).

Hi ! The CI was failing because `mauve-text` wasn't installed. I added it to the CI setup :)

I also did some minor changes to the script itself, especially to remove `**kwargs` and explicitly mentioned all the supported arguments (this way if someone does a typo with some parameters they get an error)
ConnectionError in IndicGLUE dataset

While I am trying to load IndicGLUE dataset (https://huggingface.co/datasets/indic_glue) it is giving me with the error:
```
ConnectionError: Couldn't reach https://storage.googleapis.com/ai4bharat-public-indic-nlp-corpora/evaluations/wikiann-ner.tar.gz (error 403)

@sahoodib, thanks for reporting.

Indeed, none of the data links appearing in the IndicGLUE website are working, e.g.: https://storage.googleapis.com/ai4bharat-public-indic-nlp-corpora/evaluations/soham-articles.tar.gz
```
<Error>
<Code>UserProjectAccountProblem</Code>
<Message>User project billing account not in good standing.</Message>
<Details>
The billing account for the owning project is disabled in state delinquent
</Details>
</Error>
```

We have contacted the data owners to inform them about their issue and ask them if they plan to fix it.

Yesterday I resent a reminder email with more AI4Bharat-related people in the loop.

I also opened an issue in their repos:
- https://github.com/AI4Bharat/indicnlp_corpus/issues/14
- https://github.com/AI4Bharat/ai4bharat.org/issues/71

We have received a reply from the authors reporting they have updated the URLs of their data files and opened a PR. See:
- #4978
Add the KMWP dataset (extension of #3564)

New pull request of #3564 (Add the KMWP dataset)

Sorry, I'm late to check! I'll send it to you soon!

Thanks for your contribution, @sooftware. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there, under this organization namespace: https://huggingface.co/tunib

Please, feel free to tell us if you need some help.

Close this PR. Thanks!
Add the DKTC dataset (Extension of #3564)

New pull request of #3564. (for DKTC)


I reflect your comment! @lhoestq 

Wait, the format of the data just changed, so I'll take it into consideration and commit it.

I update the code according to the dataset structure change.

Thanks ! I think the dummy data are not valid yet - the dummy train.csv file only contains a partial example (the quotes `"` start but never end).

> Thanks ! I think the dummy data are not valid yet - the dummy train.csv file only contains a partial example (the quotes `"` start but never end).

Hi! @lhoestq There is a problem.  
<img src="https://user-images.githubusercontent.com/42150335/149804142-3800e635-f5a0-44d9-9694-0c2b0c05f16b.png" width=500>
  
As shown in the picture above, the conversation is divided into "\n" in the "conversion" column.  
That's why there's an error in the file path that only saved only five lines like below. 

```
'idx', 'class', 'conversation'
'0', '협박 대화', '"지금 너 스스로를 죽여달라고 애원하는 것인가?'
아닙니다. 죄송합니다.'
죽을 거면 혼자 죽지 우리까지 사건에 휘말리게 해? 진짜 죽여버리고 싶게.'
정말 잘못했습니다.
```
   
In fact, these five lines are all one line.  
  



Hi ! I see, in this case ca you make sure that the dummy data has a full sample ?

Feel free to open the dummy train.csv in the dummy_data.zip file and add the missing lines

Sorry, I'm late to check! I'll send it to you soon!

Thanks for your contribution, @sooftware. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there, under this organization namespace: https://huggingface.co/tunib

Please, feel free to tell us if you need some help.

Close this PR. Thanks!
Fix push to hub to allow individual split push

# Description of the issue

If one decides to push a split on a datasets repo, he uploads the dataset and overrides the config. However previous config splits end up being lost despite still having the dataset necessary.

The new flow is the following:
 - query the old config from the repo
 - update into a new config (add/overwrite new split for example)
 - push the new config

# Side fix

 - `repo_id` in HfFileSystem was wrongly typed.
 - I've added `indent=2` as it becomes much easier to read now.



This has been addressed in https://github.com/huggingface/datasets/pull/4415. Closing.
Add parameter `preserve_index` to `from_pandas`

Added optional parameter, so that user can get rid of useless index preserving. [Issue](https://github.com/huggingface/datasets/issues/3563)

> 

I did `make style` and it affected over 500 files

```
All done! ✨ 🍰 ✨
575 files reformatted, 372 files left unchanged.
isort tests src benchmarks datasets/**/*.py metri
```

(result)
![image](https://user-images.githubusercontent.com/20703486/149166681-2f9d1bc4-116a-4f53-ad42-e54e3b8bd605.png)


Nvm I was using wrong black version
Dataset.from_pandas preserves useless index

## Describe the bug
Let's say that you want to create a Dataset object from pandas dataframe. Most likely you will write something like this:

```
import pandas as pd
from datasets import Dataset


df = pd.read_csv('some_dataset.csv')
# Some DataFrame preprocessing code...
dataset = Dataset.from_pandas(df)
```
If your preprocessing code contain indexing operations like this:
```
df = df[df.col1 == some_value]
```
then your df.index can be changed from (default) ```RangeIndex(start=0, stop=16590, step=1)``` to something like this ```Int64Index([    0,     1,     2,     3,     4,     5,     6,     7,     8,
                9,
            ...
            83979, 83980, 83981, 83982, 83983, 83984, 83985, 83986, 83987,
            83988],
           dtype='int64', length=16590)```

In this case, PyArrow (by default) will preserve this non-standard index. In the result, your dataset object will have the extra field that you likely don't want to have: '__index_level_0__'.

You can easily fix this by just adding extra argument ```preserve_index=False``` to call of ```InMemoryTable.from_pandas``` in ```arrow_dataset.py```.

If you approve that this isn't desirable behavior, I can make a PR fixing that.

## Environment info
- `datasets` version: 1.16.1
- Platform: Linux-5.11.0-44-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 6.0.1



Hi! That makes sense. Sure, feel free to open a PR! Just a small suggestion: let's make `preserve_index` a parameter of `Dataset.from_pandas` (which we then pass to `InMemoryTable.from_pandas`) with `None` as a default value to not have this as a breaking change.
Cannot load ‘bookcorpusopen’

## Describe the bug
Cannot load 'bookcorpusopen'


## Steps to reproduce the bug
```python
dataset = load_dataset('bookcorpusopen')
```
or
```python
dataset = load_dataset('bookcorpusopen',script_version='master')
```

## Actual results
ConnectionError: Couldn't reach https://the-eye.eu/public/AI/pile_preliminary_components/books1.tar.gz

## Environment info
- `datasets` version: 1.9.0
- Platform: Linux version 3.10.0-1160.45.1.el7.x86_64
- Python version:  3.6.13
- PyArrow version: 6.0.1


The host of this copy of the dataset (https://the-eye.eu) is down and has been down for a good amount of time ([potentially months](https://www.reddit.com/r/Roms/comments/q82s15/theeye_downdied/))

Finding this dataset is a little esoteric, as the original authors took down the official BookCorpus dataset some time ago.

There are community-created versions of BookCorpus, such as the files hosted in the link below.
https://battle.shawwn.com/sdb/bookcorpus/

And more discussion here:
https://github.com/soskek/bookcorpus

Do we want to remove this dataset entirely?  There's a fair argument for this, given that the official BookCorpus dataset was taken down by the authors.   If not, perhaps can open a PR with the link to the community-created tar above and updated dataset description.

Hi! The `bookcorpusopen` dataset is not working for the same reason as explained in this comment: https://github.com/huggingface/datasets/issues/3504#issuecomment-1004564980

Hi @HUIYINXUE, it should work now that the data owners created a mirror server with all data, and we updated the URL in our library.
Run pyupgrade for Python 3.6+

Run the command:

```bash
pyupgrade $(find . -name "*.py" -type f) --py36-plus
```

Which mainly avoids unnecessary lists creations and also removes unnecessary code for Python 3.6+.

It was originally part of #3489.

Tip for reviewing faster: use the CLI (`git diff`) and scroll.

Hi ! Thanks for the change :)
Could it be possible to only run it for the code in `src/` ? We try to not change the code in the `datasets/` directory too often since it refreshes the users cache when they upgrade `datasets`.

> Hi ! Thanks for the change :)
> Could it be possible to only run it for the code in `src/` ? We try to not change the code in the `datasets/` directory too often since it refreshes the users cache when they upgrade `datasets`.

I reverted the changes in `datasets/` instead of changing only `src/`. Does it sound good?

I just resolved some conflicts with the master branch. If the CI is green we can merge :)
Integrate Milvus (pymilvus) library

Milvus is a popular open-source vector database. We should add a new vector index to support this project.

Hi @mariosasko，Just search randomly and I found this issue~ I'm the tech lead of Milvus and we are looking forward to integrate milvus together with huggingface datasets.

Any suggestion on how we could start?


Feel free to assign to me and we probably need some guide on it

@mariosasko any updates my man?


Hi! For starters, I suggest you take a look at this file: https://github.com/huggingface/datasets/blob/master/src/datasets/search.py, which contains all the code for Faiss/ElasticSearch support. We could set up a Slack channel for additional guidance. Let me know what you prefer.

> Hi! For starters, I suggest you take a look at this file: https://github.com/huggingface/datasets/blob/master/src/datasets/search.py, which contains all the code for Faiss/ElasticSearch support. We could set up a Slack channel for additional guidance. Let me know what you prefer.

Sure, we take a look and do some research
Fix bug in `ImageClassifcation` task template

Fixes a bug in the `ImageClassification` task template which requires specifying class labels twice in dataset scripts. Additionally, this PR refactors the API around the classification task templates for cleaner `labels` handling.

CC: @lewtun @nateraw

The CI failures are unrelated to the changes in this PR.

> The CI failures are unrelated to the changes in this PR.

It seems that some of the failures are due to the tests on the dataset cards (e.g. CIFAR, MNIST, FASHION_MNIST). Perhaps it's worth addressing those in this PR to avoid confusing downstream developers who branch off `master` and suddenly have a failing CI?

@lewtun We only run these tests against the modified datasets on the PR branch, so this will not lead to errors after merging.
DuplicatedKeysError when loading tweet_qa dataset

When loading the tweet_qa dataset with `load_dataset('tweet_qa')`, the following error occurs: 

`DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 2a167f9e016ba338e1813fed275a6a1e
Keys should be unique and deterministic in nature
`
Might be related to issues #2433 and #2333

- `datasets` version: 1.17.0
- Python version: 3.8.5


Hi, we've just merged the PR with the fix. The fixed version of the dataset can be downloaded as follows:
```python
import datasets
dset = datasets.load_dataset("tweet_qa", revision="master")
```
set_format("np") no longer works for Image data

## Describe the bug
`dataset.set_format("np")` no longer works for image data, previously you could load the MNIST like this:

```python
dataset = load_dataset("mnist")
dataset.set_format("np")
X_train = dataset["train"]["image"][..., None] # <== No longer a numpy array
```
but now it doesn't work, `set_format("np")` seems to have no effect and the dataset just returns a list/array of PIL images instead of numpy arrays as requested.


A quick fix for now is doing this:

```python
X_train = np.stack(dataset["train"]["image"])[..., None]

This error also propagates to jax and is even trickier to fix, since `.with_format(type='jax')` will use numpy conversion internally (and fail). For a three line failure:

```python
dataset = datasets.load_dataset("mnist")
dataset.set_format("jax")
X_train = dataset["train"]["image"]
```

Hi! We've recently introduced a new Image feature that yields PIL Images (and caches transforms on them) instead of arrays.

However, this feature requires a custom transform to yield np arrays directly:
```python
ddict = datasets.load_dataset("mnist")

def pil_image_to_array(batch):
     return {"image": [np.array(img) for img in batch["image"]]}  # or jnp.array(img) for Jax

ddict.set_transform(pil_image_to_array, columns="image", output_all_columns=True)
```

[Docs](https://huggingface.co/docs/datasets/master/process.html#format-transform) on `set_transform`.

Also, the approach proposed by @cgarciae is not the best because it loads the entire column in memory.

@albertvillanova @lhoestq WDYT? The Audio and the Image feature currently don't support the TF/Jax/PT Formatters, but for the Numpy Formatter maybe it makes more sense to return np arrays (and not a dict in the case of the Audio feature or a PIL Image object in the case of the Image feature).

Yes I agree it should return arrays and not a PIL image (and possible an array instead of a dict for audio data).
I'm currently finishing some code refactoring of the image and audio and opening a PR today. Maybe we can look into that after the refactoring

This has been fixed in https://github.com/huggingface/datasets/pull/5072, which is included in the latest release of `datasets`.
Add more compression types for `to_json`

This PR adds `bz2`, `xz`, and `zip` (WIP) for `to_json`. I also plan to add `infer` like how `pandas` does it

@lhoestq, I looked into how to compress with `zipfile` for which few methods exist, let me know which one looks good:
1. create the file in normal `wb` mode and then zip it separately
2. use `ZipFile.write_str` to write file into the archive. For this we'll need to change how we're writing files from `_write` method  

How `pandas` handles it is that they have created a wrapper for standard library class `ZipFile` and allow the returned file-like handle to accept byte strings via `write` method instead of `write_str` (purpose was to change the name of function by creating that wrapper)

1. sounds not ideal since it creates an intermediary file.
I like pandas' approach. Is it possible to implement 2. using the pandas class ? Or maybe we can have something similar ?

Definitely, @lhoestq! I've adapted that from original code and turns out it is faster than `gz` compression. Apart from that I've also added `infer` option to automatically infer compression type from `path_or_buf` given

One small thing, currently I'm assuming that user will provide compression extension in `path_or_buf`. Is it this also possible?
`dataset.to_json("from_dataset.json", compression="zip")`? 
Should I put an `assert` to ensure the file name provided always has a compression extension?

Thanks !

> One small thing, currently I'm assuming that user will provide compression extension in path_or_buf. Is it this also possible?
>dataset.to_json("from_dataset.json", compression="zip")?
>Should I put an assert to ensure the file name provided always has a compression extension?

I think it's fine as it is right now :) No need to check the extension of the filename passed to `path_or_buf`.


> turns out it is faster than gz compression

I think the default compression level of `gzip` is 9 in python, which is very slow. Maybe we can switch to compression level 6 instead which is faster, like the `gzip` command on unix

I found that `fsspec` has something that may interest you: [fsspec.open(..., compression=...)](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.open). I don't remember if we've already mentioned it or not

It also has `zip` if I understand correctly ! see https://github.com/fsspec/filesystem_spec/blob/master/fsspec/compression.py#L70

Since `fsspec` is a dependency of `datasets` we can use all this :)

Let me know if you prefer using `fsspec` instead (I haven't tested this yet to write compressed files). IMO it sounds pretty easy to use and it would make the code base simpler

Just tried `fsspec` but I'm not able to write compressed `zip` files :/
`gzip`, `xz`, `bz2` are all working fine and it's really simple (no need for `FileWriteHandler` now!)
Bug in `openbookqa` dataset

## Describe the bug
Dataset entries contains a typo.

## Steps to reproduce the bug

```python
>>> from datasets import load_dataset
>>> obqa = load_dataset('openbookqa', 'main')
>>> obqa['train'][0]
```

## Expected results
```python
{'id': '7-980', 'question_stem': 'The sun is responsible for', 'choices': {'text': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting'], 'label': ['A', 'B', 'C', 'D']}, 'answerKey': 'D'}
```

## Actual results
```python
{'id': '7-980', 'question_stem': 'The sun is responsible for', 'choices': {'text': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting'], 'label': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting']}, 'answerKey': 'D'}
```

The bug is present in all configs and all splits.

## Environment info
- `datasets` version: 1.17.0
- Platform: Linux-5.4.0-1057-aws-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyArrow version: 4.0.1


Closed by:
- #4259
Fix sem_eval_2018_task_1 download location

This changes the download location of sem_eval_2018_task_1 files to include the test set labels as discussed in https://github.com/huggingface/datasets/issues/2745#issuecomment-954588500_ with @lhoestq.

Hi ! Thanks for pushing this :)

It seems that you created this PR from an old version of `datasets` that didn't have the sem_eval_2018_task_1.py file.

Can you try merging `master` into your branch ? Or re-create your PR from a branch that comes from a more recent version of `datasets` ?

And sorry for the late response !

Hi! No problem! I made the new branch like you said and opened https://github.com/huggingface/datasets/pull/3643 for it. I will close this one.
Specify the feature types of a dataset on the Hub without needing a dataset script

**Is your feature request related to a problem? Please describe.**
Currently if I upload a CSV with paths to audio files, the column type is string instead of Audio.

**Describe the solution you'd like**
I'd like to be able to specify the types of the column, so that when loading the dataset I directly get the features types I want.

The feature types could read from the `dataset_infos.json` for example.

**Describe alternatives you've considered**
Create a dataset script to specify the features, but that seems complicated for a simple thing.

cc @abidlabs 

After looking into this, discovered that this is already supported if the `dataset_infos.json` file is configured correctly! Here is a working example: https://huggingface.co/datasets/abidlabs/test-audio-13

This should be probably be documented, though.
fix: 🐛 pass token when retrieving the split names


Currently, it does not work with https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0/blob/main/common_voice_7_0.py#L146 (which was the goal), because `dl_manager.download_config.use_auth_token` is ignored, and the authentication is required to be use `huggingface-cli login`.
In my use case (dataset viewer), I'd prefer to use a specific "User Token Access", with only the "read" role (https://huggingface.co/settings/token).

See https://github.com/huggingface/datasets-preview-backend/issues/74#issuecomment-1007316853 for the context

> Simply passing download_config is ok :)

hmm, I prefer only passing use_auth_token. But the question is more: is it correct, in the (convoluted) case if `download_config.use_auth_token` exists and is different from `use_auth_token`? Which one should be used?

If both are passed, `use_auth_token` should have the priority (more specific parameters have the higher priority)
Ability to split a dataset in multiple files.

Hello,

**Is your feature request related to a problem? Please describe.**

My use case is that I have one writer that adds columns and multiple workers reading the same `Dataset`. Each worker should have access to columns added by the writer when they reload the dataset.

I understand that we shouldn't overwrite an arrow file as this could cause Segfault and so on. Before 1.16, I was able to overwrite the dataset and that would work most of the time with some retries.

**Describe the solution you'd like**

I was thinking that if we could append `Dataset._data_files`, when the workers reload the Dataset, they would get the new columns.


**Describe alternatives you've considered**
I currently need to
1. Save multiple "versions" of the dataset and load the latest.
2. Try working with cache files to get the latest columns.

**Additional context**

I think this would be a great addition to HFDataset as Parquet supports multi-files input out of the box!

I can make a PR myself with some pointers as needed :)
Allow loading community metrics from the hub, just like datasets

**Is your feature request related to a problem? Please describe.**
Currently, I can load a metric implemented by me by providing the local path to the file in `load_metric`.
However, there is no option to do it with the metric uploaded to the hub.
This means that if I want to allow other users to use it, they must download it first which makes the usage less smooth.

**Describe the solution you'd like**
Load metrics from the hub just like datasets are loaded.
In order to not break stuff, the convention can be to put the metric file in a "metrics" folder in the hub.



Hi ! Thanks for your message :) This is a great idea indeed. We haven't started working on this yet though. For now I guess you can host your metric on the Hub (either with your model or your dataset) and use `hf_hub_download` to download it (docs [here](https://github.com/huggingface/huggingface_hub/blob/main/docs/hub/how-to-downstream.md#cached_download))

This is a great solution in the meantime, thanks!

Here's the code I used, in case it can be of help to someone else:
```python
import os, shutil
from huggingface_hub import hf_hub_download
def download_metric(repo_id, file_path):
    # repo_id: for models "username/model_name", for datasets "datasets/username/model_name"
    local_metric_path = hf_hub_download(repo_id=repo_id, filename=file_path)
    updated_local_metric_path = (os.path.dirname(local_metric_path) + os.path.basename(local_metric_path).replace(".", "_") + ".py")
    shutil.copy(local_metric_path, updated_local_metric_path)
    return updated_local_metric_path

metric = load_metric(download_metric(REPO_ID, FILE_PATH))
```

Solved with https://github.com/huggingface/evaluate 🤗 

Yay!! cc @lvwerra @sashavor @douwekiela 

Please share your feedback @eladsegal =)
How to convert  torch.utils.data.Dataset to datasets.arrow_dataset.Dataset?

Hi,
I use torch.utils.data.Dataset to define my own data, but I need to use the 'map' function of datasets.arrow_dataset.Dataset later, so I hope to convert torch.utils.data.Dataset to datasets.arrow_dataset.Dataset.

Here is an example.
```
from torch.utils.data import Dataset
from datasets.arrow_dataset import Dataset as HFDataset

class ADataset(Dataset):
    def __init__(self, data): 
        super().__init__()
        self.data = data
    def __getitem__(self, index):
        return self.data[index]
    def __len__(self):
        return self.len
class MDataset():
    def __init__(self, tokenizer: AutoTokenizer, data_args, training_args):
        self.train_dataset = ADataset(data_args)
        self.tokenizer = tokenizer
        self.data_args = data_args

        self.train_dataset = self.train_dataset.map(
            self.process_function,
            batched=True,
            remove_columns=column_names,
            load_from_cache_file=True,
            desc="Running tokenizer on train dataset",
             )
    def process_function(self, examples):
        sentences = [" ".join(sample[0][3]) for sample in examples]

        tokenized = self.tokenizer(
            sentences,
            max_length=self.max_seq_len,
            padding=self.padding, 
            truncation=True)
```
But it would raise an ERROR, AttributeError: 'ADataset' object has no attribute 'map'.

so how to convert  torch.utils.data.Dataset to datasets.arrow_dataset.Dataset?

Thanks in advance!