KRadim/github-issues-minimal huggingface dataset:
Audiofolder creates empty Dataset if files same level as metadata
State: closed
Labels: name: bug, name: good first issue, name: hacktoberfest

## Describe the bug
When audio files are at the same level as the metadata (`metadata.csv` or `metadata.jsonl` ), the `load_dataset` returns a `DatasetDict` with no rows but the correct columns.

https://github.com/huggingface/datasets/blob/1ea4d091b7a4b83a85b2eeb8df65115d39af3766/docs/source/audio_dataset.mdx?plain=1#L88

## Steps to reproduce the bug
`metadata.csv`:
```csv
file_name,duration,transcription
./2063_fe9936e7-62b2-4e62-a276-acbd344480ce_1.wav,10.768,hello
```


```python
>>> audio_dataset = load_dataset("audiofolder", data_dir="/audio-data/")
>>> audio_dataset
DatasetDict({
    train: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
})
```

I've tried, with no success,:
- setting `split` to something else so I don't get a `DatasetDict`,
- removing the `./`,
- using `.jsonl`.


## Expected results
```
 Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 1
})
```

## Actual results
```
DatasetDict({
    train: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
})
```

## Environment info
- `datasets` version: 2.5.1
- Platform: Linux-5.13.0-1025-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.5.0
Automatically revert to last successful commit to hub when a push_to_hub is interrupted
State: closed
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
I pushed a modification of a large dataset (remove a column) to the hub. The push was interrupted after some files were committed to the repo. This left the dataset to raise an error on load_dataset() (ValueError couldn’t cast … because column names don’t match). Only by specifying the previous (complete) commit as revision=commit_hash in load_data(), I was able to repair this and after a successful, complete push, the dataset loads without error again.

**Describe the solution you'd like**
Would it make sense to detect an incomplete push_to_hub() and automatically revert to the previous commit/revision?

**Describe alternatives you've considered**
Leave everything as is, the revision parameter in load_dataset() allows to manually fix this problem.

**Additional context**
Provide useful defaults
Hendrycks Checksum
State: closed
Labels: name: dataset bug

Hi, 
The checksum for [hendrycks_test](https://huggingface.co/datasets/hendrycks_test) does not compare correctly, I guess it has been updated on the remote.

```
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://people.eecs.berkeley.edu/~hendrycks/data.tar']
```
`Dataset.unique` showing wrong output after filtering
State: closed
Labels: name: bug

## Describe the bug
After filtering a dataset, and if no samples remain, `Dataset.unique` will return the unique values of the unfiltered dataset.

## Steps to reproduce the bug
```python
from datasets import Dataset

dataset = Dataset.from_dict({'id': [0]})
dataset = dataset.filter(lambda _: False)
print(dataset.unique('id'))
```

## Expected results
The above code should return an empty list since the dataset is empty.

## Actual results
```bash
[0]
```

## Environment info
- `datasets` version: 2.5.1
- Platform: Linux-5.18.19-100.fc35.x86_64-x86_64-with-glibc2.34
- Python version: 3.9.14
- PyArrow version: 7.0.0
- Pandas version: 1.3.5
Add oversampling strategy iterable datasets interleave
State: closed

Hello everyone,
Following the issue #4893 and the PR #4831, I propose here an oversampling strategy for a `IterableDataset` list.  
The `all_exhausted` strategy stops building the new dataset as soon as all samples in each dataset have been added at least once. 
It follows roughly the same logic behind #4831, namely:
- if ``probabilities`` is `None` and the strategy is `all_exhausted`, it simply performs a round robin interleaving that stops when the longest dataset is out of samples. Here the new dataset length will be $maxLengthDataset*nbDataset$.
- if ``probabilities`` is not `None` and the strategy is `all_exhausted`, it keeps trace of the datasets which were out of samples but continues to add them to the new dataset, and stops as soons as every dataset runs out of samples at least once.

In order to be consistent and also to align with the `Dataset` behavior, please note that the behavior of the default strategy (`first_exhausted`) has been changed. Namely, it really stops when a dataset is out of samples whereas it used to stop when receiving the `StopIteration` error.  
To give an example of the last note, with the following snippet:
```
>>> from tests.test_iterable_dataset import *
>>> d1 = IterableDataset(ExamplesIterable((lambda: (yield from [(i, {"a": i}) for i in [0, 1, 2]])), {}))
>>> d2 = IterableDataset(ExamplesIterable((lambda: (yield from [(i, {"a": i}) for i in [10, 11, 12, 13]])), {}))
>>> d3 = IterableDataset(ExamplesIterable((lambda: (yield from [(i, {"a": i}) for i in [20, 21, 22, 23, 24]])), {}))
>>> dataset = interleave_datasets([d1, d2, d3])
>>> [x["a"] for x in dataset]
```
The result here will then be `[10, 0, 11, 1, 2]` instead of `[10, 0, 11, 1, 2, 20, 12, 13]`.

I modified the behavior because I found it to be consistent with the under/oversampling approach and because it unified the undersampling and oversampling code, but I stay open to any suggestions.
new dataset type: single-label and multi-label video classification
State: open
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
In my research, I am dealing with multi-modal (audio+text+frame sequence) video classification. It would be great if the datasets library supported generating multi-modal batches from a video dataset.

**Describe the solution you'd like**
Assume I have video files having single/multiple labels. I want to train a single/multi-label video classification model. I want datasets to support generating multi-modal batches (audio+frame sequence) from video files. Audio waveform and frame sequence can be extracted from each video clip then I can use any audio, image and video model from transformers library to extract features which will be fed into my model.

**Describe alternatives you've considered**
Currently, I am using https://github.com/facebookresearch/pytorchvideo dataloaders. There seems to be not much alternative.

**Additional context**
I am wiling to open a PR but don't know where to start.
Support hfh 0.10 implicit auth
State: closed

In huggingface-hub 0.10 the `token` parameter is deprecated for dataset_info and list_repo_files in favor of use_auth_token.
Moreover if use_auth_token=None then the user's token is used implicitly.

I took those two changes into account

Close https://github.com/huggingface/datasets/issues/4990

TODO:
- [x] fix tests

We should wait hfh 0.10 to be relased first to make sure it works correctly before merging
Fast dataset iter
State: closed

Use `pa.Table.to_reader` to make iteration over examples/batches faster in `Dataset.{__iter__, map}`

TODO:
* [x] benchmarking (the only benchmark for now - iterating over (single) examples of `bookcorpus` (75 mil examples) in Colab is approx. 2.3x faster)
* [x] check if iterating over bigger chunks + slicing to fetch individual examples in `_iter` yields better performance
passing parameters to the method passed to Dataset.from_generator()
State: closed
Labels: name: enhancement

Big thanks for providing dataset creation via a generator.
I want to ask whether there is any way that parameters can be passed to the method Dataset.from_generator() method, like as follows. 

```
from datasets import Dataset

def gen(param1):
    for idx in len(custom_dataset):
        yield custom_dataset[idx] + param1

ds = Dataset.from_generator(gen(param1))
```
patch CI_HUB_TOKEN_PATH with Path instead of str
State: closed

Should fix the tests for `huggingface_hub==0.10.0rc0` prerelease (see [failed CI](https://github.com/huggingface/datasets/actions/runs/3127805250/jobs/5074879144)). 
Related to [this thread](https://huggingface.slack.com/archives/C02V5EA0A95/p1664195165294559) (internal link).

Note: this should be a backward compatible fix (e.g. works also with previous versions of `huggingface_hub`)

I am not sure where to put the changes so feel free to cherry-pick the commit and close this one without merging.

cc @lhoestq
Text strings are split into lists of characters in xcsr dataset
State: closed
Labels: name: dataset bug

## Describe the bug
Text strings are split into lists of characters.

Example for "X-CSQA-en":
```
{'id': 'd3845adc08414fda',
 'lang': 'en',
 'question': {'stem': ['T',
   'h',
   'e',
   ' ',
   'd',
   'e',
   'n',
   't',
   'a',
   'l',
   ' ',
   'o',
   'f',
   'f',
   'i',
   'c',
   'e',
   ' ',
   'h',
   'a',
   'n',
   'd',
   'l',
   'e',
   'd',
   ' ',
   'a',
   ' ',
   'l',
   'o',
   't',
   ' ',
   'o',
   'f',
   ' ',
   'p',
   'a',
   't',
   'i',
   'e',
   'n',
   't',
   's',
   ' ',
   'w',
   'h',
   'o',
   ' ',
   'e',
   'x',
   'p',
   'e',
   'r',
   'i',
   'e',
   'n',
   'c',
   'e',
   'd',
   ' ',
   't',
   'r',
   'a',
   'u',
   'm',
   'a',
   't',
   'i',
   'c',
   ' ',
   'm',
   'o',
   'u',
   't',
   'h',
   ' ',
   'i',
   'n',
   'j',
   'u',
   'r',
   'y',
   ',',
   ' ',
   'w',
   'h',
   'e',
   'r',
   'e',
   ' ',
   'w',
   'e',
   'r',
   'e',
   ' ',
   't',
   'h',
   'e',
   's',
   'e',
   ' ',
   'p',
   'a',
   't',
   'i',
   'e',
   'n',
   't',
   's',
   ' ',
   'c',
   'o',
   'm',
   'i',
   'n',
   'g',
   ' ',
   'f',
   'r',
   'o',
   'm',
   '?'],
  'choices': [{'label': ['A'], 'text': ['t', 'o', 'w', 'n']},
   {'label': ['B'], 'text': ['m', 'i', 'c', 'h', 'i', 'g', 'a', 'n']},
   {'label': ['C'], 'text': ['h', 'o', 's', 'p', 'i', 't', 'a', 'l']},
   {'label': ['D'], 'text': ['s', 'c', 'h', 'o', 'o', 'l', 's']},
   {'label': ['E'],
    'text': ['o',
     'f',
     'f',
     'i',
     'c',
     'e',
     ' ',
     'b',
     'u',
     'i',
     'l',
     'd',
     'i',
     'n',
     'g']}]},
 'answerKey': 'C'}

## Steps to reproduce the bug
```python
ds = load_dataset("datasets/xcsr", "X-CSQA-en", split="validation", streaming=True)
item = next(iter(ds))
item
```

## Expected results
```
{'id': 'd3845adc08414fda',
 'lang': 'en',
 'question': {'stem': 'The dental office handled a lot of patients who experienced traumatic mouth injury, where were these patients coming from?',
  'choices': {'label': ['A', 'B', 'C', 'D', 'E'],
   'text': ['town', 'michigan', 'hospital', 'schools', 'office building']}},
 'answerKey': 'C'}
```
Split is inferred from filename and overrides metadata.jsonl
State: closed
Labels: name: bug, name: duplicate

## Describe the bug

Including the strings "test" or "train" anywhere in a filename causes `datasets` to infer the split and silently ignore all other files.

This behavior is documented for directory names but not filenames: https://huggingface.co/docs/datasets/image_dataset#imagefolder

## Steps to reproduce the bug
`metadata.jsonl`
```json
{"file_name": "photo of a cat.jpg", "text": "a photo of a cat"}
{"file_name": "photo of a dog.jpg", "text": "a photo of a dog"}
{"file_name": "photo of a train.jpg", "text": "a photo of a train"}
{"file_name": "photo of test tubes.jpg", "text": "a photo of test tubes"}
```

`bug.py`
```python
from datasets import load_dataset

dataset = load_dataset("dataset")

print(dataset)
# DatasetDict({
#     train: Dataset({
#         features: ['image', 'text'],
#         num_rows: 1
#     })
#     test: Dataset({
#         features: ['image', 'text'],
#         num_rows: 1
#     })
# })

for split in dataset:
    for n in dataset[split]:
        print(n['text'])
        # a photo of a train
        # a photo of test tubes
```

## Expected results
One single dataset with all four images / a warning for unused files / documentation of this behavior

## Actual results
Only the images with "test" or "train" in the name are loaded

## Environment info
- `datasets` version: 2.5.1
- Platform: macOS-12.5.1-x86_64-i386-64bit
- Python version: 3.10.4
- PyArrow version: 9.0.0
- Pandas version: 1.5.0
Fix URLs of sbu_captions dataset
State: closed
Labels: name: dataset contribution

Forbidden
You don't have permission to access /~vicente/sbucaptions/sbu-captions-all.tar.gz on this server.

Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.

Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/5.4.16 mod_fcgid/2.3.9 mod_wsgi/3.4 Python/2.7.5 mod_perl/2.0.11 Perl/v5.16.3 Server at [www.cs.virginia.edu](mailto:csroot@virginia.edu) Port 443
Update swiss judgment prediction
State: closed
Labels: name: dataset contribution

Hi,

I updated the dataset to include additional data made available recently. When I test it locally, it seems to work. However, I get the following error with the dummy data creation: 
`Dummy data generation done but dummy data test failed since splits ['train', 'validation', 'test'] have 0 examples for config 'fr'`. Do you know why this could be the case?

Cheers,
Joel
Create all YAML dataset_info
State: closed
Labels: name: dataset contribution

Following https://github.com/huggingface/datasets/pull/4926

Creates all the `dataset_info` YAML fields in the dataset cards
The JSON are also updated using the simplified backward compatible format added in https://github.com/huggingface/datasets/pull/4926

Needs https://github.com/huggingface/datasets/pull/4926 to be merged first
xcsr: X-CSQA simply uses english for all alleged non-english data
State: closed
Labels: name: dataset bug

## Describe the bug
All the alleged non-english subcollections for the X-CSQA task in the [xcsr benchmark dataset ](https://huggingface.co/datasets/xcsr) seem to be copies of the english subcollection, rather than translations. This is in contrast to the data description:

> we automatically translate the original CSQA and CODAH datasets, which only have English versions, to 15 other languages, forming development and test sets for studying X-CSR
 
## Steps to reproduce the bug
```python
# let's say you want to load the french X-CSQA subcollection
french = datasets.load_dataset("xcsr", "X-CSQA-fr")
# for good measure, let's load english too
english = datasets.load_dataset("xcsr", "X-CSQA-en")

# let's inspect
"".join(english['test'][0]['question']['stem'])
# output: 'The people wanted to stop the parade, so what did they set up to thwart it?'
"".join(french['test'][0]['question']['stem'])
# output: 'The people wanted to stop the parade, so what did they set up to thwart it?'
# what? Why are they both in english?

# I've checked this for validation and train splits too, across many datapoints. It's all the same english dataset
# maybe i need to look better?
french['test'].unique('lang')
# output: ['en']
# no, it's all english

```

## Expected results
Accessing a subcollection in language X should return a subcollection containg samples in language X

## Actual results
Accessing a subcollection in language X returns a subcollection containing samples in English.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 2.5.1
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.3
Fix tar extraction vuln
State: closed

Fix for CVE-2007-4559

Description:
Directory traversal vulnerability in the (1) extract and (2) extractall functions in the tarfile
module in Python allows user-assisted remote attackers to overwrite arbitrary files via a .. (dot dot)
sequence in filenames in a TAR archive, a related issue to CVE-2001-1267.

I fixed it by using the solution proposed in https://stackoverflow.com/questions/10060069/safely-extract-zip-or-tar-using-python

It blocks extraction of files with an absolute path or double dots and symlinks.
Transfer dataset scripts to Hub
State: closed

Before merging:
- #4974 

TODO:
- [x] Create label: ["dataset contribution"](https://github.com/huggingface/datasets/pulls?q=label%3A%22dataset+contribution%22)
- [x] Create project: [Datasets: Transfer datasets to Hub](https://github.com/orgs/huggingface/projects/22/)
- [x] PRs:
  - [x] Add dataset: we should recommend transfer all additions of datasets to the Hub, under the appropriate namespace; no more additions of datasets on GitHub
  - [x] Update dataset: in general, we should merge bug fixes; enhancements should be considered on a case-by-case basis, depending on whether there is a more suitable namespace on the Hub
- [ ] Issues

Finally:
- [x] #4974

Let me know what you think! :hugs:
Force JSON format regardless of file naming on S3
State: closed
Labels: name: enhancement

I have a file on S3 created by Data Version Control, it looks like `s3://dvc/ac/badff5b134382a0f25248f1b45d7b2` but contains a json file. If I run 
```python
dataset = load_dataset(
    "json",
    data_files='s3://dvc/ac/badff5b134382a0f25248f1b45d7b2' 
)
```
It gives me 
```
InvalidSchema: No connection adapters were found for 's3://dvc/ac/badff5b134382a0f25248f1b45d7b2'
```
However, I cannot go ahead and change the names of the s3 file. Is there a way to "force" load a S3 url with certain decoder (JSON, CSV, etc.) regardless of s3 URL naming?
Release 2.5.0 breaks transformers CI
State: closed
Labels: name: bug

## Describe the bug
As reported by @lhoestq:

> see https://app.circleci.com/pipelines/github/huggingface/transformers/47634/workflows/b491886b-e66e-4edb-af96-8b459e72aa25/jobs/564563
this is used here: [https://github.com/huggingface/transformers/blob/3b19c0317b6909e2d7f11b5053895ac55[…]torch/speech-pretraining/run_wav2vec2_pretraining_no_trainer.py](https://github.com/huggingface/transformers/blob/3b19c0317b6909e2d7f11b5053895ac55250e7da/examples/pytorch/speech-pretraining/run_wav2vec2_pretraining_no_trainer.py#L482-L488)
Add EmptyDatasetError
State: closed

examples:

from the hub:
```python
Traceback (most recent call last):
  File "playground/ttest.py", line 3, in <module>
    print(load_dataset("lhoestq/empty"))
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1686, in load_dataset
    **config_kwargs,
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1458, in load_dataset_builder
    data_files=data_files,
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1171, in dataset_module_factory
    raise e1 from None
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1162, in dataset_module_factory
    download_mode=download_mode,
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 760, in get_module
    else get_data_patterns_in_dataset_repository(hfh_dataset_info, self.data_dir)
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/data_files.py", line 678, in get_data_patterns_in_dataset_repository
    ) from None
datasets.data_files.EmptyDatasetError: The dataset repository at 'lhoestq/empty' doesn't contain any data file.
```

from local directory:
```python
Traceback (most recent call last):
  File "playground/ttest.py", line 3, in <module>
    print(load_dataset("playground/empty"))
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1686, in load_dataset
    **config_kwargs,
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1458, in load_dataset_builder
    data_files=data_files,
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1107, in dataset_module_factory
    path, data_dir=data_dir, data_files=data_files, download_mode=download_mode
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 625, in get_module
    else get_data_patterns_locally(base_path)
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/data_files.py", line 460, in get_data_patterns_locally
    raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data file") from None
datasets.data_files.EmptyDatasetError: The directory at playground/empty doesn't contain any data file
```

Close https://github.com/huggingface/datasets/issues/4995
Don't add a tag on the Hub on release
State: closed

Datasets with no namespace on the Hub have tags to redirect to the version of datasets where they come from.
I’m about to remove them all because I think it looks bad/unexpected in the UI and it’s not actually useful

Therefore I'm also disabling tagging.

Note that the CI job will be completely removed in https://github.com/huggingface/datasets/pull/4974 anyway
Add support for parsing JSON files in array form
State: closed

Support parsing JSON files in the array form (top-level object is an array). For simplicity, `json.load` is used for decoding. This means the entire file is loaded into memory. If requested, we can optimize this by introducing a param similar to `lines` in [`pandas.read_json`](https://pandas.pydata.org/docs/reference/api/pandas.read_json.html), which, if set to `True`, would allow us to read in chunks.

Fixes https://github.com/huggingface/datasets/issues/4963
Dataset Viewer issue for Jean-Baptiste/wikiner_fr
State: closed

### Link

https://huggingface.co/datasets/Jean-Baptiste/wikiner_fr

### Description

```
Error code:   StreamingRowsError
Exception:    FileNotFoundError
Message:      [Errno 2] No such file or directory: 'zip:/data/train::https:/huggingface.co/datasets/Jean-Baptiste/wikiner_fr/resolve/main/data.zip/state.json'
Traceback:    Traceback (most recent call last):
               File "/src/services/worker/src/worker/responses/first_rows.py", line 337, in get_first_rows_response
                 rows = get_rows(dataset, config, split, streaming=True, rows_max_number=rows_max_number, hf_token=hf_token)
               File "/src/services/worker/src/worker/utils.py", line 123, in decorator
                 return func(*args, **kwargs)
               File "/src/services/worker/src/worker/responses/first_rows.py", line 77, in get_rows
                 rows_plus_one = list(itertools.islice(ds, rows_max_number + 1))
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 718, in __iter__
                 for key, example in self._iter():
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 708, in _iter
                 yield from ex_iterable
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 112, in __iter__
                 yield from self.generate_examples_fn(**self.kwargs)
               File "/tmp/modules-cache/datasets_modules/datasets/Jean-Baptiste--wikiner_fr/683a580ba6ec769d508f7dfc603a651667b0ed3817b1ae5bfd45f97cc024923f/wikiner_fr.py", line 165, in _generate_examples
                 dataset = Dataset.load_from_disk(filepath)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1210, in load_from_disk
                 with open(Path(dataset_path, config.DATASET_STATE_JSON_FILENAME).as_posix(), encoding="utf-8") as state_file:
             FileNotFoundError: [Errno 2] No such file or directory: 'zip:/data/train::https:/huggingface.co/datasets/Jean-Baptiste/wikiner_fr/resolve/main/data.zip/state.json'
```

Is it an error with the dataset script, or the data itself, @huggingface/datasets?

https://huggingface.co/datasets/Jean-Baptiste/wikiner_fr/tree/main 

### Owner

No
Get a specific Exception when the dataset has no data
State: closed
Labels: name: enhancement, name: dataset-viewer

In the dataset viewer on the Hub (https://huggingface.co/datasets/glue/viewer), we would like (https://github.com/huggingface/moon-landing/issues/3882) to show a specific message when the repository lacks any data files.

In that case, instead of showing a complex traceback, we want to show a call to action to help the user upload data.

To do that, it would be very helpful to know for sure that the repository is missing any (supported) data files.

It could be done by raising a custom exception, for example, `NoDataError`.
delete the hardcoded license list in `datasets`
State: closed

> Feel free to delete the license list in `datasets` [...]
>
> Also FYI in #4926 I also removed all the validation steps anyway (language, license, types etc.)

_Originally posted by @lhoestq in https://github.com/huggingface/datasets/issues/4930#issuecomment-1238401662_

> [...], in my opinion we can just delete this file from `datasets`, the validation is happening hub-side anyways now?

_Originally posted by @julien-c in https://github.com/huggingface/datasets/issues/4930#issuecomment-1238390659_
Fix missing tags in dataset cards
State: closed

Fix missing tags in dataset cards:
- aeslc
- empathetic_dialogues
- event2Mind
- gap
- iwslt2017
- newsgroup
- qa4mre
- scicite

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921
- #4931
- #4979
"no-token" is passed to `huggingface_hub` when token is `None`
State: closed
Labels: name: bug

## Describe the bug

In the 2 lines listed below, a token is passed to `huggingface_hub` to get information from a dataset. If no token is provided, a "no-token" string is passed. What is the purpose of it ? If no real, I would prefer if the `None` value could be sent directly to be handle by `huggingface_hub`. I feel that here it is working because we assume the token will never be validated.

https://github.com/huggingface/datasets/blob/5b23f58535f14cc4dd7649485bce1ccc836e7bca/src/datasets/load.py#L753
https://github.com/huggingface/datasets/blob/5b23f58535f14cc4dd7649485bce1ccc836e7bca/src/datasets/load.py#L1121

## Expected results
Pass `token=None` to `huggingface_hub`.

## Actual results
`token="no-token"` is passed.

## Environment info
`huggingface_hub v0.10.0dev`
Running add_column() seems to corrupt existing sequence-type column info
State: closed
Labels: name: bug

I have a dataset that contains a column ("foo") that is a sequence type of length 4. So when I run .to_pandas() on it, the resulting dataframe correctly contains 4 columns - foo_0, foo_1, foo_2, foo_3. So the 1st row of the dataframe might look like:

ds = load_dataset(...)
df = ds.to_pandas()

df:
foo_0 | foo_1 | foo_2 | foo_3
0.0 | 1.0 | 2.0 | 3.0

If I run .add_column("new_col", data) on the dataset, and then .to_pandas() on the resulting new dataset, the resulting dataframe contains only 2 columns - foo, new_col. The values in column foo are lists of length 4, the 4 elements that should have been split into separate columns. Dataframe 1st row would be:

ds = load_dataset(...)
new_ds = ds.add_column("new_col", data)
df = new_ds.to_pandas()

df:
foo | new_col
[0.0, 1.0, 2.0, 3.0] | new_val

I've explored the 2 datasets in a debugger and haven't noticed any changes to any attributes related to the foo column, but I can't determine why the dataframes are so different.
[doc] Fix broken snippet that had too many quotes
State: closed

Hello!

### Pull request overview
* Fix broken snippet in https://huggingface.co/docs/datasets/main/en/process that has too many quotes

### Details
The snippet in question can be found here: https://huggingface.co/docs/datasets/main/en/process#map
This screenshot shows the issue, there is a quote too many, causing the snippet to be colored incorrectly:
![image](https://user-images.githubusercontent.com/37621491/190640627-f7587362-0e44-4464-a5d1-a0b98df6986f.png)

The change speaks for itself.

Thank you for the detailed documentation, by the way. 

- Tom Aarsen
docs: ✏️ add links to the Datasets API
State: closed

I added some links to the Datasets API in the docs. See https://github.com/huggingface/datasets-server/pull/566 for a companion PR in the datasets-server. The idea is to improve the discovery of the API through the docs.

I'm a bit shy about pasting a lot of links to the API in the docs, so it's minimal for now. I'm interested in ideas to integrate the API better in these docs without being too much. cc @lhoestq @julien-c @albertvillanova @stevhliu.
How to convert  torch.utils.data.Dataset to huggingface dataset?
State: closed
Labels: name: enhancement

I look through the huggingface dataset docs, and it seems that there is no offical support function to  convert  `torch.utils.data.Dataset`  to huggingface dataset. However, there is a way to convert huggingface dataset to  `torch.utils.data.Dataset`, like below:
```python
from datasets import Dataset
data = [[1, 2],[3, 4]]
ds = Dataset.from_dict({"data": data})
ds = ds.with_format("torch")
ds[0]
ds[:2]
```
So is there something I miss, or there IS no function to convert  `torch.utils.data.Dataset` to huggingface dataset. If so, is there any way to do this convert?
Thanks.
Fix missing tags in dataset cards
State: closed

Fix missing tags in dataset cards:
- amazon_us_reviews
- art
- discofuse
- indic_glue
- ubuntu_dialogs_corpus

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921
- #4931
Providing dataset size
State: open
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
Especially for big datasets like [LAION](https://huggingface.co/datasets/laion/laion2B-en/), it's hard to know exactly the downloaded size (because there are many files and you don't have their exact size when downloaded).

**Describe the solution you'd like**
Auto-populating the downloaded dataset size on the dataset page would be really useful, including that of each split (when there are some).

**Describe alternatives you've considered**
People should be adding this to dataset cards, but I don't think that is systematically the case :slightly_smiling_face: 

**Additional context**
Mentioned to @lhoestq
Hope to adapt Python3.9 as soon as possible
State: open
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context about the feature request here.
[GH->HF] Load datasets from the Hub
State: closed

Currently datasets with no namespace (e.g. squad, glue) are loaded from github.

In this PR I changed this logic to use the Hugging Face Hub instead.

This is the first step in removing all the dataset scripts in this repository

related to discussions in https://github.com/huggingface/datasets/pull/4059 (I should have continued from this PR actually)
Preserve non-`input_colums` in `Dataset.map` if `input_columns` are specified
State: closed

Currently, if the `input_columns` list in `Dataset.map` is specified, the columns not in that list are dropped after the `map` transform.

This makes the behavior inconsistent with `IterableDataset.map`.
 
(It seems this issue was introduced by mistake in https://github.com/huggingface/datasets/pull/2246) 

Fix https://github.com/huggingface/datasets/issues/4858
Fix data URL and metadata of vivos dataset
State: closed

After contacting the authors of the VIVOS dataset to report that their data server is down, we have received a reply from Hieu-Thi Luong that their data is now hosted on Zenodo: https://doi.org/10.5281/zenodo.7068130

This PR updates their data URL and some metadata (homepage, citation and license).

Fix #4936.
Column of arrays (2D+) are using unreasonably high memory
State: open
Labels: name: bug

## Describe the bug
When trying to store `Array2D, Array3D, etc` as column values in a dataset, accessing that column (or creating depending on how you create it, see code below) will cause more than 10 fold of memory usage.

## Steps to reproduce the bug
```python
from datasets import Dataset, Features, Array2D, Array3D
import numpy as np

column_name = "a"
array_shape = (64, 64, 3)

data = np.random.random((10000,) + array_shape)
dataset = Dataset.from_dict({column_name: data}, features=Features({column_name: Array3D(shape=array_shape, dtype="float64")}))
```

the code above will use about 10Gb of RAM while constructing the `dataset` object.

The code below will use roughly the same amount of memory (and time) when trying to actually access the data itself of that column.
```python
from datasets import Dataset
import numpy as np

column_name = "a"
array_shape = (64, 64, 3)

data = np.random.random((10000,) + array_shape)
dataset = Dataset.from_dict({column_name: data})
dataset[column_name]
```

## Expected results
Some memory overhead, but not like as it is now and certainly not an overhead of such runtime that is currently happening.

## Actual results
Enormous memory- and runtime overhead.

## Environment info
- `datasets` version: 2.3.2
- Platform: macOS-12.5.1-arm64-arm-64bit
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.4
Dataset without script does not support regular JSON data file
State: closed

### Link

https://huggingface.co/datasets/julien-c/label-studio-my-dogs

### Description

<img width="1115" alt="image" src="https://user-images.githubusercontent.com/326577/189422048-7e9c390f-bea7-4521-a232-43f049ccbd1f.png">


### Owner

Yes
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.4.0/datasets/jsonl/jsonl.py
State: closed
Labels: name: bug

Hi,
When I use load_dataset from local jsonl files, below error happens, and I type the link into the browser prompting me `404: Not Found`. I download the other `.py` files using the same method and it works. It seems that the server is missing the appropriate file, or it is a problem with the code version.

```
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.3.0/datasets/jsonl/jsonl.py (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /huggingface/datasets/2.3.0/datasets/jsonl/jsonl.py (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x2b08342004c0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))")))

```
Raise a more precise error when the URL is unreachable in streaming mode
State: open
Labels: name: enhancement

See for example:

- https://github.com/huggingface/datasets/issues/3191
- https://github.com/huggingface/datasets/issues/3186

It would help provide clearer information on the Hub and help the dataset maintainer solve the issue by themselves quicker. Currently:

- https://huggingface.co/datasets/compguesswhat

  <img width="1029" alt="Capture d’écran 2022-09-08 à 15 51 37" src="https://user-images.githubusercontent.com/1676121/189139946-6deffb91-f21b-4281-8825-a98026c69740.png">

- https://huggingface.co/datasets/nli_tr

  <img width="1032" alt="Capture d’écran 2022-09-08 à 15 51 44" src="https://user-images.githubusercontent.com/1676121/189139963-d26490ed-ad23-48ea-9cfc-1ab9c4d08d0c.png">


cc @albertvillanova
CI test of TensorFlow is failing
State: closed
Labels: name: bug

## Describe the bug
The following CI test fails: https://github.com/huggingface/datasets/runs/8246722693?check_suite_focus=true
```
FAILED tests/test_py_utils.py::TempSeedTest::test_tensorflow - AssertionError:
```

Details:
```
_________________________ TempSeedTest.test_tensorflow _________________________
[gw0] linux -- Python 3.7.13 /opt/hostedtoolcache/Python/3.7.13/x64/bin/python

self = <tests.test_py_utils.TempSeedTest testMethod=test_tensorflow>

    @require_tf
    def test_tensorflow(self):
        import tensorflow as tf
        from tensorflow.keras import layers
    
        def gen_random_output():
            model = layers.Dense(2)
            x = tf.random.uniform((1, 3))
            return model(x).numpy()
    
        with temp_seed(42, set_tensorflow=True):
            out1 = gen_random_output()
        with temp_seed(42, set_tensorflow=True):
            out2 = gen_random_output()
        out3 = gen_random_output()
    
>       np.testing.assert_equal(out1, out2)
E       AssertionError: 
E       Arrays are not equal
E       
E       Mismatched elements: 2 / 2 (100%)
E       Max absolute difference: 0.84619296
E       Max relative difference: 16.083529
E        x: array([[-0.793581,  0.333286]], dtype=float32)
E        y: array([[0.052612, 0.539708]], dtype=float32)

tests/test_py_utils.py:149: AssertionError
```
larger dataset, larger GPU memory in the training phase? Is that correct?
State: closed
Labels: name: bug

    from datasets import set_caching_enabled
    set_caching_enabled(False)
    for ds_name in ["squad","newsqa","nqopen","narrativeqa"]:
            train_ds = load_from_disk("../../../dall/downstream/processedproqa/{}-train.hf".format(ds_name))

        break
    train_ds = concatenate_datasets([train_ds,train_ds,train_ds,train_ds]) #operation 1


   trainer = QuestionAnsweringTrainer( #huggingface trainer
        model=model,
        args=training_args,
        train_dataset=train_ds,
        eval_dataset= None,
        eval_examples=None,
        answer_column_name=answer_column,
        dataset_name="squad",
        tokenizer=tokenizer,
        data_collator=data_collator,
        compute_metrics=compute_metrics if training_args.predict_with_generate else None,
    )

with operation 1, the GPU memory increases from 16G to 23G
Trec Dataset has incorrect labels
State: closed
Labels: name: bug

## Describe the bug
Both coarse and fine labels seem to be out of line.


## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = "trec"
raw_datasets = load_dataset(dataset)
df = pd.DataFrame(raw_datasets["test"])
df.head()
```

## Expected results
text (string) | coarse_label (class label) | fine_label (class label)
-- | -- | --
How far is it from Denver to Aspen ? | 5  	(NUM) | 40  	(NUM:dist)
What county is Modesto , California in ? | 4  	(LOC) | 32  	(LOC:city)
Who was Galileo ? | 3  	(HUM) | 31  	(HUM:desc)
What is an atom ? | 2  	(DESC) | 24  	(DESC:def)
When did Hawaii become a state ? | 5  	(NUM) | 39  	(NUM:date)

## Actual results
  index | label-coarse  |label-fine                  |                    text
-- |-- | -- | --
0    |         4     |     40     | How far is it from Denver to Aspen ?
1      |       5        |  21  | What county is Modesto , California in ?
2        |     3        |  12        |                 Who was Galileo ?
3         |    0       |    7         |                What is an atom ?
4         |    4       |    8       |   When did Hawaii become a state ?

## Environment info

- `datasets` version: 2.4.0
- Platform: Linux-5.4.0-1086-azure-x86_64-with-glibc2.27
- Python version: 3.9.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.3
Remove deprecated identical_ok
State: closed

`huggingface-hub` says that the `identical_ok` argument of `HfApi.upload_file` is now deprecated, and will be removed soon. It even has no effect at the moment when it's passed:

```python
Args:
...
    identical_ok (`bool`, *optional*, defaults to `True`):
        Deprecated: will be removed in 0.11.0.
        Changing this value has no effect.
...
```

There was only one occurence of `identical_ok=False` but it's maybe not worth adding a check ti verify if the files were the same.

cc @mariosasko
vivos (Vietnamese speech corpus) dataset not accessible
State: closed
Labels: name: dataset bug

## Describe the bug
VIVOS data is not accessible anymore, neither of these links work (at least from France):
* https://ailab.hcmus.edu.vn/assets/vivos.tar.gz (data)
* https://ailab.hcmus.edu.vn/vivos (dataset page) 

Therefore `load_dataset` doesn't work.

## Steps to reproduce the bug
```python
ds = load_dataset("vivos")
```

## Expected results
dataset loaded

## Actual results
```
ConnectionError: Couldn't reach https://ailab.hcmus.edu.vn/assets/vivos.tar.gz (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='ailab.hcmus.edu.vn', port=443): Max retries exceeded with url: /assets/vivos.tar.gz (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9d8a27d190>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))")))
```

Will try to contact the authors, as we wanted to use Vivos as an example in documentation on how to create scripts for audio datasets (https://github.com/huggingface/datasets/pull/4872), because it's small and straightforward and uses tar archives.
Dataset Viewer issue for indonesian-nlp/librivox-indonesia
State: closed

### Link

https://huggingface.co/datasets/indonesian-nlp/librivox-indonesia

### Description

I created a new speech dataset https://huggingface.co/datasets/indonesian-nlp/librivox-indonesia, but the dataset preview doesn't work with following error message:
```
Server error
Status code:   400
Exception:     TypeError
Message:       unsupported operand type(s) for +: 'NoneType' and 'str'
```
Please help, I am not sure what the problem here is. Thanks a lot.

### Owner

Yes
Dataset Viewer issue for bigscience-biomedical/biosses
State: closed

### Link

https://huggingface.co/datasets/bigscience-biomedical/biosses

### Description

I've just been working on adding the dataset loader script to this dataset and working with the relative imports. I'm not sure how to interpret the error below (show where the dataset preview used to be) . 
```
Status code:   400
Exception:     ModuleNotFoundError
Message:       No module named 'datasets_modules.datasets.bigscience-biomedical--biosses.ddbd5893bf6c2f4db06f407665eaeac619520ba41f69d94ead28f7cc5b674056.bigbiohub'
```

### Owner

Yes
Fix missing tags in dataset cards
State: closed

Fix missing tags in dataset cards:
- coqa
- hyperpartisan_news_detection
- opinosis
- scientific_papers
- scifact
- search_qa
- wiki_qa
- wiki_split
- wikisql

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921
Add ability to read-write to SQL databases.
State: closed

Fixes #3094 

Add ability to read/write to SQLite files and also read from any SQL database supported by SQLAlchemy.

I didn't add SQLAlchemy as a dependence as it is fairly big and it remains optional. 

I also recorded a Loom to showcase the feature.

https://www.loom.com/share/f0e602c2de8a46f58bca4b43333d541f
Dataset infos in yaml
State: closed
Labels: name: dataset contribution

To simplify the addition of new datasets, we'd like to have the dataset infos in the YAML and deprecate the dataset_infos.json file. YAML is readable and easy to edit, and the YAML metadata of the readme already contain dataset metadata so we would have everything in one place.

To be more specific, I moved these fields from DatasetInfo to the YAML:
- config_name (if there are several configs)
- download_size
- dataset_size
- features
- splits

Here is what I ended up with for `squad`:
```yaml
dataset_info:
  features:
  - name: id
    dtype: string
  - name: title
    dtype: string
  - name: context
    dtype: string
  - name: question
    dtype: string
  - name: answers
    sequence:
    - name: text
      dtype: string
    - name: answer_start
      dtype: int32
  splits:
  - name: train
    num_bytes: 79346360
    num_examples: 87599
  - name: validation
    num_bytes: 10473040
    num_examples: 10570
  config_name: plain_text
  download_size: 35142551
  dataset_size: 89819400
```

and it can be a list if there are several configs

I already did the change for `conll2000` and `crime_and_punish` as an example.

## Implementation details

### Load/Read

This is done via `DatasetInfosDict.write_to_directory/from_directory`

I had to implement a custom the YAML export logic for `SplitDict`, `Version` and `Features`.
The first two are trivial, but the logic for `Features` is more complicated, because I added a simplification step (or the YAML would be too long and less readable): it's just a formatting step to remove unnecessary nesting of YAML data.

### Other changes

I had to update the DatasetModule factories to also download the README.md alongside the dataset scripts/data files, and not just the dataset_infos.json

## YAML validation

I removed the old validation code that was in metadata.py, now we can just use the Hub YAML validation

## Datasets-cli

The `datasets-cli test --save_infos` command now creates a README.md file with the dataset_infos in it, instead of a datasets_infos.json file

## Backward compatibility

`dataset_infos.json` files are still supported and loaded if they exist to have full backward compatibility.
Though I removed the unnecessary keys when the value is the default (like all the `id: null` from the Value feature types) to make them easier to read.

## TODO

- [x] add comments
- [x] tests
- [x] document the new YAML fields 
- [x] try to reload the new dataset_infos.json file content with an old version of `datasets`

## EDITS

- removed "config_name" when there's only one config
- removed "version" for now (?), because it's not useful in general
- renamed the yaml field "dataset_info" instead of "dataset_infos", since it only has one by default (and because "infos" is not english)

Fix https://github.com/huggingface/datasets/issues/4876 and fix #2773
Concatenate_datasets loads everything into RAM
State: closed
Labels: name: bug

## Describe the bug
When loading the datasets seperately and saving them on disk, I want to concatenate them. But `concatenate_datasets` is filling up my RAM and the process gets killed. Is there a way to prevent this from happening or is this intended behaviour? Thanks in advance

## Steps to reproduce the bug
```python
gcs = gcsfs.GCSFileSystem(project='project')
datasets = [load_from_disk(f'path/to/slice/of/data/{i}', fs=gcs, keep_in_memory=False) for i in range(10)]

dataset = concatenate_datasets(datasets)
```

## Expected results
A concatenated dataset which is stored on my disk.

## Actual results
Concatenated dataset gets loaded into RAM and overflows it which gets the process killed.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-glibc2.10
- Python version: 3.8.13
- PyArrow version: 8.0.1
- Pandas version: 1.4.3
decode mp3 with librosa if torchaudio is > 0.12 as a temporary workaround 
State: closed

`torchaudio>0.12` fails with decoding mp3 files if `ffmpeg<4`. currently we ask users to downgrade torchaudio, but sometimes it's not possible as torchaudio version is binded to torch version. as a temporary workaround we can decode mp3 with librosa (though it 60 times slower, at least it works)

another option would be to ask users to install the required version of `ffmpeg`, but is non-trivial on colab: it's not in apt packages in ubuntu 18 and `conda` is not preinstalled (with `conda` it would be easily installable)

- [x] decode with torchaudio anyway if the version of ffmpeg is correct? it's 60 times faster
- [x] tests 
- [x] DO NOT FORGET to get back all the tests

see https://github.com/huggingface/datasets/issues/4776 and https://github.com/huggingface/datasets/issues/3663#issuecomment-1225797165 (there is a Colab notebook to reproduce the error)