jamesLeeeeeee/datasets-github-issues huggingface dataset:
feat: improve error message on Keys mismatch. closes #4917
Author: PaulLerner | Pull request: yes | State: closed

Hi @lhoestq what do you think?

Let me give you a code sample:
```py
>>> import datasets
>>> foo = datasets.Dataset.from_dict({'foo':[0,1], 'bar':[2,3]})
>>> foo.save_to_disk('foo')
# edit foo/dataset_info.json e.g. rename the 'foo' feature to 'baz'
>>> datasets.load_from_disk('foo')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-4863e606b330> in <module>
----> 1 datasets.load_from_disk('foo')

~/code/datasets/src/datasets/load.py in load_from_disk(dataset_path, fs, keep_in_memory)
   1851         raise FileNotFoundError(f"Directory {dataset_path} not found")
   1852     if fs.isfile(Path(dest_dataset_path, config.DATASET_INFO_FILENAME).as_posix()):
-> 1853         return Dataset.load_from_disk(dataset_path, fs, keep_in_memory=keep_in_memory)
   1854     elif fs.isfile(Path(dest_dataset_path, config.DATASETDICT_JSON_FILENAME).as_posix()):
   1855         return DatasetDict.load_from_disk(dataset_path, fs, keep_in_memory=keep_in_memory)

~/code/datasets/src/datasets/arrow_dataset.py in load_from_disk(dataset_path, fs, keep_in_memory)
   1230             info=dataset_info,
   1231             split=split,
-> 1232             fingerprint=state["_fingerprint"],
   1233         )
   1234 

~/code/datasets/src/datasets/arrow_dataset.py in __init__(self, arrow_table, info, split, indices_table, fingerprint)
    687             self.info.features = inferred_features
    688         else:  # make sure the nested columns are in the right order
--> 689             self.info.features = self.info.features.reorder_fields_as(inferred_features)
    690 
    691         # Infer fingerprint if None

~/code/datasets/src/datasets/features/features.py in reorder_fields_as(self, other)
   1771                 return source
   1772 
-> 1773         return Features(recursive_reorder(self, other))
   1774 
   1775     def flatten(self, max_depth=16) -> "Features":

~/code/datasets/src/datasets/features/features.py in recursive_reorder(source, target, stack)
   1760                                f"{source.keys()-target.keys()} are missing from dataset.arrow "
   1761                                f"and {target.keys()-source.keys()} are missing from dataset_info.json"+stack_position)
-> 1762                     raise ValueError(message)
   1763                 return {key: recursive_reorder(source[key], target[key], stack + f".{key}") for key in target}
   1764             elif isinstance(source, list):

ValueError: Keys mismatch: between {'baz': Value(dtype='int64', id=None), 'bar': Value(dtype='int64', id=None)} (dataset_info.json) and {'foo': Value(dtype='int64', id=None), 'bar': Value(dtype='int64', id=None)} (inferred from dataset.arrow).
{'baz'} are missing from dataset.arrow and {'foo'} are missing from dataset_info.json
```
Dataset Viewer issue for pysentimiento/spanish-targeted-sentiment-headlines
Author: finiteautomata | Pull request: no | State: closed

### Link

https://huggingface.co/datasets/pysentimiento/spanish-targeted-sentiment-headlines

### Description

After moving the dataset from my user (`finiteautomata`) to the `pysentimiento` organization, the dataset viewer says that it doesn't exist.

### Owner

_No response_
Keys mismatch: make error message more informative
Author: PaulLerner | Pull request: no | State: closed

**Is your feature request related to a problem? Please describe.**
When loading a dataset from disk with a defect in its `dataset_info.json` describing its features (I don’t know when/why/how this happens but it deserves its own issue), you will get an error message like:
`ValueError: Keys mismatch: between {'bar': Value(dtype='int64', id=None)} and {'foo': Value(dtype='int64', id=None)}`

Which is fine when you have only a few features like in the example but it gets very hard to read when you have a lot of features in your dataset.

**Describe the solution you'd like**
The error message should give the difference between the features (what keys are in A but missing in B and vice-versa). It should also tell which keys are inferred from `dataset.arrow` and which come from `dataset_info.json`.

Willing to help :)
datasets map() handles all data at a stroke and takes long time
Author: BruceStayHungry | Pull request: no | State: closed

**1. Background**

Huggingface datasets package advises using `map()`  to process data in batches. In the example code on pretraining masked language model, they use `map()` to tokenize all data at a stroke before the train loop. 

The corresponding code:
```
with accelerator.main_process_first():
    tokenized_datasets = raw_datasets.map(
        tokenize_function,
        batched=True,
        num_proc=args.preprocessing_num_workers,
        remove_columns=column_names,
        load_from_cache_file=not args.overwrite_cache,
        desc="Running tokenizer on every text in dataset"
    )
```

**2. The problem**

Thus, when I try the same pertaining code with a much larger corpus, it takes quite a long time to tokenize.

Also, we can choose to tokenize data in `data-collator`. In this way, the program only tokenizes one batch in the next training step and avoids getting stuck in tokenization.

**3. My question**

As described above, my questions are:
* **Which is better? Process in `map()` or in `data-collator`**
* **Why huggingface advises `map()` function?** There should be some advantages to using `map()`


Thanks for your answers!
[Tests] Ensure `datasets` supports renamed repositories
Author: lhoestq | Pull request: no | State: open

On https://hf.co/datasets you can rename a dataset (or sometimes move it to another user/org). The website handles redirections correctly and AFAIK `datasets` does as well.

However it would be nice to have an integration test to make sure we don't break support for renamed datasets.

To implement this we can use the /api/repos/move endpoint on hub-ci to rename/move a repo (it is documented at https://huggingface.co/docs/hub/api)
Identical keywords in build_kwargs and config_kwargs lead to TypeError in load_dataset_builder()
Author: bablf | Pull request: no | State: open

## Describe the bug
In `load_dataset_builder()`, `build_kwargs` and `config_kwargs` can contain the same keywords leading to a TypeError("type object got multiple values for keyword argument "xyz"). 

I ran into this problem with the keyword: `base_path`. It might happen with other kwargs as well. I think a quickfix would be 
```python
builder_cls = import_main_class(dataset_module.module_path)
builder_kwargs = dataset_module.builder_kwargs
data_files = builder_kwargs.pop("data_files", data_files)
config_name = builder_kwargs.pop("config_name", name)
hash = builder_kwargs.pop("hash")
base_path = builder_kwargs.pop("base_path")
```
and then pass base_path into `builder_cls`.

## Steps to reproduce the bug
```python
from datasets import load_dataset
load_dataset("rotten_tomatoes", base_path="./sample_data")
```

## Expected results
The docs state: `**config_kwargs` — Keyword arguments to be passed to the [BuilderConfig](https://huggingface.co/docs/datasets/v2.4.0/en/package_reference/builder_classes#datasets.BuilderConfig) and used in the [DatasetBuilder](https://huggingface.co/docs/datasets/v2.4.0/en/package_reference/builder_classes#datasets.DatasetBuilder).

So I would expect to be able to pass the base_path into `load_dataset()`. 
## Actual results
TypeError("type object got multiple values for keyword argument "base_path"). 

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: macOS-12.5-arm64-arm-64bit
- Python version: 3.8.9
- PyArrow version: 9.0.0
Update GLUE evaluation metadata
Author: lewtun | Pull request: yes | State: closed

This PR updates the evaluation metadata for GLUE to:

* Include defaults for all configs except `ax` (which only has a `test` split with no known labels)
* Fix the default split from `test` to `validation` since `test` splits in GLUE have no labels (they're private)
* Fix the `task_id` for some existing defaults

cc @sashavor @douwekiela
Fix missing tags in dataset cards
Author: albertvillanova | Pull request: yes | State: closed

Fix missing tags in dataset cards:
- asnq
- clue
- common_gen
- cosmos_qa
- guardian_authorship
- hindi_discourse
- py_ast
- x_stance

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
None Type error for swda datasets
Author: hannan72 | Pull request: no | State: closed

## Describe the bug
I got `'NoneType' object is not callable` error while calling the swda datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("swda")
```

## Expected results
Run without error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Python version: 3.8.10
Can't import datasets AttributeError: partially initialized module 'datasets' has no attribute 'utils' (most likely due to a circular import)
Author: OPterminator | Pull request: no | State: closed

## Describe the bug
A clear and concise description of what the bug is.
Not able to import datasets 
## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
import os
os.environ["WANDB_API_KEY"] = "0" ## to silence warning
import numpy as np
import random
import sklearn
import matplotlib.pyplot as plt
import pandas as pd
import sys
import tensorflow as tf
import plotly.express as px
import transformers
import tokenizers
import nlp as nlp
import utils
import datasets
```

## Expected results
A clear and concise description of the expected results.
import should work normal
## Actual results
Specify the actual results or traceback.
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-21-b3b5b0b62103> in <module>
     13 import nlp as nlp
     14 import utils
---> 15 import datasets

~\anaconda3\lib\site-packages\datasets\__init__.py in <module>
     44 from .fingerprint import disable_caching, enable_caching, is_caching_enabled, set_caching_enabled
     45 from .info import DatasetInfo, MetricInfo
---> 46 from .inspect import (
     47     get_dataset_config_info,
     48     get_dataset_config_names,

~\anaconda3\lib\site-packages\datasets\inspect.py in <module>
     28 from .download.streaming_download_manager import StreamingDownloadManager
     29 from .info import DatasetInfo
---> 30 from .load import dataset_module_factory, import_main_class, load_dataset_builder, metric_module_factory
     31 from .utils.file_utils import relative_to_absolute_path
     32 from .utils.logging import get_logger

~\anaconda3\lib\site-packages\datasets\load.py in <module>
     53 from .iterable_dataset import IterableDataset
     54 from .metric import Metric
---> 55 from .packaged_modules import (
     56     _EXTENSION_TO_MODULE,
     57     _MODULE_SUPPORTS_METADATA,

~\anaconda3\lib\site-packages\datasets\packaged_modules\__init__.py in <module>
      4 from typing import List
      5 
----> 6 from .csv import csv
      7 from .imagefolder import imagefolder
      8 from .json import json

~\anaconda3\lib\site-packages\datasets\packaged_modules\csv\csv.py in <module>
     13 
     14 
---> 15 logger = datasets.utils.logging.get_logger(__name__)
     16 
     17 _PANDAS_READ_CSV_NO_DEFAULT_PARAMETERS = ["names", "prefix"]

AttributeError: partially initialized module 'datasets' has no attribute 'utils' (most likely due to a circular import)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.4.0
- Platform: Windows-10-10.0.22000-SP0
- Python version: 3.8.8
- PyArrow version: 9.0.0
- Pandas version: 1.2.4
[LibriSpeech] Fix dev split local_extracted_archive for 'all' config
Author: sanchit-gandhi | Pull request: yes | State: closed

We define the keys for the `_DL_URLS` of the dev split as `dev.clean` and `dev.other`:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L60-L61

These keys get forwarded to the `dl_manager` and thus the `local_extracted_archive`.

However, when calling `SplitGenerator` for the dev sets, we query the `local_extracted_archive` keys `validation.clean` and `validation.other`:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L212
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L219

The consequence of this is that the `local_extracted_archive` arg passed to `_generate_examples` is always `None`, as the keys `validation.clean` and `validation.other` do not exists in the `local_extracted_archive`.

When defining the `audio_file` in `_generate_examples`, since `local_extracted_archive` is always `None`, we always omit the `local_extracted_archive` path from the `audio_file` path, **even** if in non-streaming mode:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L259-L263

Thus, `audio_file` will only ever be the streaming path (`audio_file`, not `os.path.join(local_extracted_archive, audio_file)`).

This PR fixes the `.get()` keys for the `local_extracted_archive` for the dev splits.
Name the default config `default`
Author: severo | Pull request: no | State: closed

Currently, if a dataset has no configuration, a default configuration is created from the dataset name.

For example, for a dataset loaded from the hub repository, such as https://huggingface.co/datasets/user/dataset (repo id is `user/dataset`), the default configuration will be `user--dataset`.

It might be easier to handle to set it to `default`, or another reserved word.
Re-add code and und language tags
Author: albertvillanova | Pull request: yes | State: closed

This PR fixes the removal of 2 language tags done by:
- #4882

The tags are:
- "code": this is not a IANA tag but needed
- "und": this is one of the special scoped tags removed by 0d53202b9abce6fd0358cb00d06fcfd904b875af
  - used in "mc4" and "udhr" datasets
datasets generate large arrow file
Author: jax11235 | Pull request: no | State: closed

Checking the large file in disk, and found the large cache file in the cifar10 data directory:

![image](https://user-images.githubusercontent.com/18533904/186830449-ba96cdeb-0fe8-4543-994d-2abe7145933f.png)

As we know, the size of cifar10 dataset is ~130MB, but the cache file has almost 30GB size, there may be some problems here.
Fix missing tags in dataset cards
Author: albertvillanova | Pull request: yes | State: closed

Fix missing tags in dataset cards:
- anli
- coarse_discourse
- commonsense_qa
- cos_e
- ilist
- lc_quad
- web_questions
- xsum

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
Oversampling strategy for iterable datasets in `interleave_datasets`
Author: lhoestq | Pull request: no | State: closed

In https://github.com/huggingface/datasets/pull/4831 @ylacombe added an oversampling strategy for `interleave_datasets`. However right now it doesn't work for datasets loaded using `load_dataset(..., streaming=True)`, which are `IterableDataset` objects.

It would be nice to expand `interleave_datasets` for iterable datasets as well to support this oversampling strategy

```python
>>> from datasets.iterable_dataset import IterableDataset, ExamplesIterable
>>> d1 = IterableDataset(ExamplesIterable(lambda: [(yield i, {"a": i}) for i in [0, 1, 2]], {}))
>>> d2 = IterableDataset(ExamplesIterable(lambda: [(yield i, {"a": i}) for i in [10, 11, 12, 13]], {}))
>>> d3 = IterableDataset(ExamplesIterable(lambda: [(yield i, {"a": i}) for i in [20, 21, 22, 23, 24]], {}))
>>> dataset = interleave_datasets([d1, d2, d3])  # is supported
>>> [x["a"] for x in dataset]
[0, 10, 20, 1, 11, 21, 2, 12, 22]
>>> dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted")  # is not supported yet
>>> [x["a"] for x in dataset]
[0, 10, 20, 1, 11, 21, 2, 12, 22, 0, 13, 23, 1, 0, 24]
```

This can be implemented by adding the strategy to both `CyclingMultiSourcesExamplesIterable` and `RandomlyCyclingMultiSourcesExamplesIterable` used in `_interleave_iterable_datasets` in `iterable_dataset.py`

I would be happy to share some guidance if anyone would like to give it a shot :)
Fix missing tags in dataset cards
Author: albertvillanova | Pull request: yes | State: closed

Fix missing tags in dataset cards:
- aslg_pc12
- librispeech_lm
- mwsc
- opus100
- qasc
- quail
- squadshifts
- winograd_wsc

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
add Dataset.from_list
Author: sanderland | Pull request: yes | State: closed

As discussed in #4885 

I initially added this bit at the end, thinking filling this field was necessary as it is done in from_dict. 
However, it seems the constructor takes care of filling info when it is empty.
```
if info.features is None:
  info.features = Features(
      {
          col: generate_from_arrow_type(coldata.type)
          for col, coldata in zip(pa_table.column_names, pa_table.columns)
      }
  )
```
torchaudio 11.0 yields different results than torchaudio 12.1 when loading MP3
Author: patrickvonplaten | Pull request: no | State: closed

## Describe the bug

When loading Common Voice with torchaudio 0.11.0 the results are different to 0.12.1 which leads to problems in transformers see: https://github.com/huggingface/transformers/pull/18749

## Steps to reproduce the bug

If you run the following code once with `torchaudio==0.11.0+cu102` and `torchaudio==0.12.1+cu102` you can see that the tensors differ. This is a pretty big breaking change and makes some integration tests fail in Transformers.

```python
#!/usr/bin/env python3
from datasets import load_dataset
import datasets
import numpy as np
import torch
import torchaudio
print("torch vesion", torch.__version__)
print("torchaudio vesion", torchaudio.__version__)

save_audio = True
load_audios = False

if save_audio:
    ds = load_dataset("common_voice", "en", split="train", streaming=True)
    ds = ds.cast_column("audio", datasets.Audio(sampling_rate=16_000))
    ds_iter = iter(ds)
    sample = next(ds_iter)

    np.save(f"audio_sample_{torch.__version__}", sample["audio"]["array"])
    print(sample["audio"]["array"])

if load_audios:
    array_torch_11 = np.load("/home/patrick/audio_sample_1.11.0+cu102.npy")
    print("Array 11 Shape", array_torch_11.shape)
    print("Array 11 abs sum", np.sum(np.abs(array_torch_11)))
    array_torch_12 = np.load("/home/patrick/audio_sample_1.12.1+cu102.npy")
    print("Array 12 Shape", array_torch_12.shape)
    print("Array 12 abs sum", np.sum(np.abs(array_torch_12)))
```

Having saved the tensors the print output yields:

```
torch vesion 1.12.1+cu102
torchaudio vesion 0.12.1+cu102
Array 11 Shape (122880,)
Array 11 abs sum 1396.4988
Array 12 Shape (123264,)
Array 12 abs sum 1396.5193
```

## Expected results
torchaudio 11.0 and 12.1 should yield same results.

## Actual results
See above.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.1.1.dev0
- Platform: Linux-5.18.10-76051810-generic-x86_64-with-glibc2.34
- Python version: 3.9.7
- PyArrow version: 6.0.1
- Pandas version: 1.4.2
Dataset Viewer issue for subjqa
Author: lewtun | Pull request: no | State: closed

### Link

https://huggingface.co/datasets/subjqa

### Description

Getting the following error for this dataset:

```
Status code:   500
Exception:     Status500Error
Message:       2 or more items returned, instead of 1
```

Not sure what's causing it though 🤔 

### Owner

Yes
Loading huggan/CelebA-HQ throws pyarrow.lib.ArrowInvalid
Author: JeanKaddour | Pull request: no | State: open

## Describe the bug
Loading huggan/CelebA-HQ throws pyarrow.lib.ArrowInvalid

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset('huggan/CelebA-HQ')
```

## Expected results
See https://colab.research.google.com/drive/141LJCcM2XyqprPY83nIQ-Zk3BbxWeahq?usp=sharing#scrollTo=N3ml_7f8kzDd

## Actual results
```
  File "/home/jean/projects/cold_diffusion/celebA.py", line 4, in <module>
    dataset = load_dataset('huggan/CelebA-HQ')
  File "/home/jean/miniconda3/envs/seq/lib/python3.10/site-packages/datasets/load.py", line 1793, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/jean/miniconda3/envs/seq/lib/python3.10/site-packages/datasets/builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "/home/jean/miniconda3/envs/seq/lib/python3.10/site-packages/datasets/builder.py", line 793, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/jean/miniconda3/envs/seq/lib/python3.10/site-packages/datasets/builder.py", line 1274, in _prepare_split
    for key, table in logging.tqdm(
  File "/home/jean/miniconda3/envs/seq/lib/python3.10/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/home/jean/miniconda3/envs/seq/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 67, in _generate_tables
    parquet_file = pq.ParquetFile(f)
  File "/home/jean/miniconda3/envs/seq/lib/python3.10/site-packages/pyarrow/parquet/__init__.py", line 286, in __init__
    self.reader.open(
  File "pyarrow/_parquet.pyx", line 1227, in pyarrow._parquet.ParquetReader.open
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: datasets-2.4.1.dev0
- Platform: Ubuntu 18.04
- Python version: 3.10
- PyArrow version: pyarrow 9.0.0
Create dataset from list of dicts
Author: sanderland | Pull request: no | State: closed

I often find myself with data from a variety of sources, and a list of dicts is very common among these.
However, converting this to a Dataset is a little awkward, requiring either

```Dataset.from_pandas(pd.DataFrame(formatted_training_data))```
Which can error out on some more exotic values as 2-d arrays for reasons that are not entirely clear
> ArrowInvalid: ('Can only convert 1-dimensional array values', 'Conversion failed for column labels with type object')

Alternatively:
```Dataset.from_dict({k: [s[k] for s in formatted_training_data] for k in  formatted_training_data[0].keys()})```
Which works, but is a little ugly.

**Describe the solution you'd like**
Either `.from_dict` accepting a list of dicts, or a `.from_records` function accepting such.

I am happy to PR this, just wanted to check you are happy to accept this I haven't missed something obvious, and which of the solutions would be prefered.