jamesLeeeeeee/datasets-github-issues huggingface dataset:
Missing MBPP splits
Author: stadlerb | Pull request: no | State: closed

(@albertvillanova)
The [MBPP dataset on the Hub](https://huggingface.co/datasets/mbpp) has only a test split for both its "full" and its "sanitized" subset, while the [paper](https://arxiv.org/abs/2108.07732) states in subsection 2.1 regarding the full split:
> In the experiments described later in the paper, we hold out 10 problems for **few-shot prompting**, another 500 as our **test** dataset (which is used to evaluate both few-shot inference and fine-tuned models), 374 problems for **fine-tuning**, and the rest for **validation**.

If the dataset on the Hub should reproduce most closely what the original authors use, I guess this four-way split should be reflected. 

The paper doesn't explicitly state the task_id ranges of the splits, but the [GitHub readme](https://github.com/google-research/google-research/tree/master/mbpp) referenced in the paper specifies exact task_id ranges, although it misstates the total number of samples:
> We specify a train and test split to use for evaluation. Specifically:
> 
> * Task IDs 11-510 are used for evaluation.
> * Task IDs 1-10 and 511-1000 are used for training and/or prompting. We typically used 1-10 for few-shot prompting, although you can feel free to use any of the training examples.

I.e. the few-shot, train and validation splits are combined into one split, with a soft suggestion of using the first ten for few-shot prompting. It is not explicitly stated whether the 374 fine-tuning samples mentioned in the paper have task_id 511 to 784 or 601 to 974 or are randomly sampled from task_id 511 to 974.

Regarding the "sanitized" split the paper states the following:
> For evaluations involving the edited dataset, we perform comparisons with 100 problems that appear in both the original and edited dataset, using the same held out 10 problems for few-shot prompting and 374 problems for fine-tuning. 

The statement doesn't appear to be very precise, as among the 10 few-shot problems, those with task_id 1, 5 and 10 are not even part of the sanitized variant, and many from the task_id range from 511 to 974 are missing (e.g. task_id 511 to 553). I suppose the idea the task_id ranges for each split remain the same, even if some of the task_ids are not present. That would result in 7 few-shot, 257 test, 141 train and 22 validation examples in the sanitized split.
Add DocVQA
Author: NielsRogge | Pull request: no | State: open

## Adding a Dataset
- **Name:** DocVQA
- **Description:** Document Visual Question Answering (DocVQA) seeks to inspire a “purpose-driven” point of view in Document Analysis and Recognition research, where the document content is extracted and used to respond to high-level tasks defined by the human consumers of this information. 
- **Paper:** https://arxiv.org/abs/2007.00398
- **Data:** https://www.docvqa.org/datasets/docvqa
- **Motivation:** Models like LayoutLM and Donut in the Transformers library are fine-tuned on DocVQA. Would be very handy to directly load this dataset from the hub.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).
Dataset Viewer issue for Team-PIXEL/rendered-wikipedia-english
Author: xplip | Pull request: no | State: closed

### Link

https://huggingface.co/datasets/Team-PIXEL/rendered-wikipedia-english/viewer/rendered-wikipedia-en/train

### Description

The dataset can be loaded fine but the viewer shows this error:

```
Server Error
Status code:   400
Exception:     Status400Error
Message:       The dataset does not exist.
```

I'm guessing this is because I recently renamed the dataset. Based on related issues (e.g. https://github.com/huggingface/datasets/issues/4759) , is there something server-side that needs to be refreshed?

### Owner

Yes
Issue with fine classes in trec dataset
Author: albertvillanova | Pull request: no | State: closed

## Describe the bug
According to their paper, the TREC dataset contains 2 kinds of classes:
- 6 coarse classes: TREC-6
- 50 fine classes: TREC-50

However, our implementation only has 47 (instead of 50) fine classes. The reason for this is that we only considered the last segment of the label, which is repeated for several coarse classes:
- We have one `desc` fine label instead of 2:
  - `DESC:desc`
  - `HUM:desc`
- We have one `other` fine label instead of 3:
  - `ENTY:other`
  - `LOC:other`
  - `NUM:other`

From their paper:
> We define a two-layered taxonomy, which represents a natural semantic classification for typical answers in the TREC task. The hierarchy contains 6 coarse classes and 50 fine classes,

> Each coarse class contains a non-overlapping set of fine classes.
NonMatchingChecksumError in mbpp dataset
Author: albertvillanova | Pull request: no | State: closed

## Describe the bug
As reported on the Hub [Fix Checksum Mismatch](https://huggingface.co/datasets/mbpp/discussions/1), there is a `NonMatchingChecksumError` when loading mbpp dataset

## Steps to reproduce the bug
```python
ds = load_dataset("mbpp", "full")
```

## Expected results
Loading of the dataset without any exception raised.

## Actual results
```
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-1-a3fbdd3ed82e> in <module>
----> 1 ds = load_dataset("mbpp", "full")

.../huggingface/datasets/src/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1791 
   1792     # Download and prepare data
-> 1793     builder_instance.download_and_prepare(
   1794         download_config=download_config,
   1795         download_mode=download_mode,
.../huggingface/datasets/src/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    702                             logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    703                     if not downloaded_from_gcs:
--> 704                         self._download_and_prepare(
    705                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    706                         )

.../huggingface/datasets/src/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos)
   1225 
   1226     def _download_and_prepare(self, dl_manager, verify_infos):
-> 1227         super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
   1228 
   1229     def _get_examples_iterable_for_split(self, split_generator: SplitGenerator) -> ExamplesIterable:

.../huggingface/datasets/src/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    773         # Checksums verification
    774         if verify_infos and dl_manager.record_checksums:
--> 775             verify_checksums(
    776                 self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
    777             )

.../huggingface/datasets/src/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42 

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://raw.githubusercontent.com/google-research/google-research/master/mbpp/mbpp.jsonl']
```
.save_to_disk('path', fs=s3) TypeError 
Author: h-k-dev | Pull request: no | State: closed

The following code:
```python
import datasets

train_dataset, test_dataset = load_dataset("imdb", split=["train", "test"])
s3 = datasets.filesystems.S3FileSystem(key=aws_access_key_id, secret=aws_secret_access_key)
train_dataset.save_to_disk("s3://datasets/", fs=s3)

```
produces following traceback:

```shell
File "C:\Users\Hong Knop\AppData\Local\Programs\Python\Python310\lib\site-packages\botocore\auth.py", line 374, in scope
    return '/'.join(scope)

```
I invoke print(scope) in <auth.py> (line 373) and find this:

```python
[('4VA08VLL3VTKQJKCAI8M',), '20220803', 'us-east-1', 's3', 'aws4_request']

```
Add Multiface dataset
Author: osanseviero | Pull request: no | State: open

## Adding a Dataset
- **Name:** Multiface dataset
- **Description:** f high quality recordings of the faces of 13 identities, each captured in a multi-view capture stage performing various facial expressions. An average of 12,200 (v1 scripts) to 23,000 (v2 scripts) frames per subject with capture rate at 30 fps
- **Data:** https://github.com/facebookresearch/multiface

The whole dataset is 65TB though, so I'm not sure

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648
Author: conceptofmind | Pull request: no | State: closed

## Describe the bug
Following the example in CodeParrot, I receive an array size limitation error when deduplicating larger datasets.

## Steps to reproduce the bug
```python
dataset_name = "the_pile"
ds = load_dataset(dataset_name, split="train")
ds = ds.map(preprocess, num_proc=num_workers)
uniques = set(ds.unique("hash"))
```
Gists for minimum reproducible example:
https://gist.github.com/conceptofmind/c5804428ea1bd89767815f9cd5f02d9a
https://gist.github.com/conceptofmind/feafb07e236f28d79c2d4b28ffbdb6e2

## Expected results
Chunking and writing out a deduplicated dataset. 

## Actual results
```
return dataset._data.column(column).unique().to_pylist()
File "pyarrow/table.pxi", line 394, in pyarrow.lib.ChunkedArray.unique
File "pyarrow/_compute.pyx", line 531, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 330, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 124, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648
```
Fix label renaming and add a battery of tests
Author: Rocketknight1 | Pull request: yes | State: closed

This PR makes some changes to label renaming in `to_tf_dataset()`, both to fix some issues when users input something we weren't expecting, and also to make it easier to deprecate label renaming in future, if/when we want to move this special-casing logic to a function in `transformers`.

The main changes are:
- Label renaming now only happens when the `auto_rename_labels` argument is set. For backward compatibility, this defaults to `True` for now.
- If the user requests "label" but the data collator renames that column to "labels", the label renaming logic will now handle that case correctly.
- Added a battery of tests to make this more reliable in future.
- Adds an optimization to loading in `to_tf_dataset()` for unshuffled datasets (uses slicing instead of a list of indices)

Fixes #4772
RuntimeError when using torchaudio 0.12.0 to load MP3 audio file
Author: albertvillanova | Pull request: no | State: closed

Current version of `torchaudio` (0.12.0) raises a RuntimeError when trying to use `sox_io` backend but non-Python dependency `sox` is not installed:
https://github.com/pytorch/audio/blob/2e1388401c434011e9f044b40bc8374f2ddfc414/torchaudio/backend/sox_io_backend.py#L21-L29
```python
def _fail_load(
    filepath: str,
    frame_offset: int = 0,
    num_frames: int = -1,
    normalize: bool = True,
    channels_first: bool = True,
    format: Optional[str] = None,
) -> Tuple[torch.Tensor, int]:
    raise RuntimeError("Failed to load audio from {}".format(filepath))
```

Maybe we should raise a more actionable error message so that the user knows how to fix it.

UPDATE:
- this is an incompatibility of latest torchaudio (0.12.0) and the sox backend

TODO:
- [x] as a temporary solution, we should recommend installing torchaudio<0.12.0
  - #4777 
  - #4785
- [ ] however, a stable solution must be found for torchaudio>=0.12.0

Related to: 
- https://github.com/huggingface/transformers/issues/18379
Training hangs at the end of epoch, with set_transform/with_transform+multiple workers
Author: memray | Pull request: no | State: open

## Describe the bug
I use load_dataset() (I tried with [wiki](https://huggingface.co/datasets/wikipedia) and my own json data) and use set_transform/with_transform for preprocessing. But it hangs at the end of the 1st epoch if dataloader_num_workers>=1. No problem with single worker. 


## Steps to reproduce the bug
```python
train_dataset = datasets.load_dataset("wikipedia", "20220301.en",
                                      split='train', 
                                      cache_dir=model_args.cache_dir,
                                      streaming=False)
train_dataset.set_transform(psg_parse_fn)
train_dataloader = DataLoader(
            train_dataset,
            batch_size=args.train_batch_size,
            sampler=DistributedSampler(train_dataset),
            collate_fn=data_collator,
            drop_last=args.dataloader_drop_last,
            num_workers=args.dataloader_num_workers,
        )
```

## Expected results


## Actual results
It simply hangs. The ending step is num_example/batch_size (one epoch).

## Environment info
- `datasets` version: 2.4.1.dev0
- Platform: Linux-5.4.170+-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 8.0.0
- Pandas version: 1.4.1
AssertionError when using label_cols in to_tf_dataset 
Author: lehrig | Pull request: no | State: closed

## Describe the bug
An incorrect `AssertionError` is raised when using `label_cols` in `to_tf_dataset` and the label's key name is `label`.

The assertion is in this line:
https://github.com/huggingface/datasets/blob/2.4.0/src/datasets/arrow_dataset.py#L475

## Steps to reproduce the bug
```python
from datasets import load_dataset
from transformers import DefaultDataCollator

dataset = load_dataset('glue', 'mrpc', split='train')

tf_dataset = dataset.to_tf_dataset(
    columns=["sentence1", "sentence2", "idx"],
    label_cols=["label"],
    batch_size=16,
    collate_fn=DefaultDataCollator(return_tensors="tf"),
)
```

## Expected results
No assertion error.

## Actual results
```
AssertionError: in user code:

    File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 475, in split_features_and_labels  *
        assert set(features.keys()).union(labels.keys()) == set(input_batch.keys())
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-4.18.0-305.45.1.el8_4.ppc64le-ppc64le-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 7.0.0
- Pandas version: 1.4.3
More rigorous shape inference in to_tf_dataset
Author: Rocketknight1 | Pull request: yes | State: closed

`tf.data` needs to know the shape of tensors emitted from a `tf.data.Dataset`. Although `None` dimensions are possible, overusing them can cause problems - Keras uses the dataset tensor spec at compile-time, and so saying that a dimension is `None` when it's actually constant can hurt performance, or even cause training to fail for dimensions that are needed to determine the shape of weight tensors!

The compromise I used here was to sample several batches from the underlying dataset and apply the `collate_fn` to them, and then to see which dimensions were "empirically variable". There's an obvious problem here, though - if you sample 10 batches and they all have the same shape on a certain dimension, there's still a small chance that the 11th batch will be different, and Keras will throw an error  if a dataset tries to emit a tensor whose shape doesn't match the spec.

I encountered this bug in practice once or twice for datasets that were mostly-but-not-totally constant on a given dimension, and I still don't have a perfect solution, but this PR should greatly reduce the risk. It samples many more batches, and also samples very small batches (size 2) - this increases the variability, making it more likely that a few outlier samples will be detected.

Ideally, of course, we'd determine the full output shape analytically, but that's surprisingly tricky when the `collate_fn` can be any arbitrary Python code!
Improve features resolution in streaming
Author: lhoestq | Pull request: yes | State: closed

`IterableDataset._resolve_features` was returning the features sorted alphabetically by column name, which is not consistent with non-streaming. I changed this and used the order of columns from the data themselves. It was causing some inconsistencies in the dataset viewer as well.

I also fixed `interleave_datasets` that was not filling missing columns with None, because it was not using the columns from `IterableDataset._resolve_features`

cc @severo
parallel searching in multi-gpu setting using faiss
Author: Jiaxin-Wen | Pull request: no | State: open

While I notice that  `add_faiss_index` has supported assigning multiple GPUs, I am still confused about how it works. 

Does the `search-batch` function automatically parallelizes the input queries to different gpus?https://github.com/huggingface/datasets/blob/d76599bdd4d186b2e7c4f468b05766016055a0a5/src/datasets/search.py#L360
Issue with offline mode
Author: SaulLu | Pull request: no | State: closed

## Describe the bug
I can't retrieve a cached dataset with offline mode enabled

## Steps to reproduce the bug

To reproduce my issue, first, you'll need to run a script that will cache the dataset
```python
import os
os.environ["HF_DATASETS_OFFLINE"] = "0"

import datasets

datasets.logging.set_verbosity_info()
ds_name = "SaulLu/toy_struc_dataset"
ds = datasets.load_dataset(ds_name)
print(ds)
```
then, you can try to reload it in offline mode:
```python
import os
os.environ["HF_DATASETS_OFFLINE"] = "1"

import datasets

datasets.logging.set_verbosity_info()
ds_name = "SaulLu/toy_struc_dataset"
ds = datasets.load_dataset(ds_name)
print(ds)
```

## Expected results
I would have expected the 2nd snippet not to return any errors

## Actual results
The 2nd snippet returns:
```
Traceback (most recent call last):
  File "/home/lucile_huggingface_co/sandbox/evaluate/test_cache_datasets.py", line 8, in <module>
    ds = datasets.load_dataset(ds_name)
  File "/home/lucile_huggingface_co/anaconda3/envs/evaluate-dev/lib/python3.8/site-packages/datasets/load.py", line 1723, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/lucile_huggingface_co/anaconda3/envs/evaluate-dev/lib/python3.8/site-packages/datasets/load.py", line 1500, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/lucile_huggingface_co/anaconda3/envs/evaluate-dev/lib/python3.8/site-packages/datasets/load.py", line 1241, in dataset_module_factory
    raise ConnectionError(f"Couln't reach the Hugging Face Hub for dataset '{path}': {e1}") from None
ConnectionError: Couln't reach the Hugging Face Hub for dataset 'SaulLu/toy_struc_dataset': Offline mode is enabled.
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.3

Maybe I'm misunderstanding something in the use of the offline mode (see [doc](https://huggingface.co/docs/datasets/v2.4.0/en/loading#offline)), is that the case?
Dataset Viewer issue for Toygar/turkish-offensive-language-detection
Author: tanyelai | Pull request: no | State: closed

### Link

https://huggingface.co/datasets/Toygar/turkish-offensive-language-detection

### Description

Status code:   400
Exception:     Status400Error
Message:       The dataset does not exist.

Hi, I provided train.csv, test.csv and valid.csv files. However, viewer says dataset does not exist. 
Should I need to do anything else?

### Owner

Yes
Document better when relative paths are transformed to URLs
Author: albertvillanova | Pull request: no | State: closed

As discussed with @ydshieh, when passing a relative path as `data_dir` to `load_dataset` of a dataset hosted on the Hub, the relative path is transformed to the corresponding URL of the Hub dataset.

Currently, we mention this in our docs here: [Create a dataset loading script > Download data files and organize splits](https://huggingface.co/docs/datasets/v2.4.0/en/dataset_script#download-data-files-and-organize-splits)
> If the data files live in the same folder or repository of the dataset script, you can just pass the relative paths to the files instead of URLs.

Maybe we should document better how relative paths are handled, not only when creating a dataset loading script, but also when passing to `load_dataset`:
- `data_dir`
- `data_files`

CC: @stevhliu
Datasets.map causes incorrect overflow_to_sample_mapping when used with tokenizers and small batch size
Author: srobertjames | Pull request: no | State: open

## Describe the bug
When using `tokenizer`, we can retrieve the field `overflow_to_sample_mapping`, since long samples will be overflown into multiple token sequences.

However, when tokenizing is done via `Dataset.map`, with `n_proc > 1`, the `overflow_to_sample_mapping` field is wrong.  This seems to be because each tokenizer only looks at its share of the samples, and maps to the index _within its share_, but then `Dataset.map` collates them together.

## Steps to reproduce the bug

1. Make a dataset of 3 strings.
2. Tokenize via Dataset.map with n_proc = 8
3. Inspect the `overflow_to_sample_mapping` field


## Expected results
`[0, 1, 2]`

## Actual results
`[0, 0, 0]`

Notes:

1. I have not yet extracted a minimal example, but the above works reliably
2. If the dataset is large, I've yet to determine if this bug still happens a. not at all b. always c. on the small, leftover batch at the end.
Add `language_bcp47` tag
Author: lhoestq | Pull request: yes | State: closed

Following (internal) https://github.com/huggingface/moon-landing/pull/3509, we need to move the bcp47 tags to `language_bcp47` and keep the `language` tag for iso 639 1-2-3 codes. In particular I made sure that all the tags in `languages` are not longer than 3 characters. I moved the rest to `language_bcp47` and fixed some of them.

After this PR is merged I think we can simplify the language validation from the DatasetMetadata class (and keep it bare-bone just for the tagging app)

PS: the CI is failing because of missing content in dataset cards that are unrelated to this PR
Easily create loading script for benchmark comprising multiple huggingface datasets
Author: JoelNiklaus | Pull request: no | State: closed

Hi,

I would like to create a loading script for a benchmark comprising multiple huggingface datasets.
The function _split_generators needs to return the files for the respective dataset. However, the files are not always in the same location for each dataset. I want to just make a wrapper dataset that provides a single interface to all the underlying datasets. 
I thought about downloading the files with the load_dataset function and then providing the link to the cached file. But this seems a bit inelegant to me. What approach would you propose to do this?

Please let me know if you have any questions.

Cheers,
Joel