dalssy/data_del huggingface dataset:
GitHub issue: Fix dl_manager.extract returning FileNotFoundError
Author: lhoestq (MEMBER)
State: closed | Created: 2023-12-30 15:24:50

The dl_manager base path is remote (e.g. a hf:// path), so local cached paths should be passed as absolute paths.
This could happen if users provide a relative path as `cache_dir`

fix https://github.com/huggingface/datasets/issues/6536
GitHub issue: Datasets : wikipedia 20220301.en error 
Author: ppx666 (NONE)
State: closed | Created: 2023-12-29 08:34:51

### Describe the bug

When I used load_dataset to download this data set, the following error occurred. The main problem was that the target data did not exist.

### Steps to reproduce the bug

1.I tried downloading directly.
```python
wiki_dataset = load_dataset("wikipedia", "20220301.en")
```
An exception occurred
```
MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). 
Example of usage: 
	`load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')`
```
2.I modified the code as prompted.
```python
wiki_dataset = load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')
```
An exception occurred:
```
FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/enwiki/20220301/dumpstatus.json
```


### Expected behavior

I searched in the parent directory of the corresponding URL, but there was no corresponding "20220301" directory.
I really need this data set and hope to provide a download method.

### Environment info

python 3.8
datasets 2.16.0
apache-beam 2.52.0
dill 0.3.7
GitHub issue: Extreme inefficiency for `save_to_disk` when merging datasets
Author: KatarinaYuan (NONE)
State: open | Created: 2023-12-29 00:44:35

### Describe the bug

Hi, I tried to merge in total 22M sequences of data, where each sequence is of maximum length 2000. I found that merging these datasets and then `save_to_disk` is extremely slow because of flattening the indices. Wondering if you have any suggestions or guidance on this. Thank you very much! 

### Steps to reproduce the bug

The source data is too big to demonstrate

### Expected behavior

The source data is too big to demonstrate

### Environment info

python                     3.9.0
datasets                  2.7.0
pytorch                   2.0.0 
tokenizers                0.13.1
transformers              4.31.0
GitHub issue: 'Repo card metadata block was not found' when loading a pragmeval dataset
Author: lambdaofgod (NONE)
State: open | Created: 2023-12-28 14:18:25

### Describe the bug

I can't load dataset subsets of 'pragmeval'.

The funny thing is I ran the dataset author's [colab notebook](https://colab.research.google.com/drive/1sg--LF4z7XR1wxAOfp0-3d4J6kQ9nj_A?usp=sharing) and it works just fine. I tried to install exactly the same packages that are installed on colab using poetry, so my environment info only differs from the one from colab in linux version - I still get the same bug outside colab.

### Steps to reproduce the bug

Install dependencies with poetry

pyproject.toml
```
[tool.poetry]
name = "project"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "^3.10"
datasets = "2.16.0"
pandas = "1.5.3"
pyarrow = "10.0.1"
huggingface-hub = "0.19.4"
fsspec = "2023.6.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
```

`poetry run python -c "import datasets; print(datasets.get_dataset_config_names('pragmeval'))`

prints ['default']

### Expected behavior

The command should print

```
['emergent',
 'emobank-arousal',
 'emobank-dominance',
 'emobank-valence',
 'gum',
 'mrda',
 'pdtb',
 'persuasiveness-claimtype',
 'persuasiveness-eloquence',
 'persuasiveness-premisetype',
 'persuasiveness-relevance',
 'persuasiveness-specificity',
 'persuasiveness-strength',
 'sarcasm',
 'squinky-formality',
 'squinky-implicature',
 'squinky-informativeness',
 'stac',
 'switchboard',
 'verifiability']
```

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0
GitHub issue: Adding support for netCDF (*.nc) files
Author: shermansiu (NONE)
State: open | Created: 2023-12-27 09:27:29
Labels: name: enhancement

### Feature request

netCDF (*.nc) is a file format for storing multidimensional scientific data, which is used by packages like `xarray` (labelled multi-dimensional arrays in Python). It would be nice to have native support for netCDF in `datasets`.

### Motivation

When uploading *.nc files onto Huggingface Hub through the `datasets` API, I would like to be able to preview the dataset without converting it to another format.

### Your contribution

I can submit a PR, provided I have the time.
GitHub issue: ted_talks_iwslt | Error: Config name is missing
Author: rayliuca (NONE)
State: closed | Created: 2023-12-26 00:38:18

### Describe the bug

Running load_dataset using the newest `datasets` library like below on the ted_talks_iwslt using year pair data will throw an error "Config name is missing"

see also:
https://huggingface.co/datasets/ted_talks_iwslt/discussions/3

likely caused by #6493, where the `and not config_kwargs` part in the if logic was removed 
 https://github.com/huggingface/datasets/blob/ef3b5dd3633995c95d77f35fb17f89ff44990bc4/src/datasets/builder.py#L512

### Steps to reproduce the bug

run: 
```python
load_dataset("ted_talks_iwslt", language_pair=("ja", "en"), year="2015")
```

### Expected behavior

Load the data without error

### Environment info

datasets 2.16.0
GitHub issue: [Feature request] Indexing datasets by a customly-defined id field to enable random access dataset items via the id
Author: Yu-Shi (NONE)
State: open | Created: 2023-12-25 11:37:10
Labels: name: enhancement

### Feature request

Some datasets may contain an id-like field, for example the `id` field in [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) and the `_id` field in [BeIR/dbpedia-entity](https://huggingface.co/datasets/BeIR/dbpedia-entity). HF datasets support efficient random access via row, but not via this kinds of id fields. I wonder if it is possible to add support for indexing by a custom "id-like" field to enable random access via such ids. The ids may be numbers or strings.

### Motivation

In some cases, especially during inference/evaluation, I may want to find out the item that has a specified id, defined by the dataset itself.

For example, in a typical re-ranking setting in information retrieval, the user may want to re-rank the set of candidate documents of each query. The input is usually presented in a TREC-style run file, with the following format:

```
<qid> Q0 <docno> <rank> <score> <tag>
```

The re-ranking program should be able to fetch the queries and documents according to the `<qid>` and `<docno>`, which are the original id defined in the query/document datasets. To accomplish this, I have to iterate over the whole HF dataset to get the mapping from real ids to row ids every time I start the program, which is time-consuming. Thus I want HF dataset to provide options for users to index by a custom id column, not by row.

### Your contribution

I'm not an expert in this project and I'm afraid that I'm not able to make contributions on the code.
GitHub issue: Add polars compatibility
Author: psmyth94 (CONTRIBUTOR)
State: closed | Created: 2023-12-24 20:03:23

Hey there,

I've just finished adding support to convert and format to `polars.DataFrame`. This was in response to the open issue about integrating Polars [#3334](https://github.com/huggingface/datasets/issues/3334). Datasets can be switched to Polars format via `Dataset.set_format("polars")`. I've also included `to_polars` and `from_polars`. All polars functions are checked via config.POLARS_AVAILABLE. 

A few notes:
This only supports `DataFrames` and not `LazyFrames`. This probably could be integrated fairly easily via `is_lazy` args in `set_format`, and `to_polars`.

Let me know your feedbacks.
GitHub issue: Impossible to save a mapped dataset to disk
Author: kopyl (NONE)
State: open | Created: 2023-12-23 15:18:27

### Describe the bug

I want to play around with different hyperparameters when training but don't want to re-map my dataset with 3 million samples each time for tens of hours when I [fully fine-tune SDXL](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py).

After I do the mapping like this:
```
train_dataset = train_dataset.map(compute_embeddings_fn, batched=True)
train_dataset = train_dataset.map(
    compute_vae_encodings_fn,
    batched=True,
    batch_size=16,
)
```
and try to save it like this:
`train_dataset.save_to_disk("test")`
i get this error ([full traceback](https://pastebin.com/kq3vt739)):
```
TypeError: Object of type function is not JSON serializable
The format kwargs must be JSON serializable, but key 'transform' isn't.
```

But what is interesting is that pushing to hub works like that:
`train_dataset.push_to_hub("kopyl/mapped-833-icons-sdxl-1024-dataset", token=True)`
Here is the link of the pushed dataset: https://huggingface.co/datasets/kopyl/mapped-833-icons-sdxl-1024-dataset

### Steps to reproduce the bug

Here is the self-contained notebook:

https://colab.research.google.com/drive/1RtCsEMVcwWcMwlWURk_cj_9xUBHz065M?usp=sharing

### Expected behavior

It should be easily saved to disk

### Environment info

NVIDIA A100, Linux (NC24ads A100 v4 from Azure), CUDA 12.2.

[pip freeze](https://pastebin.com/QTNb6iru)
GitHub issue: Impossible to only download a test split
Author: ysig (NONE)
State: open | Created: 2023-12-22 16:56:32

I've spent a significant amount of time trying to locate the split object inside my _split_generators() custom function.
Then after diving [in the code](https://github.com/huggingface/datasets/blob/5ff3670c18ed34fa8ddfa70a9aa403ae6cc9ad54/src/datasets/load.py#L2558) I realized that `download_and_prepare` is executed before! split is passed to the dataset builder in `as_dataset`.

If I'm not missing something, this seems like bad design, for the following use case:

> Imagine there is a huge dataset that has an evaluation test set and you want to just download and run just to compare your method.

Is there a current workaround that can help me achieve the same result?

Thank you,
GitHub issue: BBox type
Author: lhoestq (MEMBER)
State: closed | Created: 2023-12-21 22:13:27

see [internal discussion](https://huggingface.slack.com/archives/C02EK7C3SHW/p1703097195609209)

Draft to get some feedback on a possible `BBox` feature type that can be used to get object detection bounding boxes data in one format or another.

```python
>>> from datasets import load_dataset, BBox
>>> ds = load_dataset("svhn", "full_numbers", split="train")
>>> ds[0]
{
  'image': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=107x46 at 0x126409BE0>,
  'digits': {'bbox': [[38, 1, 21, 40], [57, 3, 16, 40]], 'label': [4, 6]}
}
>>> ds = ds.rename_column("digits", "annotations").cast_column("annotations", BBox(format="coco"))
>>> ds[0]
{
  'image': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=107x46 at 0x147730070>,
  'annotations': [{'bbox': [38, 1, 21, 40], 'category_id': 4}, {'bbox': [57, 3, 16, 40], 'category_id': 6}]
}
```

note that it's a type for a list of bounding boxes, not just one - which would be needed to switch from a format to another using type casting.
GitHub issue: Streaming the Pile: Missing Files
Author: FelixLabelle (NONE)
State: closed | Created: 2023-12-21 21:25:09

### Describe the bug

The pile does not stream, a "File not Found error" is returned. It looks like the Pile's files have been moved.

### Steps to reproduce the bug

To reproduce run the following code:

```
from datasets import load_dataset

dataset = load_dataset('EleutherAI/pile', 'en', split='train', streaming=True)
next(iter(dataset))
```

I get the following error:
`FileNotFoundError: https://the-eye.eu/public/AI/pile/train/00.jsonl.zst`

### Expected behavior

Return the data in a stream.

### Environment info

- `datasets` version: 2.12.0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.11.5
- Huggingface_hub version: 0.15.1
- PyArrow version: 11.0.0
- Pandas version: 2.0.3
GitHub issue: The order of the splits is not preserved
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-12-21 11:17:27
Labels: name: bug

We had a regression and the order of the splits is not preserved. They are alphabetically sorted, instead of preserving original "train", "validation", "test" order.

Check: In branch "main"
```python
In [9]: dataset = load_dataset("adversarial_qa", '"adversarialQA")

In [10]: dataset
Out[10]: 
DatasetDict({
    test: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
    train: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 30000
    })
    validation: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
})
```

Before (2.15.0) it was:
```python
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 30000
    })
    validation: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
    test: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
})
```

See issues: 
- https://huggingface.co/datasets/adversarial_qa/discussions/3
- https://huggingface.co/datasets/beans/discussions/4

This is a regression because it was previously fixed. See:
- #6196
- #5728
GitHub issue: Support push_to_hub canonical datasets
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-12-20 15:16:45

Support `push_to_hub` canonical datasets.

This is necessary in the Space to convert script-datasets to Parquet: https://huggingface.co/spaces/albertvillanova/convert-dataset-to-parquet

Note that before this PR, the `repo_id` "dataset_name" was transformed to "user/dataset_name". This behavior was introduced by:
- #6269
GitHub issue: Bug get_metadata_patterns arg error
Author: d710055071 (CONTRIBUTOR)
State: closed | Created: 2023-12-20 08:56:44

https://github.com/huggingface/datasets/blob/3f149204a2a5948287adcade5e90707aa5207a92/src/datasets/load.py#L1240C1-L1240C69
metadata_patterns = get_metadata_patterns(base_path, download_config=self.download_config)
GitHub issue: Support huggingface-hub pre-releases
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-12-20 07:52:29

Support `huggingface-hub` pre-releases.

This way we will have our CI green when testing `huggingface-hub` release candidates. See: https://github.com/huggingface/datasets/tree/ci-test-huggingface-hub-v0.20.0.rc1

Close #6513.
GitHub issue: Why call http_head() when fsspec_head()  succeeds
Author: d710055071 (CONTRIBUTOR)
State: closed | Created: 2023-12-20 02:25:51

https://github.com/huggingface/datasets/blob/a91582de288d98e94bcb5ab634ca1cfeeff544c5/src/datasets/utils/file_utils.py#L510C1-L523C14
GitHub issue: Cache backward compatibility with 2.15.0
Author: lhoestq (MEMBER)
State: closed | Created: 2023-12-19 23:52:25

...for datasets without scripts

It takes into account the changes in cache from
- https://github.com/huggingface/datasets/pull/6493: switch to `config/version/commit_sha` schema
- https://github.com/huggingface/datasets/pull/6454: fix `DataFilesDict` keys ordering when hashing

requires https://github.com/huggingface/datasets/pull/6493 to be merged
GitHub issue: Support huggingface-hub 0.20.0
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-12-19 15:15:46

CI to test the support of `huggingface-hub` 0.20.0: https://github.com/huggingface/datasets/compare/main...ci-test-huggingface-hub-v0.20.0.rc1

We need to merge:
- #6510 
- #6512
- #6516
GitHub issue: Implement get dataset default config name
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-12-19 11:26:19

Implement `get_dataset_default_config_name`.

Now that we support setting a configuration as default in `push_to_hub` (see #6500), we need a programmatically way to know in advance which is the default configuration. This will be used in the Space to convert script-datasets to Parquet: https://huggingface.co/spaces/albertvillanova/convert-dataset-to-parquet

Follow-up of:
- #6500

CC: @severo
GitHub issue: Replace `list_files_info` with `list_repo_tree` in `push_to_hub`
Author: mariosasko (COLLABORATOR)
State: closed | Created: 2023-12-18 15:34:19

Starting from `huggingface_hub` 0.20.0, `list_files_info` will be deprecated in favor of `list_repo_tree` (see https://github.com/huggingface/huggingface_hub/pull/1910)
GitHub issue: Read GeoParquet files using parquet reader
Author: weiji14 (CONTRIBUTOR)
State: closed | Created: 2023-12-18 04:50:37

Let GeoParquet files with the file extension `*.geoparquet` or `*.gpq` be readable by the default parquet reader.

Those two file extensions are the ones most commonly used for GeoParquet files, and is included in the `gpq` validator tool at https://github.com/planetlabs/gpq/blob/e5576b4ee7306b4d2259d56c879465a9364dab90/cmd/gpq/command/convert.go#L73-L75

Addresses https://github.com/huggingface/datasets/issues/6438
GitHub issue: where is glue_metric.py> @Frankie123421 what was the resolution to this?
Author: Mcccccc1024 (NONE)
State: closed | Created: 2023-12-17 09:58:25

              > @Frankie123421 what was the resolution to this?

use glue_metric.py instead of glue.py in load_metric

_Originally posted by @Frankie123421 in https://github.com/huggingface/datasets/issues/2117#issuecomment-905093763_
GitHub issue: Incorrect test set labels for RTE and CoLA datasets via load_dataset
Author: emreonal11 (NONE)
State: closed | Created: 2023-12-16 22:06:08

### Describe the bug

The test set labels for the RTE and CoLA datasets when loading via datasets load_dataset are all -1.
Edit: It appears this is also the case for every other dataset except for MRPC (stsb, sst2, qqp, mnli (both matched and mismatched), qnli, wnli, ax). Is this intended behavior to safeguard the test set for evaluation purposes?

### Steps to reproduce the bug

!pip install datasets
from datasets import load_dataset

rte_data = load_dataset('glue', 'rte')
cola_data = load_dataset('glue', 'cola')
print(rte_data['test'][0:30]['label'])
print(cola_data['test'][0:30]['label'])

Output:
[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]
[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]

The non-label test data seems to be fine:
e.g. rte_data['test'][1] is:
{'sentence1': "Authorities in Brazil say that more than 200 people are being held hostage in a prison in the country's remote, Amazonian-jungle state of Rondonia.",
 'sentence2': 'Authorities in Brazil hold 200 people as hostage.',
 'label': -1,
 'idx': 1}
Training and validation data are also fine:
e.g. rte_data['train][0] is:
{'sentence1': 'No Weapons of Mass Destruction Found in Iraq Yet.',
 'sentence2': 'Weapons of Mass Destruction Found in Iraq.',
 'label': 1,
 'idx': 0}


### Expected behavior

Expected the labels to be binary 0/1 values; Got all -1s instead

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0
GitHub issue: Error Pushing to Hub
Author: Jiayi-Pan (NONE)
State: closed | Created: 2023-12-16 01:05:22

### Describe the bug

Error when trying to push a dataset in a special format to hub

### Steps to reproduce the bug

```
import datasets
from datasets import Dataset
dataset_dict = {
    "filename": ["apple", "banana"],
    "token": [[[1,2],[3,4]],[[1,2],[3,4]]],
    "label": [0, 1],
}
dataset = Dataset.from_dict(dataset_dict)
dataset = dataset.cast_column("token", datasets.features.features.Array2D(shape=(2, 2),dtype="int16"))

dataset.push_to_hub("SequenceModel/imagenet_val_256")
```
Error:
```
...

ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/tuple'
  in "<unicode string>", line 8, column 16:
            shape: !!python/tuple
                   ^
```

### Expected behavior

Dataset being pushed to hub

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.19.0-1022-gcp-x86_64-with-glibc2.35
- Python version: 3.11.5
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0
GitHub issue: Fix streaming xnli
Author: lhoestq (MEMBER)
State: closed | Created: 2023-12-15 14:40:57

This code was failing

```python

In [1]: from datasets import load_dataset

In [2]: 
   ...:         ds = load_dataset("xnli", "all_languages", split="test", streaming=True)
   ...: 
   ...:         sample_data = next(iter(ds))["premise"]  # pick up one data
   ...:         input_text = list(sample_data.values())
```

```
File ~/hf/datasets/src/datasets/features/translation.py:104, in TranslationVariableLanguages.encode_example(self, translation_dict)
    102     return translation_dict
    103 elif self.languages and set(translation_dict) - lang_set:
--> 104     raise ValueError(
    105         f'Some languages in example ({", ".join(sorted(set(translation_dict) - lang_set))}) are not in valid set ({", ".join(lang_set)}).'
    106     )
    108 # Convert dictionary into tuples, splitting out cases where there are
    109 # multiple translations for a single language.
    110 translation_tuples = []

ValueError: Some languages in example (language, translation) are not in valid set (ur, fr, hi, sw, vi, el, de, th, en, tr, zh, ar, bg, ru, es).
```

because in streaming mode we expect features encode methods to be no-ops if the example is already encoded.

I fixed `TranslationVariableLanguages` to account for that
GitHub issue:  OverflowError: value too large to convert to int32_t 
Author: zhangfan-algo (NONE)
State: open | Created: 2023-12-15 10:10:21

### Describe the bug

![image](https://github.com/huggingface/datasets/assets/47747764/f58044fb-ddda-48b6-ba68-7bbfef781630)


### Steps to reproduce the bug

just loading datasets 

### Expected behavior

how can I fix it

### Environment info

pip install /mnt/cluster/zhangfan/study_info/LLaMA-Factory/peft-0.6.0-py3-none-any.whl
pip install huggingface_hub-0.19.4-py3-none-any.whl tokenizers-0.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl transformers-4.36.1-py3-none-any.whl pyarrow_hotfix-0.6-py3-none-any.whl datasets-2.15.0-py3-none-any.whl tyro-0.5.18-py3-none-any.whl trl-0.7.4-py3-none-any.whl

done
GitHub issue: Fallback on dataset script if user wants to load default config
Author: lhoestq (MEMBER)
State: closed | Created: 2023-12-14 16:46:01

Right now this code is failing on `main`:

```python
load_dataset("openbookqa")
```

This is because it tries to load the dataset from the Parquet export but the dataset has multiple configurations and the Parquet export doesn't know which one is the default one.

I fixed this by simply falling back on using the dataset script (which tells the user to pass `trust_remote_code=True`):

```python
load_dataset("openbookqa", trust_remote_code=True)
```

Note that if the user happened to specify a config name I don't fall back on the script since we can use the Parquet export in this case (no need to know which config is the default)


```python
load_dataset("openbookqa", "main")
```
GitHub issue: Error when writing a dataset to HF Hub: A commit has happened since. Please refresh and try again.
Author: GeorgesLorre (NONE)
State: open | Created: 2023-12-14 11:24:54

**Describe the bug**

Getting a `412 Client Error: Precondition Failed` when trying to write a dataset to the HF hub.

```
huggingface_hub.utils._errors.HfHubHTTPError: 412 Client Error: Precondition Failed for url: https://huggingface.co/api/datasets/GLorr/test-dask/commit/main (Request ID: Root=1-657ae26f-3bd92bf861bb254b2cc0826c;50a09ab7-9347-406a-ba49-69f98abee9cc)

A commit has happened since. Please refresh and try again.
```

**Steps to reproduce the bug**
This is a minimal reproducer:
```
import dask.dataframe as dd
import pandas as pd
import random
import os

import huggingface_hub

import datasets

huggingface_hub.login(token=os.getenv("HF_TOKEN"))

data = {"number": [random.randint(0,10) for _ in range(1000)]}
df = pd.DataFrame.from_dict(data)
dataframe = dd.from_pandas(df, npartitions=1)
dataframe = dataframe.repartition(npartitions=3)

schema = datasets.Features({"number": datasets.Value("int64")}).arrow_schema

repo_id = "GLorr/test-dask"
repo_path = f"hf://datasets/{repo_id}"
huggingface_hub.create_repo(repo_id=repo_id, repo_type="dataset", exist_ok=True)
dd.to_parquet(dataframe, path=f"{repo_path}/data", schema=schema)
```

**Expected behavior**
Would expect to write to the hub without any problem.

**Environment info**
```
datasets==2.15.0
huggingface-hub==0.19.4
```
GitHub issue: Image Data loaded Twice
Author: ArcaneLex (NONE)
State: open | Created: 2023-12-13 13:11:42

### Describe the bug

![1702472610561](https://github.com/huggingface/datasets/assets/28867010/4b7ef5e7-32c3-4b73-84cb-5de059caa0b6)
When I learn from https://huggingface.co/docs/datasets/image_load and try to load image data from a folder. I noticed that the image was read twice in the returned data. As you can see in the attached image, there are only four images in the train folder, but reading brings up eight images

### Steps to reproduce the bug

from datasets import Dataset, load_dataset

dataset = load_dataset("imagefolder", data_dir="data/", drop_labels=False)
# print(dataset["train"][0]["image"] == dataset["train"][1]["image"])

print(dataset)
print(dataset["train"]["image"])
print(len(dataset["train"]["image"]))

### Expected behavior

DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 8
    })
})
[<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D1CA8B0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D2452E0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D245310>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D2453A0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D245460>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D245430>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D2454F0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D245550>]
8

### Environment info

- `datasets` version: 2.14.5
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.9.17
- Huggingface_hub version: 0.19.4
- PyArrow version: 13.0.0
- Pandas version: 2.0.3
GitHub issue: Lazy data files resolution and offline cache reload
Author: lhoestq (MEMBER)
State: closed | Created: 2023-12-12 17:15:17

Includes both https://github.com/huggingface/datasets/pull/6458 and https://github.com/huggingface/datasets/pull/6459

This PR should be merged instead of the two individually, since they are conflicting

## Offline cache reload

 it can reload datasets that were pushed to hub if they exist in the cache.

example:

```python
>>> Dataset.from_dict({"a": [1, 2]}).push_to_hub("lhoestq/tmp")
>>> load_dataset("lhoestq/tmp")
DatasetDict({
    train: Dataset({
        features: ['a'],
        num_rows: 2
    })
})
```

and later, without connection:

```python
>>> load_dataset("lhoestq/tmp")
Using the latest cached version of the dataset since lhoestq/tmp couldn't be found on the Hugging Face Hub
Found the latest cached dataset configuration 'default' at /Users/quentinlhoest/.cache/huggingface/datasets/lhoestq___tmp/default/0.0.0/da0e902a945afeb9 (last modified on Wed Dec 13 14:55:52 2023).
DatasetDict({
    train: Dataset({
        features: ['a'],
        num_rows: 2
    })
})
```

- Updated `CachedDatasetModuleFactory` to look for datasets in the cache at `<namespace>___<dataset_name>/<config_id>`
- Since the metadata configs parameters are not available in offline mode, we don't know which folder to load (config_id and hash change), so I simply load the latest one
  - I instantiate a BuilderConfig even if there is no metadata config with the right config_name
  - Its config_id is equal to the config_name to be able to retrieve it in the cache (no more suffix for configs from metadata configs)
  - We can reload this config if offline mode by specifying the right config_name (same as online !)
- Consequences of this change:
  - Only when there are user's parameters it creates a custom builder config with config_id = config_name + user parameters hash
  - the hash used to name the cache folder takes into account the metadata config and the dataset info, so that the right cache can be reloaded when there is internet connection without redownloading the data or resolving the data files. For local directories I hash the builder configs and dataset info, and for datasets on the hub I use the commit sha as hash.
  - cache directories now look like `config/version/commit_sha` for hub datasets which is clean :)

Fix https://github.com/huggingface/datasets/issues/3547

## Lazy data files resolution

this makes this code run in 2sec instead of >10sec

```python
from datasets import load_dataset

ds = load_dataset("glue", "sst2", streaming=True, trust_remote_code=False)
```

For some datasets with many configs and files it can be up to 100x faster.
This is particularly important now that some datasets will be loaded from the Parquet export instead of the scripts.

The data files are only resolved in the builder `__init__`. To do so I added DataFilesPatternsList and DataFilesPatternsDict that have `.resolve()` to return resolved DataFilesList and DataFilesDict
GitHub issue: Make push_to_hub return CommitInfo
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-12-12 15:18:16

Make `push_to_hub` return `CommitInfo`.

This is useful, for example, if we pass `create_pr=True` and we want to know the created PR ID.

CC: @severo for the use case in https://huggingface.co/datasets/jmhessel/newyorker_caption_contest/discussions/4
GitHub issue: `load_dataset(...,save_infos=True)` not working without loading script
Author: morganveyret (NONE)
State: open | Created: 2023-12-12 08:09:18

### Describe the bug

It seems that saving a dataset infos back into the card file is not working for datasets without a loading script.

After tracking the problem a bit it looks like saving the infos uses `Builder.get_imported_module_dir()` as its destination directory.
Internally this is a call to `inspect.getfile()` but since the actual builder class used is dynamically created (cf. `datasets.load.configure_builder_class`) this method actually return te path to the parent builder class (e.g. `datasets.packaged_modules.json.JSON`). 



### Steps to reproduce the bug

1. Have a local dataset without any loading script
2. Make sure there are no dataset infos in the README.md
3. Load with `save_infos=True`
4. No change in the dataset README.md
5. A new README.md file is created in the directory of the parent builder class (e.g. for json in `.../site-packages/datasets/packaged_modules/json/README.md`)

### Expected behavior

The dataset README.md should be updated and no file should be created in the python environment.

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.3
- `fsspec` version: 2023.6.0
GitHub issue: load_dataset imageflder for aws s3 path 
Author: segalinc (NONE)
State: open | Created: 2023-12-12 00:08:43
Labels: name: enhancement

### Feature request

I would like to load a dataset from S3 using the imagefolder option 
something like 
`dataset = datasets.load_dataset('imagefolder', data_dir='s3://.../lsun/train/bedroom', fs=S3FileSystem(), streaming=True)  `

### Motivation

no need of data_files

### Your contribution

no experience with this
GitHub issue: 429 Client Error
Author: sasaadi (NONE)
State: open | Created: 2023-12-11 15:06:01

Hello, I was downloading the following dataset and after 20% of data was downloaded, I started getting error 429. It is not resolved since a few days. How should I resolve it?
Thanks

Dataset:
https://huggingface.co/datasets/cerebras/SlimPajama-627B

Error:
`requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/datasets/cerebras/SlimPajama-627B/resolve/2d0accdd58c5d5511943ca1f5ff0e3eb5e293543/train/chunk1/example_train_3300.jsonl.zst`
GitHub issue: Update builder hash with info
Author: lhoestq (MEMBER)
State: closed | Created: 2023-12-11 11:09:16

Currently if you change the `dataset_info` of a dataset (e.g. in the YAML part of the README.md), the cache ignores this change.

This is problematic because you want to regenerate a dataset if you change the features or the split sizes for example (e.g. after push_to_hub)

Ideally we should take the resolved files into account as well but this will be for another PR
GitHub issue: FileNotFoundError: [Errno 2] No such file or directory: 'nul'
Author: amanyara (NONE)
State: closed | Created: 2023-12-11 08:52:13

### Describe the bug

it seems that sth wrong with my terrible "bug body" life, When i run this code, "import datasets"
i meet this error FileNotFoundError: [Errno 2] No such file or directory: 'nul'
![image](https://github.com/huggingface/datasets/assets/73683903/3973c120-ebb1-42b7-bede-b9de053e861d)
![image](https://github.com/huggingface/datasets/assets/73683903/0496adff-a7a7-4dcb-929e-ec11ede71f04)


### Steps to reproduce the bug

1.import datasets

### Expected behavior

i just run a single line code and stuct in this bug 

### Environment info

OS: Windows10
Datasets==2.15.0
python=3.10
GitHub issue: using torchrun, save_to_disk suddenly shows SIGTERM
Author: Ariya12138 (NONE)
State: open | Created: 2023-12-08 13:22:03

### Describe the bug

When I run my code using the "torchrun" command, when the code reaches the "save_to_disk" part, suddenly I get the following warning and error messages:
Because the dataset is too large, the "save_to_disk" function splits it into 70 parts for saving. However, an error occurs suddenly when it reaches the 14th shard.
WARNING: torch.distributed.elastic.multiprocessing.api: Sending process 2224968 closing signal SIGTERM
ERROR: torch.distributed.elastic.multiprocessing.api: failed (exitcode: -7). traceback: Signal 7 (SIGBUS) received by PID 2224967.

### Steps to reproduce the bug

ds_shard = ds_shard.map(map_fn, *args, **kwargs)
ds_shard.save_to_disk(ds_shard_filepaths[rank])

Saving the dataset (14/70 shards):  20%|██        | 875350/4376702 [00:19<01:53, 30863.15 examples/s]
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 2224968 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -7) local_rank: 0 (pid: 2224967) of binary: /home/bingxing2/home/scx6964/.conda/envs/ariya235/bin/python
Traceback (most recent call last):
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/bingxing2/home/scx6964/.conda/envs/ariya235/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
==========================================================
run.py FAILED
----------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
----------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-12-08_20:09:04
  rank      : 0 (local_rank: 0)
  exitcode  : -7 (pid: 2224967)
  error_file: <N/A>
  traceback : Signal 7 (SIGBUS) received by PID 2224967



### Expected behavior

I hope it can save successfully without any issues, but it seems there is a problem.

### Environment info

`datasets` version: 2.14.6
- Platform: Linux-4.19.90-24.4.v2101.ky10.aarch64-aarch64-with-glibc2.28
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- PyArrow version: 14.0.0
- Pandas version: 2.1.2
GitHub issue: Add IterableDataset `__repr__`
Author: lhoestq (MEMBER)
State: closed | Created: 2023-12-07 16:31:50

Example for glue sst2:

Dataset

```
DatasetDict({
    test: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1821
    })
    train: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 67349
    })
    validation: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 872
    })
})
```

IterableDataset (new)

```
IterableDatasetDict({
    test: IterableDataset({
        features: ['sentence', 'label', 'idx'],
        n_shards: 1
    })
    train: IterableDataset({
        features: ['sentence', 'label', 'idx'],
        n_shards: 1
    })
    validation: IterableDataset({
        features: ['sentence', 'label', 'idx'],
        n_shards: 1
    })
})
```

IterableDataset (before)

```
{'test': <datasets.iterable_dataset.IterableDataset object at 0x130d421f0>, 'train': <datasets.iterable_dataset.IterableDataset object at 0x136f3aaf0>, 'validation': <datasets.iterable_dataset.IterableDataset object at 0x136f4b100>}
{'sentence': 'hide new secretions from the parental units ', 'label': 0, 'idx': 0}
```
GitHub issue: CI on windows is broken: PermissionError
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-12-06 08:32:53
Labels: name: bug

See: https://github.com/huggingface/datasets/actions/runs/7104781624/job/19340572394
```
FAILED tests/test_load.py::test_loading_from_the_datasets_hub - NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpfcnps56i\\hf-internal-testing___dataset_with_script\\default\\0.0.0\\c240e2be3370bdbd\\dataset_with_script-train.arrow'
```
GitHub issue: Deprecate Beam API and download from HF GCS bucket
Author: mariosasko (COLLABORATOR)
State: closed | Created: 2023-12-05 19:51:33

Deprecate the Beam API and download from the HF GCS bucked. 

TODO:
- [x] Convert the Beam-based [`wikipedia`](https://huggingface.co/datasets/wikipedia) to an Arrow-based dataset ([Hub PR](https://huggingface.co/datasets/wikipedia/discussions/19))
- [x] Make [`natural_questions`](https://huggingface.co/datasets/natural_questions) a no-code dataset ([Hub PR](https://huggingface.co/datasets/natural_questions/discussions/7))
- [x] Make [`wiki40b`](https://huggingface.co/datasets/wiki40b) a no-code dataset ([Hub PR](https://huggingface.co/datasets/wiki40b/discussions/5))
- [x] Make [`wiki_dpr`](https://huggingface.co/datasets/wiki_dpr) an Arrow-based dataset ([Hub PR](https://huggingface.co/datasets/wiki_dpr/discussions/14))
GitHub issue: CI quality is broken
Author: albertvillanova (MEMBER)
State: closed | Created: 2023-12-05 15:35:34
Labels: name: bug, name: maintenance

See: https://github.com/huggingface/datasets/actions/runs/7100835633/job/19327734359
```
Would reformat: src/datasets/features/image.py
1 file would be reformatted, 253 files left unchanged
```
GitHub issue: If an image in a dataset is corrupted, we get unescapable error
Author: chigozienri (NONE)
State: open | Created: 2023-12-04 20:58:49

### Describe the bug

Example discussed in detail here: https://huggingface.co/datasets/sasha/birdsnap/discussions/1

### Steps to reproduce the bug

```
from datasets import load_dataset, VerificationMode
dataset = load_dataset(
    'sasha/birdsnap',
    split="train",
    verification_mode=VerificationMode.ALL_CHECKS,
    streaming=True # I recommend using streaming=True when reproducing, as this dataset is large
)
for idx, row in enumerate(dataset):
    # Iterating to 9287 took 7 minutes for me
    # If you already have the data locally cached and set streaming=False, you see the same error just by with dataset[9287]
    pass
    # error at 9287 OSError: image file is truncated (45 bytes not processed)
    # note that we can't avoid the error using a try/except + continue inside the loop
```

### Expected behavior

Able to escape errors in casting to Image() without killing the whole loop

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.15.0-84-generic-x86_64-with-glibc2.31
- Python version: 3.11.5
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.3
- `fsspec` version: 2023.10.0
GitHub issue: New version release request
Author: LZHgrla (CONTRIBUTOR)
State: closed | Created: 2023-12-04 07:08:26
Labels: name: enhancement

### Feature request

Hi!
I am using `datasets` in library `xtuner` and am highly interested in the features introduced since v2.15.0.

To avoid installation from source in our pypi wheels, we are eagerly waiting for the new release. So, Does your team have a new release plan for v2.15.1 and could you please share it with us?

Thanks very much!


### Motivation

.

### Your contribution

.
GitHub issue: Can't align optional features of struct
Author: Dref360 (CONTRIBUTOR)
State: closed | Created: 2023-12-03 15:57:07

### Describe the bug

Hello!

I'm currently experiencing an issue where I can't concatenate datasets if an inner field of a Feature is Optional.

I have a column named `speaker`, and this holds some information about a speaker.

```python
@dataclass
class Speaker:
   name: str
   email: Optional[str]
```

If I have two datasets, one happens to have `email` always None, then I get `The features can't be aligned because the key email of features`




### Steps to reproduce the bug

You can run the following script:

```python

ds = Dataset.from_dict({'speaker': [{'name': 'Ben', 'email': None}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
concatenate_datasets([ds, ds2])

>>>The features can't be aligned because the key speaker of features {'speaker': {'email': Value(dtype='string', id=None), 'name': Value(dtype='string', id=None)}} has unexpected type - {'email': Value(dtype='string', id=None), 'name': Value(dtype='string', id=None)} (expected either {'email': Value(dtype='null', id=None), 'name': Value(dtype='string', id=None)} or Value("null").

```

### Expected behavior

I think this should work; if two top-level columns were in the same situation it would properly cast to `string`.

```python
ds = Dataset.from_dict({'email': [None, None]})
ds2 = Dataset.from_dict({'email': ['abc@aol.com', 'one@yahoo.com']})
concatenate_datasets([ds, ds2])
>>> # Works!
```

### Environment info

- `datasets` version: 2.15.1.dev0
- Platform: Linux-5.15.0-89-generic-x86_64-with-glibc2.35
- Python version: 3.9.13
- `huggingface_hub` version: 0.19.4
- PyArrow version: 9.0.0
- Pandas version: 1.4.4
- `fsspec` version: 2023.6.0

I would be happy to fix this issue.
GitHub issue: Fix shard retry mechanism in `push_to_hub`
Author: mariosasko (COLLABORATOR)
State: closed | Created: 2023-11-30 14:57:14

When it fails, `preupload_lfs_files` throws a [`RuntimeError`](https://github.com/huggingface/huggingface_hub/blob/5eefebee2c150a2df950ab710db350e96c711433/src/huggingface_hub/_commit_api.py#L402) error and chains the original HTTP error. This PR modifies the retry mechanism's error handling to account for that.

Fix https://github.com/huggingface/datasets/issues/6392
GitHub issue: Retrieve cached datasets that were pushed to hub when offline
Author: lhoestq (MEMBER)
State: closed | Created: 2023-11-29 16:56:15

I drafted the logic to retrieve a no-script dataset in the cache.
For example it can reload datasets that were pushed to hub if they exist in the cache.

example:

```python
>>> Dataset.from_dict({"a": [1, 2]}).push_to_hub("lhoestq/tmp")
>>> load_dataset("lhoestq/tmp")
DatasetDict({
    train: Dataset({
        features: ['a'],
        num_rows: 2
    })
})
```

and later, without connection:

```python
>>> load_dataset("lhoestq/tmp")
Using the latest cached version of the dataset from /Users/quentinlhoest/.cache/huggingface/datasets/lhoestq___tmp/*/*/0b3caccda1725efb(last modified on Wed Nov 29 16:50:27 2023) since it couldn't be found locally at lhoestq/tmp.
DatasetDict({
    train: Dataset({
        features: ['a'],
        num_rows: 2
    })
})
```

fix https://github.com/huggingface/datasets/issues/3547

## Implementation details (EDITED)

I continued in https://github.com/huggingface/datasets/pull/6493, see the changes there

TODO:
- [x] tests
- [ ] compatible with https://github.com/huggingface/datasets/pull/6458
GitHub issue: Lazy data files resolution
Author: lhoestq (MEMBER)
State: closed | Created: 2023-11-29 13:18:44

Related to discussion at https://github.com/huggingface/datasets/pull/6255

this makes this code run in 2sec instead of >10sec

```python
from datasets import load_dataset

ds = load_dataset("glue", "sst2", streaming=True, trust_remote_code=False)
```

For some datasets with many configs and files it can be up to 100x faster.
This is particularly important now that some datasets will be loaded from the Parquet export instead of the scripts.

The data files are only resolved in the builder `__init__`. To do so I added DataFilesPatternsList and DataFilesPatternsDict that have `.resolve()` to return resolved DataFilesList and DataFilesDict
GitHub issue: `TypeError`: huggingface_hub.hf_file_system.HfFileSystem.find() got multiple values for keyword argument 'maxdepth'
Author: wasertech (NONE)
State: closed | Created: 2023-11-29 01:57:36

### Describe the bug

Please see https://github.com/huggingface/huggingface_hub/issues/1872

### Steps to reproduce the bug

Please see https://github.com/huggingface/huggingface_hub/issues/1872

### Expected behavior

Please see https://github.com/huggingface/huggingface_hub/issues/1872

### Environment info

Please see https://github.com/huggingface/huggingface_hub/issues/1872
GitHub issue: Don't require trust_remote_code in inspect_dataset
Author: lhoestq (MEMBER)
State: closed | Created: 2023-11-28 19:47:07

don't require `trust_remote_code` in (deprecated) `inspect_dataset` (it defeats its purpose)

(not super important but we might as well keep it until the next major release)

this is needed to fix the tests in https://github.com/huggingface/datasets/pull/6448