jamesLeeeeeee/datasets-github-issues huggingface dataset:
[Feature request] Indexing datasets by a customly-defined id field to enable random access dataset items via the id
Author: Yu-Shi | Pull request: no | State: open

### Feature request

Some datasets may contain an id-like field, for example the `id` field in [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) and the `_id` field in [BeIR/dbpedia-entity](https://huggingface.co/datasets/BeIR/dbpedia-entity). HF datasets support efficient random access via row, but not via this kinds of id fields. I wonder if it is possible to add support for indexing by a custom "id-like" field to enable random access via such ids. The ids may be numbers or strings.

### Motivation

In some cases, especially during inference/evaluation, I may want to find out the item that has a specified id, defined by the dataset itself.

For example, in a typical re-ranking setting in information retrieval, the user may want to re-rank the set of candidate documents of each query. The input is usually presented in a TREC-style run file, with the following format:

```
<qid> Q0 <docno> <rank> <score> <tag>
```

The re-ranking program should be able to fetch the queries and documents according to the `<qid>` and `<docno>`, which are the original id defined in the query/document datasets. To accomplish this, I have to iterate over the whole HF dataset to get the mapping from real ids to row ids every time I start the program, which is time-consuming. Thus I want HF dataset to provide options for users to index by a custom id column, not by row.

### Your contribution

I'm not an expert in this project and I'm afraid that I'm not able to make contributions on the code.
Add polars compatibility
Author: psmyth94 | Pull request: yes | State: open

Hey there,

I've just finished adding support to convert and format to `polars.DataFrame`. This was in response to the open issue about integrating Polars [#3334](https://github.com/huggingface/datasets/issues/3334). Datasets can be switched to Polars format via `Dataset.set_format("polars")`. I've also included `to_polars` and `from_polars`. All polars functions are checked via config.POLARS_AVAILABLE. 

A few notes:
This only supports `DataFrames` and not `LazyFrames`. This probably could be integrated fairly easily via `is_lazy` args in `set_format`, and `to_polars`.

Let me know your feedbacks.
Impossible to save a mapped dataset to disk
Author: kopyl | Pull request: no | State: open

### Describe the bug

I want to play around with different hyperparameters when training but don't want to re-map my dataset with 3 million samples each time for tens of hours when I [fully fine-tune SDXL](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py).

After I do the mapping like this:
```
train_dataset = train_dataset.map(compute_embeddings_fn, batched=True)
train_dataset = train_dataset.map(
    compute_vae_encodings_fn,
    batched=True,
    batch_size=16,
)
```
and try to save it like this:
`train_dataset.save_to_disk("test")`
i get this error ([full traceback](https://pastebin.com/kq3vt739)):
```
TypeError: Object of type function is not JSON serializable
The format kwargs must be JSON serializable, but key 'transform' isn't.
```

But what is interesting is that pushing to hub works like that:
`train_dataset.push_to_hub("kopyl/mapped-833-icons-sdxl-1024-dataset", token=True)`
Here is the link of the pushed dataset: https://huggingface.co/datasets/kopyl/mapped-833-icons-sdxl-1024-dataset

### Steps to reproduce the bug

Here is the self-contained notebook:

https://colab.research.google.com/drive/1RtCsEMVcwWcMwlWURk_cj_9xUBHz065M?usp=sharing

### Expected behavior

It should be easily saved to disk

### Environment info

NVIDIA A100, Linux (NC24ads A100 v4 from Azure), CUDA 12.2.

[pip freeze](https://pastebin.com/QTNb6iru)
Impossible to only download a test split
Author: ysig | Pull request: no | State: open

I've spent a significant amount of time trying to locate the split object inside my _split_generators() custom function.
Then after diving [in the code](https://github.com/huggingface/datasets/blob/5ff3670c18ed34fa8ddfa70a9aa403ae6cc9ad54/src/datasets/load.py#L2558) I realized that `download_and_prepare` is executed before! split is passed to the dataset builder in `as_dataset`.

If I'm not missing something, this seems like bad design, for the following use case:

> Imagine there is a huge dataset that has an evaluation test set and you want to just download and run just to compare your method.

Is there a current workaround that can help me achieve the same result?

Thank you,
BBox type
Author: lhoestq | Pull request: yes | State: closed

see [internal discussion](https://huggingface.slack.com/archives/C02EK7C3SHW/p1703097195609209)

Draft to get some feedback on a possible `BBox` feature type that can be used to get object detection bounding boxes data in one format or another.

```python
>>> from datasets import load_dataset, BBox
>>> ds = load_dataset("svhn", "full_numbers", split="train")
>>> ds[0]
{
  'image': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=107x46 at 0x126409BE0>,
  'digits': {'bbox': [[38, 1, 21, 40], [57, 3, 16, 40]], 'label': [4, 6]}
}
>>> ds = ds.rename_column("digits", "annotations").cast_column("annotations", BBox(format="coco"))
>>> ds[0]
{
  'image': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=107x46 at 0x147730070>,
  'annotations': [{'bbox': [38, 1, 21, 40], 'category_id': 4}, {'bbox': [57, 3, 16, 40], 'category_id': 6}]
}
```

note that it's a type for a list of bounding boxes, not just one - which would be needed to switch from a format to another using type casting.
Streaming the Pile: Missing Files
Author: FelixLabelle | Pull request: no | State: closed

### Describe the bug

The pile does not stream, a "File not Found error" is returned. It looks like the Pile's files have been moved.

### Steps to reproduce the bug

To reproduce run the following code:

```
from datasets import load_dataset

dataset = load_dataset('EleutherAI/pile', 'en', split='train', streaming=True)
next(iter(dataset))
```

I get the following error:
`FileNotFoundError: https://the-eye.eu/public/AI/pile/train/00.jsonl.zst`

### Expected behavior

Return the data in a stream.

### Environment info

- `datasets` version: 2.12.0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.11.5
- Huggingface_hub version: 0.15.1
- PyArrow version: 11.0.0
- Pandas version: 2.0.3
The order of the splits is not preserved
Author: albertvillanova | Pull request: no | State: closed

We had a regression and the order of the splits is not preserved. They are alphabetically sorted, instead of preserving original "train", "validation", "test" order.

Check: In branch "main"
```python
In [9]: dataset = load_dataset("adversarial_qa", '"adversarialQA")

In [10]: dataset
Out[10]: 
DatasetDict({
    test: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
    train: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 30000
    })
    validation: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
})
```

Before (2.15.0) it was:
```python
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 30000
    })
    validation: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
    test: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
})
```

See issues: 
- https://huggingface.co/datasets/adversarial_qa/discussions/3
- https://huggingface.co/datasets/beans/discussions/4

This is a regression because it was previously fixed. See:
- #6196
- #5728
Support push_to_hub canonical datasets
Author: albertvillanova | Pull request: yes | State: closed

Support `push_to_hub` canonical datasets.

This is necessary in the Space to convert script-datasets to Parquet: https://huggingface.co/spaces/albertvillanova/convert-dataset-to-parquet

Note that before this PR, the `repo_id` "dataset_name" was transformed to "user/dataset_name". This behavior was introduced by:
- #6269
Bug get_metadata_patterns arg error
Author: d710055071 | Pull request: no | State: closed

https://github.com/huggingface/datasets/blob/3f149204a2a5948287adcade5e90707aa5207a92/src/datasets/load.py#L1240C1-L1240C69
metadata_patterns = get_metadata_patterns(base_path, download_config=self.download_config)
Support huggingface-hub pre-releases
Author: albertvillanova | Pull request: yes | State: closed

Support `huggingface-hub` pre-releases.

This way we will have our CI green when testing `huggingface-hub` release candidates. See: https://github.com/huggingface/datasets/tree/ci-test-huggingface-hub-v0.20.0.rc1

Close #6513.
Cache backward compatibility with 2.15.0
Author: lhoestq | Pull request: yes | State: closed

...for datasets without scripts

It takes into account the changes in cache from
- https://github.com/huggingface/datasets/pull/6493: switch to `config/version/commit_sha` schema
- https://github.com/huggingface/datasets/pull/6454: fix `DataFilesDict` keys ordering when hashing

requires https://github.com/huggingface/datasets/pull/6493 to be merged
Support huggingface-hub 0.20.0
Author: albertvillanova | Pull request: no | State: closed

CI to test the support of `huggingface-hub` 0.20.0: https://github.com/huggingface/datasets/compare/main...ci-test-huggingface-hub-v0.20.0.rc1

We need to merge:
- #6510 
- #6512
- #6516
Implement get dataset default config name
Author: albertvillanova | Pull request: yes | State: closed

Implement `get_dataset_default_config_name`.

Now that we support setting a configuration as default in `push_to_hub` (see #6500), we need a programmatically way to know in advance which is the default configuration. This will be used in the Space to convert script-datasets to Parquet: https://huggingface.co/spaces/albertvillanova/convert-dataset-to-parquet

Follow-up of:
- #6500

CC: @severo
Replace `list_files_info` with `list_repo_tree` in `push_to_hub`
Author: mariosasko | Pull request: yes | State: closed

Starting from `huggingface_hub` 0.20.0, `list_files_info` will be deprecated in favor of `list_repo_tree` (see https://github.com/huggingface/huggingface_hub/pull/1910)
Read GeoParquet files using parquet reader
Author: weiji14 | Pull request: yes | State: closed

Let GeoParquet files with the file extension `*.geoparquet` or `*.gpq` be readable by the default parquet reader.

Those two file extensions are the ones most commonly used for GeoParquet files, and is included in the `gpq` validator tool at https://github.com/planetlabs/gpq/blob/e5576b4ee7306b4d2259d56c879465a9364dab90/cmd/gpq/command/convert.go#L73-L75

Addresses https://github.com/huggingface/datasets/issues/6438
where is glue_metric.py> @Frankie123421 what was the resolution to this?
Author: Mcccccc1024 | Pull request: no | State: closed

              > @Frankie123421 what was the resolution to this?

use glue_metric.py instead of glue.py in load_metric

_Originally posted by @Frankie123421 in https://github.com/huggingface/datasets/issues/2117#issuecomment-905093763_
Incorrect test set labels for RTE and CoLA datasets via load_dataset
Author: emreonal11 | Pull request: no | State: closed

### Describe the bug

The test set labels for the RTE and CoLA datasets when loading via datasets load_dataset are all -1.
Edit: It appears this is also the case for every other dataset except for MRPC (stsb, sst2, qqp, mnli (both matched and mismatched), qnli, wnli, ax). Is this intended behavior to safeguard the test set for evaluation purposes?

### Steps to reproduce the bug

!pip install datasets
from datasets import load_dataset

rte_data = load_dataset('glue', 'rte')
cola_data = load_dataset('glue', 'cola')
print(rte_data['test'][0:30]['label'])
print(cola_data['test'][0:30]['label'])

Output:
[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]
[-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]

The non-label test data seems to be fine:
e.g. rte_data['test'][1] is:
{'sentence1': "Authorities in Brazil say that more than 200 people are being held hostage in a prison in the country's remote, Amazonian-jungle state of Rondonia.",
 'sentence2': 'Authorities in Brazil hold 200 people as hostage.',
 'label': -1,
 'idx': 1}
Training and validation data are also fine:
e.g. rte_data['train][0] is:
{'sentence1': 'No Weapons of Mass Destruction Found in Iraq Yet.',
 'sentence2': 'Weapons of Mass Destruction Found in Iraq.',
 'label': 1,
 'idx': 0}


### Expected behavior

Expected the labels to be binary 0/1 values; Got all -1s instead

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0
Error Pushing to Hub
Author: Jiayi-Pan | Pull request: no | State: closed

### Describe the bug

Error when trying to push a dataset in a special format to hub

### Steps to reproduce the bug

```
import datasets
from datasets import Dataset
dataset_dict = {
    "filename": ["apple", "banana"],
    "token": [[[1,2],[3,4]],[[1,2],[3,4]]],
    "label": [0, 1],
}
dataset = Dataset.from_dict(dataset_dict)
dataset = dataset.cast_column("token", datasets.features.features.Array2D(shape=(2, 2),dtype="int16"))

dataset.push_to_hub("SequenceModel/imagenet_val_256")
```
Error:
```
...

ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/tuple'
  in "<unicode string>", line 8, column 16:
            shape: !!python/tuple
                   ^
```

### Expected behavior

Dataset being pushed to hub

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.19.0-1022-gcp-x86_64-with-glibc2.35
- Python version: 3.11.5
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0
Fix streaming xnli
Author: lhoestq | Pull request: yes | State: closed

This code was failing

```python

In [1]: from datasets import load_dataset

In [2]: 
   ...:         ds = load_dataset("xnli", "all_languages", split="test", streaming=True)
   ...: 
   ...:         sample_data = next(iter(ds))["premise"]  # pick up one data
   ...:         input_text = list(sample_data.values())
```

```
File ~/hf/datasets/src/datasets/features/translation.py:104, in TranslationVariableLanguages.encode_example(self, translation_dict)
    102     return translation_dict
    103 elif self.languages and set(translation_dict) - lang_set:
--> 104     raise ValueError(
    105         f'Some languages in example ({", ".join(sorted(set(translation_dict) - lang_set))}) are not in valid set ({", ".join(lang_set)}).'
    106     )
    108 # Convert dictionary into tuples, splitting out cases where there are
    109 # multiple translations for a single language.
    110 translation_tuples = []

ValueError: Some languages in example (language, translation) are not in valid set (ur, fr, hi, sw, vi, el, de, th, en, tr, zh, ar, bg, ru, es).
```

because in streaming mode we expect features encode methods to be no-ops if the example is already encoded.

I fixed `TranslationVariableLanguages` to account for that
OverflowError: value too large to convert to int32_t 
Author: zhangfan-algo | Pull request: no | State: open

### Describe the bug

![image](https://github.com/huggingface/datasets/assets/47747764/f58044fb-ddda-48b6-ba68-7bbfef781630)


### Steps to reproduce the bug

just loading datasets 

### Expected behavior

how can I fix it

### Environment info

pip install /mnt/cluster/zhangfan/study_info/LLaMA-Factory/peft-0.6.0-py3-none-any.whl
pip install huggingface_hub-0.19.4-py3-none-any.whl tokenizers-0.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl transformers-4.36.1-py3-none-any.whl pyarrow_hotfix-0.6-py3-none-any.whl datasets-2.15.0-py3-none-any.whl tyro-0.5.18-py3-none-any.whl trl-0.7.4-py3-none-any.whl

done
Fallback on dataset script if user wants to load default config
Author: lhoestq | Pull request: yes | State: closed

Right now this code is failing on `main`:

```python
load_dataset("openbookqa")
```

This is because it tries to load the dataset from the Parquet export but the dataset has multiple configurations and the Parquet export doesn't know which one is the default one.

I fixed this by simply falling back on using the dataset script (which tells the user to pass `trust_remote_code=True`):

```python
load_dataset("openbookqa", trust_remote_code=True)
```

Note that if the user happened to specify a config name I don't fall back on the script since we can use the Parquet export in this case (no need to know which config is the default)


```python
load_dataset("openbookqa", "main")
```
Error when writing a dataset to HF Hub: A commit has happened since. Please refresh and try again.
Author: GeorgesLorre | Pull request: no | State: open

**Describe the bug**

Getting a `412 Client Error: Precondition Failed` when trying to write a dataset to the HF hub.

```
huggingface_hub.utils._errors.HfHubHTTPError: 412 Client Error: Precondition Failed for url: https://huggingface.co/api/datasets/GLorr/test-dask/commit/main (Request ID: Root=1-657ae26f-3bd92bf861bb254b2cc0826c;50a09ab7-9347-406a-ba49-69f98abee9cc)

A commit has happened since. Please refresh and try again.
```

**Steps to reproduce the bug**
This is a minimal reproducer:
```
import dask.dataframe as dd
import pandas as pd
import random
import os

import huggingface_hub

import datasets

huggingface_hub.login(token=os.getenv("HF_TOKEN"))

data = {"number": [random.randint(0,10) for _ in range(1000)]}
df = pd.DataFrame.from_dict(data)
dataframe = dd.from_pandas(df, npartitions=1)
dataframe = dataframe.repartition(npartitions=3)

schema = datasets.Features({"number": datasets.Value("int64")}).arrow_schema

repo_id = "GLorr/test-dask"
repo_path = f"hf://datasets/{repo_id}"
huggingface_hub.create_repo(repo_id=repo_id, repo_type="dataset", exist_ok=True)
dd.to_parquet(dataframe, path=f"{repo_path}/data", schema=schema)
```

**Expected behavior**
Would expect to write to the hub without any problem.

**Environment info**
```
datasets==2.15.0
huggingface-hub==0.19.4
```
Image Data loaded Twice
Author: ArcaneLex | Pull request: no | State: open

### Describe the bug

![1702472610561](https://github.com/huggingface/datasets/assets/28867010/4b7ef5e7-32c3-4b73-84cb-5de059caa0b6)
When I learn from https://huggingface.co/docs/datasets/image_load and try to load image data from a folder. I noticed that the image was read twice in the returned data. As you can see in the attached image, there are only four images in the train folder, but reading brings up eight images

### Steps to reproduce the bug

from datasets import Dataset, load_dataset

dataset = load_dataset("imagefolder", data_dir="data/", drop_labels=False)
# print(dataset["train"][0]["image"] == dataset["train"][1]["image"])

print(dataset)
print(dataset["train"]["image"])
print(len(dataset["train"]["image"]))

### Expected behavior

DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 8
    })
})
[<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D1CA8B0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D2452E0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D245310>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D2453A0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D245460>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=2877x2129 at 0x1BD1D245430>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D2454F0>, <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=4208x3120 at 0x1BD1D245550>]
8

### Environment info

- `datasets` version: 2.14.5
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.9.17
- Huggingface_hub version: 0.19.4
- PyArrow version: 13.0.0
- Pandas version: 2.0.3
Lazy data files resolution and offline cache reload
Author: lhoestq | Pull request: yes | State: closed

Includes both https://github.com/huggingface/datasets/pull/6458 and https://github.com/huggingface/datasets/pull/6459

This PR should be merged instead of the two individually, since they are conflicting

## Offline cache reload

 it can reload datasets that were pushed to hub if they exist in the cache.

example:

```python
>>> Dataset.from_dict({"a": [1, 2]}).push_to_hub("lhoestq/tmp")
>>> load_dataset("lhoestq/tmp")
DatasetDict({
    train: Dataset({
        features: ['a'],
        num_rows: 2
    })
})
```

and later, without connection:

```python
>>> load_dataset("lhoestq/tmp")
Using the latest cached version of the dataset since lhoestq/tmp couldn't be found on the Hugging Face Hub
Found the latest cached dataset configuration 'default' at /Users/quentinlhoest/.cache/huggingface/datasets/lhoestq___tmp/default/0.0.0/da0e902a945afeb9 (last modified on Wed Dec 13 14:55:52 2023).
DatasetDict({
    train: Dataset({
        features: ['a'],
        num_rows: 2
    })
})
```

- Updated `CachedDatasetModuleFactory` to look for datasets in the cache at `<namespace>___<dataset_name>/<config_id>`
- Since the metadata configs parameters are not available in offline mode, we don't know which folder to load (config_id and hash change), so I simply load the latest one
  - I instantiate a BuilderConfig even if there is no metadata config with the right config_name
  - Its config_id is equal to the config_name to be able to retrieve it in the cache (no more suffix for configs from metadata configs)
  - We can reload this config if offline mode by specifying the right config_name (same as online !)
- Consequences of this change:
  - Only when there are user's parameters it creates a custom builder config with config_id = config_name + user parameters hash
  - the hash used to name the cache folder takes into account the metadata config and the dataset info, so that the right cache can be reloaded when there is internet connection without redownloading the data or resolving the data files. For local directories I hash the builder configs and dataset info, and for datasets on the hub I use the commit sha as hash.
  - cache directories now look like `config/version/commit_sha` for hub datasets which is clean :)

Fix https://github.com/huggingface/datasets/issues/3547

## Lazy data files resolution

this makes this code run in 2sec instead of >10sec

```python
from datasets import load_dataset

ds = load_dataset("glue", "sst2", streaming=True, trust_remote_code=False)
```

For some datasets with many configs and files it can be up to 100x faster.
This is particularly important now that some datasets will be loaded from the Parquet export instead of the scripts.

The data files are only resolved in the builder `__init__`. To do so I added DataFilesPatternsList and DataFilesPatternsDict that have `.resolve()` to return resolved DataFilesList and DataFilesDict
Make push_to_hub return CommitInfo
Author: albertvillanova | Pull request: yes | State: closed

Make `push_to_hub` return `CommitInfo`.

This is useful, for example, if we pass `create_pr=True` and we want to know the created PR ID.

CC: @severo for the use case in https://huggingface.co/datasets/jmhessel/newyorker_caption_contest/discussions/4