helmo/github-issues huggingface dataset:
Columns in the dataset obtained though load_dataset do not correspond to the one in the dataset viewer since 3.4.0
State: open | Author: bruno-hays (CONTRIBUTOR)

### Describe the bug

I have noticed that on my dataset named [BrunoHays/Accueil_UBS](https://huggingface.co/datasets/BrunoHays/Accueil_UBS), since the version 3.4.0, every column except audio is missing when I load the dataset.

Interestingly, the dataset viewer still shows the correct columns

### Steps to reproduce the bug

```python
from datasets import load_dataset
ds = load_dataset("BrunoHays/Accueil_UBS", streaming=True)
print(next(iter(ds["test"])).keys())
```
With datasets >= 3.4.0: 
-> dict_keys(['audio'])
With datasets == 3.3.2:
-> dict_keys(['audio', 'id', 'speaker', 'sentence', 'raw_sentence', 'start_timestamp', 'end_timestamp', 'overlap'])


### Expected behavior

All the columns should be present

### Environment info

- `datasets` version: 3.3.2
- Platform: macOS-14.6.1-x86_64-i386-64bit
- Python version: 3.10.15
- `huggingface_hub` version: 0.30.1
- PyArrow version: 16.1.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.10.0


Comments: Hi, the dataset viewer shows all the possible columns and their types, but `load_dataset()` iterates through all the columns that you defined. It seems that you only have one column (‘audio’) defined in your dataset because when I ran `print(ds.column_names)`, the only name I got was “audio”. You need to clearly define all the other features of the dataset as columns to enable your original code to work. Furthermore, you can run this code to print out all the features of your dataset: 
```python
from datasets import load_dataset_builder
ds_builder = load_dataset_builder("BrunoHays/Accueil_UBS")
print(ds_builder.info.features)
```
Broken links in pdf loading documentation
State: closed | Author: VyoJ (NONE)

### Describe the bug

Hi, just a couple of small issues I ran into while reading the docs for [loading pdf data](https://huggingface.co/docs/datasets/main/en/document_load):

1. The link for the [`Create a pdf dataset`](https://huggingface.co/docs/datasets/main/en/document_load#pdffolder) points to https://huggingface.co/docs/datasets/main/en/pdf_dataset instead of https://huggingface.co/docs/datasets/main/en/document_dataset and hence gives a 404 error.

2. At the top of the page, it's mentioned that to work with pdf datasets we need to have the `pdfplumber` package installed but the link to its installation guide points to `pytorch/vision` [installation instructions](https://github.com/pytorch/vision#installation) instead of `pdfplumber`'s [guide](https://github.com/jsvine/pdfplumber#installation)

I love the work on enabling pdf dataset support and these small tweaks would help everyone navigate the docs better. Thanks!

### Steps to reproduce the bug

The issue is on the [Load Document Data](https://huggingface.co/docs/datasets/main/en/document_load) page of the datasets docs.

### Expected behavior

1. For solving the first issue, I went through the [source .mdx code](https://github.com/huggingface/datasets/blob/main/docs/source/document_load.mdx?plain=1#L188) of the datasets docs and found that the link is pointing to `./pdf_dataset` instead of `./document_dataset`

2. For the second issue, I went through the [source .mdx code](https://github.com/huggingface/datasets/blob/main/docs/source/document_load.mdx?plain=1#L13) of the datasets docs and found that the link is `pytorch/vision` [installation instructions](https://github.com/pytorch/vision#installation) instead of `pdfplumber`'s [guide](https://github.com/jsvine/pdfplumber#installation)

Just replacing these two links should fix the bugs

### Environment info

datasets v3.5.0 (main at the time of writing)

Comments: thanks for reporting ! I fixed the links, the docs will be updated in the next release
push_to_hub does not upload videos
State: open | Author: DominikVincent (NONE)

### Describe the bug

Hello,

I would like to upload a video dataset (some .mp4 files and some segments within them), i.e. rows correspond to subsequences from videos. Videos might be referenced by several rows.

I created a dataset locally and it references the videos and the video readers can read them correctly. I use push_to_hub() to upload the dataset to the hub.

Expectation: A user uses `load_dataset` and can load the videos.

However, the videos seem to be just referenced via paths on the computer and not uploaded to the hub. Therefore a target user cannot load the videos in the dataset.

### Steps to reproduce the bug

1. create a video dataset with paths e.g. { ["videos"]: [path1, path2, ...]}
2. dataset.push_to_hub
3. on a different computer (or same pc if relative paths are used in a different folder):
```
dataset = load_dataset("siplab/egosim", split="train")
video = dataset[0]["video_head"]
```

3. will fail

### Expected behavior

Expectation: A user uses `load_dataset` and can load the videos.

### Environment info

datasets                       3.1.0
Python 3.8.18

Comments: Hi ! the `Video` type is still experimental, and in particular `push_to_hub` doesn't upload videos at the moment (only the paths).

There is an open question to either upload the videos inside the Parquet files, or rather have them as separate files (which is great to enable remote seeking/streaming)
(refactor) remove redundant logic in _check_valid_index_key
State: open | Author: suzyahyah (NONE)

This PR contributes a minor refactor, in a small function in `src/datasets/formatting/formatting.py`. No change in logic. 

In the original code, there are separate if-conditionals for `isinstance(key, range)` and `isinstance(key, Iterable)`, with essentially the same logic. 

This PR combines these two using a single if statement.

**Considerations**

1. Although range in python is guaranteed to have integers, internally calling `int()` on an object that is already an int is negligible. (In python it returns the original object. It doesn't create a new integer object or perform any actual conversion)

2. Technically a range is already an Iterable, and we could just do `isinstance(key, Iterable)` but I explicitly did `isinstance(key, (range, Iterable))` just to be super obvious and consistent that both cases are handled because I see `slice, range, Iterable` everywhere in this `formatting.py`

3. This PR removes the `if len(key)>0` conditional. I think it is cleaner to have it this way for three reasons.
- There was originally no else statement and the code would have failed silently anyway.  
- The if len(key)>0 should be caught much earlier, rather than in `formatting.py`. 
- There are actually multiple cases where this would fail, if len(key)>0, if key is non numeric or float, or if key is a list of lists. It's clunky to state all this and the error be thrown during max or indexing. 


**Previous PR and Issues Checks**

1. No known PR or Issues (both closed or open) in hf datasets repository 

**Tests**

1. Tested using Dataset (load_dataset("wikitext", "wikitext-103-raw-v1")), Pytorch DataLoader, with a Pytorch BatchSampler (list of indexes returned instead of single index).
Support underscore int read instruction
State: closed | Author: lhoestq (MEMBER)

close https://github.com/huggingface/datasets/issues/7481

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7488). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

you rock, Quentin - thank you!
Write pdf in map
State: closed | Author: lhoestq (MEMBER)

Fix this error when mapping a PDF dataset

```
pyarrow.lib.ArrowInvalid: Could not convert <pdfplumber.pdf.PDF object at 0x13498ee40> with type PDF: did not recognize Python value type when inferring an Arrow data type
```

and also let map() outputs be lists of images or pdfs

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7487). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support skip_trying_type 
State: closed | Author: yoshitomo-matsubara (CONTRIBUTOR)

This PR addresses Issue #7472

cc: @lhoestq 

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7483). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Cool ! Can you run `make style` to fix code formatting ?

I was also thinking of naming the argument `try_original_type` and have it `True` by default

@lhoestq 

Thank you for the suggestion! I renamed the argument with `True` by default and ran `make style`
Does it look good?

Thanks @lhoestq !

Let me know if there are anything that I can do for this PR. Otherwise, looking forward to seeing this update in the package soon!

CI failures are unrelated, merging :)

Great, thanks for your support!
I can't wait for the next release :)
deal with python `10_000` legal number in slice syntax
State: closed | Author: sfc-gh-sbekman (NONE)
Labels: name: enhancement

### Feature request

```
In [6]: ds = datasets.load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:1000]")

In [7]: ds = datasets.load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:1_000]")
[dozens of frames skipped]
File /usr/local/lib/python3.10/dist-packages/datasets/arrow_reader.py:444, in _str_to_read_instruction(spec)
    442 res = _SUB_SPEC_RE.match(spec)
    443 if not res:
--> 444     raise ValueError(f"Unrecognized instruction format: {spec}")
ValueError: Unrecognized instruction format: train_sft[:1_000]
```

It took me a while to understand what the problem was. But apparently `pyarrow` doesn't allow python numbers that may include `_` as in `1_000`. The `_` aids readability since `10_000_000` vs `10000000` is obviously easier to grasp of what the actual number is.

Feature request:

ideally `datasets` being a python module will do the right thing and convert python numbers into whatever pyarrow supports - in this case stripping `_`s.

Second best it'd err and tell the user that using numbers with `_` in split slices is not acceptible, so that the user won't have to deal with a huge pyarrow assert they know nothing about.

Thank you!

Comments: should be an easy fix, I opened a PR
HF_DATASETS_CACHE ignored?
State: open | Author: stephenroller (NONE)

### Describe the bug

I'm struggling to get things to respect HF_DATASETS_CACHE.

Rationale: I'm on a system that uses NFS for homedir, so downloading to NFS is expensive, slow, and wastes valuable quota compared to local disk. Instead, it seems to rely mostly on HF_HUB_CACHE.

Current version: 3.2.1dev. In the process of testing 3.4.0

### Steps to reproduce the bug

[Currently writing using datasets 3.2.1dev. Will follow up with 3.4.0 results]

dump.py:
```python
from datasets import load_dataset
dataset = load_dataset("HuggingFaceFW/fineweb", name="sample-100BT", split="train")
```

Repro steps
```bash
# ensure no cache
$ mv ~/.cache/huggingface ~/.cache/huggingface.bak

$ export HF_DATASETS_CACHE=/tmp/roller/datasets
$ rm -rf ${HF_DATASETS_CACHE}
$ env | grep HF | grep -v TOKEN
HF_DATASETS_CACHE=/tmp/roller/datasets

$ python dump.py
# (omitted for brevity)

# (while downloading) 
$ du -hcs ~/.cache/huggingface/hub
18G     hub
18G     total

# (after downloading)
$ du -hcs ~/.cache/huggingface/hub
```

It's a shame because datasets supports s3 (which I could really use right now) but hub does not.

### Expected behavior

* ~/.cache/huggingface/hub stays empty
* /tmp/roller/datasets becomes full of stuff

### Environment info

[Currently writing using datasets 3.2.1dev. Will follow up with 3.4.0 results]

Comments: FWIW, it does eventually write to /tmp/roller/datasets when generating the final version.

Hey, I’d love to work on this issue but I am a beginner, can I work it with you?

Hi @lhoestq,
I'd like to look into this issue but I'm still learning. Could you share any quick pointers on the HF_DATASETS_CACHE behavior here? Thanks!

Hi ! `HF_DATASETS_CACHE` is only for the cache files of the `datasets` library, not for the `huggingface_hub` cache for files downloaded from the Hugging Face Hub.

You should either specify `HF_HOME` (parent cache path for everything HF) or both `HF_DATASETS_CACHE` and `HF_HUB_CACHE`



Thanks for clarifying, @lhoestq! To make sure I’ve got it right:

1. **HF_DATASETS_CACHE** only controls where the **datasets** library writes its own cache files (e.g. processed shards, Arrow files, etc.).
2. Anything downloaded via **huggingface_hub** (models, tokenizers, raw files) still goes into the Hub cache (by default `~/.cache/huggingface/hub`), unless you set **HF_HUB_CACHE** or the parent **HF_HOME**.

So if you want everything off NFS and onto local disk you have two options:

- **Set both**  
  ```bash
  export HF_DATASETS_CACHE=/tmp/roller/datasets  
  export HF_HUB_CACHE=/tmp/roller/hub
  ```
- **Or set**  
  ```bash
  export HF_HOME=/tmp/roller
  ```
  which will apply to both subdirectories.

Is that correct? And would it make sense to add a note to the docs clarifying the distinction (or even support S3 for the Hub cache in the future)? I’m happy to draft a small docs PR if that would help.

Hi, yes that's correct, thanks for the clarification ! A note in the docs would be welcome, thanks
Features.from_arrow_schema is destructive
State: open | Author: BramVanroy (CONTRIBUTOR)

### Describe the bug

I came across this, perhaps niche, bug where `Features` does not/cannot account for pyarrow's `nullable=False` option in Fields. Interestingly, I found that in regular "flat" fields this does not necessarily lead to conflicts, but when a non-nullable field is in a struct, an incompatibility arises.

It's not easy to explain in words, so the minimal example below should help I hope.

Note that I suggest a solution in the comments in the code, simply allowing `Dataset.to_parquet` to allow for a `schema` argument which, when provided, will override the default ds.features.arrow_schema.

### Steps to reproduce the bug

```python
import os
from datasets import Dataset, Features

import pyarrow as pa
import pyarrow.parquet as pq

# HF datasets is destructive when you call Features.from_arrow_schema(schema) on a schema 
# because it will not account for nullable and non-nullable fields in structs (it will always allow nullable)
# Reloading the same dataset with the original schema will raise an error because the schema is not the same anymore
non_nullable_schema = pa.schema(
    [
        pa.field("text", pa.string(), nullable=False),
        pa.field("meta",
            pa.struct(
                [
                    pa.field("date", pa.list_(pa.string()), nullable=False),
                ],
            ),
        ),

    ]
)
print("ORIGINAL SCHEMA")
print(non_nullable_schema)
print()

feats = Features.from_arrow_schema(non_nullable_schema)

print("FEATUR-IZED SCHEMA (nullable-restrictions are gone)")
print(feats.arrow_schema)
print()

ds = Dataset.from_dict(
    {
        "text": ["a", "b", "c"],
        "meta": [{"date": ["2021-01-01"]}, {"date": ["2021-01-02"]}, {"date": ["2021-01-03"]}],
    },
    features=feats,
)

fname = "tmp.parquet"

# This is not possible: TypeError: pyarrow.parquet.core.ParquetWriter() got multiple values for keyword argument 'schema'
# Though I believe this would be the easiest fix: allow schema to be passed to to_parquet and overwrite the schema in the dataset
# ds.to_parquet(fname, schema=non_nullable_schema)

ds.to_parquet(fname)

try:
    _ = pq.read_table(fname, schema=non_nullable_schema)
finally:
    os.unlink(fname)
```


### Expected behavior

- Non-destructive behavior when converting an arrow schema to Features; or
- the ability to override the default arrow schema with a custom one

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.14.0-427.20.1.el9_4.x86_64-x86_64-with-glibc2.34
- Python version: 3.11.10
- `huggingface_hub` version: 0.27.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
update fsspec 2025.3.0
State: closed | Author: peteski22 (CONTRIBUTOR)

It appears there have been two releases of fsspec since this dependency was last updated, it would be great if Datasets could be updated so that it didn't hold back the usage of newer fsspec versions in consuming projects.

PR based on https://github.com/huggingface/datasets/pull/7352

Comments: Sorry for tagging you @lhoestq but since you merged the linked PR, I wondered if you might be able to help me get this triaged so it can be reviewed/rejected etc. 🙏🏼 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7478). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
What is the canonical way to compress a Dataset?
State: open | Author: eric-czech (NONE)

Given that Arrow is the preferred backend for a Dataset, what is a user supposed to do if they want concurrent reads, concurrent writes AND on-disk compression for a larger dataset?

Parquet would be the obvious answer except that there is no native support for writing sharded, parquet datasets concurrently [[1](https://github.com/huggingface/datasets/issues/7047)].

Am I missing something?  

And if so, why is this not the standard/default way that `Dataset`'s work as they do in Xarray, Ray Data, Composer, etc.?

Comments: I saw this post by @lhoestq: https://discuss.huggingface.co/t/increased-arrow-table-size-by-factor-of-2/26561/4 suggesting that there is at least some internal code for writing sharded parquet datasets non-concurrently.  This appears to be that code: https://github.com/huggingface/datasets/blob/94ccd1b4fada8a92cea96dc8df4e915041d695b6/src/datasets/arrow_dataset.py#L5380-L5397

Is there any fundamental reason (e.g. race conditions) that this kind of operation couldn't exist as a utility or method on a `Dataset` with a `num_proc` argument?  I am not seeing any other issues explicitly for that ask. 


We simply haven't implemented a method to save as sharded parquet locally yet ^^'

Right now the only sharded parquet export method is `push_to_hub()` which writes to HF. But we can have a local one as well. 

In the meantime the easiest way to export as sharded parquet locally is to `.shard()` and `.to_parquet()` (see code from my comment [here](https://github.com/huggingface/datasets/issues/7047#issuecomment-2233163406))

> In the meantime the easiest way to export as sharded parquet locally is to .shard() and .to_parquet()

Makes sense, BUT how can it be done concurrently?  I could of course use multiprocessing myself or a dozen other libraries for parallelizing single-node/local operations like that.

What I'm asking though is, what is the way to do this that is most canonical for `datasets` specifically?  I.e. what is least likely to causing pickling or other issues because it is used frequently internally by `datasets` and already likely tests for a lot of library-native edge-cases?

Everything in `datasets` is picklable :) and even better: since the data are memory mapped from disk, pickling in one process and unpickling in another doesn't do any copy - it instantaneously reloads the memory map.

So feel free to use the library you prefer to parallelize your operations.

(it's another story in distributed setups though, because in that case you either need to copy and send the data or setup a distributed filesystem)
Priotitize json
State: closed | Author: lhoestq (MEMBER)

`datasets` should load the JSON data in https://huggingface.co/datasets/facebook/natural_reasoning, not the PDF

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7476). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove conditions for Python < 3.9
State: closed | Author: cyyever (CONTRIBUTOR)

This PR remove conditions for Python < 3.9.

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7474). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks ! can you run `make style` to fix code formatting ? then we can merge

@lhoestq Done
Webdataset data format problem
State: closed | Author: edmcman (NONE)

### Describe the bug

Please see https://huggingface.co/datasets/ejschwartz/idioms/discussions/1

Error code: FileFormatMismatchBetweenSplitsError

All three splits, train, test, and validation, use webdataset. But only the train split has more than one file. How can I force the other two splits to also be interpreted as being the webdataset format?  (I don't think there is currently a way, but happy to be told that I am wrong.)

### Steps to reproduce the bug

```
import datasets
datasets.load_dataset("ejschwartz/idioms")

### Expected behavior

The dataset loads.  Alternatively, there is a YAML syntax for manually specifying the format.

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-6.8.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.28.1
- PyArrow version: 19.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments: I was able to work around it
Adding argument to `_get_data_files_patterns`
State: closed | Author: SangbumChoi (NONE)
Labels: name: enhancement

### Feature request

How about adding if the user already know about the pattern?

https://github.com/huggingface/datasets/blob/a256b85cbc67aa3f0e75d32d6586afc507cf535b/src/datasets/data_files.py#L252

### Motivation

While using this load_dataset people might use 10M of images for the local files.
However, due to searching all the appropriate file pattern in fsspec, purely searching this pattern takes more than 10 hours (real use-case).

### Your contribution

Yeah I can make this happen if this seems valid. @lhoestq WDYT?
such like
```
def _get_data_files_patterns(pattern_resolver: Callable[[str], list[str]], patterns: PATTERNS) -> dict[str, list[str]]:
```

Comments: Hi ! The pattern can be specified in advance in YAML in the README.md of the dataset :)

For example

```
---
configs:
- config_name: default
  data_files:
  - split: train
    path: "train/*"
  - split: test
    path: "test/*"
---
```

See the docs at https://huggingface.co/docs/hub/en/datasets-manual-configuration

@lhoestq How can we choose in this case ? https://huggingface.co/datasets/datasets-examples/doc-image-5


choose what ? sorry I didn't get it ^^'
Custom split name with the web interface
State: closed | Author: vince62s (NONE)

### Describe the bug

According the doc here: https://huggingface.co/docs/hub/datasets-file-names-and-splits#custom-split-name
it should infer the split name from the subdir of data or the beg of the name of the files in data.
When doing this manually through web upload it does not work. it uses "train" as a unique split.
example: https://huggingface.co/datasets/eole-nlp/estimator_chatml

### Steps to reproduce the bug

follow the link above

### Expected behavior

there should be two splits "mlqe" and "1720_da"

### Environment info

website