noamaanMulla-03/datasets-issues huggingface dataset:
Fix docstring of Json.cast_storage

Fix docstring of `Json.cast_storage`.

This PR makes a minor change to the `cast_storage` method in the `Json` feature, updating its return type and documentation to better reflect its intended functionality.

* Updated the `cast_storage` method to return a `pa.JsonArray` instead of a `pa.Int64Array`, and revised the docstring to accurately describe the return type.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8080). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix split_dataset_by_node step

fix https://github.com/huggingface/datasets/issues/8079

I also took the opportunity to improve the arrow logic to not have to do an unnecessary python -> arrow -> python in FormattedExamplesIterable when streaming webdatasets


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8081). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix regression when loading JSON with one file = one object

was failing for datasets with JSON files that are not JSON Lines and that contain one single multi-line JSON object like https://huggingface.co/datasets/evaleval/EEE_datastore


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8086). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
all_exhausted_without_replacement working same as first_exhausted

### Describe the bug

>>> from datasets import Dataset, interleave_datasets
>>> d1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> d2 = Dataset.from_dict({"a": [10, 11, 12, 13]})
>>> d3 = Dataset.from_dict({"a": [20, 21, 22]})
>>> dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted_without_replacement")
>>> dataset["a"][:100]
[0, 10, 20, 1, 11, 21, 2, 12, 22]

expected output: [0, 10, 20, 1, 11, 21, 2, 12, 22,13]

datasets version 4.5.0

### Steps to reproduce the bug

>>> from datasets import Dataset, interleave_datasets
>>> d1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> d2 = Dataset.from_dict({"a": [10, 11, 12, 13]})
>>> d3 = Dataset.from_dict({"a": [20, 21, 22]})
>>> dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted_without_replacement")
>>> dataset["a"][:100]

### Expected behavior

 [0, 10, 20, 1, 11, 21, 2, 12, 22,13]

### Environment info

datasets version 4.5.0


https://github.com/huggingface/datasets/pull/7955

it was fixed by https://github.com/huggingface/datasets/pull/7955
ARM audio support broken

### Describe the bug

`datasets` 4.0 moved to using `torchcodec` for audio datasets, which doesn't support ARM hardware under Linux and Windows


### Steps to reproduce the bug

try: 
* Try to install `torchcodec` under Linux ARM64

or:
* Try to install `torchcodec` under Windows ARM64

or, on either of those platforms, using datasets >= 4.0:
* `dataset = load_dataset("garak-llm/audio_achilles_heel")`


### Expected behavior

dataset will load as expected

### Environment info

ARM environment

Python 3.13+

`datasets>=4.0`

Using `torchaudio`


See also https://github.com/NVIDIA/garak/issues/1622#issuecomment-3937720381

Hi @leondz , thanks for the detailed bug report — the description and reproduction steps are really clear.

I’d love to take a shot at investigating and potentially fixing this issue. Before I start digging into the codebase, I just wanted to check if there are any additional details, context, or pointers that might help get started (e.g., specific parts of the codebase related to this behavior).

Also, if there are any edge cases or constraints I should be aware of while reproducing the issue, that would be really helpful.

Thanks again for the thorough report!


torchcodec works on ARM, see the related discussion on https://github.com/meta-pytorch/torchcodec/issues/569

I see it's missing pure-CPU wheels though

Could you see with the torchcodec community what's missing for you to be able to use torchcodec ?

Thanks, also pursuing there. I want to check: in the interim, is is intended for `datasets`  be broken for audio under aarch64/woa64 when installed with `pip`, until `torchcodec` supply appropriate wheels?

If the question is: do we plan to implement a temporary workaround in the meantime, the answer is unfortunately no.
This would be a big effort to fix a temporary situation, while using older versions of `datasets` is actually a simpler workaround.

If older versions of `datasets` are not working for your case and you really have no other option, I'd be happy to review PRs that offer patches on older versions and I can make a patch release.
Broken link in documentation for "Create a video dataset"

The link to "the [WebDataset documentation](https://webdataset.github.io/webdataset)." is broken. 
https://huggingface.co/docs/datasets/main/en/video_dataset#webdataset 

<img width="2048" height="264" alt="Image" src="https://github.com/user-attachments/assets/975dd10c-aad8-42fc-9fbc-de0e2747a326" />


The URL is ok but it seems the webdataset website is down. There seems to be a related issue here: https://github.com/webdataset/webdataset/issues/155

Feel free to ask the authors there for an update. Otherwise happy to witch the link to the mirror shared in that issue

I fixed this doc problem.
PR link: https://github.com/huggingface/datasets/pull/8107
Inconsistent loading of LFS-hosted files in Dangindev/viet-cultural-vqa dataset

### Describe the bug

Describe the bug
Some files in the Dangindev/viet-cultural-vqa dataset fail to load via the Hugging Face datasets library.

xet-hosted files load fine
LFS-hosted files sometimes fail

Example:

Fails: https://huggingface.co/datasets/Dangindev/viet-cultural-vqa/blob/main/images/am_thuc/banh_chung_Tet/000004.jpg
Works: https://huggingface.co/datasets/Dangindev/viet-cultural-vqa/blob/main/images/am_thuc/banh_bao/000001.jpg



### Steps to reproduce the bug

Steps to reproduce the bug

`from datasets import load_dataset

ds = load_dataset(
    "Dangindev/viet-cultural-vqa",
    data_files="images/am_thuc/banh_chung_Tet/000001.jpg",
)`

Or I can't access in my browser: 

<img width="1338" height="955" alt="Image" src="https://github.com/user-attachments/assets/e4c24c81-ca54-4405-af3b-83f76158da9b" />

### Expected behavior

I can download this dataset

### Environment info

Python or access through browser


I also have been stuck in same problem 

This should be fixed, thanks for reporting ! let us know if you ever encounter other issues in the future

It's okay. 

Good job, thanks your support!
#5354: replace list with Sequence in from_parquet type hints

\This PR replaces `list` type hints with `Sequence` in `from_parquet` to improve type checking.

Note: Local pytest errors on Python 3.13 due to removal of `distutils` are unrelated to this change.



Hi! I'd like to work on this.
I noticed PR #7953 has been open since January with merge conflicts and no maintainer review, and only covers from_parquet in arrow_dataset.py. I'd like to open a more complete PR replacing List with Sequence across the broader codebase in src/datasets/.
Happy to start immediately if a maintainer can confirm the preferred scope.
#self-assign
add sample_by="document" support for jsonl files

With more agentic traces (_ie Claude Code / Codex / Pi etc_) being published to the Hub as JSONL files, it would be nice to easily support loading / viewing them in Dataset Viewer. 

Datasets like https://huggingface.co/datasets/cfahlgren1/agent-sessions contain one JSONL file per session. Without sample_by, each line becomes a separate row, losing the file-level grouping. With sample_by="document", each file is preserved as a single row:

```python
  ds = load_dataset("cfahlgren1/agent-sessions", sample_by="document")
  # Each row is one session file
```

  Also configurable via YAML metadata:

```yaml
  ---
  configs:
    - config_name: default
      data_files: "*.jsonl"
      sample_by: "document"
  ---
```



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8111). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

would the auto-detection of agent harness (claude vs. codex vs. opencode etc) and other metadata, live in datasets in your opinion, @lhoestq? or in dataset-server, or in the frontend?

> would the auto-detection of agent harness (claude vs. codex vs. opencode etc) and other metadata, live in datasets in your opinion, @lhoestq? or in dataset-server, or in the frontend?

would be nice personally if we could have it live in datasets-server so we can do less parsing in frontend and add some sort of detection of trace datasets 

<img width="833" height="177" alt="image" src="https://github.com/user-attachments/assets/ab76e78e-14d4-417b-8322-44de2de9d415" />


closing this in favor of https://github.com/huggingface/datasets/pull/8113
fix: replace List with Sequence in type hints for public API functions (#5354)

## Summary

Fixes #5354

Replaces `list[X]` with `Sequence[X]` in function argument type annotations 
across the public API. `List` is invariant while `Sequence` is covariant, 
which means passing a `list` to a parameter typed as `List` causes mypy errors, 
even though lists are valid sequences.

## Changes

- `src/datasets/arrow_dataset.py` — `path_or_paths`, `columns`, `input_columns`, `remove_columns`
- `src/datasets/iterable_dataset.py` — same parameters across equivalent methods  
- `src/datasets/combine.py` — `datasets` and `probabilities` parameters
- `src/datasets/dataset_dict.py` — `column_names` and `columns` parameters

## What was NOT changed

- Docstrings (separate concern)
- Internal variables and function bodies
- Return types
- Non-public/internal helpers

## Notes

PR #7953 addressed only `from_parquet` in `arrow_dataset.py` and had merge 
conflicts. This PR covers the broader public API consistently.


Updated — public Dataset.map(), Dataset.filter(), IterableDataset.map() and IterableDataset.filter() signatures now also use Sequence[str] consistently. Thanks for the pointer.

Note: PR #8067 by @biefan also addresses path_or_paths in arrow_dataset.py. This PR covers a broader scope across 4 files and additional parameters (columns, input_columns, remove_columns, map(), filter()). Happy to coordinate with @biefan or adjust scope based on maintainer preference.
feat: implement iter_arrow for skip, take and step iterables

This commit optimizes streaming operations by implementing [_iter_arrow](cci:1://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:377:4-391:57) for [SkipExamplesIterable](cci:2://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:1798:0-1892:42), [TakeExamplesIterable](cci:2://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:1944:0-2048:42), and [StepExamplesIterable](cci:2://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:667:0-743:42).

### Key Changes:
- **Fast Batch Processing**: Enabled batch-level slicing for `.skip(n)` and `.take(n)` on streaming datasets, bypassing slow row-by-row iteration.
- **Optimized Sharding**: Updated [StepExamplesIterable](cci:2://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:667:0-743:42) (used in distributed training) to use Arrow's `.take()` to extract multiple records from a batch simultaneously.
- **State Preservation**: Reinforced [_init_state_dict](cci:1://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:505:4-514:31) and [load_state_dict](cci:1://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:2382:4-2430:46) to support flawless checkpointing and resumption while using Arrow iteration.

### Performance Impact:
Users will experience significant performance gains when skipping or taking examples in streaming mode. By staying in the "Arrow path" and avoiding Python dictionary conversions, data loading overhead is drastically reduced, especially for large-scale training jobs.

### Testing:
Integrated 6 new unit tests into [tests/test_iterable_dataset.py](cci:7://file:///c:/Users/ASUS/Desktop/datasets/tests/test_iterable_dataset.py:0:0-0:0) to verify:
- Functional correctness for [skip](cci:1://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:3149:4-3191:9), [take](cci:1://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:3236:4-3271:9), and `step` using Arrow iteration.
- Reliable state checkpointing and resumption after partial iteration.
Add reshard support for JSON Lines and CSV file formats

This PR extends the IterableDataset.reshard() functionality to support JSON Lines and CSV file formats, building on the foundation laid in #7992.

### Summary
- JSON Lines : Resharding splits files by line boundaries (only when num_shards is specified)
- CSV : Resharding splits files by line boundaries (only when num_shards is specified)
- Parquet : Already supported via row groups (enhanced to support num_shards parameter)


### Usage
```python
from datasets import load_dataset

# JSON Lines
ds = load_dataset("json", data_files=["data.jsonl"], streaming=True)
resharded_ds = ds['train'].reshard(num_shards=100)

# CSV
ds = load_dataset("csv", data_files=["data.csv"], streaming=True)
resharded_ds = ds['train'].reshard(num_shards=100)

# Parquet
ds = load_dataset("parquet", data_files=["data.parquet"], streaming=True)
resharded_ds = ds['train'].reshard(num_shards=100)
```


@lhoestq , hi, lhoestq, could you please help review this PR? Thanks!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8084). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Cool ! I left a few comments. Mostly on CSV but this also applies to JSON since the logic is almost the same. Let me know what you think :)

@lhoestq Hi, lhoestq, I've made the updates accordingly, could you please take another look? 😊

Hi @lhoestq , just a gentle ping on this. I've re-verified the byte-offset logic for CSV/JSONL and confirmed it handles large files efficiently. Let me know if you have any concerns or would like me to adjust the implementation further. Looking forward to your feedback!
🔒 Pin GitHub Actions to commit SHAs

## 🔒 Pin GitHub Actions to commit SHAs

This PR pins all GitHub Actions to their exact commit SHA instead of mutable tags or branch names.

**Why?**
Pinning to a SHA prevents supply chain attacks where a tag (e.g. `v4`) could be moved to point to malicious code.

### Changes

| Workflow | Action | Avant | Après | SHA |
|---|---|---|---|---|
| `trufflehog.yml` | `actions/checkout` | `v5` | `v6.0.2` | `de0fac2e4500…` |
| `trufflehog.yml` | `trufflesecurity/trufflehog` | `main` | `main` | `6bd2d14f7a4b…` |
| `ci.yml` | `actions/checkout` | `v4` | `v6.0.2` | `de0fac2e4500…` |
| `ci.yml` | `actions/setup-python` | `v5` | `v5` | `a26af69be951…` |
| `ci.yml` | `actions/checkout` | `v4` | `v6.0.2` | `de0fac2e4500…` |
| `ci.yml` | `actions/setup-python` | `v5` | `v5` | `a26af69be951…` |
| `ci.yml` | `conda-incubator/setup-miniconda` | `v2` | `v2` | `9f54435e0e72…` |
| `ci.yml` | `actions/checkout` | `v4` | `v6.0.2` | `de0fac2e4500…` |
| `ci.yml` | `actions/setup-python` | `v5` | `v5` | `a26af69be951…` |
| `ci.yml` | `conda-incubator/setup-miniconda` | `v2` | `v2` | `9f54435e0e72…` |
| `ci.yml` | `actions/checkout` | `v4` | `v6.0.2` | `de0fac2e4500…` |
| `ci.yml` | `actions/setup-python` | `v5` | `v5` | `a26af69be951…` |
| `ci.yml` | `conda-incubator/setup-miniconda` | `v2` | `v2` | `9f54435e0e72…` |
| `build_documentation.yml` | `huggingface/doc-builder/.github/workflows/build_main_documentation.yml` | `main` | `main` | `90b4ee2c10b8…` |
| `build_pr_documentation.yml` | `huggingface/doc-builder/.github/workflows/build_pr_documentation.yml` | `main` | `main` | `90b4ee2c10b8…` |
| `upload_pr_documentation.yml` | `huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml` | `main` | `main` | `90b4ee2c10b8…` |

> 🤖 Generated by `/github-actions-audit` — [security/pin-actions-to-sha]


Closes huggingface/tracking-issues#91


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8114). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: preserve features in IterableDataset.add_column (fixes #5752)

## Summary

Fixes #5752.

`IterableDataset.add_column()` lost the dataset's feature schema on every call because it delegated to `map()` without passing the `features` argument. `map()` unconditionally writes `info.features = features`, so the default `None` silently overwrote the existing schema.

**Root cause** (`iterable_dataset.py:3459`):
```python
info = self.info.copy()
info.features = features  # None when add_column() doesn't pass features
```

**Fix**: infer the new column's Arrow type via the existing `_infer_features_from_batch` helper, merge it with the existing features, and pass the result to `map()`.

```python
new_features = None
if self._info.features is not None:
    column_features = _infer_features_from_batch({name: list(column)})
    new_features = Features({**self._info.features, **column_features})
return self.map(..., features=new_features)
```

If the dataset had no features to begin with (`_info.features is None`), behaviour is unchanged — `map()` receives `None` as before.

## Test plan

- [ ] Reproduce the original issue: create a streaming dataset with known features, call `.add_column()`, assert `.features` is not None and contains the new column
- [ ] Verify existing `test_iterable_dataset.py` tests still pass
- [ ] Smoke-test with numpy array column input (not just list)


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8115). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Document YAML configuration with "data_dir"

See https://huggingface.co/datasets/uonlp/CulturaX/discussions/15#6597e83f185db94370d6bf50 for reference

Labels: name: documentation

In particular, I would like to have an example of how to replace the following configuration (from https://huggingface.co/docs/hub/datasets-manual-configuration#splits)

```
---
configs:
- config_name: default
  data_files:
  - split: train
    path: "data/*.csv"
  - split: test
    path: "holdout/*.csv"
---
```

with the `data_dir` field.

Hey @severo can I work on this , 

Yes, thanks! I assigned you.

please open a PR instead of commenting here

Hi! I've researched this issue and found the root cause of the confusion.

Problem Analysis:
The issue is that `data_dir` cannot be used in YAML configurations (README.md), but this isn't clearly documented anywhere. Users expect it to work based on the programmatic API.

 Key Findings : 
- `data_dir` works for: `load_dataset("csv", data_dir="folder")`  
- `data_dir` does NOT work in YAML configs - must use `data_files` with `path`
- The distinction isn't explained in current documentation

 Proposed Solution : 
Add clear documentation explaining:
1. `data_dir` is for programmatic loading only
2. YAML configs must use `data_files` with glob patterns like `path: "folder/*.csv"`
3. Migration examples for common mistakes

Question for Maintainers : 
Should this documentation be added to:
 A) `huggingface/hub-docs` (covers YAML configurations)
 B) `huggingface/datasets` (covers the library)  
 C) Both repositories?

I'm ready to create a PR once you confirm the preferred location!

hmmm, @lhoestq says otherwise in https://huggingface.co/datasets/uonlp/CulturaX/discussions/15#6597e83f185db94370d6bf50

> Using data_dir in YAML is supported and is equivalent to passing data_dir= to load_dataset

Could you double check?

(maybe support has been dropped since)
fix: decode Json() values before calling DataFrame.to_json() (#8116)

# Summary
fixes (https://github.com/huggingface/datasets/issues/8116)
Basically the issue is that if a dataset column uses Json() as a feature type, calling to_json() on that dataset causes a raw json escaped string to be exported, without decoding it first. 
### root cause
`to_json` exports feature values in their stored form by default, which means that the raw json escaped string would be exported
### changes made
I tried fixing this minimally. before calling pandas `to_json()`, the writer now decodes the batch with `PandasFeaturesDecoder`. That makes Json() values serialize as native JSON. 
### test plan
- [x] reproduced this error locally
- [x] verified the fix with direct JsonDatasetWriter smoke tests and by confirming it reuses the existing PandasFeaturesDecoder path correctly.


Just committed the fix, please let me know if there are any other issues

Thanks for the quick fix!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8122). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.