noamaanMulla-03/datasets-issues huggingface dataset:
Include version in DatasetInfo YAML so push_to_hub preserves it

DatasetInfo._INCLUDED_INFO_IN_YAML controls which fields land in the README.md dataset_info block written by push_to_hub, and it currently omits version. Setting version on a Dataset (e.g. via load_dataset(..., version=...)) and pushing it to the Hub silently drops the version from the rendered front matter, so reloading the dataset returns version=None unless the user edits the README by hand. Closes #7378.

The reload side already handles a version string in DatasetInfo.__post_init__ via the Version dataclass, so adding "version" to the included keys is enough to round-trip the value through YAML without any extra coercion. The empty case is preserved because _to_yaml_dict only emits keys whose values are set on the instance.

Added test_dataset_info_to_yaml_dict_preserves_version in tests/test_info.py that asserts the string survives _to_yaml_dict and _from_yaml_dict. tests/test_info.py and tests/test_hub.py pass locally.
chore: enable Dependabot weekly GitHub Actions bumps

## Summary

Adds `.github/dependabot.yml` so this repo's pinned GitHub Action SHAs
get bumped automatically once a week.

All action updates are **grouped into one weekly PR** (not one PR per
action) to keep the noise down, and Dependabot waits **7 days after a
release before opening the bump** (cooldown). The 7-day cooldown is
aligned with the org's pinact `min_age: 7` policy — so by the time
the Dependabot PR lands, the SHA is already old enough for the security
gate to accept it. The bot opens the PR; the org-wide security gate
(pinact + denylist + deny-packages + osv-scan) runs on it; a human
merges.

## Why

GitHub Action SHAs that were safe when pinned can drift out of date —
missing security patches, bug fixes, or new features. Dependabot keeps
them current. Combined with the org-wide validation workflow (which
blocks compromised SHAs from landing), the bumps are safe by
construction.

Closes huggingface/tracking-issues#518


Labels: name: dependabot

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8221). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Checking that split name is correct happens only after the data is downloaded

### Describe the bug

Verification of split names (=indexing data by split) happens after downloading the data. So when the split name is incorrect, users learn about that only after the data is fully downloaded, for large datasets it might take a lot of time.

### Steps to reproduce the bug

Load any dataset with random split name, for example:
```python
from datasets import load_dataset

load_dataset("mozilla-foundation/common_voice_11_0", "en", split="blabla")
```
and the download will start smoothly, despite there is no split named "blabla".

### Expected behavior

Raise error when split name is incorrect. 

### Environment info

`datasets==2.9.1.dev0`

Labels: name: bug

#self-assign
Validate split name before starting download

## What this fixes

Closes #5523.

When you pass an invalid split name to `load_dataset()`, the error only surfaces after the entire dataset has already been downloaded. For large datasets that can mean waiting many minutes before seeing a one-liner like:

```python
from datasets import load_dataset
load_dataset("mozilla-foundation/common_voice_11_0", "en", split="blabla")
# ... full download happens first ...
# ValueError: Unknown split "blabla". Should be one of [...]
```

The expected behaviour is to raise immediately.

## How it works

**`splits.py`** — new `_check_split_names(split, known_splits)` helper.  It normalises all the split spec shapes the library accepts (plain name, `"train[:50%]"`, `"train+test"`, list, dict) and raises a `ValueError` with the list of available splits if any base name is not recognised.

**`load.py`** — calls `_check_split_names` right after `load_dataset_builder()` returns, before `download_and_prepare()` is ever reached.  This fires whenever split info is already available – i.e. when the builder was initialised from Hub YAML metadata or from a cached `dataset_info.json`.

**`builder.py`** — calls `_check_split_names` at the top of `as_dataset()`, before `map_nested()`.  This is the safety-net for the cases where the early check wasn't possible (first download with no Hub YAML split info): instead of an opaque error from `arrow_reader`, the user gets a clean `ValueError` with the list of available splits.

## Tests

- **`test_splits.py`**: 14 parametrised cases covering valid names, slices, compound specs, lists, dicts, `None`, `"all"`, and empty `known_splits`.
- **`test_builder.py`**: `test_builder_as_dataset_unknown_split_raises` – verifies the guard in `as_dataset()` for both a plain bad name and a compound spec with one bad part.
DOC: document missing save_infos and max_depth parameters

## What this fixes

Two public API functions were missing parameters from their docstrings:

### 1. `load_dataset()` — missing `save_infos`

The `save_infos` parameter has been in the function signature but was absent from the `Args` section of the docstring. When `save_infos=True`, it overrides `verification_mode` and forces it to `VerificationMode.ALL_CHECKS`, running full checksums/size/splits verification.

### 2. `Dataset.flatten()` — missing `max_depth`

The `max_depth` parameter (default `16`) controls how many nesting levels are flattened, but was not documented in the `Args` section.

## Files changed

- `src/datasets/load.py` — added `save_infos` to `load_dataset` docstring
- `src/datasets/arrow_dataset.py` — added `max_depth` to `Dataset.flatten` docstring
Add `.conll` / `.conllu` dataset format loader (CoNLL-2003 / 2000 / U)

Closes #7757.

## Summary

Adds a `packaged_modules/conll/` builder so `.conll` and `.conllu` files load directly via `load_dataset(...)` without manual parsing scripts. Each row of the loaded dataset corresponds to one sentence, with each configured column produced as a list aligned with the token list.

Builder shape mirrors the `text` packaged module (line-based reading) — as suggested by @namesarnav in the issue thread.

```python
from datasets import load_dataset

ds = load_dataset(
    "conll",
    data_files="train.conll",
    column_names=["tokens", "pos_tags", "chunk_tags", "ner_tags"],
)
# Each example: {"tokens": [...], "pos_tags": [...], "chunk_tags": [...], "ner_tags": [...]}
Add Apache Iceberg format support

 # Add Apache Iceberg format support 

## Motivation

  Apache Iceberg is the most widely adopted open table format for data lakes, supported by Databricks,
  Snowflake, AWS Glue, Dremio, and others. A large amount of ML training data lives in Iceberg tables.
  Currently, users must manually export Iceberg data to Parquet before loading it into HuggingFace Datasets —
  this PR removes that friction.

fix this (https://github.com/huggingface/datasets/issues/7863)

##  Usage

  Users pass a pre-configured pyiceberg Catalog object and a table identifier:

  from pyiceberg.catalog.sql import SqlCatalog
  from datasets import load_dataset

  catalog = SqlCatalog("my_catalog", uri="sqlite:///catalog.db", warehouse="/tmp/warehouse")

## Basic loading
  ds = load_dataset("iceberg", catalog=catalog, table="db.my_table")

## Column selection + row filtering (predicate pushdown)
  ds = load_dataset("iceberg", catalog=catalog, table="db.my_table",
                    columns=["text", "label"],
                    filters=[("label", ">", 0)])

## Multiple splits from different tables
  ds = load_dataset("iceberg", catalog=catalog,
                    table={"train": "db.train", "test": "db.test"})

## Time travel via snapshot_id
  ds = load_dataset("iceberg", catalog=catalog, table="db.my_table",
                    snapshot_id=7051729674881785648)

## Streaming
  ds = load_dataset("iceberg", catalog=catalog, table="db.my_table", streaming=True)

  Works with any pyiceberg-supported catalog backend (REST, Hive, Glue, SQL, etc.) — the builder is agnostic
  to how the catalog is configured.

 ## Design decisions

  - Catalog object passed in, not constructed internally. Iceberg catalog configuration varies widely across
  backends (REST, Hive, Glue, SQL each have different auth/connection params). Rather than re-implementing a
  "catalog factory" inside the builder, users bring their own catalog — similar to how the sql builder accepts
   an existing SQLAlchemy connection. This keeps the builder simple and forward-compatible with new catalog
  types.
  - No _EXTENSION_TO_MODULE registration. Unlike file-based formats (Parquet, Lance, CSV), Iceberg tables are
  addressed via catalog + table identifier, not file extensions. Users must specify "iceberg" explicitly as
  the path argument.
  - create_config_id override for fingerprinting. Catalog objects (containing SQLAlchemy engines, connection
  pools, etc.) are not picklable by dill. The override replaces the catalog with a stable string
  representation ("{ClassName}_{name}") before hashing.
  - _CountableBuilderMixin for fast row counting. Uses scan.plan_files() metadata to count rows without
  reading data files.



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8148). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add columns support to JSON loader

New fix to #7594
This PR adds support for the columns argument in the JSON dataset builder.
Added columns parameter to JsonConfig.
Applied column filtering after table creation, filling missing columns with None.

Extended tests to cover:
- Selecting a subset of columns
- Handling missing requested columns
- Column selection on list-of-strings case


Kinda strange this one is never merged and never even got any attention.
Maybe you can make another pull request again? The code base didnt change much.
This is really useful.
[BUG] Abnormal progress bar in dataset.map when load_from_cache_file=False

### Describe the bug

datasets                                 4.8.5

```python
from datasets import load_dataset, load_from_disk

data = load_from_disk('pretrain_cached_dataset/train')

def dummy_map(example):
    return example

dataset = data.map(dummy_map)
dataset = data.map(dummy_map, load_from_cache_file=True)   # works fine
dataset = data.map(dummy_map, load_from_cache_file=True, num_proc=4)   # works fine
dataset = data.map(dummy_map, load_from_cache_file=False)  # does not work

```

<img width="935" height="124" alt="Image" src="https://github.com/user-attachments/assets/2325a7f3-80e1-46bd-a8fb-eb503b7e5df3" />


<img width="954" height="58" alt="Image" src="https://github.com/user-attachments/assets/ff5f49ad-bf96-4a20-891e-bfa8018115ea" />

### Steps to reproduce the bug

-

### Expected behavior

-

### Environment info

-


https://github.com/huggingface/datasets/blob/4e3bf52ff436d841e61211dfa096b5c47197ae36/src/datasets/arrow_dataset.py#L3593-L3598

Hi, can you provide more details on the issue ?

<img width="340" height="26" alt="Image" src="https://github.com/user-attachments/assets/0dca6f47-4a8d-4871-a908-f58df856ed72" />

This exceeds the maximum value.

#self-assign

Hey, I've opened a PR for this: #8170 

thanks
fix(map): fix progress bar exceeding total when load_from_cache_file=False

## Summary

When `load_from_cache_file=False`, the progress bar in `Dataset.map()` displayed a count that exceeded the dataset size (e.g. "800 examples" for a 400-row dataset). 

The bug: `pbar_initial` was computed using `len(existing_cache_files)`, which counts cache files on disk regardless of whether they would be used. With `load_from_cache_file=False`, those files exist but are ignored, yet the bar was initialized to the full dataset size as if all work was already done. The actual processing then added on top of that, pushing the counter past the total and causing tqdm to drop the percentage display entirely.

Fix: use `num_shards - len(unprocessed_kwargs_per_job)` instead, which is 0 when `load_from_cache_file=False`, so the bar starts and counts correctly.

## Issue

Fixes #8167

## Local verification

`py -3.14 -m pytest tests/test_arrow_dataset.py -k "test_map_load_from_cache_file_false_progress_bar_starts_at_zero" -v`

2 passed


## Risk

Low risk, a one-line change to how `pbar_initial` is computed. No behaviour changes outside of the progress bar display.





The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8170). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

hello! any update?

> hello! any update?

Fixed a formatting issue and pushed an update. The workflows are awaiting approval...


Hey @lhoestq, thanks for the approval!

I looked into the 2 failing CI checks, and they seem unrelated to this PR..

- The `deps-latest` failures are caused by a breaking change in `huggingface_hub`, which no longer accepts single-segment repo IDs like `"dummy"` and `"paws"`. This affects existing tests rather than the change introduced here.
* The Windows runner also shows additional `401 Unauthorized` errors, which appear to be CI/environment-related.

The `deps-minimum` variants pass successfully, and my test (`test_map_load_from_cache_file_false_progress_bar_starts_at_zero`) passes in all runs.

Would you be able to merge this PR? I’m happy to make any additional changes if needed.


sure, the errors sound unrelated to this pr indeed :)
Security/Logic Fix: Autonomous Code Review

## Autonomous Bug Report & Patch

This vulnerability and fix were autonomously discovered by the Lucy Red Team swarm.

Upon reviewing the provided code snippet from `src/datasets/io/generator.py`, I didn't find any obvious critical bugs. However, there are a few areas that could be improved or clarified:

1. **Documentation and Type Hints**: The use of type hints is good, but adding more detailed docstrings to methods and parameters would improve readability and maintainability.

2. **Error Handling**: There is no error handling in the `read` method. If the generator function raises an exception, it will propagate up without any context or logging. Adding try-except blocks could help manage such errors gracefully.

3. **Initialization Parameters**: The `split` parameter is set to `Split.TRAIN` by default, which might not be appropriate for all use cases. It would be better to make this parameter mandatory or provide a more generic default value if it's intended to be optional.

4. **Fingerprint Handling**: The fingerprint logic is somewhat convoluted. If the `fingerprint` is provided, it sets the config_id of the builder with a hardcoded prefix. This could be simplified and made more flexible.

5. **Unused Parameters**: The `kwargs` parameter is used in the constructor but not further utilized. If these parameters are intended to be passed down to other methods or classes, they should be explicitly handled.

Here's an improved version of the code with some of these suggestions implemented:

```python
from typing import Callable, Optional

from .. import Features, NamedSplit, Split
fix: embed_external_files=True for mesh support

## Fix for #8055 

This is follow-up PR for fixing a specific issue related to `dataset.push_to_hub(repo_id, embed_external_files=True)`

### Test Conducted

```python
from datasets import Features, Image, Mesh, Value, load_dataset


repo_id = "VINAY-UMRETHE/My-Mesh-Dataset"

dataset = load_dataset("json", data_files="TEST_DATA/records.jsonl", split="train")
dataset = dataset.map(
    lambda example: {
        "image": "TEST_DATA/" + example["image_path"],
        "mesh": "TEST_DATA/" + example["mesh_path"],
    }
)
dataset = dataset.cast(
    Features(
        {
            "id": Value("string"),
            "text": Value("string"),
            "image_path": Value("string"),
            "mesh_path": Value("string"),
            "image": Image(),
            "mesh": Mesh(),
        }
    )
)

dataset.push_to_hub(repo_id, embed_external_files=True)
```

Demo: [VINAY-UMRETHE/My-Mesh-Dataset](https://huggingface.co/datasets/VINAY-UMRETHE/My-Mesh-Dataset)


@lhoestq docs are ready.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8224). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support 3D Datasets

### Feature request

In `datasets` library we have options to load `image` and `audio` datasets which can be viewed in the Dataset Viewer.

LIKE:
```bash
from datasets import load_dataset, Image
from datasets import load_dataset, Audio
```
> https://huggingface.co/docs/datasets/en/image_load & https://huggingface.co/docs/datasets/en/audio_process

---

I guess there's a need for supporting 3D datasets as well for *storing* & *loading* 3D object files like (`.glb, .ply, .obj` etc...) maybe with something like:

```bash
from datasets import load_dataset, Mesh # or `_3D` or whatever is naming convention
```

### Motivation

Multi-Modal datasets are rapidly increasing with modalites like 3D, Video etc, `datasets` already support experimental `Video` import. 

I have a dataset where I've stored everything as dict so it atleast look balanced across all those 4 columns and 4 modalites It have (Text, Image, Audio, 3D Mesh) but it don't look that good since RAW binary bytes are visible. And Needs to be decoded for that specific field containing the binary data. 

### Contribution

PUSHED THE PR: #8055 


Labels: name: enhancement

Closing, as this got implemented in:

[feat: add 3D mesh support and MeshFolder builder #8055](https://github.com/huggingface/datasets/pull/8055)

[fix: embed_external_files=True for mesh support #8224](https://github.com/huggingface/datasets/pull/8224)
Parse agent traces messages for SFT using `teich`

small breaking change (mostly for the dataset viewer on HF that has to take this into account): the field "traces" is renamed "trace" cc @cfahlgren1 

This PR adds the `teich` extracted traces data, i.e. the fields messages, prompt, tools and metadata

```python
In [1]: from datasets import load_dataset

In [2]: ds = load_dataset("lhoestq/agent-traces-example", split="train")

In [3]: ds[0]["messages"]
[{'role': 'user', 'content': 'Download a random dataset from Hugging Face, use DuckDB to inspect it, and come back with a short report about it. Be concise and include: dataset name, what files/format you found, row count or rough size if you can determine it,...'
 ...]
```


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8232). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
map reuses unchanged columns from the previous dataset to avoid disk usage

https://github.com/huggingface/datasets/issues/6013

## Summary

This PR improves Dataset.map behavior by ensuring that unchanged columns are preserved after transformation.

Previously, in some cases, columns not explicitly modified by the mapping function could be lost or reconstructed during shard concatenation.

## Changes

- Added logic to preserve unchanged columns after dataset concatenation step
- Ensures consistency between input and output schema when only partial transformation is applied

## Motivation

Improves usability of map() for partial feature transformations without requiring users to manually reattach columns.

## Tests

Added unit test verifying that untouched columns remain in the output dataset.
Fix iterable skip over full Arrow blocks

## What does this PR do?

Fixes `IterableDataset.skip(n)` for streaming datasets when the underlying iterable uses Arrow batches and `n` skips one or more complete Arrow blocks.

Previously, after a full Arrow block was counted as skipped, `_iter_arrow()` continued into the partial-slice branch and yielded rows from a block that should have been fully skipped.

## What was the issue?

`SkipExamplesIterable._iter_arrow()` handles skipping in two cases:

1. the current Arrow table is fully skipped
2. only the beginning of the current Arrow table is skipped, and the rest is yielded

The bug was that case 1 did not stop after marking the table as skipped. So the same table then fell through into case 2.

In other words, a table could first be counted as "already skipped", but then still be sliced and yielded.

For example, if Arrow tables have 4 rows each and we call `skip(6)`:

- table 1 has rows `[0, 1, 2, 3]` and should be fully skipped
- table 2 has rows `[4, 5, 6, 7]`, so only `[4, 5]` should be skipped and `[6, 7]` should be yielded

Before this PR, after table 1 was counted as skipped, the code kept processing table 1 and yielded part of it. This is why skipped rows could appear in the output.

This PR adds `continue` after a table is fully skipped, so the code moves directly to the next Arrow table.

## Tests

- `PYTHONPATH=src pytest tests/test_iterable_dataset.py::test_skip_arrow_examples_iterable -q`

The regression test covers skipping within a block, exactly one block, across blocks, and beyond the dataset length.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8236). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support composed splits in streaming datasets

Fixes #2699
Fixes #4804

This PR adds support for unsliced split composition when loading datasets in streaming mode, e.g. `split="train+validation"`.

Previously, `DatasetBuilder.as_streaming_dataset()` only accepted a single split name or returned all splits as an `IterableDatasetDict`, so composed split strings raised `ValueError: Bad split`.

The change resolves composed split instructions by building each requested streaming split and concatenating the resulting `IterableDataset`s. It also supports the `all` split sentinel in streaming mode.

This intentionally does not add support for sliced streaming expressions such as `train[:10%]`, which require separate handling.

Tests added for:
- string split composition: `"train+test"`
- object split composition: `Split.TRAIN + Split.TEST`
- `"all"`
- `Split.ALL`

Validation:
- `python -m pytest tests/test_builder.py -q`


@lhoestq could you please review this PR when you get a chance?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8220). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.