noamaanMulla-03/datasets-issues huggingface dataset:
Fix: preserve features after IterableDataset.add_column (GH#5752)

## Problem

After calling `.add_column()` on a streaming `IterableDataset`, accessing `.features` returns `None`:

```python
from datasets import load_dataset

ds = load_dataset("...", streaming=True)
print(ds.features)  # {'text': Value('string'), ...}

ds2 = ds.add_column("new_column", ["val"] * len(ds))
print(ds2.features)  # None  <-- bug
print(ds2.features.keys())  # AttributeError: 'NoneType' object has no attribute 'keys'
```

## Root Cause

`add_column` calls `self.map(...)` without passing the `features` argument. Inside `_map`, the new dataset's info is constructed as:

```python
info = self.info.copy()
info.features = features  # None if not passed
```

This unconditionally overwrites the copied features with `None`.

## Fix

Infer the feature type of the new column from the provided data, merge it with the existing features, and pass the combined `Features` object to `map()`:

```python
column_features = _infer_features_from_batch({name: list(column) if not isinstance(column, list) else column})
if self.features is not None:
    features = self.features.copy()
    features.update(column_features)
else:
    features = column_features
return self.map(..., features=features)
```

## Tests

- Added `test_iterable_dataset_add_column_preserves_features` which explicitly checks that `.features` is not `None` and contains both original and new columns after `add_column`.
- Existing `test_iterable_dataset_add_column` continues to pass.

Closes #5752
Replace distutils.util.strtobool in tests with a local helper

## What this fixes

`tests/utils.py` imports `strtobool` from `distutils.util`:

```python
from distutils.util import strtobool
```

`distutils` was removed from the standard library in **Python 3.12** ([PEP 632](https://peps.python.org/pep-0632/)). `tests/utils.py` is imported (via `from .utils import ...`) by **20 test modules**, so on an interpreter without the `distutils` shim this raises at collection time and the **entire test suite fails to run**:

```
tests/utils.py:10: in <module>
    from distutils.util import strtobool
E   ModuleNotFoundError: No module named 'distutils'
```

### When it triggers

- **setuptools absent** on Python ≥ 3.12 — increasingly common with modern venvs, `uv`, and minimal CI images: hard `ModuleNotFoundError`, whole suite uncollectable.
- **setuptools present**: it vendors a `distutils` shim so the import succeeds today, but the shim is deprecated and is **removed on Python 3.14** — which this project already targets (`setup.py` advertises `Programming Language :: Python :: 3.12/3.13/3.14`, and CI runs on 3.14).

Either way, relying on `distutils` here is on borrowed time.

## The fix

Replace the import with a small local `strtobool` that reproduces the **exact** semantics of `distutils.util.strtobool`:

```python
def strtobool(value: str) -> int:
    value = value.lower()
    if value in {"y", "yes", "t", "true", "on", "1"}:
        return 1
    if value in {"n", "no", "f", "false", "off", "0"}:
        return 0
    raise ValueError(f"invalid truth value {value!r}")
```

This is the standard PEP 632 migration for `strtobool` (the stdlib provides no replacement). It's test-only — no change to shipped code.

## Verification (local, Python 3.12.12, no setuptools)

**Before** — collection fails:
```
tests/utils.py:10: ModuleNotFoundError: No module named 'distutils'
```

**After**:
- Semantics parity checked against the `distutils` spec for every truthy/falsy value (incl. mixed case) and the `ValueError` path — all match.
- Test collection restored, and real tests run green:

```
python -m pytest tests/test_formatting.py tests/features/test_features.py
220 passed, 33 skipped
```
- `ruff check tests/utils.py` → All checks passed!
- `ruff format --check tests/utils.py` → already formatted.

## Notes

- The `strtobool` helper lives in `tests/utils.py` right next to its only caller (`parse_flag_from_env`), matching how the codebase already reads env flags like `RUN_SLOW` / `RUN_REMOTE`.
- No production code touched; behavior for the four env flags (`RUN_SLOW`, `RUN_REMOTE`, `RUN_LOCAL`, `RUN_PACKAGED`) is unchanged.
Catch RecursionError when decoding deeply nested mesh files (#8246)

Fixes #8246.

A GLB whose glTF JSON chunk is deeply nested (malformed or malicious) makes `trimesh.load` exceed Python's recursion limit while parsing it (`RecursionError: maximum recursion depth exceeded while decoding a JSON array`), crashing `Mesh.decode_example` with an uncaught error.

This routes both `trimesh.load` call sites through a small `_load_mesh` helper that catches `RecursionError` and re-raises it as a clear `ValueError`, so a malformed/adversarial mesh produces an actionable error instead of crashing the whole load. `htmlTreeAsString`-style decoding for valid meshes is unchanged.

Added a regression test that builds a GLB with a deeply-nested JSON chunk and asserts `decode_example` raises `ValueError` (it fails with the uncaught `RecursionError` without the fix).

Verified locally: `ruff check` / `ruff format` clean, and the new test passes.



Gentle bump on this one — it's a small, self-contained fix for the `RecursionError` on deeply nested mesh files (#8246), with a regression test, and CI is green. Happy to rebase or adjust if anything's needed. Thanks for taking a look!
Fix MetadataConfigs dropping parquet shards for non-consecutive configs

Fixes #8269

## Problem

`MetadataConfigs._from_exported_parquet_files_and_dataset_infos()` groups the exported parquet files with `itertools.groupby(exported_parquet_files, itemgetter("config"))`. `groupby` only groups **consecutive** equal keys, but the result is built as a dict keyed by config name:

```python
metadata_configs = {
    config_name: { ... }
    for config_name, parquet_files_for_config in groupby(exported_parquet_files, itemgetter("config"))
}
```

So if the same config appears again after a different config (e.g. `default`, `other`, `default`), `groupby` yields `default` as two separate groups and the second one **overwrites** the first in the dict — the earlier shard URLs are silently lost. The same applies to the inner `groupby` over splits.

## Fix

Sort the exported files by `(config, split)` before grouping, so all rows for a given config/split are consecutive. The sort is stable, so shard order within a split (`0000.parquet`, `0001.parquet`, …) is preserved.

## Tests

Added `test_from_exported_parquet_files_keeps_all_shards_when_configs_non_consecutive` — it feeds a non-consecutive `default` config and asserts both shard URLs survive (the test fails on `main` and passes with this change). The existing `test_split_order_...` test still passes (final ordering is still driven by `dataset_infos`). `pytest tests/test_metadata_util.py` → 9 passed; `ruff check` clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Dataset spotlight: Helium benchmarks

Helium Model Worldview: 304 cue-swap prompts. https://huggingface.co/datasets/HeliumTrades/helium-model-worldview-benchmark | https://heliumtrades.com/benchmarks/


Closing — wrong venue (Library repo; use HF dataset page.). We're focusing on eval awesome-list PRs and lm-eval/promptfoo integrations. — Helium Trades
Support concatenating multiple streaming datasets while preserving the sum of shards

### Feature request

It would be useful if concatenating multiple `IterableDataset`s could preserve the shard information from each input dataset, so that:

- the total number of shards equals the sum of the input shards;
- `split_dataset_by_node()` and worker sharding continue to work efficiently;
- downstream distributed dataloaders can fully exploit the available parallelism.

### Motivation

Currently, it's straightforward to concatenate multiple streaming datasets using `interleave_datasets` or other approaches. However, after concatenation, the resulting dataset does not preserve the original sharding information from each input dataset.

For distributed training, especially with large-scale datasets, this leads to a reduced number of effective shards compared to the sum of the input datasets' shards. Consequently, data loading parallelism may be limited, and the available worker concurrency cannot fully utilize the original dataset partitioning.

For example:

- IterableDataset A: 1 shard
- IterableDataset B: 1 shard

After concatenation, we would expect the resulting streaming dataset to expose **2 shards**, allowing distributed workers to shard the combined dataset naturally.

### Your contribution

I'd be happy to work on this feature and submit a PR if the maintainers think this would be a useful addition.

Any feedback on the proposed design would be greatly appreciated.

Labels: name: enhancement

it's already the case afaik:

```python
In [1]: from datasets import IterableDataset, concatenate_datasets

In [2]: ds1 = IterableDataset.from_dict({"i": [0, 1]})

In [3]: ds2 = IterableDataset.from_dict({"i": [2, 3]})

In [4]: ds = concatenate_datasets([ds1, ds2])

In [5]: list(ds)
Out[5]: [{'i': 0}, {'i': 1}, {'i': 2}, {'i': 3}]

In [6]: ds.num_shards  # = sum([ds1.num_shards, ds2.num_shards])
Out[6]: 2
```

On the other hand `interleave_datasets()` doesn't result in the sum of shards. Instead each new shard is made of at least one shard of every input dataset so that each shard effectively contains interleaved data.


```python
In [7]: from datasets import interleave_datasets

In [8]: ds = interleave_datasets([ds1, ds2], stopping_strategy="all_exhausted")

In [9]: list(ds)
Out[9]: [{'i': 0}, {'i': 2}, {'i': 1}, {'i': 3}]

In [10]: ds.num_shards  # = min([ds1.num_shards, ds2.num_shards])
Out[10]: 1
```
Fix lance auth

fix two bugs
- `TypeError: argument 'storage_options': 'None' is not an instance of 'str'` because "token": None is not supported in storage_options
- download_config.storage_options was empty and therefore the token for private datasets was not passed to lance


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8301). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Turkish DIK-style Syntactic Structure Benchmark Dataset

### Feature request

A structured dataset proposal introducing a Turkish syntactic structure benchmark designed for evaluating LLMs on fine-grained grammatical reasoning in a morphologically rich language.

Current NLP datasets in Hugging Face focus heavily on surface-level semantic tasks or English-centric benchmarks. There is a clear gap in evaluating deep syntactic understanding, especially for languages like Turkish where:

Embedded clause structures are highly productive
Nominalization (-DIK, -MA, -ACAK) plays a central syntactic role
Pro-drop subjects require contextual recovery
Cross-clause dependency relations are essential for meaning
Interrogative constructions involve structural reordering and embedding

This proposal introduces a DIK-style syntactic annotation benchmark dataset, where each sentence is represented in a structured JSON format capturing:

main clause decomposition
embedded clause hierarchy
verb morphology (lemma, tense, mood, polarity)
syntactic roles (subject, object, complement)
clause function mapping
case marking and nominalization tracking

A pilot dataset of 20 fully annotated sentences has already been completed with a consistent schema and manual validation. The design is fully scalable and intended for expansion to 100–300 sentences in Phase 1.

### Motivation

I am always frustrated by the lack of evaluation benchmarks that go beyond surface-level semantic correctness and actually test whether LLMs understand deep syntactic structure, especially for underrepresented morphologically rich languages like Turkish.

Existing benchmarks do not adequately evaluate:

embedded clause interpretation
nominalization-driven structure (-DIK, -MA)
pro-drop subject recovery
syntactic role consistency across clauses
dependency tracking in complex sentences
cultural and discourse-level nuances embedded in language use (e.g., pragmatic meaning, register shifts, and culturally grounded expressions in Turkish)

This limits the ability to properly measure LLM performance in languages where meaning is encoded not only in morphology and clause structure, but also in cultural context, discourse conventions, and pragmatic interpretation rather than word order alone.

This dataset aims to fill that gap by providing a structured, machine-evaluable syntactic benchmark suitable for integration into evaluation frameworks such as Hugging Face evaluate and other LLM benchmarking pipelines.****

### Your contribution

I am fully prepared to develop this dataset further and contribute a complete pull request, including:

expanding the dataset to 100–300 high-quality annotated sentences
ensuring strict schema consistency across all samples
providing full JSON dataset files compatible with Hugging Face Datasets
creating a reproducible evaluation script for LLM structural scoring
documenting dataset structure and annotation guidelines

If needed, I can also help adapt the dataset into a format compatible with existing evaluation frameworks and contribute ongoing improvements after initial review.

Labels: name: enhancement
Share fsspec filesystem across threads when resolving data files

## What does this PR do?

Fixes #8149.

`_get_origin_metadata` resolves each data file in a `ThreadPoolExecutor` and every worker builds its own filesystem via `url_to_fs` since fsspec caches filesystem instances per thread. For cloud backends like `s3://` or `gcs://` this re-resolves credentials and opens a new connection pool in each thread.

This builds the filesystem once per protocol in the main thread and passes it to the workers which just strip the protocol off the path instead of rebuilding it. `hf://` paths and chained `::` urls keep the same behavior as before
Fix `add_column` parameter order so `feature` precedes `new_fingerprint`

### What

`Dataset.add_column`'s signature is `(name, column, new_fingerprint=None, feature=None)`, but:

- its own docstring documents the third argument as `feature` (not `new_fingerprint`), and
- every sibling `@fingerprint_transform` method keeps `new_fingerprint` as the **last** parameter (e.g. `cast_column(column, feature, new_fingerprint=None)`, `rename_column`, `remove_columns`, `select_columns`).

Because `new_fingerprint` sits before `feature`, a docstring-conformant positional call raises:

```python
>>> from datasets import Dataset, Value
>>> ds = Dataset.from_dict({"a": [1, 2]})
>>> ds.add_column("b", [3, 4], Value("int32"))
TypeError: Dataset.add_column() got multiple values for argument 'new_fingerprint'
```

The `fingerprint_transform` decorator injects `new_fingerprint` as a keyword argument, so the positional value collides with it.

### Fix

Reorder the signature to `(name, column, feature=None, new_fingerprint=None)`, matching the docstring and the `cast_column` convention. The decorator references `new_fingerprint` by keyword, so this is safe for existing keyword callers, and no `Dataset.add_column` caller in the repo passes `feature`/`new_fingerprint` positionally.

### Test

Added `test_add_column_feature_argument` in `tests/test_arrow_dataset.py`, asserting that a positional `feature` is honored, the keyword form still works, and the `new_fingerprint` keyword is still respected. The test fails on `main` (`TypeError`) and passes with this change.

Fixes #7864

---
🤖 Opened with [Superhuman](https://github.com/gaurav0107/superhuman), an open-source contribution agent.
Improve missing tsfile dependency error

Related to #8256

This PR improves the error message when the optional `tsfile` dependency is unavailable.

Currently, loading a TSFile dataset without `tsfile` installed raises a raw:

ModuleNotFoundError: No module named 'tsfile'

This change raises a clearer ImportError instead:

ImportError: To support loading TSFile datasets, please install 'tsfile'.

A regression test is included for the missing dependency case.
Dataset Viewer fails on TSFile datasets

### Describe the bug

## Description

The Dataset Viewer fails when trying to display a dataset stored in TSFile format.

The error shown by the viewer is:

```python
ModuleNotFoundError: No module named 'tsfile'

### Steps to reproduce the bug

# Dataset Viewer fails to load TSFile dataset due to missing `tsfile` dependency

## Dataset

Dataset URL:

https://huggingface.co/datasets/smilegeng/pusht

## Description

The dataset is uploaded successfully, but the Dataset Viewer cannot display it.

The viewer reports:

```text
SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

After inspecting the traceback, the actual root cause appears to be a missing dependency:

```text
ModuleNotFoundError: No module named 'tsfile'
```

The error originates from the TSFile dataset loader:

```text
File "/usr/local/lib/python3.12/site-packages/datasets/packaged_modules/tsfile/tsfile.py", line 304, in _scan_metadata
    from tsfile.constants import TIME_COLUMN, ColumnCategory

ModuleNotFoundError: No module named 'tsfile'
```

This exception is then wrapped into:

```text
datasets.inspect.SplitsNotFoundError:
The split names could not be parsed from the dataset config.
```

## Expected Behavior

The Dataset Viewer should either:

1. Support TSFile datasets by installing the required `tsfile` dependency, or
2. Return a clearer error message indicating that the dependency is unavailable.

## Full Traceback

```text
Exception: SplitsNotFoundError
Message: The split names could not be parsed from the dataset config.

Traceback:
...
ModuleNotFoundError: No module named 'tsfile'
...
datasets.inspect.SplitsNotFoundError:
The split names could not be parsed from the dataset config.
```


### Expected behavior

The Dataset Viewer should support TSFile datasets.

### Environment info

D:\huggingface_to_tsfile>datasets-cli env

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 5.0.0
- Platform: Windows-11-10.0.26200-SP0
- Python version: 3.14.0
- `huggingface_hub` version: 1.7.2
- PyArrow version: 23.0.1
- Pandas version: 3.0.1
- `fsspec` version: 2026.2.0


D:\huggingface_to_tsfile>


Thanks for reporting ! I'm working on it :)

The viewer doesn't use `datasets` V5 yet, I have to do some minor adjustments first

I opened a PR that addresses the missing optional tsfile dependency error by replacing the raw ModuleNotFoundError with a clearer ImportError and adding a regression test.

This is a small improvement related to the missing-dependency path

PR: #8307
Fix symlink-following arbitrary file write in archive extraction

Extractor.extract() called shutil.rmtree(output_path, ignore_errors=True) before extracting. Since Python 3.8, shutil.rmtree() raises OSError on a symlink pointing to a directory, which ignore_errors=True silently swallows. An attacker with write access to a shared cache directory could pre-plant a symlink at the predictable extraction output path; rmtree would no-op on it, and extraction would then follow the symlink and write archive contents into the attacker-chosen target, yielding arbitrary file write as the victim.

The existing safemembers() zip-slip mitigation only validates member names and is blind to the output directory itself being a symlink.

Remove the symlink itself before extracting so contents can no longer be written through it.
#8296 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8303). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Streaming IterableDataset state stops advancing after first resume — second resume restarts from the beginning


## Description

For a streaming `IterableDataset`, calling `load_state_dict()` and then continuing to iterate leaves `state_dict()` frozen at the initial position (`shard_idx=0`, `shard_example_idx=0`). Reading resumes correctly the first time, but the tracked state never advances again.

This causes a second consecutive resume to restart the entire data stream from scratch, silently corrupting long training runs that checkpoint and resume more than once.

## Root cause

The resume call sites in `IterableDataset` (`_iter_pytorch` for `num_workers >= 1` and `_prepare_ex_iterable_for_iteration` for `num_workers == 0`) set `self._state_dict["examples_iterable"]` to reference the example-iterable's own state dict, then load the resume state into that example-iterable.

`_BaseExamplesIterable.load_state_dict` calls `self._init_state_dict()`, which rebinds `self._state_dict` to a brand-new object. After the load, `dataset._state_dict["examples_iterable"]` still points at the stale, zero-initialized state dict. During iteration the example-iterable mutates its *new* state dict, while `dataset.state_dict()` keeps deep-copying the *old* one — so the reported position never moves past zero.

On a fresh run (no resume), the `load_state_dict` branch is skipped, the reference stays linked, and tracking works correctly — which is why this only surfaces after a resume.

## Impact

We hit this training on FineWeb with torchtitan: a resumed-twice run produced a different training loss than a resume-once run, because the second resume silently replayed the dataset from the start.

## Fix

Already proposed in #8295 re-points `self._state_dict["examples_iterable"]` at the live example-iterable state immediately after the resume load, at both resume sites. Includes a regression test (`test_resume_dataloader_twice`) that fails on `main` and passes with the fix.


### Steps to reproduce the bug

```python
import datasets
def gen():
    for i in range(10): yield {"id": i}
ds = datasets.IterableDataset.from_generator(gen)
full = [x["id"] for x in ds]
it = iter(ds); [next(it) for _ in range(3)]; sd1 = ds.state_dict()
ds.load_state_dict(sd1); it = iter(ds); seen = [next(it)["id"] for _ in range(3)]; sd2 = ds.state_dict()
ds.load_state_dict(sd2); after = [x["id"] for x in ds]
print("reference       :", full)
print("after 1st resume:", seen, "(reads correctly)")
print("after 2nd resume:", after)
print(">>> BUG: 2nd resume restarted from the beginning" if after == full else ">>> OK: continued")
```

### Expected behavior

The dataloader should save the state dict properly so that the next run can be resumed accordingly


### Environment info

datasets==5.0.0
python==3.14.0b2
Make doc builds faster

Companion to the doc-builder tracker: huggingface/doc-builder#802

🤖 Generated with [Claude Code](https://claude.com/claude-code)


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8309). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.