noamaanMulla-03/datasets-issues huggingface dataset:
cannot combine splits merging and streaming?

this does not work:
`dataset = datasets.load_dataset('mc4','iw',split='train+validation',streaming=True)`
with error:
`ValueError: Bad split: train+validation. Available splits: ['train', 'validation']`

these work:
`dataset = datasets.load_dataset('mc4','iw',split='train+validation')`
`dataset = datasets.load_dataset('mc4','iw',split='train',streaming=True)`
`dataset = datasets.load_dataset('mc4','iw',split='validation',streaming=True)`

i could not find a reference to this in the documentation and the error message is confusing. also would be nice to allow streaming for the merged splits

Labels: name: bug

Hi ! That's missing indeed. We'll try to implement this for the next version :)

I guess we just need to implement #2564 first, and then we should be able to add support for splits combinations

is there an update on this? ran into the same issue on 2.17.1.

On a similar note, the keyword `split="all"` also does not work as intended when `streaming=True`. 

No update so far, especially since we haven't implemented an efficient way to query `split=train[50%:]` for example. The addition of two splits should be easy though, since we have `concatenate_datasets()`

Can you concatenate_datasets that are being streamed now? I was led to believe concatenation works on non streaming datasets only.

Yes `concatenate_datasets` works for datasets loaded in streaming mode as well

#self-assign
Warn instead of raise when user-provided data_files yields a subset

Fixes #7867.

NonMatchingSplitsSizesError currently fires whenever the loaded split size differs from the expected size, including when the user explicitly passed data_files for a known subset of the dataset. The only user-side workaround is verification_mode='no_checks', which silences ALL checks rather than just the split-size one.

This PR downgrades the split-size mismatch to a UserWarning when data_files was explicitly provided by the caller. Other mismatch paths (corrupted download, wrong size for full download) remain hard errors.


I think we can simply not raise or warn anything at all when the user provides the data_files. The number of examples is expected to be likely different

Makes sense, done in 2162efc. verify_splits now returns early when the user passed data_files, so there is no raise and no warning at all in that case. The early return also covers the split name checks, since custom data_files can define a different set of splits. I removed the warning path and updated the tests accordingly.
RecursionError when loading GLB file with deeply nested JSON in Mesh feature

## Description

Loading a GLB file with deeply nested JSON chunk causes a RecursionError, crashing the application.

## Step to Reproduce

```python
from datasets import Dataset, Mesh
import tempfile, os, struct

def make_malicious_glb(depth=5000):
    json_chunk = (b'{"a":' * depth + b'1' + b'}' * depth)
    magic = b'glTF'
    version = struct.pack('<I', 2)
    json_len = struct.pack('<I', len(json_chunk))
    json_type = struct.pack('<I', 0x4E4F534A)
    total_len = 12 + 8 + len(json_chunk)
    length = struct.pack('<I', total_len)
    return magic + version + length + json_len + json_type + json_chunk

glb_bytes = make_malicious_glb(depth=5000)

with tempfile.TemporaryDirectory() as tmpdir:
    fpath = os.path.join(tmpdir, 'test.glb')
    with open(fpath, 'wb') as f:
        f.write(glb_bytes)
    
    ds = Dataset.from_dict({'mesh': [fpath]}).cast_column('mesh', Mesh())
    result = ds[0]['mesh']
```

**Expected:** Proper error message
**Actual:** RecursionError crash

## Suggested Fix

Catch RecursionError in Mesh.decode_example() and raise ValueError instead.
VORTEXRAG benchmarked on 6 HuggingFace datasets — new open-source RAG framework

Hi HuggingFace Datasets team!

**VORTEXRAG** was evaluated on 6 datasets available on the HuggingFace Hub using datasets.load_dataset():

- NQ, TriviaQA, WebQ, PopQA, HotpotQA, 2WikiMultiHopQA

Results: EM 74.8, F1 82.6 averaged across all 6. Full evaluation code is in the repo.

- GitHub: https://github.com/vignesh2027/VORTEXRAG
- Paper: https://doi.org/10.5281/zenodo.20579702
- License: MIT
Fix partial extraction when concurrent processes share a cache (#4661)

Fixes #4661.

`ExtractManager.extract` runs the `_do_extract` "already extracted?" check outside any lock; the lock sits one level down in `Extractor.extract`. When two jobs share a cache, the second can catch a half-written output directory mid-extraction, assume it's done, and return a partial path.

This moves the lock up into `ExtractManager.extract` so the check and the extraction are atomic, the same way `get_from_cache` handles the download side. The now-redundant lock inside `Extractor.extract` is removed — keeping both would deadlock on the same lock file.

The added test forks two processes against a shared cache with a slowed extractor: it fails on `main` (a process observes a partial directory) and passes with the fix. The existing `test_extract.py` is unchanged and still passes.

One case left out of scope: if a process is killed mid-extraction, the next one can still see the partial directory. Fully addressing that would mean extracting to a temporary path and renaming it into place (the `.incomplete` approach used in `get_from_cache`). I can include that here or leave it as a follow-up.
chore: update release-conda.yml

Update `.github/workflows/release-conda.yml` workflow configuration.

cc @lhoestq

Closes huggingface/tracking-issues#808
<!--slack ts:1780939860.368259 channel:C0AJSP0D53L-->


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8251). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
docs: fix duplicate "to" in IterableDataset push-to-hub example

## What does this PR do?

The save-to-hub paragraph in `docs/source/stream.mdx` (line 584) currently reads:

> "you wish to save it **to to** [`~Dataset.push_to_hub`]"

The first `to` closes the relative clause ("the repo you wish to save it to"), and the second `to` introduces the function argument. The collision reads as a typo. Rephrase as:

> "...you wish to save it to **as an argument to** [`~Dataset.push_to_hub`]"

…which keeps both meanings explicit without the adjacent duplicate.

Single-line, doc-only change. The same file is touched by #8109 but at different lines (60, 97, 276); line 584 is not in that PR's diff.

## Before submitting

- [x] Did you read the [contributor guideline](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md)?
- [x] This PR fixes a typo in the docs.
- [ ] Did you make sure to update the documentation with your changes? _(This PR is the documentation change.)_
- [ ] Did you write any new necessary tests? _(N/A, docs-only.)_
Fix JSON loader schema inference for files starting with a UTF-8 BOM (#8241)

Fixes #8241.

## What's the bug

When a JSONL file starts with a UTF-8 BOM (`\xef\xbb\xbf`), `load_dataset("json", ...)` infers a different schema than the same file without the BOM.

The JSON builder's first-batch "mixed-struct-types" pre-scan parses each line with `ujson` to decide whether to preserve heterogeneous nested dicts as a `Json` type:

```python
try:
    examples = [ujson_loads(line) for line in batch.splitlines()]
except ValueError:
    pass
else:
    json_field_paths += find_mixed_struct_types_field_paths(examples)
```

`ujson` rejects the leading BOM with `ValueError`, the broad `except ValueError` swallows it, and `find_mixed_struct_types_field_paths` is never called. PyArrow's `read_json` tolerates the BOM and instead unifies the schema, materializing spurious `null`s for fields absent from individual records. So the user-visible content silently changes depending on whether the file happens to start with a BOM.

## Fix

Strip a leading UTF-8 BOM from the first batch (guarded to the start of the file) so the pre-scan and PyArrow operate on the same bytes.

## Test

Added a regression test: a BOM-prefixed variant of the existing `jsonl_file_with_lists_of_dicts_of_varying_keys` fixture, asserting it produces the same schema (`EXPECTED_LISTS_OF_DICTS_WITH_VARYING_KEYS`) as the non-BOM file. It fails on `main` and passes with this change.

`ruff check`, `ruff format`, and `pytest tests/packaged_modules/test_json.py::test_json_generate_tables` all pass locally.



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8243). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
docs(map): clarify that fn_kwargs yields a stable cache fingerprint vs closures

## Summary

When `Dataset.map()` or `Dataset.filter()` receives a closure that closes over an object containing non-deterministic state — for example a UUID, a random seed, or any mutable attribute of a class instance — `dill` serialises the **entire** captured object into the fingerprint.  Two calls that are logically identical but use different object instances therefore produce different fingerprints, and the computed cache is never reused.  This happens silently: no warning is emitted and the re-computation cost can be severe for large datasets.

Passing the same values through `fn_kwargs` (or `functools.partial`) avoids the problem because only the explicit, deterministic values are hashed — not the surrounding object.  The fix is already available to users today; what was missing was documentation explaining *why* the behaviour occurs and *how* to avoid it.

This PR adds a focused **Tip** block to the `fn_kwargs` parameter in `Dataset.map()` with side-by-side code examples (unstable closure vs stable `fn_kwargs`) and a back-reference in `Dataset.filter()`.  It also adds a new test (`test_fn_kwargs_produces_stable_fingerprint_vs_closure`) that pins the described behaviour and acts as a regression guard.

## Issue

Fixes #7986

## Local verification

```bash
# Linter
ruff check src/datasets/arrow_dataset.py tests/test_fingerprint.py
# All checks passed!

# New test
pytest tests/test_fingerprint.py::test_fn_kwargs_produces_stable_fingerprint_vs_closure -xvs
```

```
============================= test session starts ==============================
platform linux -- Python 3.11.15, pytest-9.0.3, pluggy-1.6.0
collecting ... collected 1 item

tests/test_fingerprint.py::test_fn_kwargs_produces_stable_fingerprint_vs_closure PASSED

============================== 1 passed in 0.06s ===============================
```

```bash
# Full fingerprint test suite
pytest tests/test_fingerprint.py -x -q
# 23 passed, 9 skipped in 3.14s
```

=== LOCAL_TEST_PASSED ===

## Risk

Changes are documentation-only in the library source (`arrow_dataset.py` docstrings) and a new test in `test_fingerprint.py`.  No behaviour is modified; the test asserts facts that have always been true about the `Hasher` API.  The only risk is that the Tip wording turns out to be misleading or incomplete — happy to revise based on reviewer feedback.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8168). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

looks like there is an error in the test

```
FAILED tests/test_fingerprint.py::test_fn_kwargs_produces_stable_fingerprint_vs_closure - _pickle.PicklingError: Can't pickle <class 'tests.test_fingerprint.test_fn_kwargs_produces_stable_fingerprint_vs_closure.<locals>.Config'>: it's not found as tests.test_fingerprint.test_fn_kwargs_produces_stable_fingerprint_vs_closure.<locals>.Config
```
Nightly OSS: Bug loading `sentence-transformers/parallel-sentences`

Automated nightly Codex contribution for https://github.com/huggingface/datasets/issues/5081.

Verification:
- See the uploaded nightly artifacts for the Codex final report.
- See this workflow run for exact commands and logs.

This PR is generated by xodn348/oss-nightly-control and is opened ready for maintainer review when the local nightly verification step succeeds.
fix(image): guard exif_transpose against corrupted EXIF rational tags

## Summary

When an `IterableDataset` contains images with corrupted EXIF metadata (e.g. a TIFF rational tag whose denominator is zero), `PIL.ImageOps.exif_transpose` raises `ZeroDivisionError` inside `Image.decode_example`. Because this exception escapes the internal generator frame, the streaming iterator dies immediately — calling `next()` again raises `StopIteration` rather than advancing to the next sample. This silently drops an unknown number of training examples and makes the data pipeline non-resumable without a full restart from sample 0, causing silent data repetition.

The fix wraps `PIL.ImageOps.exif_transpose` in a `try/except Exception` block. When PIL raises (for any malformed-EXIF reason), `decode_example` emits a `UserWarning` naming the affected file and returns the image without orientation correction. The exception is no longer allowed to escape `decode_example`, so the streaming iterator remains alive and iteration continues normally from the next sample.

A new test (`test_image_decode_example_with_corrupted_exif_orientation_tag`) mocks `PIL.ImageOps.exif_transpose` to raise `ZeroDivisionError` and asserts that (a) the returned image is the correct size/mode without orientation rotation and (b) a `UserWarning` containing "EXIF" is emitted.

## Issue

Fixes #8165

## Local verification

```
$ cd /tmp/datasets
$ ruff check src/datasets/features/image.py tests/features/test_image.py
All checks passed!
ruff exit: 0

$ python -m pytest tests/features/test_image.py::test_image_decode_example \
    tests/features/test_image.py::test_image_decode_example_with_exif_orientation_tag \
    tests/features/test_image.py::test_image_decode_example_with_corrupted_exif_orientation_tag \
    -v
============================= test session starts ==============================
platform linux -- Python 3.11.15, pytest-9.0.3, pluggy-1.6.0
plugins: datadir-1.8.0, anyio-4.13.0
collected 3 items

tests/features/test_image.py::test_image_decode_example PASSED
tests/features/test_image.py::test_image_decode_example_with_exif_orientation_tag PASSED
tests/features/test_image.py::test_image_decode_example_with_corrupted_exif_orientation_tag PASSED

============================== 3 passed in 0.12s ==============================
=== LOCAL_TEST_PASSED ===
```

## Risk

The `except Exception` catch is intentionally broad to cover any PIL internal error from `exif_transpose` (not just `ZeroDivisionError`), since future Pillow versions may raise different exceptions for similarly malformed data. The worst-case regression is that a genuine PIL bug in `exif_transpose` would now be silently swallowed — but this is mitigated by the `UserWarning`, which lets users notice and investigate affected samples. The happy path (well-formed EXIF) is unchanged and remains outside the try/except.
fix(arrow_dataset): clear stale local temp dir before re-downloading from remote FS

## What does this PR do?

Fixes #5114 — `Dataset.load_from_disk` raises a `FileNotFoundError` when called **twice** on the same remote-filesystem path within the same Python session.

### Root cause

`Dataset.load_from_disk` uses `fs.download(src, dest, recursive=True)` to copy the remote dataset to a local temp directory produced by `Dataset._build_local_temp_path`.  The path is deterministic (keyed on the remote path), so a second call within the same session re-uses the same `dest`.

`fs.download` delegates to `AbstractFileSystem.get`, which checks `LocalFileSystem().isdir(dest)`.  When `dest` already exists from the first call `isdir` returns `True`, causing fsspec to treat `dest` as the *parent* directory and append `basename(src)` to every copied path.  The dataset files end up one level deeper (e.g. `.../train/train/`) while the code still looks for them at `.../train/`, triggering a `FileNotFoundError`.

### Fix

Before calling `fs.download`, check whether the local temp directory already exists and remove it with `shutil.rmtree` if so.  `shutil` is already imported in this file.  We also add a `dest.parent.mkdir(parents=True, exist_ok=True)` guard so intermediate directories are always present for deeply nested remote paths.

```
=== LOCAL_TEST_PASSED ===
```

### Tests

Added a parametrised regression test `test_load_from_disk_remote_fs_repeated_load` covering both a flat (`mock://flat_ds`) and a nested (`mock://a/b/nested_ds`) remote path.  The test calls `load_from_disk` twice for the same path and asserts the returned dataset equals the original both times.

Before this fix both parametrisations fail with `FileNotFoundError`; after the fix all 8 tests in the relevant test class pass.

## Before submitting

- [x] The documentation is up-to-date (no user-facing API change).
- [x] The new test covers the exact failure path from issue #5114.
- [x] No new dependencies introduced.
Fix batch(by_column=...) crashing after shard/shuffle/split

## Bug

`batch(by_column=...)` (#8172) works when you iterate the dataset directly, but crashes as soon as the dataset gets copied into a new shape — which happens in very common situations: calling `.shard()` or `.shuffle()`, using `split_dataset_by_node`, or just using a torch `DataLoader` with `num_workers > 0` (each worker gets its own copy):

```python
from datasets import Dataset

ds = Dataset.from_dict({"episode": [0] * 5 + [1] * 5, "x": list(range(10))}).to_iterable_dataset(num_shards=2)
batched = ds.batch(by_column="episode")

list(batched)                               # works
list(batched.shard(num_shards=2, index=0))  # crashes:
# TypeError: _batch_accumulate_arrow_table_by_columns() missing 2 required
# positional arguments: 'tables_accumulator' and 'length'
```

## Cause

Under the hood, `batch(by_column=...)` is a map whose function needs two extra arguments (`tables_accumulator`, `length`) to collect rows until a group is complete. A boolean flag on `MappedExamplesIterable` tells the iteration loop to pass those arguments in.

When the iterable is rebuilt by `shuffle_data_sources`, `shard_data_sources`, or `reshard_data_sources`, every constructor argument is forwarded to the new copy **except that flag** — it silently falls back to its `False` default. The copy then calls the function *without* the two extra arguments, and the function (which still requires them) raises the `TypeError` above.

## Fix

Forward the flag in those three methods (one line each), and add a regression test that runs `.shard()` and `.shuffle()` on a `by_column` batched stream and checks groups stay whole.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8259). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix data file patterns starting with "./" for Hub datasets (#7727)

Fixes #7727.

`data_files` patterns starting with `./` raise `FileNotFoundError` when the dataset is loaded from the Hub, even though the same patterns work locally.

The asymmetry comes from `resolve_pattern()`: it joins relative patterns onto the base path with `xjoin` (a plain `posixpath.join`), which keeps the `.` segments, so the glob pattern ends up as `hf://datasets/<repo>/./plain_text/*.parquet`. `HfFileSystem.glob` (like fsspec's generic glob) treats `.` as a literal directory name and matches nothing, while `LocalFileSystem` normalizes the dot segments away — hence local loading works and Hub loading doesn't.

This PR strips `.` segments from relative patterns in `resolve_pattern()` before joining, so `./data/*` resolves the same as `data/*` on every filesystem. The stripping also removes any leading slashes it uncovers (e.g. in `.//data/*`), so a relative pattern can never turn absolute and resolve outside the base path. `..` segments are left untouched, as before.

Added tests for the local, mock-fs and mock-dataset-repository cases (they fail without the fix, no network needed), and checked manually that `resolve_pattern("./plain_text/*.parquet", "hf://datasets/rajpurkar/squad")` now resolves the two parquet files instead of raising.
config paths that start with ./ are not valid as hf:// accessed repos, but are valid when accessed locally

### Describe the bug

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - images/xyz/*.jpg
```
will correctly download but

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - ./images/xyz/*.jpg
```
will error with `FileNotFoundError` due to improper url joining. `load_dataset` on the same directory locally works fine.

### Steps to reproduce the bug

1. create a README.md with the front matter of the form

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - ./images/xyz/*.jpg
```

2. `touch ./images/xyz/1.jpg`
3. Observe this directory loads with `load_dataset("filesystem_path", "some_config")` correctly.
4. Observe exceptions when you load this with `load_dataset("repoid/filesystem_path", "some_config")`

### Expected behavior

`./` prefix should be interpreted correctly

### Environment info

datasets 4.0.0
datasets 3.4.0

reproduce


Tracked this down: `resolve_pattern()` joins relative patterns onto the base path with `xjoin`, which keeps the `.` segments, and `HfFileSystem.glob` (like fsspec's generic glob) treats `.` as a literal directory name — so `hf://datasets/<repo>/./...` matches nothing, while `LocalFileSystem` normalizes the dot segments away, which is why the same dataset loads locally. Fix in #8260: strip `.` segments from relative patterns before joining.
Add support audio from video

Add support for extracting video from audio. Closes https://github.com/huggingface/datasets/issues/8007

Example:
```python
from datasets import load_dataset

test_ds = load_dataset("Samoed/testds")
test_ds["train"][0]["video"].audio
# <datasets.features._torchcodec.AudioDecoder object at 0x34422fb60>
test_ds["train"][0]["video"].audio.metadata
# AudioStreamMetadata:
#   duration_seconds_from_header: 7.635
#   begin_stream_seconds_from_header: 0
#   bit_rate: 104916
#   codec: aac
#   stream_index: 1
#   duration_seconds: 7.635
#   begin_stream_seconds: 0
#   sample_rate: 48000
#   num_channels: 2
#   sample_format: fltp
```
Use cached .no_exist entries from the hub cache in cached_path (#7686)

Fixes #7686.

`cached_path` requested the Hub on every call for files that usually don't exist in dataset repos (`.huggingface.yaml`, `dataset_infos.json`) because its `hf://` branch goes straight to `HfApi.hf_hub_download`. It now consults `huggingface_hub.try_to_load_from_cache` first and raises `FileNotFoundError` right away on a cached non-existence hit — the same error the `EntryNotFoundError` path raises, so callers are unaffected.

One correction to the issue: the write half already works. `hf_hub_download` has written the `.no_exist/<sha>/<file>` entries on `EntryNotFoundError` since before our minimum pin (checked 0.25.0, 0.33.2 and 1.19.0) — they just don't show up in plain `ls` because `.no_exist` is a dot-directory. So only the read side was missing, and the fix mirrors `transformers.utils.hub.cached_files`: the negative cache is only trusted when the resolved revision is a commit hash (immutable), and it is skipped with `force_download=True`.

Running the issue's metadata-counting repro against a small dataset with a fresh `HF_HOME`: 2 missing-file metadata requests on the first `load_dataset` call and 0 on every subsequent one (previously 2 on every call).

Added regression tests in `tests/test_file_utils.py` covering the negative-cache hit (no `hf_hub_download` call), non-sha revisions (still request the Hub), and `force_download` (ignores `.no_exist`).