noamaanMulla-03/datasets-issues huggingface dataset:
Fix webdataset type inference when a field is null in the first examples (#7549)

Fixes #7549.

The webdataset builder types any JSON field that is null (or an empty list) in all of the first `NUM_EXAMPLES_FOR_FEATURES_INFERENCE = 5` examples as arrow `null`, and the first later batch with a real value then fails the strict cast in `cast_array_to_feature`. Promoting the schema mid-write isn't an option since `ArrowWriter` fixes its schema on the first write, so this resolves it at inference time.

`_split_generators` now keeps consuming examples from the already-open inference pipeline while the inferred schema contains a null type (including nested in structs/lists), unifying each example's schema with `pa.unify_schemas`. The scan resumes at example 6 — `iter_archive`'s `ArchiveIterable` restarts its generator on each `__iter__`, so this doesn't consume anything from `_generate_examples` — and only runs at all when a null type was inferred, so fully-typed datasets do exactly the same work as before.

The scan is capped at 1000 examples; if a field is still null after that (or the first shard is exhausted, i.e. the column is genuinely all-null), it logs a warning naming the fields and keeps the previous behavior of a null-typed feature.

Added regression tests with a tar whose first half has null/empty JSON values: feature inference now yields `string`/`List(string)` and `download_and_prepare` round-trips, plus an all-null-column test guarding the fallback.
Add force_sample_level to split_dataset_by_node

Allow users to explicitly request sample-level sharding for streaming `IterableDataset` even when `num_shards % world_size == 0`. Default behavior is unchanged.

When `num_physical_files < world_size`, or when shards divide evenly but are too few or too imbalanced for shard-level sharding, applying sample-level sharding before expensive `map`/`filter` avoids duplicating those transformations across nodes:

    ds = split_dataset_by_node(ds, rank, world_size, force_sample_level=True)
    ds = ds.map(tokenize_fn)  # only the examples this rank consumes

Map-style datasets ignore the flag.

Closes #8253


@lhoestq , hi, could you please help review this PR? Thanks!
MetadataConfigs drops parquet shards when exported config rows are non-consecutive

## Bug

`_from_exported_parquet_files_and_dataset_infos()` uses `itertools.groupby()` on unsorted `exported_parquet_files`.

If the same `config` appears again after another config, the earlier shard URLs are lost.

## Repro

```python
from itertools import groupby
from operator import itemgetter

exported = [
    {"config": "default", "split": "train", "url": ".../train-0000.parquet"},
    {"config": "other", "split": "train", "url": ".../other/train.parquet"},
    {"config": "default", "split": "train", "url": ".../train-0001.parquet"},
]

metadata_configs = {
    config_name: [p["url"] for _, ps in groupby(rows, itemgetter("split")) for p in ps]
    for config_name, rows in groupby(exported, itemgetter("config"))
}
# default keeps only train-0001
```

## Proof (failing before fix)

```
FAILED - default only had 1 URL instead of 2
```

## Fix

Sort by `(config, split)` before both `groupby()` calls.

I'll follow up with a PR + regression test.
Fix metadata export when config rows are non-consecutive

Fixes #8269

`groupby()` on unsorted exported parquet rows dropped earlier shards when the same config reappeared later in the list. Sort by `(config, split)` first.

Regression test: `test_non_consecutive_config_rows_are_merged_in_metadata_configs_from_exported_parquet_files`

```
rootdir: /tmp/datasets
configfile: pyproject.toml
plugins: datadir-1.8.0, xdist-3.8.0, anyio-4.13.0, hypothesis-6.155.2
collecting ... collected 1 item

tests/test_metadata_util.py::test_non_consecutive_config_rows_are_merged_in_metadata_configs_from_exported_parquet_files PASSED

============================== 1 passed in 0.39s ===============================
```
fix: report the correct variable type in fingerprint_transform ignore_kwargs validation error

The `ignore_kwargs` validation branch in `fingerprint_transform` builds its error from `type(use_kwargs)` instead of `type(ignore_kwargs)`, so an invalid `ignore_kwargs` is reported with the wrong type while `use_kwargs` is at its default. The preceding `use_kwargs` branch already uses `type(use_kwargs)` correctly; this one-token fix reports the actual offending type.
Improve `Create a dataset` tutorial 

Our [tutorial on how to create a dataset](https://huggingface.co/docs/datasets/create_dataset) is a bit misleading. 
1. In **Folder-based builders** section it says that we have two folder-based builders as standard builders, but we also have similar builders (that can be created from directory with data of required format) for `csv`, `json/jsonl`, `parquet` and `txt` files. We have info about these loaders in separate [guide for loading](https://huggingface.co/docs/datasets/loading#local-and-remote-files) but it's worth briefly mentioning them in the beginning tutorial because they are more common and for consistency. Would be helpful to add the link to the full guide.
2. **From local files** section lists methods for creating a dataset from in-memory data which are also described in [loading guide](https://huggingface.co/docs/datasets/loading#inmemory-data).  

Maybe we should actually rethink and restructure this tutorial somehow.

Labels: name: documentation

I can work on this. The link to the tutorial seems to be broken though @polinaeterna. 

@isunitha98selvan would be great, thank you! which link are you talking about? I think it should work: https://huggingface.co/docs/datasets/create_dataset

Hey I don't mind working on this issue. From my understanding, we want to let the reader know that they can build datasets from `csv`, `json/jsonl`, `parquet` and `txt` files in the **folder-based builders** section and include a link to the full guide. Then in the **from local files** section, we just want to list the methods from in-memory data section such as `.from_dict()`. 

Hey @polinaeterna, I have a pull request for this issue. Can you review and see if it needs any changes?

Hi, I'd be interested in working on this issue if it is still available. I can review the tutorial and propose documentation improvements to clarify the relationship between dataset creation and dataset loading workflows.

is this assigned , i would like to work on this
Fix imagefolder metadata with split-named files

## Summary
- Preserve folder-builder metadata when auto-inferred split filename patterns match only a file such as `train.png`
- Re-resolve default `**` patterns only for auto-inferred folder datasets when sidecar metadata would otherwise be dropped
- Add regression coverage for explicit `imagefolder`, local auto-detection, and explicit single-file `data_files`

Fixes #7201

## Tests
- `python -m pytest tests/packaged_modules/test_imagefolder.py::test_load_dataset_with_metadata_and_split_named_image_file -q`
- `python -m pytest tests/packaged_modules/test_imagefolder.py -q --basetemp=C:\tmp\hfds-imagefolder-tests`
- `python -m pytest tests/test_data_files.py::test_get_data_files_patterns tests/test_data_files.py::test_get_data_patterns_from_directory_with_the_word_data_twice -q`
- `ruff format --check src/datasets/load.py tests/packaged_modules/test_imagefolder.py`
- `ruff check src/datasets/load.py tests/packaged_modules/test_imagefolder.py`
- `git diff --check`


Hi ! Maybe this improvement can be in get_data_patterns() directly ? The split-based detection could simply ignore files that are media files (images/audio/videos) and that have a split name in their file name. This way DEFAULT_PATTERNS_SPLIT_IN_FILENAME would return no files, and DEFAULT_PATTERNS_ALL would be used. WDYT ?

Thanks for the suggestion. I reworked the PR in 7737bfa so the behavior is handled in `get_data_patterns()` instead of in the loader factories.

The split-in-filename detection now ignores exact split-named media files such as `train.png`, `valid.mp3`, and `test.mp4`, so that case falls through to `DEFAULT_PATTERNS_ALL` and metadata remains available. Explicit `data_files=` is still left untouched, covered by the imagefolder regression.

Checks run:
- `.\.venv\Scripts\python.exe -m pytest tests/test_data_files.py::test_get_data_files_patterns_ignores_split_named_media_files -q`
- `.\.venv\Scripts\python.exe -m pytest tests/packaged_modules/test_imagefolder.py::test_load_dataset_with_metadata_and_split_named_image_file -q`
- `.\.venv\Scripts\python.exe -m pytest tests/test_data_files.py::test_get_data_files_patterns tests/test_data_files.py::test_get_data_patterns_from_directory_with_the_word_data_twice -q`
- `.\.venv\Scripts\python.exe -m pytest tests/packaged_modules/test_imagefolder.py -q --basetemp=C:\tmp\hfds-imagefolder-tests`
- `.\.venv\Scripts\ruff.exe format --check src/datasets/data_files.py src/datasets/load.py tests/test_data_files.py tests/packaged_modules/test_imagefolder.py`
- `.\.venv\Scripts\ruff.exe check src/datasets/data_files.py src/datasets/load.py tests/test_data_files.py tests/packaged_modules/test_imagefolder.py`
- `git diff --check`

Thanks @lhoestq — good suggestion.

I'll explore moving the split-based detection into `get_data_patterns()` so the logic lives closer to pattern discovery rather than only at the ImageFolder layer. I'll push a follow-up commit shortly.
fix(json): write valid JSON array when lines=False with batching

## Summary
- Fix `Dataset.to_json(lines=False, orient="records")` when the dataset is larger than `batch_size`
- Previously each batch wrote a separate top-level JSON array (`[...][...]`), producing invalid JSON (`JSONDecodeError: Extra data`)
- Merge batched pandas outputs into one array by stripping per-batch brackets and joining with commas

Fixes #7037

## Test plan
- [x] `pytest tests/io/test_json.py::TestJsonDatasetWriter::test_dataset_to_json_records_batched_without_lines`
- [x] `pytest tests/io/test_json.py::TestJsonDatasetWriter::test_dataset_to_json_lines tests/io/test_json.py::TestJsonDatasetWriter::test_dataset_to_json_orient`

/cc @albertvillanova
docs: sync four docstrings with their actual signatures

## What does this PR do?

Audits docstring Args sections against actual signatures and fixes three drifts:

### `Dataset.train_test_split`

The Args section documents `train_cache_file_name` / `test_cache_file_name` — **neither is a parameter**; the real names are `train_indices_cache_file_name` / `test_indices_cache_file_name`. Following the docs (`ds.train_test_split(train_cache_file_name=...)`) raises `TypeError`. Renamed the two entries — their descriptions already describe the indices cache, so only the names change.

### `DatasetBuilder.as_dataset`

Drops the `verification_mode` entry: the parameter no longer exists (the signature is `(split, in_memory)`), so the entry documents an argument that can't be passed.

### `table.cast_table_to_schema`

The Args entry documents `features ([Features])` — carried over from the sibling `cast_table_to_features`; this function's parameter is `schema` (a `pa.Schema`). Renamed and retyped the entry.

Docstring-only; +4 / −6 across 3 files, no behavior change.


Added one more: `DatasetBuilder.as_dataset` documented a `verification_mode` parameter it doesn't accept (signature is `(self, split=None, in_memory=False)`) — dropped the stale entry. Same drift class as the others.
Fix typos in docstrings and comments

## What does this PR do?

Fixes ~20 typos in docstrings, comments, and messages (no functional changes). Examples: `informations`→`information`, `simulatenous`→`simultaneous`, `acculumation`→`accumulation`, `ambigious`→`ambiguous`, `conveniant`→`convenient`, `unstructred`→`unstructured`, `lenth`→`length`, `compresssed`→`compressed`, `DownloadManger`→`DownloadManager`, and a couple of `doesnt`/`doesnt'`→`doesn't`. A few were in user-facing strings (e.g. `ValueError("please provide valid file information")`, the fingerprint length message).

Found with `codespell` and manually triaged — I left alone the false positives: `fo` (the `fo=` parameter of `CompressedFileSystem`), `.caf` (Core Audio Format extension), `iif` (informal "iff" / if-and-only-if), and one ambiguous `cant`. Only comments, docstrings, and message strings were touched — no identifiers or public API.

## Before submitting

- [x] Did you read the [contributor guideline](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md)?
Feature Request / Discussion: Skip origin_metadata fetch when streaming=True?

Hi team!

I've been looking into the initialization process of the `DatasetBuilder` and noticed that the ETag/origin metadata caching path is executed unconditionally.

While fetching the `origin_metadata` and calculating the ETag is necessary for safely managing a local disk cache when using `as_dataset()`, it seems this might be redundant for streaming. When a user requests `streaming=True`, the pipeline drops into `as_streaming_dataset()`. Because streaming does not use a local cache, it appears the streaming pipeline ignores the generated `_cache_dir` string and never calls `_get_dataset_fingerprint()`. As a result, the origin metadata is fetched and hashed without being used.

I wanted to start a discussion to understand if there is another purpose for the etag/origin_metadata when streaming. If not, would it be possible/desirable to skip this fetch when `streaming=True` to improve initialization latency and save unnecessary network calls?

Thanks!


I think it's great that it checks that every file does exist by default. It's relatively cheap since it uses fsspec which caches file info at folder level.

On the other hand why not add skip_origin_metadata=False from @TheDerbiedOne by default to DatasetBuilder and document what it does (load_dataset passes kwargs to DatasetBuilder)

Hi @lhoestq Thanks for taking a look at this! 
Regarding the file existence check: skipping _get_origin_metadata actually doesn't remove that safeguard. I think th verification that files exist happens earlier during pattern resolution (around L372-L384 in data_files.py (https://github.com/huggingface/datasets/blob/main/src/datasets/data_files.py#L372-L384)).
The _get_single_origin_metadata function (around L498-L519 (https://github.com/huggingface/datasets/blob/main/src/datasets/data_files.py#L498-L519)) runs strictly after resolution, and its only purpose is to extract the ETag or mtime to build the local cache fingerprint. Since streaming completely bypasses the local cache, calculating this fingerprint shouldn't be necessary. Correct me if I misunderstand this part.

Regarding skip_origin_metadata flag, I think it's a good idea to add it.
Skip origin metadata fetch when streaming=True

Fixes #8197

When using `load_dataset(..., streaming=True, data_files=...)`, `DataFilesList.from_patterns` currently fetches origin metadata (ETag/mtime) for each resolved file even though the streaming pipeline doesn't use the local cache/fingerprint path.

This PR adds an opt-out (`skip_origin_metadata`) for data-file resolution and uses it in `load_dataset_builder(..., streaming=True)` so we skip the origin metadata fetch in streaming mode, reducing init latency and avoiding unnecessary network calls.

Includes a unit test that asserts `_get_origin_metadata` is not called for streaming.



Hi! Thanks for putting this PR together. Following up on the discussion in #8197 with the maintainers, instead of only applying this automatically for streaming, we should expose skip_origin_metadata=False as a public parameter (e.g., passing it through load_dataset into DatasetBuilder). Could you amend the PR?
Add option for loading audio with video

### Describe the bug

Currently, `torchcodec` don't allow extracting `Audio` from `Video` https://github.com/meta-pytorch/torchcodec/issues/1158, but when I upload videos with audio to hub using `videofolder`, then this is not possible to retrieve audio from it. Probably `VideoDecoder` can be extended with `audio` parameter to retrieve this information

### Steps to reproduce the bug

```python
from datasets import load_dataset

test_ds = load_dataset("videofolder", data_dir="/path/to/video")  
# uploaded version 
# test_ds = load_dataset("Samoed/testds")

test_ds["train"][1]["video"]
# <torchcodec.decoders._video_decoder.VideoDecoder
```

### Expected behavior

Some option to retrieve audio information.

### Environment info

```
datasets==4.5.0
```


Hi,
I’m interested in working on this issue.

From what I understand, audio decoding would involve an AudioDecoder, and previous discussions have suggested handling it separately. I’d like to explore whether integrating audio extraction in relation to the current video decoding workflow would be appropriate, or if maintaining a separate decoder is the preferred design.

Any guidance would be greatly appreciated.

Thanks!

Hi! I’m also interested in contributing to this issue.

If it would be helpful, I could try implementing a small prototype that exposes audio from the video object (for example via a `get_audio()` method), assuming the underlying decoder provides access to the audio stream.

Would you be open to a draft PR exploring this approach?


> assuming the underlying decoder provides access to the audio stream.

Torccodec.VideoDecoder doesn't support audio in video directly. To use audio from video you need to process mp4 directly and extract audio from it

Thanks for the clarification!

If Torchcodec's VideoDecoder doesn't expose audio streams, then extracting audio directly from the video file (e.g. mp4) makes sense.

Would the preferred approach be to add a helper method in the video feature that reads the file path and extracts audio separately (for example via ffmpeg or a similar backend), or should this be handled as a separate audio feature in the dataset instead?

If that direction makes sense, I’d be interested in trying a small prototype PR.


Following up on my March 13th comment — happy to open a draft PR using ffmpeg as the audio backend if that direction makes sense. Should I proceed with that approach, or is this blocked until torchcodec adds native audio support?

I created PR with addition `audio` property to `VideoDecoder` to torchcodec https://github.com/meta-pytorch/torchcodec/pull/1442

I opened PR with implementation of this feature https://github.com/huggingface/datasets/pull/8265
fix traces streaming

requires https://github.com/TeichAI/teich/pull/12 as well


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8277). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the PR. Implemented them in the latest version: 0.2.5
Add return_file_name parameter to JSON builder (#5806)

## Description
Resolves #5806.

This PR introduces an optional `return_file_name` boolean parameter to the packaged JSON builder. When enabled, it injects a `file_name` column into the generated PyArrow tables, allowing downstream users to track exactly which data shard a specific row originated from during interrupted LLM training runs.

### Changes Made:
* Added `return_file_name: bool = False` to the `JsonConfig` dataclass.
* Updated `_generate_tables` in `src/datasets/packaged_modules/json/json.py` to inspect `self.config.return_file_name`.
* Appended the string representation of the source file to the output dictionary prior to PyArrow yield.
* Added schema checks to ensure "file_name" is included in `self.features` if provided dynamically.

## Note on CI Status:
Local testing and `deps-minimum` CI matrices passed successfully. I noticed that the `ubuntu-latest, deps-latest` and `windows-latest, deps-latest` integration tests are currently failing on this branch. Reviewing the logs, these failures are occurring in `test_inspect.py` and `test_offline_util.py` due to a recent `huggingface_hub` dependency update enforcing strict `namespace/name` validation (e.g., rejecting the `'paws'` and `'dummy'` fixtures). 

Since this is an upstream dependency breakage unrelated to the `json.py` `return_file_name` implementation, I have left those test files untouched to avoid scope pollution in this PR.

 @mariosasko
Fix typos in documentation, comments, and a docstring

aFixes a handful of spelling typos across the docs and source comments/docstrings. This is a documentation/comment-only change; no functional code is affected.

Changes:

docs/source/process.mdx: colums -> columns

docs/source/video_load.mdx: effiently -> efficiently; whithin -> within

src/datasets/arrow_reader.py: informations -> information (ValueError message); conveniant -> convenient (comment)

src/datasets/utils/sharding.py: ambigious -> ambiguous (comments)

src/datasets/features/features.py: unstructred -> unstructured; abritrary -> arbitrary (docstring)
Add the supports of missing types in conversion between arrow and dtype types

The original issue is a crash in _arrow_to_datasets_dtype because the miss of support of fixed_size_binary:

```
Exception:    SplitsNotFoundError
Message:      The split names could not be parsed from the dataset config.
Traceback:    Traceback (most recent call last):
                File "/usr/local/lib/python3.12/site-packages/datasets/inspect.py", line 286, in get_dataset_config_info
                  for split_generator in builder._split_generators(
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.12/site-packages/datasets/packaged_modules/parquet/parquet.py", line 118, in _split_generators
                  self.info.features = datasets.Features.from_arrow_schema(pq.read_schema(f))
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.12/site-packages/datasets/features/features.py", line 1858, in from_arrow_schema
                  else generate_from_arrow_type(field.type)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.12/site-packages/datasets/features/features.py", line 1518, in generate_from_arrow_type
                  return Value(dtype=_arrow_to_datasets_dtype(pa_type))
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.12/site-packages/datasets/features/features.py", line 121, in _arrow_to_datasets_dtype
                  raise ValueError(f"Arrow type {arrow_type} does not have a datasets dtype equivalent.")
              ValueError: Arrow type fixed_size_binary[3] does not have a datasets dtype equivalent.
              
              The above exception was the direct cause of the following exception:
              
              Traceback (most recent call last):
                File "/src/services/worker/src/worker/job_runners/config/split_names.py", line 65, in compute_split_names_from_streaming_response
                  for split in get_dataset_split_names(
                               ^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.12/site-packages/datasets/inspect.py", line 340, in get_dataset_split_names
                  info = get_dataset_config_info(
                         ^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.12/site-packages/datasets/inspect.py", line 291, in get_dataset_config_info
                  raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
              datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.

```

In this pull request I add the support for all the missing types


@lhoestq Hi Quentin. A gentle ping to run the workflows

Hi! Can you only include the change for fixed_size_binary ? For the other types it requires discussing them in an issue first.