noamaanMulla-03/datasets-issues huggingface dataset:
Support batched=True in Dataset.to_dict

## What does this do?

`Dataset.to_dict(batch_size=..., batched=True)` silently ignores both arguments and always returns the full dict, even though its signature, docstring ("Set to `True` to return a generator that yields the dataset as batches"), and `-> Union[dict, Iterator[dict]]` return type all promise batching. The sibling methods `to_pandas` and `to_polars` implement batching; `to_dict` was missing the branch.

Because it returns a plain `dict` instead of the promised generator, iterating the result as documented walks the dict *keys* and raises:

```python
from datasets import Dataset
ds = Dataset.from_dict({"a": list(range(10))})
out = ds.to_dict(batched=True, batch_size=3)
type(out)                 # dict, not a generator
for batch in out:
    batch["a"]            # TypeError: string indices must be integers, not 'str'
```

## Fix

Mirror the `to_pandas`/`to_polars` structure: return the full dict when `batched=False`, otherwise return a generator that yields `batch_size`-row slices (defaulting `batch_size` to `config.DEFAULT_MAX_BATCH_SIZE`). A small local closure keeps the per-batch JSON-field decoding from being duplicated across the two branches.

After the fix:

```python
[len(b["a"]) for b in ds.to_dict(batched=True, batch_size=3)]   # [3, 3, 3, 1]
```

Backward compatible: `batched` defaults to `False`, so every existing `.to_dict()` call still returns a plain dict (verified byte-identical). Index mapping (`select(...)`) and `Json()`-feature decoding both work in batched and non-batched modes.

## Test

Added a "Batched" block to `tests/test_arrow_dataset.py::test_to_dict`, mirroring the existing `test_to_pandas`/`test_to_polars` coverage: it asserts each yielded batch is a `dict` with the right columns and no more than `batch_size` rows. It fails before this change (the returned dict is iterated as keys) and passes after.
Dataset Studio and Viewer down

### Describe the bug

Hello! 

I want to report a bug on the datasets viewer and data studio pages. I have attempted to access it on various datasets from various devices and it will not let me view the dataset this way.  I have tried restarting, login in and out but to no success. I use Hugging Face as a repository that I can interface with in my other work and this has become a major blocker.

Thank you for any help you can provide! If you need any more information please let me know!

<img width="2910" height="887" alt="Image" src="https://github.com/user-attachments/assets/8da511a7-a9ff-475a-aeb1-70727194793d" />

<img width="2927" height="684" alt="Image" src="https://github.com/user-attachments/assets/5c434ace-4d91-4c89-adc4-43b21ec5afa6" />

### Steps to reproduce the bug

1. Select a Dataset
2. Attempt to open Data Studio
3. Unsuccessful?

### Expected behavior

I expect the dataset to be viewable and accessible in this way.

### Environment info

This is a UI issue mainly


Hi ! Thanks for the ping, yes we are aware of the issue and actively working on it. It's taking some time though since we want to make sure it will be 100% back and for good
fix #5354

This PR updates the type hint for the `path_or_paths` argument in dataset loading methods (`from_parquet`, `from_csv`, `from_json`, and `from_text`) in `src/datasets/arrow_dataset.py`.

It replaces `list[PathLike]` with `Sequence[PathLike]`. Since `list` is invariant and `Sequence` is covariant, this prevents users from receiving false-positive type checking errors (like with `mypy`) when passing lists of specific types (like `List[str]`).

### Checklist

* [x] I have read the `CONTRIBUTING.md` file.
* [x] I have run `make style` to format my code.
Fix column loss in horizontal concatenation of arrow-backed IterableDatasets

## Bug

`concatenate_datasets([...], axis=1)` over arrow-backed `IterableDataset`s (e.g. from `.to_iterable_dataset()`) silently drops the **first** dataset's columns and duplicates the **last** dataset's column.

In `HorizontallyConcatenatedMultiSourcesExamplesIterable._iter_arrow`, the merged table is built by appending each later source's columns onto the running accumulator `new_pa_table`. But it appended onto `pa_table` — the loop variable left over from the fetch loop just above, which holds the *last* source's table — and reassigned `new_pa_table` on each iteration instead of accumulating onto it.

## Reproduction

```python
from datasets import Dataset, concatenate_datasets

a = Dataset.from_dict({"a": [1, 2, 3]}).to_iterable_dataset()
b = Dataset.from_dict({"b": [4, 5, 6]}).to_iterable_dataset()
print(list(concatenate_datasets([a, b], axis=1)))
# Expected: [{'a': 1, 'b': 4}, {'a': 2, 'b': 5}, {'a': 3, 'b': 6}]
# Actual (before fix): column 'a' is lost (nulled) and 'b' is duplicated,
#                      which then fails casting to the merged features
```

The non-arrow `__iter__` path is correct (it uses `new_example.update(...)`); only the arrow path was affected. That's why the existing `test_concatenate_datasets_axis_1*` tests — which use `ExamplesIterable` and never hit `_iter_arrow` — didn't catch it.

## Fix

Accumulate onto `new_pa_table` instead of `pa_table` (one line).

## Test

Added `test_concatenate_datasets_axis_1_arrow_backed`, which exercises the arrow-backed path and fails without the fix.



Closing this — on review the fix only covers the equal-length case. The arrow (`_iter_arrow`) path also needs to pad different-length sources with nulls the way the non-arrow path does, otherwise horizontal concat of unequal-length sources raises `ArrowInvalid`. I'd rather rework it to handle that completely before it's reviewed. Apologies for the noise.
Make Dataset generic to allow specifying column types in typehints

Closes #7719

## What does this PR do?

Makes `Dataset` optionally generic so its column types can be specified in a
typehint, similar to how `torch.utils.data.DataLoader[T]` works today (as
requested in the issue).

Before, the only way to document a dataset's columns was a docstring comment:

```python
from datasets import Dataset

QueryDataset = Dataset
"""Query dataset should have `query` and `instructions` columns as `str`"""
```

After this change, a `TypedDict` can be used instead, and static type
checkers (mypy, pyright) will infer the correct row type from `dataset[i]`:

```python
from typing import TypedDict
from datasets import Dataset

class QueryInput(TypedDict):
    query: str
    instruction: str

def queries_loader() -> Dataset[QueryInput]:
    ...

ds = queries_loader()
row = ds[0]        # inferred as QueryInput instead of `dict`
row["query"]       # str
```

### Changes

- `src/datasets/arrow_dataset.py`:
  - `Dataset` now inherits from `Generic[DatasetColumns]` (new `TypeVar`
    `DatasetColumns = TypeVar("DatasetColumns", bound=Mapping[str, Any])`).
  - The `__getitem__` overload for integer row access now returns
    `DatasetColumns` instead of the untyped `dict`. Slice/iterable and
    string-key overloads are unchanged (`dict` / `list` respectively).

### Backward compatibility

- Purely a typing change — no behavioral or runtime changes.
- `Dataset` used unparametrized (the existing convention) continues to work
  exactly as before; `dataset[i]` is inferred as `Any` in that case, matching
  today's behavior and the same pattern used by `DataLoader`.
- Verified with mypy: `Dataset[QueryInput]` correctly narrows `dataset[i]` to
  `QueryInput`, while plain `Dataset` still resolves to `Any`.

### Testing

- Ran `ruff check --fix` / `ruff format` on the touched file — no changes
  needed beyond the edit itself.
- Ran the existing `__getitem__`-related tests plus the full
  `tests/test_arrow_dataset.py` suite: 400 passed, 19 skipped, 3 pre-existing
  failures unrelated to this change (`test_from_spark*`, which fail locally
  because no Java runtime is installed for PySpark).

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md)?
- [x] Did you make sure to update the documentation with your changes?
- [x] Did you write any new necessary tests?

> **Note:** Claude Code was used to assist in drafting this fix. All changes were reviewed by the submitter.
docs(video_load): fix typos 'effiently' and 'whithin'

Fixes two typos in `docs/source/video_load.mdx`:
- "effiently" -> "efficiently" in the description of `get_frames_in_range`
- "whithin" -> "within" in the description of `get_frames_played_in_range`

Both flagged by `typos` (typos-cli 1.47.2) and verified by reading the surrounding context.

## AI assistance

This patch was drafted with the help of an AI coding assistant. The two corrections are limited to the prose on lines 40 and 49 of the file and have been reviewed line-by-line before submission.



Hi, just a gentle ping on this in case it slipped through - no rush at all. Happy to rebase, split, or adjust anything if that helps. Thanks for maintaining this project.
docs: fix typos, grammar, and branding consistency across docs and source

This commit addresses several documentation quality issues found across the repository — fixing typos, grammar errors, brand name inconsistencies, and adding modern tooling references for new contributors.

## Changes

### README.md
- Fix duplicate word: "frameworks frameworks" → "frameworks"
- Standardize brand name: "HuggingFace Datasets Hub" → "Hugging Face Datasets Hub"
- Add `uv` installation section for faster environment setup

### CONTRIBUTING.md
- Add `uv pip install -e ".[dev]"` as an alternative setup command
- Fix grammar: "To do, go" → "To do so, go"
- Fix punctuation: trailing space before period in pre-commit note (`again .` → `again.`)
- Standardize brand name: "HuggingFace [code of conduct]" → "Hugging Face [code of conduct]"

### docs/source/stream.mdx
- Fix article usage: "a [`IterableDataset`]" → "an [`IterableDataset`]" (vowel sound rule)
- Fix code comment: "shuffles the shards order and use" → "uses" (subject-verb agreement)
- Fix phrase: "as soon one of the dataset runs out" → "as soon as one of the datasets runs out"
- Fix pluralization: "every samples in every dataset" → "every sample in every dataset"
- Fix abbreviation punctuation: "i.e the" → "i.e. the"

### docs/source/quickstart.mdx
- Standardize brand name: "a HuggingFace [`~datasets.Dataset`]" → "a Hugging Face [`~datasets.Dataset`]" (3 occurrences)

### docs/README.md
- Standardize copyright notice: "The HuggingFace Team" → "The Hugging Face Team"

### notebooks/README.md
- Standardize copyright notice: "The HuggingFace Team" → "The Hugging Face Team"

### src/datasets/iterable_dataset.py
- Fix typo in `map()` docstring: "simulatenous" → "simultaneous"
- Fix typo in `filter()` docstring: "simulatenous" → "simultaneous"
- Add return type hint to `identity_func`: `(x)` → `(x: Any) -> Any`
- Add return type hint to `_rename_columns_fn`: missing `-> dict` return type
concatenate_datasets(axis=1) with Arrow format drops the first source's columns

### Describe the bug

The Arrow fast path of horizontal concatenation (`concatenate_datasets([...], axis=1)` consumed through `.with_format("arrow")`, or any Arrow/tensor consumer) drops the columns of the first source. The plain-Python path is correct; only the Arrow path diverges.

`HorizontallyConcatenatedMultiSourcesExamplesIterable._iter_arrow` builds each output batch like this:

```python
for j, table in enumerate(pa_tables):
    if j == 0:
        new_pa_table = table
    else:
        for name, col in zip(table.column_names, table.columns):
            new_pa_table = pa_table.append_column(name, col)
```

The inner loop appends onto `pa_table`, which is the loop variable that leaked from the fetch loop above (`key, pa_table = next(pa_table_iterator)`), so it holds the last source's table rather than the accumulator `new_pa_table`. Each iteration also reassigns `new_pa_table` from scratch. For sources with columns `[a, b]` and `[c]`, the batch ends up as the second source's table with `c` appended to itself (columns `[c, c]`); columns `a` and `b` are lost. When the dataset has resolved features, the resulting schema mismatch surfaces as a `CastError`.

### Steps to reproduce the bug

```python
from datasets import Dataset, concatenate_datasets

ds1 = Dataset.from_dict({"a": [1, 2], "b": [3, 4]}).to_iterable_dataset()
ds2 = Dataset.from_dict({"c": [5, 6]}).to_iterable_dataset()

concatenated = concatenate_datasets([ds1, ds2], axis=1)

# plain-Python path is correct
print(next(iter(concatenated)))            # {'a': 1, 'b': 3, 'c': 5}

# Arrow fast path drops columns a and b
print(next(iter(concatenated.with_format("arrow"))))
```

The Arrow iteration raises:

```
datasets.table.CastError: Couldn't cast
c: int64
c: int64
a: null
b: null
...
to
{'a': Value('int64'), 'b': Value('int64'), 'c': Value('int64')}
because column names don't match
```

### Expected behavior

The Arrow fast path should produce the same columns as the plain-Python path: all source columns, in order (`['a', 'b', 'c']`), with matching values.

### Environment info

- `datasets` 5.0.1.dev0 (main, commit 41adfd0f9)
- `pyarrow` 25.0.0
- Python 3.11.15, Linux

Reproduced in a clean container against current `main`. The fault line was introduced in #8068.



#self-assign
Fix column drop in Arrow path of axis=1 concatenation

Fixes #8341

### Root cause

`HorizontallyConcatenatedMultiSourcesExamplesIterable._iter_arrow` accumulates the concatenated batch with:

```python
for name, col in zip(table.column_names, table.columns):
    new_pa_table = pa_table.append_column(name, col)
```

`pa_table` here is the loop variable that leaked from the fetch loop above (`key, pa_table = next(pa_table_iterator)`), so it is the *last* source's table, not the accumulator `new_pa_table`. Each iteration also rebuilds `new_pa_table` from scratch. For sources with columns `[a, b]` and `[c]`, the batch becomes the second source's table with `c` appended to itself (columns `[c, c]`), and the first source's columns are lost. With resolved features this surfaces as a `CastError`.

The plain-Python `__iter__` path directly above accumulates correctly (`new_example.update(example)`); only the Arrow fast path diverges.

### Fix

Append onto the accumulator instead of the leaked variable:

```python
new_pa_table = new_pa_table.append_column(name, col)
```

One token. The invariant restored: the Arrow fast path yields the same column set, in the same order, as the plain-Python path.

### Verification

Run in a clean `python:3.11-slim` container against this branch, editable install, `datasets.__file__` confirmed to resolve to the source tree (not site-packages):

- Added `test_concatenate_datasets_axis_1_arrow_format` (public API: two `to_iterable_dataset()` sources with disjoint columns, `concatenate_datasets(axis=1).with_format("arrow")`, asserting columns `['a', 'b', 'c']` and values). It fails on `main` (the `CastError` above) and passes on this branch.
- The three existing `test_concatenate_datasets_axis_1*` tests pass on both `main` and this branch, so behavior of the plain-Python path is unchanged.
- `ruff check` and `ruff format --check` pass on `tests src benchmarks utils setup.py`.

I ran the axis=1 concatenation and `with_format` slice of `tests/test_iterable_dataset.py` (11 passed, 25 skipped for optional torch/tf/jax), not the full test suite.
Make the dataset fingerprint independent of Arrow chunking

Fixes #8327.

As discussed in the issue, the OOM in `Dataset.from_pandas` comes from `generate_fingerprint`, not from the Arrow conversion. Fingerprinting hashes `dataset.__dict__`, and the underlying `pa.Table` (and the `InMemoryTable` wrapping it) pickles chunk-by-chunk, so the serialized size and the hashing cost scale with the number of chunks rather than the amount of data. `shuffle().to_pandas()` produces one chunk per row, which is where it really blows up.

Following @lhoestq's suggestion to make the table hash chunk-count-independent (rather than calling `combine_chunks()` on the stored data), this PR does two things:

- `InMemoryTable` now serializes via its underlying `table` only. The `_batches`/`_offsets` index is derived and rebuilt in `__init__`, so there's no need to pickle it. This mirrors how `MemoryMappedTable` and `ConcatenationTable` already define `__getstate__`/`__setstate__`.
- `pa.Table` and `pa.ChunkedArray` get dill reducers that serialize a chunk-count-independent form. They combine one column at a time, so memory stays bounded and a table with the same data hashes the same regardless of how it's chunked.

The combine is transient and per-column, so it never materializes a full extra copy of the table.

### Result

On the repro from the issue (25k rows shuffled to one chunk per row), `Dataset.from_pandas` goes from a ~1.5 GB allocation to a few MB. I measured RSS on the scaled 2000-chunk case: the delta across `from_pandas` dropped from ~1500 MB to ~4 MB.

The fingerprint is now stable across different chunkings of the same data and still distinguishes different data. `pickle` round-trips of a dataset reconstruct correctly. `tests/test_fingerprint.py` and `tests/test_table.py` pass (the two failures I see, `test_hash_torch_compiled_module` and `test_move_script_doesnt_change_hash`, also fail on `main` in my environment and are unrelated).

I added `test_hash_arrow_table_is_independent_of_chunking` to cover the invariant.

One thing that I would flag is that this changes the fingerprint values for existing datasets, so it will invalidate caches once. This seems unavoidable for a fix that changes how the table is hashed.
Fix revision-aware cache lookup in load_dataset

Fixes #7928

When `load_dataset()` falls back to the local cache, for example when offline or when the Hub is unreachable, it ignores the `revision` argument and simply returns the most recently modified cached version. This can lead to unexpected behavior when a specific revision is requested.

This happens because `CachedDatasetModuleFactory` neither accepted nor used the `revision` parameter, and `dataset_module_factory` did not pass it along either.

In this PR, I’ve updated the cache handling so that a specific revision can be respected when loading from cache.

In particular, SHA-based revisions are now matched against cached commit directories. If a matching commit is found, the corresponding cached version is returned. If not, a `DatasetNotFoundError` is raised, listing the available cached SHAs.

For branch or tag names such as `main` or `v1.0`, I was not able to reliably resolve them without access to the Hub. In such cases, a `DatasetNotFoundError` is raised with a clear message suggesting the use of a commit SHA instead.

The existing behavior is preserved when `revision=None`, meaning the latest cached version is still used by default.

Also, I fixed the inner `except Exception` block in `dataset_module_factory`. Previously, it would swallow `DatasetNotFoundError` raised by the cache logic and fall back to the outer Hub-related error. Now, the correct error is propagated.

Test I did:

- [x] Verified that a full SHA present in cache returns the correct version
- [x] Verified that a short SHA prefix resolves correctly when unique
- [x] Confirmed that missing SHAs raise `DatasetNotFoundError` with available SHAs listed
- [x] Confirmed that branch and tag names raise an error with an offline hint
- [x] Ensured that `revision=None` continues to return the latest cached version
- [x] Confirmed that `test_load_dataset_invalid_revision_with_cache` (from PR #7929) still passes


Hi @lhoestq just a friendly follow-up on the review when you have a chance. Thanks!
Dataset.to_pandas() returns raw JSON strings for Json() columns instead of decoded objects

### Describe the bug

`Dataset.to_pandas()` returns the raw Arrow storage strings for `Json()` feature columns, while every other in-memory export path decodes them to Python objects. `to_dict()`, `to_list()`, `to_json()` and `with_format("pandas")` all run `get_json_field_paths_from_feature` + `json_decode_field`, but `to_pandas()` skips that step, so it alone yields `'{"a":1}'` (str) where the others yield `{'a': 1}` (dict). Nested `List(Json())` columns are affected the same way.

### Steps to reproduce the bug

```python
from datasets import Dataset, Features
from datasets.features import Json

ds = Dataset.from_dict({"col": [{"a": 1}]}, features=Features({"col": Json()}))

print(ds.to_dict()["col"][0])                 # {'a': 1}    (dict)
print(ds.with_format("pandas")[:]["col"][0])  # {'a': 1}    (dict)
print(ds.to_pandas()["col"][0])               # '{"a":1}'   (str)  <- inconsistent
print(type(ds.to_pandas()["col"][0]))         # <class 'str'>
```

### Expected behavior

`to_pandas()` decodes `Json()` columns to Python objects, consistent with `to_dict()` / `to_list()` / `to_json()` and `with_format("pandas")`, so that `ds.to_pandas()["col"][0] == {"a": 1}`.

### Environment info

- `datasets` version: 5.0.1.dev0 (main, commit 41adfd0f9)
- Platform: Linux, Python 3.11
- `pyarrow` version: 21.0.0
Decode Json() columns in Dataset.to_pandas()

## What does this PR do?

`Dataset.to_pandas()` returned the raw Arrow storage strings for `Json()` feature columns, while every other in-memory export path decodes them. `to_dict()`, `to_list()`, `to_json()` and `with_format("pandas")` all apply `get_json_field_paths_from_feature` + `json_decode_field`; `to_pandas()` skipped that step, so it alone produced `'{"a":1}'` (str) where the siblings produce `{'a': 1}` (dict). Nested `List(Json())` columns had the same gap.

This applies the same decode after the Arrow-to-pandas conversion, in both the single and batched code paths, so a `Json()` column materializes as decoded Python objects (pandas `object` dtype), consistent with the other export methods.

Before:

```python
ds = Dataset.from_dict({"col": [{"a": 1}]}, features=Features({"col": Json()}))
ds.to_dict()["col"][0]      # {'a': 1}
ds.to_pandas()["col"][0]    # '{"a":1}'  (str)  <- fixed to {'a': 1}
```

Fixes #8343

## How was it tested?

- Added `test_to_pandas_decode_json` and `test_to_pandas_decode_nested_json` next to the existing `to_dict`/`to_list` decode tests, covering a top-level `Json()` column (including a `None` value), a nested object, `List(Json())`, and the batched generator path. They fail on `main` (`to_pandas` returns raw strings) and pass with this change.
- Verified in a clean container on this commit: the two new tests fail before the fix and pass after; the existing `to_dict`/`to_list` decode, `json_feature`, `to_pandas` and `to_polars` tests still pass.
- `ruff check` and `ruff format --check` are clean on the changed files.

## Not covered (possible follow-up)

`to_polars()` shows the same raw-JSON-string behavior, but I left it out of this PR deliberately. Unlike pandas, whose `object` dtype holds arbitrary decoded objects faithfully, polars has no per-cell Python-object column, and letting it infer a schema is lossy for heterogeneous `Json()` data: `[{"a": 1}, {"b": [2, 3]}]` collapses to `Struct({'a': Int64})` and the second row becomes `{'a': None}`, silently dropping `b`. The faithful alternative is a `pl.Object` column, which is a representation choice I would rather leave to you. Happy to send a separate PR for `to_polars()` if you confirm the behavior you want.