noamaanMulla-03/datasets-issues huggingface dataset:
Make patch_submodule robust to __builtins__ being a module (PyPy / __main__)

Closes #7636.

### Problem

`patch_submodule` reads the builtins through `globals()["__builtins__"]`:

```python
elif target_attr in globals()["__builtins__"]:  # if it's a builtin like "open"
    self.original[target_attr] = globals()["__builtins__"][target_attr]
```

The value of a module's `__builtins__` global is a CPython implementation detail: it is the `builtins` **module** (not its dict) in the `__main__` module and under PyPy, and only *usually* a dict in imported modules ([docs](https://docs.python.org/3/reference/executionmodel.html#builtins-and-restricted-execution)). When it is the module, `target_attr in globals()["__builtins__"]` raises:

```
TypeError: argument of type 'module' is not iterable
```

so patching a builtin such as `open` fails. This is what #7636 hit running `datasets` streaming under PyPy.

### Fix

Use `builtins.__dict__` directly, which is always the builtins mapping regardless of the runtime. In the common CPython case (an imported module whose `__builtins__` already *is* `builtins.__dict__`) the behavior is unchanged; it just also works under `__main__`/PyPy.

### Test

`test_patch_submodule_builtin_when_builtins_is_module` sets `datasets.utils.patching.__builtins__` to the `builtins` module (simulating the PyPy / `__main__` runtime) and patches `open`. It raises the `TypeError` above on the current code and passes with this change; the existing patching tests are unaffected (full `tests/test_patching.py` is green).
Fix silent dataset truncation in batched map with mismatched empty column (#6879)

## What does this PR do?

Fixes #6879.

`Dataset.map(fn, batched=True)` silently truncates the dataset to **0 rows** (instead of raising) when the mapping function empties an *existing* column while another column stays non-empty.

### Reproducer (from the issue)

```python
import datasets
data = datasets.Dataset.from_dict({"test": [1]})

def mapping_fn(examples):
    return {"test": [], "y": [1]}

data = data.map(mapping_fn, batched=True)
print(len(data))   # 0  -> silent data loss
```

Returning an empty list for a *new* column (e.g. `{"x": []}`) already raises `pyarrow.lib.ArrowInvalid`, so the two cases were inconsistent.

### Root cause

`ArrowWriter._write_batch` skips a batch that "appears empty", but the emptiness check only looked at the **first** column:

```python
if batch_examples and len(next(iter(batch_examples.values()))) == 0:
    return
```

When a batched `map` overwrites an existing column with `[]`, that column stays first in the merged dict, so the whole length-mismatched batch was dropped instead of erroring.

### Fix

Skip the batch only when **every** column is empty. Genuinely empty batches (e.g. a batch that filters out all its rows) are still ignored, while mismatched batches now fall through to `pa.Table.from_arrays`, which raises the same informative `ArrowInvalid` already produced for the new-column case.

```python
if batch_examples and all(len(col_values) == 0 for col_values in batch_examples.values()):
    return
```

### Behavior

| | issue reproducer |
| --- | --- |
| before | no exception, `len(data) == 0` |
| after | `ArrowInvalid: Column 1 named y expected length 0 but got length 1` |

Normal batched `map` and legitimate "all columns empty" batches are unchanged.

### Tests

Added `test_write_batch_with_mismatched_empty_column_raises` in `tests/test_arrow_writer.py`. The existing `test_write_batch` already covers the legitimate all-empty-batch skip, so that path stays protected.

`ruff check` / `ruff format --check` pass; `tests/test_arrow_writer.py` passes locally.
Fix Dataset.take raising IndexError when taking more elements than available

### Description
`Dataset.take(n)` is implemented as `self.select(range(n))`, so when `n` is larger than the dataset length it raises `IndexError: Index N out of range for dataset of size M` instead of returning the whole dataset. This is inconsistent with `IterableDataset.take`, which clamps and returns all available elements — the two were meant to be aligned (see #6813). `Dataset.skip` already clamps correctly (`range(n, len(self))`); only `take` was affected.

This clamps `n` to `len(self)` so `take` returns the whole dataset when asked for more, matching `IterableDataset.take` and the "first `n` elements" semantics documented in the method.

### Steps to reproduce (before this change)
```python
from datasets import Dataset
Dataset.from_dict({"a": [0, 1, 2]}).take(5)   # IndexError: Index 4 out of range for dataset of size 3.
```
`IterableDataset.take(5)` on the same data returns all 3 rows.

### Test plan
- Added `test_take` to `BaseDatasetTest` (runs both in-memory and on-disk) covering fewer / exact / more-than-available / zero.
- The `n > len` case fails on current `main` (`IndexError`) and passes with this change; existing behavior for `n <= len` is unchanged.

### AI disclosure
This fix was prepared with the assistance of an AI coding assistant; the bug was reproduced and the behavior verified before submitting.
Option for Anonymous Dataset link

### Feature request

Allow for anonymized viewing of datasets. For instance, something similar to [Anonymous GitHub](https://anonymous.4open.science/).

### Motivation

We generally publish our data through Hugging Face. This has worked out very well as it's both our repository and archive (thanks to the DOI feature!). However, we have an increasing challenge when it comes to sharing our datasets for paper (both conference and journal) submissions. Due to the need to share data anonymously, we can't use the Hugging Face URLs, but datasets tend to be too large for inclusion as a zip. Being able to have an anonymous link would be great since we can't be double-publishing the data.

### Your contribution

Sorry, I don't have a contribution to make to the implementation of this. Perhaps it would be possible to work off the [Anonymous GitHub](https://github.com/tdurieux/anonymous_github) code to generate something analogous with pointers to the data still on Hugging Face's servers (instead of the duplication of data required for the GitHub version)?

Labels: name: enhancement

I created this website for that particular purpose:

Website: https://anonymous-hf.up.railway.app

GitHub: https://github.com/MagedSaeed/anonymous-hf

Please give it a look and maybe a try if you like it.
Feature request: being able to disable chmod for flat permission filesystems

### Feature request

For flat permission file systems, `os.chmod` simply fails when trying to cache datasets - in some places such as: https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L4043-L4048 for arrow datasets. 
This seems to be optional to obtained desired behavior as simply commenting it out makes our workload work as expected. Hence the feature request would be to see if it would be possible to expose an API to disable `os.chmod` operations when users asks to do so (e.g. through an env variable?)

### Motivation

Being able to save properly datasets on flat permission filesystems

### Your contribution

Can test the feature if implemented

Labels: name: enhancement

Context: the operation fails on GCS FUSE mounts, and likely will on similar s3-backed file systems.

  File "/home/user/.venv/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3747, in _map_single                                                                                                                      
    os.chmod(cache_file_name, 0o666 & ~umask)                                                                                                                                                                                                    
PermissionError: [Errno 1] Operation not permitted: '/mnt/data/cache_ 55b1844ebd9dffff_00000_of_00001.arrow'
Fix crash on millisecond-scale started_at in hermes agent traces

## Summary

Parsing a hermes-format agent trace whose `started_at` is expressed in milliseconds crashes with `OSError: [Errno 22] Invalid argument`.

## The bug

`src/datasets/packaged_modules/json/json.py` normalizes `started_at` before feeding it to `datetime.fromtimestamp` (which expects **seconds**), using a magnitude check to detect whether the input is already in milliseconds:

```python
timestamp = ujson_loads(lines[i])["started_at"]
milliseconds = timestamp if timestamp <= 10_000_000_000 else timestamp * 1_000
sent_at = datetime.fromtimestamp(milliseconds, tz=timezone.utc)...
```

The large-magnitude branch (`timestamp > 10_000_000_000`, i.e. already milliseconds) should convert **down** to seconds by dividing by 1000 — instead it multiplies by 1000 again. For a real millisecond-scale `started_at` (a common convention, e.g. `Date.now()`-style timestamps), this produces a value representing a date roughly 56 million years in the future, and `datetime.fromtimestamp` raises `OSError: [Errno 22] Invalid argument` on Windows (`OverflowError`/`ValueError` on other platforms for the same reason), aborting the whole trace-loading path.

```python
>>> from datetime import datetime, timezone
>>> ts = 1_780_665_768_307  # same instant as the existing seconds-form test fixture, in ms
>>> ms = ts if ts <= 10_000_000_000 else ts * 1_000
>>> datetime.fromtimestamp(ms, tz=timezone.utc)
OSError: [Errno 22] Invalid argument
```

## The fix

Divide instead of multiply, and rename the variable to `seconds` (what it actually holds, and what `datetime.fromtimestamp` expects):

```python
timestamp = ujson_loads(lines[i])["started_at"]
seconds = timestamp if timestamp <= 10_000_000_000 else timestamp / 1_000
sent_at = datetime.fromtimestamp(seconds, tz=timezone.utc)...
```

## Tests

Adds `HERMES_SESSION_MS` — the existing `HERMES_SESSION` fixture with `started_at` re-expressed in milliseconds for the same real-world instant — and a new `hermes-ms` parametrized case in `test_json_generate_tables_with_agent_trace_metadata`, asserting the exact same `sent_at` output as the existing seconds-form `hermes` case.

Verified fail-before / pass-after against the real library (via `PYTHONPATH` pointed at this checkout, with `teich` installed so the `@require_teich`-gated tests actually run): the new `hermes-ms` case raises `OSError` before the fix and passes after; the two pre-existing hermes cases are unaffected either way. Full file: `36 passed`.

```
pytest tests/packaged_modules/test_json.py -k hermes
```

`ruff format --check` and `ruff check` are clean.
A bug of Dataset.to_json() function

### Describe the bug

When using the Dataset.to_json() function, an unexpected error occurs if the parameter is set to lines=False. The stored data should be in the form of a list, but it actually turns into multiple lists, which causes an error when reading the data again.
The reason is that to_json() writes to the file in several segments based on the batch size. This is not a problem when lines=True, but it is incorrect when lines=False, because writing in several times will produce multiple lists(when len(dataset) > batch_size).

### Steps to reproduce the bug

try this code:
```python
from datasets import load_dataset
import json

train_dataset = load_dataset("Anthropic/hh-rlhf", data_dir="harmless-base")["train"]
output_path = "./harmless-base_hftojs.json"
print(len(train_dataset))
train_dataset.to_json(output_path, lines=False, force_ascii=False, indent=2)

with open(output_path, encoding="utf-8") as f:
    data = json.loads(f.read())

```
it raise error: json.decoder.JSONDecodeError: Extra data: line 4003 column 1 (char 1373709)

Extra square brackets have appeared here:
<img width="265" alt="image" src="https://github.com/huggingface/datasets/assets/26499566/81492332-386d-42e8-88d1-b6d4ae3682cc">


### Expected behavior

The code runs normally.

### Environment info

datasets=2.20.0

Labels: name: bug

Thanks for reporting, @LinglingGreat.

I confirm this is a bug.

@albertvillanova I would like to take a shot at this if you aren't working on it currently. Let me know!

Found the bug — to_json(lines=False) was concatenating batch JSON arrays directly, producing [...][...]... which isn't valid. Fixed by stripping outer brackets and wrapping in a single array. PR ready.
Fix size_str(0) returning "Unknown size" instead of "0 bytes"

`size_str` uses `if not size_in_bytes:` to return `"Unknown size"`, which also catches `0`. But `0` is a valid size (0 bytes), and the docstring states that only `None` means "Unknown size". So a 0-byte size (e.g. an empty split) currently renders as "Unknown size":

```python
>>> from datasets.utils.py_utils import size_str
>>> size_str(0)
'Unknown size'   # expected '0 bytes'
```
This checks `size_in_bytes is None` instead. Added a regression test covering `None`, `0`, and non-zero sizes.
Preserve features after IterableDataset.add_column

## What

`IterableDataset.add_column` delegates to `map` but never propagates the schema. After the call, `dataset.features` is `None` even when the original dataset had a typed schema, silently breaking all downstream code that inspects features.

## Root cause

The implementation is a one-liner:
```python
return self.map(partial(add_column_fn, name=name, column=column), with_indices=True)
```
`map` without an explicit `features` argument does not infer schema from the mapper output.

## Fix

Follow the same pattern used by `rename_columns`: snapshot the existing features before the map, infer the new column's type from the column data via PyArrow (one line, pyarrow is already a required dependency), merge, and write back to `_info.features`.

```python
original_features = self._info.features.copy() if self._info.features else None
ds_iterable = self.map(...)
if original_features is not None:
    col_table = pa.table({name: column})
    new_features = original_features.copy()
    new_features[name] = Features.from_arrow_schema(col_table.schema)[name]
    ds_iterable._info.features = new_features
return ds_iterable
```

If the original dataset had no features schema (`features is None`), behavior is unchanged.

Fixes #5752.
fix(json): produce valid JSON when lines=False spans multiple batches (#7037)

## Summary

Fixes #7037.

`Dataset.to_json(lines=False)` raised `NotImplementedError` (and before the guard was added, silently produced _invalid_ JSON) whenever `batch_size < num_rows` — i.e., the default case for any dataset larger than `DEFAULT_MAX_BATCH_SIZE` (1000 rows).

**Root cause:** Each batch was serialised to its own JSON array (`[{...}, ...]`) and written back-to-back in the file, producing e.g. `[{...}][{...}]` — syntactically invalid.

**Fix (two changes in `src/datasets/io/json.py`):**

1. **`write()`** — Narrow the `NotImplementedError` guard to dict-producing orientations (`split`, `index`, `columns`, `table`) only. Array-producing orientations (`records`, `values`) can now be batched safely.

2. **`_write()`** — When `lines=False` and `orient in ("records", "values")`, strip the outer `[` / `]` from each batch's bytes and emit a single wrapping `[...]`, streaming with no extra memory overhead. Dict-producing orientations fall through to the unchanged write-as-is path.

As a bonus, the single-proc and multiprocessing iteration loops are unified via `contextlib.nullcontext`, removing ~10 lines of duplication.

**Verified:**

```python
import io, json
from datasets import Dataset
from datasets.io.json import JsonDatasetWriter

ds = Dataset.from_dict({"id": list(range(25)), "value": [f"v{i}" for i in range(25)]})
buf = io.BytesIO()
JsonDatasetWriter(ds, buf, batch_size=10, lines=False).write()  # 3 batches
buf.seek(0)
result = json.loads(buf.read())   # previously raised NotImplementedError / produced invalid JSON
assert len(result) == 25          # single valid JSON array with all rows
```

## Test plan

- [x] New parametrised test `test_dataset_to_json_lines_false_batched` covers `orient="records"` and `orient="values"` with `batch_size < num_rows`
- [x] Full `TestJsonDatasetWriter` suite passes (20 tests; 3 pre-existing errors for `shared_datadir` fixture from missing `pytest-datadir` plugin, unrelated to this change)
Cast multi-dimensional numpy arrays when inferring IterableDataset features

## Summary

`IterableDataset._resolve_features()` crashes when a `map` function yields multi-dimensional numpy arrays:

```python
from datasets import Dataset
import numpy as np
ds = Dataset.from_dict({"a": [[[1, 2, 3], [1, 2, 3]]]}).to_iterable_dataset().map(lambda x: {"a": [np.array(x["a"])]})
ds._resolve_features()
# pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values
```

Feature inference goes `_resolve_features()` -> `_infer_features_from_batch(...)` -> `pa.Table.from_pydict(batch)`, and PyArrow's `from_pydict` only accepts 1-dimensional array values, so any 2D+ numpy array in a column crashes it.

Every other place in `iterable_dataset.py` that builds an Arrow table from raw examples first routes the data through `cast_to_python_objects(..., only_1d_for_numpy=True)` (lines 173 and 184), which recursively turns multi-dimensional numpy/tensor values into nested lists of 1-D arrays that PyArrow accepts. The non-streaming `Dataset`/`ArrowWriter` path does the same. `_infer_features_from_batch` was simply missing that cast.

## Fix

Route the batch through `cast_to_python_objects(batch, only_1d_for_numpy=True)` before `from_pydict`. The helper is already imported in this module, and it is a no-op for ordinary Python data, so plain batches infer identically.

Fixes #7100.

## Test

`tests/test_iterable_dataset.py::test_iterable_dataset_resolve_features_from_multidim_numpy` resolves the features of an `IterableDataset` whose `map` yields multi-dimensional numpy arrays. It raises `ArrowInvalid` before the fix and passes after (resolving to a nested `List(...)`).
Fix: Resolve Sequence type hints and add from_jsonl alias (#5354)

Fixes #5354

## Proposed Changes
- Updates features typing to use Sequence instead of List/Tuple in from_json and from_parquet parsing functions.
- Adds from_jsonl as an explicit wrapper to from_json for dataset creation, providing API symmetry with to_jsonl.
- Includes dedicated tests for the new from_jsonl alias in all standard loaders.
feat: use `ListLike` instead of `list` in path type hints

---
## Description

Changed type annotations from `list[PathLike]` to `ListLike[PathLike]` in
`from_csv`, `from_json`, `from_parquet`, and `from_text`.

`list` type hints are invariant — passing a `tuple` of paths fails mypy even
though it works fine at runtime. `ListLike` (defined in
`src/datasets/utils/typing.py`) accepts both `list` and `tuple`, and is already
used elsewhere in this file (see `_getitem`), so this follows an existing
project convention rather than introducing a new type alias.

## Changes

- `src/datasets/arrow_dataset.py`: changed 4 parameter type hints
  (`from_csv`, `from_json`, `from_parquet`, `from_text`) and updated their
  docstrings to match (`path-like or list of path-like` → `path-like or
  Sequence of path-like`).

## Verification

- Type-annotation-only change — no runtime behavior difference.
  `path_or_paths` is passed through unchanged to the underlying
  `*DatasetReader` classes.
- Ran the existing test suite: 151 passed, 0 failures related to this change
  (5 unrelated failures are pre-existing, caused by a missing
  `pytest-datadir` fixture in compression-writer tests).
- Ran mypy before and after this change and diffed the output. Mypy still
  reports a type mismatch between these methods and the internal
  `*DatasetReader` constructors (`CsvDatasetReader`, `JsonDatasetReader`,
  `ParquetDatasetReader`, `TextDatasetReader`), which are typed to accept
  `list | dict` only. Confirmed via diff that this mismatch is **pre-existing
  on `main`** (it appeared even when using `Sequence_[PathLike]` before
  switching to `ListLike`) — it is out of scope for this PR and would need a
  separate fix to the `io/*.py` reader constructor signatures.

## Related Issue

Closes #5354

## AI Disclosure

AI-assisted (Hermes Agent) — reviewed and tested manually by the author.
---
Feature Proposal: Add FunASR Chinese/Multilingual ASR Benchmark Datasets

<!-- funasr-ops:accuracy-note-20260714 -->
> [!NOTE]
> **License and capability clarification (2026-07-14):** FunASR is a toolkit, not a single checkpoint. The [FunASR](https://github.com/modelscope/FunASR#license) and [SenseVoice](https://github.com/FunAudioLLM/SenseVoice#license) repository source code is MIT; model weights follow each model card. [SenseVoiceSmall](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) supports Chinese, Cantonese, English, Japanese, and Korean, and its weights use the linked FunASR Model Open Source License Agreement. [Fun-ASR-Nano-2512](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512) is Apache-2.0. Language coverage, punctuation, and performance depend on the selected model and runtime configuration.

The HuggingFace datasets library supports audio datasets, but lacks curated speech recognition benchmark datasets for non-English languages, especially Chinese. FunASR (17.8K+ stars, https://github.com/modelscope/FunASR) provides production-grade ASR models with extensive multilingual support:

- **SenseVoice**: Ultra-fast multilingual ASR (50+ languages, strong CJK + Cantonese)
- **Paraformer**: Production-grade Chinese ASR with timestamps and punctuation

Would it be valuable to add FunASR benchmark datasets (e.g., Chinese speech recognition test sets, multilingual ASR evaluation data) to the HuggingFace datasets hub? This would benefit the broader ASR research community and provide standardized evaluation benchmarks beyond the current Whisper-centric datasets.

FunASR models are also available on the HuggingFace model hub (https://huggingface.co/FunAudioLLM), making dataset-model pairing seamless.

Would adding FunASR evaluation datasets be useful?
fix: replace list/List with Sequence in function parameter annotations

Fixes #5354

Replaces `list[T]` with `Sequence[T]` in function parameter type annotations across `arrow_dataset.py`, `dataset_dict.py`, and `iterable_dataset.py`.

The built-in `list` is invariant in its type parameter, so passing a `List[str]` where `list[PathLike]` is expected causes a mypy error.  `Sequence` is covariant and accepts both `list` and its subtypes.

### Changes

- Added `Sequence`/ `Sequence_` import from `collections.abc`
- Replaced `list[...]` with `Sequence[...]` in all function parameter annotations
- Updated corresponding docstring references
- Did **not** change return type annotations or local variable annotations (only parameters that accept input sequences)