noamaanMulla-03/datasets-issues huggingface dataset:
Typo in iterable_dataset.py

### Describe the bug

`streaming=True**kwargs,` should be `streaming=True, **kwargs,`
https://github.com/huggingface/datasets/blob/81027be09d5cd9f06a6d64ef1a8a3e9ebd0f86fd/src/datasets/iterable_dataset.py#L2950

### Steps to reproduce the bug

```
from Datasets import Dataset
IterableDataset.from_csv("file.csv")
```

### Expected behavior

Load the csv file as an iterable dataset.

### Environment info

- `datasets` version: 4.6.1
- Platform: Linux-6.14.0-37-generic-x86_64-with-glibc2.39
- Python version: 3.13.11
- `huggingface_hub` version: 1.5.0
- PyArrow version: 23.0.1
- Pandas version: 3.0.1
- `fsspec` version: 2026.2.0
Fix index out of bound error with original_shard_lengths.

I have gotten the following error

```
    original_shard_lengths[original_shard_id] += 1
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
IndexError: list index out of range
```

Not sure what causes it, but this fixes the error. This may not be the proper fix for the root cause though.


this was fixed in https://github.com/huggingface/datasets/pull/8023, sorry I didn't see your PR and took the liberty of fixing this bug as part of this improvement for `from_generator`
IterableDataset.filter() chaining fails due to features being lost when is_typed=True

### Describe the bug

Chaining multiple .filter() calls on an IterableDataset raises a TypeError: 'NoneType' object is not a mapping on the second .filter() call.

In `IterableDataset.filter()`, the internal `ex_iterable` is wrapped with FormattedExamplesIterable:

```python
ex_iterable = FormattedExamplesIterable(
    ex_iterable,
    formatting=self._formatting,
    features=None if ex_iterable.is_typed else self._info.features,
    token_per_repo_id=self._token_per_repo_id,
)
```

After the first .filter(), the resulting ex_iterable becomes typed (is_typed=True) because a ===MASK=== column is added. On the second .filter(), since is_typed=True, features=None is passed to FormattedExamplesIterable. This causes .features to return None, and FilteredExamplesIterable.__init__ fails when trying to unpack it.

The fix should preserve the existing features when is_typed=True, e.g.:

```python
features=ex_iterable.features if ex_iterable.is_typed else self._info.features
```

### Steps to reproduce the bug

```python
from datasets import load_dataset

ds = load_dataset("bigcode/the-stack-v2", "Scala", split="train", streaming=True)
ds = ds.filter(lambda row: row["src_encoding"] == "UTF-8")
ds = ds.filter(lambda row: row["branch_name"] == "refs/heads/master")  # TypeError here
```

## Error traceback
```
File ".../datasets/iterable_dataset.py", line 2975, in filter
    ex_iterable = FilteredExamplesIterable(
File ".../datasets/iterable_dataset.py", line 1606, in __init__
    features = Features({**ex_iterable.features, self.mask_column_name: Value("bool")})
TypeError: 'NoneType' object is not a mapping
```

### Expected behavior

The error should not raised in this context.

### Environment info

datasets: 4.5.0
Fix IterableDataset features being set to None on chained filter (Fixes #8037)

This PR implements the solution suggested in #8037.

It updates the features argument in IterableDataset.filter() to preserve existing features when ex_iterable.is_typed is True. This prevents the TypeError: 'NoneType' object is not a mapping from being raised when chaining multiple .filter() calls.


Thanks for the PR ! The original issue poster actually opened the same PR https://github.com/huggingface/datasets/pull/8053 but with additional tests, so I took the liberty of merging theirs - I hope you don't mind
fix: handle nested null types in feature alignment for multi-proc map

When using `Dataset.map()` with `num_proc > 1`, shards containing only empty lists get their features inferred as `List(Value('null'))`, which is incompatible with `List(Struct({...}))` from shards with actual data.

The existing `_check_if_features_can_be_aligned` and `_align_features` only handled top-level `Value('null')` compatibility. This commit adds a `_is_null_feature()` helper that recursively checks for null types inside container features (`List, LargeList, Sequence`), and uses it in both alignment functions.

Fixes #8046


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8047). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`Dataset.map()` with `num_proc > 1` fails when some shards produce `List(Value('null'))` (empty lists)

### Describe the bug

When using `Dataset.map()` with `num_proc > 1`, if the map function produces a list column where some shards contain only empty lists `[]` and other shards contain non-empty lists of structs, the feature type inference is inconsistent across shards:

- Shards with only empty lists → `List(Value('null'))`
- Shards with non-empty lists → `List(Struct({...}))`

During shard concatenation, `_check_if_features_can_be_aligned` raises a `ValueError` because it only handles top-level `Value("null")` compatibility but does **not** handle nested null types like `List(Value('null'))`.

Setting `num_proc=1` works correctly because all data is processed in a single shard.

### Steps to reproduce the bug

```python
import datasets
print(datasets.__version__)  # 4.6.1

data = {
    "id": [f"item_{i}" for i in range(8)],
    "labels": [
        [{"type": "A", "word": "x", "start": 0, "end": 5}],  # only first item has data
        [], [], [], [], [], [], []  # rest are empty
    ]
}
ds = datasets.Dataset.from_dict(data)

# Works fine with num_proc=1
result = ds.map(lambda x: {"label": x["labels"]}, num_proc=1)
print(result.features)  # OK

# Fails with num_proc=2
result = ds.map(lambda x: {"label": x["labels"]}, num_proc=2)
# ValueError: The features can't be aligned because the key label of features
# ... has unexpected type - List(Value('null'))
# (expected either List({'end': Value('int64'), ...}) or Value("null").
```


### Expected behavior

`List(Value('null'))` should be treated as compatible with any `List(...)` type during feature alignment, similar to how `Value('null')` is already treated as compatible with any other type.

### Environment info

- `datasets` version: 4.6.1 (also reproduced on 4.1.1)
- Python version: 3.12.0
- OS: Linux



### Root cause

In `src/datasets/features/features.py`, both `_check_if_features_can_be_aligned` and `_align_features` use:

```python
isinstance(v, Value) and v.dtype == "null"
```

to check for null compatibility. However, `List(Value('null'))` is **not** an instance of `Value` — it's an instance of `Sequence` (via `List`). So this check misses any container type wrapping a null, such as `List(Value('null'))`, `Sequence(Value('null'))`, or `LargeList(Value('null'))`.

### Suggested fix

Add a recursive null check function and use it in place of the current top-level-only check:

```python
def _is_null_feature(feature) -> bool:
    """Recursively check if a feature is a null type."""
    if isinstance(feature, Value) and feature.dtype == "null":
        return True
    if isinstance(feature, (Sequence, LargeList)) and hasattr(feature, "feature"):
        return _is_null_feature(feature.feature)
    return False
```

Then replace all occurrences of `isinstance(v, Value) and v.dtype == "null"` with `_is_null_feature(v)` in both `_check_if_features_can_be_aligned` and `_align_features`.
Fix unstable tokenizer fingerprinting (enables map cache reuse)

Fix unstable dataset fingerprinting when hashing `PreTrainedTokenizerFast`.

Some tokenizers backed by `tokenizers.Tokenizer` mutate runtime settings (padding/truncation) when called, which can change the serialized state and make dataset fingerprints unstable. That prevents `.map(load_from_cache_file=True)` from reusing cache files.

Fix: when hashing, temporarily disable backend padding/truncation so runtime settings don’t affect the fingerprint, then restore the original settings.

Includes a regression test showing `Hasher.hash(tokenizer)` stays stable after calling the tokenizer.


Hi! It looks like the GitHub Actions check suites for this PR are in `action_required` (no workflows actually ran). This is usually due to fork workflow approval.

Could a maintainer please approve/run the workflows so CI can execute? Happy to address anything CI flags once it runs.

Quick ping on this one. Happy to update anything if you want changes.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7982). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @itazap and @ArthurZucker for viz: this is the trick we use to hash a tokenizer

this way the hash doesn't change if the state of the tokenizer changed
Write IterableDataset to parquet incrementally instead of materializing entire shard in memory

`IterableDataset.to_parquet()` materialized the entire shard into memory via
`pa.concat_tables(list(...))` before writing. For datasets with large shards and multiprocessing (push_to_hub with num_proc > 0) the memory requirements can be really large.

This PR writes parquet incrementally by iterating Arrow batches and writing each
directly to `pq.ParquetWriter`, reducing the memory requirement from shard size to something around row group size (with lots of caveats).

**Note**

A thing to note is that this code path is only executed when `features` are set on the `IterableDataset`, in f.ex. `map` or the dataset is `cast` before `push_to_hub`.

Changes:
- Extend `ParquetDatasetWriter` to accept both `Dataset` and `IterableDataset`.
- `IterableDataset.to_parquet()` now delegates directly to `ParquetDatasetWriter`
- Use `pq.ParquetWriter` as a context manager for proper cleanup on errors
- Falls back to materialization only when `features` is `None` (schema required upfront)

That being said, I'm not entirely convinced that extending the ParquetDatasetWriter to support IterableDataset is the correct approach or even if there are other footguns related to CDC or other stuff.


I've tested this change along with the changes from my other PRs (https://github.com/HaukurPall/datasets/tree/all-fixes) and this change **dramatically** reduces the memory requirement per process.

Context:

I'm working with audio datasets, the audio is embedded in the parquet. When loading a 1GB shard with embedded mp3 audio, decoding the audio and writing it back as wav, the shard size increases to around 9GB. I run this processing with num_proc > 1, so the total memory required is large. With this change, the memory required per process drops from ~25GB to ~4GB for this use-case.

@lhoestq any feedback?
Fix silent data loss in push_to_hub when num_proc > num_shards

## Summary

When `num_proc` exceeds the number of output shards in `Dataset.push_to_hub`, workers
without assigned output shards silently drop their data. The metadata still reports the
correct dataset length (using `len(self[split])`), masking the data loss.

This fix caps `num_jobs` at `num_shards` in `_push_parquet_shards_to_hub`, matching
what the streaming path already does.

## Reproduction

1. Have a dataset with a small number of shards (e.g. 2)
2. Call `push_to_hub` with `num_proc > num_shards` (e.g. 6)
3. Only a fraction of the samples are actually written to the hub
Improve error message for invalid data_files pattern format

This PR improves the error message raised in sanitize_patterns()
when an invalid data_files entry is provided.

The new message clearly explains the expected dictionary format:
{'split': <split_name>, 'path': <path_or_list_of_paths>}

This helps users debug incorrect dataset configurations more easily.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8060). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add `IterableDataset.reshard()`

To increase the number of shards of a dataset, you can use [`IterableDataset.reshard`]:

```py
>>> dataset
IterableDataset({
    features: ['label', 'title', 'content'],
    num_shards: 4
})
>>> dataset.reshard()
IterableDataset({
    features: ['label', 'title', 'content'],
    num_shards: 3600
})
```

The resharding mechanism depends on the dataset file format.
For example for Parquet, it reshards using row groups instead of having one file per shard.

We can implement other formats later (e.g. JSON Lines, CSV can be split by recovering line boundaries from arbitrary locations)

Other details:
* fixed concatenate after shuffling, now it correctly shuffles the shards: close https://github.com/huggingface/datasets/issues/7196
* fixed interleave after split_by_node: close https://github.com/huggingface/datasets/issues/7868
* changed a bit how the effective seed works in multi node/worker/epoch situations

related to https://github.com/huggingface/datasets/issues/7917


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7992). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix schema enforcement in streaming _convert_to_arrow

When the iter_arrow chain is broken (e.g. after .map() without with_format("arrow")), _convert_to_arrow falls back to pa.Table.from_pylist() which infers types from values. If early batches contain None or empty lists, Arrow infers null/list<null> types that conflict with later batches containing real data, causing ArrowInvalid schema mismatch errors.

Add an optional features parameter to _convert_to_arrow that applies cast_table_to_features to each produced table, mirroring what ArrowWriter.write_table does in the map-style path.


I'm going to close this PR as I have a more complete branch which fixes feature handling in the streaming case. The `features` handling in the streaming case is btw completely broken at the moment, it just happens to work in some cases. I will add more test cases to show it in a later PR.
typing: accept sequences for Dataset file loaders

## Summary
- update `Dataset.from_csv`, `from_json`, `from_parquet`, and `from_text` type hints from `list[PathLike]` to `Sequence[PathLike]`
- normalize non-string sequences to lists before passing paths to readers
- add path-type tests to cover tuple input for all four loaders

## Why
`list` is invariant for static type checkers, so `list[str]` can be rejected against `list[PathLike]` even though `str` itself is valid path input. `Sequence[PathLike]` is covariant and better matches real usage.

## Validation
- `uv run --python 3.11 --with-editable . --with pytest --with setuptools --with absl-py -m pytest tests/test_arrow_dataset.py::test_dataset_from_csv_path_type tests/test_arrow_dataset.py::test_dataset_from_json_path_type tests/test_arrow_dataset.py::test_dataset_from_parquet_path_type tests/test_arrow_dataset.py::test_dataset_from_text_path_type -q`
- `uv run --python 3.11 --with ruff --with setuptools ruff check src/datasets/arrow_dataset.py tests/test_arrow_dataset.py`

Fixes #5354
Fix Dataset.map writer initialization when early examples return None

## Summary

Fixes #7990 

This PR fixes a bug in `Dataset.map()` where the writer initialization was incorrectly tied to the index being 0, causing crashes when the map function returns `None` for the first few examples and later returns a dict.

### Changes

- **Non-batched mode** (line 3676): Changed from `if i == 0:` to `if writer is None:`
- **Batched mode** (line 3701): Changed from `if i and i[0] == 0:` to `if writer is None:`

### Why This Fix Works

The original code assumed that `update_data` would always be determined by the time the first example (i=0) was processed. However, `update_data` is set lazily after processing each example - it becomes `True` when the function first returns a non-None value.

If a function returns `None` for early examples and a dict for later ones:
1. At i=0, the function returns `None`, so `update_data` remains `None`  
2. Writer is NOT initialized (because we're not updating data)
3. At i=2, the function returns a dict, so `update_data` becomes `True`
4. **Old code**: Tries to use `writer` (still None) because i != 0 → crash
5. **New code**: Checks `if writer is None` and initializes it → works correctly

### Test Plan

The fix can be verified with this minimal test case from the issue:

```python
from datasets import Dataset

ds = Dataset.from_dict({"x": [1, 2, 3]})

def fn(example, idx):
    if idx < 2:
        return None
    return {"x": [example["x"] * 10]}

# Should work without errors
result = list(ds.map(fn, with_indices=True))
print(result)  # [{'x': 1}, {'x': 2}, {'x': [30]}]
```

**Before this fix**: Crashes with `AttributeError: 'NoneType' object has no attribute 'write'`  
**After this fix**: Works correctly

### Related

This fix ensures the writer is initialized the first time a non-None value is returned, regardless of which example index that occurs at. This makes the code more robust to different map function behaviors.


Friendly ping - any chance someone could take a look at this when they get a chance? Happy to make any changes if needed.

ping - this fixes a real edge case with Dataset.map when early examples return None. would love a review

friendly bump - happy to make any changes if needed!
fix null filling in missing jsonl columns

fix regression from #8027 reported [internally](https://huggingface.slack.com/archives/C04BP5S7858/p1773661170461759) cc @gary149 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8069). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Read (and write) from HF Storage Buckets: load raw data, process and save to Dataset Repos

TODO:
- [x] load_dataset("csv", data_files=["hf://buckets/..."])
- [x] load_dataset("buckets/...")
- [x] Dataset.to_csv("hf://buckets/..."), to_json, to_parquet, etc.
- [x] DatasetDict.push_to_hub("buckets/...")
- [x] Dataset.push_to_hub("buckets/...")
- [x] DatasetDict.push_to_hub("buckets/...")
- [x] IterableDataset.push_to_hub("buckets/...")
- [x] IterableDatasetDict.push_to_hub("buckets/...")
- [x] tests
- [x] docs

e.g.

raw data in bucket to dataset repo

```python
from datasets import load_dataset

ds = load_dataset("json", data_files=["hf://buckets/..."])
ds = ds.map(...).filter(...)
ds.push_to_hub("username/dataset_name")
```


(also taking the opportunity to factorize the code a bit)


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8064). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: avoid duplicate kwargs collisions in load_dataset_builder

## Summary
- avoid duplicate keyword collisions when `config_kwargs` overlap with module-provided `builder_kwargs` in `load_dataset_builder`
- this fixes cases like passing `base_path=...` where `base_path` is already present in `builder_kwargs`
- add a regression test using a local dataset directory to ensure overriding kwargs no longer raises

## Reproduction
Before this change:
```python
from datasets import load_dataset_builder
load_dataset_builder("/path/to/local/data", base_path="/path/to/local/data")
# TypeError: ... got multiple values for keyword argument 'base_path'
```

## Validation
- `uv run --python 3.11 --with-editable . --with pytest --with setuptools -m pytest tests/test_load.py::test_load_dataset_builder_config_kwargs_can_override_builder_kwargs tests/test_load.py::test_load_dataset_builder_for_absolute_data_dir -q`
- `uv run --python 3.11 --with ruff --with setuptools ruff check src/datasets/load.py tests/test_load.py`

Fixes #4910



I believe this was fixed in https://github.com/huggingface/datasets/pull/8068
Fix formatted iter arrow double yield

I (+Claude) noticed this when reading the code. I did not encounter this bug (I think) so I can't say much more about it. Perhaps the second iteration will immediately stop so the bug is harmless, but to be safe I think a return there is safer.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8063). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Guard against duplicate builder_kwargs/config_kwargs in load_dataset_…

…builder (#4910 )

### What does this PR do?

Fixes edge case in `load_dataset_builder` by raising a `TypeError` if the same key exists in both `builder_kwargs` and `config_kwargs`.

### Implementation details

- Added a guard clause in `load_dataset_builder` to detect duplicate keys between `builder_kwargs` and `config_kwargs`
 
- Wrote a unit test in `tests/test_load_duplicate_keys.py` to verify the exception is raised correctly 

### Fixes

Closes #4910 

### Reviewers

@zach-huggingface
@SunMarc 

Would appreciate your review if you have time - thanks!


This PR fixes the duplicate-kwargs edge case and includes a unit test. Would appreciate a review when you have a moment.

@zach-huggingface
@SunMarc
Unwanted column padding in nested lists of dicts

```python
from datasets import Dataset

dataset = Dataset.from_dict({
    "messages": [
        [
            {"a": "...",},
            {"b": "...",},
        ],
    ]
})
print(dataset[0])
```

What I get:
```
{'messages': [{'a': '...', 'b': None}, {'a': None, 'b': '...'}]}
```

What I want:

```
{'messages': [{'a': '...'}, {'b': '...'}]}
```

Is there an easy way to automatically remove these auto-filled null/none values?

If not, I probably need a recursive none exclusion function, don't I?

Datasets 3.6.0


Answer from @lhoestq:

> No
> This is because Arrow and Parquet a columnar format: they require a fixed type for each column. So if you have nested dicts, each item should have the same subfields

The way around I found is the handle it after sampling with this function:

```python
def remove_padding(example):
    if isinstance(example, list):
        return [remove_padding(value) if isinstance(value, (dict, list)) else value for value in example]
    elif isinstance(example, Mapping):
        return {
            key: remove_padding(value) if isinstance(value, (dict, list)) else value
            for key, value in example.items()
            if value is not None
        }
    else:
        raise TypeError("Input must be a list or a dictionary.")

# Example:
example = next(iter(dataset))
example = remove_padding(example)
```

v4.8 introduces on_mixed_types="use_json" that you can pass to Dataset.from_dict() to fix the issue :)
Fix torchcodec audio decoding to respect 'num_channels'

Fixes torchcodec audio decoding when `num_channels` is set on `Audio`.

Before this change, `AudioDecoder["array"]` reduced multi-channel audio to mono by averaging channels, so the requested channel behavior was not respected.

With this PR:
- multi-channel decoded arrays are preserved by default;
- mono output is returned only when `num_channels == 1` is explicitly requested.

#### Previous behavior
`ds_stereo = ds.cast_column("audio", Audio(num_channels={...}))`  - None, 1, 2

Original file: '(16000, 2)' - stereo ✓  
'Audio(num_channels=None)': '(16000,)' - mono ✗  
'Audio(num_channels=2)': '(16000,)' - mono ✗  
'Audio(num_channels=1)': '(16000,)' - mono ✓

#### New behavior
'num_channels=None' preserves the original number of channels from the source file.
'num_channels=2' preserves/converts to stereo output with shape '(2, num_samples)'.
'num_channels=1' downmixes to mono with shape '(num_samples,)'.

#### Results
**Original file shape** (via soundfile): (16000, 2)
HF datasets shape with num_channels=None: (2, 16000)
HF datasets shape with num_channels=1: (16000,)
HF datasets shape with num_channels=2: (2, 16000)


Fixes #8005.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8028). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.