noamaanMulla-03/datasets-issues huggingface dataset:
Streaming the Pile: Missing Files

### Describe the bug

The pile does not stream, a "File not Found error" is returned. It looks like the Pile's files have been moved.

### Steps to reproduce the bug

To reproduce run the following code:

```
from datasets import load_dataset

dataset = load_dataset('EleutherAI/pile', 'en', split='train', streaming=True)
next(iter(dataset))
```

I get the following error:
`FileNotFoundError: https://the-eye.eu/public/AI/pile/train/00.jsonl.zst`

### Expected behavior

Return the data in a stream.

### Environment info

- `datasets` version: 2.12.0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.11.5
- Huggingface_hub version: 0.15.1
- PyArrow version: 11.0.0
- Pandas version: 2.0.3


Hello @FelixLabelle,

As you can see in the Community tab of the corresponding dataset, it is a known issue: https://huggingface.co/datasets/EleutherAI/pile/discussions/15

The data has been taken down due to reported copyright infringement.

Feel free to continue the discussion there.

Generally, I still experience this type of issue when using streaming with datasets. This is not even restricted to The Pile.
Fix tilde expansion for local data_files paths

Summary
- expand `~` before classifying local `data_files` patterns
- add a regression test covering `resolve_pattern(os.path.join("~", ...), ...)`

Impact
- `load_dataset(..., data_files="~/path/to/file.parquet")` currently treats `~` as a literal relative directory under the working tree instead of the user's home directory
- this brings `data_files` in line with the existing `data_dir` handling and fixes issue #6212

Testing
- `python -m pytest tests/test_data_files.py`
- `make quality`
- manual repro: `load_dataset("parquet", data_files="~/.codex-tmp-datasets-tilde-repro/sample.parquet")`

Fixes #6212


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8153). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix iterable map resume state


Fixes: https://github.com/huggingface/datasets/issues/8145
## Root Cause
  `FormattedExamplesIterable.__iter__` ignored `force_convert_to_python` and used Arrow batch iteration, advancing the
  underlying state by a full Arrow chunk before yielding Python examples.

  ## Fix
  Only use Arrow batch iteration when `force_convert_to_python` is false:
  `if self.ex_iterable.iter_arrow and not self.force_convert_to_python:`

  ## Tests
  - [x] Verified the issue reproduction now resumes at `[200, 201, 202, 203, 204]`.



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8147). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I took the liberty of adjusting the fix to still use the iter_arrow in `FormattedExamplesIterable.__iter__`, I hope you don't mind. Let's see if the CI is green now

hey @lhoestq I checked the source of the error for the failing integration test. Seems like all 5 failed checks stemmed from a 401 unauthorized error and does not seem to be related to the code changes. Is it possible for you to briefly check? Thank you so much!
`.map()` on a streaming IterableDataset silently skips examples after load_state_dict

### Describe the bug

After `ds.map(fn)` on a streaming IterableDataset, saving state_dict() and loading it into a fresh dataset does not resume at the correct position. It jumps forward by up to one Arrow read-chunk (~1000 rows).

### Steps to reproduce the bug

```python
from datasets import Dataset

ds1 = Dataset.from_dict({"id": list(range(5000))}).to_iterable_dataset(num_shards=1).map(
    lambda x: {**x, "tag": "X"}
)
ds2 = Dataset.from_dict({"id": list(range(5000))}).to_iterable_dataset(num_shards=1).map(
    lambda x: {**x, "tag": "X"}
)

it = iter(ds1)
for _ in range(200):
    next(it)
state = ds1.state_dict()
expected = [next(it)["id"] for _ in range(5)]

ds2.load_state_dict(state)
it2 = iter(ds2)
actual = [next(it2)["id"] for _ in range(5)]

print(f"expected={expected}")
print(f"actual  ={actual}")
assert expected == actual, f"resume skipped forward by {actual[0] - expected[0]} rows"
```

```
expected=[200, 201, 202, 203, 204]
actual  =[1000, 1001, 1002, 1003, 1004]
AssertionError: resume skipped forward by 800 rows
```

### Expected behavior

Expected behaviour:
```
expected=[200, 201, 202, 203, 204]
actual  =[200, 201, 202, 203, 204]
```

### Environment info

```
- `datasets` version: 4.8.4
- Platform: Linux-6.8.0-85-generic-x86_64-with-glibc2.39
- Python version: 3.12.12
- `huggingface_hub` version: 1.11.0
- PyArrow version: 23.0.1
- Pandas version: 3.0.2
- `fsspec` version: 2026.2.0
```


I think this line here is causing issues: 
https://github.com/huggingface/datasets/blob/2724a653e7582077be647d3fdfe8cd1fac27b85f/src/datasets/iterable_dataset.py#L2225
It ignored force_convert_to_python, so it consumed a full Arrow chunk.
I think it can be fixed if you add `self.force_convert_to_python` to the condition it forces formatting to iterate row-by-row as Python examples instead of using Arrow batches, which prevents state tracking from jumping ahead by a whole Arrow chunk.
Fix: decode JSON type before to_list or to_dict is called

**Motivation**:
There is a change in the approach to decoding JSON types, added in https://github.com/huggingface/datasets/commit/d560b58e1f1ec96591a95a1780ee118f5eea6c74. Since 4.7.0, to_list() returns raw JSON strings for columns stored as Json(), while direct access via getitem returns dicts.

In versions prior to 4.7.0, to_list() returned dicts.

**Added context**:
- Closes https://github.com/huggingface/datasets/issues/8131
- On it, the maintainer suggested taking a similar approach to https://github.com/huggingface/datasets/pull/8122

This PR contains:
- An update to the to_list and to_dict functions for the arrow dataset inspired by a (and using the utils developed for) [similar fix](https://github.com/huggingface/datasets/pull/8122) to another problem caused by the addition of the JSON type.
- Unit tests specifying the intended outcome (which was how dictionaries were processed or handled prior to v4.7.0).

**Additional questions/considerations**:
- This PR restore behaviour prior to the breaking change introduced by the addition of the JSON datatype in v4.7.0. However, does mean we now decode one data type (JSON) and not others. We could decide to decode elements of the list in the same way _get_item does but this would be a big change (i.e memory concerns with images and audio etc etc)
- I implemented the PR with a lazy import - just checking that's in line with the repo's design/ vibe?


@lhoestq Could this be reviewed and merged if it's a suitable solution?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8137). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

`datasets` 4.8.5 is out and includes the fix ! :)
The "all" split breaks streaming

## Describe the bug

Not sure if this is a bug or just the way streaming works, but setting `streaming=True` did not work when setting `split="all"`

## Steps to reproduce the bug
The following works:
```python
ds = load_dataset('super_glue', 'wsc.fixed', split='all')
```
The following throws `ValueError: Bad split: all. Available splits: ['train', 'validation', 'test']`:

```python
ds = load_dataset('super_glue', 'wsc.fixed', split='all', streaming=True)
```

## Expected results
An iterator over all splits.

## Actual results
I had to do the following to achieve the desired result:
```python
from itertools import chain
ds = load_dataset('super_glue', 'wsc.fixed', streaming=True)
it =  chain.from_iterable(ds.values())
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-4.15.0-176-generic-x86_64-with-glibc2.31
- Python version: 3.10.5
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Labels: name: bug

Thanks for reporting @cakiki.

Yes, this is a bug. We are investigating it.

@albertvillanova Nice! Let me know if it's something I can fix my self; would love to contribtue!

@cakiki I was working on this but if you would like to contribute, go ahead. I will close my PR. ;)

For the moment I just pushed the test (to see if it impacts other tests).

It impacted the test `test_generator_based_download_and_prepare` and I have fixed this.

So that you can copy the test I implemented in my PR and then implement a fix for this issue that passes the test `tests/test_builder.py::test_builder_as_streaming_dataset`.

Hi @cakiki are you still interested in working on this? Are you planning to open a PR?

Hi @albertvillanova ! Sorry it took so long; I wanted to spend this weekend working on it.
Include all test files in sdist via MANIFEST.in

Fixes #8159.

## What's wrong

The sdist on PyPI for `datasets==4.8.5` only ships **37 of the 95** files under `tests/`. Without an explicit `MANIFEST.in`, setuptools' default sdist behavior includes only top-level `tests/test_*.py` files and drops everything else.

Files missing from the current sdist:

- `tests/conftest.py`
- `tests/_test_patching.py`
- everything under `tests/commands/`, `tests/features/`, `tests/fixtures/`, `tests/io/`, `tests/packaged_modules/`, `tests/distributed_scripts/`
- the audio / image / PDF / JSON test-data files under `tests/features/data/` and `tests/io/data/`

Anyone installing from sdist (distros, mirrors, air-gapped builds) cannot run the test suite — `pytest` fails to even collect because `tests/conftest.py` is missing the `pytest_plugins` configuration that pulls in `tests.fixtures.*`.

## Fix

Add a minimal `MANIFEST.in` that grafts the whole `tests/` directory and excludes bytecode.

```
graft tests
global-exclude __pycache__
global-exclude *.py[cod]
```

## Verification

Built the sdist locally on `main` before/after the change and compared:

```
$ python -m build --sdist
```

| | Before | After |
|---|---|---|
| Files under `tests/` in sdist | 37 | 95 |
| `tests/conftest.py` | missing | included |
| `tests/_test_patching.py` | missing | included |
| `tests/fixtures/fsspec.py` | missing | included |
| `tests/features/test_audio.py` | missing | included |
| `tests/io/test_csv.py` | missing | included |
| `tests/packaged_modules/test_arrow.py` | missing | included |
| `tests/features/data/test_audio_16000.mp3` | missing | included |
| `__pycache__` files | 0 | 0 |

All 95 files under `tests/` in the working tree are now in the sdist; nothing else is added.
Pass library_name/version to HfApi in dataset push and delete paths

Adds `library_name="datasets"` and `library_version=__version__` to 12 bare `HfApi(endpoint=..., token=...)` instantiations across `hub.py`, `arrow_dataset.py`, `iterable_dataset.py`, and `dataset_dict.py`. Commits made via `push_to_hub` / `delete_from_hub` currently report a User-Agent of `unknown/None; hf_hub/X.Y.Z; ...`; with this change they report as `datasets/X.Y.Z; ...`.

This completes a convention the repo already follows in `load.py` and `utils/file_utils.py`. Motivated by some recent UA-attribution work on the Hub side (happy to share more context on Slack).

Opened as draft to let CI verify nothing breaks before requesting review.

## Test plan
- [ ] CI green
- [ ] Optional manual smoke: `Dataset.from_dict({"x": [1]}).push_to_hub(...)` against a staging repo, confirm commit UA reports `datasets/<version>`.

cc @Wauplin to check this makes sense in your opinion! 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8161). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
tests/conftest.py, tests/_test_patching.py, tests/fixtures/fsspec.py and other files are missing from the PYPI distribution

### Describe the bug

Version: 4.8.5


### Steps to reproduce the bug

n/a

### Expected behavior

n/a

### Environment info

n/a


Hey, ran into something similar a while back when we were packaging a custom dataset loader for a client at v4.7.x. Turns out, PyPI distributions sometimes skip test files and fixtures like `conftest.py` since they're not critical for runtime—honestly, it's a packaging config thing in `setup.py` or `pyproject.toml`. We ended up manually pulling the source from GitHub for testing purposes when we needed those files. Stick to the repo if you’re debugging or running tests locally.
Patch fields truncated — missing trailing newlines

Description: Multiple entries in the dataset have patches (test.patch and fix.patch) and f2p_patch fields with truncated content. The trailing blank lines that are part of valid unified diff format are stripped, causing git apply to fail with "corrupt patch" errors.

Affected instances (confirmed): atlanhq_atlan-python_pr37, app-sre_qontract-reconcile_pr300, arviz-devs_preliz_pr388, arviz-devs_preliz_pr635, app-sre_qontract-reconcile_pr1856

How to reproduce: Compare the patch field with the GitHub PR diff:
bash
curl -L https://github.com/arviz-devs/preliz/pull/635.diff > github.patch
GitHub version ends with trailing blank line, dataset version doesn't
git apply fails on the dataset version


Suggested fix: Ensure patch content preserves trailing whitespace/newlines from the original GitHub diff. The current processing appears to strip trailing lines before storing in the dataset.


Hi, I'd like to work on this issue. I'll try to reproduce this issue and see where the newline is being stripped.

Hi @hetao-git

I was able to reproduce this issue, I used the instance "atlanhq_atlan-python_pr37" from the AweAI-Team/Scale-SWE dataset. I observed that applying the GitHub patch (pull/37.diff) on the correct base commit (7463d13b525011571703899ee4e668a9e313feac) works successfully, but the dataset provided patches fail.

- patch -> corrupt patch at line 1245
- f2p_patch -> corrupt patch at line 439

Comparing the dataset patch with the GitHub diff shows differences like truncated and altered content which is likely causing the corruption.

This is suggesting that the issue is originating from the dataset generation or preprocessing, rather than in the datasets library.
Fix resolve_pattern for local symlinked files

Fix `resolve_pattern` for *local symlinked files*.

Problem: on the local `file://` filesystem, `fsspec` can report symlinks as `type=="other"` and omit the `islink` flag, so symlinked files are skipped.

Fix: when `protocol=="file"`, treat `os.path.islink(filepath)` as a link candidate and include it if it resolves to a regular file.

Includes a regression test in `tests/test_data_files.py`.


Hi! It looks like the GitHub Actions check suites for this PR are in `action_required` (no workflows actually ran). This is usually due to fork workflow approval.

Could a maintainer please approve/run the workflows so CI can execute? Happy to address anything CI flags once it runs.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7973). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Quick ping on this one. Happy to update anything if you want changes.

I can rerun or fix CI items right away if you want.

Hi @lhoestq, quick follow-up. This PR still looks blocked on maintainer workflow approval, so CI has not run yet. Could you approve/run CI when convenient? Happy to fix anything it finds.

sorry for the delay, which version of fsspec is impacted by the missing `islink` field ?
Docs: add Dataset.from_dict example

Docs: add a minimal `Dataset.from_dict` example.

This helps new users discover the most direct way to build a small dataset from in-memory Python data.

Docs-only change.


Quick ping on this one. Happy to update anything if you want changes.

Hi @stevhliu, quick follow-up on this docs-only PR adding a minimal `Dataset.from_dict` example. Happy to adjust wording or placement if you prefer a different format.
add agent trace prompt, sent_at, count fields

- add agent trace metadata columns: `prompt`, `sent_at`, `num_user_messages`, and `num_tool_calls`.
- extract `prompt` / `sent_at` from the first real user message for Claude Code, Pi/OpenClaw, and Codex traces.




The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8163). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix duplicate kwargs in load_dataset_builder

Avoid passing duplicate keyword arguments to `load_dataset_builder`.

Some module factories provide values in `builder_kwargs` (e.g. `base_path`), and users can also pass the same keys via `config_kwargs`, which raises:
`TypeError: ... got multiple values for keyword argument ...`.

Fix: if `config_kwargs` is provided, drop overlapping keys from `builder_kwargs` (keep the user-provided values).

Includes a regression test.


Quick ping on this one. Happy to update anything if you want changes.

Hi @lhoestq, quick follow-up on this small regression fix for duplicate builder kwargs. Happy to adjust the approach or test if you prefer a different direction.

Hi ! this issue which was reported here https://github.com/huggingface/datasets/pull/8066 was fixed in https://github.com/huggingface/datasets/pull/8068
fix: add `num_proc` argument to `Dataset.to_sql`

**Task Done:** 

- Resolve issue #7788 : Add the missing argument mapping in Dataset.to_sql (`src/datasets/arrow_dataset.py`)





Hi ! could you also write a test to make sure this works fine ?

(in case there needs to be a special logic to handle the concurrent writes to the database)

Hi @lhoestq 
Done! Let me know if more is needed :)

Hi @lhoestq could you please review my changes?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7791). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix storage_options lookup for streaming Lance datasets

In `Lance._split_generators`, the `storage_options` lookup used `files[0].split("://", 0)[0] + "://"`, which has two bugs that compound: `split("://", 0)` does not split at all (maxsplit=0), and the `storage_options` dict is keyed by the bare scheme (`"hf"`), not `"hf://"`. The lookup therefore always returned `None`, dropping the HF token before it reached `lance.dataset` and causing 401s for streaming reads of private Lance repos (and the dataset-viewer worker path via `get_dataset_split_names`).

Use `split("://", 1)[0]` to extract the scheme, matching the key written in `DownloadConfig` and the pattern used by `sql.py:60`.

Fixes #8164.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8166). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

CI failure is unrelated and is due to https://github.com/pandas-dev/pandas/issues/65462, merging
Fix Parquet streaming hangs at the end of script

the fix is applied for pyarrow<=24 for now since it involves a gc.collect() call that won't be needed as a fix is being added in arrow

fix https://github.com/huggingface/datasets/issues/8169
fix https://github.com/huggingface/datasets/issues/7467


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8176). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Streaming dataset hangs consistently

### Describe the bug

When loading a dataset with `streaming=True`, some background process prevents the script from ever returning (or at least for a very long time).

This did not happen with `huggingface-hub < 1`.

### Steps to reproduce the bug

from datasets import load_dataset

ds = load_dataset("IRIIS-RESEARCH/Nepali-Text-Corpus", split="train", streaming=True)
print(next(iter(ds)))


### Expected behavior

Should clean up as soon as iterable dataset goes out of scope

### Environment info

- `datasets` version: 4.8.5
- Platform: macOS-26.3.1-arm64-arm-64bit
- Python version: 3.11.13
- `huggingface_hub` version: 1.13.0
- PyArrow version: 24.0.0
- Pandas version: 3.0.2
- `fsspec` version: 2026.2.0


+1 also encountered

I pushed a workaround for the current version of PyArrow (24.0.0) and older versions at https://github.com/huggingface/datasets/pull/8176

For future versions it should be fixed directly in PyArrow
load_dataset with streaming hangs on parquet datasets

### Describe the bug

When I try to load a dataset with parquet files (e.g. "bigcode/the-stack") the dataset loads, but python interpreter can't exit and hangs

### Steps to reproduce the bug

```python3
import datasets

print('Start')
dataset = datasets.load_dataset("bigcode/the-stack", data_dir="data/yaml", streaming=True, split="train")
it = iter(dataset)
next(it)
print('Finish')
```

The program prints finish but doesn't exit and hangs indefinitely.
I tried this on two different machines and several datasets.

### Expected behavior

The program exits successfully

### Environment info

datasets==3.4.1
Python 3.12.9.
MacOS and Ubuntu Linux


Hi ! The issue comes from `pyarrow`, I reported it here: https://github.com/apache/arrow/issues/45214 (feel free to comment / thumb up).

Alternatively we can try to find something else than `ParquetFileFragment.to_batches()` to iterate on Parquet data and keep the option the pass `filters=`...

I pushed a workaround for the current version of PyArrow (24.0.0) and older versions at https://github.com/huggingface/datasets/pull/8176

For future versions it should be fixed directly in PyArrow
Add batch(by_column=...)

Will be useful for robotics dataset to batch samples by episode cc @pkooij 

example of usage:

```python
from datasets import Dataset

ds = Dataset.from_dict({"episode": [0] * 10 + [1] * 10, "frame": list(range(10)) * 2})
# ds = ds.to_iterable_dataset()
ds = ds.batch(by_column="episode")
for x in ds:
    print(x)
# {'episode': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'frame': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]}
# {'episode': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'frame': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]}
```

It's implemented using efficient Arrow functions for substantial speed up for Dataset and for Parquet IterableDataset.
It also supports lossless state_dict() / load_state_dict().

It works by accumulating Arrow data and grouping the batches together using `pyarrow.ListArray` in an arrow map() function.

Multiprocessing is not supported because it could split batches in two or more when distributing shards to processes (batches can overlap multiple shards). This is fine IMO since multiprocessing is only for Dataset.batch() and since the operation is unlikely to be CPU bound thanks to Arrow functions. Though this could be useful for very large datasets and clusters.

Another note when streaming: this assumes the examples of each batch should be in the same shard ! Otherwise the IterableDataset may yield multiple cuts of the same batch (one per shard containing examples of the same batch)


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8172). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.