noamaanMulla-03/datasets-issues huggingface dataset:
Add filename in error message when ReadError or similar occur

Please update error messages to include relevant information for debugging when loading datasets with `load_dataset()` that may have a few corrupted files. 

Whenever downloading a full dataset, some files might be corrupted (either at the source or from downloading corruption). 

However the errors often only let me know it was a tar file if `tarfile.ReadError` appears on the traceback, and I imagine similarly for other file types. 

This makes it really hard to debug which file is corrupted, and when dealing with very large datasets, it shouldn't be necessary to force download everything again.


Hi Elisa, please share the error traceback here, and if you manage to find the location in the `datasets` code where the error occurs, feel free to open a PR to add the necessary logging / improve the error message.

> please share the error traceback

I don't have access to it but it should be during [this exception](https://github.com/huggingface/datasets/blob/2049c00921c59cdeb835137a1c49639cf175af07/src/datasets/builder.py#L1643) which happens during the loading of a dataset. If one of the downloaded files is corrupted, the for loop will not yield correctly, and the error will come from, say, in the case of tar files, [this iterable](https://github.com/huggingface/datasets/blob/2049c00921c59cdeb835137a1c49639cf175af07/src/datasets/utils/file_utils.py#L1293) which has no explicit error handling that leaves clues as to which file has failed.

I only know the case for tar files, but I consider this issue could be happening across different file types too.

I think having a better error handling for this tar iterable would be useful already, maybe a simple try/except in `_iter_from_urlpath` that checks for `tarfile.ReadError` and raises an error with the `urlpath` mentioned in the error ?

I think not just from higher calls like the `_iter_from_urlpath` but directly wherever a file is attempted to be opened would be the best case, as the traceback would simply lead to that.

so maybe there should be better error messages in each dataset builder definition ? e.g. in https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/webdataset/webdataset.py for webdataset TAR archives

#self-assign

I reproduced this on current main (`41adfd0f`): iterating an empty/corrupted WebDataset TAR raises `tarfile.ReadError: empty file`, but the exception does not identify the failing archive.

Following the maintainer direction above, I will keep the first patch WebDataset-specific: add the TAR path while preserving `tarfile.ReadError` and exception chaining, and cover both feature inference and example generation so the context is present whether features are inferred or supplied.
Include archive path in WebDataset TAR read errors

Fixes #7280

## What changed

WebDataset now catches `tarfile.ReadError` around the complete TAR processing generator and re-raises the same exception type with the resolved archive path. The original exception remains available through `__cause__`.

The wrapper covers failures while advancing the archive iterator and while reading a member payload. It is used by both feature inference and normal example generation.

The diagnostic deliberately formats `str(tar_path)` rather than `tar_path.__repr__()`. Downloaded paths can be `tracked_str` values whose custom representation includes the original URL; suppressing that origin avoids placing presigned query credentials in tracebacks while retaining the resolved failing path.

## Before

```text
ReadError: empty file
```

When several shards are processed, the exception chain does not identify which TAR failed.

## After

```text
ReadError: Failed to read TAR archive "/path/to/corrupted-shard-00042.tar": empty file
```

## Regression coverage

- corrupt first shard during feature inference
- one valid shard followed by a corrupt shard during example generation
- a truncated TAR member that fails during `f.read()`
- original `tarfile.ReadError` preserved as the cause
- Windows-safe path matching
- `tracked_str` origin containing a password/signature is not exposed

## Validation

- `pytest` across WebDataset, DownloadManager, and StreamingDownloadManager: 46 passed, 10 optional-dependency skips
- `make style`: passed; 232 files unchanged
- `git diff --check`: passed
- end-to-end `load_dataset("webdataset", ...)` reproduction: exception path present and original `ReadError` chained
- independent review: submit, no remaining findings
Fix casting a nullable LargeList to a different inner type

## What's the bug

`_combine_list_array_offsets_with_mask` (in `src/datasets/table.py`) rebuilds a list array's offsets with the null bitmap applied, using an `int32` null-fill:

```python
offsets = array.offsets
if array.null_count > 0:
    offsets = pa.concat_arrays(
        [
            pc.replace_with_mask(offsets[:-1], array.is_null(), pa.nulls(len(array), pa.int32())),
            offsets[-1:],
        ]
    )
```

A `pa.ListArray` has `int32` offsets, so this happens to match. But a `pa.LargeListArray` has `int64` offsets, and `pc.replace_with_mask(int64_offsets, mask, int32_nulls)` has no matching pyarrow kernel:

```
pyarrow.lib.ArrowNotImplementedError: Function 'replace_with_mask' has no kernel matching input types (int64, bool, int32)
```

This is reached from `array_cast` (large_list branch) and `cast_array_to_feature` (`LargeList` branch) whenever a `LargeList` column **has nulls** and its **inner type changes**. A same-inner-type cast hits the no-op early return, which is why it stayed hidden.

Minimal reproduction:

```python
import pyarrow as pa
from datasets.table import cast_array_to_feature
from datasets import LargeList, Value

arr = pa.array([[1, 2], None, [3]], type=pa.large_list(pa.int64()))
cast_array_to_feature(arr, LargeList(Value("int32")))   # ArrowNotImplementedError
```

The equivalent user-facing call `Dataset.cast_column("col", LargeList(Value("int32")))` raises the same error. The same cast on a regular `List` succeeds.

## Fix

Use `offsets.type` for the null-fill instead of a hardcoded `pa.int32()`, so it matches the offset width for both list kinds (`int32` for `ListArray`, `int64` for `LargeListArray`):

```python
pc.replace_with_mask(offsets[:-1], array.is_null(), pa.nulls(len(array), offsets.type)),
```

This is the root-cause fix and covers all three caller paths; regular-list casts are unchanged.

## Tests

Added `tests/test_table.py::test_cast_array_to_feature_with_nullable_large_list_and_new_inner_type` (the existing `test_cast_array_to_feature_with_list_array_and_large_list_feature` uses non-nullable data, so it never exercised this path).

```bash
python -m pytest tests/test_table.py -q
```

- Before: the new test errors with `ArrowNotImplementedError`.
- After: `278 passed, 1 skipped`; the cast returns `large_list<int32>` with `[[1, 2], None, [3]]`.
fix: stable fingerprint for closures capturing non-deterministic state

Fixes #7986

When `.map()` uses a closure capturing `self` with non-deterministic state 
(like UUIDs or loggers), the fingerprint changes every run causing cache misses.

Fix: only hash primitive attributes of captured objects instead of the full object.

Added a regression test in `tests/test_fingerprint.py`.


I feel like this would cause headaches to many people, causing cache reuses when the function actually changed

You're right, thank you for pointing that out. The current approach is too 
broad — ignoring all non-primitive attributes could cause incorrect cache 
reuse when meaningful objects like tokenizers change between runs.

What approach would you suggest? Should we instead warn the user when a 
closure captures a non-deterministic object, or only apply this when the 
user explicitly opts in?
Include file size in dataset fingerprint

### Motivation
Currently, the dataset cache fingerprints rely entirely on the `ETag`/`etag`/`mtime` metadata fetched during `fs.info()`. If a file's byte size changes on the remote end—and the remote backend fails to reliably update the ETag—the dataset fingerprint stays improperly static. This leads to insidious bugs where an old, structurally distinct dataset cache continues mapping to the new remote file.

### Summary of Changes
* **Modified `_get_single_origin_metadata` in `src/datasets/data_files.py`**:
  Checks if `size` is present in the underlying filesystem's `info` payload, and appends it directly to the ETag string (`{ETag}-{size}`). If the size doesn't exist, it elegantly falls back to the original `ETag` string structure.
* **Updated `tests/test_data_files.py`**:
  Implemented `test_get_single_origin_metadata_includes_size` using a mocked `url_to_fs` patch to guarantee that the `size` byte integer correctly concatenates to the final string tuple emitted by `_get_single_origin_metadata()`.

### Testing
Tested unit test logic behavior locally by verifying that the mock tuple returns `("my_etag_123-1024",)`.
Add `skip_origin_metadata` flag to optimize streaming startup time

Fix #8197
This PR introduces the `skip_origin_metadata` kwarg to skip costly origin metadata fetching (ETags/mtime) when it's not needed, improving streaming initialization latency. This saves number of files `fs.info` calls per `load_dataset` call.

### Changes
- **Kwarg Plumbing**: Routes the `skip_origin_metadata` kwarg from `load_dataset(...)` safely down to `DatasetBuilder` while stripping it securely from kwargs (via `.pop()`) so it's not leaked into underlying configuration classes that don't expect it (such as `CsvConfig()`).
- **Streaming Validations**:
  - Emits a warning when `skip_origin_metadata=True` is passed to `load_dataset` without `streaming=True`.
  - Disables `download_and_prepare()` entirely when `builder._skip_origin_metadata` is true by raising a `ValueError("This function is not intended for streaming.")`.
- **Test Coverage**: Added robust test coverage spanning `tests/test_data_files.py` and `tests/test_load.py`:
  - `test_DataFilesList_from_patterns_skip_origin_metadata`
  - `test_DataFilesList_from_patterns_skip_origin_metadata_raises_FileNotFoundError`
  - `test_load_dataset_builder_skip_origin_metadata` (covers kwarg mapping & ValueError)
  - `test_load_dataset_skip_origin_metadata_warning`

### Metrics
We ran a streaming dataset (1650 shards) benchmark across 8 processes against GCS. 
1. Before PR (main branch)
Performance: Iteration execution times were highly variable, scaling anywhere between ~4.0s and ~10.8s depending.
fs.info() Calls: 1,651 per process (13,208 total across 8 processes).

2. After PR (pr-8291, with --skip-origin-metadata)
Performance: Execution times were extremely stable, consistently landing around ~1.7s.
fs.info() Calls: 1 per process (8 total across 8 processes).

### Impact 
By bypassing the unconditional ETag/origin metadata fetches during dataset builder initialization on the streaming path, we reduced the fs.info() calls from over 13,000 down to just 8. 

On average, this structural optimization reduced the initialization latency from 5.55s to 1.77s (~3.13x faster), entirely removing the extensive delays previously chained to API overhead/variability.



Hi @lhoestq, could you take a look at this PR? Thanks.

@yuxin00j Can you please add how much is this change improving streaming initialization latency ?

> @yuxin00j Can you please add how much is this change improving streaming initialization latency ?

Added in the description.