noamaanMulla-03/datasets-issues huggingface dataset:
Don't save original_shard_lengths by default for backward compat

following #7897 

but let users enable it with `datasets.config.SAVE_ORIGINAL_SHARD_LENGTHS = True`

this is useful for the Dataset Viewer to know where each row comes from after converting to parquet


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7906). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
feat(bids): Add BIDS dataset loader for neuroimaging data

## Summary

Adds native BIDS (Brain Imaging Data Structure) dataset loading support using PyBIDS, enabling `load_dataset('bids', data_dir='/path/to/bids')` workflow for neuroimaging researchers.

**Contributes to #7804** (Support scientific data formats) - BIDS is a widely-used standard for organizing neuroimaging data built on NIfTI files.

## Changes

### Core Implementation
- `src/datasets/packaged_modules/bids/bids.py` - GeneratorBasedBuilder implementation
- `src/datasets/packaged_modules/bids/__init__.py` - Module exports
- `src/datasets/packaged_modules/__init__.py` - Registration with module registry
- `src/datasets/config.py` - `PYBIDS_AVAILABLE` config flag
- `setup.py` - Optional `pybids>=0.21.0` + nibabel dependency

### Features
- Automatic BIDS structure validation
- Subject/session/datatype filtering via config
- JSON sidecar metadata extraction
- NIfTI file decoding via existing Nifti feature

### Documentation & Tests
- `docs/source/bids_dataset.mdx` - User guide with examples
- `tests/packaged_modules/test_bids.py` - Unit tests (4 tests)

## Usage

```python
from datasets import load_dataset

# Load entire BIDS dataset
ds = load_dataset('bids', data_dir='/path/to/bids_dataset')

# Filter by subject/session
ds = load_dataset('bids', 
    data_dir='/path/to/bids_dataset',
    subjects=['01', '02'],
    sessions=['baseline']
)

# Access samples
sample = ds['train'][0]
print(sample['subject'])      # '01'
print(sample['nifti'].shape)  # (176, 256, 256)
print(sample['metadata'])     # JSON sidecar data
```

## Test plan

- [x] All 4 unit tests pass (`pytest tests/packaged_modules/test_bids.py`)
- [x] `make quality` passes (ruff check)
- [x] End-to-end tested with real OpenNeuro data (ds000102)

## Context

This PR is part of the neuroimaging initiative discussed with @TobiasPitters. Follows the BIDS 1.10.1 specification and leverages the existing Nifti feature for NIfTI file handling.

Related PRs:
- #7874 (Nifti visualization support)
- #7878 (Replace papaya with niivue)
fix(tests): stabilize flaky Hub LFS integration test

## Problem

`test_push_dataset_dict_to_hub_overwrite_files` intermittently fails with:
```
BadRequestError: LFS pointer pointed to a file that does not exist
```

This has been causing the `deps-latest` integration tests to fail on main (visible in recent CI runs). I ran into this while working on the BIDS loader PR and dug into the root cause.

## Root Cause

Two race conditions in the test:

1. **LFS propagation timing** - Rapid successive `push_to_hub` calls don't wait for Hub to fully propagate LFS objects between pushes
2. **Repo name reuse** - The second test scenario reused the same repo name from scenario 1, creating a race between deletion and recreation

## Solution

- Add `_wait_for_repo_ready()` helper that polls `list_repo_files` to ensure the repo is consistent before subsequent operations
- Use a unique repo name (`ds_name_2`) for the second scenario, eliminating the delete/create race entirely

## Testing

All 4 integration test variants now pass:
- ✅ `ubuntu-latest, deps-latest` (was failing)
- ✅ `ubuntu-latest, deps-minimum`
- ✅ `windows-latest, deps-latest` (was failing)
- ✅ `windows-latest, deps-minimum`

Validated on fork: https://github.com/The-Obstacle-Is-The-Way/datasets/pull/4

## Related

- #7600 (push_to_hub concurrency)
- #6392 (push_to_hub connection robustness)

cc @lhoestq - small fix but should help CI reliability


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7889). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Maybe `push_to_hub()` can retry on error 400 when committing to the Hub instead ? This way we make sure `push_to_hub()` works without having to add the waiting step

Thanks for the pointer @lhoestq! Updated to add LFS 400 retry handling directly in `push_to_hub()` across all implementations (Dataset, DatasetDict, IterableDataset, IterableDatasetDict). Reverted the test-side waits.
Support cast_kwargs in cast_columns

### Feature request

expose `cast(**cast_kwargs)` to `cast_column()`

https://github.com/huggingface/datasets/blob/0feb65dd8733191dd2d1e74215b422fc5939a56a/src/datasets/arrow_dataset.py#L2205

### Motivation

`cast_column()` wraps `cast()` function without exposing any `cast()` args. For large multi-modal datasets, e.g.

```py
# a dataset with list[{"bytes"}: b'', ...], much more than one image
load_dataset("MLLM-CL/VTCBench").cast_column("images", List(Image(decode=False)))
```

This would fail due to #6206, #7167, where the default value `1000` for batch size in `cast()` is too large and causes `pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays`.


https://github.com/huggingface/datasets/blob/0feb65dd8733191dd2d1e74215b422fc5939a56a/src/datasets/arrow_dataset.py#L2164-L2205 

### Your contribution

#7910 

Labels: name: enhancement
Fix: Correct float feature generation in `generate_examples` 

This PR fixes a bug in the `generate_examples` function where `datasets.Value` features with a `float` dtype were incorrectly generated using `np.random.randint`. This resulted in integer values being cast to float, which is not representative of true floating-point data.


**Key changes include:**

* Added explicit handling for `float` features using `np.random.rand` to generate continuous values.
* Introduced fail-fast type checks for unsupported dtypes to improve robustness.
* Added validation for sequence features to ensure `seq_shapes` is provided.

### Before Fix

Float features were generated incorrectly as integers cast to float:

```text
- Example 0:
- int_feature: 0
- float_feature: 9.0  <-- Incorrect: An integer disguised as a float
- string_feature: The small grey turtle was surprisingly fast...
- seq_feature: [0.3048 0.4291 0.4283]
```

### After Fix

Float features are now correctly generated as continuous numbers in the range [0, 1):
```text
+ Example 0:
+ int_feature: 0
+ float_feature: 0.0183  <-- Correct: A true random float
+ string_feature: The small grey turtle was surprisingly fast...
+ seq_feature: [0.9237 0.7972 0.8526]
```
#### Note: This PR is a follow-up/fix of the previously closed PR #7769 for clarity and context.



Hi @lhoestq, just a gentle follow-up on this PR.
GDPval dataset Word docs corrupted

The [openai/gdpval](https://huggingface.co/datasets/openai/gdpval) dataset on Hugging Face contains Word .docx files with two types of corruption that cause Microsoft Word to display an "unreadable content" error.

### Root Causes

1. **Corrupted settings.xml**: The `word/settings.xml` file uses incorrect namespace prefixes (`ns0:`, `ns1:`, etc.) instead of the proper prefixes (`w:`, `mc:`, `m:`, etc.)

2. **Malformed TargetMode attributes**: Some files have `TargetMode="External"` attributes missing their closing `/>` tag in hyperlink relationships

Both issues cause Word to reject the files even though the XML structure is technically valid.

I have a fix for the issue here https://github.com/alexheat/gdpval-docx-fix


tentatively tagging @simonpfish ^

(if it's an option you could enable PRs/Discussions on your dataset on HF)
Add progress_format support for machine-readable progress output

## Summary

Adds  support to , enabling machine-readable JSON progress output similar to [huggingface/tokenizers#1921](https://github.com/huggingface/tokenizers/pull/1921).

## Motivation

When using `datasets` in automated pipelines or UI applications, it's useful to emit machine-readable progress instead of ANSI progress bars. This PR adds the same `progress_format` option that was implemented in tokenizers.

## Changes

### New Functions

- `set_progress_format(format: str)`: Set global progress format
- `get_progress_format() -> str`: Get current progress format

### Supported Formats

1. **"tqdm"** (default): Interactive progress bars
2. **"json"**: Machine-readable JSON lines to stderr
3. **"silent"**: No output

### JSON Format

When `progress_format="json"`, emits JSON every 5% progress change or completion:

```json
{"stage":"Processing","current":50,"total":100,"percent":50.0}
```

## Usage Example

```python
from datasets import load_dataset
from datasets.utils import set_progress_format

# Enable JSON output
set_progress_format("json")

# Progress will now be emitted as JSON lines
dataset = load_dataset("Goader/kobza", split="train", streaming=True)
for sample in dataset:
    process(sample)
```

## Implementation Details

- Suppresses visual output using `io.StringIO()` when format is "json"
- Keeps progress tracking active (unlike `disable=True`)
- Emits JSON to stderr every 5% progress change
- Exports new functions from `datasets.utils`

## Cross-Reference

This implementation mirrors the approach from:
- [huggingface/tokenizers#1921](https://github.com/huggingface/tokenizers/pull/1921)

## Testing

Tested with:
```python
from datasets.utils import set_progress_format, tqdm

set_progress_format('json')
for i in tqdm(range(100), desc='Test'):
    process(i)
# Outputs: {"stage":"Test","current":10,"total":100,"percent":10.0}
```

## Checklist

- [x] New functions added to `datasets.utils.tqdm`
- [x] Functions exported from `datasets.utils.__init__`
- [x] JSON format emits to stderr
- [x] Visual output suppressed when format="json"
- [x] Progress tracking remains active
- [x] Cross-referenced with tokenizers#1921
Support downloading specific splits in `load_dataset`

This is PR builds on top of #7706 to revive the unfinished #6832 but isn't just cleaning up things, here are some important changes:
 - `download_mode="FORCE_REDOWNLOAD"` is interpreted as always creating a clean slate, that means that even if we already did:
 ```python
load_dataset("<name>")
load_dataset("<name>", split="train", download_mode="force_redownload")
```
This makes sure that only the train dataset is available after executing both. This was different in the original PR, which proposed that train and test would be available. 
- `download_mode="REUSE_DATASET_IF_EXISTS"` is interpreted as only ever adding new data, never redownloading OR deleting other splits. This was different in the original PR, where  
 ```python
load_dataset("<name>", split="test")
load_dataset("<name>", split="train")
```
resulted in only the train data being available, which I deem very unintuitive and probably not what users want. Also I argue that this is just the first step to a more user friendly partial loading when specifying percentages (or maybe even single instances) via the ReadInstructions, and then doing
 ```python
load_dataset("<name>", split="test[:10%]")
load_dataset("<name>", split="test[10%:]")
```
should result IMO in the whole dataset being cached locally without redownloads.

Furthermore this PR fixes a couple issues with the previous PR, e.g. a [missing comparison](https://github.com/huggingface/datasets/pull/7706/files#diff-f933ce41f71c6c0d1ce658e27de62cbe0b45d777e9e68056dd012ac3eb9324f7R877) and adding tests for the proposed changes in behaviour, which would both fail on @ArjunJagdale's original PR. 

Todo:
 - [ ] update docs?

Future outlook (just my opinions and up for debate):
As mentioned before, I would see this as just a step towards the feature of partial percentage loading (though how the API should behave in that case is not entirely clear for me now) and maybe we could introduce another `download_mode="FORCE_REDOWNLOAD_SPLIT"`, which makes sure that even if a split is specified, only the referenced split is redownloaded and everything else left unchanged, this would then allow users more granular control over what they want to redownload. 
@lhoestq very curious to get your opinion on this.


@CloseChoice This looks great! You're absolutely right about the missing comparison - that's a critical bug I missed. 

@lhoestq just resolved the conflicts inflicted by the latest changes. Thought it might be good giving this a shot now before more changes mess with this PR.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7858). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix duplicate keyword conflict in load_dataset_builder

Fixes #4910

This PR fixes a bug where passing the same keyword in builder_kwargs and
config_kwargs caused a TypeError in load_dataset_builder.

The kwargs are now merged safely so config_kwargs override builder_kwargs
without duplication. A regression test is added to prevent this from
happening again.



Hi HuggingFace team
This PR fixes issue #4910 by safely merging builder_kwargs and config_kwargs to avoid duplicate keyword errors.  
A regression test is included to ensure this does not happen again.  

Please let me know if you’d like any changes. Thanks!
feat: Add Apache TsFile format support

# Add Apache TsFile format support

Adds support for loading `.tsfile` datasets. Closes #7922.

## What's TsFile?

[Apache TsFile](https://tsfile.apache.org/) is a columnar time-series format popular in IoT. The TsFile community requested this integration and offered to help maintain it.

## What I did

Created a new `TsFile` builder in `packaged_modules/tsfile/` following the same pattern as HDF5. Registered the module and added `.tsfile` extension mapping. Also added `tsfile>=2.0.0` as an optional dependency.

The builder uses `tsfile.to_dataframe()` with iterator mode for memory-efficient reading, then converts to PyArrow tables. Schema is inferred automatically from file metadata.

## Config options

- `batch_size` - rows per batch (default 10000)
- `table_name` - which table to read (for multi-table files)
- `columns` - filter specific columns
- `start_time` / `end_time` - time-range filtering

## Usage

```python
from datasets import load_dataset

ds = load_dataset("tsfile", data_files=["data.tsfile"], split="train")

# with filtering
ds = load_dataset("tsfile", data_files=["data.tsfile"], 
                  columns=["temperature"], start_time=1609459200000)
```

## Tests

Added 11 tests covering config validation, basic loading, data integrity, feature inference, and error handling. All passing.



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7933). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Unbounded network usage when opening Data Studio

### Describe the bug

Opening the Data Studio tab on a dataset page triggers continuous and unbounded network traffic. This issue occurs across multiple browsers and continues even without user interaction.



### Steps to reproduce the bug

https://huggingface.co/datasets/slone/nllb-200-10M-sample/viewer

### Expected behavior

Data Studio should load a limited, finite amount of data and stop further network activity unless explicitly requested by the user.

### Environment info

- OS: Windows 10
- Browsers: Chrome, Firefox, Edge
- Device: Desktop
- Network: Standard broadband connection



cc @cfahlgren1

Thanks for reporting! Looking into this!

This should be fixed. Thank you for your patience! 🤗
xPath cannot handle hdfs:///xxxx properly

### Describe the bug

_as_str('hdfs:///xxxx') would return hdfs://xxxx. Removing one / and making the path invalid.

For the use case like
```
ds = load_dataset(
    "parquet",
    data_files={
        "train": "hdfs:///user/path/to/data/train*.parquet",
    },
    streaming=True,
    storage_options={
        "host": "hostname",  
    }
)
```
would get 
```
  File "/usr/local/lib/python3.11/site-packages/datasets/load.py", line 1511, in load_dataset
    return builder_instance.as_streaming_dataset(split=split)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/datasets/builder.py", line 1193, in as_streaming_dataset
    splits_generators = {sg.name: sg for sg in self._split_generators(dl_manager)}
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/datasets/packaged_modules/parquet/parquet.py", line 123, in _split_generators
    with open(file, "rb") as f:
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/datasets/streaming.py", line 73, in wrapper
    return function(*args, download_config=download_config, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 963, in xopen
    file_obj = fsspec.open(file, mode=mode, *args, **kwargs).open()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/core.py", line 508, in open
    out = open_files(
          ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/core.py", line 295, in open_files
    fs, fs_token, paths = get_fs_token_paths(
                          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/core.py", line 672, in get_fs_token_paths
    chain = _un_chain(urlpath0, storage_options or {})
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/core.py", line 365, in _un_chain
    kw = dict(
         ^^^^^
TypeError: dict() got multiple values for keyword argument 'host'
```
Due to the file passed to fsspec.open is hdfs://user/path/to/data/trainxxx.parquet, and fsspec would take user as the hostname

### Steps to reproduce the bug

<img width="992" height="148" alt="Image" src="https://github.com/user-attachments/assets/98e1dac2-e81b-4727-bf7a-55faaf0c8168" />

### Expected behavior

Keep all three /

### Environment info

datasets 4.4.2
Add lance format support

Add lance format as one of the `packaged_modules`.

```py
import datasets

ds = datasets.load_dataset("org/lance_repo", split="train")

# Or

ds = datasets.load_dataset("./local/data.lance")
```


Mentioned https://github.com/huggingface/datasets/issues/7863 as well

@pdames for vis

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7913). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Cool ! I notice the current implementation doesn't support streaming because of the symlink hack.

I believe you can do something like this instead:

```python
def _generate_tables(self, paths: list[str]):
    for path in paths:
        ds = lance.dataset(path)
        for frag_idx, fragment in enumerate(ds.get_fragments()):
            for batch_idx, batch in enumerate(
                fragment.to_batches(columns=self.config.columns, batch_size=self.config.batch_size)
            ):
                table = pa.Table.from_batches([batch])
                table = self._cast_table(table)
                yield Key(frag_idx, batch_idx), table
```

note that path can be a local one, but also a `hf://` URI

@lhoestq Take another look? 

I took the liberty to make a few changes :)

Now I believe we should be good:
- both local and streaming work fine
- both dataset and single files work fine
- all files are properly downloaded now than all files and metadata files are included in config.data_files
- sharding is supported:
  - dataset: one shard = one fragment
  - single files: one shard = one file 
- streaming dataset resuming works fine thanks to Key()
- the two hacks are visible and with TODOs to remove them when possible
  1. remove the revision in HF uris since only "main" is supported
  2. write proper _version/* files since lance doesn't work if they are symlinks

I think this PR is ready, just let me know what you think before we merge 🚀 

The next steps are:
- open a PR in this repository to document Lance support in `datasets`
- open a PR in https://github.com/huggingface/hub-docs to add `pylance` to the list of integrated library on HF, and have some documentation on how to use it with datasets on HF (here is an example [PR](https://github.com/huggingface/hub-docs/pull/1892))
- open a PR in https://github.com/huggingface/huggingface.js to add Lance as a supported dataset library on the HF website (here is an example [PR](https://github.com/huggingface/huggingface.js/pull/1870))

Feel free to start some drafts (I noticed there are great examples in your HF account now !), I'll be happy to review :)

And once Lance is available in huggingface.js and docs are ready we'll be ready to enable the Dataset Viewer and Lance code snippets on HF !
Raise early for invalid `revision` in `load_dataset`

Solves https://github.com/huggingface/datasets/issues/7928

Raise early for invalid revisions



Passes
```sh
pytest -k "LoadTest and test_load_dataset_invalid_revision_with_cache"
```

Fails
```sh
git checkout cc2399019a3a547ebc31ec68a1ff99abd4ec93ce
pytest -k "LoadTest and test_load_dataset_invalid_revision_with_cache"
```

Ran `make test`, but failures look unrelated to the PR diff (same tests fail on `main` too)

```sh
FAILED tests/test_distributed.py::test_torch_distributed_run[False] - TypeError: Passing coroutines is forbidden...
FAILED tests/test_distributed.py::test_torch_distributed_run[True] - TypeError: Passing coroutines is forbidden...
FAILED tests/test_distributed.py::test_torch_distributed_run_streaming_with_num_workers[2-2] - TypeError: Passing coroutines is forbidden...
FAILED tests/test_distributed.py::test_torch_distributed_run_streaming_with_num_workers[3-2] - TypeError: Passing coroutines is forbidden...
= 4 failed, 3077 passed, 18 skipped, 491 warnings in 556.45s (0:09:16) =
make: *** [Makefile:20: test] Error 1
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7929). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix low but large example indexerror

Fixes #7911.

This PR simply implements the approach outlined in the corresponding issue, that if we have large examples, the number of shards should never be more than the number of samples. This is an absolute edge case, but can happen for image data.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7912). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.