AniketTec/github-issues huggingface dataset:
Title: feat(fasta): add lightweight FASTA file format support
State: open

## Summary

This PR adds support for loading FASTA files directly with `load_dataset()`, addressing feedback from #7851.

FASTA is a text-based format for representing nucleotide sequences (DNA/RNA) or peptide sequences (proteins), widely used in bioinformatics.

## Key Features

- **Zero external dependencies** - Uses a lightweight pure Python parser based on [readfq.py](https://github.com/lh3/readfq) by Heng Li
- **Streaming support** - Generator-based parsing for memory efficiency with large genomic files
- **Compression support** - Automatic detection and handling of gzip, bzip2, and xz compressed files via magic bytes
- **Large sequence support** - Uses `large_string` Arrow type to handle viral genomes and long sequences (fixes UTF-8 overflow)
- **Adaptive batching** - `max_batch_bytes` parameter (default 256MB) prevents Parquet page size errors with very large sequences

## Technical Decisions (Addressing #7851 Feedback)

| Concern | Solution |
|---------|----------|
| Long sequences → UTF-8 overflow (@apcamargo, @UriNeri) | Uses `pa.large_string()` for sequence column |
| BioPython is overkill (@apcamargo) | Pure Python parser based on Heng Li's readfq.py |
| Parquet page size limit i32::MAX (@UriNeri) | Adaptive dual-threshold batching with `max_batch_bytes` |

## Columns

| Column | Type | Description |
|--------|------|-------------|
| `id` | string | Sequence identifier (first word after `>`) |
| `description` | string | Full description line (everything after id) |
| `sequence` | large_string | The biological sequence (DNA/RNA/protein) |

## Supported Extensions

`.fa`, `.fasta`, `.fna`, `.ffn`, `.faa`, `.frn` (and compressed variants)

## Usage

```python
from datasets import load_dataset

# Load FASTA file
dataset = load_dataset("fasta", data_files="sequences.fasta")

# Load with column filtering
dataset = load_dataset("fasta", data_files="sequences.fa", columns=["id", "sequence"])

# Load gzipped file
dataset = load_dataset("fasta", data_files="sequences.fa.gz")

# Configure batching for very large genomes
dataset = load_dataset("fasta", data_files="genome.fasta", max_batch_bytes=128*1024*1024)
```

## Test Plan

- [x] Basic FASTA loading (3 sequences, multi-line)
- [x] Multiple extension support (.fa, .fasta, .fna, .ffn, .faa, .frn)
- [x] Compression formats (gzip, bz2, xz)
- [x] Long sequences with `large_string` type
- [x] Column filtering
- [x] Batch size configuration
- [x] Byte-based batching (`max_batch_bytes`)
- [x] Large genome handling (simulated 50KB sequences)
- [x] Empty description handling
- [x] Multiple files loading
- [x] Custom feature casting

All 22 tests passing.

cc: @georgia-hf

Comments:
Title: Add progress_format support for machine-readable progress output
State: open

## Summary

Adds  support to , enabling machine-readable JSON progress output similar to [huggingface/tokenizers#1921](https://github.com/huggingface/tokenizers/pull/1921).

## Motivation

When using `datasets` in automated pipelines or UI applications, it's useful to emit machine-readable progress instead of ANSI progress bars. This PR adds the same `progress_format` option that was implemented in tokenizers.

## Changes

### New Functions

- `set_progress_format(format: str)`: Set global progress format
- `get_progress_format() -> str`: Get current progress format

### Supported Formats

1. **"tqdm"** (default): Interactive progress bars
2. **"json"**: Machine-readable JSON lines to stderr
3. **"silent"**: No output

### JSON Format

When `progress_format="json"`, emits JSON every 5% progress change or completion:

```json
{"stage":"Processing","current":50,"total":100,"percent":50.0}
```

## Usage Example

```python
from datasets import load_dataset
from datasets.utils import set_progress_format

# Enable JSON output
set_progress_format("json")

# Progress will now be emitted as JSON lines
dataset = load_dataset("Goader/kobza", split="train", streaming=True)
for sample in dataset:
    process(sample)
```

## Implementation Details

- Suppresses visual output using `io.StringIO()` when format is "json"
- Keeps progress tracking active (unlike `disable=True`)
- Emits JSON to stderr every 5% progress change
- Exports new functions from `datasets.utils`

## Cross-Reference

This implementation mirrors the approach from:
- [huggingface/tokenizers#1921](https://github.com/huggingface/tokenizers/pull/1921)

## Testing

Tested with:
```python
from datasets.utils import set_progress_format, tqdm

set_progress_format('json')
for i in tqdm(range(100), desc='Test'):
    process(i)
# Outputs: {"stage":"Test","current":10,"total":100,"percent":10.0}
```

## Checklist

- [x] New functions added to `datasets.utils.tqdm`
- [x] Functions exported from `datasets.utils.__init__`
- [x] JSON format emits to stderr
- [x] Visual output suppressed when format="json"
- [x] Progress tracking remains active
- [x] Cross-referenced with tokenizers#1921

Comments:
Title: Fix load_from_disk progress bar with redirected stdout
State: closed

Fixes #7918 

## Problem
When using `load_from_disk()` with `contextlib.redirect_stdout()`, the progress bar was not showing even for datasets with >16 files.

## Root Cause  
The `disable` parameter was set to `None` which triggers TTY auto-detection. This fails when stdout is redirected, causing the progress bar to be hidden.

## Solution
Changed `disable=len(state["_data_files"]) <= 16 or None` to `disable=len(state["_data_files"]) <= 16` to force the progress bar to show for datasets with >16 files, regardless of stdout redirection.

## Testing
Verified that progress bars now appear correctly both with and without stdout redirection for datasets with >16 shards.

Comments:
this seems to contradict the comment that says 

> set `disable=None` rather than `disable=False` by default to disable progress bar when no TTY attached

I believe the right approach is to do the same as in https://github.com/huggingface/huggingface_hub/pull/2698

> this seems to contradict the comment that says
> 
> > set `disable=None` rather than `disable=False` by default to disable progress bar when no TTY attached
> 
> I believe the right approach is to do the same as in [huggingface/huggingface_hub#2698](https://github.com/huggingface/huggingface_hub/pull/2698)

Updated to check TQDM_POSITION=-1 to force-enable progress bars in cloud environments, 
following the same pattern as huggingface_hub#2698.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7919). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Moved the TQDM_POSITION check to the tqdm class in utils/tqdm.py so all progress bars 
in the codebase have consistent behavior. Thanks for the suggestion!

@lhoestq thanks again for the suggestion. I’ve applied it and everything should now be consistent across all tqdm usage. Happy to adjust anything else if needed.
Title: datasets.load_from_disk doesn't show progress bar
State: closed

### Describe the bug

This is the inverse of the bug at [https://github.com/huggingface/datasets/issues/7030](https://github.com/huggingface/datasets/issues/7030), i.e. that `datasets.load_from_disk(path)` displays no progress bar. My dataset has > 16 files in it.

I am redirecting stdout as I capture the log, could this have something to do with it? All other progress bars work fine though except for HF dataset progress bars.

### Steps to reproduce the bug

```py
with contextlib.redirect_stdout(log_file), contextlib.redirect_stderr(log_file):
    datasets.load_from_disk(path)
```

### Expected behavior

The progress bar should show when loading a dataset.

### Environment info

Python 3.13.9
Datasets 4.4.1
macOS Tahoe 26.2

Comments:
#self-assign
Title: GDPval dataset Word docs corrupted
State: open

The [openai/gdpval](https://huggingface.co/datasets/openai/gdpval) dataset on Hugging Face contains Word .docx files with two types of corruption that cause Microsoft Word to display an "unreadable content" error.

### Root Causes

1. **Corrupted settings.xml**: The `word/settings.xml` file uses incorrect namespace prefixes (`ns0:`, `ns1:`, etc.) instead of the proper prefixes (`w:`, `mc:`, `m:`, etc.)

2. **Malformed TargetMode attributes**: Some files have `TargetMode="External"` attributes missing their closing `/>` tag in hyperlink relationships

Both issues cause Word to reject the files even though the XML structure is technically valid.

I have a fix for the issue here https://github.com/alexheat/gdpval-docx-fix

Comments:
tentatively tagging @simonpfish ^

(if it's an option you could enable PRs/Discussions on your dataset on HF)
Title: Add lance format support
State: closed

Add lance format as one of the `packaged_modules`.

```py
import datasets

ds = datasets.load_dataset("org/lance_repo", split="train")

# Or

ds = datasets.load_dataset("./local/data.lance")
```

Comments:
Mentioned https://github.com/huggingface/datasets/issues/7863 as well

@pdames for vis

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7913). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Cool ! I notice the current implementation doesn't support streaming because of the symlink hack.

I believe you can do something like this instead:

```python
def _generate_tables(self, paths: list[str]):
    for path in paths:
        ds = lance.dataset(path)
        for frag_idx, fragment in enumerate(ds.get_fragments()):
            for batch_idx, batch in enumerate(
                fragment.to_batches(columns=self.config.columns, batch_size=self.config.batch_size)
            ):
                table = pa.Table.from_batches([batch])
                table = self._cast_table(table)
                yield Key(frag_idx, batch_idx), table
```

note that path can be a local one, but also a `hf://` URI

@lhoestq Take another look? 

I took the liberty to make a few changes :)

Now I believe we should be good:
- both local and streaming work fine
- both dataset and single files work fine
- all files are properly downloaded now than all files and metadata files are included in config.data_files
- sharding is supported:
  - dataset: one shard = one fragment
  - single files: one shard = one file 
- streaming dataset resuming works fine thanks to Key()
- the two hacks are visible and with TODOs to remove them when possible
  1. remove the revision in HF uris since only "main" is supported
  2. write proper _version/* files since lance doesn't work if they are symlinks

I think this PR is ready, just let me know what you think before we merge 🚀 

The next steps are:
- open a PR in this repository to document Lance support in `datasets`
- open a PR in https://github.com/huggingface/hub-docs to add `pylance` to the list of integrated library on HF, and have some documentation on how to use it with datasets on HF (here is an example [PR](https://github.com/huggingface/hub-docs/pull/1892))
- open a PR in https://github.com/huggingface/huggingface.js to add Lance as a supported dataset library on the HF website (here is an example [PR](https://github.com/huggingface/huggingface.js/pull/1870))

Feel free to start some drafts (I noticed there are great examples in your HF account now !), I'll be happy to review :)

And once Lance is available in huggingface.js and docs are ready we'll be ready to enable the Dataset Viewer and Lance code snippets on HF !
Title: fix low but large example indexerror
State: closed

Fixes #7911.

This PR simply implements the approach outlined in the corresponding issue, that if we have large examples, the number of shards should never be more than the number of samples. This is an absolute edge case, but can happen for image data.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7912). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Support cast_kwargs in cast_columns
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

expose `cast(**cast_kwargs)` to `cast_column()`

https://github.com/huggingface/datasets/blob/0feb65dd8733191dd2d1e74215b422fc5939a56a/src/datasets/arrow_dataset.py#L2205

### Motivation

`cast_column()` wraps `cast()` function without exposing any `cast()` args. For large multi-modal datasets, e.g.

```py
# a dataset with list[{"bytes"}: b'', ...], much more than one image
load_dataset("MLLM-CL/VTCBench").cast_column("images", List(Image(decode=False)))
```

This would fail due to #6206, #7167, where the default value `1000` for batch size in `cast()` is too large and causes `pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays`.


https://github.com/huggingface/datasets/blob/0feb65dd8733191dd2d1e74215b422fc5939a56a/src/datasets/arrow_dataset.py#L2164-L2205 

### Your contribution

#7910 

Comments:
Title: Don't save original_shard_lengths by default for backward compat
State: closed

following #7897 

but let users enable it with `datasets.config.SAVE_ORIGINAL_SHARD_LENGTHS = True`

this is useful for the Dataset Viewer to know where each row comes from after converting to parquet

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7906). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Unbounded network usage when opening Data Studio
State: closed

### Describe the bug

Opening the Data Studio tab on a dataset page triggers continuous and unbounded network traffic. This issue occurs across multiple browsers and continues even without user interaction.



### Steps to reproduce the bug

https://huggingface.co/datasets/slone/nllb-200-10M-sample/viewer

### Expected behavior

Data Studio should load a limited, finite amount of data and stop further network activity unless explicitly requested by the user.

### Environment info

- OS: Windows 10
- Browsers: Chrome, Firefox, Edge
- Device: Desktop
- Network: Standard broadband connection


Comments:
cc @cfahlgren1

Thanks for reporting! Looking into this!

This should be fixed. Thank you for your patience! 🤗
Title: Request: Review pending neuroimaging PRs (#7886 BIDS loader, #7887 lazy loading)
State: open

## Summary

I'm building production neuroimaging pipelines that depend on `datasets` and would benefit greatly from two pending PRs being reviewed/merged.

## Pending PRs

| PR | Description | Status | Open Since |
|----|-------------|--------|------------|
| [#7886](https://github.com/huggingface/datasets/pull/7886) | BIDS dataset loader | Open | Nov 29 |
| [#7887](https://github.com/huggingface/datasets/pull/7887) | Lazy loading for NIfTI | Open | Nov 29 |

## Use Case

The neuroimaging community uses the BIDS (Brain Imaging Data Structure) standard for organizing MRI/fMRI data. These PRs would enable:

1. **#7886**: `load_dataset('bids', data_dir='/path/to/bids')` - Load local BIDS directories directly
2. **#7887**: Memory-efficient NIfTI handling (single 4D fMRI file can be 1-2GB)

## Current Workaround

Without these, users must either:
- Upload to Hub first, then consume (works but slow iteration)
- Hand-roll BIDS parsing (duplicates effort)

## Request

Could a maintainer review these PRs? Happy to address any feedback. The BIDS loader has tests passing and was end-to-end tested with real OpenNeuro data.

Thank you for the great work on `Nifti()` support - these PRs build on that foundation.

## Related

- Contributes to #7804 (Support scientific data formats)
- Built on @TobiasPitters's Nifti feature work

Comments:
Hi ! sure I'll be happy to take a look, sorry for the delay :)
Title: The child process retrieves the dataset directly from the main process instead of executing `memory_mapped_arrow_table_from_file`.
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

The child process retrieves the dataset directly from the main process instead of executing `memory_mapped_arrow_table_from_file`.

### Motivation

Because my local disk space is insufficient, I can only store a dataset on a remote Ceph server and process it using datasets.
I used the data-juicer[https://github.com/datajuicer/data-juicer] framework as an outer layer which uses datasets, but it doesn't support streaming datasets. I then encountered a problem: for each load, map, and filter operation, I had to wait for a large number of child processes to execute `memory_mapped_arrow_table_from_file`. Since the actual file was on the remote Ceph server, this operation was limited by network I/O.
I don't know if it's a problem with my usage or if this is how datasets are currently designed.However, I think that if the instances obtained after datasets.load_datasets are directly passed to the child process instead of re-executing `memory_mapped_arrow_table_from_file`, it might solve my problem.Or datasets already support this capability, but I just didn't know it？

### Your contribution

。。。

Comments:
Memory mapping is actually the way for processes to share memory efficiently and without copy. It is efficient when on are using a local disk, and it's discouraged to use it on remote disk for the reasons you observed.

What you can do instead is save the dataset as Parquet on your remote storage (or on Hugging Face Datasets which offers fast uploads thanks to Xet), and then your can reload it in streaming mode. Streaming mode is ideal to use a dataset that is hosted in a remote storage
Title: Add inspect_ai eval logs support
State: closed

Support for .eval log files from inspect_ai

They are actually ZIP files according to the source code at https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/log/_log.py

Unfortunately their format can't be converted to Parquet, so I had to JSON-encode all the nested values

```python
ds = load_dataset("dvilasuero/kimi-bfcl")
```

this will enable the Viewer for datasets like https://huggingface.co/datasets/dvilasuero/kimi-bfcl

original tweet for context: https://x.com/dvilasuero/status/1996936988176343220?s=20

cc @dvsrepo @julien-c @davanstrien 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7899). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Very cool! 

Any reason not to directly use inspect for loading/converting from the binary format to JSON? https://inspect.aisi.org.uk/reference/inspect_ai.log.html#convert_eval_logs 

The format is simple enough to not have to rely on an additional dependency :)
Title: Save input shard lengths
State: closed

will be useful for the Viewer, to know what (original) shard each row belongs to

cc @cfahlgren1 

next step is use it in Dataset Viewer and expose an API that returns the file containing the row at rowId

(took the opportunity to remove unusued code)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7897). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: fix: use temp files in push_to_hub to prevent OOM on large datasets
State: closed

## Summary

Fixes memory accumulation in `_push_parquet_shards_to_hub_single` that causes OOM when uploading large datasets with many shards.

**Root cause**: The current implementation stores ALL parquet shard bytes in memory via `BytesIO`, accumulating in the `additions` list. For N shards of ~300MB each, this requires N × 300MB RAM.

**Fix**: Write parquet to temp file instead of `BytesIO`, pass file path to `CommitOperationAdd`. Delete temp file after `preupload_lfs_files` completes (for LFS uploads only - regular uploads need the file until `create_commit`).

## Changes

- Replace `BytesIO` with `tempfile.NamedTemporaryFile` in `_push_parquet_shards_to_hub_single`
- Use file path in `CommitOperationAdd.path_or_fileobj` instead of bytes
- Delete temp file after upload (only for LFS mode - regular uploads keep file for `create_commit`)
- Add `try...finally` for safe cleanup even on errors
- Remove unused `BytesIO` import

## Test plan

- [x] Added `tests/test_push_to_hub_memory.py` with 4 tests:
  - `test_push_to_hub_uses_file_path_not_bytes_in_commit_operation`
  - `test_push_to_hub_cleans_up_temp_files_for_lfs_uploads`
  - `test_push_to_hub_keeps_temp_files_for_regular_uploads`
  - `test_push_to_hub_uploaded_size_still_calculated`
- [x] All tests pass locally
- [x] ruff check passes

## Context

This was discovered while uploading a 270GB neuroimaging dataset (ARC) with 902 shards. The process was killed by OOM after accumulating ~270GB in the `additions` list.

Fixes #7893
Related: #5990, #7400

Comments:
Closing this PR.

After further investigation (prompted by @lhoestq's feedback on #7893), I discovered that `free_memory=True` does work correctly - memory doesn't actually accumulate in the `additions` list as I originally thought.

I tested standard `push_to_hub()` with 902 shards and it processed 625 shards (69%) without OOM issues. The crash we were experiencing was actually **#7894** (`embed_table_storage` crash on `Sequence()` types), not memory accumulation.

I've closed #7893 as invalid. This PR is no longer needed.

Apologies for the noise - lesson learned about testing before filing.
Title: encode nifti correctly when uploading lazily
State: closed

When trying to upload nifti datasets lazily I got the error:
```python
from pathlib import Path
from datasets import load_dataset

nifti_dir = Path("<local_path>")

dataset = load_dataset(
        "niftifolder",
        data_dir=str(nifti_dir.absolute()),
        streaming=True,
)

dataset.push_to_hub(repo_id="TobiasPitters/test-nifti-papaya-testdata")
```


```python
pyarrow.lib.ArrowInvalid: Could not convert <datasets.features.nifti.Nifti1ImageWrapper object at 0x77633407af90> with type Nifti1ImageWrapper: did not recognize Python value type when inferring an Arrow data type
```

This PR fixes that by encoding the Nifti1ImageWrappers correctly to bytes. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7892). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.