noamaanMulla-03/datasets-issues huggingface dataset:
Add fasta support

This PR adds support for FASTA files conversion to Parquet. 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7851). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

A few comments:

- Have you tried using this with longer sequences? @UriNeri developed something similar internally and used it with viral genomes. He got some Parquet errors due to genomes not fitting in a `utf8` column. This was fixed by using `large_utf8`.
- If you're only using it to read FASTA files, I think that having BioPython as a dependency is overkill. The library is very large and the FASTA parser isn't particularly fast. I have an example of a fast parser with no external references [here](https://gist.github.com/apcamargo/d039aa04a2cbbcbb14e2d34a0963b862) (this is actually based on [`readfq.py`](https://github.com/lh3/readfq/blob/master/readfq.py), with a couple of extra functions that might not be useful in the context of this PR)

> * If you're only using it to read FASTA files, I think that having BioPython as a dependency is overkill. The library is very large and the FASTA parser isn't particularly fast. I have an example of a fast parser with no external references [here](https://gist.github.com/apcamargo/d039aa04a2cbbcbb14e2d34a0963b862) (this is actually based on [`readfq.py`](https://github.com/lh3/readfq/blob/master/readfq.py), with a couple of extra functions that might not be useful in the context of this PR)

What @apcamargo said, plus FWIW in **our approach** (so might not be relevant here) we use polars (with custom fasta io parser) or polars-bio (that has a `scan_fasta` function) and we foudn out that the page size sometimes need to be adjusted:
```
envs/default/lib/python3.9/site-packages/polars/lazyframe/frame.py:2422, in LazyFrame.collect(self, type_coercion, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, collapse_joins, no_optimization, engine, background, optimizations, **_kwargs)
   2420 # Only for testing purposes
   2421 callback = _kwargs.get("post_opt_callback", callback)
-> 2422 return wrap_df(ldf.collect(engine, callback))
ComputeError: parquet: File out of specification: A page can only contain i32::MAX uncompressed bytes. This one contains 4544943557
```

Which in polars can be solved with:
```
df.write_parquet(
    "test1.patquet",
   compression="zstd",
    row_group_size=10_000,  #  smaller row groups
     data_page_size=1024*1024  # 1MB page size
)
```
Add lightweight FASTQ file format support

## Summary

This PR adds support for loading FASTQ files directly with `load_dataset()`.

FASTQ is an extension of FASTA that includes quality scores for each base, widely used for storing output from high-throughput sequencing instruments.

### Key Features
- **Zero external dependencies** - Pure Python parser based on [readfq.py](https://github.com/lh3/readfq) by Heng Li
- **Quality score support** - Preserves per-base quality scores as ASCII-encoded strings
- **Streaming support** - Generator-based parsing for memory efficiency with large NGS files
- **Compression support** - Automatic detection of gzip, bzip2, and xz compressed files
- **Large sequence support** - Uses `large_string` for both sequence and quality columns
- **Parquet-safe batching** - Dual-threshold batching (batch_size + max_batch_bytes) prevents page size errors

### Columns
| Column | Type | Description |
|--------|------|-------------|
| `id` | string | Sequence identifier (first word after `@`) |
| `description` | string | Full description line (everything after id) |
| `sequence` | large_string | The nucleotide sequence |
| `quality` | large_string | ASCII-encoded quality scores (Phred+33 by default) |

### Supported Extensions
`.fq`, `.fastq` (and compressed variants: `.fq.gz`, `.fastq.gz`, `.fq.bz2`, `.fq.xz`)

### Usage
```python
from datasets import load_dataset

# Load FASTQ file
dataset = load_dataset("fastq", data_files="reads.fastq")

# Load gzipped file
dataset = load_dataset("fastq", data_files="reads.fq.gz")

# Filter columns
dataset = load_dataset("fastq", data_files="reads.fq", columns=["sequence", "quality"])
```

### Quality Score Format
Quality scores use Sanger/Illumina 1.8+ encoding (Phred+33):
- ASCII character `\!` (33) = quality 0
- ASCII character `I` (73) = quality 40

### Testing
- 22 comprehensive tests covering basic loading, multi-line sequences, compression, batching, schema types, and edge cases
- All tests passing
- Linting clean

### References
- Follows pattern established in #7923 (FASTA support)
- Parser based on: https://github.com/lh3/readfq
- Addresses feedback from #7851

cc: @georgia-hf


same comment as in https://github.com/huggingface/datasets/pull/7923

> As fo the other PR I think we can have biopython as an optional dependency for the parsing
> 
> Also I mentioned in https://github.com/huggingface/datasets/pull/7851 that maybe it would be nice to have a dedicated type for this kind of data and decode the data as `SeqRecord` objects from biopython, which are nicer than simple python dictionaries for practitioners. wdyt ?

(btw since the two formats are similar this could also be one single PR)
Support Cursor agent traces

## Summary

- detect Teich Cursor trace rows using Cursor-specific metadata and preserved `raw_cursor` data
- normalize Cursor sessions into the shared agent traces schema with session id, timestamp, prompt, user count, tool-call count, and per-row raw trace preservation
- update the Teich test dependency to `0.2.8` for Cursor trace conversion support

## Validation

- `PYTHONPATH=src uv run --python 3.12 --with pytest --with setuptools --with pyarrow --with pandas --with teich==0.2.6 --with fsspec --with aiohttp --with xxhash --with multiprocess --with dill --with huggingface-hub --with requests --with packaging --with tqdm pytest -s tests/packaged_modules/test_json.py::test_json_generate_tables_with_agent_trace_metadata tests/packaged_modules/test_json.py::test_json_load_dataset_with_agent_trace_metadata tests/packaged_modules/test_json.py::test_json_load_dataset_without_droid_marker_stays_ordinary_json -q`
- loaded `armand0e/cursor-traces-example`'s `cursor-sessions.jsonl` through the patched JSON loader and confirmed 13 rows with the shared agent trace columns



Here's an svg that should look good for the icon:

<img width="167" height="190" alt="cursor" src="https://github.com/user-attachments/assets/669c71b8-1025-4c99-a540-04f5ada1eead" />

```
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 166.79 190" width="166.79" height="190" fill="none">
  <path
    fill="#EDECEC"
    transform="translate(-400 -395)"
    d="M563.463 439.971L487.344 396.057C484.899 394.646 481.883 394.646 479.439 396.057L403.323 439.971C401.269 441.156 400 443.349 400 445.723V534.276C400 536.647 401.269 538.843 403.323 540.029L479.443 583.943C481.887 585.353 484.903 585.353 487.347 583.943L563.466 540.029C565.521 538.843 566.79 536.651 566.79 534.276V445.723C566.79 443.352 565.521 441.156 563.466 439.971H563.463ZM558.681 449.273L485.199 576.451C484.703 577.308 483.391 576.958 483.391 575.966V492.691C483.391 491.027 482.501 489.488 481.058 488.652L408.887 447.016C408.03 446.52 408.38 445.209 409.373 445.209H556.337C558.424 445.209 559.728 447.47 558.685 449.276H558.681V449.273Z"
  />
</svg>
```

Also let me know if the trace shape is acceptable. The hard part with cursor traces is that it's all stored in a db so there's not really a standard trace format. If there's a different shape for the dataset that you would prefer feel free to let me know and I can update the cursor extraction logic accordingly.

Test repository: [armand0e/cursor-traces-example](https://huggingface.co/datasets/armand0e/cursor-traces-example)

Switching the shape to be more native by following the event based session files found in .cursor/projects. 

By themselves these session files are useless, but in the name of keeping huggingface traces as "native" exports I'm going to follow these transcript file structures.

If an end user uses teich to extract their cursor traces (`teich extract cursor`) the internal db's will also be scanned and the redacted transcripts will be populated with the raw unredacted data.
Prefer torchvision to decode images instead of PIL

This PR uses `torchvision.io.decode_image` instead of `PIL.open()` to decode images when the `torch` formatter is used.

TorchVision is typically faster than PIL to decode images, and this also saves the `pil_to_tensor` conversion. See https://github.com/huggingface/transformers/pull/45195 for benchmarks on `transfomers` where the same changes were applied.

----

Very open to any feedback regarding the implem. I went for creating a new `TorchFeaturesDecoder` class which mirrors what exists for `pandas`. There are other ways to enable this, like making the `Image` class 'formatter-aware'.

As discussed offline, I considered making `Image.decode_example()` return a lazy `PIL.Image` (no .load()) so the torch formatter could extract the path/bytes and pass them to torchvision. But this doesn't work cleanly because `PIL.Image.open(path)` without .load() keeps OS file handles open, which caused https://github.com/huggingface/datasets/issues/3985. Additionally, EXIF/mode destroys lazy state:[ `decode_example()` needs to apply EXIF transpose and mode conversion](https://github.com/NicolasHug/datasets/blame/e88a484cf05869e13f535e097ff116ffb00b2608/src/datasets/features/image.py#L194-L198). Both trigger `.load()` and produce a new image that loses filename and fp. So we can't both do EXIF/mode AND keep the image lazy.




Sounds good ! Maybe a way to make it work even on List(Image()) would be to integrate it in decode_nested_example() directly (or copy and adapt decode_nested_example())

E.g. decode_nested_example() could accept an argument that makes it use `_decode_image_for_torch` for Image() type instead of `Image.decode_example`

This way the TorchDecoder can call `decode_nested_example(self.features[column_name], value, overrides={Image: _decode_image_for_torch})` or something like that

Other that that looks good to me ! Excited by the speed ups this will bring to dataloaders 🚀 

cc @vasqu for viz, this speeds up datasets loaded with

```python
ds = load_dataset(...).with_format("torch")
# much faster than before
for image_tensor in ds["image"]:
    ...
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8254). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix fixed size binary

The original issue is a crash in _arrow_to_datasets_dtype because the miss of support of fixed_size_binary:

```
Exception:    SplitsNotFoundError
Message:      The split names could not be parsed from the dataset config.
Traceback:    Traceback (most recent call last):
                File "/usr/local/lib/python3.12/site-packages/datasets/inspect.py", line 286, in get_dataset_config_info
                  for split_generator in builder._split_generators(
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.12/site-packages/datasets/packaged_modules/parquet/parquet.py", line 118, in _split_generators
                  self.info.features = datasets.Features.from_arrow_schema(pq.read_schema(f))
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.12/site-packages/datasets/features/features.py", line 1858, in from_arrow_schema
                  else generate_from_arrow_type(field.type)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.12/site-packages/datasets/features/features.py", line 1518, in generate_from_arrow_type
                  return Value(dtype=_arrow_to_datasets_dtype(pa_type))
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.12/site-packages/datasets/features/features.py", line 121, in _arrow_to_datasets_dtype
                  raise ValueError(f"Arrow type {arrow_type} does not have a datasets dtype equivalent.")
              ValueError: Arrow type fixed_size_binary[3] does not have a datasets dtype equivalent.
              
              The above exception was the direct cause of the following exception:
              
              Traceback (most recent call last):
                File "/src/services/worker/src/worker/job_runners/config/split_names.py", line 65, in compute_split_names_from_streaming_response
                  for split in get_dataset_split_names(
                               ^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.12/site-packages/datasets/inspect.py", line 340, in get_dataset_split_names
                  info = get_dataset_config_info(
                         ^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.12/site-packages/datasets/inspect.py", line 291, in get_dataset_config_info
                  raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
              datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.

```

In this pull request I add the support for fixed size binary


Subset of #8258 
Ping @lhoestq
Fix Mypy typing invariance by updating List to Sequence in dataset loading functions (#5354)

## Description
This PR addresses issue #5354. The standard `typing.List` (or built-in `list`) is invariant in Python, meaning static analyzers like Mypy will complain if a user tries to pass a tuple or another valid sequence into the dataset loading parameters (e.g. `data_files`). By refactoring these type hints from `list` to `typing.Sequence` (which is covariant) across `from_csv`, `from_json`, `from_parquet`, and `from_text`, we satisfy type checkers for read-only sequence data paths.

This change ensures cleaner integration for downstream users who use tuples or custom sequence subclasses when loading datasets.

## Checklist
- [x] Tested locally with type hinting validation.
- [x] Code follows the repository's style guidelines.
Fix CI: commit operation equality (hfh 1.20.0) and pytest parametrize collection error

Fixes two unrelated CI failures (seen e.g. in the `huggingface_hub` 1.21.0.rc0 test run, but **the first one already affects `main` with the latest stable `huggingface_hub`**).

## 1. `tests/test_hub.py::test_delete_from_hub`

Since `huggingface_hub` 1.20.0 (introduced in https://github.com/huggingface/huggingface_hub/pull/4331), `UploadInfo` is no longer a dataclass and therefore `CommitOperationAdd` / `CommitOperationDelete` no longer implement value equality. Two commit operations with identical content are no longer considered equal, which broke:

```python
assert mock_method.call_args.kwargs.get("operations") == expected_operations
```

with an `AssertionError` between two byte-identical `CommitOperationAdd` objects.

Updated the test to compare operations attribute by attribute (type, `path_in_repo`, `is_folder`, `path_or_fileobj`) instead of relying on `==`.

## 2. `tests/test_load.py` collection error

```
TypeError: object of type 'bool' has no len()
```

A trailing comma in `@pytest.mark.parametrize("stream_from_cache, ", [False, True])` made recent `pytest` versions parse it as **two** argnames and expect each parameter value to be a 2-tuple, failing at collection time (which errors out the whole `test_load.py` module). This is unrelated to `huggingface_hub`. Fixed by dropping the trailing comma.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8283). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Hash frozensets deterministically

While looking at how transforms get fingerprinted I noticed `frozenset` isn't hashed deterministically, unlike `set`.

`set` has a custom reducer (`_save_set`) that pickles its elements sorted, so the fingerprint is stable across runs. `frozenset` has no such reducer and falls back to the default one, which keeps the iteration order. Because Python randomizes the string hash seed per process, a transform that captures a `frozenset` of strings gets a different fingerprint every session, so the cache never hits.

Repro:
```python
from datasets.fingerprint import Hasher
# prints a different hash on every run (PYTHONHASHSEED is random by default)
print(Hasher.hash(frozenset("abcdefghij")))
```

I registered the same sorted reducer for `frozenset` that `set` already uses (falling back to sorting by `Hasher.hash` for unorderable elements). Added a test that hashes a frozenset in two subprocesses with different `PYTHONHASHSEED` and checks the hashes match, mirroring the existing `set` coverage.
fix: pass hub token when embedding external files during push_to_hub

## Summary
- Fixes gated-repo failures when `Dataset.push_to_hub(..., embed_external_files=True)` embeds `hf://` image/audio paths into Parquet shards.
- `_push_parquet_shards_to_hub_single` now builds a `token_per_repo_id` map from shard paths (and destination repo metadata) and passes it to `embed_table_storage`, matching the existing `IterableDataset.push_to_hub` behavior.

## Root cause
`IterableDataset.push_to_hub` already used `partial(embed_table_storage, token_per_repo_id=...)`, but the map-style `Dataset` push path called `embed_table_storage` bare, so downloads from gated source repos failed during parquet conversion.

## Test plan
- [x] Added `test_get_token_per_repo_id_for_embed` unit test
- [ ] Manual: push dataset with `hf://` image paths from a gated repo using `token=...`

Fixes #6348
datasets.load_from_disk progress bar optional manual control

### Feature request

This is tangentially related to [https://github.com/huggingface/datasets/issues/7918](https://github.com/huggingface/datasets/issues/7918).

When loading a dataset with > 16 files a progress bar is shown (unless stdout is redirected or [https://github.com/huggingface/datasets/pull/7919](https://github.com/huggingface/datasets/pull/7919) is merged).

However, if you use multiple processes with data sharding, where each core loads the dataset, you get multiple copies of the progress bar (all fighting each other). It would be greatly appreciated if `datasets.load_from_disk` accepted an argument for whether to show a progress bar; default could be `None` which would retain current functionality (i.e. show if > 16 files in dataset), but user could also force the progress bar on or off as needed. Essentially just expose the progress bar visibility argument to the method's argument so that user can control it instead of it being hardcoded, where `None` would be default argument and retain current functionality.

### Motivation

Progress bar could be forced off in all processes than one, to avoid progress bar fighting and log spam.
Progress bar could also be manually forced on and off for other use cases.

### Your contribution

Possibly do a PR if this is accepted.

Labels: name: enhancement
Add regression test suite for datasets (QA)

## What does this PR do?

Adds a data-driven manual/semi-automated **regression test suite** for `datasets` at `tests/regression/regression_suite.md` (128 test cases).

The suite was built by mining 984 historical issues (2020-04 → 2021-09) to find the bug categories that keep repeating, cross-checking the currently-open issues via the GitHub Search API, and mapping the result onto the documented QA focus areas. It defines a 30-minute smoke pass (24 P0 cases) and a 180-minute full regression run, each case with preconditions, numbered steps, and expected results, plus a traceability matrix back to specific issues.

### Categories driving the most bugs (historical)
1. Data loading (48%) 2. Caching (29%) 3. Splits/slicing (29%) 4. Arrow serialization (26%)

### Coverage gaps filled
- `trust_remote_code` (post-dates the historical snapshot; 36 open today)
- Streaming (low historical volume but high-growth) — deliberately over-weighted
- Multiprocessing hangs/freezes (e.g. #6936, #6357)

Before submitting this PR, please make sure you have read the contribution guidelines.
Fix docstring example for select_columns in IterableDatasetDict

Fix docstring example for select_columns in IterableDatasetDict.

This PR updates the usage example for selecting columns in a streaming dataset. The example now correctly uses the `select_columns` method instead of the outdated `select` method.

### Changes

- **Documentation update:**
  * Updated the example in the docstring of `select_columns` in `dataset_dict.py` to use `select_columns("text")` instead of the deprecated `select("text")`.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_8290). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: skip filename-based split detection for media files in get_data_…

Fixes #7201

## Problem
When a user has an image file named `train.png` in a flat folder, 
the library mistakenly treats it as a split name instead of an image,
resulting in only that one file being loaded.

## Fix
Modified `_get_data_files_patterns` in `data_files.py` to skip 
filename-based split detection when all matched files are media files
(images, audio, video). This causes it to fall through to 
`DEFAULT_PATTERNS_ALL` which correctly loads all files.

## Test
Added `test_data_files_with_image_named_after_split` to verify
that all images are loaded when one is named after a split.
docs(py_utils): add missing docstrings to 5 public utility functions

## What does this PR do?

Adds missing docstrings to five public utility functions in `src/datasets/utils/py_utils.py`. Each docstring follows the HuggingFace documentation style with `Args`, `Returns`, `Raises`, and `Example` sections.

**Functions documented:**

| Function | What it does |
|---|---|
| `glob_pattern_to_regex` | Converts a glob pattern to a regex string |
| `has_sufficient_disk_space` | Checks whether a directory has enough free space |
| `copyfunc` | Returns a shallow copy of a Python function |
| `iflatmap_unordered` | Parallel flatmap yielding results in completion order |
| `iter_batched` | Iterates over an iterable in fixed-size batches |

## Motivation

These functions are used throughout the library but lacked any documentation, making it harder for contributors and users to understand their behaviour without reading the implementation.

## Tests

No logic changed — documentation only. Existing test suite covers these functions.

`ruff check` and `ruff format --check` both pass with no errors.