AniketTec/github-issues huggingface dataset:
Title: feat: implement iter_arrow for skip, take and step iterables
State: open

This commit optimizes streaming operations by implementing [_iter_arrow](cci:1://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:377:4-391:57) for [SkipExamplesIterable](cci:2://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:1798:0-1892:42), [TakeExamplesIterable](cci:2://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:1944:0-2048:42), and [StepExamplesIterable](cci:2://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:667:0-743:42).

### Key Changes:
- **Fast Batch Processing**: Enabled batch-level slicing for `.skip(n)` and `.take(n)` on streaming datasets, bypassing slow row-by-row iteration.
- **Optimized Sharding**: Updated [StepExamplesIterable](cci:2://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:667:0-743:42) (used in distributed training) to use Arrow's `.take()` to extract multiple records from a batch simultaneously.
- **State Preservation**: Reinforced [_init_state_dict](cci:1://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:505:4-514:31) and [load_state_dict](cci:1://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:2382:4-2430:46) to support flawless checkpointing and resumption while using Arrow iteration.

### Performance Impact:
Users will experience significant performance gains when skipping or taking examples in streaming mode. By staying in the "Arrow path" and avoiding Python dictionary conversions, data loading overhead is drastically reduced, especially for large-scale training jobs.

### Testing:
Integrated 6 new unit tests into [tests/test_iterable_dataset.py](cci:7://file:///c:/Users/ASUS/Desktop/datasets/tests/test_iterable_dataset.py:0:0-0:0) to verify:
- Functional correctness for [skip](cci:1://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:3149:4-3191:9), [take](cci:1://file:///c:/Users/ASUS/Desktop/datasets/src/datasets/iterable_dataset.py:3236:4-3271:9), and `step` using Arrow iteration.
- Reliable state checkpointing and resumption after partial iteration.

Comments:
Title: push_to_hub() for videos
State: closed

possible now that row group sizes are auto-determined based on the content size after https://github.com/huggingface/datasets/pull/7589

Videos are uploaded as PLAIN in Parquet to make sure they can be seeked remotely and with random access to frames in https://github.com/huggingface/datasets/pull/7976

In the future it could be cool to have the same behavior as when videos are separate files, i.e. lazily load them instead of downloading them completely in streaming mode.

Right now there is this discrepency:
- `load_dataset("username/my-folder-of-videos", streaming=True)` -> videos are lazy loaded one by one when iterating, and only actually downloaded when accessing frames in `torchcodec`
- `load_dataset("username/my-video-dataset-in-parquet", streaming=True)` -> videos are downloaded one by one when iterating, even if no frame is accessed in `torchcodec`

close https://github.com/huggingface/datasets/issues/7493

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7971). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Count examples in lance
State: closed

```python
In [1]: from datasets import load_dataset_builder, StreamingDownloadManager

In [2]: b = load_dataset_builder("lance-format/openvid-lance")
Resolving data files: 100%|█| 240/240 [00:00<00:00, 42675.64it/s

In [3]: b.count_examples(StreamingDownloadManager())
Out[3]: {'train': 937957}
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7969). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Potential conflicting type checks and dead code in `/src/datasets/table.py`
State: open

When statically analyzing and manually reviewing the code, I noticed a potential logic conflicting in `/src/datasets/table.py` as follows:

```
def to_blocks(table: Union[pa.Table, Table]) -> list[list[TableBlock]]:
      if isinstance(table, pa.Table):
          return [[InMemoryTable(table)]]
      elif isinstance(table, ConcatenationTable):  # dead code
          return copy.deepcopy(table.blocks)
      else:
          return [[table]]
```

Within the function, the condition `isinstance(table, ConcatenationTable)` at line 4 will never be True because the previous condition `isinstance(table, pa.Table)` at line 2 would have already caught all instances of `ConcatenationTable` (since `ConcatenationTable` is a subtype of `pa.Table`). This creates a logical conflict in the type checking flow.

Please verify if this logic is intentional or it is an issue warranting a refactoring or fixing.

Comments:
ConcatenationTable is a subclass of datasets.table.Table but not pa.Table, so it should be fine
Title: Infer types from lance blobs
State: closed

Ex: infer Video() type in https://huggingface.co/datasets/lance-format/openvid-lance and Image() type in https://huggingface.co/datasets/lance-format/laion-1m

```python
from datasets import load_dataset

ds = load_dataset("lance-format/laion-1m", streaming=True, split="train")
print(ds.feature["image"])
# Image()

ds = load_dataset("lance-format/openvid-lance", streaming=True, split="train")
print(ds.feature["video_blob"])
# Video()
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7966). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: `huggingface_hub.errors.HfHubHTTPError: 404 Client Error: Not Found for url` when fetching a dataset with `datasets.load_dataset`
State: closed

Not a bug but a question. We started getting the following error:

https://github.com/mlflow/mlflow/actions/runs/21368603305/job/61506951617

```
ests/data/test_huggingface_dataset_and_source.py::test_from_huggingface_dataset_constructs_expected_dataset_with_revision - huggingface_hub.errors.HfHubHTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/datasets/rotten_tomatoes/revision/aa13bc287fa6fcab6daf52f0dfb9994269ffea28 (Request ID: Root=1-6977aeca-35bc2b5b605884926a9224d0;aa2391f3-26e8-4975-b9bb-114b2fa40223)
```

Adding a user id fixed the issue (https://github.com/mlflow/mlflow/pull/20350). `https://huggingface.co/api/datasets` no longer accepts a name-only path like `rotten_tomatoes`? Just wondering what changed. Thanks!

Comments:
Hi ! Yes you should use `cornell-movie-review-data/rotten_tomatoes` instead of `rotten_tomatoes`, which is the legacy name. Those datasets have been moved under their actual owners accounts some time ago (but we were keeping the old names as aliases)

Some other impacted names are:
- `imdb` -> `stanfordnlp/imdb`
- `wikitext` -> `Salesforce/wikitext`
- `gsm8k` -> `openai/gsm8k`
- `winogrande` -> `allenai/winogrande`

We're working on re-enabling them as aliases for backward compatibility. I'll post updates here, sorry for the inconvenience.

**Using the actual name instead of the old legacy name is more future proof though**

Thanks for the heads up @lhoestq ! fyi, this change is likely breaking a lot of repos that have legacy names hardcoded ([example](https://github.com/allenai/olmes/pull/40)) Would be helpful to many to share this update in a more visible way if it is likely to persist for a while.

[internal tracking link](https://github.com/huggingface-internal/moon-landing/pull/16539)

@lhoestq Thanks for clarifying!

The aliases are re-enabled :)

Thanks!

Can I close this issue?

Yep :)
Title: Support null in json string cols
State: closed

fix for https://huggingface.co/datasets/arcprize/arc_agi_v2_public_eval

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7963). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Use Sequence instead of list in Dataset.from_parquet type hints
State: open

This PR updates type annotations in Dataset.from_parquet to use Sequence instead of list to avoid mypy invariant type issues, as discussed in issue #5354.


Comments:
Thank you for the review!

I’ve updated the implementation to:
- Use Sequence from collections.abc as per project conventions
- Restore backward compatibility with Union[PathLike, Sequence[PathLike]]
- Keep the columns annotation as Optional[Sequence[str]]

The fixes are pushed. Please let me know if anything else is needed.
Title: Revert "feat: avoid some copies in torch formatter (#7787)"
State: closed

This reverts commit c412a6f5a50955e141c5169bf7abe005d10228d2 (I assume it was ai generated which makes it hard for me to review and make sure it doesn't have bad edge cases, but lmk if it wasn't, anyways it didn't take into account the torch kwargs which are responsible for sending the data to the correct device)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7961). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: [CUDA Tensors Not working in ~v4.5.0] set_format(type="torch", device="cuda") returns cpu
State: open

### Describe the bug

The problem is that when calling:
```ds.set_format(type="torch", columns = ["input", "labels"], device="cuda")```
The device type of the individual datapoints is now: `cpu` as opposed to `cuda:0`.

With `v4.0.0` it still works. With `v4.5.0` it doesn't work anymore.


Related Issue:
https://github.com/huggingface/datasets/issues/1762

### Steps to reproduce the bug

Steps to reproduce the bug:

```
ds.set_format(type="torch", columns = ["input", "labels"], device="cuda")
print(ds["train"][0]["input"].device) #outputs cpu
```

The above should be `cuda:0`.

### Expected behavior

`set_format` should be able to move data to GPU.

### Environment info

datasets==4.5.0

Comments:
@lhoestq tagging you here as you were on the previous issue, hope that's fine. 

Thanks for reporting, let me take a look

I reverted this change which caused the issue #7961 , I'll do a new release soon but in the meantime feel free to install `datasets` from source

@lhoestq thanks a lot. I am actually checking older versions of datasets and it seems that it doesn't work with 4.2.0 as well.

(hopefully that's relevant).
Title: Fix all exhausted without replacement
State: closed

Fix interleave_datasets "all_exhausted_without_replacement" stopping strategy

- Corrected logic to ensure each sample is picked exactly once when using 
  stopping_strategy="all_exhausted_without_replacement".
- Adjusted boolean stopping condition to properly track dataset exhaustion.
- Added test to verify the last element is included as expected.


Comments:
Hi maintainers 👋  
This PR fixes `all_exhausted_without_replacement` to ensure all samples are emitted exactly once,  
and adds a regression test reproducing the reported issue.

Happy to adjust the implementation or split changes if you’d prefer.
Thanks!


We just merged https://github.com/huggingface/datasets/pull/7955 which also fix the issue, thanks for investigating !

I'm closing this one if you don't mind
Title: Is the 10k files / folder limit a hard limit for a dataset repo?
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

Can the hard limit of 10k files per folder be extended with acceptable loss in performance?

### Motivation

I'm uploading a lance dataset to huggingface hub and have a folder inside lance internals (`data/*.lance/_transactions`) that has > 20k atomic transaction records and my commits are being rejected.

```
Bad request for commit endpoint:
Your push was rejected because it contains too many files per directory. Each directory in your git repo can only contain up to 10000 files. Offending reference: refs/heads/main Offending directories: /data/expression.lance/_transactions/
```


### Your contribution

Open to suggestions for how to make a PR

Comments:
Yes, that's a hard limit. Can you split your files into different folders? Or we'll probably have a new repo type in the near to mid future that will relax this limit a bit. 

Thanks! Working around this with a different sharding parameter to have fewer overall fragments (and therefore fewer files in `*.lance/data/` and `*.lance/_transactions/`
Title: Fix interleave_datasets with all_exhausted_without_replacement strategy
State: closed

When using interleave_datasets with stopping_strategy="all_exhausted_without_replacement" and probabilities=None, the function was incorrectly falling into the undersampling branch, causing it to stop at min(lengths) instead of continuing until all datasets were exhausted.

This fix adds a specific branch to handle the all_exhausted_without_replacement case when probabilities=None. The new logic cycles through all datasets round by round, adding elements from each dataset until all are exhausted, ensuring each element appears exactly once.

Example fix:
- Input: d1=[0,1,2], d2=[10,11,12,13], d3=[20,21,22]
- Before: [0, 10, 20, 1, 11, 21, 2, 12, 22]
- After: [0, 10, 20, 1, 11, 21, 2, 12, 22, 13]

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7955). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: all_exhausted_without_replacement working same as first_exhausted
State: open

### Describe the bug

>>> from datasets import Dataset, interleave_datasets
>>> d1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> d2 = Dataset.from_dict({"a": [10, 11, 12, 13]})
>>> d3 = Dataset.from_dict({"a": [20, 21, 22]})
>>> dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted_without_replacement")
>>> dataset["a"][:100]
[0, 10, 20, 1, 11, 21, 2, 12, 22]

expected output: [0, 10, 20, 1, 11, 21, 2, 12, 22,13]

datasets version 4.5.0

### Steps to reproduce the bug

>>> from datasets import Dataset, interleave_datasets
>>> d1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> d2 = Dataset.from_dict({"a": [10, 11, 12, 13]})
>>> d3 = Dataset.from_dict({"a": [20, 21, 22]})
>>> dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted_without_replacement")
>>> dataset["a"][:100]

### Expected behavior

 [0, 10, 20, 1, 11, 21, 2, 12, 22,13]

### Environment info

datasets version 4.5.0

Comments:
https://github.com/huggingface/datasets/pull/7955
Title: feat: Add GenBank file format support for biological sequence data
State: open

## Summary
Add native support for loading GenBank (.gb, .gbk, .genbank) files, a standard format for biological sequence data with annotations maintained by NCBI.

## Changes
- Add `genbank` packaged module with pure Python state machine parser
- Register GenBank extensions in `_PACKAGED_DATASETS_MODULES` and `_EXTENSION_TO_MODULE`
- Add comprehensive test suite (28 tests)

## Features
- **Metadata parsing**: LOCUS, DEFINITION, ACCESSION, VERSION, KEYWORDS, ORGANISM, taxonomy
- **Feature parsing**: Structured JSON output with location parsing (complement, join)
- **Sequence parsing**: ORIGIN section with automatic length calculation
- **Compression support**: gzip, bz2, xz via magic bytes detection
- **Memory efficiency**: Dual-threshold batching (batch_size + max_batch_bytes)
- **Large sequences**: Uses `large_string` Arrow type for sequences/features

## Usage
```python
from datasets import load_dataset

# Load GenBank files
ds = load_dataset("genbank", data_files="sequences.gb")

# With options
ds = load_dataset("genbank", data_files="*.gbk", 
                  columns=["sequence", "organism", "features"],
                  parse_features=True)
```

## Test plan
- [x] All 28 unit tests pass
- [x] Tests cover: basic loading, multi-record, compression, feature parsing, column filtering, batching, schema types

Comments:
Title: Add examples for Lance datasets
State: closed

## Summary

Updated the Lance integration docs to match the official dataset cards and expand coverage of multimodal workflows in `use_with_lance.mdx`.

## Details

- Added a Lance format-focused guidance page for multimodal examples:
   - Stream from the Hub via `datasets` API
   - Use Lance's dataset API to pass in a `hf://...` path specifier,
   - Scan, vector search, schema evolution examples
   - "Export to Lance" example for materializing filtered subsets locally
- Aligned all dataset names/paths to existing datasets so users can immediately test it out: `lance-format/laion-1m` (image) and the `lance-format/openvid-lance` (video)

Depending on the questions we get from users, we may add more descriptive examples as we along, but this should be a good start!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7950). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @prrao87 ! All the main examples with pylance should be in their dedicated documentation page IMO. Let's document Lance as a supported format in the `datasets` docs, and move the `pylance` docs and examples there instead if you don't mind: https://github.com/huggingface/hub-docs/pull/2164

I think for the `datasets` docs we can simply make sure Lance is mentioned in the lists of supported formats, and have a section in the Video documentation pages alongside WebDataset for example:

* [video_load.md](https://huggingface.co/docs/datasets/video_load) for loading 
* [video_dataset.md](https://huggingface.co/docs/datasets/video_dataset) for dataset creation
* [loading_methods.md](https://huggingface.co/docs/datasets/package_reference/loading_methods) the reference which lists all loading methods and configs

Hi @lhoestq, what do you mean by this?
> Let's document Lance as a supported format in the datasets docs

Where exactly would Lance be mentioned? I wasn't able to find a section titled "supported formats" in the docs. The rest of the parts of your response made sense (can make those updates). Thanks!

Hi @lhoestq, I've updated the docs in various places, to show how to use Lance for image creation/loading and also video creation/loading. I've also removed the `use_with_lance.mdx` page's contents over to the Hub docs by enhancing that page, as you mentioned.

I think this should cover all your feedback points! Please let me know if anything's missing. I'll be back in a future PR with audio examples, I think the HF community would benefit from using Lance format for audio storage/retrieval, too.

Thanks! We'll get those fixed and send more PRs ;)
Title: json: add optional return_file_name parameter
State: open

This PR adds an optional `return_file_name` parameter to the JSON dataset loader.

When enabled, a new `file_name` column is added containing the source file name
for each row. Default behavior is unchanged.

Changes:
- Add `return_file_name` to JsonConfig
- Append file name during JSON table generation
- Add tests covering default and enabled behavior, and ensures other functions are not affected

Motivation:
This helps resume training from checkpoints by identifying already-consumed data shards.

Fixes #5806


Comments:
Title: Add _generate_shards
State: closed

Useful to list a dataset's shards:

```python
from datasets import load_dataset_builder, StreamingDownloadManager

dlm = StreamingDownloadManager()

def get_shards(dataset_name, *args, **kwargs):
    b = load_dataset_builder(dataset_name, *args, **kwargs)
    splits = b._split_generators(dlm)
    return list(b._generate_shards(**splits[0].gen_kwargs))

print(get_shards("username/dataset_name"))
# ['hf://datasets/...', ...]
```

I'll use this in combination with https://github.com/huggingface/datasets/pull/7897 in the Dataset Viewer for the and API endpoint that does {dataset, config, split, offset, limit} -> [{fileUri, offset, limit}]. This will be useful to edit datasets since your can get a row's location inside a dataset cc @cfahlgren1 

This will be similar to https://github.com/huggingface/dataset-viewer/pull/3276 but works for any dataset format: csv, json, webdataset, images etc.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7943). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Improve readability and documentation of indexing integration tests
State: open

### Summary
This PR improves the readability and maintainability of the indexing integration tests by adding clear, detailed comments throughout the test suite.

### Motivation
The indexing tests cover multiple backends (FAISS and Elasticsearch) and involve non-trivial workflows such as vector creation, indexing, querying, and serialization. Adding explanatory comments helps new contributors and reviewers understand the intent of each test case more easily.

### What’s Changed
- Added descriptive docstrings to test classes and methods
- Included inline comments explaining:
  - Dataset construction
  - Index creation and configuration
  - Search and batch search behavior
  - Error handling and validation logic
  - Serialization and cleanup steps
- No functional or behavioral changes

### Scope
- Documentation and readability improvements only
- No changes to test logic, APIs, or expected behavior

### Impact
This change lowers the barrier for new contributors, improves code comprehension, and makes future maintenance easier without affecting test coverage or performance.

### Checklist
- [x] No functional changes introduced
- [x] Tests pass locally
- [x] Follows existing project style and conventions


Comments:
Title: datasets.load_from_disk progress bar optional manual control
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

This is tangentially related to [https://github.com/huggingface/datasets/issues/7918](https://github.com/huggingface/datasets/issues/7918).

When loading a dataset with > 16 files a progress bar is shown (unless stdout is redirected or [https://github.com/huggingface/datasets/pull/7919](https://github.com/huggingface/datasets/pull/7919) is merged).

However, if you use multiple processes with data sharding, where each core loads the dataset, you get multiple copies of the progress bar (all fighting each other). It would be greatly appreciated if `datasets.load_from_disk` accepted an argument for whether to show a progress bar; default could be `None` which would retain current functionality (i.e. show if > 16 files in dataset), but user could also force the progress bar on or off as needed. Essentially just expose the progress bar visibility argument to the method's argument so that user can control it instead of it being hardcoded, where `None` would be default argument and retain current functionality.

### Motivation

Progress bar could be forced off in all processes than one, to avoid progress bar fighting and log spam.
Progress bar could also be manually forced on and off for other use cases.

### Your contribution

Possibly do a PR if this is accepted.

Comments: