NyssaW/github-issues huggingface dataset:
`huggingface_hub.errors.HfHubHTTPError: 404 Client Error: Not Found for url` when fetching a dataset with `datasets.load_dataset`

Not a bug but a question. We started getting the following error:

https://github.com/mlflow/mlflow/actions/runs/21368603305/job/61506951617

```
ests/data/test_huggingface_dataset_and_source.py::test_from_huggingface_dataset_constructs_expected_dataset_with_revision - huggingface_hub.errors.HfHubHTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/datasets/rotten_tomatoes/revision/aa13bc287fa6fcab6daf52f0dfb9994269ffea28 (Request ID: Root=1-6977aeca-35bc2b5b605884926a9224d0;aa2391f3-26e8-4975-b9bb-114b2fa40223)
```

Adding a user id fixed the issue (https://github.com/mlflow/mlflow/pull/20350). `https://huggingface.co/api/datasets` no longer accepts a name-only path like `rotten_tomatoes`? Just wondering what changed. Thanks!

Comments:
Hi ! Yes you should use `cornell-movie-review-data/rotten_tomatoes` instead of `rotten_tomatoes`, which is the legacy name. Those datasets have been moved under their actual owners accounts some time ago (but we were keeping the old names as aliases)

Some other impacted names are:
- `imdb` -> `stanfordnlp/imdb`
- `wikitext` -> `Salesforce/wikitext`
- `gsm8k` -> `openai/gsm8k`
- `winogrande` -> `allenai/winogrande`

We're working on re-enabling them as aliases for backward compatibility. I'll post updates here, sorry for the inconvenience.

**Using the actual name instead of the old legacy name is more future proof though**

Thanks for the heads up @lhoestq ! fyi, this change is likely breaking a lot of repos that have legacy names hardcoded ([example](https://github.com/allenai/olmes/pull/40)) Would be helpful to many to share this update in a more visible way if it is likely to persist for a while.

[internal tracking link](https://github.com/huggingface-internal/moon-landing/pull/16539)

@lhoestq Thanks for clarifying!

The aliases are re-enabled :)

Thanks!

Can I close this issue?

Yep :)
Use Sequence instead of list in Dataset.from_parquet type hints

This PR updates type annotations in Dataset.from_parquet to use Sequence instead of list to avoid mypy invariant type issues, as discussed in issue #5354.


Comments:
Thank you for the review!

I’ve updated the implementation to:
- Use Sequence from collections.abc as per project conventions
- Restore backward compatibility with Union[PathLike, Sequence[PathLike]]
- Keep the columns annotation as Optional[Sequence[str]]

The fixes are pushed. Please let me know if anything else is needed.
Revert "feat: avoid some copies in torch formatter (#7787)"

This reverts commit c412a6f5a50955e141c5169bf7abe005d10228d2 (I assume it was ai generated which makes it hard for me to review and make sure it doesn't have bad edge cases, but lmk if it wasn't, anyways it didn't take into account the torch kwargs which are responsible for sending the data to the correct device)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7961). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[CUDA Tensors Not working in ~v4.5.0] set_format(type="torch", device="cuda") returns cpu

### Describe the bug

The problem is that when calling:
```ds.set_format(type="torch", columns = ["input", "labels"], device="cuda")```
The device type of the individual datapoints is now: `cpu` as opposed to `cuda:0`.

With `v4.0.0` it still works. With `v4.5.0` it doesn't work anymore.


Related Issue:
https://github.com/huggingface/datasets/issues/1762

### Steps to reproduce the bug

Steps to reproduce the bug:

```
ds.set_format(type="torch", columns = ["input", "labels"], device="cuda")
print(ds["train"][0]["input"].device) #outputs cpu
```

The above should be `cuda:0`.

### Expected behavior

`set_format` should be able to move data to GPU.

### Environment info

datasets==4.5.0

Comments:
@lhoestq tagging you here as you were on the previous issue, hope that's fine. 

Thanks for reporting, let me take a look

I reverted this change which caused the issue #7961 , I'll do a new release soon but in the meantime feel free to install `datasets` from source

@lhoestq thanks a lot. I am actually checking older versions of datasets and it seems that it doesn't work with 4.2.0 as well.

(hopefully that's relevant).

This has been fixed in the latest version :)
Fix all exhausted without replacement

Fix interleave_datasets "all_exhausted_without_replacement" stopping strategy

- Corrected logic to ensure each sample is picked exactly once when using 
  stopping_strategy="all_exhausted_without_replacement".
- Adjusted boolean stopping condition to properly track dataset exhaustion.
- Added test to verify the last element is included as expected.


Comments:
Hi maintainers 👋  
This PR fixes `all_exhausted_without_replacement` to ensure all samples are emitted exactly once,  
and adds a regression test reproducing the reported issue.

Happy to adjust the implementation or split changes if you’d prefer.
Thanks!


We just merged https://github.com/huggingface/datasets/pull/7955 which also fix the issue, thanks for investigating !

I'm closing this one if you don't mind
Is the 10k files / folder limit a hard limit for a dataset repo?

### Feature request

Can the hard limit of 10k files per folder be extended with acceptable loss in performance?

### Motivation

I'm uploading a lance dataset to huggingface hub and have a folder inside lance internals (`data/*.lance/_transactions`) that has > 20k atomic transaction records and my commits are being rejected.

```
Bad request for commit endpoint:
Your push was rejected because it contains too many files per directory. Each directory in your git repo can only contain up to 10000 files. Offending reference: refs/heads/main Offending directories: /data/expression.lance/_transactions/
```


### Your contribution

Open to suggestions for how to make a PR

Comments:
Yes, that's a hard limit. Can you split your files into different folders? Or we'll probably have a new repo type in the near to mid future that will relax this limit a bit. 

Thanks! Working around this with a different sharding parameter to have fewer overall fragments (and therefore fewer files in `*.lance/data/` and `*.lance/_transactions/`
Fix interleave_datasets with all_exhausted_without_replacement strategy

When using interleave_datasets with stopping_strategy="all_exhausted_without_replacement" and probabilities=None, the function was incorrectly falling into the undersampling branch, causing it to stop at min(lengths) instead of continuing until all datasets were exhausted.

This fix adds a specific branch to handle the all_exhausted_without_replacement case when probabilities=None. The new logic cycles through all datasets round by round, adding elements from each dataset until all are exhausted, ensuring each element appears exactly once.

Example fix:
- Input: d1=[0,1,2], d2=[10,11,12,13], d3=[20,21,22]
- Before: [0, 10, 20, 1, 11, 21, 2, 12, 22]
- After: [0, 10, 20, 1, 11, 21, 2, 12, 22, 13]

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7955). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
all_exhausted_without_replacement working same as first_exhausted

### Describe the bug

>>> from datasets import Dataset, interleave_datasets
>>> d1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> d2 = Dataset.from_dict({"a": [10, 11, 12, 13]})
>>> d3 = Dataset.from_dict({"a": [20, 21, 22]})
>>> dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted_without_replacement")
>>> dataset["a"][:100]
[0, 10, 20, 1, 11, 21, 2, 12, 22]

expected output: [0, 10, 20, 1, 11, 21, 2, 12, 22,13]

datasets version 4.5.0

### Steps to reproduce the bug

>>> from datasets import Dataset, interleave_datasets
>>> d1 = Dataset.from_dict({"a": [0, 1, 2]})
>>> d2 = Dataset.from_dict({"a": [10, 11, 12, 13]})
>>> d3 = Dataset.from_dict({"a": [20, 21, 22]})
>>> dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted_without_replacement")
>>> dataset["a"][:100]

### Expected behavior

 [0, 10, 20, 1, 11, 21, 2, 12, 22,13]

### Environment info

datasets version 4.5.0

Comments:
https://github.com/huggingface/datasets/pull/7955

it was fixed by https://github.com/huggingface/datasets/pull/7955
#5354: replace list with Sequence in from_parquet type hints

\This PR replaces `list` type hints with `Sequence` in `from_parquet` to improve type checking.

Note: Local pytest errors on Python 3.13 due to removal of `distutils` are unrelated to this change.


Comments:
Hi! I'd like to work on this.
I noticed PR #7953 has been open since January with merge conflicts and no maintainer review, and only covers from_parquet in arrow_dataset.py. I'd like to open a more complete PR replacing List with Sequence across the broader codebase in src/datasets/.
Happy to start immediately if a maintainer can confirm the preferred scope.
#self-assign
feat: Add GenBank file format support for biological sequence data

## Summary
Add native support for loading GenBank (.gb, .gbk, .genbank) files, a standard format for biological sequence data with annotations maintained by NCBI.

## Changes
- Add `genbank` packaged module with pure Python state machine parser
- Register GenBank extensions in `_PACKAGED_DATASETS_MODULES` and `_EXTENSION_TO_MODULE`
- Add comprehensive test suite (28 tests)

## Features
- **Metadata parsing**: LOCUS, DEFINITION, ACCESSION, VERSION, KEYWORDS, ORGANISM, taxonomy
- **Feature parsing**: Structured JSON output with location parsing (complement, join)
- **Sequence parsing**: ORIGIN section with automatic length calculation
- **Compression support**: gzip, bz2, xz via magic bytes detection
- **Memory efficiency**: Dual-threshold batching (batch_size + max_batch_bytes)
- **Large sequences**: Uses `large_string` Arrow type for sequences/features

## Usage
```python
from datasets import load_dataset

# Load GenBank files
ds = load_dataset("genbank", data_files="sequences.gb")

# With options
ds = load_dataset("genbank", data_files="*.gbk", 
                  columns=["sequence", "organism", "features"],
                  parse_features=True)
```

## Test plan
- [x] All 28 unit tests pass
- [x] Tests cover: basic loading, multi-record, compression, feature parsing, column filtering, batching, schema types

Comments:
Add examples for Lance datasets

## Summary

Updated the Lance integration docs to match the official dataset cards and expand coverage of multimodal workflows in `use_with_lance.mdx`.

## Details

- Added a Lance format-focused guidance page for multimodal examples:
   - Stream from the Hub via `datasets` API
   - Use Lance's dataset API to pass in a `hf://...` path specifier,
   - Scan, vector search, schema evolution examples
   - "Export to Lance" example for materializing filtered subsets locally
- Aligned all dataset names/paths to existing datasets so users can immediately test it out: `lance-format/laion-1m` (image) and the `lance-format/openvid-lance` (video)

Depending on the questions we get from users, we may add more descriptive examples as we along, but this should be a good start!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7950). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @prrao87 ! All the main examples with pylance should be in their dedicated documentation page IMO. Let's document Lance as a supported format in the `datasets` docs, and move the `pylance` docs and examples there instead if you don't mind: https://github.com/huggingface/hub-docs/pull/2164

I think for the `datasets` docs we can simply make sure Lance is mentioned in the lists of supported formats, and have a section in the Video documentation pages alongside WebDataset for example:

* [video_load.md](https://huggingface.co/docs/datasets/video_load) for loading 
* [video_dataset.md](https://huggingface.co/docs/datasets/video_dataset) for dataset creation
* [loading_methods.md](https://huggingface.co/docs/datasets/package_reference/loading_methods) the reference which lists all loading methods and configs

Hi @lhoestq, what do you mean by this?
> Let's document Lance as a supported format in the datasets docs

Where exactly would Lance be mentioned? I wasn't able to find a section titled "supported formats" in the docs. The rest of the parts of your response made sense (can make those updates). Thanks!

Hi @lhoestq, I've updated the docs in various places, to show how to use Lance for image creation/loading and also video creation/loading. I've also removed the `use_with_lance.mdx` page's contents over to the Hub docs by enhancing that page, as you mentioned.

I think this should cover all your feedback points! Please let me know if anything's missing. I'll be back in a future PR with audio examples, I think the HF community would benefit from using Lance format for audio storage/retrieval, too.

Thanks! We'll get those fixed and send more PRs ;)
json: add optional return_file_name parameter

This PR adds an optional `return_file_name` parameter to the JSON dataset loader.

When enabled, a new `file_name` column is added containing the source file name
for each row. Default behavior is unchanged.

Changes:
- Add `return_file_name` to JsonConfig
- Append file name during JSON table generation
- Add tests covering default and enabled behavior, and ensures other functions are not affected

Motivation:
This helps resume training from checkpoints by identifying already-consumed data shards.

Fixes #5806


Comments:
Add _generate_shards

Useful to list a dataset's shards:

```python
from datasets import load_dataset_builder, StreamingDownloadManager

dlm = StreamingDownloadManager()

def get_shards(dataset_name, *args, **kwargs):
    b = load_dataset_builder(dataset_name, *args, **kwargs)
    splits = b._split_generators(dlm)
    return list(b._generate_shards(**splits[0].gen_kwargs))

print(get_shards("username/dataset_name"))
# ['hf://datasets/...', ...]
```

I'll use this in combination with https://github.com/huggingface/datasets/pull/7897 in the Dataset Viewer for the and API endpoint that does {dataset, config, split, offset, limit} -> [{fileUri, offset, limit}]. This will be useful to edit datasets since your can get a row's location inside a dataset cc @cfahlgren1 

This will be similar to https://github.com/huggingface/dataset-viewer/pull/3276 but works for any dataset format: csv, json, webdataset, images etc.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7943). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Improve readability and documentation of indexing integration tests

### Summary
This PR improves the readability and maintainability of the indexing integration tests by adding clear, detailed comments throughout the test suite.

### Motivation
The indexing tests cover multiple backends (FAISS and Elasticsearch) and involve non-trivial workflows such as vector creation, indexing, querying, and serialization. Adding explanatory comments helps new contributors and reviewers understand the intent of each test case more easily.

### What’s Changed
- Added descriptive docstrings to test classes and methods
- Included inline comments explaining:
  - Dataset construction
  - Index creation and configuration
  - Search and batch search behavior
  - Error handling and validation logic
  - Serialization and cleanup steps
- No functional or behavioral changes

### Scope
- Documentation and readability improvements only
- No changes to test logic, APIs, or expected behavior

### Impact
This change lowers the barrier for new contributors, improves code comprehension, and makes future maintenance easier without affecting test coverage or performance.

### Checklist
- [x] No functional changes introduced
- [x] Tests pass locally
- [x] Follows existing project style and conventions


Comments:
datasets.load_from_disk progress bar optional manual control

### Feature request

This is tangentially related to [https://github.com/huggingface/datasets/issues/7918](https://github.com/huggingface/datasets/issues/7918).

When loading a dataset with > 16 files a progress bar is shown (unless stdout is redirected or [https://github.com/huggingface/datasets/pull/7919](https://github.com/huggingface/datasets/pull/7919) is merged).

However, if you use multiple processes with data sharding, where each core loads the dataset, you get multiple copies of the progress bar (all fighting each other). It would be greatly appreciated if `datasets.load_from_disk` accepted an argument for whether to show a progress bar; default could be `None` which would retain current functionality (i.e. show if > 16 files in dataset), but user could also force the progress bar on or off as needed. Essentially just expose the progress bar visibility argument to the method's argument so that user can control it instead of it being hardcoded, where `None` would be default argument and retain current functionality.

### Motivation

Progress bar could be forced off in all processes than one, to avoid progress bar fighting and log spam.
Progress bar could also be manually forced on and off for other use cases.

### Your contribution

Possibly do a PR if this is accepted.

Comments:
Fix duplicate log messages by disabling log propagation by default

This PR fixes an issue where applications that configure logging see duplicate messages from `datasets`:

```python
import logging
logging.basicConfig(level=logging.WARNING)
from datasets.utils.logging import get_logger
get_logger("datasets.load").warning("This appears twice")
```

Outputs:
```
This appears twice
WARNING:datasets.load:This appears twice
```

This non-standard behaviour breaks default logging behaviour. The docstring for `disable_propagation()` incorrectly says: [Note that log propagation is disabled by default](https://github.com/huggingface/datasets/blob/6a1bc355a0ca2c8f9f5c10698215212f0f14e7b7/src/datasets/utils/logging.py#L161C1-L162C1)

Perhaps this was copied over from `transformers`, which disables log propogation by default, unless it's running under CI? [library_root_logger.propagate = is_ci
](https://github.com/huggingface/transformers/blob/37974267efefe020168ff27081fbab8bbce04720/src/transformers/utils/logging.py#L103)

To restore the old behaviour, users can do:
```
import datasets
datasets.logging.enable_propagation()
```

Comments:
_add_retries_to_file_obj_read_method makes file_obj invalid for pyarrow

### Describe the bug

I'm trying to use `load_dataset` to construct a dataset that read parquet data on HDFS streamingly, like
```python
ds = load_dataset(
    "parquet",
    data_files={
        "train": "hdfs://xxx/train*.parquet",
        "test": "hdfs://xxx/test*.parquet"
    },
    streaming=True,
)
```
I encountered an error

<img width="1784" height="662" alt="Image" src="https://github.com/user-attachments/assets/14f25602-ef37-4a84-83fc-dac426451163" />


In file src/datasets/packaged_modules/parquet/parquet.py, 
```python
with open(file, "rb") as f:
    self.info.features = datasets.Features.from_arrow_schema(pq.read_schema(f))
```
The `open` is replaced with `xopen` in src/datasets/utils/file_utils.py

In the func `_add_retries_to_file_obj_read_method`, the original file object would be replaced by io.RawIOBase(). Even though it tried to proxy all methods back to the original file object, it still unusable for pyarrow.
```python
try:
    file_obj.read = read_with_retries
except AttributeError:  # read-only attribute
    orig_file_obj = file_obj
    file_obj = io.RawIOBase()
    file_obj.read = read_with_retries
    file_obj.__getattr__ = lambda _, attr: getattr(orig_file_obj, attr)
return file_obj
```
For example, the original `file_obj.readable() == True`, while the new `file_obj.readable() == False`

### Steps to reproduce the bug

```python
from datasets.utils.file_utils import xopen
f = xopen('hdfs://xxxx.parquet', 'rb')
f.readable()
```

### Expected behavior

Not sure

### Environment info

Datasets 4.4.2

Comments:
hmm not sure how to fix this, I believe `file_obj.__getattr__ = lambda _, attr: getattr(orig_file_obj, attr)` would make all the methods point to the original file_obj

> hmm not sure how to fix this, I believe `file_obj.__getattr__ = lambda _, attr: getattr(orig_file_obj, attr)` would make all the methods point to the original file_obj

Could you verify by executing
```python
from datasets.utils.file_utils import xopen
f = xopen('hdfs://xxxx.parquet', 'rb')
f.readable()
```
If it's indeed a bug, I think all data files that using pyarrow would break.

Just found the issue and merged a quick fix, feel free to install `datasets` from source and let me know if it works !

> Just found the issue and merged a quick fix, feel free to install `datasets` from source and let me know if it works !

It still not working 🥹

<img width="1216" height="348" alt="Image" src="https://github.com/user-attachments/assets/a68e8f3d-2491-4616-9777-951c02c88580" />

<img width="1780" height="962" alt="Image" src="https://github.com/user-attachments/assets/9ae8f799-0d24-40ac-8cae-6f5a77d84dec" />

Arf sorry ! I opened https://github.com/huggingface/datasets/pull/7942, hopefully it's alright now ^^' feel free to try it out

It works. Thx a lot.

Yay !
Bug fix: Add HDFS hostname to protocol prefix

For HDFS url with hostname like `hdfs://hostname/user/xxx`, the function `resolve_pattern` would drop the hostname, and outputs  `hdfs:///user/xxx`. This may break later file operations by trying to connect to wrong HDFS cluster.

Comments:
Hi ! is it related to https://github.com/huggingface/datasets/issues/7934 ?

It's not clear to me why the protocol would need this, given hostname should be present in `pattern` already

```python
resolve_pattern("hdfs://hostname/user/xxx", ...)
```

> Hi ! is it related to #7934 ?
> 
> It's not clear to me why the protocol would need this, given hostname should be present in `pattern` already
> 
> ```python
> resolve_pattern("hdfs://hostname/user/xxx", ...)
> ```

It's related to #7934 in a subttle way. In my use case, I need to specify the hdfs hostname. In theory, I can do it by
```python
ds = load_dataset(
    "parquet",
    data_files={
        "train": "hdfs://hostname/xxx*.parquet",
    },
    streaming=True,
)
```
or
```python
ds = load_dataset(
    "parquet",
    data_files={
        "train": "hdfs:///xxx*.parquet",
    },
    streaming=True,
    storage_options={
        "host": "hostname"
    }
)
```
None of them work.
The first one does not work due to what this PR trying to fix, and the second one due to #7934.

Yes, `resolve_pattern` would be called like `resolve_pattern("hdfs://hostname/user/xxx", ...)`, but its out put would be like `hdfs:///user/xxx`, no hostname in it. This output would be passed to later file operation like `fsspec.open()`. It needs the hostname in the url to find the HDFS cluster correctly.

@lhoestq 
Hi! Is there any concern?🙃

I see, I think the path forward is to fix https://github.com/huggingface/datasets/issues/7934 which sounds like an actual xPath bug, while resolve_pattern dropping the hostname comes from fsspec HDFS implementation that we should probably try to follow

Fixing #7934 alone can solve my problem. 

But I don't think fsspec intends to drop the hostname. Function `resolve_pattern` here is supposed to convert a pattern to absolute file paths, and keeping the protocol intouched. `fs.glob` just returns the absolute paths to files, of which no hostname should in the result. The problem is how the function `resolve_pattern` reconstructs the whole path, ignoring the HDFS hostname in the protocol.

From another point of view, in `resolve_pattern` `fs.glob` is call with `hdfs://hostname/user/xxx` but latter `fs.open` is called with `hdfs:///user/xxx`, which is inconsistent.
xPath cannot handle hdfs:///xxxx properly

### Describe the bug

_as_str('hdfs:///xxxx') would return hdfs://xxxx. Removing one / and making the path invalid.

For the use case like
```
ds = load_dataset(
    "parquet",
    data_files={
        "train": "hdfs:///user/path/to/data/train*.parquet",
    },
    streaming=True,
    storage_options={
        "host": "hostname",  
    }
)
```
would get 
```
  File "/usr/local/lib/python3.11/site-packages/datasets/load.py", line 1511, in load_dataset
    return builder_instance.as_streaming_dataset(split=split)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/datasets/builder.py", line 1193, in as_streaming_dataset
    splits_generators = {sg.name: sg for sg in self._split_generators(dl_manager)}
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/datasets/packaged_modules/parquet/parquet.py", line 123, in _split_generators
    with open(file, "rb") as f:
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/datasets/streaming.py", line 73, in wrapper
    return function(*args, download_config=download_config, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 963, in xopen
    file_obj = fsspec.open(file, mode=mode, *args, **kwargs).open()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/core.py", line 508, in open
    out = open_files(
          ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/core.py", line 295, in open_files
    fs, fs_token, paths = get_fs_token_paths(
                          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/core.py", line 672, in get_fs_token_paths
    chain = _un_chain(urlpath0, storage_options or {})
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/core.py", line 365, in _un_chain
    kw = dict(
         ^^^^^
TypeError: dict() got multiple values for keyword argument 'host'
```
Due to the file passed to fsspec.open is hdfs://user/path/to/data/trainxxx.parquet, and fsspec would take user as the hostname

### Steps to reproduce the bug

<img width="992" height="148" alt="Image" src="https://github.com/user-attachments/assets/98e1dac2-e81b-4727-bf7a-55faaf0c8168" />

### Expected behavior

Keep all three /

### Environment info

datasets 4.4.2

Comments:
feat: Add Apache TsFile format support

# Add Apache TsFile format support

Adds support for loading `.tsfile` datasets. Closes #7922.

## What's TsFile?

[Apache TsFile](https://tsfile.apache.org/) is a columnar time-series format popular in IoT. The TsFile community requested this integration and offered to help maintain it.

## What I did

Created a new `TsFile` builder in `packaged_modules/tsfile/` following the same pattern as HDF5. Registered the module and added `.tsfile` extension mapping. Also added `tsfile>=2.0.0` as an optional dependency.

The builder uses `tsfile.to_dataframe()` with iterator mode for memory-efficient reading, then converts to PyArrow tables. Schema is inferred automatically from file metadata.

## Config options

- `batch_size` - rows per batch (default 10000)
- `table_name` - which table to read (for multi-table files)
- `columns` - filter specific columns
- `start_time` / `end_time` - time-range filtering

## Usage

```python
from datasets import load_dataset

ds = load_dataset("tsfile", data_files=["data.tsfile"], split="train")

# with filtering
ds = load_dataset("tsfile", data_files=["data.tsfile"], 
                  columns=["temperature"], start_time=1609459200000)
```

## Tests

Added 11 tests covering config validation, basic loading, data integrity, feature inference, and error handling. All passing.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7933). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix duplicate keyword conflict in load_dataset_builder

Fixes #4910

This PR fixes a bug where passing the same keyword in builder_kwargs and
config_kwargs caused a TypeError in load_dataset_builder.

The kwargs are now merged safely so config_kwargs override builder_kwargs
without duplication. A regression test is added to prevent this from
happening again.


Comments:
Hi HuggingFace team
This PR fixes issue #4910 by safely merging builder_kwargs and config_kwargs to avoid duplicate keyword errors.  
A regression test is included to ensure this does not happen again.  

Please let me know if you’d like any changes. Thanks!
Raise early for invalid `revision` in `load_dataset`

Solves https://github.com/huggingface/datasets/issues/7928

Raise early for invalid revisions


Comments:
Passes
```sh
pytest -k "LoadTest and test_load_dataset_invalid_revision_with_cache"
```

Fails
```sh
git checkout cc2399019a3a547ebc31ec68a1ff99abd4ec93ce
pytest -k "LoadTest and test_load_dataset_invalid_revision_with_cache"
```

Ran `make test`, but failures look unrelated to the PR diff (same tests fail on `main` too)

```sh
FAILED tests/test_distributed.py::test_torch_distributed_run[False] - TypeError: Passing coroutines is forbidden...
FAILED tests/test_distributed.py::test_torch_distributed_run[True] - TypeError: Passing coroutines is forbidden...
FAILED tests/test_distributed.py::test_torch_distributed_run_streaming_with_num_workers[2-2] - TypeError: Passing coroutines is forbidden...
FAILED tests/test_distributed.py::test_torch_distributed_run_streaming_with_num_workers[3-2] - TypeError: Passing coroutines is forbidden...
= 4 failed, 3077 passed, 18 skipped, 491 warnings in 556.45s (0:09:16) =
make: *** [Makefile:20: test] Error 1
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7929). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`load_dataset` `revision` param not respected when fetching from cache

### Describe the bug

`datasets.load_dataset` `revision` semantics are a bit inconsistent when the dataset is not found on the huggingface hub. When fetching the latest cached version of the dataset, the `revision` argument is ignored, so long as any cached versions of the dataset already exist in the HF cache.

### Steps to reproduce the bug

```python
import datasets
datasets.load_dataset(
    "sentientfutures/ahb",
    "dimensions",
    split="train",
    revision="main"
)
# would expect some error to raise here
datasets.load_dataset(
    "sentientfutures/ahb",
    "dimensions",
    split="train",
    revision="invalid_revision"
)
```



### Expected behavior

On the second call to `datasets.load_dataset` in the 'steps to reproduce the bug' example, expect something like:

```sh
raise DatasetNotFoundError(
datasets.exceptions.DatasetNotFoundError: Revision 'invalid_revision' doesn't exist for dataset 'sentientfutures/ahb' on the Hub.
```

### Environment info

- `datasets` version: 4.4.1
- Platform: Linux-6.2.0-39-generic-x86_64-with-glibc2.37
- Python version: 3.12.12
- `huggingface_hub` version: 0.36.0
- PyArrow version: 22.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2025.9.0

Comments:
This might be better placed as a feature request not a bug, since the logging `Using the latest cached version of the dataset since sentientfutures/ahb couldn't be found on the Hugging Face Hub` is clear.

https://github.com/huggingface/datasets/pull/7929 This only solves the case of invalid revisions. Fetching a specific revision from the cache would be more work but I think this is a good start and solves issues like https://github.com/UKGovernmentBEIS/inspect_evals/pull/834#issuecomment-3704689637

#self-assign
Add lightweight PDB (Protein Data Bank) file support

## Summary

This PR adds support for loading PDB (Protein Data Bank) files with `load_dataset()`, following the **ImageFolder pattern** where **one row = one structure**.

Based on feedback from @lhoestq in #7930, this approach makes datasets more practical for ML workflows:
- Each row is independent, enabling train/test splits and shuffling
- Easy to add labels (folder-based) and metadata (metadata.jsonl)
- Compatible with Dataset Viewer (one 3D render per row)

### Architecture

Uses `FolderBasedBuilder` pattern (like `ImageFolder`, `AudioFolder`):

```python
class PdbFolder(FolderBasedBuilder):
    BASE_FEATURE = ProteinStructure
    BASE_COLUMN_NAME = "structure"
    EXTENSIONS = [".pdb", ".ent"]
```

### New `ProteinStructure` Feature Type

```python
# Arrow schema for lazy loading
pa.struct({"bytes": pa.binary(), "path": pa.string()})

# Decoded: returns structure file content as string
dataset = load_dataset("pdb", data_dir="structures/")
print(dataset[0]["structure"])  # Full PDB file content
```

### Supported Extensions
`.pdb`, `.ent`

### Usage

```python
from datasets import load_dataset

# Load from directory
dataset = load_dataset("pdb", data_dir="protein_structures/")

# Load with folder-based labels
# structures/
#   enzymes/
#     1abc.pdb
#   receptors/
#     2def.pdb
dataset = load_dataset("pdb", data_dir="structures/")
print(dataset[0])  # {"structure": "HEADER...", "label": "enzymes"}

# Load with metadata
# structures/
#   1abc.pdb
#   metadata.jsonl  # {"file_name": "1abc.pdb", "resolution": 2.5}
dataset = load_dataset("pdb", data_dir="structures/")
print(dataset[0])  # {"structure": "HEADER...", "resolution": 2.5}

# Drop labels or metadata
dataset = load_dataset("pdb", data_dir="structures/", drop_labels=True)
dataset = load_dataset("pdb", data_dir="structures/", drop_metadata=True)
```

### Test Results
All 28 PDB tests + 15 ProteinStructure feature tests pass.

### Related PRs
- #7925 - mmCIF support (same pattern)
- #7930 - Protein 3D visualization proposal

cc @lhoestq @georgia-hf

Comments:
feat: Add mmCIF file support for macromolecular structures

## Summary

This PR adds support for loading mmCIF (macromolecular Crystallographic Information File) files with `load_dataset()`, following the **ImageFolder pattern** where **one row = one structure**.

Based on feedback from @lhoestq in #7930, this approach makes datasets more practical for ML workflows:
- Each row is independent, enabling train/test splits and shuffling
- Easy to add labels (folder-based) and metadata (metadata.jsonl)
- Compatible with Dataset Viewer (one 3D render per row)

### Architecture

Uses `FolderBasedBuilder` pattern (like `ImageFolder`, `AudioFolder`):

```python
class MmcifFolder(FolderBasedBuilder):
    BASE_FEATURE = ProteinStructure
    BASE_COLUMN_NAME = "structure"
    EXTENSIONS = [".cif", ".mmcif"]
```

### New `ProteinStructure` Feature Type

```python
# Arrow schema for lazy loading
pa.struct({"bytes": pa.binary(), "path": pa.string()})

# Decoded: returns structure file content as string
dataset = load_dataset("mmcif", data_dir="structures/")
print(dataset[0]["structure"])  # Full mmCIF file content
```

### Supported Extensions
`.cif`, `.mmcif`

### Usage

```python
from datasets import load_dataset

# Load from directory
dataset = load_dataset("mmcif", data_dir="protein_structures/")

# Load with folder-based labels
# structures/
#   enzymes/
#     1abc.cif
#   receptors/
#     2def.cif
dataset = load_dataset("mmcif", data_dir="structures/")
print(dataset[0])  # {"structure": "data_...", "label": "enzymes"}

# Load with metadata
# structures/
#   1abc.cif
#   metadata.jsonl  # {"file_name": "1abc.cif", "resolution": 2.5}
dataset = load_dataset("mmcif", data_dir="structures/")
print(dataset[0])  # {"structure": "data_...", "resolution": 2.5}

# Drop labels or metadata
dataset = load_dataset("mmcif", data_dir="structures/", drop_labels=True)
dataset = load_dataset("mmcif", data_dir="structures/", drop_metadata=True)
```

### Test Results
All 24 mmCIF tests + 15 ProteinStructure feature tests pass.

### Related PRs
- #7926 - PDB support (same pattern)
- #7930 - Protein 3D visualization proposal

### References
- mmCIF specification: https://mmcif.wwpdb.org/
- PDB archive: https://www.rcsb.org/

cc @lhoestq @georgia-hf

Comments:
Add lightweight FASTQ file format support

## Summary

This PR adds support for loading FASTQ files directly with `load_dataset()`.

FASTQ is an extension of FASTA that includes quality scores for each base, widely used for storing output from high-throughput sequencing instruments.

### Key Features
- **Zero external dependencies** - Pure Python parser based on [readfq.py](https://github.com/lh3/readfq) by Heng Li
- **Quality score support** - Preserves per-base quality scores as ASCII-encoded strings
- **Streaming support** - Generator-based parsing for memory efficiency with large NGS files
- **Compression support** - Automatic detection of gzip, bzip2, and xz compressed files
- **Large sequence support** - Uses `large_string` for both sequence and quality columns
- **Parquet-safe batching** - Dual-threshold batching (batch_size + max_batch_bytes) prevents page size errors

### Columns
| Column | Type | Description |
|--------|------|-------------|
| `id` | string | Sequence identifier (first word after `@`) |
| `description` | string | Full description line (everything after id) |
| `sequence` | large_string | The nucleotide sequence |
| `quality` | large_string | ASCII-encoded quality scores (Phred+33 by default) |

### Supported Extensions
`.fq`, `.fastq` (and compressed variants: `.fq.gz`, `.fastq.gz`, `.fq.bz2`, `.fq.xz`)

### Usage
```python
from datasets import load_dataset

# Load FASTQ file
dataset = load_dataset("fastq", data_files="reads.fastq")

# Load gzipped file
dataset = load_dataset("fastq", data_files="reads.fq.gz")

# Filter columns
dataset = load_dataset("fastq", data_files="reads.fq", columns=["sequence", "quality"])
```

### Quality Score Format
Quality scores use Sanger/Illumina 1.8+ encoding (Phred+33):
- ASCII character `\!` (33) = quality 0
- ASCII character `I` (73) = quality 40

### Testing
- 22 comprehensive tests covering basic loading, multi-line sequences, compression, batching, schema types, and edge cases
- All tests passing
- Linting clean

### References
- Follows pattern established in #7923 (FASTA support)
- Parser based on: https://github.com/lh3/readfq
- Addresses feedback from #7851

cc: @georgia-hf

Comments:
feat(fasta): add lightweight FASTA file format support

## Summary

This PR adds support for loading FASTA files directly with `load_dataset()`, addressing feedback from #7851.

FASTA is a text-based format for representing nucleotide sequences (DNA/RNA) or peptide sequences (proteins), widely used in bioinformatics.

## Key Features

- **Zero external dependencies** - Uses a lightweight pure Python parser based on [readfq.py](https://github.com/lh3/readfq) by Heng Li
- **Streaming support** - Generator-based parsing for memory efficiency with large genomic files
- **Compression support** - Automatic detection and handling of gzip, bzip2, and xz compressed files via magic bytes
- **Large sequence support** - Uses `large_string` Arrow type to handle viral genomes and long sequences (fixes UTF-8 overflow)
- **Adaptive batching** - `max_batch_bytes` parameter (default 256MB) prevents Parquet page size errors with very large sequences

## Technical Decisions (Addressing #7851 Feedback)

| Concern | Solution |
|---------|----------|
| Long sequences → UTF-8 overflow (@apcamargo, @UriNeri) | Uses `pa.large_string()` for sequence column |
| BioPython is overkill (@apcamargo) | Pure Python parser based on Heng Li's readfq.py |
| Parquet page size limit i32::MAX (@UriNeri) | Adaptive dual-threshold batching with `max_batch_bytes` |

## Columns

| Column | Type | Description |
|--------|------|-------------|
| `id` | string | Sequence identifier (first word after `>`) |
| `description` | string | Full description line (everything after id) |
| `sequence` | large_string | The biological sequence (DNA/RNA/protein) |

## Supported Extensions

`.fa`, `.fasta`, `.fna`, `.ffn`, `.faa`, `.frn` (and compressed variants)

## Usage

```python
from datasets import load_dataset

# Load FASTA file
dataset = load_dataset("fasta", data_files="sequences.fasta")

# Load with column filtering
dataset = load_dataset("fasta", data_files="sequences.fa", columns=["id", "sequence"])

# Load gzipped file
dataset = load_dataset("fasta", data_files="sequences.fa.gz")

# Configure batching for very large genomes
dataset = load_dataset("fasta", data_files="genome.fasta", max_batch_bytes=128*1024*1024)
```

## Test Plan

- [x] Basic FASTA loading (3 sequences, multi-line)
- [x] Multiple extension support (.fa, .fasta, .fna, .ffn, .faa, .frn)
- [x] Compression formats (gzip, bz2, xz)
- [x] Long sequences with `large_string` type
- [x] Column filtering
- [x] Batch size configuration
- [x] Byte-based batching (`max_batch_bytes`)
- [x] Large genome handling (simulated 50KB sequences)
- [x] Empty description handling
- [x] Multiple files loading
- [x] Custom feature casting

All 22 tests passing.

cc: @georgia-hf

Comments:
Add progress_format support for machine-readable progress output

## Summary

Adds  support to , enabling machine-readable JSON progress output similar to [huggingface/tokenizers#1921](https://github.com/huggingface/tokenizers/pull/1921).

## Motivation

When using `datasets` in automated pipelines or UI applications, it's useful to emit machine-readable progress instead of ANSI progress bars. This PR adds the same `progress_format` option that was implemented in tokenizers.

## Changes

### New Functions

- `set_progress_format(format: str)`: Set global progress format
- `get_progress_format() -> str`: Get current progress format

### Supported Formats

1. **"tqdm"** (default): Interactive progress bars
2. **"json"**: Machine-readable JSON lines to stderr
3. **"silent"**: No output

### JSON Format

When `progress_format="json"`, emits JSON every 5% progress change or completion:

```json
{"stage":"Processing","current":50,"total":100,"percent":50.0}
```

## Usage Example

```python
from datasets import load_dataset
from datasets.utils import set_progress_format

# Enable JSON output
set_progress_format("json")

# Progress will now be emitted as JSON lines
dataset = load_dataset("Goader/kobza", split="train", streaming=True)
for sample in dataset:
    process(sample)
```

## Implementation Details

- Suppresses visual output using `io.StringIO()` when format is "json"
- Keeps progress tracking active (unlike `disable=True`)
- Emits JSON to stderr every 5% progress change
- Exports new functions from `datasets.utils`

## Cross-Reference

This implementation mirrors the approach from:
- [huggingface/tokenizers#1921](https://github.com/huggingface/tokenizers/pull/1921)

## Testing

Tested with:
```python
from datasets.utils import set_progress_format, tqdm

set_progress_format('json')
for i in tqdm(range(100), desc='Test'):
    process(i)
# Outputs: {"stage":"Test","current":10,"total":100,"percent":10.0}
```

## Checklist

- [x] New functions added to `datasets.utils.tqdm`
- [x] Functions exported from `datasets.utils.__init__`
- [x] JSON format emits to stderr
- [x] Visual output suppressed when format="json"
- [x] Progress tracking remains active
- [x] Cross-referenced with tokenizers#1921

Comments:
Fix load_from_disk progress bar with redirected stdout

Fixes #7918 

## Problem
When using `load_from_disk()` with `contextlib.redirect_stdout()`, the progress bar was not showing even for datasets with >16 files.

## Root Cause  
The `disable` parameter was set to `None` which triggers TTY auto-detection. This fails when stdout is redirected, causing the progress bar to be hidden.

## Solution
Changed `disable=len(state["_data_files"]) <= 16 or None` to `disable=len(state["_data_files"]) <= 16` to force the progress bar to show for datasets with >16 files, regardless of stdout redirection.

## Testing
Verified that progress bars now appear correctly both with and without stdout redirection for datasets with >16 shards.

Comments:
this seems to contradict the comment that says 

> set `disable=None` rather than `disable=False` by default to disable progress bar when no TTY attached

I believe the right approach is to do the same as in https://github.com/huggingface/huggingface_hub/pull/2698

> this seems to contradict the comment that says
> 
> > set `disable=None` rather than `disable=False` by default to disable progress bar when no TTY attached
> 
> I believe the right approach is to do the same as in [huggingface/huggingface_hub#2698](https://github.com/huggingface/huggingface_hub/pull/2698)

Updated to check TQDM_POSITION=-1 to force-enable progress bars in cloud environments, 
following the same pattern as huggingface_hub#2698.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7919). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Moved the TQDM_POSITION check to the tqdm class in utils/tqdm.py so all progress bars 
in the codebase have consistent behavior. Thanks for the suggestion!

@lhoestq thanks again for the suggestion. I’ve applied it and everything should now be consistent across all tqdm usage. Happy to adjust anything else if needed.
datasets.load_from_disk doesn't show progress bar

### Describe the bug

This is the inverse of the bug at [https://github.com/huggingface/datasets/issues/7030](https://github.com/huggingface/datasets/issues/7030), i.e. that `datasets.load_from_disk(path)` displays no progress bar. My dataset has > 16 files in it.

I am redirecting stdout as I capture the log, could this have something to do with it? All other progress bars work fine though except for HF dataset progress bars.

### Steps to reproduce the bug

```py
with contextlib.redirect_stdout(log_file), contextlib.redirect_stderr(log_file):
    datasets.load_from_disk(path)
```

### Expected behavior

The progress bar should show when loading a dataset.

### Environment info

Python 3.13.9
Datasets 4.4.1
macOS Tahoe 26.2

Comments:
#self-assign
GDPval dataset Word docs corrupted

The [openai/gdpval](https://huggingface.co/datasets/openai/gdpval) dataset on Hugging Face contains Word .docx files with two types of corruption that cause Microsoft Word to display an "unreadable content" error.

### Root Causes

1. **Corrupted settings.xml**: The `word/settings.xml` file uses incorrect namespace prefixes (`ns0:`, `ns1:`, etc.) instead of the proper prefixes (`w:`, `mc:`, `m:`, etc.)

2. **Malformed TargetMode attributes**: Some files have `TargetMode="External"` attributes missing their closing `/>` tag in hyperlink relationships

Both issues cause Word to reject the files even though the XML structure is technically valid.

I have a fix for the issue here https://github.com/alexheat/gdpval-docx-fix

Comments:
tentatively tagging @simonpfish ^

(if it's an option you could enable PRs/Discussions on your dataset on HF)