AniketTec/github-issues huggingface dataset:
Title: Fix duplicate log messages by disabling log propagation by default
State: open

This PR fixes an issue where applications that configure logging see duplicate messages from `datasets`:

```python
import logging
logging.basicConfig(level=logging.WARNING)
from datasets.utils.logging import get_logger
get_logger("datasets.load").warning("This appears twice")
```

Outputs:
```
This appears twice
WARNING:datasets.load:This appears twice
```

This non-standard behaviour breaks default logging behaviour. The docstring for `disable_propagation()` incorrectly says: [Note that log propagation is disabled by default](https://github.com/huggingface/datasets/blob/6a1bc355a0ca2c8f9f5c10698215212f0f14e7b7/src/datasets/utils/logging.py#L161C1-L162C1)

Perhaps this was copied over from `transformers`, which disables log propogation by default, unless it's running under CI? [library_root_logger.propagate = is_ci
](https://github.com/huggingface/transformers/blob/37974267efefe020168ff27081fbab8bbce04720/src/transformers/utils/logging.py#L103)

To restore the old behaviour, users can do:
```
import datasets
datasets.logging.enable_propagation()
```

Comments:
Title: Add option for loading audio with video
State: open

### Describe the bug

Currently, `torchcodec` don't allow extracting `Audio` from `Video` https://github.com/meta-pytorch/torchcodec/issues/1158, but when I upload videos with audio to hub using `videofolder`, then this is not possible to retrieve audio from it. Probably `VideoDecoder` can be extended with `audio` parameter to retrieve this information

### Steps to reproduce the bug

```python
from datasets import load_dataset

test_ds = load_dataset("videofolder", data_dir="/path/to/video")  
# uploaded version 
# test_ds = load_dataset("Samoed/testds")

test_ds["train"][1]["video"]
# <torchcodec.decoders._video_decoder.VideoDecoder
```

### Expected behavior

Some option to retrieve audio information.

### Environment info

```
datasets==4.5.0
```

Comments:
Hi,
I’m interested in working on this issue.

From what I understand, audio decoding would involve an AudioDecoder, and previous discussions have suggested handling it separately. I’d like to explore whether integrating audio extraction in relation to the current video decoding workflow would be appropriate, or if maintaining a separate decoder is the preferred design.

Any guidance would be greatly appreciated.

Thanks!
Title: _add_retries_to_file_obj_read_method makes file_obj invalid for pyarrow
State: closed

### Describe the bug

I'm trying to use `load_dataset` to construct a dataset that read parquet data on HDFS streamingly, like
```python
ds = load_dataset(
    "parquet",
    data_files={
        "train": "hdfs://xxx/train*.parquet",
        "test": "hdfs://xxx/test*.parquet"
    },
    streaming=True,
)
```
I encountered an error

<img width="1784" height="662" alt="Image" src="https://github.com/user-attachments/assets/14f25602-ef37-4a84-83fc-dac426451163" />


In file src/datasets/packaged_modules/parquet/parquet.py, 
```python
with open(file, "rb") as f:
    self.info.features = datasets.Features.from_arrow_schema(pq.read_schema(f))
```
The `open` is replaced with `xopen` in src/datasets/utils/file_utils.py

In the func `_add_retries_to_file_obj_read_method`, the original file object would be replaced by io.RawIOBase(). Even though it tried to proxy all methods back to the original file object, it still unusable for pyarrow.
```python
try:
    file_obj.read = read_with_retries
except AttributeError:  # read-only attribute
    orig_file_obj = file_obj
    file_obj = io.RawIOBase()
    file_obj.read = read_with_retries
    file_obj.__getattr__ = lambda _, attr: getattr(orig_file_obj, attr)
return file_obj
```
For example, the original `file_obj.readable() == True`, while the new `file_obj.readable() == False`

### Steps to reproduce the bug

```python
from datasets.utils.file_utils import xopen
f = xopen('hdfs://xxxx.parquet', 'rb')
f.readable()
```

### Expected behavior

Not sure

### Environment info

Datasets 4.4.2

Comments:
hmm not sure how to fix this, I believe `file_obj.__getattr__ = lambda _, attr: getattr(orig_file_obj, attr)` would make all the methods point to the original file_obj

> hmm not sure how to fix this, I believe `file_obj.__getattr__ = lambda _, attr: getattr(orig_file_obj, attr)` would make all the methods point to the original file_obj

Could you verify by executing
```python
from datasets.utils.file_utils import xopen
f = xopen('hdfs://xxxx.parquet', 'rb')
f.readable()
```
If it's indeed a bug, I think all data files that using pyarrow would break.

Just found the issue and merged a quick fix, feel free to install `datasets` from source and let me know if it works !

> Just found the issue and merged a quick fix, feel free to install `datasets` from source and let me know if it works !

It still not working 🥹

<img width="1216" height="348" alt="Image" src="https://github.com/user-attachments/assets/a68e8f3d-2491-4616-9777-951c02c88580" />

<img width="1780" height="962" alt="Image" src="https://github.com/user-attachments/assets/9ae8f799-0d24-40ac-8cae-6f5a77d84dec" />

Arf sorry ! I opened https://github.com/huggingface/datasets/pull/7942, hopefully it's alright now ^^' feel free to try it out

It works. Thx a lot.

Yay !
Title: Bug fix: Add HDFS hostname to protocol prefix
State: open

For HDFS url with hostname like `hdfs://hostname/user/xxx`, the function `resolve_pattern` would drop the hostname, and outputs  `hdfs:///user/xxx`. This may break later file operations by trying to connect to wrong HDFS cluster.

Comments:
Hi ! is it related to https://github.com/huggingface/datasets/issues/7934 ?

It's not clear to me why the protocol would need this, given hostname should be present in `pattern` already

```python
resolve_pattern("hdfs://hostname/user/xxx", ...)
```

> Hi ! is it related to #7934 ?
> 
> It's not clear to me why the protocol would need this, given hostname should be present in `pattern` already
> 
> ```python
> resolve_pattern("hdfs://hostname/user/xxx", ...)
> ```

It's related to #7934 in a subttle way. In my use case, I need to specify the hdfs hostname. In theory, I can do it by
```python
ds = load_dataset(
    "parquet",
    data_files={
        "train": "hdfs://hostname/xxx*.parquet",
    },
    streaming=True,
)
```
or
```python
ds = load_dataset(
    "parquet",
    data_files={
        "train": "hdfs:///xxx*.parquet",
    },
    streaming=True,
    storage_options={
        "host": "hostname"
    }
)
```
None of them work.
The first one does not work due to what this PR trying to fix, and the second one due to #7934.

Yes, `resolve_pattern` would be called like `resolve_pattern("hdfs://hostname/user/xxx", ...)`, but its out put would be like `hdfs:///user/xxx`, no hostname in it. This output would be passed to later file operation like `fsspec.open()`. It needs the hostname in the url to find the HDFS cluster correctly.

@lhoestq 
Hi! Is there any concern?🙃

I see, I think the path forward is to fix https://github.com/huggingface/datasets/issues/7934 which sounds like an actual xPath bug, while resolve_pattern dropping the hostname comes from fsspec HDFS implementation that we should probably try to follow

Fixing #7934 alone can solve my problem. 

But I don't think fsspec intends to drop the hostname. Function `resolve_pattern` here is supposed to convert a pattern to absolute file paths, and keeping the protocol intouched. `fs.glob` just returns the absolute paths to files, of which no hostname should in the result. The problem is how the function `resolve_pattern` reconstructs the whole path, ignoring the HDFS hostname in the protocol.

From another point of view, in `resolve_pattern` `fs.glob` is call with `hdfs://hostname/user/xxx` but latter `fs.open` is called with `hdfs:///user/xxx`, which is inconsistent.
Title: xPath cannot handle hdfs:///xxxx properly
State: open

### Describe the bug

_as_str('hdfs:///xxxx') would return hdfs://xxxx. Removing one / and making the path invalid.

For the use case like
```
ds = load_dataset(
    "parquet",
    data_files={
        "train": "hdfs:///user/path/to/data/train*.parquet",
    },
    streaming=True,
    storage_options={
        "host": "hostname",  
    }
)
```
would get 
```
  File "/usr/local/lib/python3.11/site-packages/datasets/load.py", line 1511, in load_dataset
    return builder_instance.as_streaming_dataset(split=split)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/datasets/builder.py", line 1193, in as_streaming_dataset
    splits_generators = {sg.name: sg for sg in self._split_generators(dl_manager)}
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/datasets/packaged_modules/parquet/parquet.py", line 123, in _split_generators
    with open(file, "rb") as f:
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/datasets/streaming.py", line 73, in wrapper
    return function(*args, download_config=download_config, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/datasets/utils/file_utils.py", line 963, in xopen
    file_obj = fsspec.open(file, mode=mode, *args, **kwargs).open()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/core.py", line 508, in open
    out = open_files(
          ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/core.py", line 295, in open_files
    fs, fs_token, paths = get_fs_token_paths(
                          ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/core.py", line 672, in get_fs_token_paths
    chain = _un_chain(urlpath0, storage_options or {})
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fsspec/core.py", line 365, in _un_chain
    kw = dict(
         ^^^^^
TypeError: dict() got multiple values for keyword argument 'host'
```
Due to the file passed to fsspec.open is hdfs://user/path/to/data/trainxxx.parquet, and fsspec would take user as the hostname

### Steps to reproduce the bug

<img width="992" height="148" alt="Image" src="https://github.com/user-attachments/assets/98e1dac2-e81b-4727-bf7a-55faaf0c8168" />

### Expected behavior

Keep all three /

### Environment info

datasets 4.4.2

Comments:
Title: feat: Add Apache TsFile format support
State: open

# Add Apache TsFile format support

Adds support for loading `.tsfile` datasets. Closes #7922.

## What's TsFile?

[Apache TsFile](https://tsfile.apache.org/) is a columnar time-series format popular in IoT. The TsFile community requested this integration and offered to help maintain it.

## What I did

Created a new `TsFile` builder in `packaged_modules/tsfile/` following the same pattern as HDF5. Registered the module and added `.tsfile` extension mapping. Also added `tsfile>=2.0.0` as an optional dependency.

The builder uses `tsfile.to_dataframe()` with iterator mode for memory-efficient reading, then converts to PyArrow tables. Schema is inferred automatically from file metadata.

## Config options

- `batch_size` - rows per batch (default 10000)
- `table_name` - which table to read (for multi-table files)
- `columns` - filter specific columns
- `start_time` / `end_time` - time-range filtering

## Usage

```python
from datasets import load_dataset

ds = load_dataset("tsfile", data_files=["data.tsfile"], split="train")

# with filtering
ds = load_dataset("tsfile", data_files=["data.tsfile"], 
                  columns=["temperature"], start_time=1609459200000)
```

## Tests

Added 11 tests covering config validation, basic loading, data integrity, feature inference, and error handling. All passing.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7933). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Fix duplicate keyword conflict in load_dataset_builder
State: open

Fixes #4910

This PR fixes a bug where passing the same keyword in builder_kwargs and
config_kwargs caused a TypeError in load_dataset_builder.

The kwargs are now merged safely so config_kwargs override builder_kwargs
without duplication. A regression test is added to prevent this from
happening again.


Comments:
Hi HuggingFace team
This PR fixes issue #4910 by safely merging builder_kwargs and config_kwargs to avoid duplicate keyword errors.  
A regression test is included to ensure this does not happen again.  

Please let me know if you’d like any changes. Thanks!
Title: Raise early for invalid `revision` in `load_dataset`
State: closed

Solves https://github.com/huggingface/datasets/issues/7928

Raise early for invalid revisions


Comments:
Passes
```sh
pytest -k "LoadTest and test_load_dataset_invalid_revision_with_cache"
```

Fails
```sh
git checkout cc2399019a3a547ebc31ec68a1ff99abd4ec93ce
pytest -k "LoadTest and test_load_dataset_invalid_revision_with_cache"
```

Ran `make test`, but failures look unrelated to the PR diff (same tests fail on `main` too)

```sh
FAILED tests/test_distributed.py::test_torch_distributed_run[False] - TypeError: Passing coroutines is forbidden...
FAILED tests/test_distributed.py::test_torch_distributed_run[True] - TypeError: Passing coroutines is forbidden...
FAILED tests/test_distributed.py::test_torch_distributed_run_streaming_with_num_workers[2-2] - TypeError: Passing coroutines is forbidden...
FAILED tests/test_distributed.py::test_torch_distributed_run_streaming_with_num_workers[3-2] - TypeError: Passing coroutines is forbidden...
= 4 failed, 3077 passed, 18 skipped, 491 warnings in 556.45s (0:09:16) =
make: *** [Makefile:20: test] Error 1
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7929). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: `load_dataset` `revision` param not respected when fetching from cache
State: open

### Describe the bug

`datasets.load_dataset` `revision` semantics are a bit inconsistent when the dataset is not found on the huggingface hub. When fetching the latest cached version of the dataset, the `revision` argument is ignored, so long as any cached versions of the dataset already exist in the HF cache.

### Steps to reproduce the bug

```python
import datasets
datasets.load_dataset(
    "sentientfutures/ahb",
    "dimensions",
    split="train",
    revision="main"
)
# would expect some error to raise here
datasets.load_dataset(
    "sentientfutures/ahb",
    "dimensions",
    split="train",
    revision="invalid_revision"
)
```



### Expected behavior

On the second call to `datasets.load_dataset` in the 'steps to reproduce the bug' example, expect something like:

```sh
raise DatasetNotFoundError(
datasets.exceptions.DatasetNotFoundError: Revision 'invalid_revision' doesn't exist for dataset 'sentientfutures/ahb' on the Hub.
```

### Environment info

- `datasets` version: 4.4.1
- Platform: Linux-6.2.0-39-generic-x86_64-with-glibc2.37
- Python version: 3.12.12
- `huggingface_hub` version: 0.36.0
- PyArrow version: 22.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2025.9.0

Comments:
This might be better placed as a feature request not a bug, since the logging `Using the latest cached version of the dataset since sentientfutures/ahb couldn't be found on the Hugging Face Hub` is clear.

https://github.com/huggingface/datasets/pull/7929 This only solves the case of invalid revisions. Fetching a specific revision from the cache would be more work but I think this is a good start and solves issues like https://github.com/UKGovernmentBEIS/inspect_evals/pull/834#issuecomment-3704689637
Title: Add lightweight PDB (Protein Data Bank) file support
State: open

## Summary

This PR adds support for loading PDB (Protein Data Bank) files with `load_dataset()`, following the **ImageFolder pattern** where **one row = one structure**.

Based on feedback from @lhoestq in #7930, this approach makes datasets more practical for ML workflows:
- Each row is independent, enabling train/test splits and shuffling
- Easy to add labels (folder-based) and metadata (metadata.jsonl)
- Compatible with Dataset Viewer (one 3D render per row)

### Architecture

Uses `FolderBasedBuilder` pattern (like `ImageFolder`, `AudioFolder`):

```python
class PdbFolder(FolderBasedBuilder):
    BASE_FEATURE = ProteinStructure
    BASE_COLUMN_NAME = "structure"
    EXTENSIONS = [".pdb", ".ent"]
```

### New `ProteinStructure` Feature Type

```python
# Arrow schema for lazy loading
pa.struct({"bytes": pa.binary(), "path": pa.string()})

# Decoded: returns structure file content as string
dataset = load_dataset("pdb", data_dir="structures/")
print(dataset[0]["structure"])  # Full PDB file content
```

### Supported Extensions
`.pdb`, `.ent`

### Usage

```python
from datasets import load_dataset

# Load from directory
dataset = load_dataset("pdb", data_dir="protein_structures/")

# Load with folder-based labels
# structures/
#   enzymes/
#     1abc.pdb
#   receptors/
#     2def.pdb
dataset = load_dataset("pdb", data_dir="structures/")
print(dataset[0])  # {"structure": "HEADER...", "label": "enzymes"}

# Load with metadata
# structures/
#   1abc.pdb
#   metadata.jsonl  # {"file_name": "1abc.pdb", "resolution": 2.5}
dataset = load_dataset("pdb", data_dir="structures/")
print(dataset[0])  # {"structure": "HEADER...", "resolution": 2.5}

# Drop labels or metadata
dataset = load_dataset("pdb", data_dir="structures/", drop_labels=True)
dataset = load_dataset("pdb", data_dir="structures/", drop_metadata=True)
```

### Test Results
All 28 PDB tests + 15 ProteinStructure feature tests pass.

### Related PRs
- #7925 - mmCIF support (same pattern)
- #7930 - Protein 3D visualization proposal

cc @lhoestq @georgia-hf

Comments:
Title: feat: Add mmCIF file support for macromolecular structures
State: open

## Summary

This PR adds support for loading mmCIF (macromolecular Crystallographic Information File) files with `load_dataset()`, following the **ImageFolder pattern** where **one row = one structure**.

Based on feedback from @lhoestq in #7930, this approach makes datasets more practical for ML workflows:
- Each row is independent, enabling train/test splits and shuffling
- Easy to add labels (folder-based) and metadata (metadata.jsonl)
- Compatible with Dataset Viewer (one 3D render per row)

### Architecture

Uses `FolderBasedBuilder` pattern (like `ImageFolder`, `AudioFolder`):

```python
class MmcifFolder(FolderBasedBuilder):
    BASE_FEATURE = ProteinStructure
    BASE_COLUMN_NAME = "structure"
    EXTENSIONS = [".cif", ".mmcif"]
```

### New `ProteinStructure` Feature Type

```python
# Arrow schema for lazy loading
pa.struct({"bytes": pa.binary(), "path": pa.string()})

# Decoded: returns structure file content as string
dataset = load_dataset("mmcif", data_dir="structures/")
print(dataset[0]["structure"])  # Full mmCIF file content
```

### Supported Extensions
`.cif`, `.mmcif`

### Usage

```python
from datasets import load_dataset

# Load from directory
dataset = load_dataset("mmcif", data_dir="protein_structures/")

# Load with folder-based labels
# structures/
#   enzymes/
#     1abc.cif
#   receptors/
#     2def.cif
dataset = load_dataset("mmcif", data_dir="structures/")
print(dataset[0])  # {"structure": "data_...", "label": "enzymes"}

# Load with metadata
# structures/
#   1abc.cif
#   metadata.jsonl  # {"file_name": "1abc.cif", "resolution": 2.5}
dataset = load_dataset("mmcif", data_dir="structures/")
print(dataset[0])  # {"structure": "data_...", "resolution": 2.5}

# Drop labels or metadata
dataset = load_dataset("mmcif", data_dir="structures/", drop_labels=True)
dataset = load_dataset("mmcif", data_dir="structures/", drop_metadata=True)
```

### Test Results
All 24 mmCIF tests + 15 ProteinStructure feature tests pass.

### Related PRs
- #7926 - PDB support (same pattern)
- #7930 - Protein 3D visualization proposal

### References
- mmCIF specification: https://mmcif.wwpdb.org/
- PDB archive: https://www.rcsb.org/

cc @lhoestq @georgia-hf

Comments:
Title: Add lightweight FASTQ file format support
State: open

## Summary

This PR adds support for loading FASTQ files directly with `load_dataset()`.

FASTQ is an extension of FASTA that includes quality scores for each base, widely used for storing output from high-throughput sequencing instruments.

### Key Features
- **Zero external dependencies** - Pure Python parser based on [readfq.py](https://github.com/lh3/readfq) by Heng Li
- **Quality score support** - Preserves per-base quality scores as ASCII-encoded strings
- **Streaming support** - Generator-based parsing for memory efficiency with large NGS files
- **Compression support** - Automatic detection of gzip, bzip2, and xz compressed files
- **Large sequence support** - Uses `large_string` for both sequence and quality columns
- **Parquet-safe batching** - Dual-threshold batching (batch_size + max_batch_bytes) prevents page size errors

### Columns
| Column | Type | Description |
|--------|------|-------------|
| `id` | string | Sequence identifier (first word after `@`) |
| `description` | string | Full description line (everything after id) |
| `sequence` | large_string | The nucleotide sequence |
| `quality` | large_string | ASCII-encoded quality scores (Phred+33 by default) |

### Supported Extensions
`.fq`, `.fastq` (and compressed variants: `.fq.gz`, `.fastq.gz`, `.fq.bz2`, `.fq.xz`)

### Usage
```python
from datasets import load_dataset

# Load FASTQ file
dataset = load_dataset("fastq", data_files="reads.fastq")

# Load gzipped file
dataset = load_dataset("fastq", data_files="reads.fq.gz")

# Filter columns
dataset = load_dataset("fastq", data_files="reads.fq", columns=["sequence", "quality"])
```

### Quality Score Format
Quality scores use Sanger/Illumina 1.8+ encoding (Phred+33):
- ASCII character `\!` (33) = quality 0
- ASCII character `I` (73) = quality 40

### Testing
- 22 comprehensive tests covering basic loading, multi-line sequences, compression, batching, schema types, and edge cases
- All tests passing
- Linting clean

### References
- Follows pattern established in #7923 (FASTA support)
- Parser based on: https://github.com/lh3/readfq
- Addresses feedback from #7851

cc: @georgia-hf

Comments: