noamaanMulla-03/datasets-issues huggingface dataset:
Improve readability and documentation of indexing integration tests

### Summary
This PR improves the readability and maintainability of the indexing integration tests by adding clear, detailed comments throughout the test suite.

### Motivation
The indexing tests cover multiple backends (FAISS and Elasticsearch) and involve non-trivial workflows such as vector creation, indexing, querying, and serialization. Adding explanatory comments helps new contributors and reviewers understand the intent of each test case more easily.

### What’s Changed
- Added descriptive docstrings to test classes and methods
- Included inline comments explaining:
  - Dataset construction
  - Index creation and configuration
  - Search and batch search behavior
  - Error handling and validation logic
  - Serialization and cleanup steps
- No functional or behavioral changes

### Scope
- Documentation and readability improvements only
- No changes to test logic, APIs, or expected behavior

### Impact
This change lowers the barrier for new contributors, improves code comprehension, and makes future maintenance easier without affecting test coverage or performance.

### Checklist
- [x] No functional changes introduced
- [x] Tests pass locally
- [x] Follows existing project style and conventions
Fix duplicate log messages by disabling log propagation by default

This PR fixes an issue where applications that configure logging see duplicate messages from `datasets`:

```python
import logging
logging.basicConfig(level=logging.WARNING)
from datasets.utils.logging import get_logger
get_logger("datasets.load").warning("This appears twice")
```

Outputs:
```
This appears twice
WARNING:datasets.load:This appears twice
```

This non-standard behaviour breaks default logging behaviour. The docstring for `disable_propagation()` incorrectly says: [Note that log propagation is disabled by default](https://github.com/huggingface/datasets/blob/6a1bc355a0ca2c8f9f5c10698215212f0f14e7b7/src/datasets/utils/logging.py#L161C1-L162C1)

Perhaps this was copied over from `transformers`, which disables log propogation by default, unless it's running under CI? [library_root_logger.propagate = is_ci
](https://github.com/huggingface/transformers/blob/37974267efefe020168ff27081fbab8bbce04720/src/transformers/utils/logging.py#L103)

To restore the old behaviour, users can do:
```
import datasets
datasets.logging.enable_propagation()
```
_add_retries_to_file_obj_read_method makes file_obj invalid for pyarrow

### Describe the bug

I'm trying to use `load_dataset` to construct a dataset that read parquet data on HDFS streamingly, like
```python
ds = load_dataset(
    "parquet",
    data_files={
        "train": "hdfs://xxx/train*.parquet",
        "test": "hdfs://xxx/test*.parquet"
    },
    streaming=True,
)
```
I encountered an error

<img width="1784" height="662" alt="Image" src="https://github.com/user-attachments/assets/14f25602-ef37-4a84-83fc-dac426451163" />


In file src/datasets/packaged_modules/parquet/parquet.py, 
```python
with open(file, "rb") as f:
    self.info.features = datasets.Features.from_arrow_schema(pq.read_schema(f))
```
The `open` is replaced with `xopen` in src/datasets/utils/file_utils.py

In the func `_add_retries_to_file_obj_read_method`, the original file object would be replaced by io.RawIOBase(). Even though it tried to proxy all methods back to the original file object, it still unusable for pyarrow.
```python
try:
    file_obj.read = read_with_retries
except AttributeError:  # read-only attribute
    orig_file_obj = file_obj
    file_obj = io.RawIOBase()
    file_obj.read = read_with_retries
    file_obj.__getattr__ = lambda _, attr: getattr(orig_file_obj, attr)
return file_obj
```
For example, the original `file_obj.readable() == True`, while the new `file_obj.readable() == False`

### Steps to reproduce the bug

```python
from datasets.utils.file_utils import xopen
f = xopen('hdfs://xxxx.parquet', 'rb')
f.readable()
```

### Expected behavior

Not sure

### Environment info

Datasets 4.4.2


hmm not sure how to fix this, I believe `file_obj.__getattr__ = lambda _, attr: getattr(orig_file_obj, attr)` would make all the methods point to the original file_obj

> hmm not sure how to fix this, I believe `file_obj.__getattr__ = lambda _, attr: getattr(orig_file_obj, attr)` would make all the methods point to the original file_obj

Could you verify by executing
```python
from datasets.utils.file_utils import xopen
f = xopen('hdfs://xxxx.parquet', 'rb')
f.readable()
```
If it's indeed a bug, I think all data files that using pyarrow would break.

Just found the issue and merged a quick fix, feel free to install `datasets` from source and let me know if it works !

> Just found the issue and merged a quick fix, feel free to install `datasets` from source and let me know if it works !

It still not working 🥹

<img width="1216" height="348" alt="Image" src="https://github.com/user-attachments/assets/a68e8f3d-2491-4616-9777-951c02c88580" />

<img width="1780" height="962" alt="Image" src="https://github.com/user-attachments/assets/9ae8f799-0d24-40ac-8cae-6f5a77d84dec" />

Arf sorry ! I opened https://github.com/huggingface/datasets/pull/7942, hopefully it's alright now ^^' feel free to try it out

It works. Thx a lot.

Yay !
Bug fix: Add HDFS hostname to protocol prefix

For HDFS url with hostname like `hdfs://hostname/user/xxx`, the function `resolve_pattern` would drop the hostname, and outputs  `hdfs:///user/xxx`. This may break later file operations by trying to connect to wrong HDFS cluster.


Hi ! is it related to https://github.com/huggingface/datasets/issues/7934 ?

It's not clear to me why the protocol would need this, given hostname should be present in `pattern` already

```python
resolve_pattern("hdfs://hostname/user/xxx", ...)
```

> Hi ! is it related to #7934 ?
> 
> It's not clear to me why the protocol would need this, given hostname should be present in `pattern` already
> 
> ```python
> resolve_pattern("hdfs://hostname/user/xxx", ...)
> ```

It's related to #7934 in a subttle way. In my use case, I need to specify the hdfs hostname. In theory, I can do it by
```python
ds = load_dataset(
    "parquet",
    data_files={
        "train": "hdfs://hostname/xxx*.parquet",
    },
    streaming=True,
)
```
or
```python
ds = load_dataset(
    "parquet",
    data_files={
        "train": "hdfs:///xxx*.parquet",
    },
    streaming=True,
    storage_options={
        "host": "hostname"
    }
)
```
None of them work.
The first one does not work due to what this PR trying to fix, and the second one due to #7934.

Yes, `resolve_pattern` would be called like `resolve_pattern("hdfs://hostname/user/xxx", ...)`, but its out put would be like `hdfs:///user/xxx`, no hostname in it. This output would be passed to later file operation like `fsspec.open()`. It needs the hostname in the url to find the HDFS cluster correctly.

@lhoestq 
Hi! Is there any concern?🙃

I see, I think the path forward is to fix https://github.com/huggingface/datasets/issues/7934 which sounds like an actual xPath bug, while resolve_pattern dropping the hostname comes from fsspec HDFS implementation that we should probably try to follow

Fixing #7934 alone can solve my problem. 

But I don't think fsspec intends to drop the hostname. Function `resolve_pattern` here is supposed to convert a pattern to absolute file paths, and keeping the protocol intouched. `fs.glob` just returns the absolute paths to files, of which no hostname should in the result. The problem is how the function `resolve_pattern` reconstructs the whole path, ignoring the HDFS hostname in the protocol.

From another point of view, in `resolve_pattern` `fs.glob` is call with `hdfs://hostname/user/xxx` but latter `fs.open` is called with `hdfs:///user/xxx`, which is inconsistent.
Add _generate_shards

Useful to list a dataset's shards:

```python
from datasets import load_dataset_builder, StreamingDownloadManager

dlm = StreamingDownloadManager()

def get_shards(dataset_name, *args, **kwargs):
    b = load_dataset_builder(dataset_name, *args, **kwargs)
    splits = b._split_generators(dlm)
    return list(b._generate_shards(**splits[0].gen_kwargs))

print(get_shards("username/dataset_name"))
# ['hf://datasets/...', ...]
```

I'll use this in combination with https://github.com/huggingface/datasets/pull/7897 in the Dataset Viewer for the and API endpoint that does {dataset, config, split, offset, limit} -> [{fileUri, offset, limit}]. This will be useful to edit datasets since your can get a row's location inside a dataset cc @cfahlgren1 

This will be similar to https://github.com/huggingface/dataset-viewer/pull/3276 but works for any dataset format: csv, json, webdataset, images etc.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7943). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix load_from_disk progress bar with redirected stdout

Fixes #7918 

## Problem
When using `load_from_disk()` with `contextlib.redirect_stdout()`, the progress bar was not showing even for datasets with >16 files.

## Root Cause  
The `disable` parameter was set to `None` which triggers TTY auto-detection. This fails when stdout is redirected, causing the progress bar to be hidden.

## Solution
Changed `disable=len(state["_data_files"]) <= 16 or None` to `disable=len(state["_data_files"]) <= 16` to force the progress bar to show for datasets with >16 files, regardless of stdout redirection.

## Testing
Verified that progress bars now appear correctly both with and without stdout redirection for datasets with >16 shards.


this seems to contradict the comment that says 

> set `disable=None` rather than `disable=False` by default to disable progress bar when no TTY attached

I believe the right approach is to do the same as in https://github.com/huggingface/huggingface_hub/pull/2698

> this seems to contradict the comment that says
> 
> > set `disable=None` rather than `disable=False` by default to disable progress bar when no TTY attached
> 
> I believe the right approach is to do the same as in [huggingface/huggingface_hub#2698](https://github.com/huggingface/huggingface_hub/pull/2698)

Updated to check TQDM_POSITION=-1 to force-enable progress bars in cloud environments, 
following the same pattern as huggingface_hub#2698.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7919). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Moved the TQDM_POSITION check to the tqdm class in utils/tqdm.py so all progress bars 
in the codebase have consistent behavior. Thanks for the suggestion!

@lhoestq thanks again for the suggestion. I’ve applied it and everything should now be consistent across all tqdm usage. Happy to adjust anything else if needed.
datasets.load_from_disk doesn't show progress bar

### Describe the bug

This is the inverse of the bug at [https://github.com/huggingface/datasets/issues/7030](https://github.com/huggingface/datasets/issues/7030), i.e. that `datasets.load_from_disk(path)` displays no progress bar. My dataset has > 16 files in it.

I am redirecting stdout as I capture the log, could this have something to do with it? All other progress bars work fine though except for HF dataset progress bars.

### Steps to reproduce the bug

```py
with contextlib.redirect_stdout(log_file), contextlib.redirect_stderr(log_file):
    datasets.load_from_disk(path)
```

### Expected behavior

The progress bar should show when loading a dataset.

### Environment info

Python 3.13.9
Datasets 4.4.1
macOS Tahoe 26.2


#self-assign
json: add optional return_file_name parameter

This PR adds an optional `return_file_name` parameter to the JSON dataset loader.

When enabled, a new `file_name` column is added containing the source file name
for each row. Default behavior is unchanged.

Changes:
- Add `return_file_name` to JsonConfig
- Append file name during JSON table generation
- Add tests covering default and enabled behavior, and ensures other functions are not affected

Motivation:
This helps resume training from checkpoints by identifying already-consumed data shards.

Fixes #5806
Add C4

The old code for the C4 dataset was to generate the C4 with Apache Beam, as in Tensorflow Datasets.
However AllenAI is now hosting the processed C4 dataset in this repo: https://huggingface.co/datasets/allenai/c4
Thanks a lot to them for their amazing work !

In this PR I changed the script to download and prepare the data directly from this repo.
It has 4 variants: en, en.noblocklist, en.noclean, realnewslike

You can load it with
```python
from datasets import load_dataset

c4 = load_dataset("c4", "en")
```

It also supports streaming, if you don't want to download hundreds of GB of data:
```python
c4 = load_dataset("c4", "en", streaming=True)
```

Regarding the dataset_infos.json, I haven't added the infos for en.noclean. I will add them once I have them.

Also we can work on the dataset card at https://huggingface.co/datasets/c4
For now I just added a link to https://huggingface.co/datasets/allenai/c4 as well as a few sections


C4 has broken:

In offline mode C4 downloader will request a hash from nowhere

> ValueError: Couldn't find cache for allenai/c4 for config 'default-c2cff5d7da675c1c'


while the cache files:

> Available configs in the cache: ['default-b04fc8a0b8562884', 'default-c7bc8b0aefc5e48f']


see issue https://huggingface.co/datasets/allenai/c4/discussions/7

@albertvillanova @lhoestq @dirkgr
Is the 10k files / folder limit a hard limit for a dataset repo?

### Feature request

Can the hard limit of 10k files per folder be extended with acceptable loss in performance?

### Motivation

I'm uploading a lance dataset to huggingface hub and have a folder inside lance internals (`data/*.lance/_transactions`) that has > 20k atomic transaction records and my commits are being rejected.

```
Bad request for commit endpoint:
Your push was rejected because it contains too many files per directory. Each directory in your git repo can only contain up to 10000 files. Offending reference: refs/heads/main Offending directories: /data/expression.lance/_transactions/
```


### Your contribution

Open to suggestions for how to make a PR

Labels: name: enhancement

Yes, that's a hard limit. Can you split your files into different folders? Or we'll probably have a new repo type in the near to mid future that will relax this limit a bit. 

Thanks! Working around this with a different sharding parameter to have fewer overall fragments (and therefore fewer files in `*.lance/data/` and `*.lance/_transactions/`
Unable to format dataset to CUDA Tensors

Hi,

I came across this [link](https://huggingface.co/docs/datasets/torch_tensorflow.html) where the docs show show to convert a dataset to a particular format. I see that there is an option to convert it to tensors, but I don't see any option to convert it to CUDA tensors.

I tried this, but Dataset doesn't support assignment:
```
  columns=['input_ids', 'token_type_ids', 'attention_mask', 'start_positions','end_positions']

        samples.set_format(type='torch', columns = columns)
        for column in columns:
            samples[column].to(torch.device(self.config.device))
```
There should be an option to do so, or if there is already a way to do this, please let me know.

Thanks,
Gunjan


Hi ! You can get CUDA tensors with

```python
dataset.set_format("torch", columns=columns, device="cuda")
```

Indeed `set_format` passes the `**kwargs` to `torch.tensor`

Hi @lhoestq,

Thanks a lot. Is this true for all format types?

As in, for 'torch', I can have `**kwargs` to `torch.tensor` and for 'tf' those args are passed to `tf.Tensor`, and the same for 'numpy' and 'pandas'?

Yes the keywords arguments are passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
We don't support the kwargs for pandas on the other hand.

Thanks @lhoestq,
Would it be okay if I added this to the docs and made a PR?

Sure ! Feel free to open a PR to improve the documentation :) 

Closing this issue as it has been resolved.

@lhoestq 

Hi,

I tried the above with 4.5.0, but this seems to not work anymore..
Revert "feat: avoid some copies in torch formatter (#7787)"

This reverts commit c412a6f5a50955e141c5169bf7abe005d10228d2 (I assume it was ai generated which makes it hard for me to review and make sure it doesn't have bad edge cases, but lmk if it wasn't, anyways it didn't take into account the torch kwargs which are responsible for sending the data to the correct device)


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7961). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix all exhausted without replacement

Fix interleave_datasets "all_exhausted_without_replacement" stopping strategy

- Corrected logic to ensure each sample is picked exactly once when using 
  stopping_strategy="all_exhausted_without_replacement".
- Adjusted boolean stopping condition to properly track dataset exhaustion.
- Added test to verify the last element is included as expected.



Hi maintainers 👋  
This PR fixes `all_exhausted_without_replacement` to ensure all samples are emitted exactly once,  
and adds a regression test reproducing the reported issue.

Happy to adjust the implementation or split changes if you’d prefer.
Thanks!


We just merged https://github.com/huggingface/datasets/pull/7955 which also fix the issue, thanks for investigating !

I'm closing this one if you don't mind
Add examples for Lance datasets

## Summary

Updated the Lance integration docs to match the official dataset cards and expand coverage of multimodal workflows in `use_with_lance.mdx`.

## Details

- Added a Lance format-focused guidance page for multimodal examples:
   - Stream from the Hub via `datasets` API
   - Use Lance's dataset API to pass in a `hf://...` path specifier,
   - Scan, vector search, schema evolution examples
   - "Export to Lance" example for materializing filtered subsets locally
- Aligned all dataset names/paths to existing datasets so users can immediately test it out: `lance-format/laion-1m` (image) and the `lance-format/openvid-lance` (video)

Depending on the questions we get from users, we may add more descriptive examples as we along, but this should be a good start!


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7950). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @prrao87 ! All the main examples with pylance should be in their dedicated documentation page IMO. Let's document Lance as a supported format in the `datasets` docs, and move the `pylance` docs and examples there instead if you don't mind: https://github.com/huggingface/hub-docs/pull/2164

I think for the `datasets` docs we can simply make sure Lance is mentioned in the lists of supported formats, and have a section in the Video documentation pages alongside WebDataset for example:

* [video_load.md](https://huggingface.co/docs/datasets/video_load) for loading 
* [video_dataset.md](https://huggingface.co/docs/datasets/video_dataset) for dataset creation
* [loading_methods.md](https://huggingface.co/docs/datasets/package_reference/loading_methods) the reference which lists all loading methods and configs

Hi @lhoestq, what do you mean by this?
> Let's document Lance as a supported format in the datasets docs

Where exactly would Lance be mentioned? I wasn't able to find a section titled "supported formats" in the docs. The rest of the parts of your response made sense (can make those updates). Thanks!

Hi @lhoestq, I've updated the docs in various places, to show how to use Lance for image creation/loading and also video creation/loading. I've also removed the `use_with_lance.mdx` page's contents over to the Hub docs by enhancing that page, as you mentioned.

I think this should cover all your feedback points! Please let me know if anything's missing. I'll be back in a future PR with audio examples, I think the HF community would benefit from using Lance format for audio storage/retrieval, too.

Thanks! We'll get those fixed and send more PRs ;)
Fix interleave_datasets with all_exhausted_without_replacement strategy

When using interleave_datasets with stopping_strategy="all_exhausted_without_replacement" and probabilities=None, the function was incorrectly falling into the undersampling branch, causing it to stop at min(lengths) instead of continuing until all datasets were exhausted.

This fix adds a specific branch to handle the all_exhausted_without_replacement case when probabilities=None. The new logic cycles through all datasets round by round, adding elements from each dataset until all are exhausted, ensuring each element appears exactly once.

Example fix:
- Input: d1=[0,1,2], d2=[10,11,12,13], d3=[20,21,22]
- Before: [0, 10, 20, 1, 11, 21, 2, 12, 22]
- After: [0, 10, 20, 1, 11, 21, 2, 12, 22, 13]

🤖 Generated with [Claude Code](https://claude.com/claude-code)


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7955). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
datasets.map(f, num_proc=N) hangs with N>1 when run on import

### Describe the bug

If you `import` a module that runs `datasets.map(f, num_proc=N)` at the top-level, Python hangs.


### Steps to reproduce the bug

1. Create a file that runs datasets.map at the top-level:

```bash
cat <<EOF > import_me.py
import datasets

the_dataset = datasets.load_dataset("openai/openai_humaneval")
the_dataset = the_dataset.map(lambda item: item, num_proc=2)
EOF
```

2. Start Python REPL:

```bash
uv run --python 3.12.3 --with "datasets==4.0.0"  python3
Python 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
```

3. Import the file:

```python
import import_me
````

Observe hang.

### Expected behavior

Ideally would not hang, or would fallback to num_proc=1 with a warning.

### Environment info


- `datasets` version: 4.0.0
- Platform: Linux-6.14.0-29-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.34.4
- PyArrow version: 21.0.0
- Pandas version: 2.3.2
- `fsspec` version: 2025.3.0



Hii !! Frosh from India taking issue #7756. 

Plan - Add try/except multiprocessing. Start Method, Context +  Fallback to num_proc = 1. 

Forked, coding now... PR in a few
Count examples in lance

```python
In [1]: from datasets import load_dataset_builder, StreamingDownloadManager

In [2]: b = load_dataset_builder("lance-format/openvid-lance")
Resolving data files: 100%|█| 240/240 [00:00<00:00, 42675.64it/s

In [3]: b.count_examples(StreamingDownloadManager())
Out[3]: {'train': 937957}
```


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7969). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Infer types from lance blobs

Ex: infer Video() type in https://huggingface.co/datasets/lance-format/openvid-lance and Image() type in https://huggingface.co/datasets/lance-format/laion-1m

```python
from datasets import load_dataset

ds = load_dataset("lance-format/laion-1m", streaming=True, split="train")
print(ds.feature["image"])
# Image()

ds = load_dataset("lance-format/openvid-lance", streaming=True, split="train")
print(ds.feature["video_blob"])
# Video()
```


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7966). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.