AniketTec/github-issues huggingface dataset:
Title: Fix: make environment variable naming consistent (issue #7998)
State: closed

Fix: https://github.com/huggingface/datasets/issues/7998

**Summary**

Addressed the inconsistent environment variable name HF_DATASETS_DISABLE_PROGRESS_BARS used for toggling progress bars, particularly in docstrings and warning messages.

**Changes**

Updated function docstrings to use the correct environment variable name

Corrected warning messages in tqdm.py to ensure consistency

Comments:
Hi @lhoestq ,
I’ve addressed the inconsistent environment variable name HF_DATASETS_DISABLE_PROGRESS_BARS in docstrings and warning messages. Please review the changes and let me know if there’s anything else I should update to complete this issue fix.
Title: Too many dataloader workers: 4 (max is dataset.num_shards=3). Stopping 1 dataloader workers.
State: open

Hi !
I’m working on training with a large-scale dataset (100+ Parquet files) using lazy loading, and I’m struggling to understand/optimize the num_shards setting— in the lerobot repo: streaming_datasets.py:
```
from datasets import load_dataset
self.hf_dataset: datasets.IterableDataset = load_dataset(
            self.repo_id if not self.streaming_from_local else str(self.root),
            split="train",
            streaming=self.streaming,
            data_files="data/*/*.parquet",
            revision=self.revision,
        )

self.num_shards = min(self.hf_dataset.num_shards, max_num_shards)
```

```
dataloader = torch.utils.data.DataLoader(
            datasets[sub_idx],
            num_workers=datasets[sub_idx].num_shards, #cfg.num_workers,
            batch_size=cfg.batch_size,
            shuffle=shuffle and not cfg.dataset.streaming,
            sampler=sampler,
            collate_fn=FlowerDataCollator(),
            pin_memory=device.type == "cuda",
            drop_last=True,
            prefetch_factor=2 if cfg.num_workers > 0 else None,
            )
```

What exactly does hf_dataset.**num_shards** represent? Is it safe to manually override/edit num_shards?

My batch loading is slower than expected (2-3s per batch) despite num_worker cannot be bigger with warning: Too many dataloader workers: 4 (max is dataset.num_shards=3). Stopping 1 dataloader workers. 

Even use num_workers=datasets[sub_idx].num_shards, the waring is still exist! (my num_worker is 4 and hf_dataset.num_shards is 100+, so the datasets[sub_idx].num_shards=4)
Why does the "too many workers" warning persist even when num_workers equals dataset.num_shards—and how do I fix this?

Thanks so much for any insights or help with this! Really appreciate your time and expertise 😊



Comments:
Hi, thanks for the clear question and code snippet!

From my understanding, **hf_dataset.num_shards** represents the number of actual iterable partitions that the streaming dataset can be read from in parallel. This is not the same as the number of underlying Parquet files—many files can be grouped into a much smaller number of shards internally.

The warning seems to persist because **datasets[sub_idx]** is a sliced or sharded view of the original dataset, and its effective num_shards can be smaller than expected after this step. So even if **num_workers** is set equal to **datasets[sub_idx].num_shards**, additional internal sharding can still result in fewer usable shards, which triggers the warning.

Because of this, manually overriding **num_shards** does not appear to increase real parallelism and may just create extra workers that stay idle. For streaming datasets, it seems safer to let the dataset control sharding and to cap the number of workers to the effective shard count (or even use fewer workers and adjust batch size instead), since performance is often I/O-bound.

Hope this helps clarify what’s going on!
Title: [doc] Inconsistant ENV VAR Name for Progress Bar Toggle
State: closed

Code uses env var name `HF_DATASETS_DISABLE_PROGRESS_BARS`.

https://github.com/huggingface/datasets/blob/025593f2f0722f31fc136e0ae45da4ff44d4416a/src/datasets/config.py#L221-L226

Docstrings and warnings report env var name `HF_DATASETS_DISABLE_PROGRESS_BAR` without the ending `S`.

https://github.com/huggingface/datasets/blob/025593f2f0722f31fc136e0ae45da4ff44d4416a/src/datasets/utils/tqdm.py#L61-L73

https://github.com/huggingface/datasets/blob/025593f2f0722f31fc136e0ae45da4ff44d4416a/src/datasets/utils/tqdm.py#L78-L90

https://github.com/huggingface/datasets/blob/025593f2f0722f31fc136e0ae45da4ff44d4416a/src/datasets/utils/tqdm.py#L95-L100

This affects doc webpages as well, e.g., see https://huggingface.co/docs/datasets/en/package_reference/utilities#datasets.enable_progress_bars.

Comments:
Title: fix: Dataset.map writer initialization when first examples return None
State: closed

Fixes #7990

## Summary

When Dataset.map is called and the first examples processed return None, the writer is never properly initialized, causing a ValueError.

## Changes

- Modified _map_single to initialize the writer early if the first batch returns empty results
- Ensures writer is set before the first call to writer.write_batch

## Test Plan

- Added test case that reproduces the bug
- Verified the fix resolves the issue
- Existing tests still pass

Comments:
Closing in favor of #7996 which addresses the same issue with the same fix.
Title: Fix Dataset.map writer initialization when early examples return None
State: open

## Summary

Fixes #7990 

This PR fixes a bug in `Dataset.map()` where the writer initialization was incorrectly tied to the index being 0, causing crashes when the map function returns `None` for the first few examples and later returns a dict.

### Changes

- **Non-batched mode** (line 3676): Changed from `if i == 0:` to `if writer is None:`
- **Batched mode** (line 3701): Changed from `if i and i[0] == 0:` to `if writer is None:`

### Why This Fix Works

The original code assumed that `update_data` would always be determined by the time the first example (i=0) was processed. However, `update_data` is set lazily after processing each example - it becomes `True` when the function first returns a non-None value.

If a function returns `None` for early examples and a dict for later ones:
1. At i=0, the function returns `None`, so `update_data` remains `None`  
2. Writer is NOT initialized (because we're not updating data)
3. At i=2, the function returns a dict, so `update_data` becomes `True`
4. **Old code**: Tries to use `writer` (still None) because i != 0 → crash
5. **New code**: Checks `if writer is None` and initializes it → works correctly

### Test Plan

The fix can be verified with this minimal test case from the issue:

```python
from datasets import Dataset

ds = Dataset.from_dict({"x": [1, 2, 3]})

def fn(example, idx):
    if idx < 2:
        return None
    return {"x": [example["x"] * 10]}

# Should work without errors
result = list(ds.map(fn, with_indices=True))
print(result)  # [{'x': 1}, {'x': 2}, {'x': [30]}]
```

**Before this fix**: Crashes with `AttributeError: 'NoneType' object has no attribute 'write'`  
**After this fix**: Works correctly

### Related

This fix ensures the writer is initialized the first time a non-None value is returned, regardless of which example index that occurs at. This makes the code more robust to different map function behaviors.

Comments:
Friendly ping - any chance someone could take a look at this when they get a chance? Happy to make any changes if needed.
Title: Bump fsspec upper bound to 2026.2.0 (fixes #7994)
State: closed

Fixes #7994 and e.g. “Bumps fsspec upper bound so the latest version can be used; CI will validate compatibility.”

Comments:
Apologize for the ping @lhoestq but I wanted to know whether this is something you could look into? And more generally whether you have a policy to regularly bump fsspec? Thanks 🙏

thanks for opening a PR, I'm triggering the CI and I'll merge if it's all green :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7995). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Bump fsspec upper bound constraint
State: closed

Would it be possible to bump fsspec upper bound to the latest (2026.2.0)?

I saw you had some API compat issues in the past (https://github.com/huggingface/datasets/issues/7326) and I understand the need for an upper bound.

But I wonder if you think your CI and tests are a good proxy to catch fsspec API breakage? If that's the case then triggering dataset CI with the latest version of fsspec should tell us whether it's all good right?

Happy to open a PR if needed.

Comments:
Title: :sparkles: Add 'SparseCsv' builder and 'sparse_collate_fn' for efficient high-dimensional sparse data loading
State: open

This PR introduces a new dataset builder, SparseCsv, designed to handle "wide" tabular datasets (e.g., 100k+ columns common in transcriptomics, sparse NLP features, or recommender systems) that are typically too large to load into memory as dense Arrow tables.

It also adds a utility function, `sparse_collate_fn`, to seamlessly convert these sparse examples into `torch.sparse` or `scipy.sparse` matrices during training.

This PR should fix #7377 

Comments:
Title: Add `IterableDataset.reshard()`
State: closed

To increase the number of shards of a dataset, you can use [`IterableDataset.reshard`]:

```py
>>> dataset
IterableDataset({
    features: ['label', 'title', 'content'],
    num_shards: 4
})
>>> dataset.reshard()
IterableDataset({
    features: ['label', 'title', 'content'],
    num_shards: 3600
})
```

The resharding mechanism depends on the dataset file format.
For example for Parquet, it reshards using row groups instead of having one file per shard.

We can implement other formats later (e.g. JSON Lines, CSV can be split by recovering line boundaries from arbitrary locations)

Other details:
* fixed concatenate after shuffling, now it correctly shuffles the shards: close https://github.com/huggingface/datasets/issues/7196
* fixed interleave after split_by_node: close https://github.com/huggingface/datasets/issues/7868
* changed a bit how the effective seed works in multi node/worker/epoch situations

related to https://github.com/huggingface/datasets/issues/7917

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7992). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: list(api.list_datasets()) giving jsondecode error
State: closed

i am using the python api wrapper to list all datasets available on hugging face. This is for research, and i need all datasets to determine which % have language tags and other related questions requiring the total list. However, the following code that worked a few months ago:

```
from huggingface_hub import HfApi
api = HfApi(token=token)
datasets = list(api.list_datasets())
```

now gives a JSONDecodeError when reaching 2000 results. My understanding is that this is a pagination issue, as there is no cursor exposed to the API and so it doesnt know where to read from after the limit. 

Is there any way to find all datasets available and combine them into a list?



Comments:
Title: Dataset.map crashes when first examples return None and later examples return dict — writer not initialized
State: open

### Describe the bug

I detected a serious [bug from datasets/arrow_dataset.py](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L3676)
---

**Description of the bug**
`Dataset.map` crashes with `writer is None` when the map function returns `None` for the first few examples and a dictionary (or `pa.Table` / DataFrame) for later examples. This happens because the internal writer is initialized only when `i == 0` (or `i[0] == 0` in batched mode), but `update_data` is determined lazily after processing the first example/batch.

**Steps to reproduce**

```python
from datasets import Dataset

ds = Dataset.from_dict({"x": [1, 2, 3]})

def fn(example, idx):
    if idx < 2:
        return None
    return {"x": [example["x"] * 10]}

list(ds.map(fn, with_indices=True))
```

**Expected behavior**

* The function should work regardless of when `update_data` becomes `True`.
* Writer should be initialized the first time a non-`None` return occurs, not tied to the first index.

**Environment info**

* `datasets` version: <insert your version>
* Python version: 3.12
* OS: <insert your OS>

**Suggested fix**
Replace `if i == 0` / `if i[0] == 0` checks with `if writer is None` when initializing the writer.

---

### Steps to reproduce the bug

Here's a ready-to-use version you can paste into that section:

---

### Steps to reproduce the bug

```python
from datasets import Dataset

# Create a minimal dataset
ds = Dataset.from_dict({"x": [1, 2, 3]})

# Define a map function that returns None for first examples, dict later
def fn(example, idx):
    if idx < 2:
        return None
    return {"x": [example["x"] * 10]}

# Apply map with indices
list(ds.map(fn, with_indices=True))
```

**Expected:** function executes without errors.
**Observed:** crashes with `AttributeError: 'NoneType' object has no attribute 'write'` because the internal writer is not initialized when the first non-None return happens after i > 0.

---

This is minimal and clearly demonstrates the exact failure condition (`None` early, `dict` later).

### Expected behavior

---

**Expected behavior**
The `Dataset.map` function should handle map functions that return `None` for some examples and a dictionary (or `pa.Table` / DataFrame) for later examples. In this case, the internal writer should be initialized when the first non-`None` value is returned, so that the dataset can be updated without crashing. The code should run successfully for all examples and return the updated dataset.

---

### Environment info

- python3.12
- datasets==3.6.0 [but the latest version still has this problem]
- transformers==4.55.2


Comments:
Title: Remove pre-release workaround in CI for `transformers v5` and `huggingface_hub v1`
State: closed

This PR removes workaround for pre-release `transformers v5.*` / `huggingface_hub v1.*` in `test_py314_future` job since they are now officially released.

cc @Wauplin just for viz since you introduced this in https://github.com/huggingface/datasets/pull/7783.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7989). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Remove unused data files optims
State: closed

this fixes module inference when there are many metadata files

e.g. the lance dataset at https://huggingface.co/datasets/davanstrien/encyclopaedia-britannica-lance has > 200 metadata files

those optims are not used anymore, they come from a time we were dealing with slow data files iterators instead of lists

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7985). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Add Zarr streaming support (POC)
State: open

Add initial Zarr streaming support (POC).

This introduces a `zarr` packaged module and docs/tests to validate basic loading.

Note: I pushed a follow-up commit to fix an accidental duplication in `benchmarks/benchmark_zarr_streaming.py` (file now contains a single benchmark script).

Comments:
Hi! It looks like the GitHub Actions check suites for this PR are in `action_required` (no workflows actually ran). This is usually due to fork workflow approval.

Could a maintainer please approve/run the workflows so CI can execute? Happy to address anything CI flags once it runs.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7983). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Great PR ! It looks mostly good to me, I added a fes suggestions.
> 
> Btw the current implementation for streaming returns a StreamingIterableDataset with `num_shards=1`, which corresponds to 1 metadata file.
> 
> For large datasets it's maybe more practical to have a more finegrained sharding, e.g. at data file level. Wdyt ?

Thanks for the review. I agree on sharding: num_shards currently tracks the number of input Zarr stores, so it is often 1, which is not ideal for large datasets.

I can implement finer-grained sharding using row-range shards aligned to axis-0 chunk boundaries (instead of one shard per metadata file), with a config knob like rows_per_shard / target_num_shards.

I can add this in this PR if you want it before merge, or do it as a focused follow-up PR right after.

> Feel free to include it in this PR if it's ok for you.
> 
> Thanks for the changes, I just have one last comment I didn't think of earlier. The rest looks good to me :)
> 
> btw it seems some tests are failing in `tests/packaged_modules/test_zarr.py`

Done  ✅ 🤗
included the sharding/counting follow-ups in this PR
including _generate_num_examples, _CountableBuilderMixin, finer-grained sharding controls, and the robustness fixes/tests/docs.

@lhoestq quick ping. I pushed the requested updates including sharding and _generate_num_examples. Could you take another look when you have time?

I can rerun or fix CI items right away if you want.
Title: Fix unstable tokenizer fingerprinting (enables map cache reuse)
State: open

Fix unstable dataset fingerprinting when hashing `PreTrainedTokenizerFast`.

Some tokenizers backed by `tokenizers.Tokenizer` mutate runtime settings (padding/truncation) when called, which can change the serialized state and make dataset fingerprints unstable. That prevents `.map(load_from_cache_file=True)` from reusing cache files.

Fix: when hashing, temporarily disable backend padding/truncation so runtime settings don’t affect the fingerprint, then restore the original settings.

Includes a regression test showing `Hasher.hash(tokenizer)` stays stable after calling the tokenizer.

Comments:
Hi! It looks like the GitHub Actions check suites for this PR are in `action_required` (no workflows actually ran). This is usually due to fork workflow approval.

Could a maintainer please approve/run the workflows so CI can execute? Happy to address anything CI flags once it runs.

Quick ping on this one. Happy to update anything if you want changes.
Title: Use temp files in push_to_hub to save memory
State: closed

write parquet data to temp files on disk prior to upload to save memory

this is enabled for for datasets loaded using streaming=True/False

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7979). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

And they are never deleted?

Just to be clear @lhoestq, by not deleting the files on successful shard push they will accumulate and eventually fill the disk. By accumulating files, stream processing is effectively broken. Why stream files if you end up with all of them on disk?

Is there no review process in place for PRs? Even an automated one would have gotten this.

arf what an oversight, thanks for reporting I'll do a patch release asap

4.6.1 is released with a fix !
Title: Fix 4910 kwargs
State: open

Fix #4910 : Merge duplicate kwargs in `load_dataset_builder()`

Problem: load_dataset("dataset", base_path="./data")` gives TypeError("multiple values for keyword 'base_path')

Fix: {**builder_kwargs, **config_kwargs} to user kwargs override dataset defaults

Repro : python

Before: TypeError

load_dataset("rotten_tomatoes", base_path="./sample_data")

After: Works 



Comments:
Title: Updated get_dataset_config_names returning default in offline mode 
State: closed

When a dataset is cached and accessed in offline mode, get_dataset_config_names was returning default instead of the actual cached config names. This happened because CachedDatasetModuleFactory.get_module returned a DatasetModule without builder_configs_parameters, causing the fallback to default in get_dataset_config_names.                                                                                             
                                                                                                                                                   
The fix reads config_name from each dataset_info file in the cache directory and includes them as builder_configs_parameters in the returned DatasetModule. Invalid or missing dataset_info.json files are handled.

**Testing:**                                                                                                                                                                                                                                                                                         
1. Download a dataset in online mode so it gets cached                                                                                           
2. Switch to offline mode and call get_dataset_config_names                                                                                      
3. Verify it returns the cached config names instead of ['default']                                                                              
                                                                                                                                                   
**Example:**                                                                                                                                                                                                                                                                                         

- HF_DATASETS_OFFLINE=0 HF_HOME="/tmp/hftemp" python -c "import datasets; datasets.load_dataset('cais/mmlu', 'all')"                                                                                                                                                                                
- HF_DATASETS_OFFLINE=1 HF_HOME="/tmp/hftemp" python -c "import datasets; print(datasets.get_dataset_config_names('cais/mmlu'))"                                                                                                                                                                    
- -> Expected output: ['all'] 

Fixes https://github.com/huggingface/datasets/issues/7947 

Comments:
Title: Fix duplicate kwargs in load_dataset_builder
State: open

Avoid passing duplicate keyword arguments to `load_dataset_builder`.

Some module factories provide values in `builder_kwargs` (e.g. `base_path`), and users can also pass the same keys via `config_kwargs`, which raises:
`TypeError: ... got multiple values for keyword argument ...`.

Fix: if `config_kwargs` is provided, drop overlapping keys from `builder_kwargs` (keep the user-provided values).

Includes a regression test.

Comments:
Quick ping on this one. Happy to update anything if you want changes.
Title: Fix resolve_pattern for local symlinked files
State: open

Fix `resolve_pattern` for *local symlinked files*.

Problem: on the local `file://` filesystem, `fsspec` can report symlinks as `type=="other"` and omit the `islink` flag, so symlinked files are skipped.

Fix: when `protocol=="file"`, treat `os.path.islink(filepath)` as a link candidate and include it if it resolves to a regular file.

Includes a regression test in `tests/test_data_files.py`.

Comments:
Hi! It looks like the GitHub Actions check suites for this PR are in `action_required` (no workflows actually ran). This is usually due to fork workflow approval.

Could a maintainer please approve/run the workflows so CI can execute? Happy to address anything CI flags once it runs.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7973). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Quick ping on this one. Happy to update anything if you want changes.

I can rerun or fix CI items right away if you want.