AniketTec/github-issues huggingface dataset:
Title: Fix Audio docstring by removing unsupported mono argument
State: closed

This PR fixes issue #7745.

Who can review:
@lhoestq 

Comments:
I think we can keep the arg and add the missing torch.mean() in the Audio.decode_example method

> I think we can keep the arg and add the missing torch.mean() in the Audio.decode_example method

Thank you @lhoestq. I will add torch.mean().

fixed by #7840
Title: Support huggingface_hub v0.x and v1.x
State: closed

Related to https://github.com/huggingface/huggingface_hub/issues/3340.

This PR adapts `datasets` to be compatible with both huggingface_hub v0.x and v1.x. 

In practice nothing else should change (I've checked the codebase). The `HfHubHTTPError` is a base error defined in `huggingface_hub` that inherits from `requests.HTTPError` in v0.x and will inherit from `httpx.HTTPError` in v1.x. It has been introduced ~2 years ago so it's fine to use it right now (i.e. no need to wait for v1.x release or bump minimal version).

Most of the changes have been around the test suite to make sure that tests are passing with both `requests` and `httpx` backends. Mid-term it would be good to completely remove the `requests` dependency from `datasets` but that's an orthogonal topic. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7783). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq could you have a look at this PR please? It makes `datasets` compatible with the upcoming `huggingface_hub` v1.0 release while staying compatible with 0.x. Look at PR description for more details.

The CI is currently failing because of 429 rate limit errors but otherwise everything should be fine (made extensive local and ci tests to ensure that). Let me know if you notice anything weird. PR is ready to be merged "as-is" in my opinion.
Title: BIGPATENT dataset inaccessible (deprecated script loader)
State: closed

dataset: https://huggingface.co/datasets/NortheasternUniversity/big_patent

When I try to load it with the datasets library, it fails with:

RuntimeError: Dataset scripts are no longer supported, but found big_patent.py


Could you please publish a Parquet/Arrow export of BIGPATENT on the Hugging Face so that it can be accessed with datasets>=4.x.



Comments:
Hi ! I opened https://huggingface.co/datasets/NortheasternUniversity/big_patent/discussions/7 to update the dataset, hopefully it's merged soon !

The dataset now works with `datasets` v4 ! closing this issue
Title: [FIX] force spawning pool for MacOS
State: open

This PR gets multiprocessing to work on mac os:

```python
from datasets import load_dataset

ds = load_dataset("fka/awesome-chatgpt-prompts", split="train").take(100)

ds = ds.map(lambda x: x, num_proc=4)

ds.push_to_hub("burtenshaw/dataset-test", num_proc=4)
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7778). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

After more discussions on slack, we can switch the default to spawn.

Let's use multiprocess instead of multiprocessing and maybe add a check to apply this only on Macos ?
Title: push_to_hub not overwriting but stuck in a loop when there are existing commits
State: closed

### Describe the bug

`get_deletions_and_dataset_card` stuck at error a commit has happened error since push to hub for http error 412 for tag 4.1.0. The error does not exists in 4.0.0. 

### Steps to reproduce the bug

Create code to use push_to_hub, ran twice each time with different content for datasets.Dataset.
The code will stuck in time.sleep loop for `get_deletions_and_dataset_card`. If error is explicitly printed, the error is HTTP 412.

### Expected behavior

New datasets overwrite existing one on repo.

### Environment info

datasets 4.1.0

Comments:
HTTP 412 means a commit happened in the meantime, so `get_deletions_and_dataset_card` has to retry to get the latest version of the dataset card and what files to delete based on the latest version of the dataset repository

Are you running other operations in the dataset repo for your push_to_hub ?

There was only a map() followed by a push_to_hub(). The repo had one prior commit also by using push_to_hub(). The error disappeared when I downgraded datasets to 4.0.0.

It is reproducible if you use finegrained token with Read+Write (Open pull request) access to only that repo.

Ah it was due to the use of requests_cache with POST methods, closing this.
Title: [docs] Fix broken WebDataset link on “Create a video dataset” page
State: closed

### What
Fix the "WebDataset documentation" link on the Create a video dataset page to point
to the correct section on the video load guide.

### Why
The link currently points to an external repo, but the Hugging Face docs
have an internal "WebDataset" section under video_load.

### How
- docs/source/video_dataset.mdx: updated link to
  `https://huggingface.co/docs/datasets/main/en/video_load#webdataset`

### Issue
Fixes #7699


Comments:
Title: Error processing scalar columns using tensorflow.
State: open

`datasets==4.0.0`
```
columns_to_return = ['input_ids','attention_mask', 'start_positions', 'end_positions']
train_ds.set_format(type='tf', columns=columns_to_return)
```
`train_ds`:
```
train_ds type: <class 'datasets.arrow_dataset.Dataset'>, shape: (1000, 9)
columns: ['question', 'sentences', 'answer', 'str_idx', 'end_idx', 'input_ids', 'attention_mask', 'start_positions', 'end_positions']
features:{'question': Value('string'), 'sentences': Value('string'), 'answer': Value('string'), 'str_idx': Value('int64'), 'end_idx': Value('int64'), 'input_ids': List(Value('int32')), 'attention_mask': List(Value('int8')), 'start_positions': Value('int64'), 'end_positions': Value('int64')}
```
`train_ds_tensor = train_ds['start_positions'].to_tensor(shape=(-1,1))` hits the following error:
```
AttributeError: 'Column' object has no attribute 'to_tensor'
```
`tf.reshape(train_ds['start_positions'], shape=[-1,1])` hits the following error:
```
TypeError: Scalar tensor has no `len()`
```

Comments:
Using tf.convert_to_tensor works fine:

```
import tensorflow as tf

start_pos = tf.convert_to_tensor(train_ds['start_positions'], dtype=tf.int64)
start_pos = tf.reshape(start_pos, [-1, 1])
```


Alternatively, using the built-in to_tf_dataset also avoids the issue:

```
train_tf = train_ds.to_tf_dataset(
    columns=['input_ids','attention_mask'],
    label_cols=['start_positions','end_positions'],
    shuffle=True,
    batch_size=32
)
```

```
    start_pos = tf.convert_to_tensor(self._train_ds['start_positions'], dtype=tf.int64)
  File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.13/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.13/site-packages/tensorflow/python/framework/constant_op.py", line 108, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: TypeError: Scalar tensor has no `len()`
Traceback (most recent call last):

  File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.13/site-packages/tensorflow/python/framework/ops.py", line 361, in __len__
    raise TypeError("Scalar tensor has no `len()`")

TypeError: Scalar tensor has no `len()`
```

`to_tf_dataset` works perfectly.
Title: Add support for arrow iterable when concatenating or interleaving
State: closed

Fixes a case when concatenating or interleaving datasets with `with_format(...)` call was slower.
Details here: https://github.com/huggingface/datasets/issues/6637

@lhoestq I tried to minimize the duplication between iter and iter_arrow methods, not sure if this is against the design, can separate those if needed.

Comments:
Seeing the following numbers on the script shared in the original issue. (MacBook Pro M4)

```
1000it [00:00, 4074.63it/s] # ds_a.with_format("torch")
1000it [00:01, 593.39it/s] # ds_a.shuffle()
1999it [00:03, 594.09it/s] # datasets.interleave_datasets([ds_a, ds_b])
1000it [00:00, 5382.45it/s] # ds_a.shuffle().with_format("torch") <--- Was slow <2it/s
1999it [00:00, 4743.45it/s] # datasets.interleave_datasets([ds_a, ds_b]).with_format("torch") <--- Was slow <2it/s
1999it [00:20, 98.94it/s]  # torch.tensor(example["tensor"])
```


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7771). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq I've implemented the iteration on arrow as separate methods, can you take another look/trigger ci? 

@lhoestq Any idea why the integration tests are failing, is this expected? Anything I can do on my side?

They seem unrelated to your changes. Merging :)
Title: Fix: Correct float feature generation in `generate_examples` 
State: closed

This PR fixes a bug in the `generate_examples` function where `datasets.Value` features with a `float` dtype were incorrectly generated using `np.random.randint`. This resulted in integer values being cast to float, which is not representative of true floating-point data.


**Key changes include:**

* Added explicit handling for `float` features using `np.random.rand` to generate continuous values.
* Introduced fail-fast type checks for unsupported dtypes to improve robustness.
* Added validation for sequence features to ensure `seq_shapes` is provided.

### Before Fix

Float features were generated incorrectly as integers cast to float:

```text
- Example 0:
- int_feature: 0
- float_feature: 9.0  <-- Incorrect: An integer disguised as a float
- string_feature: The small grey turtle was surprisingly fast...
- seq_feature: [0.3048 0.4291 0.4283]
```

### After Fix

Float features are now correctly generated as continuous numbers in the range [0, 1):
```text
+ Example 0:
+ int_feature: 0
+ float_feature: 0.0183  <-- Correct: A true random float
+ string_feature: The small grey turtle was surprisingly fast...
+ seq_feature: [0.9237 0.7972 0.8526]
```
#### Note: This PR is a follow-up/fix of the previously closed PR #7769 for clarity and context.


Comments:
Hi @lhoestq, just a gentle follow-up on this PR.
Title: Fix: Correct float feature generation in `generate_examples`
State: closed

This PR fixes a bug in the `generate_examples` function where `datasets.Value` features with a `float` dtype were incorrectly generated using `np.random.randint`. This resulted in integer values being cast to float, which is not representative of true floating-point data.

**Key changes include:**
1.  Added explicit handling for float features using `np.random.rand` to generate continuous values.
2.  Introduced fail-fast type checks for unsupported dtypes to improve robustness.
3.  Added validation for sequence features to ensure `seq_shapes` is provided.


### Before Fix
Float features were generated incorrectly as integers cast to float:

```text
Example 0:
int_feature: 0
float_feature: 9.0  <-- Incorrect: An integer disguised as a float
string_feature: The small grey turtle was surprisingly fast...
seq_feature: [0.3048 0.4291 0.4283]
```
### After Fix
Float features are now correctly generated as continuous numbers in the range [0, 1):
```text

Example 0:
int_feature: 0
float_feature: 0.0183  <-- Correct: A true random float
string_feature: The small grey turtle was surprisingly fast...
seq_feature: [0.9237 0.7972 0.8526]

Comments:
Title: Custom `dl_manager` in `load_dataset`
State: open
Labels: description: New feature or request
name: enhancement

### Feature request
https://github.com/huggingface/datasets/blob/4.0.0/src/datasets/load.py#L1411-L1418

```

def load_dataset(
    ...
    dl_manager: Optional[DownloadManager] = None,  # add this new argument
    **config_kwargs,
) -> Union[DatasetDict, Dataset, IterableDatasetDict, IterableDataset]:

    ...

    # Create a dataset builder
    builder_instance = load_dataset_builder(
        path=path,
        name=name,
        data_dir=data_dir,
        data_files=data_files,
        cache_dir=cache_dir,
        features=features,
        download_config=download_config,
        download_mode=download_mode,
        revision=revision,
        token=token,
        storage_options=storage_options,
        **config_kwargs,
    )

    # Return iterable dataset in case of streaming
    if streaming:
        return builder_instance.as_streaming_dataset(split=split)

    # Note: This is the revised part
    if dl_manager is None:
        if download_config is None:
            download_config = DownloadConfig(
                cache_dir=builder_instance._cache_downloaded_dir,
                force_download=download_mode == DownloadMode.FORCE_REDOWNLOAD,
                force_extract=download_mode == DownloadMode.FORCE_REDOWNLOAD,
                use_etag=False,
                num_proc=num_proc,
                token=builder_instance.token,
                storage_options=builder_instance.storage_options,
            )  # We don't use etag for data files to speed up the process

        dl_manager = DownloadManager(
            dataset_name=builder_instance.dataset_name,
            download_config=download_config,
            data_dir=builder_instance.config.data_dir,
            record_checksums=(
                builder_instance._record_infos or verification_mode == VerificationMode.ALL_CHECKS
            ),
        )

    # Download and prepare data
    builder_instance.download_and_prepare(
        download_config=download_config,
        download_mode=download_mode,
        verification_mode=verification_mode,
        dl_manager=dl_manager,  # pass the new argument
        num_proc=num_proc,
        storage_options=storage_options,
    )

    ...

```

### Motivation

In my case, I'm hoping to deal with the cache files downloading manually (not using hash filenames and save to another location, or using potential existing local files).

### Your contribution

It's already implemented above. If maintainers think this should be considered, I'll open a PR.

Comments:
Title: cast columns to Image/Audio/Video with `storage_options`
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

Allow `storage_options` to be passed in
1. `cast` related operations (e.g., `cast_columns, cast`)
2. `info` related reading (e.g., `from_dict, from_pandas, from_polars`) together with `info.features`
```python3
import datasets

image_path = "s3://bucket/sample.png"
dataset = datasets.Dataset.from_dict({"image_path": [image_path]})

# dataset = dataset.cast_column("image_path", datasets.Image())  # now works without `storage_options`

# expected behavior
dataset = dataset.cast_column("image_path", datasets.Image(), storage_options={"anon": True})
```

### Motivation

I'm using my own registered fsspec filesystem (s3 with customized local cache support). I need to pass cache folder paths `cache_dirs: list[str]` to the filesystem when I read the remote images (cast from file_paths).

### Your contribution

Could help with a PR at weekends

Comments:
A

1

1

Ok

> ### Feature request
> Allow `storage_options` to be passed in
> 
> 1. `cast` related operations (e.g., `cast_columns, cast`)
> 2. `info` related reading (e.g., `from_dict, from_pandas, from_polars`) together with `info.features`
> 
> import datasets
> 
> image_path = "s3://bucket/sample.png"
> dataset = datasets.Dataset.from_dict({"image_path": [image_path]})
> 
> # dataset = dataset.cast_column("image_path", datasets.Image())  # now works without `storage_options`
> 
> # expected behavior
> dataset = dataset.cast_column("image_path", datasets.Image(), storage_options={"anon": True})
> ### Motivation
> I'm using my own registered fsspec filesystem (s3 with customized local cache support). I need to pass cache folder paths `cache_dirs: list[str]` to the filesystem when I read the remote images (cast from file_paths).
> 
> ### Your contribution
> Could help with a PR at weekends



>

#7910
Title: polars dataset cannot cast column to Image/Audio/Video
State: closed

### Describe the bug

`from_polars` dataset cannot cast column to Image/Audio/Video, while it works on `from_pandas` and `from_dict`

### Steps to reproduce the bug

```python3
import datasets
import pandas as pd
import polars as pl

image_path = "./sample.png"

# polars
df = pl.DataFrame({"image_path": [image_path]})
dataset = datasets.Dataset.from_polars(df)
dataset = dataset.cast_column("image_path", datasets.Image())

# # raises Error
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from large_string to struct using function cast_struct


# pandas 
df = pd.DataFrame({"image_path": [image_path]})
dataset = datasets.Dataset.from_pandas(df)
dataset = dataset.cast_column("image_path", datasets.Image())

# # pass
{'image_path': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=338x277 at 0x7FBA719D4050>}


# dict
dataset = datasets.Dataset.from_dict({"image_path": [image_path]})
dataset = dataset.cast_column("image_path", datasets.Image())

# # pass
{'image_path': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=338x277 at 0x7FBA719D4050>}
```


### Expected behavior

`from_polars` case shouldn't raise error and have the same outputs as `from_pandas` and `from_dict`

### Environment info

```
# Name                    Version                   Build  Channel
datasets                  4.0.0                    pypi_0    pypi
pandas                    2.3.1                    pypi_0    pypi
polars                    1.32.3                   pypi_0    pypi
```

Comments:
I fixed this with a combination of `to_dict` and `from_dict`:

```py
datasets.Dataset.from_dict(df.to_dict(as_series=False))
```

@samuelstevens Yeah, I'm using similar workaround as well. But it would be ideal if we can avoid the copy.
Title: Bump dill to 0.4.0
State: closed

This bumps `dill` to 0.3.9 and closes #7510 

It turns out the only thing required to make the tests pass was to extend the version checks to include 0.3.9.

Comments:
Have you tried to run `pytest tests/test_fingerprint.py` ? It seems dill 0.3.9 breaks a lot of tests

```
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_regex - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_tokenizer - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_tokenizer_with_cache - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ignores_line_definition_of_function - AssertionError: 'c48ebfacf8768f50' != '27e49d047c02c83b'
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ipython_function - AssertionError: '65edc6b6d425a8e9' != '9f364fe298fb286a'
FAILED tests/test_fingerprint.py::HashingTest::test_hash_tiktoken_encoding - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_compiled_module - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_generator - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_tensor - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_doesnt_depend_on_order - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_stable - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::test_move_script_doesnt_change_hash - AssertionError: assert b'93072ca404a697db\n' == b'cf89a7e497a97e32\n'
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7763). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @lhoestq! Yes, I did. It's not really `dill` that breaks things. Rather the shims that `datasets` has in place did not include the next version. 

FYI: I also tested it with `dill-0.4.0` and the changes would need to be analogous, but I wanted to be conservative in this PR. 

The NameError is fixed in your PR since it defines the right `log()` function for 0.3.9.

But I'm less sure about the AssertionError that may be related to deterministic hashing or ipython/shell function hashing. We would need to solve these

EDIT: ah actually it does ! cool ! let me update the branch and re-run the CI
Title: Parquet: use data page v2 for efficient page pruning
State: closed

This is needed to enable page pruning with DataFusion, which will be useful for the Dataset Viewer.

Indeed page pruning with DataFusion allows to download only certain pages of a row group, reducing the I/O required to read just a few rows.

But while data page v1 generally works, it's not easy with DataFusion to do page pruning on datasets with nested data. This is because rows can span multiple pages in v1, contrary to v2.

cc @severo for viz

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7762). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Closing this since it looks like the page offset index is enough :)
Title: Comment/feature request: Huggingface 502s from GHA
State: open

This is no longer a pressing issue, but for completeness I am reporting that in August 26th, GET requests to `https://datasets-server.huggingface.co/info\?dataset\=livebench/math` were returning 502s when invoked from [github actions](https://github.com/UKGovernmentBEIS/inspect_evals/actions/runs/17241892475/job/48921123754) (that link will expire eventually, [here are the logs](https://github.com/user-attachments/files/22233578/logs_44225296943.zip)). 

When invoked from actions, it appeared to be consistently failing for ~6 hours. However, these 502s never occurred when the request was invoked from my local machine in that same time period.

I suspect that this is related to how the requests are routed with github actions versus locally. 

Its not clear to me if the request even reached huggingface servers or if its the github proxy that stopped it from going through, but I wanted to report it nonetheless in case this is helpful information. I'm curious if huggingface can do anything on their end to confirm cause. 

And a feature request for if this happens in the future (assuming huggingface has visibilty on it):  A "datasets status" page highlighting if 502s occur for  specific individual datasets could be useful for people debugging on the other end of this!




Comments:
Title: Option for Anonymous Dataset link
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

Allow for anonymized viewing of datasets. For instance, something similar to [Anonymous GitHub](https://anonymous.4open.science/).

### Motivation

We generally publish our data through Hugging Face. This has worked out very well as it's both our repository and archive (thanks to the DOI feature!). However, we have an increasing challenge when it comes to sharing our datasets for paper (both conference and journal) submissions. Due to the need to share data anonymously, we can't use the Hugging Face URLs, but datasets tend to be too large for inclusion as a zip. Being able to have an anonymous link would be great since we can't be double-publishing the data.

### Your contribution

Sorry, I don't have a contribution to make to the implementation of this. Perhaps it would be possible to work off the [Anonymous GitHub](https://github.com/tdurieux/anonymous_github) code to generate something analogous with pointers to the data still on Hugging Face's servers (instead of the duplication of data required for the GitHub version)?

Comments: