noamaanMulla-03/datasets-issues huggingface dataset:
Incompatibile dill version (0.3.9) in datasets 2.18.0 - 3.5.0

### Describe the bug

Datasets 2.18.0 - 3.5.0 has a dependency on dill < 0.3.9. This causes errors with dill >= 0.3.9.

Could you please take a look into it and make it compatible?

### Steps to reproduce the bug

1. Install setuptools >= 2.18.0
2. Install dill >=0.3.9
3. Run pip check
4. Output:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.18.0 requires dill<0.3.9,>=0.3.0, but you have dill 0.3.9 which is incompatible.

### Expected behavior

Pip install both libraries without any errors

### Environment info

Datasets version: 2.18 - 3.5
Python: 3.11


Hi ! We can bump `dill` to 0.3.9 if we make sure it's deterministic and doesn't break the caching mechanism in `datasets`.

Would you be interested in opening a PR ? Then we can run the CI to see if it works

Hi!. Yeah I can do it. Should I make any changes besides dill versions?

There are probably some usage of internal functions from `dill` that we'll need to update in `datasets`

If you run `pytest tests/test_fingerprint.py` you should already have a good idea of what works and what doesn't.
But feel free to open a PR anyway, this way we can run the full CI and see the results


Hi, sorry for no response from my side. I will try to do it today.

Created pull request: [LINK](https://github.com/huggingface/datasets/pull/7535)
Tried to run tests by using command you have send and got few errors:

![Image](https://github.com/user-attachments/assets/acbf1feb-4dd1-416e-a118-c91abe0d188b)

Thanks for running the test ! So it appears we have two issues to fix:
1. 'log' is not defined: it seems an internal `dill` function has disappeared, so we should adapt the `datasets` code that was using it
2. there are some hashes mismatches, which means `dill` doesn't seem to output the same dump when passed the same ipython function twice, or the same function but located at a different line in a python file
Add support for arrow iterable when concatenating or interleaving

Fixes a case when concatenating or interleaving datasets with `with_format(...)` call was slower.
Details here: https://github.com/huggingface/datasets/issues/6637

@lhoestq I tried to minimize the duplication between iter and iter_arrow methods, not sure if this is against the design, can separate those if needed.


Seeing the following numbers on the script shared in the original issue. (MacBook Pro M4)

```
1000it [00:00, 4074.63it/s] # ds_a.with_format("torch")
1000it [00:01, 593.39it/s] # ds_a.shuffle()
1999it [00:03, 594.09it/s] # datasets.interleave_datasets([ds_a, ds_b])
1000it [00:00, 5382.45it/s] # ds_a.shuffle().with_format("torch") <--- Was slow <2it/s
1999it [00:00, 4743.45it/s] # datasets.interleave_datasets([ds_a, ds_b]).with_format("torch") <--- Was slow <2it/s
1999it [00:20, 98.94it/s]  # torch.tensor(example["tensor"])
```


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7771). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq I've implemented the iteration on arrow as separate methods, can you take another look/trigger ci? 

@lhoestq Any idea why the integration tests are failing, is this expected? Anything I can do on my side?

They seem unrelated to your changes. Merging :)
push_to_hub not overwriting but stuck in a loop when there are existing commits

### Describe the bug

`get_deletions_and_dataset_card` stuck at error a commit has happened error since push to hub for http error 412 for tag 4.1.0. The error does not exists in 4.0.0. 

### Steps to reproduce the bug

Create code to use push_to_hub, ran twice each time with different content for datasets.Dataset.
The code will stuck in time.sleep loop for `get_deletions_and_dataset_card`. If error is explicitly printed, the error is HTTP 412.

### Expected behavior

New datasets overwrite existing one on repo.

### Environment info

datasets 4.1.0


HTTP 412 means a commit happened in the meantime, so `get_deletions_and_dataset_card` has to retry to get the latest version of the dataset card and what files to delete based on the latest version of the dataset repository

Are you running other operations in the dataset repo for your push_to_hub ?

There was only a map() followed by a push_to_hub(). The repo had one prior commit also by using push_to_hub(). The error disappeared when I downgraded datasets to 4.0.0.

It is reproducible if you use finegrained token with Read+Write (Open pull request) access to only that repo.

Ah it was due to the use of requests_cache with POST methods, closing this.
[FIX] force spawning pool for MacOS

This PR gets multiprocessing to work on mac os:

```python
from datasets import load_dataset

ds = load_dataset("fka/awesome-chatgpt-prompts", split="train").take(100)

ds = ds.map(lambda x: x, num_proc=4)

ds.push_to_hub("burtenshaw/dataset-test", num_proc=4)
```


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7778). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

After more discussions on slack, we can switch the default to spawn.

Let's use multiprocess instead of multiprocessing and maybe add a check to apply this only on Macos ?
Make DatasetDict.column_names return a list instead of dict

Currently, `DatasetDict.column_names` returns a dict, with each split name as keys and the corresponding list of column names as values.

However, by construction, all splits have the same column names.

I think it makes more sense to return a single list with the column names, which is the same for all the split keys.

Labels: name: enhancement

Hi @albertvillanova, can I work on this issue?
OSError: libcudart.so.11.0: cannot open shared object file: No such file or directory

> Hi  is there any solution for that eror i try to install this one 
pip install torch==1.12.1+cpu torchaudio==0.12.1+cpu -f https://download.pytorch.org/whl/torch_stable.html  
this is working fine but tell me how to install pytorch version that is fit for gpu 


Is this related to the "datasets" library? @SaleemMalikAI
`Dataset.to_sql` doesn't utilize `num_proc`

The underlying `SqlDatasetWriter` has `num_proc` as an available argument [here](https://github.com/huggingface/datasets/blob/5dc1a179783dff868b0547c8486268cfaea1ea1f/src/datasets/io/sql.py#L63) , but `Dataset.to_sql()` does not accept it, therefore it is always using one process for the SQL conversion.
BIGPATENT dataset inaccessible (deprecated script loader)

dataset: https://huggingface.co/datasets/NortheasternUniversity/big_patent

When I try to load it with the datasets library, it fails with:

RuntimeError: Dataset scripts are no longer supported, but found big_patent.py


Could you please publish a Parquet/Arrow export of BIGPATENT on the Hugging Face so that it can be accessed with datasets>=4.x.




Hi ! I opened https://huggingface.co/datasets/NortheasternUniversity/big_patent/discussions/7 to update the dataset, hopefully it's merged soon !

The dataset now works with `datasets` v4 ! closing this issue
Error processing scalar columns using tensorflow.

`datasets==4.0.0`
```
columns_to_return = ['input_ids','attention_mask', 'start_positions', 'end_positions']
train_ds.set_format(type='tf', columns=columns_to_return)
```
`train_ds`:
```
train_ds type: <class 'datasets.arrow_dataset.Dataset'>, shape: (1000, 9)
columns: ['question', 'sentences', 'answer', 'str_idx', 'end_idx', 'input_ids', 'attention_mask', 'start_positions', 'end_positions']
features:{'question': Value('string'), 'sentences': Value('string'), 'answer': Value('string'), 'str_idx': Value('int64'), 'end_idx': Value('int64'), 'input_ids': List(Value('int32')), 'attention_mask': List(Value('int8')), 'start_positions': Value('int64'), 'end_positions': Value('int64')}
```
`train_ds_tensor = train_ds['start_positions'].to_tensor(shape=(-1,1))` hits the following error:
```
AttributeError: 'Column' object has no attribute 'to_tensor'
```
`tf.reshape(train_ds['start_positions'], shape=[-1,1])` hits the following error:
```
TypeError: Scalar tensor has no `len()`
```


Using tf.convert_to_tensor works fine:

```
import tensorflow as tf

start_pos = tf.convert_to_tensor(train_ds['start_positions'], dtype=tf.int64)
start_pos = tf.reshape(start_pos, [-1, 1])
```


Alternatively, using the built-in to_tf_dataset also avoids the issue:

```
train_tf = train_ds.to_tf_dataset(
    columns=['input_ids','attention_mask'],
    label_cols=['start_positions','end_positions'],
    shuffle=True,
    batch_size=32
)
```

```
    start_pos = tf.convert_to_tensor(self._train_ds['start_positions'], dtype=tf.int64)
  File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.13/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.13/site-packages/tensorflow/python/framework/constant_op.py", line 108, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: TypeError: Scalar tensor has no `len()`
Traceback (most recent call last):

  File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.13/site-packages/tensorflow/python/framework/ops.py", line 361, in __len__
    raise TypeError("Scalar tensor has no `len()`")

TypeError: Scalar tensor has no `len()`
```

`to_tf_dataset` works perfectly.
[docs] Fix broken WebDataset link on “Create a video dataset” page

### What
Fix the "WebDataset documentation" link on the Create a video dataset page to point
to the correct section on the video load guide.

### Why
The link currently points to an external repo, but the Hugging Face docs
have an internal "WebDataset" section under video_load.

### How
- docs/source/video_dataset.mdx: updated link to
  `https://huggingface.co/docs/datasets/main/en/video_load#webdataset`

### Issue
Fixes #7699
Support huggingface_hub v0.x and v1.x

Related to https://github.com/huggingface/huggingface_hub/issues/3340.

This PR adapts `datasets` to be compatible with both huggingface_hub v0.x and v1.x. 

In practice nothing else should change (I've checked the codebase). The `HfHubHTTPError` is a base error defined in `huggingface_hub` that inherits from `requests.HTTPError` in v0.x and will inherit from `httpx.HTTPError` in v1.x. It has been introduced ~2 years ago so it's fine to use it right now (i.e. no need to wait for v1.x release or bump minimal version).

Most of the changes have been around the test suite to make sure that tests are passing with both `requests` and `httpx` backends. Mid-term it would be good to completely remove the `requests` dependency from `datasets` but that's an orthogonal topic. 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7783). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq could you have a look at this PR please? It makes `datasets` compatible with the upcoming `huggingface_hub` v1.0 release while staying compatible with 0.x. Look at PR description for more details.

The CI is currently failing because of 429 rate limit errors but otherwise everything should be fine (made extensive local and ci tests to ensure that). Let me know if you notice anything weird. PR is ready to be merged "as-is" in my opinion.
Audio dataset is not decoding on 4.1.1

### Describe the bug

The audio column remain as non-decoded objects even when accessing them.

```python
dataset = load_dataset("MrDragonFox/Elise", split = "train")
dataset[0] # see that it doesn't show 'array' etc...
```

Works fine with `datasets==3.6.0`

Followed the docs in 

- https://huggingface.co/docs/datasets/en/audio_load

### Steps to reproduce the bug

```python
dataset = load_dataset("MrDragonFox/Elise", split = "train")
dataset[0] # see that it doesn't show 'array' etc...
```


### Expected behavior

It should decode when accessing the elemenet

### Environment info

4.1.1
ubuntu 22.04


Related

- https://github.com/huggingface/datasets/issues/7707


Previously (datasets<=3.6.0), audio columns were decoded automatically when accessing a row. Now, for performance reasons, audio decoding is lazy by default: you just see the file path unless you explicitly cast the column to Audio.

Here’s the fix (following the current [datasets audio docs](https://huggingface.co/docs/datasets/en/audio_load)
):

```
from datasets import load_dataset, Audio

dataset = load_dataset("MrDragonFox/Elise", split="train")

# Explicitly decode the audio column
dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))

print(dataset[0]["audio"])
# {'path': '...', 'array': array([...], dtype=float32), 'sampling_rate': 16000}
```

@haitam03-yo's comment is right that the data is not decoded by default anymore indeed, but here is how it works in practice now:

From `datasets` v4, audio data are read as [AudioDecoder](https://meta-pytorch.org/torchcodec/0.4/generated/torchcodec.decoders.AudioDecoder.html) objects from torchcodec. This doesn't decode the data by default, but you can call `audio.get_all_samples()` to decode the audio.

See the documentation on how to process audio data here: https://huggingface.co/docs/datasets/audio_process

To resolve this, you need to explicitly cast the audio column to the Audio feature. This will decode the audio data and make it accessible as an array. Here is the corrected code snippet


from datasets import load_dataset, Audio

# Load your dataset
dataset = load_dataset("MrDragonFox/Elise", split="train")

# Explicitly cast the 'audio' column to the Audio feature
dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))

# Now you can access the decoded audio array
print(dataset[0]["audio"])

By adding the cast_column step, you are telling the datasets library to decode the audio data with the specified sampling rate, and you will then be able to access the audio array as you were used to in previous versions.
NonMatchingSplitsSizesError and ExpectedMoreSplitsError

### Describe the bug

When loading dataset, the info specified by `data_files` did not overwrite the original info.


### Steps to reproduce the bug

```python
from datasets import load_dataset

traindata = load_dataset(
            "allenai/c4",
            "en",
            data_files={"train": "en/c4-train.00000-of-01024.json.gz", 
                        "validation": "en/c4-validation.00000-of-00008.json.gz"},
        )
```


```log
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=828589180707, num_examples=364868892, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='train', num_bytes=809262831, num_examples=356317, shard_lengths=[223006, 133311], dataset_name='c4')}, {'expected': SplitInfo(name='validation', num_bytes=825767266, num_examples=364608, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='validation', num_bytes=102199431, num_examples=45576, shard_lengths=None, dataset_name='c4')}]
```

```python
from datasets import load_dataset

traindata = load_dataset(
            "allenai/c4",
            "en",
            data_files={"train": "en/c4-train.00000-of-01024.json.gz"},
            split="train"
        )
```

```log
ExpectedMoreSplitsError: {'validation'}
```

### Expected behavior

No error

### Environment info

datasets 4.0.0


To load just one shard without errors, you should use data_files directly with split set to "train", but don’t specify "allenai/c4", since that points to the full dataset with all shards.

Instead, do this:
```
from datasets import load_dataset
from datasets import load_dataset

# Load only one shard of C4
traindata = load_dataset(
    "json",   # <-- use "json" since you’re directly passing JSON files
    data_files={"train": "https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-train.00000-of-01024.json.gz"},
    split="train"
)

print(traindata)
```
If you want both train and validation but only a subset of shards, do:
```
traindata = load_dataset(
    "json",
    data_files={
        "train": "https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-train.00000-of-01024.json.gz",
        "validation": "https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-validation.00000-of-00008.json.gz"
    }
)

print(traindata)
```

I just want to load a few files from allenai/c4.
If I do not specify allenai/c4, where will the files be loaded from?

My apologies, I’ve modified my previous answer.
You just need to specify the full path, for example:

https://huggingface.co/datasets/allenai/c4/resolve/main/en/c4-train.00000-of-01024.json.gz

<img width="1843" height="633" alt="Image" src="https://github.com/user-attachments/assets/b2922958-9d87-4b62-a00e-c5ca02e31c27" />

I hope this updated answer is helpful.
Less api calls when resolving data_files

There ~10 unnecessary `/api/datasets/user/dataset/revision`calls due to multithreading in data files resolution.

I disabled multithreading, which was actually not useful anymore since `HfFileSystem` has been using caching for a while now.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7805). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Parquet: add `on_bad_file` argument to error/warn/skip bad files

```python
from datasets import load_dataset

on_bad_file = "error"  # default
# on_bad_file = "warn"  # warn and skip bad file
# on_bad_file = "skip"  # skip bad file
ds = load_dataset(parquet_dataset_id, on_bad_file=on_bad_file)
```


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7806). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Datasets: Add WMT21 & WMT22 loaders (basic TSV loaders, sample data, tests)

- Implemented TSV-based dataset loaders:
   - WMT21Dataset (local_datasets/wmt21/wmt21_dataset.py)
   - WMT22Dataset (local_datasets/wmt22/wmt22_dataset.py)
   These classes load source-target pairs from .tsv files for train, validation, and test splits.
- Created sample dummy data for both datasets:
   - dummy_data/train.tsv, dummy_data/validation.tsv, dummy_data/test.tsv
   - Includes a few realistic example lines to allow CI and local tests to pass without downloading full datasets.
- Added automated tests for robust validation:
   - tests/test_wmt21.py and tests/test_wmt22.py
   - Checks that all splits load correctly, empty lines are ignored, and the number of examples matches the number of lines in the .tsv files.
   - Edge cases handled: empty lines, malformed lines, extra tabs.
- Added README.md files for both datasets:
   - Provides dataset structure, usage instructions, and placeholders for citation & license information.
   - Ensures that other developers and reviewers can understand dataset usage immediately.
- Ensured easy local testing:
   - Load datasets programmatically using WMT21Dataset / WMT22Dataset.
   - Verified train/validation/test splits are correctly returned as Python dictionaries of Dataset objects.


-Provides initial support for WMT21 and WMT22 NLP/translation experiments.
-Allows contributors and reviewers to test dataset loading locally or in CI without downloading large datasets.
-Serves as a template to extend to other WMT datasets in the future.

Testing instructions:
```
# Activate your environment
pytest tests/test_wmt21.py -v
pytest tests/test_wmt22.py -v
```
```
from local_datasets.wmt21.wmt21_dataset import WMT21Dataset
from local_datasets.wmt22.wmt22_dataset import WMT22Dataset

# WMT21
wmt21 = WMT21Dataset("local_datasets/wmt21/dummy_data")
ds21 = wmt21.load()
print(ds21["train"][0])
print(ds21["validation"][0])
print(ds21["test"][0])

# WMT22
wmt22 = WMT22Dataset("local_datasets/wmt22/dummy_data")
ds22 = wmt22.load()
print(ds22["train"][0])
print(ds22["validation"][0])
print(ds22["test"][0])
```


closing since datasets should be added on https://huggingface.co directly
Add pyarrow's binary view to features

Basically https://github.com/huggingface/datasets/pull/7718 just for binary view instead of string view


@lhoestq 🙏 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7795). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
polars dataset cannot cast column to Image/Audio/Video

### Describe the bug

`from_polars` dataset cannot cast column to Image/Audio/Video, while it works on `from_pandas` and `from_dict`

### Steps to reproduce the bug

```python3
import datasets
import pandas as pd
import polars as pl

image_path = "./sample.png"

# polars
df = pl.DataFrame({"image_path": [image_path]})
dataset = datasets.Dataset.from_polars(df)
dataset = dataset.cast_column("image_path", datasets.Image())

# # raises Error
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from large_string to struct using function cast_struct


# pandas 
df = pd.DataFrame({"image_path": [image_path]})
dataset = datasets.Dataset.from_pandas(df)
dataset = dataset.cast_column("image_path", datasets.Image())

# # pass
{'image_path': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=338x277 at 0x7FBA719D4050>}


# dict
dataset = datasets.Dataset.from_dict({"image_path": [image_path]})
dataset = dataset.cast_column("image_path", datasets.Image())

# # pass
{'image_path': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=338x277 at 0x7FBA719D4050>}
```


### Expected behavior

`from_polars` case shouldn't raise error and have the same outputs as `from_pandas` and `from_dict`

### Environment info

```
# Name                    Version                   Build  Channel
datasets                  4.0.0                    pypi_0    pypi
pandas                    2.3.1                    pypi_0    pypi
polars                    1.32.3                   pypi_0    pypi
```


I fixed this with a combination of `to_dict` and `from_dict`:

```py
datasets.Dataset.from_dict(df.to_dict(as_series=False))
```

@samuelstevens Yeah, I'm using similar workaround as well. But it would be ideal if we can avoid the copy.
Allow streaming hdf5 files

Add streaming support after https://github.com/huggingface/datasets/pull/7690, cc @klamike :)

## Details

in `datasets` loaders, `open()` is extended to work with files that are on disk but also on HF. Files on HF are streamed using HTTP range requests using the `HfFileSystem` implementation in the `huggingface_hub` library.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7814). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
disable_progress_bar() not working as expected

### Describe the bug

Hi,

I'm trying to load a dataset on Kaggle TPU image. There is some known compat issue with progress bar on Kaggle, so I'm trying to disable the progress bar globally. This does not work as you can see in [here](https://www.kaggle.com/code/windmaple/hf-datasets-issue).

In contract, disabling progress bar for snapshot_download() works as expected as in [here](https://www.kaggle.com/code/windmaple/snapshot-download-error).

### Steps to reproduce the bug

See this [notebook](https://www.kaggle.com/code/windmaple/hf-datasets-issue). 

There is sth. wrong with `shell_paraent`.

### Expected behavior

The downloader should disable progress bar and move forward w/ no error.

### Environment info

The latest version as I did:

!pip install -U datasets ipywidgets ipykernel


@xianbaoqian 

Closing this one since it's a Xet issue.