Minimartzz/github-issues huggingface dataset:
`datasets.map(..., num_proc=4)` multi-processing fails
State: open

The following code fails in python 3.11+ 

```python
tokenized_datasets = datasets.map(tokenize_function, batched=True, num_proc=4, remove_columns=["text"])  
```
Error log:
```bash
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/multiprocess/process.py", line 315, in _bootstrap
self.run()
File "/usr/local/lib/python3.12/dist-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.12/dist-packages/multiprocess/pool.py", line 114, in worker
task = get()
^^^^^
File "/usr/local/lib/python3.12/dist-packages/multiprocess/queues.py", line 371, in get
return _ForkingPickler.loads(res)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/dill/_dill.py", line 327, in loads
return load(file, ignore, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/dill/_dill.py", line 313, in load
return Unpickler(file, ignore=ignore, **kwds).load()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/dill/_dill.py", line 525, in load
obj = StockUnpickler.load(self)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/dill/_dill.py", line 659, in _create_code
if len(args) == 16: return CodeType(*args)
^^^^^^^^^^^^^^^
TypeError: code() argument 13 must be str, not int
```

After upgrading dill to the latest 0.4.0 with "pip install --upgrade dill", it can pass. So it seems that there is a compatibility issue between dill 0.3.4 and python 3.11+, because python 3.10 works fine. 


Is the dill deterministic issue mentioned in https://github.com/huggingface/datasets/blob/main/setup.py#L117) still valid? Any plan to unpin? 


Comments:
related: https://github.com/huggingface/datasets/issues/7510

we need to do more tests to see if latest `dill` is deterministic
TensorFlow RaggedTensor Support (batch-level)
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

Hi,

Currently datasets does not support RaggedTensor output on batch-level.
When building a Object Detection Dataset (with TensorFlow) I need to enable RaggedTensors as that's how BBoxes & classes are expected from the Keras Model POV.

Currently there's a error thrown saying that "Nested Data is not supported".

It'd be very helpful if this was fixed! :)

### Motivation

Enabling Object Detection pipelines for TensorFlow.

### Your contribution

With guidance I'd happily help making the PR.

The current implementation with DataCollator and later enforcing `np.array` is the problematic part (at the end of `np_get_batch` in `tf_utils.py`). As `numpy` don't support "Raggednes"

Comments:
Keras doesn't support other inputs other than tf.data.Dataset objects ? it's a bit painful to have to support and maintain this kind of integration

Is there a way to use a `datasets.Dataset` with outputs formatted as tensors / ragged tensors instead ? like in https://huggingface.co/docs/datasets/use_with_tensorflow#dataset-format

I'll give it a try when I get the time. But quite sure I already tested the `with_format` approach.

Keras when using TF as backend converts the datasets into `tf.data.Dataset`, much like you do.

Hi @Lundez! Thanks for raising this — very valid point, especially for Object Detection use-cases.

You're right that np_get_batch currently enforces numpy batching, which breaks RaggedTensor support due to its inability to handle nested structures. This likely needs a redesign to allow TensorFlow-native batching in specific formats.

Before diving into a code change though, could you confirm:

Does `.with_format("tensorflow")` (without batching) return a `tf.data.Dataset` that works if batching is deferred to `model.fit()`?

Have you tried something like:

```python
tf_dataset = dataset.with_format("tensorflow").to_tf_dataset(
    columns=["image", "labels"],
    label_cols=None,
    batch_size=None  # No batching here
)
model.fit(tf_dataset.batch(BATCH_SIZE))  # Use RaggedTensor batching here
```

If this works, it might be worth updating the documentation rather than changing batching logic inside datasets itself.

That said, happy to explore changes if batching needs to be supported natively for RaggedTensor. Just flagging that it’d require some careful design due to existing numpy assumptions.

Hi, we've had to move on for now. 

We have actually also moved to dense tensors to make it possible to xla complie the training. 

But I'll check when I'm back from vacation which is far into the future. 

Thanks
Add custom fingerprint support to `from_generator`
State: closed

This PR adds `dataset_id_suffix` parameter to 'Dataset.from_generator' function.

`Dataset.from_generator` function passes all of its arguments to `BuilderConfig.create_config_id`, including generator function itself. `BuilderConfig.create_config_id` function tries to hash all the args, which can take a large amount of time or even cause MemoryError if the dataset processed in a generator function is large enough.

This PR allows user to pass a custom fingerprint (`dataset_id_suffix`) to be used as a suffix in a dataset name instead of the one generated by hashing the args.

This PR is a possible solution of #7513 

Comments:
This is great !

What do you think of passing `config_id=` directly to the builder instead of just the suffix ? This would be a power user argument though, or for internal use. And in from_generator the new argument can be `fingerprint=` as in `Dataset.__init__()`

The `config_id` can be defined using something like `config_id = "default-fingerprint=" + fingerprint`

I feel ike this could make the Dataset API more coherent if we avoid introducing a new argument while we can juste use `fingerprint=`

@lhoestq could you please re-review the changes I made?

@lhoestq ping
I also added a simple test for the `fingerprint` parameter

@lhoestq could you please review the PR? I implemented the requested changes and added a test for the added `fingerprint` arg

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7533). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Document the HF_DATASETS_CACHE environment variable in the datasets cache documentation
State: closed


This pull request updates the Datasets documentation to include the `HF_DATASETS_CACHE` environment variable. While the current documentation only mentions `HF_HOME` for overriding the default cache directory, `HF_DATASETS_CACHE` is also a supported and useful option for specifying a custom cache location for datasets stored in Arrow format.

This addition is based on the discussion in (https://github.com/huggingface/datasets/issues/7457), where users noted the absence of this variable in the documentation despite its functionality. The update adds a new section to `cache.mdx` that explains how to use `HF_DATASETS_CACHE` with an example.

This change aims to improve clarity and help users better manage their cache directories when working in shared environments or with limited local storage.

Closes #7457.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7532). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Your clarification in your comment at https://github.com/huggingface/datasets/issues/7480#issuecomment-2833640084 sounds great, would you like to update this PR to include it ?

Hi @lhoestq, I’ve updated the documentation to reflect the clarifications discussed in #7480. Let me know if anything else is needed!
How to solve "Spaces stuck in Building" problems
State: closed

### Describe the bug

Public spaces may stuck in Building after restarting, error log as follows:

build error
Unexpected job error

ERROR: failed to push spaces-registry.huggingface.tech/spaces/*:cpu-*-*: unexpected status from HEAD request to https://spaces-registry.huggingface.tech/v2/spaces/*/manifests/cpu-*-*: 401 Unauthorized

### Steps to reproduce the bug

Restart space / Factory rebuild cannot avoid it

### Expected behavior

Fix this problem

### Environment info

no requirements.txt can still happen
python gradio spaces

Comments:
I'm facing the same issue—Space stuck in "Building" even after restart and Factory rebuild. Any fix?


> I'm facing the same issue—Space stuck in "Building" even after restart and Factory rebuild. Any fix?

Also see https://github.com/huggingface/huggingface_hub/issues/3019

I'm facing the same issue. The build fails with the same error, and restarting won't help. Is there a fix or ETA? 

Same error happens today, Nov 10th.
audio folder builder cannot detect custom split name
State: open

### Describe the bug

when using audio folder builder (`load_dataset("audiofolder", data_dir="/path/to/folder")`), it cannot detect custom split name other than train/validation/test

### Steps to reproduce the bug

i have the following folder structure
```
my_dataset/
├── train/
│   ├── lorem.wav
│   ├── …
│   └── metadata.csv
├── test/
│   ├── ipsum.wav
│   ├── …
│   └── metadata.csv
├── validation/
│   ├── dolor.wav
│   ├── …
│   └── metadata.csv
└── custom/
    ├── sit.wav
    ├── …
    └── metadata.csv
```
using `ds = load_dataset("audiofolder", data_dir="/path/to/my_dataset")`

### Expected behavior

i got `ds` with only 3 splits train/validation/test, whenever i rename train/validation/test folder it also disappear if i re-create `ds`

### Environment info

- `datasets` version: 3.5.0
- Platform: Windows-11-10.0.26100-SP0
- Python version: 3.12.8
- `huggingface_hub` version: 0.30.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments:
Data Studio Error: Convert JSONL incorrectly
State: open

### Describe the bug

Hi there,

I uploaded a dataset here https://huggingface.co/datasets/V-STaR-Bench/V-STaR, but I found that Data Studio incorrectly convert the "bboxes" value for the whole dataset. Therefore, anyone who downloaded the dataset via the API would get the wrong "bboxes" value in the data file.

Could you help me address the issue?

Many thanks,



### Steps to reproduce the bug

The JSONL file of [V_STaR_test_release.jsonl](https://huggingface.co/datasets/V-STaR-Bench/V-STaR/blob/main/V_STaR_test_release.jsonl) has the correct values of every "bboxes" for each sample.

But in the Data Studio, we can see that the values of "bboxes" have changed, and load the dataset via API will also get the wrong values.

### Expected behavior

Fix the bug to correctly download my dataset.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.14.0-427.22.1.el9_4.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.16
- `huggingface_hub` version: 0.29.3
- PyArrow version: 19.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2023.10.0

Comments:
Hi ! Your JSONL file is incompatible with Arrow / Parquet. Indeed in Arrow / Parquet every dict should have the same keys, while in your dataset the bboxes have varying keys.

This causes the Data Studio to treat the bboxes as if each row was missing the keys from other rows.

Feel free to take a look at the docs on object segmentation to see how to format a dataset with bboxes: https://huggingface.co/docs/datasets/object_detection
Auto-merge option for `convert-to-parquet`
State: closed
Labels: description: New feature or request
name: enhancement

### Feature request

Add a command-line option, e.g. `--auto-merge-pull-request` that enables automatic merging of the commits created by the `convert-to-parquet` tool.

### Motivation

Large datasets may result in dozens of PRs due to the splitting mechanism. Each of these has to be manually accepted via the website.

### Your contribution

Happy to look into submitting a PR if this is of interest to maintainers.

Comments:
Alternatively, there could be an option to switch from submitting PRs to just committing changes directly to `main`.

Why not, I'd be in favor of `--merge-pull-request` to call `HfApi().merge_pull_request()` at the end of the conversion :) feel free to open a PR if you'd like

#self-assign

Closing since convert to parquet has been removed... https://github.com/huggingface/datasets/pull/7592#issuecomment-3073053138
Faster downloads/uploads with Xet storage
State: open

![Image](https://github.com/user-attachments/assets/6e247f4a-d436-4428-a682-fe18ebdc73a9)

## Xet is out !

Over the past few weeks, Hugging Face’s [Xet Team](https://huggingface.co/xet-team) took a major step forward by [migrating the first Model and Dataset repositories off LFS and to Xet storage](https://huggingface.co/posts/jsulz/911431940353906).

See more information on the HF blog: https://huggingface.co/blog/xet-on-the-hub

You can already enable Xet on Hugging Face account to benefit from faster downloads and uploads :)

We finalized an official integration with the `huggingface_hub` library that means you get the benefits of Xet without any significant changes to your current workflow.


## Previous versions of `datasets`

For older versions of `datasets` you might see this warning in `push_to_hub()`:

```
Uploading files as bytes or binary IO objects is not supported by Xet Storage.
```

This means the `huggingface_hub` + Xet integration isn't enabled for your version of `datasets`.

You can fix this by updating to `datasets>=3.6.0` and `huggingface_hub>=0.31.0`

```
pip install -U datasets huggingface_hub
```

## The future

Stay tuned for more Xet optimizations, especially on [Xet-optimized Parquet](https://huggingface.co/blog/improve_parquet_dedupe)



Comments:
Fix indexing in split commit messages
State: closed

When a large commit is split up, it seems the commit index in the message is zero-based while the total number is one-based. I came across this running `convert-to-parquet` and was wondering why there was no `6-of-6` commit. This PR fixes that by adding one to the commit index, so both are one-based.

Current behavior:
<img width="463" alt="Screenshot 2025-04-17 at 1 00 17 PM" src="https://github.com/user-attachments/assets/7f3d389e-cb92-405d-a3c2-f2b1cdf0cb79" />

Comments:
Hi ! this is expected and is coherent with other naming conventions in `datasets` such as parquet shards naming
fix: Image Feature in Datasets Library Fails to Handle bytearray Objects from Spark DataFrames (#7517)
State: closed

## Task
Support bytes-like objects (bytes and bytearray) in Features classes

### Description
The `Features` classes only accept `bytes` objects for binary data, but not `bytearray`. This leads to errors when using `IterableDataset.from_spark()` with Spark DataFrames as they contain `bytearray` objects, even though both `bytes` and `bytearray` are valid [*bytes-like objects* in Python](https://docs.python.org/3/glossary.html#term-bytes-like-object).

### Changes
- Updated `Features` classes to accept both `bytes` and `bytearray` types for binary data fields.

### Reasoning
- `bytes` and `bytearray` serve the same purpose for binary data, with the only difference being mutability.
- Modifying the Spark iterator to convert `bytearray` to `bytes` would be a workaround, not a true fix. I think the correct solution is to accept all bytes-like objects as input.
- This approach is more robust and future-proof since Python 3.12+ provides a [standard way to check for buffer protocol](https://docs.python.org/3/c-api/buffer.html#bufferobjects).

### Testing
- Added tests to cover `bytearray` inputs for image features.

### Related Issues
- Fixes: #7517 

Comments:
@lhoestq let me know if you prefer to change the spark iterator so it outputs `bytes`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7521). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update items in the dataset without `map`
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

I would like to be able to update items in my dataset without affecting all rows. At least if there was a range option, I would be able to process those items, save the dataset, and then continue.

If I am supposed to split the dataset first, that is not clear, since the docs suggest that any of those functions returns a new object, so I don't think I can do that.

### Motivation

I am applying an extremely time-consuming function to each item in my `Dataset`. Unfortunately, datasets only supports updating values via `map`, so if my computer dies in the middle of this long-running process, I lose all progress. This is far from ideal. I would like to use `datasets` throughout this processing, but this limitation is now forcing me to write my own dataset format just to do this intermediary operation.

It would be less intuitive but I suppose I could split and then concatenate the dataset before saving? But this feels very inefficient.

### Your contribution

I can test the feature.

Comments:
Hello!

Have you looked at `Dataset.shard`? [Docs](https://huggingface.co/docs/datasets/en/process#shard)

Using this method you could break your dataset in N shards. Apply `map` on each shard and concatenate them back.
num_proc parallelization works only for first ~10s.
State: open

### Describe the bug

When I try to load an already downloaded dataset with num_proc=64, the speed is very high for the first 10-20 seconds acheiving 30-40K samples / s, and 100% utilization for all cores but it soon drops to <= 1000 with almost 0% utilization for most cores. 

### Steps to reproduce the bug

```
// download dataset with cli
!huggingface-cli download --repo-type dataset timm/imagenet-1k-wds --max-workers 32

from datasets import load_dataset
ds = load_dataset("timm/imagenet-1k-wds", num_proc=64)
```


### Expected behavior

100% core utilization throughout. 

### Environment info

Azure A100-80GB, 16 cores VM

![Image](https://github.com/user-attachments/assets/69d00fe3-d720-4474-9439-21e046d85034)

Comments:
Hi, can you check if the processes are still alive ? It's a bit weird because `datasets` does check if processes crash and return an error in that case

Thank you for reverting quickly. I digged a bit, and realized my disk's IOPS is also limited - which is causing this. will check further and report if it's an issue of hf datasets' side or mine.
Image Feature in Datasets Library Fails to Handle bytearray Objects from Spark DataFrames
State: closed

### Describe the bug

When using `IterableDataset.from_spark()` with a Spark DataFrame containing image data, the `Image` feature class fails to properly process this data type, causing an `AttributeError: 'bytearray' object has no attribute 'get'`

### Steps to reproduce the bug

1. Create a Spark DataFrame with a column containing image data as bytearray objects
2. Define a Feature schema with an Image feature
3. Create an IterableDataset using `IterableDataset.from_spark()`
4. Attempt to iterate through the dataset

```
from pyspark.sql import SparkSession
from datasets import Dataset, IterableDataset, Features, Image, Value

# initialize spark
spark = SparkSession.builder.appName("MinimalRepro").getOrCreate()

# create spark dataframe
data = [(0, open("image.png", "rb").read())]
df = spark.createDataFrame(data, "idx: int, image: binary")

# convert to dataset
features = Features({"idx": Value("int64"), "image": Image()})
ds = Dataset.from_spark(df, features=features)
ds_iter = IterableDataset.from_spark(df, features=features)

# iterate
print(next(iter(ds)))
print(next(iter(ds_iter)))
```

### Expected behavior

The features should work on `IterableDataset` the same way they work on `Dataset`

### Environment info

- `datasets` version: 3.5.0
- Platform: macOS-15.3.2-arm64-arm-64bit
- Python version: 3.12.7
- `huggingface_hub` version: 0.30.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0


Comments:
Hi ! The `Image()` type accepts either
- a `bytes` object containing the image bytes
- a `str` object containing the image path
- a `PIL.Image` object

but it doesn't support `bytearray`, maybe you can convert to `bytes` beforehand ?

Hi @lhoestq, 
converting to bytes is certainly possible and would work around the error. However, the core issue is that `Dataset` and `IterableDataset` behave differently with the features.

I’d be happy to work on a fix for this issue.

I see, that's an issue indeed. Feel free to ping me if I can help with reviews or any guidance

If it can help, the code that takes a Spark DataFrame and iterates on the rows for `IterableDataset` is here: 

https://github.com/huggingface/datasets/blob/6a96bf313085d7538a999b929a550e14e1d406c9/src/datasets/packaged_modules/spark/spark.py#L49-L53

#self-assign
unsloth/DeepSeek-R1-Distill-Qwen-32B server error
State: closed

### Describe the bug

hfhubhttperror: 500 server error: internal server error for url: https://huggingface.co/api/models/unsloth/deepseek-r1-distill-qwen-32b-bnb-4bit/commits/main (request id: root=1-67fe23fa-3a2150eb444c2a823c388579;de3aed68-c397-4da5-94d4-6565efd3b919) internal error - we're working hard to fix this as soon as possible!

### Steps to reproduce the bug

unsloth/DeepSeek-R1-Distill-Qwen-32B server error

### Expected behavior

Network repair

### Environment info

The web side is also unavailable

Comments:
`concatenate_datasets` does not preserve Pytorch format for IterableDataset
State: closed

### Describe the bug

When concatenating datasets with `concatenate_datasets`, I would expect the resulting combined dataset to be in the same format as the inputs (assuming it's consistent). This is indeed the behavior when combining `Dataset`, but not when combining `IterableDataset`. Specifically, when applying `concatenate_datasets` to a list of `IterableDataset` in Pytorch format (i.e. using `.with_format(Pytorch)`), the output `IterableDataset` is not in Pytorch format.

### Steps to reproduce the bug

```
import datasets
ds = datasets.Dataset.from_dict({"a": [1,2,3]})
iterable_ds = ds.to_iterable_dataset()
datasets.concatenate_datasets([ds.with_format("torch")]) # <- this preserves Pytorch format
datasets.concatenate_datasets([iterable_ds.with_format("torch")]) # <- this does NOT preserves Pytorch format
```

### Expected behavior

Pytorch format should be preserved when combining IterableDataset in Pytorch format.

### Environment info

datasets==3.5.0, Python 3.11.11, torch==2.2.2

Comments:
Hi ! Oh indeed it would be cool to return the same format in that case. Would you like to submit a PR ? The function that does the concatenation is here:

https://github.com/huggingface/datasets/blob/90e5bf8a8599b625d6103ee5ac83b98269991141/src/datasets/iterable_dataset.py#L3375-L3380

Thank you for the pointer, @lhoestq ! See #7522
Do not hash `generator` in `BuilderConfig.create_config_id`
State: closed

`Dataset.from_generator` function passes all of its arguments to `BuilderConfig.create_config_id`, including generator function itself. `BuilderConfig.create_config_id` function tries to hash all the args, and hashing a `generator` can take a large amount of time or even cause MemoryError if the dataset processed in a generator function is large enough.

Maybe we should pop generator from `config_kwargs_to_add_to_suffix` before hashing to avoid it.

There is a more detailed description of the problem this PR solves in #7513 

Comments: