josteece/github-issues huggingface dataset:
Support for sparse arrays with the Arrow Sparse Tensor format?
Opened by JulesGM (open)

### Feature request

AI in biology is becoming a big thing. One thing that would be a huge benefit to the field that Huggingface Datasets doesn't currently have is native support for **sparse arrays**. 


Arrow has support for sparse tensors. 
https://arrow.apache.org/docs/format/Other.html#sparse-tensor


It would be a big deal if Hugging Face Datasets supported sparse tensors as a feature type, natively. 


### Motivation

This is important for example in the field of transcriptomics (modeling and understanding gene expression), because a large fraction of the genes are not expressed (zero). More generally, in science, sparse arrays are very common, so adding support for them would be very benefitial, it would make just using Hugging Face Dataset objects a lot more straightforward and clean.


### Your contribution

We can discuss this further once the team comments of what they think about the feature, and if there were previous attempts at making it work, and understanding their evaluation of how hard it would be. My intuition is that it should be fairly straightforward, as the Arrow backend already supports it.

Comments:
Hi ! Unfortunately the Sparse Tensor structure in Arrow is not part of the Arrow format (yes it's confusing...), so it's not possible to use it in `datasets`. It's a separate structure that doesn't correspond to any type or extension type in Arrow.

The Arrow community recently added an extension type for fixed shape tensors at https://arrow.apache.org/docs/format/CanonicalExtensions.html#fixed-shape-tensor, it should be possible to contribute an extension type for sparse tensors as well.
vllm批量推理报错
Opened by YuShengzuishuai (open)

### Describe the bug

![Image](https://github.com/user-attachments/assets/3d958e43-28dc-4467-9333-5990c7af3b3f)

### Steps to reproduce the bug

![Image](https://github.com/user-attachments/assets/3067eeca-a54d-4956-b0fd-3fc5ea93dabb)

### Expected behavior

![Image](https://github.com/user-attachments/assets/77d32936-488f-4572-9365-bfb4170e555b)

### Environment info

![Image](https://github.com/user-attachments/assets/230335c4-825f-4db1-b07d-4776ef63ead8)

Comments:
Make sure you have installed a recent version of `soundfile`
Excessive RAM Usage After Dataset Concatenation concatenate_datasets
Opened by sam-hey (open)

### Describe the bug

When loading a dataset from disk, concatenating it, and starting the training process, the RAM usage progressively increases until the kernel terminates the process due to excessive memory consumption.

https://github.com/huggingface/datasets/issues/2276

### Steps to reproduce the bug

```python
from datasets import  DatasetDict, concatenate_datasets

dataset = DatasetDict.load_from_disk("data")

...
...

combined_dataset = concatenate_datasets(
        [dataset[split] for split in dataset]
    )

#start SentenceTransformer training
```

### Expected behavior

I would not expect RAM utilization to increase after concatenation. Removing the concatenation step resolves the issue

### Environment info

sentence-transformers==3.1.1
datasets==3.2.0

python3.10

Comments:
![Image](https://github.com/user-attachments/assets/b6f8bcbd-44af-413e-bc06-65380eb0f746)

![Image](https://github.com/user-attachments/assets/a241fcd8-4b62-495c-926c-685f82015dfb)

Adding a img from memray
https://gist.github.com/sam-hey/00c958f13fb0f7b54d17197fe353002f

I'm having the same issue where concatenation seems to use a huge amount of RAM.

```python
# Load all chunks and concatenate them into a final dataset.
        chunk_datasets = [
            Dataset.load_from_disk(file, keep_in_memory=False)
            for file in tqdm(chunk_files, desc="Loading chunk datasets")
        ]
        logging.info("Concatenating chunk datasets...")
        final_dataset = concatenate_datasets(chunk_datasets)
```

This is a real issue for me as the final dataset is a few terabytes in size. I'm using datasets version `3.1.0`. Also tested with version `3.4.1`

I did have a short look, the error seems to be from `memory_map` and the stream not being closed. 

https://github.com/huggingface/datasets/blob/5f8d2ad9a1b0bccfd962d998987228addfd5be9f/src/datasets/table.py#L48-L50


Did not have the time to test jet: https://github.com/sam-hey/datasets/tree/fix/concatenate_datasets

I will probably have a better look in a couple of days.
Inconsistent Behavior Between `load_dataset` and `load_from_disk` When Loading Sharded Datasets
Opened by gaohongkui (open)

### Description

I encountered an inconsistency in behavior between `load_dataset` and `load_from_disk` when loading sharded datasets. Here is a minimal example to reproduce the issue:

#### Code 1: Using `load_dataset`
```python
from datasets import Dataset, load_dataset

# First save with max_shard_size=10
Dataset.from_dict({"id": range(1000)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Second save with max_shard_size=10
Dataset.from_dict({"id": range(500)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Load the DatasetDict
loaded_datasetdict = load_dataset("my_sharded_datasetdict")
print(loaded_datasetdict)
```
**Output**:
- `train` has 1350 samples.
- `test` has 150 samples.

#### Code 2: Using `load_from_disk`
```python
from datasets import Dataset, load_from_disk

# First save with max_shard_size=10
Dataset.from_dict({"id": range(1000)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Second save with max_shard_size=10
Dataset.from_dict({"id": range(500)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Load the DatasetDict
loaded_datasetdict = load_from_disk("my_sharded_datasetdict")
print(loaded_datasetdict)
```
**Output**:
- `train` has 450 samples.
- `test` has 50 samples.

### Expected Behavior
I expected both `load_dataset` and `load_from_disk` to load the same dataset, as they are pointing to the same directory. However, the results differ significantly:
- `load_dataset` seems to merge all shards, resulting in a combined dataset.
- `load_from_disk` only loads the last saved dataset, ignoring previous shards.

### Questions
1. Is this behavior intentional? If so, could you clarify the difference between `load_dataset` and `load_from_disk` in the documentation?
2. If this is not intentional, could this be considered a bug?
3. What is the recommended way to handle cases where multiple datasets are saved to the same directory?


Thank you for your time and effort in maintaining this great library! I look forward to your feedback.

Comments:
Support faster processing using pandas or polars functions in `IterableDataset.map()`
Opened by lhoestq (closed)

Following the polars integration :)

Allow super fast processing using pandas or polars functions in `IterableDataset.map()` by adding support to pandas and polars formatting in `IterableDataset`

```python
import polars as pl
from datasets import Dataset

ds = Dataset.from_dict({"i": range(10)}).to_iterable_dataset()
ds = ds.with_format("polars")
ds = ds.map(lambda df: df.with_columns(pl.col("i").add(1).alias("i+1")), batched=True)
ds = ds.with_format(None)
print(next(iter(ds)))
# {'i': 0, 'i+1': 1}
```

It leverages arrow's zero-copy features from/to pandas and polars.

related to https://github.com/huggingface/datasets/issues/3444 #6762 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7370). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

merging this and will make some docs and communications around using polars for optimizing data processing :)
Importing dataset gives unhelpful error message when filenames in metadata.csv are not found in the directory
Opened by svencornetsdegroot (open)

### Describe the bug

While importing an audiofolder dataset, where the names of the audiofiles don't correspond to the filenames in the metadata.csv, we get an unclear error message that is not helpful for the debugging, i.e. 
```
ValueError: Instruction "train" corresponds to no data!
```

### Steps to reproduce the bug

Assume an audiofolder with audiofiles, filename1.mp3, filename2.mp3 etc and a file metadata.csv which contains the columns file_name and sentence. The file_names are formatted like filename1.mp3, filename2.mp3 etc. 

Load the audio
```
from datasets import load_dataset
load_dataset("audiofolder", data_dir='/path/to/audiofolder')
```
When the file_names in the csv are not in sync with the filenames in the audiofolder, then we get an Error message:
```
File /opt/conda/lib/python3.12/site-packages/datasets/arrow_reader.py:251, in BaseReader.read(self, name, instructions, split_infos, in_memory)
    249 if not files:
    250     msg = f'Instruction "{instructions}" corresponds to no data!'
--> 251     raise ValueError(msg)
    252 return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)

ValueError: Instruction "train" corresponds to no data!
```
load_dataset has a default value for the argument split = 'train'. 


### Expected behavior

It would be better to get an error report something like:
```
The metadata.csv file has different filenames than the files in the datadirectory. 
```
It would have saved me 4 hours of debugging. 

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.14.0-427.40.1.el9_4.x86_64-x86_64-with-glibc2.39
- Python version: 3.12.8
- `huggingface_hub` version: 0.27.0
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments:
I'd prefer even more verbose errors; like `"file123.mp3" is referenced in metadata.csv, but not found in the data directory '/path/to/audiofolder' ! (and 100+ more missing files)` Or something along those lines.
Add with_split to DatasetDict.map
Opened by jp1924 (closed)

#7356

Comments:
Can you check this out, @lhoestq?

cc @lhoestq @albertvillanova 

@lhoestq


@lhoestq


@lhoestq

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7368). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq 

@lhoestq please...

Thank you so much for reviewing this PR! Have a great weekend~
Dataset.from_dict() can't handle large dict
Opened by CSU-OSS (open)

### Describe the bug

I have 26,000,000 3-tuples. When I use Dataset.from_dict() to load, neither. py nor Jupiter notebook can run successfully. This is my code:
```
    # len(example_data) is 26,000,000, 'diff' is a text
    diff1_list = [example_data[i].texts[0] for i in range(len(example_data))]
    diff2_list = [example_data[i].texts[1] for i in range(len(example_data))]
    label_list = [example_data[i].label for i in range(len(example_data))]

    embedding_dataset = Dataset.from_dict({
        "diff1": diff1_list,
        "diff2": diff2_list,
        "label": label_list
    })
```

### Steps to reproduce the bug

1. Initialize a large 3-tuple, e.g. 26,000,000
2. Use Dataset.from_dict() to load

### Expected behavior

Dataset.from_dict() run successfully

### Environment info

sentence-transformers                   3.3.1

Comments:
A parameter is specified but not used in datasets.arrow_dataset.Dataset.from_pandas()
Opened by NourOM02 (open)

### Describe the bug

I am interested in creating train, test and eval splits from a pandas Dataframe, therefore I was looking at the possibilities I can follow. I noticed the split parameter and was hopeful to use it in order to generate the 3 at once, however, while trying to understand the code, i noticed that it has no added value (correct me if I am wrong or misunderstood the code). 


from_pandas function code :

```python
    if info is not None and features is not None and info.features != features:
            raise ValueError(
                f"Features specified in `features` and `info.features` can't be different:\n{features}\n{info.features}"
            )
        features = features if features is not None else info.features if info is not None else None
        if info is None:
            info = DatasetInfo()
        info.features = features
        table = InMemoryTable.from_pandas(
            df=df,
            preserve_index=preserve_index,
        )
        if features is not None:
            # more expensive cast than InMemoryTable.from_pandas(..., schema=features.arrow_schema)
            # needed to support the str to Audio conversion for instance
            table = table.cast(features.arrow_schema)
        return cls(table, info=info, split=split)
```

### Steps to reproduce the bug

```python
from datasets import Dataset
# Filling the split parameter with whatever causes no harm at all
data = Dataset.from_pandas(self.raw_data, split='egiojegoierjgoiejgrefiergiuorenvuirgurthgi')
```

### Expected behavior

Would be great if there is no split parameter (if it isn't working), or to add a concrete example of how it can be used.

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.15.0-127-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.27.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments:
ImportError: To support decoding images, please install 'Pillow'.
Opened by jamessdixon (open)

### Describe the bug

Following this tutorial locally using a macboko and VSCode: https://huggingface.co/docs/diffusers/en/tutorials/basic_training
This line of code: for i, image in enumerate(dataset[:4]["image"]):
throws: ImportError: To support decoding images, please install 'Pillow'.

Pillow is installed.

### Steps to reproduce the bug

Run the tutorial


### Expected behavior

Images should be rendered

### Environment info

MacBook, VSCode

Comments:
what's your `pip show Pillow` output

same issue.. my pip show Pillow output as below:

```
Name: pillow
Version: 11.1.0
Summary: Python Imaging Library (Fork)
Home-page: https://python-pillow.github.io/
Author: 
Author-email: "Jeffrey A. Clark" <aclark@aclark.net>
License: MIT-CMU
Location: [/opt/homebrew/lib/python3.10/site-packages](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/lib/python3.10/site-packages)
Requires: 
Required-by:
```

I encountered the same problem on Ubuntu system, my pip show Pillow output as below:

```
Name: pillow
Version: 10.4.0
Summary: Python Imaging Library (Fork)
Home-page: https://python-pillow.org/
Author: 
Author-email: "Jeffrey A. Clark" <[aclark@aclark.net](mailto:aclark@aclark.net)>
License: HPND
Location: /home/shunying/.local/lib/python3.8/site-packages
Requires: 
Required-by: 
```

Well, solved this by specifying the pip version to my conda virtual environment :)

I have also encountered this. It's a strange thing that's happening.

Checking the code `datasets` it uses `importlib.util.find_spec("PIL")` to verify if `PIL` is installed. While both `pip show` and `importlib` work correctly, I still got the error.

In my case, restarting and redoing the `datasets` import helped. Seems weird to me.
Fix lock permission
Opened by cih9088 (open)

All files except lock file have proper permission obeying `ACL` property if it is set.

If the cache directory has `ACL` property, it should be respected instead of just using `umask` for permission.
To fix it, just create a lock file and pass the created `mode`.

By creating a lock file with `touch()` before `FileLock` create it with `mode`,
- if `ACL` is not set, same as before
- if `ACL` is set, `ACL` is respected

If it is acceptable, it should be also applied to [`huggingface_hub`](https://github.com/huggingface/huggingface_hub/blob/2702ec2a2bd0124cc1fddfd72ccb1297b2478148/src/huggingface_hub/utils/_fixes.py#L95) I guess.

Comments:
Fix remove_columns in the formatted case
Opened by lhoestq (open)

`remove_columns` had no effect when running a function in `.map()` on dataset that is formatted

This aligns the logic of `map()` with the non formatted case and also with with https://github.com/huggingface/datasets/pull/7353

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7358). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
How about adding a feature to pass the key when performing map on DatasetDict?
Opened by jp1924 (closed)

### Feature request

Add a feature to pass the key of the DatasetDict when performing map

### Motivation

I often preprocess using map on DatasetDict. 
Sometimes, I need to preprocess train and valid data differently depending on the task. 
So, I thought it would be nice to pass the key (like train, valid) when performing map on DatasetDict. 

What do you think?

### Your contribution

I can submit a pull request to add the feature to pass the key of the DatasetDict when performing map.

Comments:
@lhoestq 
If it's okay with you, can I work on this?

Hi ! Can you give an example of what it would look like to use this new feature ?

Note that currently you can already do

```python
ds["train"] = ds["train"].map(process_train)
ds["test"] = ds["test"].map(process_test)
```

@lhoestq 
Thanks for the response! 
Let me clarify what I'm looking for with an example:

Currently, we need to write separate processing functions or call .map() separately:
```python
# Current approach
def process_train(example):
    # Training-specific processing
    return example

def process_valid(example):
    # Validation-specific processing
    return example

ds["train"] = ds["train"].map(process_train)
ds["valid"] = ds["valid"].map(process_valid)
```

What I'm proposing is to have a single processing function that knows which split it's processing:

```python
# Proposed feature
def process(example, split_key):
    if split_key == "train":
        # Training-specific processing
    elif split_key == "valid":
        # Validation-specific processing
    return example

# Using with_key=True to pass the split information
ds = ds.map(process, with_key=True)
```

This becomes particularly useful when:
1. The processing logic is heavily shared between splits but needs minor adjustments
2. You want to maintain the processing logic in one place for better maintainability
3. The processing function is complex and you want to avoid duplicating code

So I wanted to request this feature to achieve this kind of functionality. 
I've created a draft PR implementing this: https://github.com/huggingface/datasets/pull/7240/files


I see ! I think it makes sense, and it's more readable than doing something like this:
```python
from functools import partial
ds = DatasetDict({key: ds[key].map(partial(process, split_key=key)) for key in ds})
```

PS: you named the argument `with_key` in your example, but it might be even clearer with it's named `with_split` maybe no ?

@lhoestq I agree. 
It seems better to use `with_split`.
So can I open a PR with this change?

Sure !
A module that was compiled using NumPy 1.x cannot be run in NumPy 2.0.2 as it may crash. To support both 1.x and 2.x versions of NumPy, modules must be compiled with NumPy 2.0. Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
Opened by jamessdixon (closed)

### Describe the bug

Following this tutorial: https://huggingface.co/docs/diffusers/en/tutorials/basic_training and running it locally using VSCode on my MacBook.  The first line in the tutorial fails:  from datasets import load_dataset
dataset = load_dataset('huggan/smithsonian_butterflies_subset', split="train"). with this error: 
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2. and ImportError: numpy.core.multiarray failed to import.   

Does from datasets import load_dataset really use NumPy 1.x?  

### Steps to reproduce the bug

Open VSCode.  create a new venv.  Create a new ipynb file.  Import pip install diffusers[training] try to run this line of code: from datasets import load_dataset

### Expected behavior

data is loaded

### Environment info

ran this: datasets-cli env
and got A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Comments:
recreated .venv and run this: pip install diffusers[training]==0.11.1
changes to MappedExamplesIterable to resolve #7345
Opened by vttrifonov (closed)

modified `MappedExamplesIterable` and `test_iterable_dataset.py::test_mapped_examples_iterable_with_indices`

fix #7345

@lhoestq 

Comments:
I noticed that `Dataset.map` has a more complex output depending on `remove_columns`. In particular [this](https://github.com/huggingface/datasets/blob/6457be66e2ef88411281eddc4e7698866a3977f1/src/datasets/arrow_dataset.py#L3371) line removes columns from output if the input is being modified in place (i.e. `input_columns = None`). I tried to mimic this behaviour in `MappedExamplesIterable` by checking if the input and output point to the same dictionary object.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7353). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Webdataset special columns in last position
Opened by lhoestq (closed)

Place columns "__key__" and "__url__" in last position in the Dataset Viewer since they are not the main content

before:

<img width="1012" alt="image" src="https://github.com/user-attachments/assets/b556c1fe-2674-4ba0-9643-c074aa9716fd" />


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7349). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Catch OSError for arrow
Opened by lhoestq (closed)

fixes https://github.com/huggingface/datasets/issues/7346

(also updated `ruff` and appleid style changes)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7348). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.