ahmedfarazsyk/datasets-github-issues huggingface dataset:
Dynamic adjusting dataloader sampling weight
Author: whc688
State: open

Hi,
Thanks for your wonderful work! I'm wondering is there a way to dynamically adjust the sampling weight of each data in the dataset during training? Looking forward to your reply, thanks again.

Comments: You mean based on a condition that has to be checked on-the-fly during training ? Otherwise if you know in advance after how many samples you need to change the sampling you can simply concatenate the two mixes

Yes, like during training, if one data sample's prediction is consistently wrong, its sampling weight gets higher and higher, and if one data sample's prediction is already correct, then we rarely sample it

it's not possible to use `interleave_datasets()` and modify the probabilities while iterating on the dataset at the moment, so you'd have to implement your own torch `Sampler` or your own`IterableDataset` to implement this logic
Add bookfolder Dataset Builder for Digital Book Formats
Author: shikanime
State: closed
Labels: name: enhancement

### Feature request

This feature proposes adding a new dataset builder called bookfolder to the datasets library. This builder would allow users to easily load datasets consisting of various digital book formats, including: AZW, AZW3, CB7, CBR, CBT, CBZ, EPUB, MOBI, and PDF.

### Motivation

Currently, loading datasets of these digital book files requires manual effort. This would also lower the barrier to entry for working with these formats, enabling more diverse and interesting datasets to be used within the Hugging Face ecosystem.

### Your contribution

This feature is rather simple as it will be based on the folder-based builder, similar to imagefolder. I'm willing to contribute to this feature by submitting a PR

Comments: On second thought, probably not a good idea.
Allow pushing config version to hub
Author: momeara
State: open
Labels: name: enhancement

### Feature request

Currently, when datasets are created, they can be versioned by passing the `version` argument to `load_dataset(...)`. For example creating `outcomes.csv` on the command line

```
echo "id,value\n1,0\n2,0\n3,1\n4,1\n" > outcomes.csv
```
and creating it
```
import datasets
dataset = datasets.load_dataset(
    "csv",
    data_files ="outcomes.csv",
    keep_in_memory = True,
    version = '1.0.0')
```

The version info is stored in the `info` and can be accessed e.g. by `next(iter(dataset.values())).info.version`

This dataset can be uploaded to the hub with `dataset.push_to_hub(repo_id = "maomlab/example_dataset")`. This will create a dataset  on the hub with the following in the `README.md`, but it doesn't upload the version information:

```
---
dataset_info:
  features:
  - name: id
    dtype: int64
  - name: value
    dtype: int64
  splits:
  - name: train
    num_bytes: 64
    num_examples: 4
  download_size: 1332
  dataset_size: 64
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
---
```

However, when I download from the hub, the version information is missing:

```
dataset_from_hub_no_version = datasets.load_dataset("maomlab/example_dataset")
next(iter(dataset.values())).info.version
```

I can add the version information manually to the hub, by appending it to the end of config section:

```
...
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  version: 1.0.0
---
```

And then when I download it, the version information is correct.





### Motivation

### Why adding version information for each config makes sense
  1. The version information is already recorded in the dataset config info data structure and is able to parse it correctly, so it makes sense to sync it with `push_to_hub`. 
  2. Keeping the version info in at the config level is different from version info at the branch level. As the former relates to the version of the specific dataset the config refers to rather than the version of the dataset curation itself.

## A explanation for the current behavior:
In [datasets/src/datasets/info.py:159](https://github.com/huggingface/datasets/blob/fb91fd3c9ea91a818681a777faf8d0c46f14c680/src/datasets/info.py#L159C1-L160C1
), the `_INCLUDED_INFO_IN_YAML` variable doesn't include `"version"`.

If my reading of the code is right, adding `"version"` to `_INCLUDED_INFO_IN_YAML`, would allow the version information to be uploaded to the hub.

### Your contribution

Request: add `"version"` to `_INCLUDE_INFO_IN_YAML` in [datasets/src/datasets/info.py:159](https://github.com/huggingface/datasets/blob/fb91fd3c9ea91a818681a777faf8d0c46f14c680/src/datasets/info.py#L159C1-L160C1
)


Comments: Hi ! This sounds reasonable to me, feel free to open a PR :)
Support for sparse arrays with the Arrow Sparse Tensor format?
Author: JulesGM
State: open
Labels: name: enhancement

### Feature request

AI in biology is becoming a big thing. One thing that would be a huge benefit to the field that Huggingface Datasets doesn't currently have is native support for **sparse arrays**. 


Arrow has support for sparse tensors. 
https://arrow.apache.org/docs/format/Other.html#sparse-tensor


It would be a big deal if Hugging Face Datasets supported sparse tensors as a feature type, natively. 


### Motivation

This is important for example in the field of transcriptomics (modeling and understanding gene expression), because a large fraction of the genes are not expressed (zero). More generally, in science, sparse arrays are very common, so adding support for them would be very benefitial, it would make just using Hugging Face Dataset objects a lot more straightforward and clean.


### Your contribution

We can discuss this further once the team comments of what they think about the feature, and if there were previous attempts at making it work, and understanding their evaluation of how hard it would be. My intuition is that it should be fairly straightforward, as the Arrow backend already supports it.

Comments: Hi ! Unfortunately the Sparse Tensor structure in Arrow is not part of the Arrow format (yes it's confusing...), so it's not possible to use it in `datasets`. It's a separate structure that doesn't correspond to any type or extension type in Arrow.

The Arrow community recently added an extension type for fixed shape tensors at https://arrow.apache.org/docs/format/CanonicalExtensions.html#fixed-shape-tensor, it should be possible to contribute an extension type for sparse tensors as well.
vllm批量推理报错
Author: YuShengzuishuai
State: open

### Describe the bug

![Image](https://github.com/user-attachments/assets/3d958e43-28dc-4467-9333-5990c7af3b3f)

### Steps to reproduce the bug

![Image](https://github.com/user-attachments/assets/3067eeca-a54d-4956-b0fd-3fc5ea93dabb)

### Expected behavior

![Image](https://github.com/user-attachments/assets/77d32936-488f-4572-9365-bfb4170e555b)

### Environment info

![Image](https://github.com/user-attachments/assets/230335c4-825f-4db1-b07d-4776ef63ead8)

Comments: Make sure you have installed a recent version of `soundfile`
Excessive RAM Usage After Dataset Concatenation concatenate_datasets
Author: sam-hey
State: open

### Describe the bug

When loading a dataset from disk, concatenating it, and starting the training process, the RAM usage progressively increases until the kernel terminates the process due to excessive memory consumption.

https://github.com/huggingface/datasets/issues/2276

### Steps to reproduce the bug

```python
from datasets import  DatasetDict, concatenate_datasets

dataset = DatasetDict.load_from_disk("data")

...
...

combined_dataset = concatenate_datasets(
        [dataset[split] for split in dataset]
    )

#start SentenceTransformer training
```

### Expected behavior

I would not expect RAM utilization to increase after concatenation. Removing the concatenation step resolves the issue

### Environment info

sentence-transformers==3.1.1
datasets==3.2.0

python3.10

Comments: ![Image](https://github.com/user-attachments/assets/b6f8bcbd-44af-413e-bc06-65380eb0f746)

![Image](https://github.com/user-attachments/assets/a241fcd8-4b62-495c-926c-685f82015dfb)

Adding a img from memray
https://gist.github.com/sam-hey/00c958f13fb0f7b54d17197fe353002f

I'm having the same issue where concatenation seems to use a huge amount of RAM.

```python
# Load all chunks and concatenate them into a final dataset.
        chunk_datasets = [
            Dataset.load_from_disk(file, keep_in_memory=False)
            for file in tqdm(chunk_files, desc="Loading chunk datasets")
        ]
        logging.info("Concatenating chunk datasets...")
        final_dataset = concatenate_datasets(chunk_datasets)
```

This is a real issue for me as the final dataset is a few terabytes in size. I'm using datasets version `3.1.0`. Also tested with version `3.4.1`

I did have a short look, the error seems to be from `memory_map` and the stream not being closed. 

https://github.com/huggingface/datasets/blob/5f8d2ad9a1b0bccfd962d998987228addfd5be9f/src/datasets/table.py#L48-L50


Did not have the time to test jet: https://github.com/sam-hey/datasets/tree/fix/concatenate_datasets

I will probably have a better look in a couple of days.
Inconsistent Behavior Between `load_dataset` and `load_from_disk` When Loading Sharded Datasets
Author: gaohongkui
State: open

### Description

I encountered an inconsistency in behavior between `load_dataset` and `load_from_disk` when loading sharded datasets. Here is a minimal example to reproduce the issue:

#### Code 1: Using `load_dataset`
```python
from datasets import Dataset, load_dataset

# First save with max_shard_size=10
Dataset.from_dict({"id": range(1000)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Second save with max_shard_size=10
Dataset.from_dict({"id": range(500)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Load the DatasetDict
loaded_datasetdict = load_dataset("my_sharded_datasetdict")
print(loaded_datasetdict)
```
**Output**:
- `train` has 1350 samples.
- `test` has 150 samples.

#### Code 2: Using `load_from_disk`
```python
from datasets import Dataset, load_from_disk

# First save with max_shard_size=10
Dataset.from_dict({"id": range(1000)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Second save with max_shard_size=10
Dataset.from_dict({"id": range(500)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Load the DatasetDict
loaded_datasetdict = load_from_disk("my_sharded_datasetdict")
print(loaded_datasetdict)
```
**Output**:
- `train` has 450 samples.
- `test` has 50 samples.

### Expected Behavior
I expected both `load_dataset` and `load_from_disk` to load the same dataset, as they are pointing to the same directory. However, the results differ significantly:
- `load_dataset` seems to merge all shards, resulting in a combined dataset.
- `load_from_disk` only loads the last saved dataset, ignoring previous shards.

### Questions
1. Is this behavior intentional? If so, could you clarify the difference between `load_dataset` and `load_from_disk` in the documentation?
2. If this is not intentional, could this be considered a bug?
3. What is the recommended way to handle cases where multiple datasets are saved to the same directory?


Thank you for your time and effort in maintaining this great library! I look forward to your feedback.
Importing dataset gives unhelpful error message when filenames in metadata.csv are not found in the directory
Author: svencornetsdegroot
State: open

### Describe the bug

While importing an audiofolder dataset, where the names of the audiofiles don't correspond to the filenames in the metadata.csv, we get an unclear error message that is not helpful for the debugging, i.e. 
```
ValueError: Instruction "train" corresponds to no data!
```

### Steps to reproduce the bug

Assume an audiofolder with audiofiles, filename1.mp3, filename2.mp3 etc and a file metadata.csv which contains the columns file_name and sentence. The file_names are formatted like filename1.mp3, filename2.mp3 etc. 

Load the audio
```
from datasets import load_dataset
load_dataset("audiofolder", data_dir='/path/to/audiofolder')
```
When the file_names in the csv are not in sync with the filenames in the audiofolder, then we get an Error message:
```
File /opt/conda/lib/python3.12/site-packages/datasets/arrow_reader.py:251, in BaseReader.read(self, name, instructions, split_infos, in_memory)
    249 if not files:
    250     msg = f'Instruction "{instructions}" corresponds to no data!'
--> 251     raise ValueError(msg)
    252 return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)

ValueError: Instruction "train" corresponds to no data!
```
load_dataset has a default value for the argument split = 'train'. 


### Expected behavior

It would be better to get an error report something like:
```
The metadata.csv file has different filenames than the files in the datadirectory. 
```
It would have saved me 4 hours of debugging. 

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.14.0-427.40.1.el9_4.x86_64-x86_64-with-glibc2.39
- Python version: 3.12.8
- `huggingface_hub` version: 0.27.0
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments: I'd prefer even more verbose errors; like `"file123.mp3" is referenced in metadata.csv, but not found in the data directory '/path/to/audiofolder' ! (and 100+ more missing files)` Or something along those lines.
Dataset.from_dict() can't handle large dict
Author: CSU-OSS
State: open

### Describe the bug

I have 26,000,000 3-tuples. When I use Dataset.from_dict() to load, neither. py nor Jupiter notebook can run successfully. This is my code:
```
    # len(example_data) is 26,000,000, 'diff' is a text
    diff1_list = [example_data[i].texts[0] for i in range(len(example_data))]
    diff2_list = [example_data[i].texts[1] for i in range(len(example_data))]
    label_list = [example_data[i].label for i in range(len(example_data))]

    embedding_dataset = Dataset.from_dict({
        "diff1": diff1_list,
        "diff2": diff2_list,
        "label": label_list
    })
```

### Steps to reproduce the bug

1. Initialize a large 3-tuple, e.g. 26,000,000
2. Use Dataset.from_dict() to load

### Expected behavior

Dataset.from_dict() run successfully

### Environment info

sentence-transformers                   3.3.1
A parameter is specified but not used in datasets.arrow_dataset.Dataset.from_pandas()
Author: NourOM02
State: open

### Describe the bug

I am interested in creating train, test and eval splits from a pandas Dataframe, therefore I was looking at the possibilities I can follow. I noticed the split parameter and was hopeful to use it in order to generate the 3 at once, however, while trying to understand the code, i noticed that it has no added value (correct me if I am wrong or misunderstood the code). 


from_pandas function code :

```python
    if info is not None and features is not None and info.features != features:
            raise ValueError(
                f"Features specified in `features` and `info.features` can't be different:\n{features}\n{info.features}"
            )
        features = features if features is not None else info.features if info is not None else None
        if info is None:
            info = DatasetInfo()
        info.features = features
        table = InMemoryTable.from_pandas(
            df=df,
            preserve_index=preserve_index,
        )
        if features is not None:
            # more expensive cast than InMemoryTable.from_pandas(..., schema=features.arrow_schema)
            # needed to support the str to Audio conversion for instance
            table = table.cast(features.arrow_schema)
        return cls(table, info=info, split=split)
```

### Steps to reproduce the bug

```python
from datasets import Dataset
# Filling the split parameter with whatever causes no harm at all
data = Dataset.from_pandas(self.raw_data, split='egiojegoierjgoiejgrefiergiuorenvuirgurthgi')
```

### Expected behavior

Would be great if there is no split parameter (if it isn't working), or to add a concrete example of how it can be used.

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.15.0-127-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.27.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
ImportError: To support decoding images, please install 'Pillow'.
Author: jamessdixon
State: open

### Describe the bug

Following this tutorial locally using a macboko and VSCode: https://huggingface.co/docs/diffusers/en/tutorials/basic_training
This line of code: for i, image in enumerate(dataset[:4]["image"]):
throws: ImportError: To support decoding images, please install 'Pillow'.

Pillow is installed.

### Steps to reproduce the bug

Run the tutorial


### Expected behavior

Images should be rendered

### Environment info

MacBook, VSCode

Comments: what's your `pip show Pillow` output

same issue.. my pip show Pillow output as below:

```
Name: pillow
Version: 11.1.0
Summary: Python Imaging Library (Fork)
Home-page: https://python-pillow.github.io/
Author: 
Author-email: "Jeffrey A. Clark" <aclark@aclark.net>
License: MIT-CMU
Location: [/opt/homebrew/lib/python3.10/site-packages](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/lib/python3.10/site-packages)
Requires: 
Required-by:
```

I encountered the same problem on Ubuntu system, my pip show Pillow output as below:

```
Name: pillow
Version: 10.4.0
Summary: Python Imaging Library (Fork)
Home-page: https://python-pillow.org/
Author: 
Author-email: "Jeffrey A. Clark" <[aclark@aclark.net](mailto:aclark@aclark.net)>
License: HPND
Location: /home/shunying/.local/lib/python3.8/site-packages
Requires: 
Required-by: 
```

Well, solved this by specifying the pip version to my conda virtual environment :)

I have also encountered this. It's a strange thing that's happening.

Checking the code `datasets` it uses `importlib.util.find_spec("PIL")` to verify if `PIL` is installed. While both `pip show` and `importlib` work correctly, I still got the error.

In my case, restarting and redoing the `datasets` import helped. Seems weird to me.
How about adding a feature to pass the key when performing map on DatasetDict?
Author: jp1924
State: closed
Labels: name: enhancement

### Feature request

Add a feature to pass the key of the DatasetDict when performing map

### Motivation

I often preprocess using map on DatasetDict. 
Sometimes, I need to preprocess train and valid data differently depending on the task. 
So, I thought it would be nice to pass the key (like train, valid) when performing map on DatasetDict. 

What do you think?

### Your contribution

I can submit a pull request to add the feature to pass the key of the DatasetDict when performing map.

Comments: @lhoestq 
If it's okay with you, can I work on this?

Hi ! Can you give an example of what it would look like to use this new feature ?

Note that currently you can already do

```python
ds["train"] = ds["train"].map(process_train)
ds["test"] = ds["test"].map(process_test)
```

@lhoestq 
Thanks for the response! 
Let me clarify what I'm looking for with an example:

Currently, we need to write separate processing functions or call .map() separately:
```python
# Current approach
def process_train(example):
    # Training-specific processing
    return example

def process_valid(example):
    # Validation-specific processing
    return example

ds["train"] = ds["train"].map(process_train)
ds["valid"] = ds["valid"].map(process_valid)
```

What I'm proposing is to have a single processing function that knows which split it's processing:

```python
# Proposed feature
def process(example, split_key):
    if split_key == "train":
        # Training-specific processing
    elif split_key == "valid":
        # Validation-specific processing
    return example

# Using with_key=True to pass the split information
ds = ds.map(process, with_key=True)
```

This becomes particularly useful when:
1. The processing logic is heavily shared between splits but needs minor adjustments
2. You want to maintain the processing logic in one place for better maintainability
3. The processing function is complex and you want to avoid duplicating code

So I wanted to request this feature to achieve this kind of functionality. 
I've created a draft PR implementing this: https://github.com/huggingface/datasets/pull/7240/files


I see ! I think it makes sense, and it's more readable than doing something like this:
```python
from functools import partial
ds = DatasetDict({key: ds[key].map(partial(process, split_key=key)) for key in ds})
```

PS: you named the argument `with_key` in your example, but it might be even clearer with it's named `with_split` maybe no ?

@lhoestq I agree. 
It seems better to use `with_split`.
So can I open a PR with this change?

Sure !
A module that was compiled using NumPy 1.x cannot be run in NumPy 2.0.2 as it may crash. To support both 1.x and 2.x versions of NumPy, modules must be compiled with NumPy 2.0. Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
Author: jamessdixon
State: closed

### Describe the bug

Following this tutorial: https://huggingface.co/docs/diffusers/en/tutorials/basic_training and running it locally using VSCode on my MacBook.  The first line in the tutorial fails:  from datasets import load_dataset
dataset = load_dataset('huggan/smithsonian_butterflies_subset', split="train"). with this error: 
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2. and ImportError: numpy.core.multiarray failed to import.   

Does from datasets import load_dataset really use NumPy 1.x?  

### Steps to reproduce the bug

Open VSCode.  create a new venv.  Create a new ipynb file.  Import pip install diffusers[training] try to run this line of code: from datasets import load_dataset

### Expected behavior

data is loaded

### Environment info

ran this: datasets-cli env
and got A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Comments: recreated .venv and run this: pip install diffusers[training]==0.11.1