NyssaW/github-issues huggingface dataset:
Add bookfolder Dataset Builder for Digital Book Formats

### Feature request

This feature proposes adding a new dataset builder called bookfolder to the datasets library. This builder would allow users to easily load datasets consisting of various digital book formats, including: AZW, AZW3, CB7, CBR, CBT, CBZ, EPUB, MOBI, and PDF.

### Motivation

Currently, loading datasets of these digital book files requires manual effort. This would also lower the barrier to entry for working with these formats, enabling more diverse and interesting datasets to be used within the Hugging Face ecosystem.

### Your contribution

This feature is rather simple as it will be based on the folder-based builder, similar to imagefolder. I'm willing to contribute to this feature by submitting a PR

Comments:
On second thought, probably not a good idea.
Make IterableDataset (optionally) resumable

### What does this PR do?
This PR introduces a new `stateful` option to the `dataset.shuffle` method, which defaults to `False`. 
When enabled, this option allows for resumable shuffling of `IterableDataset` instances, albeit with some additional memory overhead.

Key points:
* All tests have passed
* Docstrings have been updated to reflect the new functionality

I'm very looking forward to receiving feedback on this implementation! @lhoestq

Comments:
@lhoestq Hi again~ Just circling back on this
Wondering if there’s anything I can do to help move this forward. 🤗 
Thanks!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7385). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support async functions in map()

e.g. to download images or call an inference API like HF or vLLM

```python
import asyncio
import random

from datasets import Dataset


async def f(x):
    await asyncio.sleep(random.random())

ds = Dataset.from_dict({"data": range(100)})
ds.map(f)
# Map: 100%|█████████████████████████████| 100/100 [00:01<00:00, 99.81 examples/s]
```

TODO
- [x] clean code (right now it's a big copy paste)
- [x] batched
- [x] Dataset.map()
- [x] IterableDataset.map()
- [x] Dataset.filter()
- [x] IterableDataset.filter()
- [x] test
- [x] docs

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7384). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

example of what you can do with it:

```python
import aiohttp
from huggingface_hub import get_token

from datasets import Dataset


API_URL = "https://api-inference.huggingface.co/models/microsoft/Phi-3-mini-4k-instruct/v1/chat/completions"
PROMPT = "What is this text mainly about ? Here is the text:\n\n```\n{Problem}\n```\n\nReply in one or two words."

async def query(example):
    headers = {"Authorization": f"Bearer {get_token()}", "Content-Type": "application/json"}
    json = {"messages": [{"role": "user", "content": PROMPT.format(Problem=example["Problem"])}], "max_tokens": 20, "seed": 42}
    async with aiohttp.ClientSession() as session, session.post(API_URL, headers=headers, json=json) as response:
        output = await response.json()
        return {"output": output["choices"][0]["message"]["content"]}

ds = Dataset.from_dict({"Problem": ["1 + 1"] * 10})
ds = ds.map(query)
print(ds[0])
# {'Problem': '1 + 1', 'output': 'Arithmetic'}
```
fix: dill default for version bigger 0.3.8

Fixes def log for dill version >= 0.3.9 
https://pypi.org/project/dill/

This project uses dill with the release of version 0.3.9 the datasets lib. 

Comments:
`datasets` doesn't support `dill` 0.3.9 yet afaik since `dill` made some changes related to the determinism of dumps

It would be cool to investigate (maybe run the `datasets` test) with recent `dill` to see excactly what breaks and if we can make `dill` 0.3.9 work with `datasets`
Allow pushing config version to hub

### Feature request

Currently, when datasets are created, they can be versioned by passing the `version` argument to `load_dataset(...)`. For example creating `outcomes.csv` on the command line

```
echo "id,value\n1,0\n2,0\n3,1\n4,1\n" > outcomes.csv
```
and creating it
```
import datasets
dataset = datasets.load_dataset(
    "csv",
    data_files ="outcomes.csv",
    keep_in_memory = True,
    version = '1.0.0')
```

The version info is stored in the `info` and can be accessed e.g. by `next(iter(dataset.values())).info.version`

This dataset can be uploaded to the hub with `dataset.push_to_hub(repo_id = "maomlab/example_dataset")`. This will create a dataset  on the hub with the following in the `README.md`, but it doesn't upload the version information:

```
---
dataset_info:
  features:
  - name: id
    dtype: int64
  - name: value
    dtype: int64
  splits:
  - name: train
    num_bytes: 64
    num_examples: 4
  download_size: 1332
  dataset_size: 64
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
---
```

However, when I download from the hub, the version information is missing:

```
dataset_from_hub_no_version = datasets.load_dataset("maomlab/example_dataset")
next(iter(dataset.values())).info.version
```

I can add the version information manually to the hub, by appending it to the end of config section:

```
...
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  version: 1.0.0
---
```

And then when I download it, the version information is correct.





### Motivation

### Why adding version information for each config makes sense
  1. The version information is already recorded in the dataset config info data structure and is able to parse it correctly, so it makes sense to sync it with `push_to_hub`. 
  2. Keeping the version info in at the config level is different from version info at the branch level. As the former relates to the version of the specific dataset the config refers to rather than the version of the dataset curation itself.

## A explanation for the current behavior:
In [datasets/src/datasets/info.py:159](https://github.com/huggingface/datasets/blob/fb91fd3c9ea91a818681a777faf8d0c46f14c680/src/datasets/info.py#L159C1-L160C1
), the `_INCLUDED_INFO_IN_YAML` variable doesn't include `"version"`.

If my reading of the code is right, adding `"version"` to `_INCLUDED_INFO_IN_YAML`, would allow the version information to be uploaded to the hub.

### Your contribution

Request: add `"version"` to `_INCLUDE_INFO_IN_YAML` in [datasets/src/datasets/info.py:159](https://github.com/huggingface/datasets/blob/fb91fd3c9ea91a818681a777faf8d0c46f14c680/src/datasets/info.py#L159C1-L160C1
)


Comments:
Hi ! This sounds reasonable to me, feel free to open a PR :)
Support for sparse arrays with the Arrow Sparse Tensor format?

### Feature request

AI in biology is becoming a big thing. One thing that would be a huge benefit to the field that Huggingface Datasets doesn't currently have is native support for **sparse arrays**. 


Arrow has support for sparse tensors. 
https://arrow.apache.org/docs/format/Other.html#sparse-tensor


It would be a big deal if Hugging Face Datasets supported sparse tensors as a feature type, natively. 


### Motivation

This is important for example in the field of transcriptomics (modeling and understanding gene expression), because a large fraction of the genes are not expressed (zero). More generally, in science, sparse arrays are very common, so adding support for them would be very benefitial, it would make just using Hugging Face Dataset objects a lot more straightforward and clean.


### Your contribution

We can discuss this further once the team comments of what they think about the feature, and if there were previous attempts at making it work, and understanding their evaluation of how hard it would be. My intuition is that it should be fairly straightforward, as the Arrow backend already supports it.

Comments:
Hi ! Unfortunately the Sparse Tensor structure in Arrow is not part of the Arrow format (yes it's confusing...), so it's not possible to use it in `datasets`. It's a separate structure that doesn't correspond to any type or extension type in Arrow.

The Arrow community recently added an extension type for fixed shape tensors at https://arrow.apache.org/docs/format/CanonicalExtensions.html#fixed-shape-tensor, it should be possible to contribute an extension type for sparse tensors as well.

Thanks for the clarification on the Arrow format limitations @lhoestq 

Since a native Arrow Extension Type for sparse tensors might be a longer-term upstream effort, I wanted to propose a pragmatic solution that could be implemented in datasets immediately to help users in domains such as biology or NLP.

Instead of waiting for a native sparse type, we can efficiently handle sparse data by formalizing a schema based on nested lists (COO/CSR style) using existing Arrow types. This avoids OOM issues caused by loading dense zeros while maintaining full compatibility with the current Arrow backend.

The Strategy: We could add a reader (or a "best practice" utility) that ingests sparse formats (like wide CSVs or Matrix Market) and stores them as a struct of sequences:
```
features = Features({
    "row_id": Value("int32"),
    "indices": Sequence(Value("int32")),
    "values": Sequence(Value("float32")),
    "shape": Sequence(Value("int32"), length=2) # Store original shape
})
```

#self-assign
vllm批量推理报错

### Describe the bug

![Image](https://github.com/user-attachments/assets/3d958e43-28dc-4467-9333-5990c7af3b3f)

### Steps to reproduce the bug

![Image](https://github.com/user-attachments/assets/3067eeca-a54d-4956-b0fd-3fc5ea93dabb)

### Expected behavior

![Image](https://github.com/user-attachments/assets/77d32936-488f-4572-9365-bfb4170e555b)

### Environment info

![Image](https://github.com/user-attachments/assets/230335c4-825f-4db1-b07d-4776ef63ead8)

Comments:
Make sure you have installed a recent version of `soundfile`
Excessive RAM Usage After Dataset Concatenation concatenate_datasets

### Describe the bug

When loading a dataset from disk, concatenating it, and starting the training process, the RAM usage progressively increases until the kernel terminates the process due to excessive memory consumption.

https://github.com/huggingface/datasets/issues/2276

### Steps to reproduce the bug

```python
from datasets import  DatasetDict, concatenate_datasets

dataset = DatasetDict.load_from_disk("data")

...
...

combined_dataset = concatenate_datasets(
        [dataset[split] for split in dataset]
    )

#start SentenceTransformer training
```

### Expected behavior

I would not expect RAM utilization to increase after concatenation. Removing the concatenation step resolves the issue

### Environment info

sentence-transformers==3.1.1
datasets==3.2.0

python3.10

Comments:
![Image](https://github.com/user-attachments/assets/b6f8bcbd-44af-413e-bc06-65380eb0f746)

![Image](https://github.com/user-attachments/assets/a241fcd8-4b62-495c-926c-685f82015dfb)

Adding a img from memray
https://gist.github.com/sam-hey/00c958f13fb0f7b54d17197fe353002f

I'm having the same issue where concatenation seems to use a huge amount of RAM.

```python
# Load all chunks and concatenate them into a final dataset.
        chunk_datasets = [
            Dataset.load_from_disk(file, keep_in_memory=False)
            for file in tqdm(chunk_files, desc="Loading chunk datasets")
        ]
        logging.info("Concatenating chunk datasets...")
        final_dataset = concatenate_datasets(chunk_datasets)
```

This is a real issue for me as the final dataset is a few terabytes in size. I'm using datasets version `3.1.0`. Also tested with version `3.4.1`

I did have a short look, the error seems to be from `memory_map` and the stream not being closed. 

https://github.com/huggingface/datasets/blob/5f8d2ad9a1b0bccfd962d998987228addfd5be9f/src/datasets/table.py#L48-L50


Did not have the time to test jet: https://github.com/sam-hey/datasets/tree/fix/concatenate_datasets

I will probably have a better look in a couple of days.
Inconsistent Behavior Between `load_dataset` and `load_from_disk` When Loading Sharded Datasets

### Description

I encountered an inconsistency in behavior between `load_dataset` and `load_from_disk` when loading sharded datasets. Here is a minimal example to reproduce the issue:

#### Code 1: Using `load_dataset`
```python
from datasets import Dataset, load_dataset

# First save with max_shard_size=10
Dataset.from_dict({"id": range(1000)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Second save with max_shard_size=10
Dataset.from_dict({"id": range(500)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Load the DatasetDict
loaded_datasetdict = load_dataset("my_sharded_datasetdict")
print(loaded_datasetdict)
```
**Output**:
- `train` has 1350 samples.
- `test` has 150 samples.

#### Code 2: Using `load_from_disk`
```python
from datasets import Dataset, load_from_disk

# First save with max_shard_size=10
Dataset.from_dict({"id": range(1000)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Second save with max_shard_size=10
Dataset.from_dict({"id": range(500)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Load the DatasetDict
loaded_datasetdict = load_from_disk("my_sharded_datasetdict")
print(loaded_datasetdict)
```
**Output**:
- `train` has 450 samples.
- `test` has 50 samples.

### Expected Behavior
I expected both `load_dataset` and `load_from_disk` to load the same dataset, as they are pointing to the same directory. However, the results differ significantly:
- `load_dataset` seems to merge all shards, resulting in a combined dataset.
- `load_from_disk` only loads the last saved dataset, ignoring previous shards.

### Questions
1. Is this behavior intentional? If so, could you clarify the difference between `load_dataset` and `load_from_disk` in the documentation?
2. If this is not intentional, could this be considered a bug?
3. What is the recommended way to handle cases where multiple datasets are saved to the same directory?


Thank you for your time and effort in maintaining this great library! I look forward to your feedback.

Comments:
Support faster processing using pandas or polars functions in `IterableDataset.map()`

Following the polars integration :)

Allow super fast processing using pandas or polars functions in `IterableDataset.map()` by adding support to pandas and polars formatting in `IterableDataset`

```python
import polars as pl
from datasets import Dataset

ds = Dataset.from_dict({"i": range(10)}).to_iterable_dataset()
ds = ds.with_format("polars")
ds = ds.map(lambda df: df.with_columns(pl.col("i").add(1).alias("i+1")), batched=True)
ds = ds.with_format(None)
print(next(iter(ds)))
# {'i': 0, 'i+1': 1}
```

It leverages arrow's zero-copy features from/to pandas and polars.

related to https://github.com/huggingface/datasets/issues/3444 #6762 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7370). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

merging this and will make some docs and communications around using polars for optimizing data processing :)
Importing dataset gives unhelpful error message when filenames in metadata.csv are not found in the directory

### Describe the bug

While importing an audiofolder dataset, where the names of the audiofiles don't correspond to the filenames in the metadata.csv, we get an unclear error message that is not helpful for the debugging, i.e. 
```
ValueError: Instruction "train" corresponds to no data!
```

### Steps to reproduce the bug

Assume an audiofolder with audiofiles, filename1.mp3, filename2.mp3 etc and a file metadata.csv which contains the columns file_name and sentence. The file_names are formatted like filename1.mp3, filename2.mp3 etc. 

Load the audio
```
from datasets import load_dataset
load_dataset("audiofolder", data_dir='/path/to/audiofolder')
```
When the file_names in the csv are not in sync with the filenames in the audiofolder, then we get an Error message:
```
File /opt/conda/lib/python3.12/site-packages/datasets/arrow_reader.py:251, in BaseReader.read(self, name, instructions, split_infos, in_memory)
    249 if not files:
    250     msg = f'Instruction "{instructions}" corresponds to no data!'
--> 251     raise ValueError(msg)
    252 return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)

ValueError: Instruction "train" corresponds to no data!
```
load_dataset has a default value for the argument split = 'train'. 


### Expected behavior

It would be better to get an error report something like:
```
The metadata.csv file has different filenames than the files in the datadirectory. 
```
It would have saved me 4 hours of debugging. 

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.14.0-427.40.1.el9_4.x86_64-x86_64-with-glibc2.39
- Python version: 3.12.8
- `huggingface_hub` version: 0.27.0
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments:
I'd prefer even more verbose errors; like `"file123.mp3" is referenced in metadata.csv, but not found in the data directory '/path/to/audiofolder' ! (and 100+ more missing files)` Or something along those lines.
Add with_split to DatasetDict.map

#7356

Comments:
Can you check this out, @lhoestq?

cc @lhoestq @albertvillanova 

@lhoestq


@lhoestq


@lhoestq

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7368). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq 

@lhoestq please...

Thank you so much for reviewing this PR! Have a great weekend~
Dataset.from_dict() can't handle large dict

### Describe the bug

I have 26,000,000 3-tuples. When I use Dataset.from_dict() to load, neither. py nor Jupiter notebook can run successfully. This is my code:
```
    # len(example_data) is 26,000,000, 'diff' is a text
    diff1_list = [example_data[i].texts[0] for i in range(len(example_data))]
    diff2_list = [example_data[i].texts[1] for i in range(len(example_data))]
    label_list = [example_data[i].label for i in range(len(example_data))]

    embedding_dataset = Dataset.from_dict({
        "diff1": diff1_list,
        "diff2": diff2_list,
        "label": label_list
    })
```

### Steps to reproduce the bug

1. Initialize a large 3-tuple, e.g. 26,000,000
2. Use Dataset.from_dict() to load

### Expected behavior

Dataset.from_dict() run successfully

### Environment info

sentence-transformers                   3.3.1

Comments:
A parameter is specified but not used in datasets.arrow_dataset.Dataset.from_pandas()

### Describe the bug

I am interested in creating train, test and eval splits from a pandas Dataframe, therefore I was looking at the possibilities I can follow. I noticed the split parameter and was hopeful to use it in order to generate the 3 at once, however, while trying to understand the code, i noticed that it has no added value (correct me if I am wrong or misunderstood the code). 


from_pandas function code :

```python
    if info is not None and features is not None and info.features != features:
            raise ValueError(
                f"Features specified in `features` and `info.features` can't be different:\n{features}\n{info.features}"
            )
        features = features if features is not None else info.features if info is not None else None
        if info is None:
            info = DatasetInfo()
        info.features = features
        table = InMemoryTable.from_pandas(
            df=df,
            preserve_index=preserve_index,
        )
        if features is not None:
            # more expensive cast than InMemoryTable.from_pandas(..., schema=features.arrow_schema)
            # needed to support the str to Audio conversion for instance
            table = table.cast(features.arrow_schema)
        return cls(table, info=info, split=split)
```

### Steps to reproduce the bug

```python
from datasets import Dataset
# Filling the split parameter with whatever causes no harm at all
data = Dataset.from_pandas(self.raw_data, split='egiojegoierjgoiejgrefiergiuorenvuirgurthgi')
```

### Expected behavior

Would be great if there is no split parameter (if it isn't working), or to add a concrete example of how it can be used.

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.15.0-127-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.27.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments:
ImportError: To support decoding images, please install 'Pillow'.

### Describe the bug

Following this tutorial locally using a macboko and VSCode: https://huggingface.co/docs/diffusers/en/tutorials/basic_training
This line of code: for i, image in enumerate(dataset[:4]["image"]):
throws: ImportError: To support decoding images, please install 'Pillow'.

Pillow is installed.

### Steps to reproduce the bug

Run the tutorial


### Expected behavior

Images should be rendered

### Environment info

MacBook, VSCode

Comments:
what's your `pip show Pillow` output

same issue.. my pip show Pillow output as below:

```
Name: pillow
Version: 11.1.0
Summary: Python Imaging Library (Fork)
Home-page: https://python-pillow.github.io/
Author: 
Author-email: "Jeffrey A. Clark" <aclark@aclark.net>
License: MIT-CMU
Location: [/opt/homebrew/lib/python3.10/site-packages](https://file+.vscode-resource.vscode-cdn.net/opt/homebrew/lib/python3.10/site-packages)
Requires: 
Required-by:
```

I encountered the same problem on Ubuntu system, my pip show Pillow output as below:

```
Name: pillow
Version: 10.4.0
Summary: Python Imaging Library (Fork)
Home-page: https://python-pillow.org/
Author: 
Author-email: "Jeffrey A. Clark" <[aclark@aclark.net](mailto:aclark@aclark.net)>
License: HPND
Location: /home/shunying/.local/lib/python3.8/site-packages
Requires: 
Required-by: 
```

Well, solved this by specifying the pip version to my conda virtual environment :)

I have also encountered this. It's a strange thing that's happening.

Checking the code `datasets` it uses `importlib.util.find_spec("PIL")` to verify if `PIL` is installed. While both `pip show` and `importlib` work correctly, I still got the error.

In my case, restarting and redoing the `datasets` import helped. Seems weird to me.
Fix lock permission

All files except lock file have proper permission obeying `ACL` property if it is set.

If the cache directory has `ACL` property, it should be respected instead of just using `umask` for permission.
To fix it, just create a lock file and pass the created `mode`.

By creating a lock file with `touch()` before `FileLock` create it with `mode`,
- if `ACL` is not set, same as before
- if `ACL` is set, `ACL` is respected

If it is acceptable, it should be also applied to [`huggingface_hub`](https://github.com/huggingface/huggingface_hub/blob/2702ec2a2bd0124cc1fddfd72ccb1297b2478148/src/huggingface_hub/utils/_fixes.py#L95) I guess.

Comments:
Fix remove_columns in the formatted case

`remove_columns` had no effect when running a function in `.map()` on dataset that is formatted

This aligns the logic of `map()` with the non formatted case and also with with https://github.com/huggingface/datasets/pull/7353

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7358). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
How about adding a feature to pass the key when performing map on DatasetDict?

### Feature request

Add a feature to pass the key of the DatasetDict when performing map

### Motivation

I often preprocess using map on DatasetDict. 
Sometimes, I need to preprocess train and valid data differently depending on the task. 
So, I thought it would be nice to pass the key (like train, valid) when performing map on DatasetDict. 

What do you think?

### Your contribution

I can submit a pull request to add the feature to pass the key of the DatasetDict when performing map.

Comments:
@lhoestq 
If it's okay with you, can I work on this?

Hi ! Can you give an example of what it would look like to use this new feature ?

Note that currently you can already do

```python
ds["train"] = ds["train"].map(process_train)
ds["test"] = ds["test"].map(process_test)
```

@lhoestq 
Thanks for the response! 
Let me clarify what I'm looking for with an example:

Currently, we need to write separate processing functions or call .map() separately:
```python
# Current approach
def process_train(example):
    # Training-specific processing
    return example

def process_valid(example):
    # Validation-specific processing
    return example

ds["train"] = ds["train"].map(process_train)
ds["valid"] = ds["valid"].map(process_valid)
```

What I'm proposing is to have a single processing function that knows which split it's processing:

```python
# Proposed feature
def process(example, split_key):
    if split_key == "train":
        # Training-specific processing
    elif split_key == "valid":
        # Validation-specific processing
    return example

# Using with_key=True to pass the split information
ds = ds.map(process, with_key=True)
```

This becomes particularly useful when:
1. The processing logic is heavily shared between splits but needs minor adjustments
2. You want to maintain the processing logic in one place for better maintainability
3. The processing function is complex and you want to avoid duplicating code

So I wanted to request this feature to achieve this kind of functionality. 
I've created a draft PR implementing this: https://github.com/huggingface/datasets/pull/7240/files


I see ! I think it makes sense, and it's more readable than doing something like this:
```python
from functools import partial
ds = DatasetDict({key: ds[key].map(partial(process, split_key=key)) for key in ds})
```

PS: you named the argument `with_key` in your example, but it might be even clearer with it's named `with_split` maybe no ?

@lhoestq I agree. 
It seems better to use `with_split`.
So can I open a PR with this change?

Sure !
A module that was compiled using NumPy 1.x cannot be run in NumPy 2.0.2 as it may crash. To support both 1.x and 2.x versions of NumPy, modules must be compiled with NumPy 2.0. Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

### Describe the bug

Following this tutorial: https://huggingface.co/docs/diffusers/en/tutorials/basic_training and running it locally using VSCode on my MacBook.  The first line in the tutorial fails:  from datasets import load_dataset
dataset = load_dataset('huggan/smithsonian_butterflies_subset', split="train"). with this error: 
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2. and ImportError: numpy.core.multiarray failed to import.   

Does from datasets import load_dataset really use NumPy 1.x?  

### Steps to reproduce the bug

Open VSCode.  create a new venv.  Create a new ipynb file.  Import pip install diffusers[training] try to run this line of code: from datasets import load_dataset

### Expected behavior

data is loaded

### Environment info

ran this: datasets-cli env
and got A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Comments:
recreated .venv and run this: pip install diffusers[training]==0.11.1
changes to MappedExamplesIterable to resolve #7345

modified `MappedExamplesIterable` and `test_iterable_dataset.py::test_mapped_examples_iterable_with_indices`

fix #7345

@lhoestq 

Comments:
I noticed that `Dataset.map` has a more complex output depending on `remove_columns`. In particular [this](https://github.com/huggingface/datasets/blob/6457be66e2ef88411281eddc4e7698866a3977f1/src/datasets/arrow_dataset.py#L3371) line removes columns from output if the input is being modified in place (i.e. `input_columns = None`). I tried to mimic this behaviour in `MappedExamplesIterable` by checking if the input and output point to the same dictionary object.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7353). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Webdataset special columns in last position

Place columns "__key__" and "__url__" in last position in the Dataset Viewer since they are not the main content

before:

<img width="1012" alt="image" src="https://github.com/user-attachments/assets/b556c1fe-2674-4ba0-9643-c074aa9716fd" />


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7349). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Catch OSError for arrow

fixes https://github.com/huggingface/datasets/issues/7346

(also updated `ruff` and appleid style changes)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7348). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Different behaviour of IterableDataset.map vs Dataset.map with remove_columns

### Describe the bug

The following code

```python
import datasets as hf
ds1 = hf.Dataset.from_list([{'i': i} for i in [0,1]])
#ds1 = ds1.to_iterable_dataset()
ds2 = ds1.map(
    lambda i: {'i': i+1},
    input_columns = ['i'],
    remove_columns = ['i']
)   
list(ds2)
```

produces

```python
[{'i': 1}, {'i': 2}]
```

as expected. If the line that converts `ds1` to iterable is uncommented so that the `ds2` is a map of an `IterableDataset`, the result is 

```python
[{},{}]
```

I expected the output to be the same as before. It seems that in the second case the removed column is not added back into the output.

The issue seems to be [here](https://github.com/huggingface/datasets/blob/6c6a82a573f946c4a81069f56446caed15cee9c2/src/datasets/iterable_dataset.py#L1093): the columns are removed after the mapping which is not what we want (or what the [documentation says](https://github.com/huggingface/datasets/blob/6c6a82a573f946c4a81069f56446caed15cee9c2/src/datasets/iterable_dataset.py#L2370)) because we want the columns removed from the transformed example but then added if the map produced them.

This is `datasets==3.2.0` and `python==3.10`



### Steps to reproduce the bug

see above

### Expected behavior

see above

### Environment info

see above

Comments:
Good catch ! Do you think you can open a PR to fix this issue ?
[Bug] Inconsistent behavior of data_files and data_dir in load_dataset method.

### Describe the bug

Inconsistent operation of data_files and data_dir in load_dataset method.


### Steps to reproduce the bug

# First
I have three files, named 'train.json', 'val.json', 'test.json'.
Each one has a simple dict `{text:'aaa'}`.
Their path are `/data/train.json`,  `/data/val.json`,  `/data/test.json`
I load dataset with `data_files` argument:
```py
files = [os.path.join('./data',file) for file in os.listdir('./data')]
ds = load_dataset(
            path='json',
            data_files=files,)
```
And I get:
```py
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 3
    })
})
```
However, If I load dataset with `data_dir` argument:
```py
ds = load_dataset(
            path='json',
            data_dir='./data',)
```
And I get:
```py
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 1
    })
    validation: Dataset({
        features: ['text'],
        num_rows: 1
    })
    test: Dataset({
        features: ['text'],
        num_rows: 1
    })
})
```
Two results are not the same. Their behaviors are not equal, even if the statement [here](https://github.com/huggingface/datasets/blob/d0c152a979d91cc34b605c0298aebc650ab7dd27/src/datasets/load.py#L1790) said that their behaviors are equal. 


# Second
If some filename include 'test' while others do not, `load_dataset` only return `test` dataset and others files are **abandoned**.
Given two files named `test.json` and `1.json`
Each one has a simple dict `{text:'aaa'}`.
I load the dataset using:
```py
ds = load_dataset(
            path='json',
            data_dir='./data',)
```
Only `test` is returned, `1.json` is missing:
```py
DatasetDict({
    test: Dataset({
        features: ['text'],
        num_rows: 1
    })
})
```
Things do not change even I manually set `split='train'`

 



### Expected behavior

1. Fix the above bugs.
2. Although the document says that load_dataset method will `Find which file goes into which split (e.g. train/test) based on file and directory names or on the YAML configuration`, I hope I can manually decide whether to do so. Sometimes users may accidentally put a `test` string in the filename but they just want a single `train` dataset. If the number of files in `data_dir` is huge, it's not easy to find out what cause the second situation metioned above.

### Environment info

datasets==3.2.0
Ubuntu18.84

Comments:
Hi ! `data_files` with a list is equivalent to `data_files={"train": data_files}` with a train test only.

When no split are specified, they are inferred based on file names, and files with no apparent split are ignored

Thanks for your reply!
`files with no apparent split are ignored`. Is there a option that I can choose to ignored it or not as I mention aboved? Thanks!

To include all the files, the best way is to pass `data_files` yourself. There is no option to disable split detection at the moment

Thanks! I hope you guys can consider adding this option in the future. :)
One or several metadata.jsonl were found, but not in the same directory or in a parent directory of

### Describe the bug

ImageFolder with metadata.jsonl error. I downloaded liuhaotian/LLaVA-CC3M-Pretrain-595K locally from Hugging Face. According to the tutorial in https://huggingface.co/docs/datasets/image_dataset#image-captioning, only put images.zip and metadata.jsonl containing information in the same folder. However, after loading, an error was reported: One or several metadata.jsonl were found, but not in the same directory or in a parent directory of.

The data in my jsonl file is as follows:

> {"id": "GCC_train_002448550", "file_name": "GCC_train_002448550.jpg", "conversations": [{"from": "human", "value": "<image>\nProvide a brief description of the given image."}, {"from": "gpt", "value": "a view of a city , where the flyover was proposed to reduce the increasing traffic on thursday ."}]}


### Steps to reproduce the bug

from datasets import load_dataset
image = load_dataset("imagefolder",data_dir='data/opensource_data')

### Expected behavior

success

### Environment info

datasets==3.2.0

Comments:
Hmmm I double checked in the source code and I found a contradiction: in the current implementation the metadata file is ignored if it's not in the same archive as the zip image somehow:

https://github.com/huggingface/datasets/blob/caa705e8bf4bedf1a956f48b545283b2ca14170a/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L352-L353

in the tests suite the metadata file is placed inside the archive:

https://github.com/huggingface/datasets/blob/caa705e8bf4bedf1a956f48b545283b2ca14170a/tests/packaged_modules/test_imagefolder.py#L223-L223

Thanks for reporting this issue, it seems the documentation is wrong and we never implemented the support for zip + metadata outside zip. We might rewrite part of this code soon though to make it more flexible, it can be a good occasion to fix this. In the meantime feel free to open a PR to fix the documentation if you'd like
Clarify documentation or Create DatasetCard

### Feature request

I noticed that you can use a Model Card instead of a Dataset Card when pushing a dataset to the Hub, but this isn’t clearly mentioned in [the docs.](https://huggingface.co/docs/datasets/dataset_card)

- Update the docs to clarify that a Model Card can work for datasets too.
- It might be worth creating a dedicated DatasetCard module, similar to the ModelCard module, for consistency and better support.

Not sure if this belongs here or on the [Hub repo](https://github.com/huggingface/huggingface_hub), but thought I’d bring it up!

### Motivation

I just spent an hour like on [this issue](https://github.com/huggingface/trl/pull/2491) trying to create a `DatasetCard` for a script.

### Your contribution

might later

Comments:
Too many open files: '/root/.cache/huggingface/token'

### Describe the bug

I ran this code:
```
from datasets import load_dataset
dataset = load_dataset("common-canvas/commoncatalog-cc-by", cache_dir="/datadrive/datasets/cc", num_proc=1000)
```

And got this error.
Before it was some other file though (lie something...incomplete)

runnting
```
ulimit -n 8192
```
did not help at all.

### Steps to reproduce the bug

Run the code i sent

### Expected behavior

Should be no errors 

### Environment info

linux, jupyter lab.

Comments:
TypeError: Value.__init__() missing 1 required positional argument: 'dtype'

### Describe the bug

ds = load_dataset(
    "./xxx.py",
    name="default",
    split="train",
)

The datasets does not support debugging locally anymore...

### Steps to reproduce the bug

```
from datasets import load_dataset

ds = load_dataset(
    "./repo.py",
    name="default",
    split="train",
)

for item in ds:
    print(item)
```

It works fine for "username/repo", but it does not work for "./repo.py" when debugging locally...
Running above code template will report TypeError: Value.__init__() missing 1 required positional argument: 'dtype'

### Expected behavior

fix this bug

### Environment info

python 3.10 datasets==2.21

Comments:
same error 
```
data = load_dataset('/opt/deepseek_R1_finetune/hf_datasets/openai/gsm8k', 'main')[split] 
```

> same error
> 
> ```
> data = load_dataset('/opt/deepseek_R1_finetune/hf_datasets/openai/gsm8k', 'main')[split] 
> ```

https://github.com/huggingface/open-r1/issues/204  this help me

Solved by delete `dataset_infos.json` file in dataset dir, or you can transfer datasets from Hugginface to Modelscope by [hf-ms-transfer](https://github.com/wa008/hf-ms-transfer), which will solve this problem by default.
.map() is not caching and ram goes OOM

### Describe the bug

Im trying to run a fairly simple map that is converting a dataset into numpy arrays. however, it just piles up on memory and doesnt write to disk. Ive tried multiple cache techniques such as specifying the cache dir, setting max mem, +++ but none seem to work. What am I missing here?

### Steps to reproduce the bug

```
from pydub import AudioSegment
import io
import base64
import numpy as np
import os
CACHE_PATH = "/mnt/extdisk/cache" # "/root/.cache/huggingface/"# 
os.environ["HF_HOME"] = CACHE_PATH
import datasets
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# Create a handler for Jupyter notebook
handler = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)

#datasets.config.IN_MEMORY_MAX_SIZE= 1000#*(2**30) #50 gb
print(datasets.config.HF_CACHE_HOME)
print(datasets.config.HF_DATASETS_CACHE)
# Decode the base64 string into bytes
def convert_mp3_to_audio_segment(example):
    """
    example = ds['train'][0]
    """
    try:
        audio_data_bytes = base64.b64decode(example['audio'])
        # Use pydub to load the MP3 audio from the decoded bytes
        audio_segment = AudioSegment.from_file(io.BytesIO(audio_data_bytes), format="mp3")
        # Resample to 24_000
        audio_segment = audio_segment.set_frame_rate(24_000)
        audio = {'sampling_rate' : audio_segment.frame_rate,
        'array' : np.array(audio_segment.get_array_of_samples(), dtype="float")}
        del audio_segment
        duration = len(audio['array']) / audio['sampling_rate']
    except Exception as e:
        logger.warning(f"Failed to convert audio for {example['id']}. Error: {e}")
        audio = {'sampling_rate' : 0,
        'array' : np.array([]), duration : 0}
    return {'audio' : audio, 'duration' : duration}

ds = datasets.load_dataset("NbAiLab/nb_distil_speech_noconcat_stortinget", cache_dir=CACHE_PATH, keep_in_memory=False)

#%%
num_proc=32
ds_processed = (
    ds
    #.select(range(10))
    .map(convert_mp3_to_audio_segment, num_proc=num_proc, desc="Converting mp3 to audio segment") #, cache_file_name=f"{CACHE_PATH}/stortinget_audio" # , cache_file_name="test"
)
```

### Expected behavior

the map should write to disk

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-6.8.0-45-generic-x86_64-with-glibc2.39
- Python version: 3.12.7
- `huggingface_hub` version: 0.26.3
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments:
I have the same issue - any update on this?
Remove upper bound for fsspec

### Describe the bug

As also raised by @cyyever in https://github.com/huggingface/datasets/pull/7296 and @NeilGirdhar in https://github.com/huggingface/datasets/commit/d5468836fe94e8be1ae093397dd43d4a2503b926#commitcomment-140952162 , `datasets` has a problematic version constraint on `fsspec`.

In our case this causes (unnecessary?) troubles due to a race condition bug in that version of the corresponding `gcsfs` plugin, that causes deadlocks: https://github.com/fsspec/gcsfs/pull/643

We just use a version override to ignore the constraint from `datasets`, but imho the version constraint could just be removed in the first place?

The last few PRs bumping the upper bound were basically uneventful:

* https://github.com/huggingface/datasets/pull/7219
* https://github.com/huggingface/datasets/pull/6921
* https://github.com/huggingface/datasets/pull/6747

### Steps to reproduce the bug

-

### Expected behavior

Installing `fsspec>=2024.10.0` along `datasets` should be possible without overwriting constraints.

### Environment info

All recent datasets versions

Comments:
Unfortunately `fsspec` versioning allows breaking changes across version and there is no way we can keep it without constrains at the moment. It already broke `datasets` once in the past. Maybe one day once `fsspec` decides on a stable and future proof API but I don't think this will happen anytime soon

edit: bumped to 2024.10.0 in https://github.com/huggingface/datasets/pull/7352
Introduce pdf support (#7318)

First implementation of the Pdf feature to support pdfs (#7318) . Using [pdfplumber](https://github.com/jsvine/pdfplumber?tab=readme-ov-file#python-library) as the default library to work with pdfs. 
@lhoestq and @AndreaFrancis 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7325). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @AndreaFrancis and @lhoestq ! Thanks for looking at the code and for all the changes and suggestions. I have worked on all your suggestions. I need to work a bit more on the tests, but I created a first version and uploaded a simple pdf to use for testing. The pdf has a couple of pages some of which have images, figures, and tables (this is exactly the pdfs for which I think keeping the pdf format is very interesting because they have multiple types of content). Will try to finish the tests as soon as possible.

(I updated the CI a bit, I took the liberty to update your branch and re-run it on your PR @yabramuvdi)
Unexpected cache behaviour using load_dataset

### Describe the bug

Following the (Cache management)[https://huggingface.co/docs/datasets/en/cache] docu and previous behaviour from datasets version 2.18.0, one is able to change the cache directory. Previously, all downloaded/extracted/etc files were found in this folder. As i have recently update to the latest version this is not the case anymore. Downloaded files are stored in `~/.cache/huggingface/hub`.
Providing the `cache_dir` argument in `load_dataset` the cache directory is created and there are some files but the bulk is still in `~/.cache/huggingface/hub`.

I believe this could be solved by adding the cache_dir argument [here](https://github.com/huggingface/datasets/blob/fdda5585ab18ea1292547f36c969d12c408ab842/src/datasets/utils/file_utils.py#L188)


### Steps to reproduce the bug

For example using https://huggingface.co/datasets/ashraq/esc50:
```python
from datasets import load_dataset
ds = load_dataset("ashraq/esc50", "default", cache_dir="~/custom/cache/path/esc50")
```


### Expected behavior

I would expect the bulk of files related to the dataset to be stored somewhere in `~/custom/cache/path/esc50`, but it seems they are in `~/.cache/huggingface/hub/datasets--ashraq--esc50`. 

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.14.0-503.15.1.el9_5.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.14
- `huggingface_hub` version: 0.26.5
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1

Comments:
Hi ! Since `datasets` 3.x,  the `datasets` specific files are in `cache_dir=` and the HF files are cached using `huggingface_hub` and you can set its cache directory using the `HF_HOME` environment variable.

They are independent, for example you can delete the Hub cache (containing downloaded files) but still reload your cached datasets from the `datasets` cache (containing prepared datasets in Arrow format)
ImportError: cannot import name 'set_caching_enabled' from 'datasets' 

### Describe the bug

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/local/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/home/Medusa/axolotl/src/axolotl/cli/__init__.py", line 23, in <module>
    from axolotl.train import TrainDatasetMeta
  File "/home/Medusa/axolotl/src/axolotl/train.py", line 23, in <module>
    from axolotl.utils.trainer import setup_trainer
  File "/home/Medusa/axolotl/src/axolotl/utils/trainer.py", line 13, in <module>
    from datasets import set_caching_enabled
ImportError: cannot import name 'set_caching_enabled' from 'datasets' (/usr/local/lib/python3.10/site-packages/datasets/__init__.py)

### Steps to reproduce the bug

1、axolotl
2、accelerate launch -m axolotl.cli.train examples/medusa/qwen_lora_stage1.yml

### Expected behavior

enable datasets

### Environment info

python3.10

Comments:
pip install datasets==2.18.0

Hi ! I think you need to update axolotl