noamaanMulla-03/datasets-issues huggingface dataset:
`map()` stops every 1000 steps

### Describe the bug

I am passing the following `prepare_dataset` function to `Dataset.map` (code is inspired from [here](https://github.com/huggingface/community-events/blob/main/whisper-fine-tuning-event/run_speech_recognition_seq2seq_streaming.py#L454))
```python3
def prepare_dataset(batch):
    # load and resample audio data from 48 to 16kHz
    audio = batch["audio"]

    # compute log-Mel input features from input audio array 
    batch["input_features"] = feature_extractor(audio["array"], sampling_rate=audio["sampling_rate"]).input_features[0]

    # encode target text to label ids 
    batch["labels"] = tokenizer(batch[text_column]).input_ids
    return batch
...
train_ds = train_ds.map(prepare_dataset)
```

Here is the exact code I am running https://github.com/bayartsogt-ya/whisper-multiple-hf-datasets/blob/main/train.py#L70-L71

It starts using all the cores (I am not sure why because I did not pass `num_proc`)
then progress bar stops at every 1k steps. (starts using a single core)
then come back to using all the cores again.

link to [screen record](https://youtu.be/jPQpQQGp6Gc)

Can someone explain this process and maybe provide a way to improve this pipeline? cc: @lhoestq 

### Steps to reproduce the bug

1. load the dataset
2. create a Whisper processor
3. create a `prepare_dataset` function
4. pass the function to `dataset.map(prepare_dataset)`

### Expected behavior

- Use a single core per a function
- not to stop at some point?

### Environment info

- `datasets` version: 2.7.1.dev0
- Platform: Linux-5.4.0-109-generic-x86_64-with-glibc2.27
- Python version: 3.8.10
- PyArrow version: 10.0.1
- Pandas version: 1.5.2


Hi !

> It starts using all the cores (I am not sure why because I did not pass num_proc)

The tokenizer uses Rust code that is multithreaded. And maybe the `feature_extractor` might run some things in parallel as well - but I'm not super familiar with its internals.

> then progress bar stops at every 1k steps. (starts using a single core)

Every 1000 examples we flush the processed examples to disk. It is this way because Arrow is a columnar format: you must write data chunk by chunk. The processing in on hold while writing right now - maybe this can be improved in the future.

Hi @lhoestq 
Thanks for the explanation! it was so helpful! Let me check why `feature_extractor` is running on multiple cpus.

Hey @lhoestq, any news about this flush operation ? This really really slow down most of my .map and .filter operation. I'm sometime taking 10x more time waiting for the flush than actually processing the 1k items. My current workarround is to set `writer_batch_size=len(dataset)`.
Lazy loading of environment variables

### Describe the bug

Loading a `.env` file after an `import datasets` call does not correctly use the environment variables. 

This is due the fact that environment variables are read at import time:
https://github.com/huggingface/datasets/blob/de062f0552a810c52077543c1169c38c1f0c53fc/src/datasets/config.py#L155C1-L155C80

### Steps to reproduce the bug

```bash
# make tmp dir
mkdir -p /tmp/debug-env
# make .env file
echo HF_HOME=/tmp/debug-env/data > /tmp/debug-env/.env
# first load dotenv, downloads to /tmp/debug-env/data
uv run --with datasets,python-dotenv python3 -c \
'import dotenv; dotenv.load_dotenv("/tmp/debug-env/.env"); import datasets; datasets.load_dataset("Anthropic/hh-rlhf")' 
# first import datasets, downloads to `~/.cache/huggingface`
uv run --with datasets,python-dotenv python3 -c \
'import datasets; import dotenv; dotenv.load_dotenv("/tmp/debug-env/.env"); datasets.load_dataset("Anthropic/hh-rlhf")' 

```

### Expected behavior

I expect that setting environment variables with something like this:

```python3
if __name__ == "__main__":
    load_dotenv()
    main()
```
works correctly. 

### Environment info

    "datasets>=3.3.0",



Many python packages out there, including `huggingface_hub`, do load the environment variables on import.
You should `load_dotenv()` before importing the libraries.

For example you can move all you imports inside your `main()` function
Performance regression in `dataset.filter`

### Describe the bug

We're filtering dataset of ~1M (small-ish) records. At some point in the code we do `dataset.filter`, before (including 3.2.0) it was taking couple of seconds, and now it takes 4 hours.

We use 16 threads/workers, and stack trace at them look as follows:
```
Traceback (most recent call last):
  File "/python/lib/python3.12/site-packages/multiprocess/process.py", line 314, in _bootstrap
    self.run()
  File "/python/lib/python3.12/site-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/python/lib/python3.12/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/python/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 678, in _write_generator_to_queue
    for i, result in enumerate(func(**kwargs)):
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3511, in _map_single
    for i, batch in iter_outputs(shard_iterable):
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3461, in iter_outputs
    yield i, apply_function(example, i, offset=offset)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3390, in apply_function
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 6416, in get_indices_from_mask_function
    indices_array = indices_mapping.column(0).take(indices_array)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/table.pxi", line 1079, in pyarrow.lib.ChunkedArray.take
  File "/python/lib/python3.12/site-packages/pyarrow/compute.py", line 458, in take
    def take(data, indices, *, boundscheck=True, memory_pool=None):
```

### Steps to reproduce the bug

1. Save dataset of 1M records in arrow
2. Filter it with 16 threads
3. Watch it take too long

### Expected behavior

Filtering done fast

### Environment info

datasets 3.3.0, python 3.12


Thanks for reporting, I'll fix the regression today

I just released `datasets` 3.3.1 with a fix, let me know if it's good now :)

@lhoestq it fixed the issue.

This was (very) fast, thank you very much!
Index Error Invalid Ket is out of bounds for size 0 for code-search-net/code_search_net dataset

### Describe the bug

I am trying to do model pruning on sentence-transformers/all-mini-L6-v2 for the code-search-net/code_search_net dataset using INCTrainer class 
However I am getting below error

```
raise IndexError(f"Invalid Key: {key is our of bounds for size {size}")
IndexError: Invalid key: 1840208 is out of bounds for size 0
```

### Steps to reproduce the bug

Model pruning on the above dataset using the below guide 

https://huggingface.co/docs/optimum/en/intel/neural_compressor/optimization#pruning
### Expected behavior

The modsl should be successfully pruned 

### Environment info

Torch version: 2.4.1
Python version: 3.8.10
Synchronize parameters for various datasets

### Describe the bug

[IterableDatasetDict](https://huggingface.co/docs/datasets/v3.2.0/en/package_reference/main_classes#datasets.IterableDatasetDict.map) map function is missing the `desc` parameter. You can see the equivalent map function for [Dataset here](https://huggingface.co/docs/datasets/v3.2.0/en/package_reference/main_classes#datasets.Dataset.map).

There might be other parameters missing - I haven't checked.

### Steps to reproduce the bug

from datasets import Dataset, IterableDataset, IterableDatasetDict

ds = IterableDatasetDict({"train": Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=3), 
                          "validate": Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=3)})

for d in ds["train"]:
    print(d)

ds = ds.map(lambda x: {k: v+1 for k, v in x.items()}, desc="increment")

for d in ds["train"]:
    print(d)

### Expected behavior

The description parameter should be available for all datasets (or none). 

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.11.11
- `huggingface_hub` version: 0.28.1
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.9.0


Hi ! the `desc` parameter is only available for Dataset / DatasetDict for the progress bar of `map()``

Since IterableDataset only runs the map functions when you iterate over the dataset, there is no progress bar and `desc` is useless. We could still add the argument for parity but it wouldn't be used for anything

I think you should add it. It doesn't hurt. The reason I ran into it was because I re-wrote a pipeline to use either a stream or a fully loaded dataset. Of course I can simply remove it but it is nice to have on the memory loaded dataset.
Attempt to fix multiprocessing hang by closing and joining the pool before termination

https://github.com/huggingface/datasets/issues/6393 has plagued me on and off for a very long time. I have had various workarounds (one time combining two filter calls into one filter call removed the issue, another time making rank 0 go first resolved a cache race condition, one time i think upgrading the version of something resolved it). I don't know hf datasets well enough to fully understand the root cause, but I _think_ this PR fixes it.

Evidence: I have an LLM Foundry training yaml/script (datasets version 3.2.0) that results in a hang ~1/10 times (for a baseline for this testing, it was 2/36 runs that hung). I also reran with the latest datasets version (3.3.1) and got 4/36 hung. Installing datasets from this PR, I was able to successful run the script 144 times without a hang occurring. Assuming the base probability is 1/10, this should be more than enough times to have confidence it works.

After adding some logging, I could see that the code hung during the __exit__ of the mp pool context manager, after all shards had been processed, and the tqdm context manager had exited.

My best explanation: When multiprocessing pool __exit__ is called, it calls pool.terminate, which forcefully exits all the processes (and calls code related to this that I haven't looked at closely). I'm guessing this forceful termination has a bad interaction with some multithreading/multiprocessing that hf datasets does. If we instead call pool.close and pool.join before the pool.terminate happens, perhaps whatever that bad interaction is is able to complete gracefully, and then terminate call proceeds without issue.

If this PR seems good to you, I'd be very appreciative if you were able to do a patch release including it. Thank you!

@lhoestq 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7411). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the fix! We have been affected by this a lot when we try to use LLM Foundry with larger multimodal ICL datasets. 

@lorabit110 are you able to test it out for your case as well? Would be great to get a second validation that it actually fixes the issue. Thanks!
Documentation on multiple media files of the same type with WebDataset

The [current documentation](https://huggingface.co/docs/datasets/en/video_dataset) on a creating a video dataset includes only examples with one media file and one json. It would be useful to have examples where multiple files of the same type are included. For example, in a sign language dataset, you may have a base video and a video annotation of the extracted pose. According to the WebDataset documentation, this should be able to be done with period separated filenames. For example:

```e39871fd9fd74f55.base.mp4
e39871fd9fd74f55.pose.mp4
e39871fd9fd74f55.json
f18b91585c4d3f3e.base.mp4
f18b91585c4d3f3e.pose.mp4
f18b91585c4d3f3e.json
... 
```

If you can confirm that this method of including multiple media files works with huggingface datasets and include an example in the documentation, I'd appreciate it.


Yes this is correct and it works with huggingface datasets as well ! Feel free to include an example here: https://github.com/huggingface/datasets/blob/main/docs/source/video_dataset.mdx
Shard Dataset at specific indices

I have a dataset of sequences, where each example in the sequence is a separate row in the dataset (similar to LeRobotDataset). When running `Dataset.save_to_disk` how can I provide indices where it's possible to shard the dataset such that no episode spans more than 1 shard. Consequently, when I run `Dataset.load_from_disk`, how can I load just a subset of the shards to save memory and time on different ranks?

I guess an alternative to this would be, given a loaded `Dataset`, how can I run `Dataset.shard` such that sharding doesn't split any episode across shards?


Hi ! if it's an option I'd suggest to have one sequence per row instead.

Otherwise you'd have to make your own save/load mechanism

Saving one sequence per row is very difficult and heavy and makes all the optimizations pointless. How would a custom save/load mechanism look like?

You can use `pyarrow` for example to save/load individual arrow or parquet files and control what they contain
Import order crashes script execution

### Describe the bug

Hello,

I'm trying to convert an HF dataset into a TFRecord so I'm importing `tensorflow` and `datasets` to do so.
Depending in what order I'm importing those librairies, my code hangs forever and is unkillable (CTRL+C doesn't work, I need to kill my shell entirely).

Thank you for your help
🙏 

### Steps to reproduce the bug

If you run the following script, this will hang forever : 
```python
import tensorflow as tf
import datasets

dataset = datasets.load_dataset("imagenet-1k", split="validation", streaming=True)
print(next(iter(dataset)))
```

however running the following will work fine (I just changed the order of the imports) :
```python
import datasets
import tensorflow as tf

dataset = datasets.load_dataset("imagenet-1k", split="validation", streaming=True)
print(next(iter(dataset)))
```

### Expected behavior

I'm expecting the script to reach the end and my case print the content of the first item in the dataset 
```
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=408x500 at 0x70C646A03110>, 'label': 91}
```

### Environment info

```
$ datasets-cli env
- `datasets` version: 3.3.2
- Platform: Linux-6.8.0-1017-aws-x86_64-with-glibc2.35
- Python version: 3.11.7
- `huggingface_hub` version: 0.29.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
```

I'm also using `tensorflow==2.18.0`.
load_dataset: TypeError: 'NoneType' object is not callable, on local dataset filename changes

### Describe the bug

I've adapted Databrick's [train_dolly.py](/databrickslabs/dolly/blob/master/train_dolly.py) to train using a local dataset, which has been working. Upon changing the filenames of the `.json` & `.py` files in my local dataset directory, `dataset = load_dataset(path_or_dataset)["train"]` throws the error:

```python
2023-04-30 09:10:52 INFO [training.trainer] Loading dataset from dushowxa-characters
Traceback (most recent call last):
  File "/data/dushowxa-dolly/train_dushowxa.py", line 26, in <module>
    load_training_dataset()
  File "/data/dushowxa-dolly/training/trainer.py", line 89, in load_training_dataset
    dataset = load_dataset(path_or_dataset)["train"]
  File "/data/dushowxa-dolly/.venv/lib/python3.10/site-packages/datasets/load.py", line 1773, in load_dataset
    builder_instance = load_dataset_builder(
  File "/data/dushowxa-dolly/.venv/lib/python3.10/site-packages/datasets/load.py", line 1528, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
TypeError: 'NoneType' object is not callable
```
The local dataset filenames were of the form `dushowxa-characters/expanse-dushowxa-characters.json` and are now of the form `dushowxa-characters/dushowxa-characters.json` (the word `expanse-` was removed from the filenames). Is this perhaps a dataset caching issue?

I have attempted to manually clear caches, but to no effect:

```sh
rm -rfv ~/.cache/huggingface/datasets/*
rm -rfv ~/.cache/huggingface/modules/* 
```

### Steps to reproduce the bug

Run `python3 train_dushowxa.py` (adapted from Databrick's [train_dolly.py](/databrickslabs/dolly/blob/master/train_dolly.py)).

### Expected behavior

Training succeeds as before local dataset filenames were changed.

### Environment info

Ubuntu 22.04, Python 3.10.6, venv
```python
accelerate>=0.16.0,<1
click>=8.0.4,<9
datasets>=2.10.0,<3
deepspeed>=0.9.0,<1
transformers[torch]>=4.28.1,<5
langchain>=0.0.139
```


This error means a `DatasetBuilder` subclass that generates the dataset could not be found inside the script, so make sure `dushowxa-characters/dushowxa-characters.py `is a valid dataset script (assuming `path_or_dataset` is `dushowxa-characters`)

Also, we should improve the error to make it more obvious what the problem is.

from datasets import load_dataset
lcb_codegen = load_dataset("livecodebench/code_generation_lite", version_tag="release_v2")

or  
configs = get_dataset_config_names("livecodebench/code_generation_lite", trust_remote_code=True)

**both  error:**
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/workspace/miniconda/envs/grpo/lib/python3.10/site-packages/datasets/load.py", line 2131, in load_dataset
    builder_instance = load_dataset_builder(
  File "/workspace/miniconda/envs/grpo/lib/python3.10/site-packages/datasets/load.py", line 1888, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
TypeError: 'NoneType' object is not callable
ds.map(f, num_proc=10) is slower than df.apply

### Describe the bug

pandas columns: song_id, song_name
ds = Dataset.from_pandas(df)

def has_cover(song_name):
    if song_name is None or pd.isna(song_name):
        return False
    return 'cover' in song_name.lower()

df['has_cover'] = df.song_name.progress_apply(has_cover)
ds = ds.map(lambda x: {'has_cover': has_cover(x['song_name'])}, num_proc=10)

time cost: 
1. df.apply: 100%|██████████| 12500592/12500592 [00:13<00:00, 959825.47it/s]
2. ds.map: Map (num_proc=10):  31%
 3899028/12500592 [00:28<00:38, 222532.89 examples/s]

### Steps to reproduce the bug

pandas columns: song_id, song_name
ds = Dataset.from_pandas(df)

def has_cover(song_name):
    if song_name is None or pd.isna(song_name):
        return False
    return 'cover' in song_name.lower()

df['has_cover'] = df.song_name.progress_apply(has_cover)
ds = ds.map(lambda x: {'has_cover': has_cover(x['song_name'])}, num_proc=10)

### Expected behavior

ds.map is ~num_proc faster than df.apply

### Environment info

pandas: 2.2.2
datasets: 2.19.1


Hi ! `map()` reads all the columns and writes the resulting dataset with all the columns as well, while df.column_name.apply only reads and writes one column and does it in memory. So this is speed difference is actually expected.

Moreover using multiprocessing on a dataset that lives in memory (from_pandas uses the same in-memory data as the pandas DataFrame while load_dataset or from_generator load from disk) requires to copy the data to each subprocess which can also be slow. Data loaded from disk don't need to be copied though since they work as a form of shared memory thanks to memory mapping.

However you can make you map() call much faster by making it read and write only the column you want:

```python
has_cover_ds = ds.map(lambda song_name: {'has_cover': has_cover(song_name)}, input_columns=["song_name"], remove_columns=ds.column_names)  # outputs a dataset with 1 column
ds = ds.concatenate_datasets([ds, has_cover_ds], axis=1)
```

and if your dataset is loaded from disk you can pass num_proc=10 and get a nice speed up as well (no need to copy the data to subprocesses)

Isn't there a way to do memory mapping with the in-memory dataset without saving it to disk?

Maybe saving it to a memory-mapped filesystem? It'd be like a trick to make datasets save to "disk" but actually it's memory. But it feels like there should be a better "automatic" way provided by `datasets`.
Make IterableDataset (optionally) resumable

### What does this PR do?
This PR introduces a new `stateful` option to the `dataset.shuffle` method, which defaults to `False`. 
When enabled, this option allows for resumable shuffling of `IterableDataset` instances, albeit with some additional memory overhead.

Key points:
* All tests have passed
* Docstrings have been updated to reflect the new functionality

I'm very looking forward to receiving feedback on this implementation! @lhoestq


@lhoestq Hi again~ Just circling back on this
Wondering if there’s anything I can do to help move this forward. 🤗 
Thanks!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7385). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Error in code "Time to slice and dice" from course "NLP Course"

### Describe the bug

When we execute code 
```
frequencies = (
    train_df["condition"]
    .value_counts()
    .to_frame()
    .reset_index()
    .rename(columns={"index": "condition", "condition": "frequency"})
)
frequencies.head()
```
answer should be like this

condition      | frequency
birth control | 27655
depression    | 8023
acne              | 5209
anxiety          | 4991
pain               | 4744

but he is different

frequency      | count
birth control  | 27655
depression    | 8023
acne              | 5209
anxiety          | 4991
pain               | 4744

this is not correct, correct code
```
frequencies = (
    train_df["condition"]
    .value_counts()
    .to_frame()
    .reset_index()
    .rename(columns={"index": "condition", "count": "frequency"})
)
````

### Steps to reproduce the bug

```
frequencies = (
train_df["condition"]
.value_counts()
.to_frame()
.reset_index()
.rename(columns={"index": "condition", "condition": "frequency"})
)
frequencies.head()
```

### Expected behavior


condition      | frequency
birth control | 27655
depression    | 8023
acne              | 5209
anxiety          | 4991
pain               | 4744

### Environment info

Google Colab


You should open an issue in the NLP course website / github page. I'm closing this issue if you don't mind

ok, i don't mind, i'll mark the error there