AniketTec/github-issues huggingface dataset:
Title: Memory leak / Large memory usage with num_workers = 0 and numerous dataset within DatasetDict
State: open

### Describe the bug

Hi team, first off, I love the datasets library! 🥰

I'm encountering a potential memory leak / increasing memory usage when training a model on a very large DatasetDict.

Setup: I have a DatasetDict containing 362 distinct datasets, which sum up to ~2.8 billion rows.

Training Task: I'm performing contrastive learning with SentenceTransformer and Accelerate on a single node with 4 H100, which requires me to sample from only one dataset at a time.

Training Loop: At each training step, I sample ~16,000 examples from a single dataset, and then switch to a different dataset for the next step. I iterate through all 362 datasets this way. 

Problem: The process's memory usage continuously increases over time, eventually causing a stale status where GPUs would stop working. It seems memory from previously sampled datasets isn't being released. I've set num_workers=0 for all experiments.

Chart 1: Standard DatasetDict The memory usage grows steadily until it make the training stale (RSS memory) <img width="773" height="719" alt="Image" src="https://github.com/user-attachments/assets/6606bef5-1153-4f2d-bf08-82da249d6e8d" />

Chart 2: IterableDatasetDict I also tried  to use IterableDatasetDict and IterableDataset. The memory curve is "smoother," but the result is the same: it grows indefinitely and the training become stale. <img width="339" height="705" alt="Image" src="https://github.com/user-attachments/assets/ee90c1a1-6c3b-4135-9edc-90955cb1695a" />

Any feedback or guidance on how to manage this memory would be greatly appreciated!

### Steps to reproduce the bug

WIP, I'll add some code that manage to reproduce this error, but not straightforward. 

### Expected behavior

The memory usage should remain relatively constant or plateau after a few steps. Memory used for sampling one dataset should be released before or during the sampling of the next dataset.

### Environment info

Python: 3.12
Datasets: 4.3.0
SentenceTransformers: 5.1.1

Comments:
Thanks for the report, this is possibly related #7722 and #7694.

Could you pls provide steps to reproduce this?

To overcome this issue right now I did simply reduce the size of the dataset and ended up running a for loop (my training has now a constant learning rate schedule). From what I understood, and I don't know if it's possible, the solution would be to tell the backend of `datasets` to leave x% of the memory free  (including memory mapping). Can't release the data right now but I will and then allow to reproduce this issue. But it will involve to have some free TB of disk

@raphaelsty thanks for coming back to this. I assume you are running in streaming mode? That should prevent these errors but it looks like more people than just you have this problem, so a clearly reproducing example (including data + code) is highly appreciated.

This could be related to this issue: https://github.com/huggingface/datasets/issues/4883 in which we discussed how RSS and memory mapping works and depends on the OS and disk type.
Title: Fix ArrowInvalid for large variables
State: open

Fixes #7821

In addition to the solution proposed in the issue, I encountered we also need to support 64bit types when writing using the `RecordBatchStreamWriter`. 

Not sure if we want to create such large objects in the CI, but this is the only way to test that the issue is fixed, therefore I added the unit test.

Comments:
@lhoestq : from what I see in the CI logs ( tests/test_arrow_dataset.py::test_map_int32_overflow 
/opt/hostedtoolcache/Python/3.9.23/x64/lib/python3.9/site-packages/multiprocess/resource_tracker.py:219: UserWarning: resource_tracker: There appear to be 24 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '), see [here](https://github.com/huggingface/datasets/actions/runs/18720027796/job/53591652242#step:13:6530), the test crashes probably since it consumes too much memory, it works locally for me though. 
  
  Shall we keep the test, remove ti, or skip it, or is there a specific pytest flag you have for slow/memory-consuming tests I can add, so that it gets skipped automatically on the CI but might be run locally?
  
  EDIT: I added the `high_memory` marker (see [here](https://github.com/huggingface/datasets/pull/7825/files#diff-c480dc36aca482db963c43ee66ba21705111f62f826352c96558ef5ed1619791R4789)), so this will not be run in the CI, which is not ideal but I guess the way if we want to keep the test
Title: Fix batch_size default description in to_polars docstrings
State: closed

Fix batch_size default description in `to_polars` docstrings.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7824). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Fix random seed on shuffle and interleave_datasets
State: closed

closes #7567

Add `shift_rngs` method to `ExamplesIterable` that is called directly after sharding. If a generator is available (not the case for all subclasses) we update the seed of the generator by shifting by the worker_id.

~This is just the fix for `shuffle`, in the corresponding issue `interleave_datasets` is mentioned as well, which won't be fixed with this approach.~

EDIT: This is a fix for `shuffle` and `interleave_datasets`. Adding recursivity to `shift_rngs` solved `interleave_datasets` as well. Not sure though if this is completely safe or if we could destroy something with that. I don't think so but could be wrong and appreciate some guidance from the maintainers. I also checked, on a single_worker we are always handing over `index=0` so that case preserves the seed the user specified. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7823). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Cool ! To avoid unwanted side effects it could be implemented for every class instead of using `hasattr`, and also return a new instance instead of modifying it in place. Some other functions work like that already like `shuffle_data_sources` and `shard_data_sources`.

Returning a new object is actually quite important otherwise iterating on the dataset multiple times would shift the RNGs every time

> Cool ! To avoid unwanted side effects it could be implemented for every class instead of using `hasattr`, and also return a new instance instead of modifying it in place. Some other functions work like that already like `shuffle_data_sources` and `shard_data_sources`.
> 
> Returning a new object is actually quite important otherwise iterating on the dataset multiple times would shift the RNGs every time

Thanks for the review. 

I managed to return instances of `_BaseExamplesIterable` instead of modifying inplace and this shifts the seed correctly even in the case of multiple iterations (see additional test). But I kept the `hasattr` for the reason that we have chained ex_iterables (so an ex_iterable can have other ex_iterables) and for the child ex_iterable it is not clear if they do have a generator attribute (e.g. `ArrowExamplesIterable`, `RebatchedArrowExamplesIterable`, `SelectColumnsIterable`, `StepExamplesIterable`, `SkipExamplesIterable`, `RepeatExamplesIterable` all don't have that). We could implement a generic method `shift_rngs` on the baseclass and then just return the ex_iterable and get rid of `hasattr` but I'd prefer the `hasattr` solution here. Let me now what you think about that.
Title: Keep hffs cache in workers when streaming
State: closed

(and also reorder the hffs args to improve caching)

When using `DataLoader(iterable_dataset, num_workers=...)` the dataset is pickled and passed to the worker. However previously the resulting dataset would be in a process with an empty hffs cache. By keeping the cache attached to `IterableDataset`, the cached hffs instances are pickled with the dataset and re-populates the cache in the DataLoader workers

this requires https://github.com/huggingface/huggingface_hub/pull/3443 to work effectively though, otherwise the unpickled hffs cache would start empty


cc @andimarafioti @ltmeyer

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7820). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Cannot download opus dataset
State: open

When I tried to download opus_books using:

from datasets import load_dataset
dataset = load_dataset("Helsinki-NLP/opus_books")

I got the following errors:
FileNotFoundError: Couldn't find any data file at /workspace/Helsinki-NLP/opus_books. Couldn't find 'Helsinki-NLP/opus_books' on the Hugging Face Hub either: LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

I also tried:
dataset = load_dataset("opus_books", "en-zh")

and the errors remain the same. However, I can download "mlabonne/FineTome-100k" successfully. 
My datasets is version 4.2.0

Any clues? Big thanks.

Comments:
Hi ! it seems "en-zh" doesn't exist for this dataset

You can see the list of subsets here: https://huggingface.co/datasets/Helsinki-NLP/opus_books
Title: train_test_split and stratify breaks with Numpy 2.0
State: closed

### Describe the bug

As stated in the title, since Numpy changed in version >2.0 with copy, the stratify parameters break.

e.g. `all_dataset.train_test_split(test_size=0.2,stratify_by_column="label")` returns a Numpy error.

It works if you downgrade Numpy to a version lower than 2.0.

### Steps to reproduce the bug

1. Numpy > 2.0
2. `all_dataset.train_test_split(test_size=0.2,stratify_by_column="label")`

### Expected behavior

It returns a stratified split as per the results of Numpy < 2.0

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-6.8.0-85-generic-x86_64-with-glibc2.35
- Python version: 3.13.7
- Huggingface_hub version: 0.34.4
- PyArrow version: 19.0.0
- Pandas version: 2.3.2


Comments:
I can't reproduce this. Could you pls provide an example with a public dataset/artificial dataset and show how you loaded that?

This works for me:

```python
import numpy as np
from datasets import Dataset, Features, ClassLabel, Value

data = {"text": [f"sample_{i}" for i in range(100)], "label": [i % 3 for i in range(100)]}
features = Features({"text": Value("string"),
                     "label": ClassLabel(names=["class_0", "class_1", "class_2"])})
dataset = Dataset.from_dict(data, features=features)
splits = dataset.train_test_split(test_size=0.2, stratify_by_column="label")
print(f"Success with numpy {np.__version__}")
```
but it also works for `numpy<2`

@davebulaval tried with numpy 2.3.4, and maybe i have successfully reproduced the bug!
```
ValueError: Unable to avoid copy while creating an array as requested.
If using `np.array(obj, copy=False)` replace it with `np.asarray(obj)` to allow a copy when needed (no behavior change in NumPy 1.x).
For more details, see https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword.
```

Also i downgraded to numpy 1.26.4
```
(hf-reproduce) F:\Python\Machine learning\reproducing>python repro.py
DatasetDict({
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 16
    })
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 4
    })
})
```

Also @CloseChoice The bug only appears in cases where the Arrow array cannot be represented as a contiguous NumPy array without copying.

So closing the discussion loop here - 

The error occurs because `train_test_split(..., stratify_by_column=...)` attempts to convert
an Arrow column to a NumPy array using `np.array(..., copy=False)`.

In NumPy <2.0 this silently allowed a copy if needed.
In NumPy ≥2.0 this raises:
ValueError: Unable to avoid copy while creating an array as requested.

This only happens when the Arrow column is not contiguous in memory, which explains
why some datasets reproduce it and others do not.
Title: disable_progress_bar() not working as expected
State: closed

### Describe the bug

Hi,

I'm trying to load a dataset on Kaggle TPU image. There is some known compat issue with progress bar on Kaggle, so I'm trying to disable the progress bar globally. This does not work as you can see in [here](https://www.kaggle.com/code/windmaple/hf-datasets-issue).

In contract, disabling progress bar for snapshot_download() works as expected as in [here](https://www.kaggle.com/code/windmaple/snapshot-download-error).

### Steps to reproduce the bug

See this [notebook](https://www.kaggle.com/code/windmaple/hf-datasets-issue). 

There is sth. wrong with `shell_paraent`.

### Expected behavior

The downloader should disable progress bar and move forward w/ no error.

### Environment info

The latest version as I did:

!pip install -U datasets ipywidgets ipykernel

Comments:
@xianbaoqian 

Closing this one since it's a Xet issue.
Title: Allow streaming hdf5 files
State: closed

Add streaming support after https://github.com/huggingface/datasets/pull/7690, cc @klamike :)

## Details

in `datasets` loaders, `open()` is extended to work with files that are on disk but also on HF. Files on HF are streamed using HTTP range requests using the `HfFileSystem` implementation in the `huggingface_hub` library.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7814). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Parquet: add `on_bad_file` argument to error/warn/skip bad files
State: closed

```python
from datasets import load_dataset

on_bad_file = "error"  # default
# on_bad_file = "warn"  # warn and skip bad file
# on_bad_file = "skip"  # skip bad file
ds = load_dataset(parquet_dataset_id, on_bad_file=on_bad_file)
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7806). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Less api calls when resolving data_files
State: closed

There ~10 unnecessary `/api/datasets/user/dataset/revision`calls due to multithreading in data files resolution.

I disabled multithreading, which was actually not useful anymore since `HfFileSystem` has been using caching for a while now.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7805). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Support scientific data formats
State: open

List of formats and libraries we can use to load the data in `datasets`:

- [ ] DICOMs: pydicom
- [x] NIfTIs: nibabel
- [ ] WFDB: wfdb

cc @zaRizk7 for viz

Feel free to comment / suggest other formats and libs you'd like to see or to share your interest in one of the mentioned format

Comments:
Title: [Docs] Missing documentation for `Dataset.from_dict`
State: open

Documentation link: https://huggingface.co/docs/datasets/en/package_reference/main_classes

Link to method (docstring present): https://github.com/huggingface/datasets/blob/6f2502c5a026caa89839713f6f7c8b958e5e83eb/src/datasets/arrow_dataset.py#L1029

The docstring is present for the function, but seems missing from the official documentation for the `Dataset` class on HuggingFace.

The method in question:
```python
    @classmethod
    def from_dict(
        cls,
        mapping: dict,
        features: Optional[Features] = None,
        info: Optional[DatasetInfo] = None,
        split: Optional[NamedSplit] = None,
    ) -> "Dataset":
        """
        Convert `dict` to a `pyarrow.Table` to create a [`Dataset`].

        Important: a dataset created with from_dict() lives in memory
        and therefore doesn't have an associated cache directory.
        This may change in the future, but in the meantime if you
        want to reduce memory usage you should write it back on disk
        and reload using e.g. save_to_disk / load_from_disk.

        Args:
            mapping (`Mapping`):
                Mapping of strings to Arrays or Python lists.
            features ([`Features`], *optional*):
                Dataset features.
            info (`DatasetInfo`, *optional*):
                Dataset information, like description, citation, etc.
            split (`NamedSplit`, *optional*):
                Name of the dataset split.

        Returns:
            [`Dataset`]
        """
```

Comments:
I'd like to work on this documentation issue.

Hi I'd like to work on this. I can see the docstring is already in the code. 
Could you confirm:
1. Is this still available?
2. Should I add this to the main_classes.md file, or is there a specific 
   documentation file I should update?
3. Are there any formatting guidelines I should follow?

I'm new to contributing but eager to learn!
Title: Add parquet scan options and docs
State: closed

I added scan options, useful to control buffering and caching when streaming and docs, including how to select a subset of columns and apply filters

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7801). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Audio dataset is not decoding on 4.1.1
State: open

### Describe the bug

The audio column remain as non-decoded objects even when accessing them.

```python
dataset = load_dataset("MrDragonFox/Elise", split = "train")
dataset[0] # see that it doesn't show 'array' etc...
```

Works fine with `datasets==3.6.0`

Followed the docs in 

- https://huggingface.co/docs/datasets/en/audio_load

### Steps to reproduce the bug

```python
dataset = load_dataset("MrDragonFox/Elise", split = "train")
dataset[0] # see that it doesn't show 'array' etc...
```


### Expected behavior

It should decode when accessing the elemenet

### Environment info

4.1.1
ubuntu 22.04


Related

- https://github.com/huggingface/datasets/issues/7707

Comments:
Previously (datasets<=3.6.0), audio columns were decoded automatically when accessing a row. Now, for performance reasons, audio decoding is lazy by default: you just see the file path unless you explicitly cast the column to Audio.

Here’s the fix (following the current [datasets audio docs](https://huggingface.co/docs/datasets/en/audio_load)
):

```
from datasets import load_dataset, Audio

dataset = load_dataset("MrDragonFox/Elise", split="train")

# Explicitly decode the audio column
dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))

print(dataset[0]["audio"])
# {'path': '...', 'array': array([...], dtype=float32), 'sampling_rate': 16000}
```

@haitam03-yo's comment is right that the data is not decoded by default anymore indeed, but here is how it works in practice now:

From `datasets` v4, audio data are read as [AudioDecoder](https://meta-pytorch.org/torchcodec/0.4/generated/torchcodec.decoders.AudioDecoder.html) objects from torchcodec. This doesn't decode the data by default, but you can call `audio.get_all_samples()` to decode the audio.

See the documentation on how to process audio data here: https://huggingface.co/docs/datasets/audio_process

To resolve this, you need to explicitly cast the audio column to the Audio feature. This will decode the audio data and make it accessible as an array. Here is the corrected code snippet


from datasets import load_dataset, Audio

# Load your dataset
dataset = load_dataset("MrDragonFox/Elise", split="train")

# Explicitly cast the 'audio' column to the Audio feature
dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))

# Now you can access the decoded audio array
print(dataset[0]["audio"])

By adding the cast_column step, you are telling the datasets library to decode the audio data with the specified sampling rate, and you will then be able to access the audio array as you were used to in previous versions.
Title: Datasets: Add WMT21 & WMT22 loaders (basic TSV loaders, sample data, tests)
State: closed

- Implemented TSV-based dataset loaders:
   - WMT21Dataset (local_datasets/wmt21/wmt21_dataset.py)
   - WMT22Dataset (local_datasets/wmt22/wmt22_dataset.py)
   These classes load source-target pairs from .tsv files for train, validation, and test splits.
- Created sample dummy data for both datasets:
   - dummy_data/train.tsv, dummy_data/validation.tsv, dummy_data/test.tsv
   - Includes a few realistic example lines to allow CI and local tests to pass without downloading full datasets.
- Added automated tests for robust validation:
   - tests/test_wmt21.py and tests/test_wmt22.py
   - Checks that all splits load correctly, empty lines are ignored, and the number of examples matches the number of lines in the .tsv files.
   - Edge cases handled: empty lines, malformed lines, extra tabs.
- Added README.md files for both datasets:
   - Provides dataset structure, usage instructions, and placeholders for citation & license information.
   - Ensures that other developers and reviewers can understand dataset usage immediately.
- Ensured easy local testing:
   - Load datasets programmatically using WMT21Dataset / WMT22Dataset.
   - Verified train/validation/test splits are correctly returned as Python dictionaries of Dataset objects.


-Provides initial support for WMT21 and WMT22 NLP/translation experiments.
-Allows contributors and reviewers to test dataset loading locally or in CI without downloading large datasets.
-Serves as a template to extend to other WMT datasets in the future.

Testing instructions:
```
# Activate your environment
pytest tests/test_wmt21.py -v
pytest tests/test_wmt22.py -v
```
```
from local_datasets.wmt21.wmt21_dataset import WMT21Dataset
from local_datasets.wmt22.wmt22_dataset import WMT22Dataset

# WMT21
wmt21 = WMT21Dataset("local_datasets/wmt21/dummy_data")
ds21 = wmt21.load()
print(ds21["train"][0])
print(ds21["validation"][0])
print(ds21["test"][0])

# WMT22
wmt22 = WMT22Dataset("local_datasets/wmt22/dummy_data")
ds22 = wmt22.load()
print(ds22["train"][0])
print(ds22["validation"][0])
print(ds22["test"][0])
```

Comments:
closing since datasets should be added on https://huggingface.co directly
Title: Add pyarrow's binary view to features
State: closed

Basically https://github.com/huggingface/datasets/pull/7718 just for binary view instead of string view

Comments:
@lhoestq 🙏 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7795). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: fix: add `num_proc` argument to `Dataset.to_sql`
State: open

**Task Done:** 

- Resolve issue #7788 : Add the missing argument mapping in Dataset.to_sql (`src/datasets/arrow_dataset.py`)




Comments:
Hi ! could you also write a test to make sure this works fine ?

(in case there needs to be a special logic to handle the concurrent writes to the database)

Hi @lhoestq 
Done! Let me know if more is needed :)

Hi @lhoestq could you please review my changes?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7791). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: `Dataset.to_sql` doesn't utilize `num_proc`
State: open

The underlying `SqlDatasetWriter` has `num_proc` as an available argument [here](https://github.com/huggingface/datasets/blob/5dc1a179783dff868b0547c8486268cfaea1ea1f/src/datasets/io/sql.py#L63) , but `Dataset.to_sql()` does not accept it, therefore it is always using one process for the SQL conversion.

Comments: