Minimartzz/github-issues huggingface dataset:
Transform with columns parameter triggers on non-specified column access
State: closed

### Describe the bug

Iterating over a [`Column`](https://github.com/huggingface/datasets/blob/8b1bd4ec1cc9e9ce022f749abb6485ef984ae7c0/src/datasets/arrow_dataset.py#L633-L692) iterates through the parent [`Dataset`](https://github.com/huggingface/datasets/blob/8b1bd4ec1cc9e9ce022f749abb6485ef984ae7c0/src/datasets/arrow_dataset.py#L695) and applies all formatting/transforms on each row, regardless of which column is being accessed. This causes an error when transforms depend on columns not present in the projection.

### Steps to reproduce the bug

### Load a dataset with multiple columns

```python
ds = load_dataset("mrbrobot/isic-2024", split="train")
```

### Define a transform that specifies an input column

```python
def image_transform(batch):
    batch["image"] = batch["image"]  # KeyError when batch doesn't contain "image"
    return batch

# apply transform only to image column
ds = ds.with_format("torch")
ds = ds.with_transform(image_transform, columns=["image"], output_all_columns=True)
```

### Iterate over non-specified column

```python
# iterate over a different column, triggers the transform on each row, but batch doesn't contain "image"
for t in ds["target"]:  # KeyError: 'image'
    print(t)
```

### Expected behavior

If a user iterates over `ds["target"]` and the transform specifies `columns=["image"]`, the transform should be skipped.

### Environment info

`datasets`: 4.2.0
Python: 3.12.12
Linux: Debian 11.11

Comments:
DOC: `mode` parameter on pdf and video features unused
State: closed

Following up on https://github.com/huggingface/datasets/pull/7840 I asked claude code to check for undocumented parameters for other features and it found:
- mode parameter on video is documented but unused: https://github.com/huggingface/datasets/blob/main/src/datasets/features/video.py#L48-L49
- the same goes for the mode parameter on the pdf feature: https://github.com/huggingface/datasets/blob/main/src/datasets/features/pdf.py#L47-L48

I assume checking if these modes can be supported and otherwise removing them is the way to go here.


Comments:
They seem to be artefacts from a copy-paste of the Image feature ^^' we should remove them
Add num channels to audio
State: closed

Fixes #7837

We currently have the [mono attribute for Audio documented](https://github.com/huggingface/datasets/blob/41c05299348a499807432ab476e1cdc4143c8772/src/datasets/features/audio.py#L52C1-L54C22) but not used anywhere resulting in confusion for users. Since torchcodec does not know this attribute I suggest using `num_channels` (currently supported `None` (leave unchanged), mono: `1`, stereo: `2`).

I could also add a mono attribute but found that to be more confusing for developers and would restrict us if at any point in the future more than 2 channels are supported.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7840). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
mono parameter to the Audio feature is missing
State: closed

According to the docs, there is a "mono" parameter to the Audio feature, which turns any stereo into mono. In practice the signal is not touched and the mono parameter, even though documented, does not exist.


https://github.com/huggingface/datasets/blob/41c05299348a499807432ab476e1cdc4143c8772/src/datasets/features/audio.py#L52C1-L54C22

Comments:
Hey,  we removed the misleading passage in the docstring and enabled support for `num_channels` as torchcodec does

thanks!
Add DICOM support
State: open

supports #7804
Add support for the dicom file format. 

This PR follows PR #7815 and PR #7325 closely.
Remarkable differences:
I made sure that we can load all of pydicom's test data, and encountered the `force=True` parameter that we explicitly support here. This allows to trying to load corrupted dicom files, we explicitly test this!

There is one dataset with all of dicom's test data on huggingface which can be loaded using this branch with the following script:
```python
from datasets import load_dataset
from datasets import Features, ClassLabel
from datasets.features import Dicom

features = Features({
    "dicom": Dicom(force=True),  # necessary to be able to load one corrupted file
    "label": ClassLabel(num_classes=2)
})

ds = load_dataset("TobiasPitters/dicom-sample-dataset",
                  features=features)

error_count = 0

for idx, item in enumerate(ds["test"]):
    dicom = item["dicom"]

    try:
        print(f"Type: {type(dicom)}")
        if hasattr(dicom, 'PatientID'):
            print(f"PatientID: {dicom.PatientID}")
        if hasattr(dicom, 'StudyInstanceUID'):
            print(f"StudyInstanceUID: {dicom.StudyInstanceUID}")
        if hasattr(dicom, 'Modality'):
            print(f"Modality: {dicom.Modality}")
    except Exception as e:
        error_count += 1
        print(e)

print(f"Finished processing with {error_count} errors.")
```

todo: 
- [x] add docs (will do so soon)

Comments:
Awesome ! For the docs should we rename https://huggingface.co/docs/datasets/nifti_dataset to medical_imaging_dataset and have both DICOM and NIfTI together or have separate pages in you opinion ?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7835). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Awesome ! For the docs should we rename https://huggingface.co/docs/datasets/nifti_dataset to medical_imaging_dataset and have both DICOM and NIfTI together or have separate pages in you opinion ?

Makes sense, is more intuitive for the user and the pages as proposed in this branch have a lot of overlap. I would then structure it in such a way to write some brief things about medical imaging, then introduce the formats (so basically concatenating the two pages together and removing duplicates).

Pls don't merge currently, since we'll need an `embed_storage` function in here as well. See
https://github.com/huggingface/datasets/pull/7815#issuecomment-3494094692 and the following conversation

@lhoestq, this is ready for a first round of review.
Fix: Properly render [!TIP] block in stream.shuffle documentation
State: closed

The second line starting with the bracket doesn't properly render on huggingface/docs. Added "> " to address it.

In the client documentation, the markdown 'TIP' paragraph in docs/stream#shuffle is not well executed, not as the other in the same page / while markdown is correctly considering it.

Documentation:
https://huggingface.co/docs/datasets/v4.3.0/en/stream#shuffle:~:text=%5B!TIP%5D%5BIterableDataset.shuffle()%5D(/docs/datasets/v4.3.0/en/package_reference/main_classes%23datasets.IterableDataset.shuffle)%20will%20also%20shuffle%20the%20order%20of%20the%20shards%20if%20the%20dataset%20is%20sharded%20into%20multiple%20files.

Github source:
https://github.com/huggingface/datasets/blob/main/docs/source/stream.mdx#:~:text=Casting%20only%20works%20if%20the%20original%20feature%20type%20and%20new%20feature%20type%20are%20compatible.%20For%20example%2C%20you%20can%20cast%20a%20column%20with%20the%20feature%20type%20Value(%27int32%27)%20to%20Value(%27bool%27)%20if%20the%20original%20column%20only%20contains%20ones%20and%20zeros.

Comments:
[DOCS][minor] TIPS paragraph not compiled in docs/stream
State: closed

In the client documentation, the markdown 'TIP' paragraph for paragraph in docs/stream#shuffle is not well executed — not as the other in the same page / while markdown is correctly considering it.

Documentation:
https://huggingface.co/docs/datasets/v4.3.0/en/stream#shuffle:~:text=%5B!TIP%5D%5BIterableDataset.shuffle()%5D(/docs/datasets/v4.3.0/en/package_reference/main_classes%23datasets.IterableDataset.shuffle)%20will%20also%20shuffle%20the%20order%20of%20the%20shards%20if%20the%20dataset%20is%20sharded%20into%20multiple%20files.

Github source:
https://github.com/huggingface/datasets/blob/main/docs/source/stream.mdx#:~:text=Casting%20only%20works%20if%20the%20original%20feature%20type%20and%20new%20feature%20type%20are%20compatible.%20For%20example%2C%20you%20can%20cast%20a%20column%20with%20the%20feature%20type%20Value(%27int32%27)%20to%20Value(%27bool%27)%20if%20the%20original%20column%20only%20contains%20ones%20and%20zeros.

Comments:
Memory leak / Large memory usage with num_workers = 0 and numerous dataset within DatasetDict
State: open

### Describe the bug

Hi team, first off, I love the datasets library! 🥰

I'm encountering a potential memory leak / increasing memory usage when training a model on a very large DatasetDict.

Setup: I have a DatasetDict containing 362 distinct datasets, which sum up to ~2.8 billion rows.

Training Task: I'm performing contrastive learning with SentenceTransformer and Accelerate on a single node with 4 H100, which requires me to sample from only one dataset at a time.

Training Loop: At each training step, I sample ~16,000 examples from a single dataset, and then switch to a different dataset for the next step. I iterate through all 362 datasets this way. 

Problem: The process's memory usage continuously increases over time, eventually causing a stale status where GPUs would stop working. It seems memory from previously sampled datasets isn't being released. I've set num_workers=0 for all experiments.

Chart 1: Standard DatasetDict The memory usage grows steadily until it make the training stale (RSS memory) <img width="773" height="719" alt="Image" src="https://github.com/user-attachments/assets/6606bef5-1153-4f2d-bf08-82da249d6e8d" />

Chart 2: IterableDatasetDict I also tried  to use IterableDatasetDict and IterableDataset. The memory curve is "smoother," but the result is the same: it grows indefinitely and the training become stale. <img width="339" height="705" alt="Image" src="https://github.com/user-attachments/assets/ee90c1a1-6c3b-4135-9edc-90955cb1695a" />

Any feedback or guidance on how to manage this memory would be greatly appreciated!

### Steps to reproduce the bug

WIP, I'll add some code that manage to reproduce this error, but not straightforward. 

### Expected behavior

The memory usage should remain relatively constant or plateau after a few steps. Memory used for sampling one dataset should be released before or during the sampling of the next dataset.

### Environment info

Python: 3.12
Datasets: 4.3.0
SentenceTransformers: 5.1.1

Comments:
Thanks for the report, this is possibly related #7722 and #7694.

Could you pls provide steps to reproduce this?

To overcome this issue right now I did simply reduce the size of the dataset and ended up running a for loop (my training has now a constant learning rate schedule). From what I understood, and I don't know if it's possible, the solution would be to tell the backend of `datasets` to leave x% of the memory free  (including memory mapping). Can't release the data right now but I will and then allow to reproduce this issue. But it will involve to have some free TB of disk

@raphaelsty thanks for coming back to this. I assume you are running in streaming mode? That should prevent these errors but it looks like more people than just you have this problem, so a clearly reproducing example (including data + code) is highly appreciated.

This could be related to this issue: https://github.com/huggingface/datasets/issues/4883 in which we discussed how RSS and memory mapping works and depends on the OS and disk type.
Fix ArrowInvalid for large variables
State: open

Fixes #7821

In addition to the solution proposed in the issue, I encountered we also need to support 64bit types when writing using the `RecordBatchStreamWriter`. 

Not sure if we want to create such large objects in the CI, but this is the only way to test that the issue is fixed, therefore I added the unit test.

Comments:
@lhoestq : from what I see in the CI logs ( tests/test_arrow_dataset.py::test_map_int32_overflow 
/opt/hostedtoolcache/Python/3.9.23/x64/lib/python3.9/site-packages/multiprocess/resource_tracker.py:219: UserWarning: resource_tracker: There appear to be 24 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '), see [here](https://github.com/huggingface/datasets/actions/runs/18720027796/job/53591652242#step:13:6530), the test crashes probably since it consumes too much memory, it works locally for me though. 
  
  Shall we keep the test, remove ti, or skip it, or is there a specific pytest flag you have for slow/memory-consuming tests I can add, so that it gets skipped automatically on the CI but might be run locally?
  
  EDIT: I added the `high_memory` marker (see [here](https://github.com/huggingface/datasets/pull/7825/files#diff-c480dc36aca482db963c43ee66ba21705111f62f826352c96558ef5ed1619791R4789)), so this will not be run in the CI, which is not ideal but I guess the way if we want to keep the test
Fix random seed on shuffle and interleave_datasets
State: closed

closes #7567

Add `shift_rngs` method to `ExamplesIterable` that is called directly after sharding. If a generator is available (not the case for all subclasses) we update the seed of the generator by shifting by the worker_id.

~This is just the fix for `shuffle`, in the corresponding issue `interleave_datasets` is mentioned as well, which won't be fixed with this approach.~

EDIT: This is a fix for `shuffle` and `interleave_datasets`. Adding recursivity to `shift_rngs` solved `interleave_datasets` as well. Not sure though if this is completely safe or if we could destroy something with that. I don't think so but could be wrong and appreciate some guidance from the maintainers. I also checked, on a single_worker we are always handing over `index=0` so that case preserves the seed the user specified. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7823). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Cool ! To avoid unwanted side effects it could be implemented for every class instead of using `hasattr`, and also return a new instance instead of modifying it in place. Some other functions work like that already like `shuffle_data_sources` and `shard_data_sources`.

Returning a new object is actually quite important otherwise iterating on the dataset multiple times would shift the RNGs every time

> Cool ! To avoid unwanted side effects it could be implemented for every class instead of using `hasattr`, and also return a new instance instead of modifying it in place. Some other functions work like that already like `shuffle_data_sources` and `shard_data_sources`.
> 
> Returning a new object is actually quite important otherwise iterating on the dataset multiple times would shift the RNGs every time

Thanks for the review. 

I managed to return instances of `_BaseExamplesIterable` instead of modifying inplace and this shifts the seed correctly even in the case of multiple iterations (see additional test). But I kept the `hasattr` for the reason that we have chained ex_iterables (so an ex_iterable can have other ex_iterables) and for the child ex_iterable it is not clear if they do have a generator attribute (e.g. `ArrowExamplesIterable`, `RebatchedArrowExamplesIterable`, `SelectColumnsIterable`, `StepExamplesIterable`, `SkipExamplesIterable`, `RepeatExamplesIterable` all don't have that). We could implement a generic method `shift_rngs` on the baseclass and then just return the ex_iterable and get rid of `hasattr` but I'd prefer the `hasattr` solution here. Let me now what you think about that.
Keep hffs cache in workers when streaming
State: closed

(and also reorder the hffs args to improve caching)

When using `DataLoader(iterable_dataset, num_workers=...)` the dataset is pickled and passed to the worker. However previously the resulting dataset would be in a process with an empty hffs cache. By keeping the cache attached to `IterableDataset`, the cached hffs instances are pickled with the dataset and re-populates the cache in the DataLoader workers

this requires https://github.com/huggingface/huggingface_hub/pull/3443 to work effectively though, otherwise the unpickled hffs cache would start empty


cc @andimarafioti @ltmeyer

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7820). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Cannot download opus dataset
State: open

When I tried to download opus_books using:

from datasets import load_dataset
dataset = load_dataset("Helsinki-NLP/opus_books")

I got the following errors:
FileNotFoundError: Couldn't find any data file at /workspace/Helsinki-NLP/opus_books. Couldn't find 'Helsinki-NLP/opus_books' on the Hugging Face Hub either: LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

I also tried:
dataset = load_dataset("opus_books", "en-zh")

and the errors remain the same. However, I can download "mlabonne/FineTome-100k" successfully. 
My datasets is version 4.2.0

Any clues? Big thanks.

Comments:
Hi ! it seems "en-zh" doesn't exist for this dataset

You can see the list of subsets here: https://huggingface.co/datasets/Helsinki-NLP/opus_books
train_test_split and stratify breaks with Numpy 2.0
State: closed

### Describe the bug

As stated in the title, since Numpy changed in version >2.0 with copy, the stratify parameters break.

e.g. `all_dataset.train_test_split(test_size=0.2,stratify_by_column="label")` returns a Numpy error.

It works if you downgrade Numpy to a version lower than 2.0.

### Steps to reproduce the bug

1. Numpy > 2.0
2. `all_dataset.train_test_split(test_size=0.2,stratify_by_column="label")`

### Expected behavior

It returns a stratified split as per the results of Numpy < 2.0

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-6.8.0-85-generic-x86_64-with-glibc2.35
- Python version: 3.13.7
- Huggingface_hub version: 0.34.4
- PyArrow version: 19.0.0
- Pandas version: 2.3.2


Comments:
I can't reproduce this. Could you pls provide an example with a public dataset/artificial dataset and show how you loaded that?

This works for me:

```python
import numpy as np
from datasets import Dataset, Features, ClassLabel, Value

data = {"text": [f"sample_{i}" for i in range(100)], "label": [i % 3 for i in range(100)]}
features = Features({"text": Value("string"),
                     "label": ClassLabel(names=["class_0", "class_1", "class_2"])})
dataset = Dataset.from_dict(data, features=features)
splits = dataset.train_test_split(test_size=0.2, stratify_by_column="label")
print(f"Success with numpy {np.__version__}")
```
but it also works for `numpy<2`

@davebulaval tried with numpy 2.3.4, and maybe i have successfully reproduced the bug!
```
ValueError: Unable to avoid copy while creating an array as requested.
If using `np.array(obj, copy=False)` replace it with `np.asarray(obj)` to allow a copy when needed (no behavior change in NumPy 1.x).
For more details, see https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword.
```

Also i downgraded to numpy 1.26.4
```
(hf-reproduce) F:\Python\Machine learning\reproducing>python repro.py
DatasetDict({
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 16
    })
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 4
    })
})
```

Also @CloseChoice The bug only appears in cases where the Arrow array cannot be represented as a contiguous NumPy array without copying.

So closing the discussion loop here - 

The error occurs because `train_test_split(..., stratify_by_column=...)` attempts to convert
an Arrow column to a NumPy array using `np.array(..., copy=False)`.

In NumPy <2.0 this silently allowed a copy if needed.
In NumPy ≥2.0 this raises:
ValueError: Unable to avoid copy while creating an array as requested.

This only happens when the Arrow column is not contiguous in memory, which explains
why some datasets reproduce it and others do not.
disable_progress_bar() not working as expected
State: closed

### Describe the bug

Hi,

I'm trying to load a dataset on Kaggle TPU image. There is some known compat issue with progress bar on Kaggle, so I'm trying to disable the progress bar globally. This does not work as you can see in [here](https://www.kaggle.com/code/windmaple/hf-datasets-issue).

In contract, disabling progress bar for snapshot_download() works as expected as in [here](https://www.kaggle.com/code/windmaple/snapshot-download-error).

### Steps to reproduce the bug

See this [notebook](https://www.kaggle.com/code/windmaple/hf-datasets-issue). 

There is sth. wrong with `shell_paraent`.

### Expected behavior

The downloader should disable progress bar and move forward w/ no error.

### Environment info

The latest version as I did:

!pip install -U datasets ipywidgets ipykernel

Comments:
@xianbaoqian 

Closing this one since it's a Xet issue.
Allow streaming hdf5 files
State: closed

Add streaming support after https://github.com/huggingface/datasets/pull/7690, cc @klamike :)

## Details

in `datasets` loaders, `open()` is extended to work with files that are on disk but also on HF. Files on HF are streamed using HTTP range requests using the `HfFileSystem` implementation in the `huggingface_hub` library.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7814). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Parquet: add `on_bad_file` argument to error/warn/skip bad files
State: closed

```python
from datasets import load_dataset

on_bad_file = "error"  # default
# on_bad_file = "warn"  # warn and skip bad file
# on_bad_file = "skip"  # skip bad file
ds = load_dataset(parquet_dataset_id, on_bad_file=on_bad_file)
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7806). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Less api calls when resolving data_files
State: closed

There ~10 unnecessary `/api/datasets/user/dataset/revision`calls due to multithreading in data files resolution.

I disabled multithreading, which was actually not useful anymore since `HfFileSystem` has been using caching for a while now.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7805). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.