NyssaW/github-issues huggingface dataset:
DOC: `mode` parameter on pdf and video features unused

Following up on https://github.com/huggingface/datasets/pull/7840 I asked claude code to check for undocumented parameters for other features and it found:
- mode parameter on video is documented but unused: https://github.com/huggingface/datasets/blob/main/src/datasets/features/video.py#L48-L49
- the same goes for the mode parameter on the pdf feature: https://github.com/huggingface/datasets/blob/main/src/datasets/features/pdf.py#L47-L48

I assume checking if these modes can be supported and otherwise removing them is the way to go here.


Comments:
They seem to be artefacts from a copy-paste of the Image feature ^^' we should remove them
Add num channels to audio

Fixes #7837

We currently have the [mono attribute for Audio documented](https://github.com/huggingface/datasets/blob/41c05299348a499807432ab476e1cdc4143c8772/src/datasets/features/audio.py#L52C1-L54C22) but not used anywhere resulting in confusion for users. Since torchcodec does not know this attribute I suggest using `num_channels` (currently supported `None` (leave unchanged), mono: `1`, stereo: `2`).

I could also add a mono attribute but found that to be more confusing for developers and would restrict us if at any point in the future more than 2 channels are supported.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7840). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
mono parameter to the Audio feature is missing

According to the docs, there is a "mono" parameter to the Audio feature, which turns any stereo into mono. In practice the signal is not touched and the mono parameter, even though documented, does not exist.


https://github.com/huggingface/datasets/blob/41c05299348a499807432ab476e1cdc4143c8772/src/datasets/features/audio.py#L52C1-L54C22

Comments:
Hey,  we removed the misleading passage in the docstring and enabled support for `num_channels` as torchcodec does

thanks!
Add DICOM support

supports #7804
Add support for the dicom file format. 

This PR follows PR #7815 and PR #7325 closely.
Remarkable differences:
I made sure that we can load all of pydicom's test data, and encountered the `force=True` parameter that we explicitly support here. This allows to trying to load corrupted dicom files, we explicitly test this!

There is one dataset with all of dicom's test data on huggingface which can be loaded using this branch with the following script:
```python
from datasets import load_dataset
from datasets import Features, ClassLabel
from datasets.features import Dicom

features = Features({
    "dicom": Dicom(force=True),  # necessary to be able to load one corrupted file
    "label": ClassLabel(num_classes=2)
})

ds = load_dataset("TobiasPitters/dicom-sample-dataset",
                  features=features)

error_count = 0

for idx, item in enumerate(ds["test"]):
    dicom = item["dicom"]

    try:
        print(f"Type: {type(dicom)}")
        if hasattr(dicom, 'PatientID'):
            print(f"PatientID: {dicom.PatientID}")
        if hasattr(dicom, 'StudyInstanceUID'):
            print(f"StudyInstanceUID: {dicom.StudyInstanceUID}")
        if hasattr(dicom, 'Modality'):
            print(f"Modality: {dicom.Modality}")
    except Exception as e:
        error_count += 1
        print(e)

print(f"Finished processing with {error_count} errors.")
```

todo: 
- [x] add docs (will do so soon)

Comments:
Awesome ! For the docs should we rename https://huggingface.co/docs/datasets/nifti_dataset to medical_imaging_dataset and have both DICOM and NIfTI together or have separate pages in you opinion ?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7835). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Awesome ! For the docs should we rename https://huggingface.co/docs/datasets/nifti_dataset to medical_imaging_dataset and have both DICOM and NIfTI together or have separate pages in you opinion ?

Makes sense, is more intuitive for the user and the pages as proposed in this branch have a lot of overlap. I would then structure it in such a way to write some brief things about medical imaging, then introduce the formats (so basically concatenating the two pages together and removing duplicates).

Pls don't merge currently, since we'll need an `embed_storage` function in here as well. See
https://github.com/huggingface/datasets/pull/7815#issuecomment-3494094692 and the following conversation

@lhoestq, this is ready for a first round of review.
Fix: Properly render [!TIP] block in stream.shuffle documentation

The second line starting with the bracket doesn't properly render on huggingface/docs. Added "> " to address it.

In the client documentation, the markdown 'TIP' paragraph in docs/stream#shuffle is not well executed, not as the other in the same page / while markdown is correctly considering it.

Documentation:
https://huggingface.co/docs/datasets/v4.3.0/en/stream#shuffle:~:text=%5B!TIP%5D%5BIterableDataset.shuffle()%5D(/docs/datasets/v4.3.0/en/package_reference/main_classes%23datasets.IterableDataset.shuffle)%20will%20also%20shuffle%20the%20order%20of%20the%20shards%20if%20the%20dataset%20is%20sharded%20into%20multiple%20files.

Github source:
https://github.com/huggingface/datasets/blob/main/docs/source/stream.mdx#:~:text=Casting%20only%20works%20if%20the%20original%20feature%20type%20and%20new%20feature%20type%20are%20compatible.%20For%20example%2C%20you%20can%20cast%20a%20column%20with%20the%20feature%20type%20Value(%27int32%27)%20to%20Value(%27bool%27)%20if%20the%20original%20column%20only%20contains%20ones%20and%20zeros.

Comments:
[DOCS][minor] TIPS paragraph not compiled in docs/stream

In the client documentation, the markdown 'TIP' paragraph for paragraph in docs/stream#shuffle is not well executed — not as the other in the same page / while markdown is correctly considering it.

Documentation:
https://huggingface.co/docs/datasets/v4.3.0/en/stream#shuffle:~:text=%5B!TIP%5D%5BIterableDataset.shuffle()%5D(/docs/datasets/v4.3.0/en/package_reference/main_classes%23datasets.IterableDataset.shuffle)%20will%20also%20shuffle%20the%20order%20of%20the%20shards%20if%20the%20dataset%20is%20sharded%20into%20multiple%20files.

Github source:
https://github.com/huggingface/datasets/blob/main/docs/source/stream.mdx#:~:text=Casting%20only%20works%20if%20the%20original%20feature%20type%20and%20new%20feature%20type%20are%20compatible.%20For%20example%2C%20you%20can%20cast%20a%20column%20with%20the%20feature%20type%20Value(%27int32%27)%20to%20Value(%27bool%27)%20if%20the%20original%20column%20only%20contains%20ones%20and%20zeros.

Comments:
Memory leak / Large memory usage with num_workers = 0 and numerous dataset within DatasetDict

### Describe the bug

Hi team, first off, I love the datasets library! 🥰

I'm encountering a potential memory leak / increasing memory usage when training a model on a very large DatasetDict.

Setup: I have a DatasetDict containing 362 distinct datasets, which sum up to ~2.8 billion rows.

Training Task: I'm performing contrastive learning with SentenceTransformer and Accelerate on a single node with 4 H100, which requires me to sample from only one dataset at a time.

Training Loop: At each training step, I sample ~16,000 examples from a single dataset, and then switch to a different dataset for the next step. I iterate through all 362 datasets this way. 

Problem: The process's memory usage continuously increases over time, eventually causing a stale status where GPUs would stop working. It seems memory from previously sampled datasets isn't being released. I've set num_workers=0 for all experiments.

Chart 1: Standard DatasetDict The memory usage grows steadily until it make the training stale (RSS memory) <img width="773" height="719" alt="Image" src="https://github.com/user-attachments/assets/6606bef5-1153-4f2d-bf08-82da249d6e8d" />

Chart 2: IterableDatasetDict I also tried  to use IterableDatasetDict and IterableDataset. The memory curve is "smoother," but the result is the same: it grows indefinitely and the training become stale. <img width="339" height="705" alt="Image" src="https://github.com/user-attachments/assets/ee90c1a1-6c3b-4135-9edc-90955cb1695a" />

Any feedback or guidance on how to manage this memory would be greatly appreciated!

### Steps to reproduce the bug

WIP, I'll add some code that manage to reproduce this error, but not straightforward. 

### Expected behavior

The memory usage should remain relatively constant or plateau after a few steps. Memory used for sampling one dataset should be released before or during the sampling of the next dataset.

### Environment info

Python: 3.12
Datasets: 4.3.0
SentenceTransformers: 5.1.1

Comments:
Thanks for the report, this is possibly related #7722 and #7694.

Could you pls provide steps to reproduce this?

To overcome this issue right now I did simply reduce the size of the dataset and ended up running a for loop (my training has now a constant learning rate schedule). From what I understood, and I don't know if it's possible, the solution would be to tell the backend of `datasets` to leave x% of the memory free  (including memory mapping). Can't release the data right now but I will and then allow to reproduce this issue. But it will involve to have some free TB of disk

@raphaelsty thanks for coming back to this. I assume you are running in streaming mode? That should prevent these errors but it looks like more people than just you have this problem, so a clearly reproducing example (including data + code) is highly appreciated.

This could be related to this issue: https://github.com/huggingface/datasets/issues/4883 in which we discussed how RSS and memory mapping works and depends on the OS and disk type.
Fix ArrowInvalid for large variables

Fixes #7821

In addition to the solution proposed in the issue, I encountered we also need to support 64bit types when writing using the `RecordBatchStreamWriter`. 

Not sure if we want to create such large objects in the CI, but this is the only way to test that the issue is fixed, therefore I added the unit test.

Comments:
@lhoestq : from what I see in the CI logs ( tests/test_arrow_dataset.py::test_map_int32_overflow 
/opt/hostedtoolcache/Python/3.9.23/x64/lib/python3.9/site-packages/multiprocess/resource_tracker.py:219: UserWarning: resource_tracker: There appear to be 24 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '), see [here](https://github.com/huggingface/datasets/actions/runs/18720027796/job/53591652242#step:13:6530), the test crashes probably since it consumes too much memory, it works locally for me though. 
  
  Shall we keep the test, remove ti, or skip it, or is there a specific pytest flag you have for slow/memory-consuming tests I can add, so that it gets skipped automatically on the CI but might be run locally?
  
  EDIT: I added the `high_memory` marker (see [here](https://github.com/huggingface/datasets/pull/7825/files#diff-c480dc36aca482db963c43ee66ba21705111f62f826352c96558ef5ed1619791R4789)), so this will not be run in the CI, which is not ideal but I guess the way if we want to keep the test
Fix random seed on shuffle and interleave_datasets

closes #7567

Add `shift_rngs` method to `ExamplesIterable` that is called directly after sharding. If a generator is available (not the case for all subclasses) we update the seed of the generator by shifting by the worker_id.

~This is just the fix for `shuffle`, in the corresponding issue `interleave_datasets` is mentioned as well, which won't be fixed with this approach.~

EDIT: This is a fix for `shuffle` and `interleave_datasets`. Adding recursivity to `shift_rngs` solved `interleave_datasets` as well. Not sure though if this is completely safe or if we could destroy something with that. I don't think so but could be wrong and appreciate some guidance from the maintainers. I also checked, on a single_worker we are always handing over `index=0` so that case preserves the seed the user specified. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7823). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Cool ! To avoid unwanted side effects it could be implemented for every class instead of using `hasattr`, and also return a new instance instead of modifying it in place. Some other functions work like that already like `shuffle_data_sources` and `shard_data_sources`.

Returning a new object is actually quite important otherwise iterating on the dataset multiple times would shift the RNGs every time

> Cool ! To avoid unwanted side effects it could be implemented for every class instead of using `hasattr`, and also return a new instance instead of modifying it in place. Some other functions work like that already like `shuffle_data_sources` and `shard_data_sources`.
> 
> Returning a new object is actually quite important otherwise iterating on the dataset multiple times would shift the RNGs every time

Thanks for the review. 

I managed to return instances of `_BaseExamplesIterable` instead of modifying inplace and this shifts the seed correctly even in the case of multiple iterations (see additional test). But I kept the `hasattr` for the reason that we have chained ex_iterables (so an ex_iterable can have other ex_iterables) and for the child ex_iterable it is not clear if they do have a generator attribute (e.g. `ArrowExamplesIterable`, `RebatchedArrowExamplesIterable`, `SelectColumnsIterable`, `StepExamplesIterable`, `SkipExamplesIterable`, `RepeatExamplesIterable` all don't have that). We could implement a generic method `shift_rngs` on the baseclass and then just return the ex_iterable and get rid of `hasattr` but I'd prefer the `hasattr` solution here. Let me now what you think about that.
Keep hffs cache in workers when streaming

(and also reorder the hffs args to improve caching)

When using `DataLoader(iterable_dataset, num_workers=...)` the dataset is pickled and passed to the worker. However previously the resulting dataset would be in a process with an empty hffs cache. By keeping the cache attached to `IterableDataset`, the cached hffs instances are pickled with the dataset and re-populates the cache in the DataLoader workers

this requires https://github.com/huggingface/huggingface_hub/pull/3443 to work effectively though, otherwise the unpickled hffs cache would start empty


cc @andimarafioti @ltmeyer

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7820). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Cannot download opus dataset

When I tried to download opus_books using:

from datasets import load_dataset
dataset = load_dataset("Helsinki-NLP/opus_books")

I got the following errors:
FileNotFoundError: Couldn't find any data file at /workspace/Helsinki-NLP/opus_books. Couldn't find 'Helsinki-NLP/opus_books' on the Hugging Face Hub either: LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

I also tried:
dataset = load_dataset("opus_books", "en-zh")

and the errors remain the same. However, I can download "mlabonne/FineTome-100k" successfully. 
My datasets is version 4.2.0

Any clues? Big thanks.

Comments:
Hi ! it seems "en-zh" doesn't exist for this dataset

You can see the list of subsets here: https://huggingface.co/datasets/Helsinki-NLP/opus_books
train_test_split and stratify breaks with Numpy 2.0

### Describe the bug

As stated in the title, since Numpy changed in version >2.0 with copy, the stratify parameters break.

e.g. `all_dataset.train_test_split(test_size=0.2,stratify_by_column="label")` returns a Numpy error.

It works if you downgrade Numpy to a version lower than 2.0.

### Steps to reproduce the bug

1. Numpy > 2.0
2. `all_dataset.train_test_split(test_size=0.2,stratify_by_column="label")`

### Expected behavior

It returns a stratified split as per the results of Numpy < 2.0

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-6.8.0-85-generic-x86_64-with-glibc2.35
- Python version: 3.13.7
- Huggingface_hub version: 0.34.4
- PyArrow version: 19.0.0
- Pandas version: 2.3.2


Comments:
I can't reproduce this. Could you pls provide an example with a public dataset/artificial dataset and show how you loaded that?

This works for me:

```python
import numpy as np
from datasets import Dataset, Features, ClassLabel, Value

data = {"text": [f"sample_{i}" for i in range(100)], "label": [i % 3 for i in range(100)]}
features = Features({"text": Value("string"),
                     "label": ClassLabel(names=["class_0", "class_1", "class_2"])})
dataset = Dataset.from_dict(data, features=features)
splits = dataset.train_test_split(test_size=0.2, stratify_by_column="label")
print(f"Success with numpy {np.__version__}")
```
but it also works for `numpy<2`

@davebulaval tried with numpy 2.3.4, and maybe i have successfully reproduced the bug!
```
ValueError: Unable to avoid copy while creating an array as requested.
If using `np.array(obj, copy=False)` replace it with `np.asarray(obj)` to allow a copy when needed (no behavior change in NumPy 1.x).
For more details, see https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword.
```

Also i downgraded to numpy 1.26.4
```
(hf-reproduce) F:\Python\Machine learning\reproducing>python repro.py
DatasetDict({
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 16
    })
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 4
    })
})
```

Also @CloseChoice The bug only appears in cases where the Arrow array cannot be represented as a contiguous NumPy array without copying.

So closing the discussion loop here - 

The error occurs because `train_test_split(..., stratify_by_column=...)` attempts to convert
an Arrow column to a NumPy array using `np.array(..., copy=False)`.

In NumPy <2.0 this silently allowed a copy if needed.
In NumPy ≥2.0 this raises:
ValueError: Unable to avoid copy while creating an array as requested.

This only happens when the Arrow column is not contiguous in memory, which explains
why some datasets reproduce it and others do not.
disable_progress_bar() not working as expected

### Describe the bug

Hi,

I'm trying to load a dataset on Kaggle TPU image. There is some known compat issue with progress bar on Kaggle, so I'm trying to disable the progress bar globally. This does not work as you can see in [here](https://www.kaggle.com/code/windmaple/hf-datasets-issue).

In contract, disabling progress bar for snapshot_download() works as expected as in [here](https://www.kaggle.com/code/windmaple/snapshot-download-error).

### Steps to reproduce the bug

See this [notebook](https://www.kaggle.com/code/windmaple/hf-datasets-issue). 

There is sth. wrong with `shell_paraent`.

### Expected behavior

The downloader should disable progress bar and move forward w/ no error.

### Environment info

The latest version as I did:

!pip install -U datasets ipywidgets ipykernel

Comments:
@xianbaoqian 

Closing this one since it's a Xet issue.
Allow streaming hdf5 files

Add streaming support after https://github.com/huggingface/datasets/pull/7690, cc @klamike :)

## Details

in `datasets` loaders, `open()` is extended to work with files that are on disk but also on HF. Files on HF are streamed using HTTP range requests using the `HfFileSystem` implementation in the `huggingface_hub` library.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7814). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Parquet: add `on_bad_file` argument to error/warn/skip bad files

```python
from datasets import load_dataset

on_bad_file = "error"  # default
# on_bad_file = "warn"  # warn and skip bad file
# on_bad_file = "skip"  # skip bad file
ds = load_dataset(parquet_dataset_id, on_bad_file=on_bad_file)
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7806). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Less api calls when resolving data_files

There ~10 unnecessary `/api/datasets/user/dataset/revision`calls due to multithreading in data files resolution.

I disabled multithreading, which was actually not useful anymore since `HfFileSystem` has been using caching for a while now.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7805). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Docs] Missing documentation for `Dataset.from_dict`

Documentation link: https://huggingface.co/docs/datasets/en/package_reference/main_classes

Link to method (docstring present): https://github.com/huggingface/datasets/blob/6f2502c5a026caa89839713f6f7c8b958e5e83eb/src/datasets/arrow_dataset.py#L1029

The docstring is present for the function, but seems missing from the official documentation for the `Dataset` class on HuggingFace.

The method in question:
```python
    @classmethod
    def from_dict(
        cls,
        mapping: dict,
        features: Optional[Features] = None,
        info: Optional[DatasetInfo] = None,
        split: Optional[NamedSplit] = None,
    ) -> "Dataset":
        """
        Convert `dict` to a `pyarrow.Table` to create a [`Dataset`].

        Important: a dataset created with from_dict() lives in memory
        and therefore doesn't have an associated cache directory.
        This may change in the future, but in the meantime if you
        want to reduce memory usage you should write it back on disk
        and reload using e.g. save_to_disk / load_from_disk.

        Args:
            mapping (`Mapping`):
                Mapping of strings to Arrays or Python lists.
            features ([`Features`], *optional*):
                Dataset features.
            info (`DatasetInfo`, *optional*):
                Dataset information, like description, citation, etc.
            split (`NamedSplit`, *optional*):
                Name of the dataset split.

        Returns:
            [`Dataset`]
        """
```

Comments:
I'd like to work on this documentation issue.

Hi I'd like to work on this. I can see the docstring is already in the code. 
Could you confirm:
1. Is this still available?
2. Should I add this to the main_classes.md file, or is there a specific 
   documentation file I should update?
3. Are there any formatting guidelines I should follow?

I'm new to contributing but eager to learn!
Audio dataset is not decoding on 4.1.1

### Describe the bug

The audio column remain as non-decoded objects even when accessing them.

```python
dataset = load_dataset("MrDragonFox/Elise", split = "train")
dataset[0] # see that it doesn't show 'array' etc...
```

Works fine with `datasets==3.6.0`

Followed the docs in 

- https://huggingface.co/docs/datasets/en/audio_load

### Steps to reproduce the bug

```python
dataset = load_dataset("MrDragonFox/Elise", split = "train")
dataset[0] # see that it doesn't show 'array' etc...
```


### Expected behavior

It should decode when accessing the elemenet

### Environment info

4.1.1
ubuntu 22.04


Related

- https://github.com/huggingface/datasets/issues/7707

Comments:
Previously (datasets<=3.6.0), audio columns were decoded automatically when accessing a row. Now, for performance reasons, audio decoding is lazy by default: you just see the file path unless you explicitly cast the column to Audio.

Here’s the fix (following the current [datasets audio docs](https://huggingface.co/docs/datasets/en/audio_load)
):

```
from datasets import load_dataset, Audio

dataset = load_dataset("MrDragonFox/Elise", split="train")

# Explicitly decode the audio column
dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))

print(dataset[0]["audio"])
# {'path': '...', 'array': array([...], dtype=float32), 'sampling_rate': 16000}
```

@haitam03-yo's comment is right that the data is not decoded by default anymore indeed, but here is how it works in practice now:

From `datasets` v4, audio data are read as [AudioDecoder](https://meta-pytorch.org/torchcodec/0.4/generated/torchcodec.decoders.AudioDecoder.html) objects from torchcodec. This doesn't decode the data by default, but you can call `audio.get_all_samples()` to decode the audio.

See the documentation on how to process audio data here: https://huggingface.co/docs/datasets/audio_process

To resolve this, you need to explicitly cast the audio column to the Audio feature. This will decode the audio data and make it accessible as an array. Here is the corrected code snippet


from datasets import load_dataset, Audio

# Load your dataset
dataset = load_dataset("MrDragonFox/Elise", split="train")

# Explicitly cast the 'audio' column to the Audio feature
dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))

# Now you can access the decoded audio array
print(dataset[0]["audio"])

By adding the cast_column step, you are telling the datasets library to decode the audio data with the specified sampling rate, and you will then be able to access the audio array as you were used to in previous versions.
Datasets: Add WMT21 & WMT22 loaders (basic TSV loaders, sample data, tests)

- Implemented TSV-based dataset loaders:
   - WMT21Dataset (local_datasets/wmt21/wmt21_dataset.py)
   - WMT22Dataset (local_datasets/wmt22/wmt22_dataset.py)
   These classes load source-target pairs from .tsv files for train, validation, and test splits.
- Created sample dummy data for both datasets:
   - dummy_data/train.tsv, dummy_data/validation.tsv, dummy_data/test.tsv
   - Includes a few realistic example lines to allow CI and local tests to pass without downloading full datasets.
- Added automated tests for robust validation:
   - tests/test_wmt21.py and tests/test_wmt22.py
   - Checks that all splits load correctly, empty lines are ignored, and the number of examples matches the number of lines in the .tsv files.
   - Edge cases handled: empty lines, malformed lines, extra tabs.
- Added README.md files for both datasets:
   - Provides dataset structure, usage instructions, and placeholders for citation & license information.
   - Ensures that other developers and reviewers can understand dataset usage immediately.
- Ensured easy local testing:
   - Load datasets programmatically using WMT21Dataset / WMT22Dataset.
   - Verified train/validation/test splits are correctly returned as Python dictionaries of Dataset objects.


-Provides initial support for WMT21 and WMT22 NLP/translation experiments.
-Allows contributors and reviewers to test dataset loading locally or in CI without downloading large datasets.
-Serves as a template to extend to other WMT datasets in the future.

Testing instructions:
```
# Activate your environment
pytest tests/test_wmt21.py -v
pytest tests/test_wmt22.py -v
```
```
from local_datasets.wmt21.wmt21_dataset import WMT21Dataset
from local_datasets.wmt22.wmt22_dataset import WMT22Dataset

# WMT21
wmt21 = WMT21Dataset("local_datasets/wmt21/dummy_data")
ds21 = wmt21.load()
print(ds21["train"][0])
print(ds21["validation"][0])
print(ds21["test"][0])

# WMT22
wmt22 = WMT22Dataset("local_datasets/wmt22/dummy_data")
ds22 = wmt22.load()
print(ds22["train"][0])
print(ds22["validation"][0])
print(ds22["test"][0])
```

Comments:
closing since datasets should be added on https://huggingface.co directly
Add pyarrow's binary view to features

Basically https://github.com/huggingface/datasets/pull/7718 just for binary view instead of string view

Comments:
@lhoestq 🙏 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7795). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: add `num_proc` argument to `Dataset.to_sql`

**Task Done:** 

- Resolve issue #7788 : Add the missing argument mapping in Dataset.to_sql (`src/datasets/arrow_dataset.py`)




Comments:
Hi ! could you also write a test to make sure this works fine ?

(in case there needs to be a special logic to handle the concurrent writes to the database)

Hi @lhoestq 
Done! Let me know if more is needed :)

Hi @lhoestq could you please review my changes?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7791). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`Dataset.to_sql` doesn't utilize `num_proc`

The underlying `SqlDatasetWriter` has `num_proc` as an available argument [here](https://github.com/huggingface/datasets/blob/5dc1a179783dff868b0547c8486268cfaea1ea1f/src/datasets/io/sql.py#L63) , but `Dataset.to_sql()` does not accept it, therefore it is always using one process for the SQL conversion.

Comments:
Fix Audio docstring by removing unsupported mono argument

This PR fixes issue #7745.

Who can review:
@lhoestq 

Comments:
I think we can keep the arg and add the missing torch.mean() in the Audio.decode_example method

> I think we can keep the arg and add the missing torch.mean() in the Audio.decode_example method

Thank you @lhoestq. I will add torch.mean().

fixed by #7840
Support huggingface_hub v0.x and v1.x

Related to https://github.com/huggingface/huggingface_hub/issues/3340.

This PR adapts `datasets` to be compatible with both huggingface_hub v0.x and v1.x. 

In practice nothing else should change (I've checked the codebase). The `HfHubHTTPError` is a base error defined in `huggingface_hub` that inherits from `requests.HTTPError` in v0.x and will inherit from `httpx.HTTPError` in v1.x. It has been introduced ~2 years ago so it's fine to use it right now (i.e. no need to wait for v1.x release or bump minimal version).

Most of the changes have been around the test suite to make sure that tests are passing with both `requests` and `httpx` backends. Mid-term it would be good to completely remove the `requests` dependency from `datasets` but that's an orthogonal topic. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7783). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq could you have a look at this PR please? It makes `datasets` compatible with the upcoming `huggingface_hub` v1.0 release while staying compatible with 0.x. Look at PR description for more details.

The CI is currently failing because of 429 rate limit errors but otherwise everything should be fine (made extensive local and ci tests to ensure that). Let me know if you notice anything weird. PR is ready to be merged "as-is" in my opinion.
BIGPATENT dataset inaccessible (deprecated script loader)

dataset: https://huggingface.co/datasets/NortheasternUniversity/big_patent

When I try to load it with the datasets library, it fails with:

RuntimeError: Dataset scripts are no longer supported, but found big_patent.py


Could you please publish a Parquet/Arrow export of BIGPATENT on the Hugging Face so that it can be accessed with datasets>=4.x.



Comments:
Hi ! I opened https://huggingface.co/datasets/NortheasternUniversity/big_patent/discussions/7 to update the dataset, hopefully it's merged soon !

The dataset now works with `datasets` v4 ! closing this issue
[FIX] force spawning pool for MacOS

This PR gets multiprocessing to work on mac os:

```python
from datasets import load_dataset

ds = load_dataset("fka/awesome-chatgpt-prompts", split="train").take(100)

ds = ds.map(lambda x: x, num_proc=4)

ds.push_to_hub("burtenshaw/dataset-test", num_proc=4)
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7778). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

After more discussions on slack, we can switch the default to spawn.

Let's use multiprocess instead of multiprocessing and maybe add a check to apply this only on Macos ?
push_to_hub not overwriting but stuck in a loop when there are existing commits

### Describe the bug

`get_deletions_and_dataset_card` stuck at error a commit has happened error since push to hub for http error 412 for tag 4.1.0. The error does not exists in 4.0.0. 

### Steps to reproduce the bug

Create code to use push_to_hub, ran twice each time with different content for datasets.Dataset.
The code will stuck in time.sleep loop for `get_deletions_and_dataset_card`. If error is explicitly printed, the error is HTTP 412.

### Expected behavior

New datasets overwrite existing one on repo.

### Environment info

datasets 4.1.0

Comments:
HTTP 412 means a commit happened in the meantime, so `get_deletions_and_dataset_card` has to retry to get the latest version of the dataset card and what files to delete based on the latest version of the dataset repository

Are you running other operations in the dataset repo for your push_to_hub ?

There was only a map() followed by a push_to_hub(). The repo had one prior commit also by using push_to_hub(). The error disappeared when I downgraded datasets to 4.0.0.

It is reproducible if you use finegrained token with Read+Write (Open pull request) access to only that repo.

Ah it was due to the use of requests_cache with POST methods, closing this.
[docs] Fix broken WebDataset link on “Create a video dataset” page

### What
Fix the "WebDataset documentation" link on the Create a video dataset page to point
to the correct section on the video load guide.

### Why
The link currently points to an external repo, but the Hugging Face docs
have an internal "WebDataset" section under video_load.

### How
- docs/source/video_dataset.mdx: updated link to
  `https://huggingface.co/docs/datasets/main/en/video_load#webdataset`

### Issue
Fixes #7699


Comments:
Error processing scalar columns using tensorflow.

`datasets==4.0.0`
```
columns_to_return = ['input_ids','attention_mask', 'start_positions', 'end_positions']
train_ds.set_format(type='tf', columns=columns_to_return)
```
`train_ds`:
```
train_ds type: <class 'datasets.arrow_dataset.Dataset'>, shape: (1000, 9)
columns: ['question', 'sentences', 'answer', 'str_idx', 'end_idx', 'input_ids', 'attention_mask', 'start_positions', 'end_positions']
features:{'question': Value('string'), 'sentences': Value('string'), 'answer': Value('string'), 'str_idx': Value('int64'), 'end_idx': Value('int64'), 'input_ids': List(Value('int32')), 'attention_mask': List(Value('int8')), 'start_positions': Value('int64'), 'end_positions': Value('int64')}
```
`train_ds_tensor = train_ds['start_positions'].to_tensor(shape=(-1,1))` hits the following error:
```
AttributeError: 'Column' object has no attribute 'to_tensor'
```
`tf.reshape(train_ds['start_positions'], shape=[-1,1])` hits the following error:
```
TypeError: Scalar tensor has no `len()`
```

Comments:
Using tf.convert_to_tensor works fine:

```
import tensorflow as tf

start_pos = tf.convert_to_tensor(train_ds['start_positions'], dtype=tf.int64)
start_pos = tf.reshape(start_pos, [-1, 1])
```


Alternatively, using the built-in to_tf_dataset also avoids the issue:

```
train_tf = train_ds.to_tf_dataset(
    columns=['input_ids','attention_mask'],
    label_cols=['start_positions','end_positions'],
    shuffle=True,
    batch_size=32
)
```

```
    start_pos = tf.convert_to_tensor(self._train_ds['start_positions'], dtype=tf.int64)
  File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.13/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.13/site-packages/tensorflow/python/framework/constant_op.py", line 108, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: TypeError: Scalar tensor has no `len()`
Traceback (most recent call last):

  File "/home/khteh/.local/share/virtualenvs/pAIthon-GaqEDHQT/lib/python3.13/site-packages/tensorflow/python/framework/ops.py", line 361, in __len__
    raise TypeError("Scalar tensor has no `len()`")

TypeError: Scalar tensor has no `len()`
```

`to_tf_dataset` works perfectly.
Add support for arrow iterable when concatenating or interleaving

Fixes a case when concatenating or interleaving datasets with `with_format(...)` call was slower.
Details here: https://github.com/huggingface/datasets/issues/6637

@lhoestq I tried to minimize the duplication between iter and iter_arrow methods, not sure if this is against the design, can separate those if needed.

Comments:
Seeing the following numbers on the script shared in the original issue. (MacBook Pro M4)

```
1000it [00:00, 4074.63it/s] # ds_a.with_format("torch")
1000it [00:01, 593.39it/s] # ds_a.shuffle()
1999it [00:03, 594.09it/s] # datasets.interleave_datasets([ds_a, ds_b])
1000it [00:00, 5382.45it/s] # ds_a.shuffle().with_format("torch") <--- Was slow <2it/s
1999it [00:00, 4743.45it/s] # datasets.interleave_datasets([ds_a, ds_b]).with_format("torch") <--- Was slow <2it/s
1999it [00:20, 98.94it/s]  # torch.tensor(example["tensor"])
```


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7771). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq I've implemented the iteration on arrow as separate methods, can you take another look/trigger ci? 

@lhoestq Any idea why the integration tests are failing, is this expected? Anything I can do on my side?

They seem unrelated to your changes. Merging :)
Fix: Correct float feature generation in `generate_examples` 

This PR fixes a bug in the `generate_examples` function where `datasets.Value` features with a `float` dtype were incorrectly generated using `np.random.randint`. This resulted in integer values being cast to float, which is not representative of true floating-point data.


**Key changes include:**

* Added explicit handling for `float` features using `np.random.rand` to generate continuous values.
* Introduced fail-fast type checks for unsupported dtypes to improve robustness.
* Added validation for sequence features to ensure `seq_shapes` is provided.

### Before Fix

Float features were generated incorrectly as integers cast to float:

```text
- Example 0:
- int_feature: 0
- float_feature: 9.0  <-- Incorrect: An integer disguised as a float
- string_feature: The small grey turtle was surprisingly fast...
- seq_feature: [0.3048 0.4291 0.4283]
```

### After Fix

Float features are now correctly generated as continuous numbers in the range [0, 1):
```text
+ Example 0:
+ int_feature: 0
+ float_feature: 0.0183  <-- Correct: A true random float
+ string_feature: The small grey turtle was surprisingly fast...
+ seq_feature: [0.9237 0.7972 0.8526]
```
#### Note: This PR is a follow-up/fix of the previously closed PR #7769 for clarity and context.


Comments:
Hi @lhoestq, just a gentle follow-up on this PR.
Fix: Correct float feature generation in `generate_examples`

This PR fixes a bug in the `generate_examples` function where `datasets.Value` features with a `float` dtype were incorrectly generated using `np.random.randint`. This resulted in integer values being cast to float, which is not representative of true floating-point data.

**Key changes include:**
1.  Added explicit handling for float features using `np.random.rand` to generate continuous values.
2.  Introduced fail-fast type checks for unsupported dtypes to improve robustness.
3.  Added validation for sequence features to ensure `seq_shapes` is provided.


### Before Fix
Float features were generated incorrectly as integers cast to float:

```text
Example 0:
int_feature: 0
float_feature: 9.0  <-- Incorrect: An integer disguised as a float
string_feature: The small grey turtle was surprisingly fast...
seq_feature: [0.3048 0.4291 0.4283]
```
### After Fix
Float features are now correctly generated as continuous numbers in the range [0, 1):
```text

Example 0:
int_feature: 0
float_feature: 0.0183  <-- Correct: A true random float
string_feature: The small grey turtle was surprisingly fast...
seq_feature: [0.9237 0.7972 0.8526]

Comments:
Custom `dl_manager` in `load_dataset`

### Feature request
https://github.com/huggingface/datasets/blob/4.0.0/src/datasets/load.py#L1411-L1418

```

def load_dataset(
    ...
    dl_manager: Optional[DownloadManager] = None,  # add this new argument
    **config_kwargs,
) -> Union[DatasetDict, Dataset, IterableDatasetDict, IterableDataset]:

    ...

    # Create a dataset builder
    builder_instance = load_dataset_builder(
        path=path,
        name=name,
        data_dir=data_dir,
        data_files=data_files,
        cache_dir=cache_dir,
        features=features,
        download_config=download_config,
        download_mode=download_mode,
        revision=revision,
        token=token,
        storage_options=storage_options,
        **config_kwargs,
    )

    # Return iterable dataset in case of streaming
    if streaming:
        return builder_instance.as_streaming_dataset(split=split)

    # Note: This is the revised part
    if dl_manager is None:
        if download_config is None:
            download_config = DownloadConfig(
                cache_dir=builder_instance._cache_downloaded_dir,
                force_download=download_mode == DownloadMode.FORCE_REDOWNLOAD,
                force_extract=download_mode == DownloadMode.FORCE_REDOWNLOAD,
                use_etag=False,
                num_proc=num_proc,
                token=builder_instance.token,
                storage_options=builder_instance.storage_options,
            )  # We don't use etag for data files to speed up the process

        dl_manager = DownloadManager(
            dataset_name=builder_instance.dataset_name,
            download_config=download_config,
            data_dir=builder_instance.config.data_dir,
            record_checksums=(
                builder_instance._record_infos or verification_mode == VerificationMode.ALL_CHECKS
            ),
        )

    # Download and prepare data
    builder_instance.download_and_prepare(
        download_config=download_config,
        download_mode=download_mode,
        verification_mode=verification_mode,
        dl_manager=dl_manager,  # pass the new argument
        num_proc=num_proc,
        storage_options=storage_options,
    )

    ...

```

### Motivation

In my case, I'm hoping to deal with the cache files downloading manually (not using hash filenames and save to another location, or using potential existing local files).

### Your contribution

It's already implemented above. If maintainers think this should be considered, I'll open a PR.

Comments:
Any feedback from maintainers on this?
cast columns to Image/Audio/Video with `storage_options`

### Feature request

Allow `storage_options` to be passed in
1. `cast` related operations (e.g., `cast_columns, cast`)
2. `info` related reading (e.g., `from_dict, from_pandas, from_polars`) together with `info.features`
```python3
import datasets

image_path = "s3://bucket/sample.png"
dataset = datasets.Dataset.from_dict({"image_path": [image_path]})

# dataset = dataset.cast_column("image_path", datasets.Image())  # now works without `storage_options`

# expected behavior
dataset = dataset.cast_column("image_path", datasets.Image(), storage_options={"anon": True})
```

### Motivation

I'm using my own registered fsspec filesystem (s3 with customized local cache support). I need to pass cache folder paths `cache_dirs: list[str]` to the filesystem when I read the remote images (cast from file_paths).

### Your contribution

Could help with a PR at weekends

Comments:
A

1

1

Ok

> ### Feature request
> Allow `storage_options` to be passed in
> 
> 1. `cast` related operations (e.g., `cast_columns, cast`)
> 2. `info` related reading (e.g., `from_dict, from_pandas, from_polars`) together with `info.features`
> 
> import datasets
> 
> image_path = "s3://bucket/sample.png"
> dataset = datasets.Dataset.from_dict({"image_path": [image_path]})
> 
> # dataset = dataset.cast_column("image_path", datasets.Image())  # now works without `storage_options`
> 
> # expected behavior
> dataset = dataset.cast_column("image_path", datasets.Image(), storage_options={"anon": True})
> ### Motivation
> I'm using my own registered fsspec filesystem (s3 with customized local cache support). I need to pass cache folder paths `cache_dirs: list[str]` to the filesystem when I read the remote images (cast from file_paths).
> 
> ### Your contribution
> Could help with a PR at weekends



>

#7910
polars dataset cannot cast column to Image/Audio/Video

### Describe the bug

`from_polars` dataset cannot cast column to Image/Audio/Video, while it works on `from_pandas` and `from_dict`

### Steps to reproduce the bug

```python3
import datasets
import pandas as pd
import polars as pl

image_path = "./sample.png"

# polars
df = pl.DataFrame({"image_path": [image_path]})
dataset = datasets.Dataset.from_polars(df)
dataset = dataset.cast_column("image_path", datasets.Image())

# # raises Error
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from large_string to struct using function cast_struct


# pandas 
df = pd.DataFrame({"image_path": [image_path]})
dataset = datasets.Dataset.from_pandas(df)
dataset = dataset.cast_column("image_path", datasets.Image())

# # pass
{'image_path': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=338x277 at 0x7FBA719D4050>}


# dict
dataset = datasets.Dataset.from_dict({"image_path": [image_path]})
dataset = dataset.cast_column("image_path", datasets.Image())

# # pass
{'image_path': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=338x277 at 0x7FBA719D4050>}
```


### Expected behavior

`from_polars` case shouldn't raise error and have the same outputs as `from_pandas` and `from_dict`

### Environment info

```
# Name                    Version                   Build  Channel
datasets                  4.0.0                    pypi_0    pypi
pandas                    2.3.1                    pypi_0    pypi
polars                    1.32.3                   pypi_0    pypi
```

Comments:
I fixed this with a combination of `to_dict` and `from_dict`:

```py
datasets.Dataset.from_dict(df.to_dict(as_series=False))
```

@samuelstevens Yeah, I'm using similar workaround as well. But it would be ideal if we can avoid the copy.