josteece/github-issues huggingface dataset:
Fix ArrowInvalid for large variables
Opened by CloseChoice (open)

Fixes #7821

In addition to the solution proposed in the issue, I encountered we also need to support 64bit types when writing using the `RecordBatchStreamWriter`. 

Not sure if we want to create such large objects in the CI, but this is the only way to test that the issue is fixed, therefore I added the unit test.

Comments:
@lhoestq : from what I see in the CI logs ( tests/test_arrow_dataset.py::test_map_int32_overflow 
/opt/hostedtoolcache/Python/3.9.23/x64/lib/python3.9/site-packages/multiprocess/resource_tracker.py:219: UserWarning: resource_tracker: There appear to be 24 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '), see [here](https://github.com/huggingface/datasets/actions/runs/18720027796/job/53591652242#step:13:6530), the test crashes probably since it consumes too much memory, it works locally for me though. 
  
  Shall we keep the test, remove ti, or skip it, or is there a specific pytest flag you have for slow/memory-consuming tests I can add, so that it gets skipped automatically on the CI but might be run locally?
  
  EDIT: I added the `high_memory` marker (see [here](https://github.com/huggingface/datasets/pull/7825/files#diff-c480dc36aca482db963c43ee66ba21705111f62f826352c96558ef5ed1619791R4789)), so this will not be run in the CI, which is not ideal but I guess the way if we want to keep the test
Fix batch_size default description in to_polars docstrings
Opened by albertvillanova (closed)

Fix batch_size default description in `to_polars` docstrings.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7824). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix random seed on shuffle and interleave_datasets
Opened by CloseChoice (closed)

closes #7567

Add `shift_rngs` method to `ExamplesIterable` that is called directly after sharding. If a generator is available (not the case for all subclasses) we update the seed of the generator by shifting by the worker_id.

~This is just the fix for `shuffle`, in the corresponding issue `interleave_datasets` is mentioned as well, which won't be fixed with this approach.~

EDIT: This is a fix for `shuffle` and `interleave_datasets`. Adding recursivity to `shift_rngs` solved `interleave_datasets` as well. Not sure though if this is completely safe or if we could destroy something with that. I don't think so but could be wrong and appreciate some guidance from the maintainers. I also checked, on a single_worker we are always handing over `index=0` so that case preserves the seed the user specified. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7823). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Cool ! To avoid unwanted side effects it could be implemented for every class instead of using `hasattr`, and also return a new instance instead of modifying it in place. Some other functions work like that already like `shuffle_data_sources` and `shard_data_sources`.

Returning a new object is actually quite important otherwise iterating on the dataset multiple times would shift the RNGs every time

> Cool ! To avoid unwanted side effects it could be implemented for every class instead of using `hasattr`, and also return a new instance instead of modifying it in place. Some other functions work like that already like `shuffle_data_sources` and `shard_data_sources`.
> 
> Returning a new object is actually quite important otherwise iterating on the dataset multiple times would shift the RNGs every time

Thanks for the review. 

I managed to return instances of `_BaseExamplesIterable` instead of modifying inplace and this shifts the seed correctly even in the case of multiple iterations (see additional test). But I kept the `hasattr` for the reason that we have chained ex_iterables (so an ex_iterable can have other ex_iterables) and for the child ex_iterable it is not clear if they do have a generator attribute (e.g. `ArrowExamplesIterable`, `RebatchedArrowExamplesIterable`, `SelectColumnsIterable`, `StepExamplesIterable`, `SkipExamplesIterable`, `RepeatExamplesIterable` all don't have that). We could implement a generic method `shift_rngs` on the baseclass and then just return the ex_iterable and get rid of `hasattr` but I'd prefer the `hasattr` solution here. Let me now what you think about that.
Keep hffs cache in workers when streaming
Opened by lhoestq (closed)

(and also reorder the hffs args to improve caching)

When using `DataLoader(iterable_dataset, num_workers=...)` the dataset is pickled and passed to the worker. However previously the resulting dataset would be in a process with an empty hffs cache. By keeping the cache attached to `IterableDataset`, the cached hffs instances are pickled with the dataset and re-populates the cache in the DataLoader workers

this requires https://github.com/huggingface/huggingface_hub/pull/3443 to work effectively though, otherwise the unpickled hffs cache would start empty


cc @andimarafioti @ltmeyer

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7820). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Cannot download opus dataset
Opened by liamsun2019 (open)

When I tried to download opus_books using:

from datasets import load_dataset
dataset = load_dataset("Helsinki-NLP/opus_books")

I got the following errors:
FileNotFoundError: Couldn't find any data file at /workspace/Helsinki-NLP/opus_books. Couldn't find 'Helsinki-NLP/opus_books' on the Hugging Face Hub either: LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

I also tried:
dataset = load_dataset("opus_books", "en-zh")

and the errors remain the same. However, I can download "mlabonne/FineTome-100k" successfully. 
My datasets is version 4.2.0

Any clues? Big thanks.

Comments:
Hi ! it seems "en-zh" doesn't exist for this dataset

You can see the list of subsets here: https://huggingface.co/datasets/Helsinki-NLP/opus_books
train_test_split and stratify breaks with Numpy 2.0
Opened by davebulaval (closed)

### Describe the bug

As stated in the title, since Numpy changed in version >2.0 with copy, the stratify parameters break.

e.g. `all_dataset.train_test_split(test_size=0.2,stratify_by_column="label")` returns a Numpy error.

It works if you downgrade Numpy to a version lower than 2.0.

### Steps to reproduce the bug

1. Numpy > 2.0
2. `all_dataset.train_test_split(test_size=0.2,stratify_by_column="label")`

### Expected behavior

It returns a stratified split as per the results of Numpy < 2.0

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-6.8.0-85-generic-x86_64-with-glibc2.35
- Python version: 3.13.7
- Huggingface_hub version: 0.34.4
- PyArrow version: 19.0.0
- Pandas version: 2.3.2


Comments:
I can't reproduce this. Could you pls provide an example with a public dataset/artificial dataset and show how you loaded that?

This works for me:

```python
import numpy as np
from datasets import Dataset, Features, ClassLabel, Value

data = {"text": [f"sample_{i}" for i in range(100)], "label": [i % 3 for i in range(100)]}
features = Features({"text": Value("string"),
                     "label": ClassLabel(names=["class_0", "class_1", "class_2"])})
dataset = Dataset.from_dict(data, features=features)
splits = dataset.train_test_split(test_size=0.2, stratify_by_column="label")
print(f"Success with numpy {np.__version__}")
```
but it also works for `numpy<2`

@davebulaval tried with numpy 2.3.4, and maybe i have successfully reproduced the bug!
```
ValueError: Unable to avoid copy while creating an array as requested.
If using `np.array(obj, copy=False)` replace it with `np.asarray(obj)` to allow a copy when needed (no behavior change in NumPy 1.x).
For more details, see https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword.
```

Also i downgraded to numpy 1.26.4
```
(hf-reproduce) F:\Python\Machine learning\reproducing>python repro.py
DatasetDict({
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 16
    })
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 4
    })
})
```

Also @CloseChoice The bug only appears in cases where the Arrow array cannot be represented as a contiguous NumPy array without copying.

So closing the discussion loop here - 

The error occurs because `train_test_split(..., stratify_by_column=...)` attempts to convert
an Arrow column to a NumPy array using `np.array(..., copy=False)`.

In NumPy <2.0 this silently allowed a copy if needed.
In NumPy ≥2.0 this raises:
ValueError: Unable to avoid copy while creating an array as requested.

This only happens when the Arrow column is not contiguous in memory, which explains
why some datasets reproduce it and others do not.
disable_progress_bar() not working as expected
Opened by windmaple (closed)

### Describe the bug

Hi,

I'm trying to load a dataset on Kaggle TPU image. There is some known compat issue with progress bar on Kaggle, so I'm trying to disable the progress bar globally. This does not work as you can see in [here](https://www.kaggle.com/code/windmaple/hf-datasets-issue).

In contract, disabling progress bar for snapshot_download() works as expected as in [here](https://www.kaggle.com/code/windmaple/snapshot-download-error).

### Steps to reproduce the bug

See this [notebook](https://www.kaggle.com/code/windmaple/hf-datasets-issue). 

There is sth. wrong with `shell_paraent`.

### Expected behavior

The downloader should disable progress bar and move forward w/ no error.

### Environment info

The latest version as I did:

!pip install -U datasets ipywidgets ipykernel

Comments:
@xianbaoqian 

Closing this one since it's a Xet issue.
Allow streaming hdf5 files
Opened by lhoestq (closed)

Add streaming support after https://github.com/huggingface/datasets/pull/7690, cc @klamike :)

## Details

in `datasets` loaders, `open()` is extended to work with files that are on disk but also on HF. Files on HF are streamed using HTTP range requests using the `HfFileSystem` implementation in the `huggingface_hub` library.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7814). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Parquet: add `on_bad_file` argument to error/warn/skip bad files
Opened by lhoestq (closed)

```python
from datasets import load_dataset

on_bad_file = "error"  # default
# on_bad_file = "warn"  # warn and skip bad file
# on_bad_file = "skip"  # skip bad file
ds = load_dataset(parquet_dataset_id, on_bad_file=on_bad_file)
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7806). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Less api calls when resolving data_files
Opened by lhoestq (closed)

There ~10 unnecessary `/api/datasets/user/dataset/revision`calls due to multithreading in data files resolution.

I disabled multithreading, which was actually not useful anymore since `HfFileSystem` has been using caching for a while now.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7805). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Docs] Missing documentation for `Dataset.from_dict`
Opened by aaronshenhao (open)

Documentation link: https://huggingface.co/docs/datasets/en/package_reference/main_classes

Link to method (docstring present): https://github.com/huggingface/datasets/blob/6f2502c5a026caa89839713f6f7c8b958e5e83eb/src/datasets/arrow_dataset.py#L1029

The docstring is present for the function, but seems missing from the official documentation for the `Dataset` class on HuggingFace.

The method in question:
```python
    @classmethod
    def from_dict(
        cls,
        mapping: dict,
        features: Optional[Features] = None,
        info: Optional[DatasetInfo] = None,
        split: Optional[NamedSplit] = None,
    ) -> "Dataset":
        """
        Convert `dict` to a `pyarrow.Table` to create a [`Dataset`].

        Important: a dataset created with from_dict() lives in memory
        and therefore doesn't have an associated cache directory.
        This may change in the future, but in the meantime if you
        want to reduce memory usage you should write it back on disk
        and reload using e.g. save_to_disk / load_from_disk.

        Args:
            mapping (`Mapping`):
                Mapping of strings to Arrays or Python lists.
            features ([`Features`], *optional*):
                Dataset features.
            info (`DatasetInfo`, *optional*):
                Dataset information, like description, citation, etc.
            split (`NamedSplit`, *optional*):
                Name of the dataset split.

        Returns:
            [`Dataset`]
        """
```

Comments:
I'd like to work on this documentation issue.

Hi I'd like to work on this. I can see the docstring is already in the code. 
Could you confirm:
1. Is this still available?
2. Should I add this to the main_classes.md file, or is there a specific 
   documentation file I should update?
3. Are there any formatting guidelines I should follow?

I'm new to contributing but eager to learn!
Add parquet scan options and docs
Opened by lhoestq (closed)

I added scan options, useful to control buffering and caching when streaming and docs, including how to select a subset of columns and apply filters

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7801). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Define CI future
Opened by lhoestq (closed)

this should fix the CI which currently uses transformers on 3.9 while it's now unsupported

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7799). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Audio dataset is not decoding on 4.1.1
Opened by thewh1teagle (open)

### Describe the bug

The audio column remain as non-decoded objects even when accessing them.

```python
dataset = load_dataset("MrDragonFox/Elise", split = "train")
dataset[0] # see that it doesn't show 'array' etc...
```

Works fine with `datasets==3.6.0`

Followed the docs in 

- https://huggingface.co/docs/datasets/en/audio_load

### Steps to reproduce the bug

```python
dataset = load_dataset("MrDragonFox/Elise", split = "train")
dataset[0] # see that it doesn't show 'array' etc...
```


### Expected behavior

It should decode when accessing the elemenet

### Environment info

4.1.1
ubuntu 22.04


Related

- https://github.com/huggingface/datasets/issues/7707

Comments:
Previously (datasets<=3.6.0), audio columns were decoded automatically when accessing a row. Now, for performance reasons, audio decoding is lazy by default: you just see the file path unless you explicitly cast the column to Audio.

Here’s the fix (following the current [datasets audio docs](https://huggingface.co/docs/datasets/en/audio_load)
):

```
from datasets import load_dataset, Audio

dataset = load_dataset("MrDragonFox/Elise", split="train")

# Explicitly decode the audio column
dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))

print(dataset[0]["audio"])
# {'path': '...', 'array': array([...], dtype=float32), 'sampling_rate': 16000}
```

@haitam03-yo's comment is right that the data is not decoded by default anymore indeed, but here is how it works in practice now:

From `datasets` v4, audio data are read as [AudioDecoder](https://meta-pytorch.org/torchcodec/0.4/generated/torchcodec.decoders.AudioDecoder.html) objects from torchcodec. This doesn't decode the data by default, but you can call `audio.get_all_samples()` to decode the audio.

See the documentation on how to process audio data here: https://huggingface.co/docs/datasets/audio_process

To resolve this, you need to explicitly cast the audio column to the Audio feature. This will decode the audio data and make it accessible as an array. Here is the corrected code snippet


from datasets import load_dataset, Audio

# Load your dataset
dataset = load_dataset("MrDragonFox/Elise", split="train")

# Explicitly cast the 'audio' column to the Audio feature
dataset = dataset.cast_column("audio", Audio(sampling_rate=16_000))

# Now you can access the decoded audio array
print(dataset[0]["audio"])

By adding the cast_column step, you are telling the datasets library to decode the audio data with the specified sampling rate, and you will then be able to access the audio array as you were used to in previous versions.
Datasets: Add WMT21 & WMT22 loaders (basic TSV loaders, sample data, tests)
Opened by tanisha-samant (closed)

- Implemented TSV-based dataset loaders:
   - WMT21Dataset (local_datasets/wmt21/wmt21_dataset.py)
   - WMT22Dataset (local_datasets/wmt22/wmt22_dataset.py)
   These classes load source-target pairs from .tsv files for train, validation, and test splits.
- Created sample dummy data for both datasets:
   - dummy_data/train.tsv, dummy_data/validation.tsv, dummy_data/test.tsv
   - Includes a few realistic example lines to allow CI and local tests to pass without downloading full datasets.
- Added automated tests for robust validation:
   - tests/test_wmt21.py and tests/test_wmt22.py
   - Checks that all splits load correctly, empty lines are ignored, and the number of examples matches the number of lines in the .tsv files.
   - Edge cases handled: empty lines, malformed lines, extra tabs.
- Added README.md files for both datasets:
   - Provides dataset structure, usage instructions, and placeholders for citation & license information.
   - Ensures that other developers and reviewers can understand dataset usage immediately.
- Ensured easy local testing:
   - Load datasets programmatically using WMT21Dataset / WMT22Dataset.
   - Verified train/validation/test splits are correctly returned as Python dictionaries of Dataset objects.


-Provides initial support for WMT21 and WMT22 NLP/translation experiments.
-Allows contributors and reviewers to test dataset loading locally or in CI without downloading large datasets.
-Serves as a template to extend to other WMT datasets in the future.

Testing instructions:
```
# Activate your environment
pytest tests/test_wmt21.py -v
pytest tests/test_wmt22.py -v
```
```
from local_datasets.wmt21.wmt21_dataset import WMT21Dataset
from local_datasets.wmt22.wmt22_dataset import WMT22Dataset

# WMT21
wmt21 = WMT21Dataset("local_datasets/wmt21/dummy_data")
ds21 = wmt21.load()
print(ds21["train"][0])
print(ds21["validation"][0])
print(ds21["test"][0])

# WMT22
wmt22 = WMT22Dataset("local_datasets/wmt22/dummy_data")
ds22 = wmt22.load()
print(ds22["train"][0])
print(ds22["validation"][0])
print(ds22["test"][0])
```

Comments:
closing since datasets should be added on https://huggingface.co directly
Docs: fix typo, improve readability, add code comments
Opened by tanisha-samant (open)

What I did:
- Fixed a small typo in README to improve clarity
- Fixed repeated word "frameworks frameworks"
- Split long paragraphs into shorter sentences for readability
- Added # Example comments before code blocks for clarity

Why:
- Helps new users avoid confusion

How I tested:
- Checked locally in Markdown preview

Comments:
Add pyarrow's binary view to features
Opened by delta003 (closed)

Basically https://github.com/huggingface/datasets/pull/7718 just for binary view instead of string view

Comments:
@lhoestq 🙏 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7795). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: add `num_proc` argument to `Dataset.to_sql`
Opened by EricSaikali (open)

**Task Done:** 

- Resolve issue #7788 : Add the missing argument mapping in Dataset.to_sql (`src/datasets/arrow_dataset.py`)




Comments:
Hi ! could you also write a test to make sure this works fine ?

(in case there needs to be a special logic to handle the concurrent writes to the database)

Hi @lhoestq 
Done! Let me know if more is needed :)

Hi @lhoestq could you please review my changes?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7791). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
`Dataset.to_sql` doesn't utilize `num_proc`
Opened by tcsmaster (open)

The underlying `SqlDatasetWriter` has `num_proc` as an available argument [here](https://github.com/huggingface/datasets/blob/5dc1a179783dff868b0547c8486268cfaea1ea1f/src/datasets/io/sql.py#L63) , but `Dataset.to_sql()` does not accept it, therefore it is always using one process for the SQL conversion.

Comments:
Fix Audio docstring by removing unsupported mono argument
Opened by tanuj-rai (closed)

This PR fixes issue #7745.

Who can review:
@lhoestq 

Comments:
I think we can keep the arg and add the missing torch.mean() in the Audio.decode_example method

> I think we can keep the arg and add the missing torch.mean() in the Audio.decode_example method

Thank you @lhoestq. I will add torch.mean().

fixed by #7840
Support huggingface_hub v0.x and v1.x
Opened by Wauplin (closed)

Related to https://github.com/huggingface/huggingface_hub/issues/3340.

This PR adapts `datasets` to be compatible with both huggingface_hub v0.x and v1.x. 

In practice nothing else should change (I've checked the codebase). The `HfHubHTTPError` is a base error defined in `huggingface_hub` that inherits from `requests.HTTPError` in v0.x and will inherit from `httpx.HTTPError` in v1.x. It has been introduced ~2 years ago so it's fine to use it right now (i.e. no need to wait for v1.x release or bump minimal version).

Most of the changes have been around the test suite to make sure that tests are passing with both `requests` and `httpx` backends. Mid-term it would be good to completely remove the `requests` dependency from `datasets` but that's an orthogonal topic. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7783). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq could you have a look at this PR please? It makes `datasets` compatible with the upcoming `huggingface_hub` v1.0 release while staying compatible with 0.x. Look at PR description for more details.

The CI is currently failing because of 429 rate limit errors but otherwise everything should be fine (made extensive local and ci tests to ensure that). Let me know if you notice anything weird. PR is ready to be merged "as-is" in my opinion.
BIGPATENT dataset inaccessible (deprecated script loader)
Opened by ishmaifan (closed)

dataset: https://huggingface.co/datasets/NortheasternUniversity/big_patent

When I try to load it with the datasets library, it fails with:

RuntimeError: Dataset scripts are no longer supported, but found big_patent.py


Could you please publish a Parquet/Arrow export of BIGPATENT on the Hugging Face so that it can be accessed with datasets>=4.x.



Comments:
Hi ! I opened https://huggingface.co/datasets/NortheasternUniversity/big_patent/discussions/7 to update the dataset, hopefully it's merged soon !

The dataset now works with `datasets` v4 ! closing this issue