noamaanMulla-03/datasets-issues huggingface dataset:
Keep hffs cache in workers when streaming

(and also reorder the hffs args to improve caching)

When using `DataLoader(iterable_dataset, num_workers=...)` the dataset is pickled and passed to the worker. However previously the resulting dataset would be in a process with an empty hffs cache. By keeping the cache attached to `IterableDataset`, the cached hffs instances are pickled with the dataset and re-populates the cache in the DataLoader workers

this requires https://github.com/huggingface/huggingface_hub/pull/3443 to work effectively though, otherwise the unpickled hffs cache would start empty


cc @andimarafioti @ltmeyer


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7820). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Docs] Missing documentation for `Dataset.from_dict`

Documentation link: https://huggingface.co/docs/datasets/en/package_reference/main_classes

Link to method (docstring present): https://github.com/huggingface/datasets/blob/6f2502c5a026caa89839713f6f7c8b958e5e83eb/src/datasets/arrow_dataset.py#L1029

The docstring is present for the function, but seems missing from the official documentation for the `Dataset` class on HuggingFace.

The method in question:
```python
    @classmethod
    def from_dict(
        cls,
        mapping: dict,
        features: Optional[Features] = None,
        info: Optional[DatasetInfo] = None,
        split: Optional[NamedSplit] = None,
    ) -> "Dataset":
        """
        Convert `dict` to a `pyarrow.Table` to create a [`Dataset`].

        Important: a dataset created with from_dict() lives in memory
        and therefore doesn't have an associated cache directory.
        This may change in the future, but in the meantime if you
        want to reduce memory usage you should write it back on disk
        and reload using e.g. save_to_disk / load_from_disk.

        Args:
            mapping (`Mapping`):
                Mapping of strings to Arrays or Python lists.
            features ([`Features`], *optional*):
                Dataset features.
            info (`DatasetInfo`, *optional*):
                Dataset information, like description, citation, etc.
            split (`NamedSplit`, *optional*):
                Name of the dataset split.

        Returns:
            [`Dataset`]
        """
```


I'd like to work on this documentation issue.

Hi I'd like to work on this. I can see the docstring is already in the code. 
Could you confirm:
1. Is this still available?
2. Should I add this to the main_classes.md file, or is there a specific 
   documentation file I should update?
3. Are there any formatting guidelines I should follow?

I'm new to contributing but eager to learn!
Cannot download opus dataset

When I tried to download opus_books using:

from datasets import load_dataset
dataset = load_dataset("Helsinki-NLP/opus_books")

I got the following errors:
FileNotFoundError: Couldn't find any data file at /workspace/Helsinki-NLP/opus_books. Couldn't find 'Helsinki-NLP/opus_books' on the Hugging Face Hub either: LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

I also tried:
dataset = load_dataset("opus_books", "en-zh")

and the errors remain the same. However, I can download "mlabonne/FineTome-100k" successfully. 
My datasets is version 4.2.0

Any clues? Big thanks.


Hi ! it seems "en-zh" doesn't exist for this dataset

You can see the list of subsets here: https://huggingface.co/datasets/Helsinki-NLP/opus_books
Add option to ignore keys/columns when loading a dataset from jsonl(or any other data format)

### Feature request

Hi, I would like the option to ignore keys/columns when loading a dataset from files (e.g. jsonl).

### Motivation

I am working on a dataset which is built on jsonl. It seems the dataset is unclean and a column has different types in each row. I can't clean this or remove the column (It is not my data and it is too big for me to clean and save on my own hardware). 
I would like the option to just ignore this column when using `load_dataset`, since i don't need it.
I tried to look if this is already possible but couldn't find a solution. if there is I would love some help. If it is not currently possible, I would love this feature

### Your contribution

I don't think I can help this time, unfortunately.

Labels: name: enhancement

Good point, I'd be in favor of having the `columns` argument in `JsonConfig` (and the others) to align with `ParquetConfig` to let users choose which columns to load and ignore the rest

Is it possible to ignore columns when using parquet? 

Yes, you can pass `columns=...` to load_dataset to select which columns to load, and it is passed to `ParquetConfig` :)

Ok, i didn't know that. 
Anyway, it would be good to add this to others

Hi @lhoestq 

I'd like to take this up!

As you suggested, I’ll extend the support for the columns parameter (currently used in ParquetConfig) to JsonConfig as well. This will allow users to selectively load specific keys/columns from .jsonl (or .json) files and ignore the rest — solving the type inconsistency issues in unclean datasets.

Hi @avishaiElmakies and @lhoestq 

Just wanted to let you know that this is now implemented in #7594
As suggested, support for the `columns=...` argument (previously available for Parquet) has now been extended to **JSON and JSONL** loading via `load_dataset(...)`. You can now load only specific keys/columns and skip the rest — which should help in cases where some fields are unclean, inconsistent, or just unnecessary.

### ✅ Example:

```python
from datasets import load_dataset

dataset = load_dataset("json", data_files="your_data.jsonl", columns=["id", "title"])
print(dataset["train"].column_names)
# Output: ['id', 'title']
```

### 🔧 Summary of changes:

* Added `columns: Optional[List[str]]` to `JsonConfig`
* Updated `_generate_tables()` to filter selected columns
* Forwarded `columns` argument from `load_dataset()` to the config
* Added test case to validate behavior

Let me know if you'd like the same to be added for CSV or others as a follow-up — happy to help.

@ArjunJagdale this looks great! Thanks!
I believe that every format that is supported by `datasets` should probably have this feature since it is very useful and will streamline the api (people will know that they can just use `columns` to select the columns they want, and it will  not be dependent on the data format) 

Thanks @avishaiElmakies — totally agree, making `columns=...` support consistent across all formats would be really helpful for users.

#Codex Fix

#Codex Fix
Add custom fingerprint support to `from_generator`

This PR adds `dataset_id_suffix` parameter to 'Dataset.from_generator' function.

`Dataset.from_generator` function passes all of its arguments to `BuilderConfig.create_config_id`, including generator function itself. `BuilderConfig.create_config_id` function tries to hash all the args, which can take a large amount of time or even cause MemoryError if the dataset processed in a generator function is large enough.

This PR allows user to pass a custom fingerprint (`dataset_id_suffix`) to be used as a suffix in a dataset name instead of the one generated by hashing the args.

This PR is a possible solution of #7513 


This is great !

What do you think of passing `config_id=` directly to the builder instead of just the suffix ? This would be a power user argument though, or for internal use. And in from_generator the new argument can be `fingerprint=` as in `Dataset.__init__()`

The `config_id` can be defined using something like `config_id = "default-fingerprint=" + fingerprint`

I feel ike this could make the Dataset API more coherent if we avoid introducing a new argument while we can juste use `fingerprint=`

@lhoestq could you please re-review the changes I made?

@lhoestq ping
I also added a simple test for the `fingerprint` parameter

@lhoestq could you please review the PR? I implemented the requested changes and added a test for the added `fingerprint` arg

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7533). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix random seed on shuffle and interleave_datasets

closes #7567

Add `shift_rngs` method to `ExamplesIterable` that is called directly after sharding. If a generator is available (not the case for all subclasses) we update the seed of the generator by shifting by the worker_id.

~This is just the fix for `shuffle`, in the corresponding issue `interleave_datasets` is mentioned as well, which won't be fixed with this approach.~

EDIT: This is a fix for `shuffle` and `interleave_datasets`. Adding recursivity to `shift_rngs` solved `interleave_datasets` as well. Not sure though if this is completely safe or if we could destroy something with that. I don't think so but could be wrong and appreciate some guidance from the maintainers. I also checked, on a single_worker we are always handing over `index=0` so that case preserves the seed the user specified. 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7823). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Cool ! To avoid unwanted side effects it could be implemented for every class instead of using `hasattr`, and also return a new instance instead of modifying it in place. Some other functions work like that already like `shuffle_data_sources` and `shard_data_sources`.

Returning a new object is actually quite important otherwise iterating on the dataset multiple times would shift the RNGs every time

> Cool ! To avoid unwanted side effects it could be implemented for every class instead of using `hasattr`, and also return a new instance instead of modifying it in place. Some other functions work like that already like `shuffle_data_sources` and `shard_data_sources`.
> 
> Returning a new object is actually quite important otherwise iterating on the dataset multiple times would shift the RNGs every time

Thanks for the review. 

I managed to return instances of `_BaseExamplesIterable` instead of modifying inplace and this shifts the seed correctly even in the case of multiple iterations (see additional test). But I kept the `hasattr` for the reason that we have chained ex_iterables (so an ex_iterable can have other ex_iterables) and for the child ex_iterable it is not clear if they do have a generator attribute (e.g. `ArrowExamplesIterable`, `RebatchedArrowExamplesIterable`, `SelectColumnsIterable`, `StepExamplesIterable`, `SkipExamplesIterable`, `RepeatExamplesIterable` all don't have that). We could implement a generic method `shift_rngs` on the baseclass and then just return the ex_iterable and get rid of `hasattr` but I'd prefer the `hasattr` solution here. Let me now what you think about that.
[DOCS][minor] TIPS paragraph not compiled in docs/stream

In the client documentation, the markdown 'TIP' paragraph for paragraph in docs/stream#shuffle is not well executed — not as the other in the same page / while markdown is correctly considering it.

Documentation:
https://huggingface.co/docs/datasets/v4.3.0/en/stream#shuffle:~:text=%5B!TIP%5D%5BIterableDataset.shuffle()%5D(/docs/datasets/v4.3.0/en/package_reference/main_classes%23datasets.IterableDataset.shuffle)%20will%20also%20shuffle%20the%20order%20of%20the%20shards%20if%20the%20dataset%20is%20sharded%20into%20multiple%20files.

Github source:
https://github.com/huggingface/datasets/blob/main/docs/source/stream.mdx#:~:text=Casting%20only%20works%20if%20the%20original%20feature%20type%20and%20new%20feature%20type%20are%20compatible.%20For%20example%2C%20you%20can%20cast%20a%20column%20with%20the%20feature%20type%20Value(%27int32%27)%20to%20Value(%27bool%27)%20if%20the%20original%20column%20only%20contains%20ones%20and%20zeros.
Fix: Properly render [!TIP] block in stream.shuffle documentation

The second line starting with the bracket doesn't properly render on huggingface/docs. Added "> " to address it.

In the client documentation, the markdown 'TIP' paragraph in docs/stream#shuffle is not well executed, not as the other in the same page / while markdown is correctly considering it.

Documentation:
https://huggingface.co/docs/datasets/v4.3.0/en/stream#shuffle:~:text=%5B!TIP%5D%5BIterableDataset.shuffle()%5D(/docs/datasets/v4.3.0/en/package_reference/main_classes%23datasets.IterableDataset.shuffle)%20will%20also%20shuffle%20the%20order%20of%20the%20shards%20if%20the%20dataset%20is%20sharded%20into%20multiple%20files.

Github source:
https://github.com/huggingface/datasets/blob/main/docs/source/stream.mdx#:~:text=Casting%20only%20works%20if%20the%20original%20feature%20type%20and%20new%20feature%20type%20are%20compatible.%20For%20example%2C%20you%20can%20cast%20a%20column%20with%20the%20feature%20type%20Value(%27int32%27)%20to%20Value(%27bool%27)%20if%20the%20original%20column%20only%20contains%20ones%20and%20zeros.
train_test_split and stratify breaks with Numpy 2.0

### Describe the bug

As stated in the title, since Numpy changed in version >2.0 with copy, the stratify parameters break.

e.g. `all_dataset.train_test_split(test_size=0.2,stratify_by_column="label")` returns a Numpy error.

It works if you downgrade Numpy to a version lower than 2.0.

### Steps to reproduce the bug

1. Numpy > 2.0
2. `all_dataset.train_test_split(test_size=0.2,stratify_by_column="label")`

### Expected behavior

It returns a stratified split as per the results of Numpy < 2.0

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-6.8.0-85-generic-x86_64-with-glibc2.35
- Python version: 3.13.7
- Huggingface_hub version: 0.34.4
- PyArrow version: 19.0.0
- Pandas version: 2.3.2



I can't reproduce this. Could you pls provide an example with a public dataset/artificial dataset and show how you loaded that?

This works for me:

```python
import numpy as np
from datasets import Dataset, Features, ClassLabel, Value

data = {"text": [f"sample_{i}" for i in range(100)], "label": [i % 3 for i in range(100)]}
features = Features({"text": Value("string"),
                     "label": ClassLabel(names=["class_0", "class_1", "class_2"])})
dataset = Dataset.from_dict(data, features=features)
splits = dataset.train_test_split(test_size=0.2, stratify_by_column="label")
print(f"Success with numpy {np.__version__}")
```
but it also works for `numpy<2`

@davebulaval tried with numpy 2.3.4, and maybe i have successfully reproduced the bug!
```
ValueError: Unable to avoid copy while creating an array as requested.
If using `np.array(obj, copy=False)` replace it with `np.asarray(obj)` to allow a copy when needed (no behavior change in NumPy 1.x).
For more details, see https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword.
```

Also i downgraded to numpy 1.26.4
```
(hf-reproduce) F:\Python\Machine learning\reproducing>python repro.py
DatasetDict({
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 16
    })
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 4
    })
})
```

Also @CloseChoice The bug only appears in cases where the Arrow array cannot be represented as a contiguous NumPy array without copying.

So closing the discussion loop here - 

The error occurs because `train_test_split(..., stratify_by_column=...)` attempts to convert
an Arrow column to a NumPy array using `np.array(..., copy=False)`.

In NumPy <2.0 this silently allowed a copy if needed.
In NumPy ≥2.0 this raises:
ValueError: Unable to avoid copy while creating an array as requested.

This only happens when the Arrow column is not contiguous in memory, which explains
why some datasets reproduce it and others do not.
Reimplemented partial split download support (revival of #6832)

(revival of #6832)

https://github.com/huggingface/datasets/pull/7648#issuecomment-3084050130

Close https://github.com/huggingface/datasets/issues/4101, and more

---
### PR under work!!!!



 Mario’s Patch (in PR #6832):
```
def _make_split_generators_kwargs(self, prepare_split_kwargs):
    # Pass `pipeline` into `_split_generators()` from `prepare_split_kwargs` if
    # it's in the call signature of `_split_generators()`.
    # This allows for global preprocessing in beam.
    split_generators_kwargs = {}
    if "pipeline" in inspect.signature(self._split_generators).parameters:
        split_generators_kwargs["pipeline"] = prepare_split_kwargs["pipeline"]
    split_generators_kwargs.update(super()._make_split_generators_kwargs(prepare_split_kwargs))
    return split_generators_kwargs
```

In the latest main(in my fork and og repo's main):
```
def _make_split_generators_kwargs(self, prepare_split_kwargs):
    """Get kwargs for `self._split_generators()` from `prepare_split_kwargs`."""
    splits = prepare_split_kwargs.pop("splits", None)
    if self._supports_partial_generation():
        return {"splits": splits}
    return {}
```
It enables passing splits into _split_generators() only for builders that support it(if i am not wrong..). So ignored Beam logic for now!

Awesome ! btw we can modify the GeneratorBasedBuilder and ArrowBasedBuilder if needed now that custom loading scripts are not supported anymore :)

I'll review this in a bit

@lhoestq @ArjunJagdale is this still work in progress or is just a review missing? Anything I can help with here? This would indeed be a cool feature

I did a preliminary pass and it looks good but we should check the CI, could you run `make style` @ArjunJagdale so we can run the CI ?

Done! Also some parts may be incomplete because I had to focus on important exams and semester activities so couldn’t finish the work fully. I will still try my best.
TypeError: Value.__init__() missing 1 required positional argument: 'dtype'

### Describe the bug

ds = load_dataset(
    "./xxx.py",
    name="default",
    split="train",
)

The datasets does not support debugging locally anymore...

### Steps to reproduce the bug

```
from datasets import load_dataset

ds = load_dataset(
    "./repo.py",
    name="default",
    split="train",
)

for item in ds:
    print(item)
```

It works fine for "username/repo", but it does not work for "./repo.py" when debugging locally...
Running above code template will report TypeError: Value.__init__() missing 1 required positional argument: 'dtype'

### Expected behavior

fix this bug

### Environment info

python 3.10 datasets==2.21


same error 
```
data = load_dataset('/opt/deepseek_R1_finetune/hf_datasets/openai/gsm8k', 'main')[split] 
```

> same error
> 
> ```
> data = load_dataset('/opt/deepseek_R1_finetune/hf_datasets/openai/gsm8k', 'main')[split] 
> ```

https://github.com/huggingface/open-r1/issues/204  this help me

Solved by delete `dataset_infos.json` file in dataset dir, or you can transfer datasets from Hugginface to Modelscope by [hf-ms-transfer](https://github.com/wa008/hf-ms-transfer), which will solve this problem by default.
Transform with columns parameter triggers on non-specified column access

### Describe the bug

Iterating over a [`Column`](https://github.com/huggingface/datasets/blob/8b1bd4ec1cc9e9ce022f749abb6485ef984ae7c0/src/datasets/arrow_dataset.py#L633-L692) iterates through the parent [`Dataset`](https://github.com/huggingface/datasets/blob/8b1bd4ec1cc9e9ce022f749abb6485ef984ae7c0/src/datasets/arrow_dataset.py#L695) and applies all formatting/transforms on each row, regardless of which column is being accessed. This causes an error when transforms depend on columns not present in the projection.

### Steps to reproduce the bug

### Load a dataset with multiple columns

```python
ds = load_dataset("mrbrobot/isic-2024", split="train")
```

### Define a transform that specifies an input column

```python
def image_transform(batch):
    batch["image"] = batch["image"]  # KeyError when batch doesn't contain "image"
    return batch

# apply transform only to image column
ds = ds.with_format("torch")
ds = ds.with_transform(image_transform, columns=["image"], output_all_columns=True)
```

### Iterate over non-specified column

```python
# iterate over a different column, triggers the transform on each row, but batch doesn't contain "image"
for t in ds["target"]:  # KeyError: 'image'
    print(t)
```

### Expected behavior

If a user iterates over `ds["target"]` and the transform specifies `columns=["image"]`, the transform should be skipped.

### Environment info

`datasets`: 4.2.0
Python: 3.12.12
Linux: Debian 11.11
Fix Audio docstring by removing unsupported mono argument

This PR fixes issue #7745.

Who can review:
@lhoestq 


I think we can keep the arg and add the missing torch.mean() in the Audio.decode_example method

> I think we can keep the arg and add the missing torch.mean() in the Audio.decode_example method

Thank you @lhoestq. I will add torch.mean().

fixed by #7840
Fix: Update Dill Version in Setup py

Currently the DIll version is less than 3.9 and now major libraries like Multiprocess, gepa requires Dill version as 0.4.0 and this is making a conflict in installation. So added this small PR to update the DIll. 


https://github.com/huggingface/datasets/issues/7751

same as https://github.com/huggingface/datasets/pull/7763: some tests need to be fixed to support 0.4.0

fixed by #7763
mono parameter to the Audio feature is missing

According to the docs, there is a "mono" parameter to the Audio feature, which turns any stereo into mono. In practice the signal is not touched and the mono parameter, even though documented, does not exist.


https://github.com/huggingface/datasets/blob/41c05299348a499807432ab476e1cdc4143c8772/src/datasets/features/audio.py#L52C1-L54C22


Hey,  we removed the misleading passage in the docstring and enabled support for `num_channels` as torchcodec does

thanks!
Add num channels to audio

Fixes #7837

We currently have the [mono attribute for Audio documented](https://github.com/huggingface/datasets/blob/41c05299348a499807432ab476e1cdc4143c8772/src/datasets/features/audio.py#L52C1-L54C22) but not used anywhere resulting in confusion for users. Since torchcodec does not know this attribute I suggest using `num_channels` (currently supported `None` (leave unchanged), mono: `1`, stereo: `2`).

I could also add a mono attribute but found that to be more confusing for developers and would restrict us if at any point in the future more than 2 channels are supported.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7840). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.