noamaanMulla-03/datasets-issues huggingface dataset:
`huggingface_hub.errors.HfHubHTTPError: 404 Client Error: Not Found for url` when fetching a dataset with `datasets.load_dataset`

Not a bug but a question. We started getting the following error:

https://github.com/mlflow/mlflow/actions/runs/21368603305/job/61506951617

```
ests/data/test_huggingface_dataset_and_source.py::test_from_huggingface_dataset_constructs_expected_dataset_with_revision - huggingface_hub.errors.HfHubHTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/datasets/rotten_tomatoes/revision/aa13bc287fa6fcab6daf52f0dfb9994269ffea28 (Request ID: Root=1-6977aeca-35bc2b5b605884926a9224d0;aa2391f3-26e8-4975-b9bb-114b2fa40223)
```

Adding a user id fixed the issue (https://github.com/mlflow/mlflow/pull/20350). `https://huggingface.co/api/datasets` no longer accepts a name-only path like `rotten_tomatoes`? Just wondering what changed. Thanks!


Hi ! Yes you should use `cornell-movie-review-data/rotten_tomatoes` instead of `rotten_tomatoes`, which is the legacy name. Those datasets have been moved under their actual owners accounts some time ago (but we were keeping the old names as aliases)

Some other impacted names are:
- `imdb` -> `stanfordnlp/imdb`
- `wikitext` -> `Salesforce/wikitext`
- `gsm8k` -> `openai/gsm8k`
- `winogrande` -> `allenai/winogrande`

We're working on re-enabling them as aliases for backward compatibility. I'll post updates here, sorry for the inconvenience.

**Using the actual name instead of the old legacy name is more future proof though**

Thanks for the heads up @lhoestq ! fyi, this change is likely breaking a lot of repos that have legacy names hardcoded ([example](https://github.com/allenai/olmes/pull/40)) Would be helpful to many to share this update in a more visible way if it is likely to persist for a while.

[internal tracking link](https://github.com/huggingface-internal/moon-landing/pull/16539)

@lhoestq Thanks for clarifying!

The aliases are re-enabled :)

Thanks!

Can I close this issue?

Yep :)
push_to_hub does not upload videos

### Describe the bug

Hello,

I would like to upload a video dataset (some .mp4 files and some segments within them), i.e. rows correspond to subsequences from videos. Videos might be referenced by several rows.

I created a dataset locally and it references the videos and the video readers can read them correctly. I use push_to_hub() to upload the dataset to the hub.

Expectation: A user uses `load_dataset` and can load the videos.

However, the videos seem to be just referenced via paths on the computer and not uploaded to the hub. Therefore a target user cannot load the videos in the dataset.

### Steps to reproduce the bug

1. create a video dataset with paths e.g. { ["videos"]: [path1, path2, ...]}
2. dataset.push_to_hub
3. on a different computer (or same pc if relative paths are used in a different folder):
```
dataset = load_dataset("siplab/egosim", split="train")
video = dataset[0]["video_head"]
```

3. will fail

### Expected behavior

Expectation: A user uses `load_dataset` and can load the videos.

### Environment info

datasets                       3.1.0
Python 3.8.18


Hi ! the `Video` type is still experimental, and in particular `push_to_hub` doesn't upload videos at the moment (only the paths).

There is an open question to either upload the videos inside the Parquet files, or rather have them as separate files (which is great to enable remote seeking/streaming)

im having the same issue (btw i mistook this to be xet error https://huggingface.co/spaces/xet-team/README/discussions/4 )

@jsulz suggested me to use `upload_folder` but it exceeds hf limits (>10k files per folder and >100k files in total)

from my reading of the docs, in my case i have to save as either parquet or webdataset and then use `upload_folder`

i tried `ds.to_parquet("...")` but the parquet file also doesnt contain video, as of `datasets` v4.0

so i think the only workaround for my case is webdataset

in that case you can create a VideoFolder dataset instead, see the docs at http://huggingface.co/docs/datasets/video_dataset#videofolder

I just added support in push_to_hub() for videos :)

Note a small discrepancy for video datasets I hope to fix soon:

- `load_dataset("username/my-folder-of-videos", streaming=True)` -> videos are lazy loaded one by one when iterating, and only actually downloaded when accessing frames in torchcodec (only the requested frames are downloaded)
- `load_dataset("username/my-video-dataset-in-parquet-saved-with-push_to_hub", streaming=True)` -> videos are downloaded one by one when iterating. Even if no frame is accessed in torchcodec, the videos are downloaded completely
push_to_hub() for videos

possible now that row group sizes are auto-determined based on the content size after https://github.com/huggingface/datasets/pull/7589

Videos are uploaded as PLAIN in Parquet to make sure they can be seeked remotely and with random access to frames in https://github.com/huggingface/datasets/pull/7976

In the future it could be cool to have the same behavior as when videos are separate files, i.e. lazily load them instead of downloading them completely in streaming mode.

Right now there is this discrepency:
- `load_dataset("username/my-folder-of-videos", streaming=True)` -> videos are lazy loaded one by one when iterating, and only actually downloaded when accessing frames in `torchcodec`
- `load_dataset("username/my-video-dataset-in-parquet", streaming=True)` -> videos are downloaded one by one when iterating, even if no frame is accessed in `torchcodec`

close https://github.com/huggingface/datasets/issues/7493


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7971). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Memory leak / Large memory usage with num_workers = 0 and numerous dataset within DatasetDict

### Describe the bug

Hi team, first off, I love the datasets library! 🥰

I'm encountering a potential memory leak / increasing memory usage when training a model on a very large DatasetDict.

Setup: I have a DatasetDict containing 362 distinct datasets, which sum up to ~2.8 billion rows.

Training Task: I'm performing contrastive learning with SentenceTransformer and Accelerate on a single node with 4 H100, which requires me to sample from only one dataset at a time.

Training Loop: At each training step, I sample ~16,000 examples from a single dataset, and then switch to a different dataset for the next step. I iterate through all 362 datasets this way. 

Problem: The process's memory usage continuously increases over time, eventually causing a stale status where GPUs would stop working. It seems memory from previously sampled datasets isn't being released. I've set num_workers=0 for all experiments.

Chart 1: Standard DatasetDict The memory usage grows steadily until it make the training stale (RSS memory) <img width="773" height="719" alt="Image" src="https://github.com/user-attachments/assets/6606bef5-1153-4f2d-bf08-82da249d6e8d" />

Chart 2: IterableDatasetDict I also tried  to use IterableDatasetDict and IterableDataset. The memory curve is "smoother," but the result is the same: it grows indefinitely and the training become stale. <img width="339" height="705" alt="Image" src="https://github.com/user-attachments/assets/ee90c1a1-6c3b-4135-9edc-90955cb1695a" />

Any feedback or guidance on how to manage this memory would be greatly appreciated!

### Steps to reproduce the bug

WIP, I'll add some code that manage to reproduce this error, but not straightforward. 

### Expected behavior

The memory usage should remain relatively constant or plateau after a few steps. Memory used for sampling one dataset should be released before or during the sampling of the next dataset.

### Environment info

Python: 3.12
Datasets: 4.3.0
SentenceTransformers: 5.1.1


Thanks for the report, this is possibly related #7722 and #7694.

Could you pls provide steps to reproduce this?

To overcome this issue right now I did simply reduce the size of the dataset and ended up running a for loop (my training has now a constant learning rate schedule). From what I understood, and I don't know if it's possible, the solution would be to tell the backend of `datasets` to leave x% of the memory free  (including memory mapping). Can't release the data right now but I will and then allow to reproduce this issue. But it will involve to have some free TB of disk

@raphaelsty thanks for coming back to this. I assume you are running in streaming mode? That should prevent these errors but it looks like more people than just you have this problem, so a clearly reproducing example (including data + code) is highly appreciated.

This could be related to this issue: https://github.com/huggingface/datasets/issues/4883 in which we discussed how RSS and memory mapping works and depends on the OS and disk type.
Updated get_dataset_config_names returning default in offline mode 

When a dataset is cached and accessed in offline mode, get_dataset_config_names was returning default instead of the actual cached config names. This happened because CachedDatasetModuleFactory.get_module returned a DatasetModule without builder_configs_parameters, causing the fallback to default in get_dataset_config_names.                                                                                             
                                                                                                                                                   
The fix reads config_name from each dataset_info file in the cache directory and includes them as builder_configs_parameters in the returned DatasetModule. Invalid or missing dataset_info.json files are handled.

**Testing:**                                                                                                                                                                                                                                                                                         
1. Download a dataset in online mode so it gets cached                                                                                           
2. Switch to offline mode and call get_dataset_config_names                                                                                      
3. Verify it returns the cached config names instead of ['default']                                                                              
                                                                                                                                                   
**Example:**                                                                                                                                                                                                                                                                                         

- HF_DATASETS_OFFLINE=0 HF_HOME="/tmp/hftemp" python -c "import datasets; datasets.load_dataset('cais/mmlu', 'all')"                                                                                                                                                                                
- HF_DATASETS_OFFLINE=1 HF_HOME="/tmp/hftemp" python -c "import datasets; print(datasets.get_dataset_config_names('cais/mmlu'))"                                                                                                                                                                    
- -> Expected output: ['all'] 

Fixes https://github.com/huggingface/datasets/issues/7947
Fix 4910 kwargs

Fix #4910 : Merge duplicate kwargs in `load_dataset_builder()`

Problem: load_dataset("dataset", base_path="./data")` gives TypeError("multiple values for keyword 'base_path')

Fix: {**builder_kwargs, **config_kwargs} to user kwargs override dataset defaults

Repro : python

Before: TypeError

load_dataset("rotten_tomatoes", base_path="./sample_data")

After: Works
Remove unused data files optims

this fixes module inference when there are many metadata files

e.g. the lance dataset at https://huggingface.co/datasets/davanstrien/encyclopaedia-britannica-lance has > 200 metadata files

those optims are not used anymore, they come from a time we were dealing with slow data files iterators instead of lists


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7985). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Dataset Preprocessing Cache with .map() function not working as expected

I've been having issues with reproducibility when loading and processing datasets with the `.map` function. I was only able to resolve them by clearing all of the cache files on my system. 

Is there a way to disable using the cache when processing a dataset? As I make minor processing changes on the same dataset, I want to be able to be certain the data is being re-processed rather than loaded from a cached file. 

Could you also help me understand a bit more about how the caching functionality is used for pre-processing? E.g. how is it determined when to load from a cache vs. reprocess. 
I was particularly having an issue where the correct dataset splits were loaded, but as soon as I applied the `.map()` function to each split independently, they somehow all exited this process having been converted to the test set.
Thanks!


When you're processing a dataset with `.map`, it checks whether it has already done this computation using a hash based on the function and the input (using some fancy serialization with `dill`). If you found that it doesn't work as expected in some cases, let us know !

Given that, you can still force to re-process using `.map(my_func, load_from_cache_file=False)` if you want to.

I am curious about the problem you have with splits. It makes me think about #160 that was an issue of version 0.1.0. What version of `nlp` are you running ? Could you give me more details ?

Thanks, that's helpful! I was running 0.1.0, but since upgraded to 0.2.1. I can't reproduce the issue anymore as I've cleared the cache & everything now seems to be running fine since the upgrade. I've added some checks to my code, so if I do encounter it again I will reopen this issue.

Just checking in, the cache sometimes still does not work when I make changes in my processing function in version `1.2.1`. The changes made to my data processing function only propagate to the dataset when I use `load_from_cache_file=False` or clear the cache. Is this a system-specific issue?

Hi @sarahwie 
The data are reloaded from the cache if the hash of the function you provide is the same as a computation you've done before. The hash is computed by recursively looking at the python objects of the function you provide.

If you think there's an issue, can you share the function you used or a google colab please ?

I can't reproduce it, so I'll close for now.
Support for sparse arrays with the Arrow Sparse Tensor format?

### Feature request

AI in biology is becoming a big thing. One thing that would be a huge benefit to the field that Huggingface Datasets doesn't currently have is native support for **sparse arrays**. 


Arrow has support for sparse tensors. 
https://arrow.apache.org/docs/format/Other.html#sparse-tensor


It would be a big deal if Hugging Face Datasets supported sparse tensors as a feature type, natively. 


### Motivation

This is important for example in the field of transcriptomics (modeling and understanding gene expression), because a large fraction of the genes are not expressed (zero). More generally, in science, sparse arrays are very common, so adding support for them would be very benefitial, it would make just using Hugging Face Dataset objects a lot more straightforward and clean.


### Your contribution

We can discuss this further once the team comments of what they think about the feature, and if there were previous attempts at making it work, and understanding their evaluation of how hard it would be. My intuition is that it should be fairly straightforward, as the Arrow backend already supports it.

Labels: name: enhancement

Hi ! Unfortunately the Sparse Tensor structure in Arrow is not part of the Arrow format (yes it's confusing...), so it's not possible to use it in `datasets`. It's a separate structure that doesn't correspond to any type or extension type in Arrow.

The Arrow community recently added an extension type for fixed shape tensors at https://arrow.apache.org/docs/format/CanonicalExtensions.html#fixed-shape-tensor, it should be possible to contribute an extension type for sparse tensors as well.

Thanks for the clarification on the Arrow format limitations @lhoestq 

Since a native Arrow Extension Type for sparse tensors might be a longer-term upstream effort, I wanted to propose a pragmatic solution that could be implemented in datasets immediately to help users in domains such as biology or NLP.

Instead of waiting for a native sparse type, we can efficiently handle sparse data by formalizing a schema based on nested lists (COO/CSR style) using existing Arrow types. This avoids OOM issues caused by loading dense zeros while maintaining full compatibility with the current Arrow backend.

The Strategy: We could add a reader (or a "best practice" utility) that ingests sparse formats (like wide CSVs or Matrix Market) and stores them as a struct of sequences:
```
features = Features({
    "row_id": Value("int32"),
    "indices": Sequence(Value("int32")),
    "values": Sequence(Value("float32")),
    "shape": Sequence(Value("int32"), length=2) # Store original shape
})
```

#self-assign
Use Sequence instead of list in Dataset.from_parquet type hints

This PR updates type annotations in Dataset.from_parquet to use Sequence instead of list to avoid mypy invariant type issues, as discussed in issue #5354.



Thank you for the review!

I’ve updated the implementation to:
- Use Sequence from collections.abc as per project conventions
- Restore backward compatibility with Union[PathLike, Sequence[PathLike]]
- Keep the columns annotation as Optional[Sequence[str]]

The fixes are pushed. Please let me know if anything else is needed.
Remove pre-release workaround in CI for `transformers v5` and `huggingface_hub v1`

This PR removes workaround for pre-release `transformers v5.*` / `huggingface_hub v1.*` in `test_py314_future` job since they are now officially released.

cc @Wauplin just for viz since you introduced this in https://github.com/huggingface/datasets/pull/7783.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7989). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
concatenate_datasets does not preserve shuffling state

### Describe the bug

After concatenate datasets on an iterable dataset, the shuffling state is destroyed, similar to #7156 

This means concatenation cant be used for resolving uneven numbers of samples across devices when using iterable datasets in a distributed setting as discussed in #6623 

I also noticed that the number of shards is the same after concatenation, which I found surprising, but I don't understand the internals well enough to know whether this is actually surprising or not

### Steps to reproduce the bug

```python
import datasets
import torch.utils.data


def gen(shards):
    yield {"shards": shards}


def main():
    dataset1 = datasets.IterableDataset.from_generator(
        gen, gen_kwargs={"shards": list(range(25))}  # TODO: how to understand this?
    )
    dataset2 = datasets.IterableDataset.from_generator(
        gen, gen_kwargs={"shards": list(range(25, 50))}  # TODO: how to understand this?
    )
    dataset1 = dataset1.shuffle(buffer_size=1)
    dataset2 = dataset2.shuffle(buffer_size=1)
    print(dataset1.n_shards)
    print(dataset2.n_shards)

    dataset = datasets.concatenate_datasets(
        [dataset1, dataset2]
    )
    print(dataset.n_shards)
    # dataset = dataset1

    dataloader = torch.utils.data.DataLoader(
        dataset,
        batch_size=8,
        num_workers=0,
    )

    for i, batch in enumerate(dataloader):
        print(batch)
    print("\nNew epoch")

    dataset = dataset.set_epoch(1)

    for i, batch in enumerate(dataloader):
        print(batch)


if __name__ == "__main__":
    main()
```

### Expected behavior

Shuffling state should be preserved

### Environment info

Latest datasets


It also does preserve `split_by_node`, so in the meantime you should call `shuffle` or `split_by_node` AFTER `interleave_datasets` or `concatenate_datasets`
:sparkles: Add 'SparseCsv' builder and 'sparse_collate_fn' for efficient high-dimensional sparse data loading

This PR introduces a new dataset builder, SparseCsv, designed to handle "wide" tabular datasets (e.g., 100k+ columns common in transcriptomics, sparse NLP features, or recommender systems) that are typically too large to load into memory as dense Arrow tables.

It also adds a utility function, `sparse_collate_fn`, to seamlessly convert these sparse examples into `torch.sparse` or `scipy.sparse` matrices during training.

This PR should fix #7377
list(api.list_datasets()) giving jsondecode error

i am using the python api wrapper to list all datasets available on hugging face. This is for research, and i need all datasets to determine which % have language tags and other related questions requiring the total list. However, the following code that worked a few months ago:

```
from huggingface_hub import HfApi
api = HfApi(token=token)
datasets = list(api.list_datasets())
```

now gives a JSONDecodeError when reaching 2000 results. My understanding is that this is a pagination issue, as there is no cursor exposed to the API and so it doesnt know where to read from after the limit. 

Is there any way to find all datasets available and combine them into a list?