josteece/github-issues huggingface dataset:
File not found error
Opened by MichielBontenbal (open)

### Describe the bug

I get a FileNotFoundError:
<img width="944" alt="image" src="https://github.com/user-attachments/assets/1336bc08-06f6-4682-a3c0-071ff65efa87">


### Steps to reproduce the bug

See screenshot.



### Expected behavior

I want to load one audiofile from the dataset. 

### Environment info

MacOs Intel 14.6.1 (23G93)
Python 3.10.9
Numpy 1.23
Datasets latest version

Comments:
Link to the dataset: https://huggingface.co/datasets/MichielBontenbal/UrbanSounds
Add filename in error message when ReadError or similar occur
Opened by elisa-aleman (open)

Please update error messages to include relevant information for debugging when loading datasets with `load_dataset()` that may have a few corrupted files. 

Whenever downloading a full dataset, some files might be corrupted (either at the source or from downloading corruption). 

However the errors often only let me know it was a tar file if `tarfile.ReadError` appears on the traceback, and I imagine similarly for other file types. 

This makes it really hard to debug which file is corrupted, and when dealing with very large datasets, it shouldn't be necessary to force download everything again.

Comments:
Hi Elisa, please share the error traceback here, and if you manage to find the location in the `datasets` code where the error occurs, feel free to open a PR to add the necessary logging / improve the error message.

> please share the error traceback

I don't have access to it but it should be during [this exception](https://github.com/huggingface/datasets/blob/2049c00921c59cdeb835137a1c49639cf175af07/src/datasets/builder.py#L1643) which happens during the loading of a dataset. If one of the downloaded files is corrupted, the for loop will not yield correctly, and the error will come from, say, in the case of tar files, [this iterable](https://github.com/huggingface/datasets/blob/2049c00921c59cdeb835137a1c49639cf175af07/src/datasets/utils/file_utils.py#L1293) which has no explicit error handling that leaves clues as to which file has failed.

I only know the case for tar files, but I consider this issue could be happening across different file types too.

I think having a better error handling for this tar iterable would be useful already, maybe a simple try/except in `_iter_from_urlpath` that checks for `tarfile.ReadError` and raises an error with the `urlpath` mentioned in the error ?

I think not just from higher calls like the `_iter_from_urlpath` but directly wherever a file is attempted to be opened would be the best case, as the traceback would simply lead to that.

so maybe there should be better error messages in each dataset builder definition ? e.g. in https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/webdataset/webdataset.py for webdataset TAR archives
Add link to video dataset
Opened by NielsRogge (closed)

This PR updates https://huggingface.co/docs/datasets/loading to also link to the new video loading docs.

cc @mfarre 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7277). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
load_dataset
Opened by santiagobp99 (open)

### Describe the bug

I am performing two operations I see on a hugging face tutorial (Fine-tune a language model), and I am defining every aspect inside the mapped functions, also some imports of the library because it doesnt identify anything not defined outside that function where the dataset elements are being mapped:

https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb#scrollTo=iaAJy5Hu3l_B

`- lm_datasets = tokenized_datasets.map(
    group_texts,
    batched=True,
    batch_size=batch_size,
    num_proc=4,
)

- tokenized_datasets = datasets.map(tokenize_function, batched=True, num_proc=4, remove_columns=["text"])
def tokenize_function(examples):
    model_checkpoint = 'gpt2'
    from transformers import AutoTokenizer
    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)
    return tokenizer(examples["text"])`


### Steps to reproduce the bug

Currently handle all the imports inside the function

### Expected behavior

The code must work es expected in the notebook, but currently this is not happening.

https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb#scrollTo=iaAJy5Hu3l_B

### Environment info

print(transformers.__version__)

4.46.1

Comments:
Raise error for incorrect JSON serialization
Opened by varadhbhatnagar (closed)

Raise error when `lines = False` and `batch_size < Dataset.num_rows` in `Dataset.to_json()`.

Issue: #7037 

Related PRs:
#7039 #7181 

Comments:
PTAL @lhoestq @albertvillanova 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7273). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
load_from_disk
Opened by ghaith-mq (open)

### Describe the bug

I have data saved with save_to_disk. The data is big (700Gb). When I try loading it, the only option is load_from_disk, and this function copies the data to a tmp directory, causing me to run out of disk space. Is there an alternative solution to that?

### Steps to reproduce the bug

when trying to load data using load_From_disk after being saved using save_to_disk 

### Expected behavior

run out of disk space

### Environment info

lateest version

Comments:
Hello, It's an interesting issue here. I have the same problem, I have a local dataset and I want to push the dataset to the hub but huggingface does a copy of it.

```py
from datasets import load_dataset

dataset = load_dataset("webdataset", data_files="/media/works/data/*.tar") # copy here
dataset.push_to_hub("WaveGenAI/audios2")
```

Edit: I can use HfApi for my use case


Is there any update on this issue? I found the same behavior too.
My datasets version is `2.13.2`

Updating to the newest version of datasets lib resolved the issue.
Source installation fails on Macintosh with python 3.10
Opened by mayankagarwals (open)

### Describe the bug

Hi, 

Decord is a dev dependency not maintained since couple years. 
It does not have an ARM package available rendering it uninstallable on non-intel based macs 

Suggestion is to move to eva-decord (https://github.com/georgia-tech-db/eva-decord)  which doesnt have this problem. 

Happy to raise a PR


### Steps to reproduce the bug

Source installation as mentioned in contributinog.md

### Expected behavior

Installation without decord failing to be installed.

### Environment info

python=3.10, M3 Mac 

Comments:
I encountered the same problem on M1, a workaround I did was to simply comment out the dependency:

```python
...
    "zstandard",
    "polars[timezone]>=0.20.0",
    # "decord==0.6.0",
]
```

This worked for me as the adjustments I did to the code do not use the dependency, but I do not know if the same holds for you.

I also do not think it is a good idea to rely on a dependency (I mean decord) that has not been maintained for 2 years, but I saw that even eva-decord hasn't been maintained since last year.

Did you get it to work with eva-decord?
The dataset viewer should be available soon. Please retry later.
Opened by viiika (closed)

### Describe the bug

After waiting for 2 hours, it still presents ``The dataset viewer should be available soon. Please retry later.''

### Steps to reproduce the bug

dataset link: https://huggingface.co/datasets/BryanW/HI_EDIT

### Expected behavior

Present the dataset viewer.

### Environment info

NA

Comments:
Waiting is all you need. 10 hours later, it works.
Allow video with disabeld decoding without decord
Opened by lhoestq (closed)

for the viewer, this way it can use Video(decode=False) and doesn't need decord (which causes segfaults)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7262). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Cannot load the cache when mapping the dataset
Opened by zhangn77 (open)

### Describe the bug

I'm training the flux controlnet. The train_dataset.map() takes long time to finish. However, when I killed one training process and want to restart a new training with the same dataset. I can't reuse the mapped result even I defined the cache dir for the dataset.

with accelerator.main_process_first():
        from datasets.fingerprint import Hasher

        # fingerprint used by the cache for the other processes to load the result
        # details: https://github.com/huggingface/diffusers/pull/4038#discussion_r1266078401
        new_fingerprint = Hasher.hash(args)
        train_dataset = train_dataset.map(
            compute_embeddings_fn, batched=True, new_fingerprint=new_fingerprint, batch_size=10,
        )

### Steps to reproduce the bug

train flux controlnet and start again



### Expected behavior

will not map again

### Environment info

latest diffusers



Comments:
@zhangn77 Hi ，have you solved this problem? I encountered the same issue during training. Could we discuss it?

I also encountered the same problem, why is that？
cache can't cleaned or disabled
Opened by charliedream1 (open)

### Describe the bug

I tried following ways, the cache can't be disabled. 

I got 2T data, but I also got more than 2T cache file. I got pressure on storage. I need to diable cache or cleaned immediately after processed. Following ways are all not working, please give some help!

```python
from datasets import disable_caching
from transformers import AutoTokenizer
disable_caching()

tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_path)
def tokenization_fn(examples):
    column_name = 'text' if 'text' in examples else 'data'
    tokenized_inputs = tokenizer(
        examples[column_name], return_special_tokens_mask=True, truncation=False,
        max_length=tokenizer.model_max_length
    )
    return tokenized_inputs

data = load_dataset('json', data_files=save_local_path, split='train', cache_dir=None)
data.cleanup_cache_files()
updated_dataset = data.map(tokenization_fn, load_from_cache_file=False)
updated_dataset .cleanup_cache_files()
```

### Expected behavior

no cache file generated

### Environment info

Ubuntu 20.04.6 LTS
datasets  3.0.2

Comments:
Hey I have a similar problem and found a workaround using [temporary directories](https://docs.python.org/3/library/tempfile.html):

```python
from tempfile import TemporaryDirectory

with TemporaryDirectory() as cache_dir:
    data = load_dataset('json', data_files=save_local_path, split='train', cache_dir=cache_dir)
```

But I do agree that it would be more intuitive if `datasets` supported this directly. Especially `disable_caching` is confusing, since it basically doesn't disable caching.
Don't embed videos
Opened by lhoestq (closed)

don't include video bytes when running download_and_prepare(format="parquet")

this also affects push_to_hub which will just upload the local paths of the videos though

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7259). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Retry all requests timeouts
Opened by lhoestq (closed)

as reported in https://github.com/huggingface/datasets/issues/6843

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7256). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix decord import
Opened by lhoestq (closed)

delay the import until Video() is instantiated + also import duckdb first (otherwise importing duckdb later causes a segfault)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7255). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
mismatch for datatypes when providing `Features` with `Array2D` and user specified `dtype` and using with_format("numpy")
Opened by Akhil-CM (open)

### Describe the bug

If the user provides a `Features` type value to `datasets.Dataset` with members having `Array2D` with a value for `dtype`, it is not respected during `with_format("numpy")` which should return a `np.array` with `dtype` that the user provided for `Array2D`. It seems for floats, it will be set to `float32` and for ints it will be set to `int64`

### Steps to reproduce the bug

```python
import numpy as np
import datasets
from datasets import Dataset, Features, Array2D

print(f"datasets version: {datasets.__version__}")

data_info = {
    "arr_float" : "float64",
    "arr_int" : "int32"
}

sample = {key : [np.zeros([4, 5], dtype=dtype)] for key, dtype in data_info.items()}

features = {key : Array2D(shape=(None, 5), dtype=dtype) for key, dtype in data_info.items()}
features = Features(features)

dataset = Dataset.from_dict(sample, features=features)

ds = dataset.with_format("numpy")
for key in features:
    print(f"{key} feature dtype: ", ds.features[key].dtype)
    print(f"{key} dtype:", ds[key].dtype)
```

Output:
```bash
datasets version: 3.0.2
arr_float feature dtype:  float64
arr_float dtype: float32
arr_int feature dtype:  int32
arr_int dtype: int64
```

### Expected behavior

It should return a `np.array` with `dtype` that the user provided for the corresponding member in the `Features` type value

### Environment info

- `datasets` version: 3.0.2
- Platform: Linux-6.11.5-arch1-1-x86_64-with-glibc2.40
- Python version: 3.12.7
- `huggingface_hub` version: 0.26.1
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0

Comments:
It seems that https://github.com/huggingface/datasets/issues/5517 is exactly the same issue.

It was mentioned there that this would be fixed in version 3.x
Unable to upload a large dataset zip either from command line or UI
Opened by vakyansh (open)

### Describe the bug

Unable to upload a large dataset zip from command line or UI. UI simply says error. I am trying to a upload a tar.gz file of 17GB.

<img width="550" alt="image" src="https://github.com/user-attachments/assets/f9d29024-06c8-49c4-a109-0492cff79d34">
<img width="755" alt="image" src="https://github.com/user-attachments/assets/a8d4acda-7f02-4279-9c2d-b2e0282b4faa">


### Steps to reproduce the bug

Upload a large file 

### Expected behavior

The file should upload without any issue.

### Environment info

None

Comments:
Add IterableDataset.shard()
Opened by lhoestq (closed)

Will be useful to distribute a dataset across workers (other than pytorch) like spark

I also renamed `.n_shards` -> `.num_shards` for consistency and kept the old name for backward compatibility. And a few changes in internal functions for consistency as well (rank, world_size -> num_shards, index)

Breaking change: the new default for `contiguous` in `Dataset.shard()` is `True`, but imo not a big deal since I couldn't find any usage of `contiguous=False` internally (we always do contiguous=True for map-style datasets since its more optimized) or in the wild

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7252). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Is there some way to get this to work for pytorch dataloader workers?

eg. start with a single sharded IterableDataset.from_generator(), then reshard before calling map() to do expensive processing over multiple workers
Basic XML support (mostly copy pasted from text)
Opened by lhoestq (closed)

enable the viewer for datasets like https://huggingface.co/datasets/FrancophonIA/e-calm (there will be more and more apparently)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7250). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
How to debugging
Opened by ShDdu (open)

### Describe the bug

I wanted to use my own script to handle the processing, and followed the tutorial documentation by rewriting the MyDatasetConfig and MyDatasetBuilder (which contains the _info,_split_generators and _generate_examples methods) classes. Testing with simple data was able to output the results of the processing, but when I wished to do more complex processing, I found that I was unable to debug (even the simple samples were inaccessible). There are no errors reported, and I am able to print the _info,_split_generators and _generate_examples messages, but I am unable to access the breakpoints.

### Steps to reproduce the bug

# my_dataset.py
import json
import datasets


class MyDatasetConfig(datasets.BuilderConfig):
    def __init__(self, **kwargs):
        super(MyDatasetConfig, self).__init__(**kwargs)


class MyDataset(datasets.GeneratorBasedBuilder):
    VERSION = datasets.Version("1.0.0")

    BUILDER_CONFIGS = [
        MyDatasetConfig(
            name="default",
            version=VERSION,
            description="myDATASET"
        ),
    ]

    def _info(self):
        print("info")  # breakpoints
        return datasets.DatasetInfo(
            description="myDATASET",
            features=datasets.Features(
                {
                    "id": datasets.Value("int32"),
                    "text": datasets.Value("string"),
                    "label": datasets.ClassLabel(names=["negative", "positive"]),
                }
            ),
            supervised_keys=("text", "label"),
        )

    def _split_generators(self, dl_manager):
 
        print("generate")  # breakpoints
        data_file = "data.json"  

        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN, gen_kwargs={"filepath": data_file}
            ),
        ]

    def _generate_examples(self, filepath):
        print("example")  # breakpoints
        with open(filepath, encoding="utf-8") as f:
            data = json.load(f)
            for idx, sample in enumerate(data):
                yield idx, {
                    "id": sample["id"],
                    "text": sample["text"],
                    "label": sample["label"],
                }

#main.py
import os
os.environ["TRANSFORMERS_NO_MULTIPROCESSING"] = "1" 

from datasets import load_dataset

dataset = load_dataset("my_dataset.py", split="train", cache_dir=None)

print(dataset[:5])

### Expected behavior

Pause at breakpoints while running debugging

### Environment info

pycharm


Comments:
Adding column with dict struction when mapping lead to wrong order
Opened by chchch0109 (open)

### Describe the bug

in `map()` function, I want to add a new column with a dict structure.
```
def map_fn(example):
  example['text'] = {'user': ..., 'assistant': ...}
  return example
```
However this leads to a wrong order `{'assistant':..., 'user':...}` in the dataset.
Thus I can't concatenate two datasets due to the different feature structures.
[Here](https://colab.research.google.com/drive/1zeaWq9Ith4DKWP_EiBNyLfc8S8I68LyY?usp=sharing) is a minimal reproducible example
This seems an issue in low level pyarrow library instead of datasets, however, I think datasets should allow concatenate two datasets actually in the same structure.

### Steps to reproduce the bug

[Here](https://colab.research.google.com/drive/1zeaWq9Ith4DKWP_EiBNyLfc8S8I68LyY?usp=sharing) is a minimal reproducible example

### Expected behavior

two datasets could be concatenated.

### Environment info

N/A

Comments:
use huggingface_hub offline mode
Opened by lhoestq (closed)

and better handling of LocalEntryNotfoundError cc @Wauplin 

follow up to #7234 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7244). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
ArrayXD with None as leading dim incompatible with DatasetCardData
Opened by alex-hh (open)

### Describe the bug

Creating a dataset with ArrayXD features leads to errors when downloading from hub due to DatasetCardData removing the Nones

@lhoestq 

### Steps to reproduce the bug

```python
import numpy as np
from datasets import Array2D, Dataset, Features, load_dataset

def examples_generator():
    for i in range(4):
        yield {
            "array_1d": np.zeros((10,1), dtype="uint16"),
            "array_2d": np.zeros((10, 1), dtype="uint16"),
        }

features = Features(array_1d=Array2D((None,1), "uint16"), array_2d=Array2D((None, 1), "uint16"))
dataset = Dataset.from_generator(examples_generator, features=features)
dataset.push_to_hub("alex-hh/test_array_1d2d")
ds = load_dataset("alex-hh/test_array_1d2d")
```

Source of error appears to be DatasetCardData.to_dict invoking DatasetCardData._remove_none
```python
from huggingface_hub import DatasetCardData
from datasets.info import DatasetInfosDict

dataset_card_data = DatasetCardData()
DatasetInfosDict({"default": dataset.info.copy()}).to_dataset_card_data(dataset_card_data)
print(dataset_card_data.to_dict())  # removes Nones in shape
```

### Expected behavior

Should be possible to load datasets saved with shape None in leading dimension

### Environment info

3.0.2 and latest huggingface_hub

Comments:
It looks like `CardData` in `huggingface_hub` removes None values where it shouldn't. Indeed it calls `_remove_none` on the return of `to_dict()`:

```python
    def to_dict(self) -> Dict[str, Any]:
        """Converts CardData to a dict.

        Returns:
            `dict`: CardData represented as a dictionary ready to be dumped to a YAML
            block for inclusion in a README.md file.
        """

        data_dict = copy.deepcopy(self.__dict__)
        self._to_dict(data_dict)
        return _remove_none(data_dict)
```

Would it be ok to remove `list()` from being scanned in `_remove_none` ? it could also be a specific behavior to DatasetCardData if necessary @Wauplin 

I have actually no idea why none values are removed in model and dataset card data... :see_no_evil:
Looks like `_remove_none` has been introduced at the same time as the entire repocard module (see https://github.com/huggingface/huggingface_hub/pull/940). I would be tempted to remove `_remove_none` entirely actually and only remove "top-level" None values (i.e. if something like `pipeline_tag=None` due to a default value in kwargs => we remove it). Hard to tell what could be the side effects but I'm not against trying.


However, I'm not really in favor in making an exception only for lists. It would mean that tuples, sets and dicts are filtered but not lists, which is pretty inconsistent.

let's do it for top level attributes yes

I opened https://github.com/huggingface/huggingface_hub/pull/2626 to address it :)

thanks !