helmo/github-issues huggingface dataset:
`datasets/downloads` cleanup tool
State: open | Author: stas00 (CONTRIBUTOR)
Labels: name: enhancement

### Feature request

Splitting off https://github.com/huggingface/huggingface_hub/issues/1997 - currently `huggingface-cli delete-cache` doesn't take care of cleaning `datasets` temp files

e.g. I discovered having millions of files under `datasets/downloads` cache, I had to do:

```
sudo find /data/huggingface/datasets/downloads -type f -mtime +3 -exec rm {} \+
sudo find /data/huggingface/datasets/downloads -type d -empty -delete
```
 
could the cleanup be integrated into `huggingface-cli` or a different tool provided to keep the folders tidy and not consume inodes and space 

e.g. there were tens of thousands of `.lock` files - I don't know why they never get removed - lock files should be temporary for the duration of the operation requiring the lock and not remain after the operation finished, IMHO.

Also I think one should be able to nuke `datasets/downloads` w/o hurting the cache, but I think there are some datasets that rely on files extracted under this dir - or at least they did in the past - which is very difficult to manage since one has no idea what is safe to delete and what not.

Thank you

@Wauplin (requested to be tagged)
cnn_dailymail repeats itself
State: closed | Author: KeremZaman (NONE)

### Describe the bug

When I try to load `cnn_dailymail` dataset, it takes longer than usual and when I checked the dataset it's 3x bigger than it's supposed to be.

Check https://huggingface.co/datasets/cnn_dailymail: it says 287k rows for train. But when I check length of train split it says 861339.

Also I checked data:

```
>>> ds['train']['highlights'][0]                                           
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday . Young actor says he has no plans to fritter his cash away . Radcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][0]                                           
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday . Young actor says he has no plans to fritter his cash away . Radcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][287113]
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\nYoung actor says he has no plans to fritter his cash away .\nRadcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][574226]
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\nYoung actor says he has no plans to fritter his cash away .\nRadcliffe's earnings from first five Potter films have been held in trust fund ."
```

The datasets seems to be updated 6 days ago to convert it to Parquet. Probably, there is some issue with backward compatability.



### Steps to reproduce the bug

1.
```
from datasets import load_dataset
ds = load_dataset('cnn_dailymail', '3.0.0')
len(ds['train'])
```

### Expected behavior

It should not repeat itself.

### Environment info

datasets==2.13.2
Python==3.7.13

Comments: Hi ! We recently updated `cnn_dailymail` and now `datasets>=2.14` is needed to load it.

You can update `datasets` with

```
pip install -U datasets
```
cast_column to Sequence(subfeatures_dict) has err
State: closed | Author: neiblegy (NONE)

### Describe the bug

I am working with the following demo code:
```
from datasets import load_dataset
from datasets.features import Sequence, Value, ClassLabel, Features

ais_dataset = load_dataset("/data/ryan.gao/ais_dataset_cache/raw/1978/")
ais_dataset = ais_dataset["train"]

def add_class(example):
    example["my_labeled_bbox"] = {"bbox": [100,100,200,200], "label": "cat"}
    return example

ais_dataset = ais_dataset.map(add_class, batched=False, num_proc=32)
ais_dataset = ais_dataset.cast_column("my_labeled_bbox", Sequence(
                                                        {
                                                            "bbox": Sequence(Value(dtype="int64")),
                                                            "label": ClassLabel(names=["cat", "dog"])
                                                        }))

print(ais_dataset[0])
```
However, executing this code results in an error:

```
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 2111, in cast_array_to_feature
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
TypeError: Couldn't cast array of type
int64
to
Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)
```
Upon examining the source code in datasets/table.py at line 2035:

```
        if isinstance(feature, Sequence) and isinstance(feature.feature, dict):
            feature = {
                name: Sequence(subfeature, length=feature.length) for name, subfeature in feature.feature.items()
            }
```

I noticed that if subfeature is of type Sequence, the code results in Sequence(Sequence(...), ...) and Sequence(ClassLabel(...), ...), which appears to be the source of the error.


### Steps to reproduce the bug

run my demo code

### Expected behavior

no exception

### Environment info

python 3.9 
datasets: 2.16.1

Comments: Hi! You are passing the wrong feature type to `cast_column`. This is the fixed call:
```python
ais_dataset = ais_dataset.cast_column("my_labeled_bbox", {"bbox": Sequence(Value(dtype="int64")), "label": ClassLabel(names=["cat", "dog"])})
```

> Hi! You are passing the wrong feature type to `cast_column`. This is the fixed call:
> 
> ```python
> ais_dataset = ais_dataset.cast_column("my_labeled_bbox", {"bbox": Sequence(Value(dtype="int64")), "label": ClassLabel(names=["cat", "dog"])})
> ```

thanks
Wrong path for cache directory in offline mode
State: closed | Author: je-santos (NONE)

### Describe the bug

Dear huggingfacers,

I'm trying to use a subset of the-stack dataset. When I run the command the first time

```
dataset = load_dataset(
    path='bigcode/the-stack',
    data_dir='data/fortran',
    split='train' )
```

It downloads the files and caches them normally.

Nevertheless, since my compute nodes are not online (`HF_DATASETS_OFFLINE=1`) . Whenever I try to run the command again, the library is passing the wrong cache path:


`Cache directory for the-stack doesn't exist at /Users/user/.cache/huggingface/datasets/bigcode___the-stack/default-data_dir=data%2Ffortran-data_dir=data%2Ffortran`

when the right path is:

`'/Users/user/.cache/huggingface/datasets/bigcode___the-stack/default-data_dir=data\%2Ffortran`

Not sure why those redundancies are included in the path.  If I try adding the correct path through the the cache_dir argument it throws an error:

ConnectionError: Couldn't reach the Hugging Face Hub for dataset 'bigcode/the-stack': Offline mode is enabled.


Your help with this issue is greatly appreciated. Thanks a lot for the great work.


### Steps to reproduce the bug

1: 

`dataset = load_dataset(
    path='bigcode/the-stack',
    data_dir='data/fortran',
    split='train' )`


2: 

`HF_DATASETS_OFFLINE=1`

3:

`dataset = load_dataset(
    path='bigcode/the-stack',
    data_dir='data/fortran',
    split='train' )`


### Expected behavior

being able to use the cached data

### Environment info

several different systems

Comments: +1

same error in 2.16.1

@kongjiellx any luck with the issue?

I opened https://github.com/huggingface/datasets/pull/6632 to fix this issue. Once it's merged we'll do a new release of `datasets`

Thanks @lhoestq !
ELI5 no longer available, but referenced in example code
State: closed | Author: drdsgvo (NONE)

Here, an example code is given:
https://huggingface.co/docs/transformers/tasks/language_modeling

This code + article references the ELI5 dataset.
ELI5 is no longer available, as the ELI5 dataset page states: https://huggingface.co/datasets/eli5
"Defunct: Dataset "eli5" is defunct and no longer accessible due to unavailability of the source data.
Reddit recently [changed the terms of access](https://www.reddit.com/r/reddit/comments/12qwagm/an_update_regarding_reddits_api/) to its API, making the source data for this dataset unavailable.
"

Please change the example code to use a different dataset.

Comments: Addressed in https://github.com/huggingface/transformers/pull/28715.
Transform fingerprint collisions due to setting fixed random seed
State: closed | Author: normster (NONE)

### Describe the bug

The transform fingerprinting logic relies on the `random` library for random bits when the function is not hashable (e.g. bound methods as used in `trl`: https://github.com/huggingface/trl/blob/main/trl/trainer/dpo_trainer.py#L356). This causes collisions when the training code sets a fixed random seed, which is common practice: https://github.com/huggingface/alignment-handbook/blob/main/recipes/zephyr-7b-beta/sft/config_full.yaml#L45.

This results in fingerprint collisions which leads to silently loading incorrect cache files corresponding to completely different datasets.

### Steps to reproduce the bug

n/a

### Expected behavior

Use `uuid` v4 instead of `random.getrandbits()`

### Environment info

`datasets` main branch

Comments: I've opened a PR with a fix.

I don't think the PR fixes the root cause, since it still relies on the `random` library which will often have its seed fixed. I think the builtin `uuid.uuid4()` is a better choice: https://docs.python.org/3/library/uuid.html
datasets map `cache_file_name` does not work
State: open | Author: ChenchaoZhao (NONE)

### Describe the bug

In the documentation `datasets.Dataset.map` arg `cache_file_name` is said to be a string, but it doesn't work.

### Steps to reproduce the bug

1. pick a dataset
2. write a map function
3. do `ds.map(..., cache_file_name='some_filename')`
4. it crashes

### Expected behavior

It will tell you the filename you specified does not exist or it will generate a new file and tell you the filename does not exist.

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.10.201-168.748.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.12.2

Comments: Unfortunately, I'm unable to reproduce this error. Can you share the reproducer?

```
ds = datasets.Dataset.from_dict(dict(a=[i for i in range(100)]))
ds.map(lambda item: dict(b=item['a'] * 2), cache_file_name="/tmp/whatever-fn") # this worked
ds.map(lambda item: dict(b=item['a'] * 2), cache_file_name="/tmp/whatever-folder/filename") # this failed
ds.map(lambda item: dict(b=item['a'] * 2), cache_file_name="/tmp/whatever-folder/") # this failed


FileNotFoundError: [Errno 2] No such file or directory: '/tmp/whatever-folder/tmp1_izxvoo'
```

It will fail if the filename parents do not exists. If we have `os.makedirs("/tmp/whatever-folder")`, then it worked.

Maybe add the `mkdir -p` into the map function?
Index error when data is large
State: open | Author: ChenchaoZhao (NONE)

### Describe the bug

At `save_to_disk` step, the `max_shard_size` by default is `500MB`. However, one row of the dataset might be larger than `500MB` then the saving will throw an index error. Without looking at the source code, the bug is due to wrong calculation of number of shards which i think is 
`total_size / min(max_shard_size, row_size)` which should be `total_size / max(max_shard_size, row_size)`

The fix is setting a larger `max_shard_size`

### Steps to reproduce the bug

1. create a dataset with large dense tensors per row
2. set a small `max_shard_size` say 1MB
3. `save_to_disk`

### Expected behavior

```
raise IndexError(f"Index {index} out of range for dataset of size {size}.")

IndexError: Index 10 out of range for dataset of size 10.
```

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.10.201-168.748.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.12.2

Comments: I'm facing this problem while doing my translation of [mteb/stackexchange-clustering](https://huggingface.co/datasets/mteb/stackexchange-clustering). each row has lots of samples (up to 100k samples), because in this dataset, each row represent multiple clusters.
my hack is to setting `max_shard_size` to 20Gb or even larger
```py
final_dataset.push_to_hub(
        output_dataset, 
        private=True,
        max_shard_size="20GB"  # This will ensure appropriate sharding based on data size
    )
```
It will work, but depends on your data size.
add safety checks when using only part of dataset
State: open | Author: benseddikismail (NONE)

Added some checks to prevent errors that arrise when using evaluate.py on only a portion of the squad 2.0 dataset.

Comments: Hi ! The metrics in `datasets` are deprecated in favor of https://github.com/huggingface/evaluate

You can open a PR here instead: https://huggingface.co/spaces/evaluate-metric/squad_v2/tree/main
Easy way to segment into 30s snippets given an m4a file and a vtt file 
State: closed | Author: RonanKMcGovern (NONE)
Labels: name: enhancement

### Feature request

Uploading datasets is straightforward thanks to the ability to push Audio to hub. However, it would be nice if the data (text and audio) could be segmented when being pushed (if not possible already).

### Motivation

It's easy to create a vtt file from an audio file. If there could be auto-segmenting, this would make the creation of datasets much faster.

### Your contribution

I have made a custom script to do this but it's not all that clean - uses librosa and pydub.

Comments: Hi! Non-generic data processing is out of this library's scope, so it's downstream libraries/users' responsibility to implement such logic.

That's fair. Thanks
Logs are delayed when doing .map when `docker logs`
State: closed | Author: kopyl (NONE)

### Describe the bug

When I run my SD training in a Docker image and then listen to logs like `docker logs train -f`, the progress bar is delayed.
It's updating every few percent.
When you have a large dataset that has to be mapped (like 1+ million samples), it's crucial to see the updates in real-time, not every couple hours to make sure nothing got frozen or broken

### Steps to reproduce the bug

1. Run any huge dataset processing as a Docker image
2. `docker logs image_name` to it

### Expected behavior

...

### Environment info

...

Comments: Hi! `tqdm` doesn't work well in non-interactive environments, so there isn't much we can do about this. It's best to [disable it](https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/utilities#datasets.disable_progress_bars) in such environments and instead use logging to track progress.
The datasets models housed in Dropbox can't support a lot of users downloading them
State: closed | Author: RDaneelOlivav (NONE)

### Describe the bug

I'm using the datasets 


```
from datasets import load_dataset, Audio
dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
```


And it seems that sometimes when I imagine a lot of users are accessing the same resources, the Dropbox host fails:

`raise ConnectionError(f"Couldn't reach {url} (error {response.status_code})") ConnectionError: Couldn't reach https://www.dropbox.com/s/e2us0hcs3ilr20e/MInDS-14.zip?dl=1 (error 429)`

My question is if we can somehow host these files elsewhere or can you change the limit of simultaneous users accessing those resources or any other solution?

Also, has anyone had this issue before?

Thanks

### Steps to reproduce the bug

1: Create a python script like so:
```

from datasets import load_dataset, Audio
dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
```

2: Execute this by a certain number of users at the same time

### Expected behavior

I woudl expect that this shouldnt happen unless its a huge amount of users, which it is not the case

### Environment info

This was done in an Ubuntu 22 environment.

Comments: Hi! Indeed, Dropbox is not a reliable host. I've just merged https://huggingface.co/datasets/PolyAI/minds14/discussions/24 to fix this by hosting the data files inside the repo.
Feature request: Multi-GPU dataset mapping for SDXL training
State: open | Author: kopyl (NONE)
Labels: name: enhancement

### Feature request

We need to speed up SDXL dataset pre-process. Please make it possible to use multiple GPUs for the [official SDXL trainer](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py) :)

### Motivation

Pre-computing 3 million of images takes around 2 days.
Would be nice to be able to be able to do multi-GPU (or even better – multi-GPU + multi-node) vae and embedding precompute...

### Your contribution

I'm not sure i can wrap my head around the multi-GPU mapping...
Plus it's too expensive for me to take x2 A100 and spend a day just figuring out the staff since I don't have a job right now.
After `2.16.0` version, there are `PermissionError` when users use shared cache_dir
State: closed | Author: minhopark-neubla (NONE)

### Describe the bug

- We use shared `cache_dir` using `HF_HOME="{shared_directory}"` 
- After dataset version 2.16.0, datasets uses `filelock` package for file locking #6445 
- But, `filelock` package make `.lock` file with `644` permission
- Dataset is not available to other users except the user who created the lock file via `load_dataset`.

### Steps to reproduce the bug

1. `pip install datasets==2.16.0`
2. `export HF_HOME="{shared_directory}"`
3. download dataset with `load_dataset`
4. logout and login another user
5. `pip install datasets==2.16.0`
6. `export HF_HOME="{shared_directory}"`
7. download dataset with `load_dataset`
8. `PermissionError` occurs

### Expected behavior

- Users can share `cache_dir` using environment variable `HF_HOME`

### Environment info

- python == 3.9.10
- datasets == 2.16.0
- ubuntu 22.04
- shared_directory has ACL
![image (1)](https://github.com/huggingface/datasets/assets/106717516/5ca759db-ad0c-4883-9a97-9c8fccd00d8a)
- users are same group (developers)


Comments: We'll do a new release of `datasets` in the coming days with a fix !

@lhoestq Thank you very much!
fix os.listdir return name is empty string
State: closed | Author: d710055071 (CONTRIBUTOR)

### Describe the bug

xlistdir return name is empty string 
Overloaded os.listdir

### Steps to reproduce the bug

```python
from datasets.download.streaming_download_manager import xjoin
from datasets.download.streaming_download_manager import xlistdir
config = DownloadConfig(storage_options=options)
manger = StreamingDownloadManager("ILSVRC2012",download_config=config)
input_path = "lakefs://datalab/main/imagenet/ILSVRC2012.zip"

download_files = manger.download_and_extract(input_path)

current_dir = xjoin(download_files,"ILSVRC2012/Images/ILSVRC2012_img_train")
folder_list = xlistdir(current_dir)
```

in xlistdir function

Obj ["name"] ends with "/"

last return ""

### Expected behavior

Obj ["name"] ends with "/" 
return folder name

### Environment info

no
losing DatasetInfo in Dataset.map when num_proc > 1
State: open | Author: JochenSiegWork (CONTRIBUTOR)

### Describe the bug

Hello and thanks for developing this package!

When I process a Dataset with the map function using multiple processors some set attributes of the DatasetInfo get lost and are None in the resulting Dataset.    

### Steps to reproduce the bug

```python
from datasets import Dataset, DatasetInfo


def run_map(num_proc):
    dataset = Dataset.from_dict(
        {"col1": [0, 1], "col2": [3, 4]},
        info=DatasetInfo(
            dataset_name="my_dataset",
        ),
    )

    ds = dataset.map(lambda x: x, num_proc=num_proc)
    print(ds.info.dataset_name)


run_map(1)
run_map(2)
```

This puts out:

```bash
Map: 100%|██████████| 2/2 [00:00<00:00, 724.66 examples/s]
my_dataset
Map (num_proc=2): 100%|██████████| 2/2 [00:00<00:00, 18.25 examples/s]
None
```

### Expected behavior

I expect the DatasetInfo to be kept as it was and there should be no difference in the output of running map with num_proc=1 and num_proc=2. 

Expected output:
```bash
Map: 100%|██████████| 2/2 [00:00<00:00, 724.66 examples/s]
my_dataset
Map (num_proc=2): 100%|██████████| 2/2 [00:00<00:00, 18.25 examples/s]
my_dataset
```

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.17
- Python version: 3.8.18
- `huggingface_hub` version: 0.20.2
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.9.2

Comments: Hi ! This issue comes from the fact that `map()` with `num_proc>1` shards the dataset in multiple chunks to be processed (one per process) and merges them. The DatasetInfos of each chunk are then merged together, but for some fields like `dataset_name` it's not been implemented and default to None.

The DatasetInfo merge is defined here, in case you'd like to contribute an improvement: 

https://github.com/huggingface/datasets/blob/d2e0034122a788015c0834a72e6c6279e7ecbac5/src/datasets/info.py#L269-L270

#self-assign
dataset cache only stores one config of the dataset in parquet dir, and uses that for all other configs resulting in showing same data in all configs.
State: closed | Author: kartikgupta321 (NONE)

### Describe the bug

ds = load_dataset("ai2_arc", "ARC-Easy"), i have tried to force redownload, delete cache and changing the cache dir.

### Steps to reproduce the bug

dataset = []
dataset_name = "ai2_arc"
possible_configs = [
    'ARC-Challenge',
    'ARC-Easy'
    ]
for config in possible_configs:
    dataset_slice = load_dataset(dataset_name, config,ignore_verifications=True,cache_dir='ai2_arc_files')
    dataset.append(dataset_slice)

### Expected behavior

all configs should get saved in cache with their respective names.

### Environment info

ai2_arc
Unable to load `eli5` dataset with streaming
State: closed | Author: haok1402 (NONE)

### Describe the bug

Unable to load `eli5` dataset with streaming.

### Steps to reproduce the bug

This fails with FileNotFoundError: https://files.pushshift.io/reddit/submissions 
```
from datasets import load_dataset
load_dataset("eli5", streaming=True)
```

This works correctly.
```
from datasets import load_dataset
load_dataset("eli5")
```

### Expected behavior

- Loading `eli5` dataset should not raise an error under the streaming mode.
- Or at the very least, show a warning that streaming mode is not supported with `eli5` dataset.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.2.0-39-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.6.0


Comments: Hi @haok1402, I have created an issue in the Discussion tab of the corresponding dataset: https://huggingface.co/datasets/eli5/discussions/7
Let's continue the discussion there!