Kokolipa/github-issues huggingface dataset:
Issue: Support fsspec 2023.12
State: closed
Labels: name: enhancement

Support fsspec 2023.12 by handling previous and new glob behavior.

The pinned fsspec version range dependency conflict has been affecting several of our users in https://github.com/iterative/dvc. I've opened an initial PR that I think should resolve the glob behavior changes with using datasets + the latest fsspec release.

Please let us know if there's any other fsspec related behavior in datasets that needs to be updated to get 2024.2 supported, we'd like to get this conflict resolved as quickly as possible and we're willing to contribute any additional work that's required here.

cc @dberenbaum
Issue: Differently dataset object saved than it is loaded.
State: closed

### Describe the bug

Differently sized object is saved than it is loaded.

### Steps to reproduce the bug

Hi, I save dataset in a following way:
```
        dataset = load_dataset("json",
                               data_files={
                                   "train": os.path.join(input_folder, f"{task_meta_type}_{task_type}_train.jsonl"),
                                   "test": os.path.join(input_folder, f"{task_meta_type}_{task_type}_test.jsonl")})
        print(os.path.join(output_folder, f"{task_meta_type}_{task_type}"))
        print(f"Length of train dataset: {len(dataset['train'])}")
        print(f"Length of test dataset: {len(dataset['test'])}")
        dataset.save_to_disk(os.path.join(output_folder, f"{task_meta_type}_{task_type}"))
```

this yields output
```
.data/hf_dataset/propaganda_zanr
Length of train dataset: 7642
Length of test dataset: 1000
```

Everything looks fine.

Then I load the dataset
```python
from datasets import load_dataset

dataset_path = ".data/hf_dataset/propaganda_zanr"
dataset = load_dataset(dataset_path)

print(f"Length of train dataset: {len(dataset['train'])}")
print(f"Length of test dataset: {len(dataset['test'])}")

```

this prints
```
Generating train split: 1 examples [00:00, 72.10 examples/s]
Generating test split: 1 examples [00:00, 100.69 examples/s]
Length of train dataset: 1
Length of test dataset: 1
```

I dont' understand :(


### Expected behavior

same object is loaded

### Environment info

datasets==2.16.1

I see now, that I have to use `load_from_disk`, in order to load dataset properly, not `load_dataset`. Why is this behavior split? Why do we need both, `load_dataset` and `load_from_disk`?

Unless answered, I believe this might be helpful for other hf datasets newbies.

Anyway, made a `load_dataset` compatible dataset in a following way. I created a directory, and just copied jsonl there as `train.jsonl/test.jsonl`.
```python
output_folder = os.path.join(args.output_folder, f"{task_meta_type}_{task_type}")
os.makedirs(output_folder, exist_ok=True)
file = f"{task_meta_type}_{task_type}_train.jsonl"
shutil.copy(os.path.join(input_folder, file),
            os.path.join(output_folder, "train.jsonl"))
# now test
file = f"{task_meta_type}_{task_type}_test.jsonl"
shutil.copy(os.path.join(input_folder, file),
            os.path.join(output_folder, "test.jsonl"))
```


Hi @MFajcik, 

You can find information about save_to_disk/load_from_disk in our docs:
- https://huggingface.co/docs/datasets/v2.16.1/en/process#save
- https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/main_classes#datasets.Dataset.save_to_disk
- https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/main_classes#datasets.Dataset.load_from_disk
Issue: Sign Language Support
State: open
Labels: name: enhancement

### Feature request

Currently, there are only several Sign Language labels, I would like to propose adding all the Signed Languages as new labels which are described in this ISO standard: https://www.evertype.com/standards/iso639/sign-language.html

### Motivation

Datasets currently only have labels for several signed languages. There are more signed languages in the world. Furthermore, some signed languages that have a lot of online data cannot be found because of this reason (for instance, German Sign Language, and there is no German Sign Language label on huggingface datasets even though there are a lot of readily available sign language datasets exist for German Sign Language, which are used very frequently in Sign Language Processing papers, and models.)

### Your contribution

I can submit a PR for this as well, adding the ISO codes and languages to the labels in datasets.
Issue: multi-GPU map does not work
State: closed

### Describe the bug

Here is the code for single-GPU processing: https://pastebin.com/bfmEeK2y
Here is the code for multi-GPU processing: https://pastebin.com/gQ7i5AQy

Here is the video showing that the multi-GPU mapping does not work as expected (there are so many things wrong here, it's better to watch the 3-minute video than explain here):

https://youtu.be/RNbdPkSppc4

### Steps to reproduce the bug

-

### Expected behavior

-

### Environment info

x2 RTX A4000

This should now be fixed by https://github.com/huggingface/datasets/pull/6550 and updated with https://github.com/huggingface/datasets/pull/6646

Feel free to re-open if you're still having issues :)
Issue: wiki_dpr.py error (ID mismatch between lines {id} and vector {vec_id}
State: closed

### Describe the bug

I'm trying to run a rag example, and the dataset is wiki_dpr.
wiki_dpr download and extracting have been completed successfully.
However, at the generating train split stage, an error from wiki_dpr.py keeps popping up.

Especially in "_generate_examples" :
1. The following error occurs in the line **id, text, title = line.strip().split("\t")**
ValueError: not enough values ​​to unpack (expected 3, got 2)
-> This part handles exceptions so that even if an error occurs, it passes.

2. **ID mismatch between lines {id} and vector {vec_id}**
This error seems to occur at the line " assert int(id) == int(vec_id),".
After I handled the exception in the split error, generating train split progressed to 80%, but an id mismatch error occurred at about the 16200000th vector id.

Debugging is even more difficult because it takes a long time to download and split wiki_dpr. I need help. thank you in advance!!

### Steps to reproduce the bug

Occurs in the generating train split step when running the rag example in the transformers repository.
Specifically, it is an error in wiki_dpr.py.

### Expected behavior

.

### Environment info

python 3.8

Thanks for reporting, @kiehls90.

As this seems an issue with the specific "wiki_dpr" dataset, I am transferring the issue to the corresponding dataset page: https://huggingface.co/datasets/wiki_dpr/discussions/13
Issue: While importing load_dataset from datasets 
State: closed

### Describe the bug

cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' this is the error i received

### Steps to reproduce the bug

from datasets import load_dataset

### Expected behavior

No errors

### Environment info

python 3.11.5

Hi! Can you please share the error's stack trace so we can see where it comes from?

We cannot reproduce the issue and we do not have enough information: environment info (need to run `datasets-cli env`), stack trace,...

I am closing the issue. Feel free to reopen it (with additional information) if the problem persists.

Yeah 👍

On Tue, 6 Feb 2024 at 2:56 PM, Albert Villanova del Moral <
***@***.***> wrote:

> We cannot reproduce the issue and we do not have enough information:
> environment info (need to run datasets-cli env), stack trace,...
>
> I am closing the issue. Feel free to reopen it (with additional
> information) if the problem persists.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/6618#issuecomment-1929102334>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ASS4PJ3XOIIWISPY3VX3QRTYSHZK5AVCNFSM6AAAAABCL3BT4SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRZGEYDEMZTGQ>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


Please downgrade the version of urllib3 if you have the same issue:

!pip install urllib3==1.25.11

> Please downgrade the version of urllib3 if you have the same issue:
> 
> !pip install urllib3==1.25.11

This worked for me. Thanks.

I use python 3.11 and datasets==2.20.0. Downgrading urllib3 to 1.25.11 worked in my case.
Issue: `datasets/downloads` cleanup tool
State: open
Labels: name: enhancement

### Feature request

Splitting off https://github.com/huggingface/huggingface_hub/issues/1997 - currently `huggingface-cli delete-cache` doesn't take care of cleaning `datasets` temp files

e.g. I discovered having millions of files under `datasets/downloads` cache, I had to do:

```
sudo find /data/huggingface/datasets/downloads -type f -mtime +3 -exec rm {} \+
sudo find /data/huggingface/datasets/downloads -type d -empty -delete
```
 
could the cleanup be integrated into `huggingface-cli` or a different tool provided to keep the folders tidy and not consume inodes and space 

e.g. there were tens of thousands of `.lock` files - I don't know why they never get removed - lock files should be temporary for the duration of the operation requiring the lock and not remain after the operation finished, IMHO.

Also I think one should be able to nuke `datasets/downloads` w/o hurting the cache, but I think there are some datasets that rely on files extracted under this dir - or at least they did in the past - which is very difficult to manage since one has no idea what is safe to delete and what not.

Thank you

@Wauplin (requested to be tagged)
Issue: cnn_dailymail repeats itself
State: closed

### Describe the bug

When I try to load `cnn_dailymail` dataset, it takes longer than usual and when I checked the dataset it's 3x bigger than it's supposed to be.

Check https://huggingface.co/datasets/cnn_dailymail: it says 287k rows for train. But when I check length of train split it says 861339.

Also I checked data:

```
>>> ds['train']['highlights'][0]                                           
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday . Young actor says he has no plans to fritter his cash away . Radcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][0]                                           
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday . Young actor says he has no plans to fritter his cash away . Radcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][287113]
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\nYoung actor says he has no plans to fritter his cash away .\nRadcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][574226]
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\nYoung actor says he has no plans to fritter his cash away .\nRadcliffe's earnings from first five Potter films have been held in trust fund ."
```

The datasets seems to be updated 6 days ago to convert it to Parquet. Probably, there is some issue with backward compatability.



### Steps to reproduce the bug

1.
```
from datasets import load_dataset
ds = load_dataset('cnn_dailymail', '3.0.0')
len(ds['train'])
```

### Expected behavior

It should not repeat itself.

### Environment info

datasets==2.13.2
Python==3.7.13

Hi ! We recently updated `cnn_dailymail` and now `datasets>=2.14` is needed to load it.

You can update `datasets` with

```
pip install -U datasets
```
Issue: cast_column to Sequence(subfeatures_dict) has err
State: closed

### Describe the bug

I am working with the following demo code:
```
from datasets import load_dataset
from datasets.features import Sequence, Value, ClassLabel, Features

ais_dataset = load_dataset("/data/ryan.gao/ais_dataset_cache/raw/1978/")
ais_dataset = ais_dataset["train"]

def add_class(example):
    example["my_labeled_bbox"] = {"bbox": [100,100,200,200], "label": "cat"}
    return example

ais_dataset = ais_dataset.map(add_class, batched=False, num_proc=32)
ais_dataset = ais_dataset.cast_column("my_labeled_bbox", Sequence(
                                                        {
                                                            "bbox": Sequence(Value(dtype="int64")),
                                                            "label": ClassLabel(names=["cat", "dog"])
                                                        }))

print(ais_dataset[0])
```
However, executing this code results in an error:

```
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 2111, in cast_array_to_feature
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
TypeError: Couldn't cast array of type
int64
to
Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)
```
Upon examining the source code in datasets/table.py at line 2035:

```
        if isinstance(feature, Sequence) and isinstance(feature.feature, dict):
            feature = {
                name: Sequence(subfeature, length=feature.length) for name, subfeature in feature.feature.items()
            }
```

I noticed that if subfeature is of type Sequence, the code results in Sequence(Sequence(...), ...) and Sequence(ClassLabel(...), ...), which appears to be the source of the error.


### Steps to reproduce the bug

run my demo code

### Expected behavior

no exception

### Environment info

python 3.9 
datasets: 2.16.1

Hi! You are passing the wrong feature type to `cast_column`. This is the fixed call:
```python
ais_dataset = ais_dataset.cast_column("my_labeled_bbox", {"bbox": Sequence(Value(dtype="int64")), "label": ClassLabel(names=["cat", "dog"])})
```

> Hi! You are passing the wrong feature type to `cast_column`. This is the fixed call:
> 
> ```python
> ais_dataset = ais_dataset.cast_column("my_labeled_bbox", {"bbox": Sequence(Value(dtype="int64")), "label": ClassLabel(names=["cat", "dog"])})
> ```

thanks
Issue: Wrong path for cache directory in offline mode
State: closed

### Describe the bug

Dear huggingfacers,

I'm trying to use a subset of the-stack dataset. When I run the command the first time

```
dataset = load_dataset(
    path='bigcode/the-stack',
    data_dir='data/fortran',
    split='train' )
```

It downloads the files and caches them normally.

Nevertheless, since my compute nodes are not online (`HF_DATASETS_OFFLINE=1`) . Whenever I try to run the command again, the library is passing the wrong cache path:


`Cache directory for the-stack doesn't exist at /Users/user/.cache/huggingface/datasets/bigcode___the-stack/default-data_dir=data%2Ffortran-data_dir=data%2Ffortran`

when the right path is:

`'/Users/user/.cache/huggingface/datasets/bigcode___the-stack/default-data_dir=data\%2Ffortran`

Not sure why those redundancies are included in the path.  If I try adding the correct path through the the cache_dir argument it throws an error:

ConnectionError: Couldn't reach the Hugging Face Hub for dataset 'bigcode/the-stack': Offline mode is enabled.


Your help with this issue is greatly appreciated. Thanks a lot for the great work.


### Steps to reproduce the bug

1: 

`dataset = load_dataset(
    path='bigcode/the-stack',
    data_dir='data/fortran',
    split='train' )`


2: 

`HF_DATASETS_OFFLINE=1`

3:

`dataset = load_dataset(
    path='bigcode/the-stack',
    data_dir='data/fortran',
    split='train' )`


### Expected behavior

being able to use the cached data

### Environment info

several different systems

+1

same error in 2.16.1

@kongjiellx any luck with the issue?

I opened https://github.com/huggingface/datasets/pull/6632 to fix this issue. Once it's merged we'll do a new release of `datasets`

Thanks @lhoestq !
Issue: ELI5 no longer available, but referenced in example code
State: closed

Here, an example code is given:
https://huggingface.co/docs/transformers/tasks/language_modeling

This code + article references the ELI5 dataset.
ELI5 is no longer available, as the ELI5 dataset page states: https://huggingface.co/datasets/eli5
"Defunct: Dataset "eli5" is defunct and no longer accessible due to unavailability of the source data.
Reddit recently [changed the terms of access](https://www.reddit.com/r/reddit/comments/12qwagm/an_update_regarding_reddits_api/) to its API, making the source data for this dataset unavailable.
"

Please change the example code to use a different dataset.

Addressed in https://github.com/huggingface/transformers/pull/28715.
Issue: Transform fingerprint collisions due to setting fixed random seed
State: closed

### Describe the bug

The transform fingerprinting logic relies on the `random` library for random bits when the function is not hashable (e.g. bound methods as used in `trl`: https://github.com/huggingface/trl/blob/main/trl/trainer/dpo_trainer.py#L356). This causes collisions when the training code sets a fixed random seed, which is common practice: https://github.com/huggingface/alignment-handbook/blob/main/recipes/zephyr-7b-beta/sft/config_full.yaml#L45.

This results in fingerprint collisions which leads to silently loading incorrect cache files corresponding to completely different datasets.

### Steps to reproduce the bug

n/a

### Expected behavior

Use `uuid` v4 instead of `random.getrandbits()`

### Environment info

`datasets` main branch

I've opened a PR with a fix.

I don't think the PR fixes the root cause, since it still relies on the `random` library which will often have its seed fixed. I think the builtin `uuid.uuid4()` is a better choice: https://docs.python.org/3/library/uuid.html
Issue: datasets map `cache_file_name` does not work
State: open

### Describe the bug

In the documentation `datasets.Dataset.map` arg `cache_file_name` is said to be a string, but it doesn't work.

### Steps to reproduce the bug

1. pick a dataset
2. write a map function
3. do `ds.map(..., cache_file_name='some_filename')`
4. it crashes

### Expected behavior

It will tell you the filename you specified does not exist or it will generate a new file and tell you the filename does not exist.

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.10.201-168.748.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.12.2

Unfortunately, I'm unable to reproduce this error. Can you share the reproducer?

```
ds = datasets.Dataset.from_dict(dict(a=[i for i in range(100)]))
ds.map(lambda item: dict(b=item['a'] * 2), cache_file_name="/tmp/whatever-fn") # this worked
ds.map(lambda item: dict(b=item['a'] * 2), cache_file_name="/tmp/whatever-folder/filename") # this failed
ds.map(lambda item: dict(b=item['a'] * 2), cache_file_name="/tmp/whatever-folder/") # this failed


FileNotFoundError: [Errno 2] No such file or directory: '/tmp/whatever-folder/tmp1_izxvoo'
```

It will fail if the filename parents do not exists. If we have `os.makedirs("/tmp/whatever-folder")`, then it worked.

Maybe add the `mkdir -p` into the map function?
Issue: Index error when data is large
State: open

### Describe the bug

At `save_to_disk` step, the `max_shard_size` by default is `500MB`. However, one row of the dataset might be larger than `500MB` then the saving will throw an index error. Without looking at the source code, the bug is due to wrong calculation of number of shards which i think is 
`total_size / min(max_shard_size, row_size)` which should be `total_size / max(max_shard_size, row_size)`

The fix is setting a larger `max_shard_size`

### Steps to reproduce the bug

1. create a dataset with large dense tensors per row
2. set a small `max_shard_size` say 1MB
3. `save_to_disk`

### Expected behavior

```
raise IndexError(f"Index {index} out of range for dataset of size {size}.")

IndexError: Index 10 out of range for dataset of size 10.
```

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.10.201-168.748.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.12.2

I'm facing this problem while doing my translation of [mteb/stackexchange-clustering](https://huggingface.co/datasets/mteb/stackexchange-clustering). each row has lots of samples (up to 100k samples), because in this dataset, each row represent multiple clusters.
my hack is to setting `max_shard_size` to 20Gb or even larger
```py
final_dataset.push_to_hub(
        output_dataset, 
        private=True,
        max_shard_size="20GB"  # This will ensure appropriate sharding based on data size
    )
```
It will work, but depends on your data size.
Issue: Easy way to segment into 30s snippets given an m4a file and a vtt file 
State: closed
Labels: name: enhancement

### Feature request

Uploading datasets is straightforward thanks to the ability to push Audio to hub. However, it would be nice if the data (text and audio) could be segmented when being pushed (if not possible already).

### Motivation

It's easy to create a vtt file from an audio file. If there could be auto-segmenting, this would make the creation of datasets much faster.

### Your contribution

I have made a custom script to do this but it's not all that clean - uses librosa and pydub.

Hi! Non-generic data processing is out of this library's scope, so it's downstream libraries/users' responsibility to implement such logic.

That's fair. Thanks
Issue: Logs are delayed when doing .map when `docker logs`
State: closed

### Describe the bug

When I run my SD training in a Docker image and then listen to logs like `docker logs train -f`, the progress bar is delayed.
It's updating every few percent.
When you have a large dataset that has to be mapped (like 1+ million samples), it's crucial to see the updates in real-time, not every couple hours to make sure nothing got frozen or broken

### Steps to reproduce the bug

1. Run any huge dataset processing as a Docker image
2. `docker logs image_name` to it

### Expected behavior

...

### Environment info

...

Hi! `tqdm` doesn't work well in non-interactive environments, so there isn't much we can do about this. It's best to [disable it](https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/utilities#datasets.disable_progress_bars) in such environments and instead use logging to track progress.
Issue: The datasets models housed in Dropbox can't support a lot of users downloading them
State: closed

### Describe the bug

I'm using the datasets 


```
from datasets import load_dataset, Audio
dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
```


And it seems that sometimes when I imagine a lot of users are accessing the same resources, the Dropbox host fails:

`raise ConnectionError(f"Couldn't reach {url} (error {response.status_code})") ConnectionError: Couldn't reach https://www.dropbox.com/s/e2us0hcs3ilr20e/MInDS-14.zip?dl=1 (error 429)`

My question is if we can somehow host these files elsewhere or can you change the limit of simultaneous users accessing those resources or any other solution?

Also, has anyone had this issue before?

Thanks

### Steps to reproduce the bug

1: Create a python script like so:
```

from datasets import load_dataset, Audio
dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
```

2: Execute this by a certain number of users at the same time

### Expected behavior

I woudl expect that this shouldnt happen unless its a huge amount of users, which it is not the case

### Environment info

This was done in an Ubuntu 22 environment.

Hi! Indeed, Dropbox is not a reliable host. I've just merged https://huggingface.co/datasets/PolyAI/minds14/discussions/24 to fix this by hosting the data files inside the repo.
Issue: Feature request: Multi-GPU dataset mapping for SDXL training
State: open
Labels: name: enhancement

### Feature request

We need to speed up SDXL dataset pre-process. Please make it possible to use multiple GPUs for the [official SDXL trainer](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py) :)

### Motivation

Pre-computing 3 million of images takes around 2 days.
Would be nice to be able to be able to do multi-GPU (or even better – multi-GPU + multi-node) vae and embedding precompute...

### Your contribution

I'm not sure i can wrap my head around the multi-GPU mapping...
Plus it's too expensive for me to take x2 A100 and spend a day just figuring out the staff since I don't have a job right now.
Issue: After `2.16.0` version, there are `PermissionError` when users use shared cache_dir
State: closed

### Describe the bug

- We use shared `cache_dir` using `HF_HOME="{shared_directory}"` 
- After dataset version 2.16.0, datasets uses `filelock` package for file locking #6445 
- But, `filelock` package make `.lock` file with `644` permission
- Dataset is not available to other users except the user who created the lock file via `load_dataset`.

### Steps to reproduce the bug

1. `pip install datasets==2.16.0`
2. `export HF_HOME="{shared_directory}"`
3. download dataset with `load_dataset`
4. logout and login another user
5. `pip install datasets==2.16.0`
6. `export HF_HOME="{shared_directory}"`
7. download dataset with `load_dataset`
8. `PermissionError` occurs

### Expected behavior

- Users can share `cache_dir` using environment variable `HF_HOME`

### Environment info

- python == 3.9.10
- datasets == 2.16.0
- ubuntu 22.04
- shared_directory has ACL
![image (1)](https://github.com/huggingface/datasets/assets/106717516/5ca759db-ad0c-4883-9a97-9c8fccd00d8a)
- users are same group (developers)


We'll do a new release of `datasets` in the coming days with a fix !

@lhoestq Thank you very much!
Issue: fix os.listdir return name is empty string
State: closed

### Describe the bug

xlistdir return name is empty string 
Overloaded os.listdir

### Steps to reproduce the bug

```python
from datasets.download.streaming_download_manager import xjoin
from datasets.download.streaming_download_manager import xlistdir
config = DownloadConfig(storage_options=options)
manger = StreamingDownloadManager("ILSVRC2012",download_config=config)
input_path = "lakefs://datalab/main/imagenet/ILSVRC2012.zip"

download_files = manger.download_and_extract(input_path)

current_dir = xjoin(download_files,"ILSVRC2012/Images/ILSVRC2012_img_train")
folder_list = xlistdir(current_dir)
```

in xlistdir function

Obj ["name"] ends with "/"

last return ""

### Expected behavior

Obj ["name"] ends with "/" 
return folder name

### Environment info

no
Issue: losing DatasetInfo in Dataset.map when num_proc > 1
State: open

### Describe the bug

Hello and thanks for developing this package!

When I process a Dataset with the map function using multiple processors some set attributes of the DatasetInfo get lost and are None in the resulting Dataset.    

### Steps to reproduce the bug

```python
from datasets import Dataset, DatasetInfo


def run_map(num_proc):
    dataset = Dataset.from_dict(
        {"col1": [0, 1], "col2": [3, 4]},
        info=DatasetInfo(
            dataset_name="my_dataset",
        ),
    )

    ds = dataset.map(lambda x: x, num_proc=num_proc)
    print(ds.info.dataset_name)


run_map(1)
run_map(2)
```

This puts out:

```bash
Map: 100%|██████████| 2/2 [00:00<00:00, 724.66 examples/s]
my_dataset
Map (num_proc=2): 100%|██████████| 2/2 [00:00<00:00, 18.25 examples/s]
None
```

### Expected behavior

I expect the DatasetInfo to be kept as it was and there should be no difference in the output of running map with num_proc=1 and num_proc=2. 

Expected output:
```bash
Map: 100%|██████████| 2/2 [00:00<00:00, 724.66 examples/s]
my_dataset
Map (num_proc=2): 100%|██████████| 2/2 [00:00<00:00, 18.25 examples/s]
my_dataset
```

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.17
- Python version: 3.8.18
- `huggingface_hub` version: 0.20.2
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.9.2

Hi ! This issue comes from the fact that `map()` with `num_proc>1` shards the dataset in multiple chunks to be processed (one per process) and merges them. The DatasetInfos of each chunk are then merged together, but for some fields like `dataset_name` it's not been implemented and default to None.

The DatasetInfo merge is defined here, in case you'd like to contribute an improvement: 

https://github.com/huggingface/datasets/blob/d2e0034122a788015c0834a72e6c6279e7ecbac5/src/datasets/info.py#L269-L270

#self-assign
Issue: dataset cache only stores one config of the dataset in parquet dir, and uses that for all other configs resulting in showing same data in all configs.
State: closed

### Describe the bug

ds = load_dataset("ai2_arc", "ARC-Easy"), i have tried to force redownload, delete cache and changing the cache dir.

### Steps to reproduce the bug

dataset = []
dataset_name = "ai2_arc"
possible_configs = [
    'ARC-Challenge',
    'ARC-Easy'
    ]
for config in possible_configs:
    dataset_slice = load_dataset(dataset_name, config,ignore_verifications=True,cache_dir='ai2_arc_files')
    dataset.append(dataset_slice)

### Expected behavior

all configs should get saved in cache with their respective names.

### Environment info

ai2_arc
Issue: Unable to load `eli5` dataset with streaming
State: closed

### Describe the bug

Unable to load `eli5` dataset with streaming.

### Steps to reproduce the bug

This fails with FileNotFoundError: https://files.pushshift.io/reddit/submissions 
```
from datasets import load_dataset
load_dataset("eli5", streaming=True)
```

This works correctly.
```
from datasets import load_dataset
load_dataset("eli5")
```

### Expected behavior

- Loading `eli5` dataset should not raise an error under the streaming mode.
- Or at the very least, show a warning that streaming mode is not supported with `eli5` dataset.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.2.0-39-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.6.0


Hi @haok1402, I have created an issue in the Discussion tab of the corresponding dataset: https://huggingface.co/datasets/eli5/discussions/7
Let's continue the discussion there!
Issue: document page 404 not found after redirection
State: closed

### Describe the bug

The redirected page encountered 404 not found.

### Steps to reproduce the bug

1. In this tutorial: https://huggingface.co/learn/nlp-course/chapter5/4?fw=pt 
original md: https://github.com/huggingface/course/blob/2c733c2246b8b7e0e6f19a9e5d15bb12df43b2a3/chapters/en/chapter5/4.mdx#L49

```
By default, 🤗 Datasets will decompress the files needed to load a dataset. If you want to preserve hard drive space, you can pass `DownloadConfig(delete_extracted=True)` to the `download_config` argument of `load_dataset()`. See the [documentation](https://huggingface.co/docs/datasets/package_reference/builder_classes.html?#datasets.utils.DownloadConfig) for more details.
```
The documentation points to `https://huggingface.co/docs/datasets/package_reference/builder_classes.html?#datasets.utils.DownloadConfig`

it shows `The documentation page PACKAGE_REFERENCE/BUILDER_CLASSES.HTML doesn’t exist in v2.16.1, but exists on the main version. Click [here](https://huggingface.co/docs/datasets/main/en/package_reference/builder_classes.html) to redirect to the main version of the documentation.`

But the redirected website `https://huggingface.co/docs/datasets/main/en/package_reference/builder_classes.html` is 404 not found.

### Expected behavior

I Guess the redirected webisite should be 
`https://huggingface.co/docs/datasets/main/en/package_reference/builder_classes`    (without `.html`)
or `https://huggingface.co/docs/datasets/main/en/package_reference/builder_classes#datasets.DownloadConfig`.

### Environment info

Datasets main

Thanks for reporting! I've opened a PR with a fix.
Issue: Make DatasetDict.column_names return a list instead of dict
State: open
Labels: name: enhancement

Currently, `DatasetDict.column_names` returns a dict, with each split name as keys and the corresponding list of column names as values.

However, by construction, all splits have the same column names.

I think it makes more sense to return a single list with the column names, which is the same for all the split keys.

Hi @albertvillanova, can I work on this issue?
Issue: No online docs for 2.16 release
State: closed
Labels: name: bug

name: documentation

We do not have the online docs for the latest minor release 2.16 (2.16.0 nor 2.16.1).

In the online docs, the latest version appearing is 2.15.0: https://huggingface.co/docs/datasets/index

![Screenshot from 2024-01-09 08-43-08](https://github.com/huggingface/datasets/assets/8515462/83613222-867f-41f4-8833-7a4a76582f44)


Though the `build / build_main_documentation` CI job ran for 2.16.0: https://github.com/huggingface/datasets/actions/runs/7300836845/job/19896275099 🤔 

Yes, I saw it. Maybe @mishig25 can give us some hint...

fixed https://huggingface.co/docs/datasets/v2.16.0/en/index

Still missing 2.16.1.

> Still missing 2.16.1.

re-running the doc-buld job for the missing ones should fix



Re-running the job for the 2.16.1 release: https://github.com/huggingface/datasets/actions/runs/7365231552/job/20310278583

Fixed for 2.16.1: https://huggingface.co/docs/datasets/v2.16.1/en/index
Issue: keep_in_memory=True does not seem to work
State: open

UPD: [Fixed](https://github.com/huggingface/datasets/issues/6568#issuecomment-1880817794) . But a new issue came up :(

Seems like I just used the old code which did not have `keep_in_memory=True` argument, sorry.

Although i encountered a different problem – at 97% my python process just hung for around 11 minutes with no logs (when running dataset.map without `keep_in_memory=True` over around 3 million of dataset samples)...

Can you open a new issue and provide a bit more details ? What kind of map operations did you run ?

Hey. I will try to find some free time to describe it.

(can't do it now, cause i need to reproduce it myself to be sure about everything, which requires spinning a new Azuree VM, copying a huge dataset to drive from network disk for a long time etc...)

@lhoestq loading dataset like this does not spawn 50 python processes:

```
datasets.load_dataset("/preprocessed_2256k/train", num_proc=50)
```

I have 64 vCPU so i hoped it could speed up the dataset loading...

My dataset onlly has images and metadata.csv with text column alongside image file path column

now noticed
```
'Setting num_proc from 50 back to 1 for the train split to disable multiprocessing as it only contains one shard
```

Any way to work around this?

@lhoestq thanks, [this helped](https://github.com/huggingface/datasets/blob/9d6d16117a30ba345b0236407975f701c5b288d4/src/datasets/arrow_dataset.py#L1053)
Issue: AttributeError: 'str' object has no attribute 'to'
State: closed

### Describe the bug

```
--------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-6-80c6086794e8>](https://localhost:8080/#) in <cell line: 10>()
      8 		  report_to="wandb")
      9 
---> 10 trainer = Trainer(
     11     model=model,
     12     args=training_args,

1 frames
[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _move_model_to_device(self, model, device)
    688 
    689     def _move_model_to_device(self, model, device):
--> 690         model = model.to(device)
    691         # Moving a model to an XLA device disconnects the tied weights, so we have to retie them.
    692         if self.args.parallel_mode == ParallelMode.TPU and hasattr(model, "tie_weights"):

AttributeError: 'str' object has no attribute 'to'
```

### Steps to reproduce the bug

here is the notebook:
```
https://colab.research.google.com/drive/10JDBNsLlYrQdnI2FWfDK3F5M8wvVUDXG?usp=sharing

```


### Expected behavior

run the Training 

### Environment info

Colab Notebook , T4 

I think you are reporting an issue with the `transformers` library. Note this is the repository of the `datasets` library. I recommend that you open an issue in their repository: https://github.com/huggingface/transformers/issues

EDIT: I have not the rights to transfer the issue
~~I am transferring your issue to their repository.~~

Thanks, I hope someone from transformers library addresses this issue.

On Mon, Jan 8, 2024 at 15:29 Albert Villanova del Moral <
***@***.***> wrote:

> I think you are reporting an issue with the transformers library. Note
> this is the repository of the datasets library. I am transferring your
> issue to their repository.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/6567#issuecomment-1880688586>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AE4LJNOYMD6WJMXFKPMH6DLYNO7PJAVCNFSM6AAAAABBQ63HWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBQGY4DQNJYGY>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


@andysingal, I recommend that you open an issue in their repository: https://github.com/huggingface/transformers/issues
I don't have the rights to transfer this issue to their repo.
Issue: `Dataset.filter` missing `with_rank` parameter
State: closed

### Describe the bug

The issue shall be open: https://github.com/huggingface/datasets/issues/6435

When i try to pass `with_rank` to `Dataset.filter()`, i get this:

`Dataset.filter() got an unexpected keyword argument 'with_rank'`

### Steps to reproduce the bug

Run notebook:

https://colab.research.google.com/drive/1WUNKph8BdP0on5ve3gQnh_PE0cFLQqTn?usp=sharing

### Expected behavior

Should work?

### Environment info

NVIDIA RTX 4090

Thanks for reporting! I've opened a PR with a fix

@mariosasko thank you very much :)
Issue: datasets.DownloadMode.FORCE_REDOWNLOAD use cache to download dataset features with load_dataset function
State: open

### Describe the bug

I have updated my dataset by adding a new feature, and push it to the hub. When I want to download it on my machine which contain the old version by using `datasets.load_dataset("your_dataset_name", download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)` I get an error (paste bellow).

Seems that the load_dataset function still use the old features schema instead of downloading everything new from the HUB.

I find a way to go around this issue by manually deleting the old dataset cache. But from my understanding of `datasets.DownloadMode.FORCE_REDOWNLOAD` option, the dataset cache should be ignored.

### Steps to reproduce the bug

1. Download your dataset in your machine using `datasets.load_dataset`
2. Create a new feature in your dataset and push it to the hub
3. On the same machine redownload your dataset using `datasets.load_dataset("your_dataset_name", download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)`

### Expected behavior

`
ValueError: Couldn't cast
id: string
level: string
context: list<element: string>
  child 0, element: string
type: string
answer: string
question: string
supporting_facts: list<element: string>
  child 0, element: string
fra_answer: string
fra_question: string
-- schema metadata --
huggingface: '{"info": {"features": {"id": {"dtype": "string", "_type": "' + 490
to
{'id': Value(dtype='string', id=None), 'level': Value(dtype='string', id=None), 'context': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'type': Value(dtype='string', id=None), 'answer': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'supporting_facts': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
because column names don't match

The above exception was the direct cause of the following exception:

DatasetGenerationError
...
DatasetGenerationError: An error occurred while generating the dataset`

### Environment info

datasets-2.16.1 huggingface-hub-0.20.2
Issue: Document YAML configuration with "data_dir"
State: open
Labels: name: documentation

See https://huggingface.co/datasets/uonlp/CulturaX/discussions/15#6597e83f185db94370d6bf50 for reference

In particular, I would like to have an example of how to replace the following configuration (from https://huggingface.co/docs/hub/datasets-manual-configuration#splits)

```
---
configs:
- config_name: default
  data_files:
  - split: train
    path: "data/*.csv"
  - split: test
    path: "holdout/*.csv"
---
```

with the `data_dir` field.

Hey @severo can I work on this , 

Yes, thanks! I assigned you.

please open a PR instead of commenting here

Hi! I've researched this issue and found the root cause of the confusion.

Problem Analysis:
The issue is that `data_dir` cannot be used in YAML configurations (README.md), but this isn't clearly documented anywhere. Users expect it to work based on the programmatic API.

 Key Findings : 
- `data_dir` works for: `load_dataset("csv", data_dir="folder")`  
- `data_dir` does NOT work in YAML configs - must use `data_files` with `path`
- The distinction isn't explained in current documentation

 Proposed Solution : 
Add clear documentation explaining:
1. `data_dir` is for programmatic loading only
2. YAML configs must use `data_files` with glob patterns like `path: "folder/*.csv"`
3. Migration examples for common mistakes

Question for Maintainers : 
Should this documentation be added to:
 A) `huggingface/hub-docs` (covers YAML configurations)
 B) `huggingface/datasets` (covers the library)  
 C) Both repositories?

I'm ready to create a PR once you confirm the preferred location!

hmmm, @lhoestq says otherwise in https://huggingface.co/datasets/uonlp/CulturaX/discussions/15#6597e83f185db94370d6bf50

> Using data_dir in YAML is supported and is equivalent to passing data_dir= to load_dataset

Could you double check?

(maybe support has been dropped since)
Issue: Cannot import name 'load_dataset' from .... module ‘datasets’
State: closed

### Describe the bug


use python -m pip install datasets to install

### Steps to reproduce the bug

from datasets import load_dataset

### Expected behavior

it doesn't work

### Environment info

datasets version==2.15.0
python == 3.10.12
linux  version I don't know??

I don't know  My conpany conputer cannot work. but in my computer, it work?

Do you have a folder in your working directory called datasets?
Issue: Loading a dataset from Google Colab hangs at "Resolving data files".
State: closed

### Describe the bug

Hello,

I'm trying to load a dataset from Google Colab but the process hangs at `Resolving data files`:

![image](https://github.com/huggingface/datasets/assets/99779/7175ad85-e571-46ed-9f87-92653985777d)

It is happening when the `_get_origin_metadata` definition is invoked:

```python
def _get_origin_metadata(
    data_files: List[str],
    max_workers=64,
    download_config: Optional[DownloadConfig] = None,
) -> Tuple[str]:
    return thread_map(
        partial(_get_single_origin_metadata, download_config=download_config),
        data_files,
        max_workers=max_workers,
        tqdm_class=hf_tqdm,
        desc="Resolving data files",
        disable=len(data_files) <= 16,
```

The thread is then stuck at `waiter.acquire()` in the builtin `threading.py` file.

I can load the dataset just fine on my machine.

Cheers,

Thomas


### Steps to reproduce the bug

In Google Colab:

```python
!pip install datasets
from datasets import load_dataset

dataset = load_dataset("colour-science/color-checker-detection-dataset")
```

### Expected behavior

The dataset should be loaded.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.1
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0

This bug comes from the `huggingface_hub` library, see: https://github.com/huggingface/huggingface_hub/issues/1952

A fix is provided at https://github.com/huggingface/huggingface_hub/pull/1953. Feel free to install `huggingface_hub` from this PR, or wait for it to be merged and the new version of `huggingface_hub` to be released

Thanks!
Issue: Loading from hf hub with clearer error message
State: open
Labels: name: enhancement

### Feature request

Shouldn't this kinda work ?
```
Dataset.from_json("hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json")
```
I got an error
```
File ~/miniconda3/envs/datatrove/lib/python3.10/site-packages/datasets/data_files.py:380, in resolve_pattern(pattern, base_path, allowed_extensions, download_config)
    378     if allowed_extensions is not None:
    379         error_msg += f" with any supported extension {list(allowed_extensions)}"
--> 380     raise FileNotFoundError(error_msg)
    381 return out

FileNotFoundError: Unable to find 'hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json'
(I'm logged in)
```

Fix: the correct path is
```
hf://datasets/HuggingFaceTB/eval_data/eval_data_context_and_answers.json
```

Proposal: raise a clearer error

### Motivation

Clearer error message

### Your contribution

Can open a PR

Maybe we can add a helper message like `Maybe try again using "hf://path/without/resolve"` if the path contains `/resolve/` ?

e.g.

```
FileNotFoundError: Unable to find 'hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json'
It looks like you used parts of the URL of the file from the Hugging Face website, but you should remove the "/resolve/<revision>" part to have a valid `hf://` path.
Please try again using this path instead:
    hf://datasets/HuggingFaceTB/eval_data/eval_data_context_and_answers.json
```

and suggest `f"hf://datasets/HuggingFaceTB/eval_data@{revision}/eval_data_context_and_answers.json"` if revision != "main"

EDIT: I think this message should also be raised from the  `huggingface_hub`'s `HfFileSystem` implementation
Issue: Skip if a dataset has issues
State: open

### Describe the bug

Hello everyone,
I'm using **load_datasets** from **huggingface** to download the datasets and I'm facing an issue, the download starts but it reaches some state and then  fails with the following error:
Couldn't reach https://huggingface.co/datasets/wikimedia/wikipedia/resolve/4cb9b0d719291f1a10f96f67d609c5d442980dc9/20231101.ext/train-00000-of-00001.parquet

Failed to resolve \'huggingface.co\' ([Errno -3] Temporary failure in name resolution)"))')))


![image](https://github.com/huggingface/datasets/assets/143214684/8847d9cb-529e-4eda-9c76-282713dfa3af)

so I was wondering is there a parameter to be passed to load_dataset() to skip files that can't be downloaded??

### Steps to reproduce the bug

Parameter to be passed to load_dataset() of huggingface to skip files that can't be downloaded??

### Expected behavior

load_dataset() finishes without error

### Environment info

None

It looks like a transient DNS issue. It should work fine now if you try again.

There is no parameter in load_dataset to skip failed downloads. In your case it would have skipped every single subsequent download until the DNS issue was resolved anyway.