KRadim/github-issues-minimal huggingface dataset:
Transform fingerprint collisions due to setting fixed random seed
State: closed

### Describe the bug

The transform fingerprinting logic relies on the `random` library for random bits when the function is not hashable (e.g. bound methods as used in `trl`: https://github.com/huggingface/trl/blob/main/trl/trainer/dpo_trainer.py#L356). This causes collisions when the training code sets a fixed random seed, which is common practice: https://github.com/huggingface/alignment-handbook/blob/main/recipes/zephyr-7b-beta/sft/config_full.yaml#L45.

This results in fingerprint collisions which leads to silently loading incorrect cache files corresponding to completely different datasets.

### Steps to reproduce the bug

n/a

### Expected behavior

Use `uuid` v4 instead of `random.getrandbits()`

### Environment info

`datasets` main branch
datasets map `cache_file_name` does not work
State: open

### Describe the bug

In the documentation `datasets.Dataset.map` arg `cache_file_name` is said to be a string, but it doesn't work.

### Steps to reproduce the bug

1. pick a dataset
2. write a map function
3. do `ds.map(..., cache_file_name='some_filename')`
4. it crashes

### Expected behavior

It will tell you the filename you specified does not exist or it will generate a new file and tell you the filename does not exist.

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.10.201-168.748.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.12.2
Index error when data is large
State: open

### Describe the bug

At `save_to_disk` step, the `max_shard_size` by default is `500MB`. However, one row of the dataset might be larger than `500MB` then the saving will throw an index error. Without looking at the source code, the bug is due to wrong calculation of number of shards which i think is 
`total_size / min(max_shard_size, row_size)` which should be `total_size / max(max_shard_size, row_size)`

The fix is setting a larger `max_shard_size`

### Steps to reproduce the bug

1. create a dataset with large dense tensors per row
2. set a small `max_shard_size` say 1MB
3. `save_to_disk`

### Expected behavior

```
raise IndexError(f"Index {index} out of range for dataset of size {size}.")

IndexError: Index 10 out of range for dataset of size 10.
```

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.10.201-168.748.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.12.2
Loading CSV exported dataset has unexpected format
State: open

### Describe the bug

I wanted to be able to save a HF dataset for translations and load it again in another script, but I'm a bit confused with the documentation and the result I've got so I'm opening this issue to ask if this behavior is as expected.

### Steps to reproduce the bug

The documentation I've mainly consulted is https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/loading_methods#datasets.load_dataset and https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset (where I've found `.to_csv()`)

```python
# Load a dataset of translations
test_dataset = load_dataset("opus100", name="en-fr", split="test")

# Save with .to_csv()
test_csv_path = "try_testset_save.csv"
test_dataset.to_csv(test_csv_path)

# Load dataset from the CSV
loaded_dataset = load_dataset("csv", data_files=test_csv_path)
print(test_dataset_fromfile[0]["translation"])
print(test_dataset_fromfile[0]["translation"]["en"])
```
```
Creating CSV from Arrow format: 100%
2/2 [00:00<00:00, 47.99ba/s]
Downloading data files: 100%
1/1 [00:00<00:00, 65.33it/s]
Extracting data files: 100%
1/1 [00:00<00:00, 42.10it/s]
Generating train split:
2000/0 [00:00<00:00, 47486.09 examples/s]

{'en': "She wasn't going to vaccinate her kid against polio,  no way.", 'fr': 'Elle ne vaccinerait pas son enfant contre la polio. Pas question.'}

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[29], line 11
      9 loaded_dataset = load_dataset("csv", data_files=test_csv_path)
     10 print(test_dataset_fromfile[0]["translation"])
---> 11 print(test_dataset_fromfile[0]["translation"]["en"])

TypeError: string indices must be integers, not 'str'
```


### Expected behavior

Each translation was saved as a stringified dict like `"{'en': ""She wasn't going to vaccinate her kid against polio,  no way."", 'fr': 'Elle ne vaccinerait pas son enfant contre la polio. Pas question.'}"` where I would have expected 2 columns (1st with English segments, and 2nd with French segments), and I was expecting `load_dataset` to infer the type of feature automatically as I haven't seen anything about it in the documentation.

Do you have an example of how to effectively save and load datasets of translations ?

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-3.10.0-1160.36.2.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.11.5
- `huggingface_hub` version: 0.16.4
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0
Easy way to segment into 30s snippets given an m4a file and a vtt file 
State: closed
Labels: name: enhancement

### Feature request

Uploading datasets is straightforward thanks to the ability to push Audio to hub. However, it would be nice if the data (text and audio) could be segmented when being pushed (if not possible already).

### Motivation

It's easy to create a vtt file from an audio file. If there could be auto-segmenting, this would make the creation of datasets much faster.

### Your contribution

I have made a custom script to do this but it's not all that clean - uses librosa and pydub.
Dataset.push_to_hub of a canonical dataset creates an additional dataset under the user namespace
State: closed
Labels: name: bug

While using `Dataset.push_to_hub` of a canonical dataset, an additional dataset was created under my user namespace.

## Steps to reproduce the bug
The command:
```python
commit_info = ds.push_to_hub(
    "caner",
    config_name="default",
    commit_message="Convert dataset to Parquet",
    commit_description="Convert dataset to Parquet.",
    create_pr=True,
    token=token,
)
```
creates the additional dataset `albertvillanova/caner`.
IterableDataset sharding logic needs improvement
State: open

### Describe the bug

The sharding of IterableDatasets with respect to distributed and dataloader worker processes appears problematic with significant performance traps and inconsistencies wrt to distributed train processes vs worker processes.

Splitting across num_workers (per train process loader processes) and world_size (distributed training processes) appears inconsistent.
* worker split: https://github.com/huggingface/datasets/blob/9d6d16117a30ba345b0236407975f701c5b288d4/src/datasets/iterable_dataset.py#L1266-L1283
* distributed split: https://github.com/huggingface/datasets/blob/9d6d16117a30ba345b0236407975f701c5b288d4/src/datasets/iterable_dataset.py#L1335-L1356

In the case of the distributed split, there is a modulus check that flips between two very different behaviours, why is this different than splitting across the data loader workers? For IterableDatasets the DataLoaders worker processes are independent, so whether it's workers within one train process or across a distributed world the shards should be distributed the same, across `world_size * num_worker` independent workers in either case... 

Further, the fallback case when the `n_shards % world_size == 0` check fails is a rather extreme change. I argue it is not desirable to do that implicitly, it should be an explicit case for specific scenarios (ie reliable validation). A train scenario would likely be much better handled with improved wrapping / stopping behaviour to eg also fix #6437. Changing from stepping shards to stepping samples means that every single process reads ALL of the shards. This was never an intended default for sharded training, shards gain their performance advantage in large scale distributed training by explicitly avoiding the need to have every process overlapping in the data they read, by default, only the data allocated to each process via their assigned shards should be read in each pass of the dataset.

Using a large scale CLIP example, some of the larger datasets have 10-20k shards across 100+TB of data. Training with 1000 GPUs we are switching between reading 100 terabytes per epoch to 100 petabytes if say change 20k % 1000 and drop one gpu-node to 20k % 992.

The 'step over samples' case might be worth the overhead in specific validation scenarios where gaurantees of at least/most once samples seen are more important and do not make up a significant portion of train time or are done in smaller world sizes outside of train.

### Steps to reproduce the bug

N/A

### Expected behavior

We have an iterable dataset with N shards, to split across workers
* shuffle shards (same seed across all train processes)
* step shard iterator across distributed processes
* step shard iterator across dataloader worker processes
* shuffle samples in every worker via shuffle buffer (different seed in each worker, but ideally controllable (based on base seed + worker id + epoch).
* end up with (possibly uneven) number of shards per worker but each shard only ever accessed by 1 worker per pass (epoch)



### Environment info

N/A
Logs are delayed when doing .map when `docker logs`
State: closed

### Describe the bug

When I run my SD training in a Docker image and then listen to logs like `docker logs train -f`, the progress bar is delayed.
It's updating every few percent.
When you have a large dataset that has to be mapped (like 1+ million samples), it's crucial to see the updates in real-time, not every couple hours to make sure nothing got frozen or broken

### Steps to reproduce the bug

1. Run any huge dataset processing as a Docker image
2. `docker logs image_name` to it

### Expected behavior

...

### Environment info

...
The datasets models housed in Dropbox can't support a lot of users downloading them
State: closed

### Describe the bug

I'm using the datasets 


```
from datasets import load_dataset, Audio
dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
```


And it seems that sometimes when I imagine a lot of users are accessing the same resources, the Dropbox host fails:

`raise ConnectionError(f"Couldn't reach {url} (error {response.status_code})") ConnectionError: Couldn't reach https://www.dropbox.com/s/e2us0hcs3ilr20e/MInDS-14.zip?dl=1 (error 429)`

My question is if we can somehow host these files elsewhere or can you change the limit of simultaneous users accessing those resources or any other solution?

Also, has anyone had this issue before?

Thanks

### Steps to reproduce the bug

1: Create a python script like so:
```

from datasets import load_dataset, Audio
dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
```

2: Execute this by a certain number of users at the same time

### Expected behavior

I woudl expect that this shouldnt happen unless its a huge amount of users, which it is not the case

### Environment info

This was done in an Ubuntu 22 environment.
Feature request: Multi-GPU dataset mapping for SDXL training
State: open
Labels: name: enhancement

### Feature request

We need to speed up SDXL dataset pre-process. Please make it possible to use multiple GPUs for the [official SDXL trainer](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py) :)

### Motivation

Pre-computing 3 million of images takes around 2 days.
Would be nice to be able to be able to do multi-GPU (or even better – multi-GPU + multi-node) vae and embedding precompute...

### Your contribution

I'm not sure i can wrap my head around the multi-GPU mapping...
Plus it's too expensive for me to take x2 A100 and spend a day just figuring out the staff since I don't have a job right now.
After `2.16.0` version, there are `PermissionError` when users use shared cache_dir
State: closed

### Describe the bug

- We use shared `cache_dir` using `HF_HOME="{shared_directory}"` 
- After dataset version 2.16.0, datasets uses `filelock` package for file locking #6445 
- But, `filelock` package make `.lock` file with `644` permission
- Dataset is not available to other users except the user who created the lock file via `load_dataset`.

### Steps to reproduce the bug

1. `pip install datasets==2.16.0`
2. `export HF_HOME="{shared_directory}"`
3. download dataset with `load_dataset`
4. logout and login another user
5. `pip install datasets==2.16.0`
6. `export HF_HOME="{shared_directory}"`
7. download dataset with `load_dataset`
8. `PermissionError` occurs

### Expected behavior

- Users can share `cache_dir` using environment variable `HF_HOME`

### Environment info

- python == 3.9.10
- datasets == 2.16.0
- ubuntu 22.04
- shared_directory has ACL
![image (1)](https://github.com/huggingface/datasets/assets/106717516/5ca759db-ad0c-4883-9a97-9c8fccd00d8a)
- users are same group (developers)
fix os.listdir return name is empty string
State: closed

### Describe the bug

xlistdir return name is empty string 
Overloaded os.listdir

### Steps to reproduce the bug

```python
from datasets.download.streaming_download_manager import xjoin
from datasets.download.streaming_download_manager import xlistdir
config = DownloadConfig(storage_options=options)
manger = StreamingDownloadManager("ILSVRC2012",download_config=config)
input_path = "lakefs://datalab/main/imagenet/ILSVRC2012.zip"

download_files = manger.download_and_extract(input_path)

current_dir = xjoin(download_files,"ILSVRC2012/Images/ILSVRC2012_img_train")
folder_list = xlistdir(current_dir)
```

in xlistdir function

Obj ["name"] ends with "/"

last return ""

### Expected behavior

Obj ["name"] ends with "/" 
return folder name

### Environment info

no
Allow concatenation of datasets with mixed structs
State: closed

Fixes #6466 

The idea is to do a recursive check for structs. PyArrow handles it well enough.


For a demo you can do:

```python
from datasets import Dataset, concatenate_datasets

ds = Dataset.from_dict({'speaker': [{'name': 'Ben', 'email': None}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
print(concatenate_datasets([ds, ds2]).features)
print(concatenate_datasets([ds, ds2]).to_dict())
```

Now both the features and the rows are fixed.

I note that Sequence suffers from the same problem, so I can fix that in a future PR once this one is merged.
losing DatasetInfo in Dataset.map when num_proc > 1
State: open

### Describe the bug

Hello and thanks for developing this package!

When I process a Dataset with the map function using multiple processors some set attributes of the DatasetInfo get lost and are None in the resulting Dataset.    

### Steps to reproduce the bug

```python
from datasets import Dataset, DatasetInfo


def run_map(num_proc):
    dataset = Dataset.from_dict(
        {"col1": [0, 1], "col2": [3, 4]},
        info=DatasetInfo(
            dataset_name="my_dataset",
        ),
    )

    ds = dataset.map(lambda x: x, num_proc=num_proc)
    print(ds.info.dataset_name)


run_map(1)
run_map(2)
```

This puts out:

```bash
Map: 100%|██████████| 2/2 [00:00<00:00, 724.66 examples/s]
my_dataset
Map (num_proc=2): 100%|██████████| 2/2 [00:00<00:00, 18.25 examples/s]
None
```

### Expected behavior

I expect the DatasetInfo to be kept as it was and there should be no difference in the output of running map with num_proc=1 and num_proc=2. 

Expected output:
```bash
Map: 100%|██████████| 2/2 [00:00<00:00, 724.66 examples/s]
my_dataset
Map (num_proc=2): 100%|██████████| 2/2 [00:00<00:00, 18.25 examples/s]
my_dataset
```

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.17
- Python version: 3.8.18
- `huggingface_hub` version: 0.20.2
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.9.2
np.fromfile not supported
State: open

How to do np.fromfile  to use it like np.load


```python
def xnumpy_fromfile(filepath_or_buffer, *args, download_config: Optional[DownloadConfig] = None, **kwargs):
    import numpy as np

    if hasattr(filepath_or_buffer, "read"):
        return np.fromfile(filepath_or_buffer, *args, **kwargs)
    else:
        filepath_or_buffer = str(filepath_or_buffer)
        return np.fromfile(xopen(filepath_or_buffer, "rb", download_config=download_config).read(), *args, **kwargs)
```
this is not work
Fix for Incorrect ex_iterable used with multi num_worker
State: closed

Corrects an issue where `self._ex_iterable` was erroneously used instead of `ex_iterable`, when both Distributed Data Parallel (DDP) and multi num_worker are used concurrently. This improper usage led to the generation of incorrect `shards_indices`, subsequently causing issues with the control flow responsible for worker creation. The fix ensures the appropriate iterable is used, thus providing a more accurate determination of whether a new worker should be instantiated or not.
fix os.listdir return name is empty string
State: closed

fix #6588

xlistdir return name is empty string

for example：
`
from datasets.download.streaming_download_manager import xjoin
from datasets.download.streaming_download_manager import xlistdir
config = DownloadConfig(storage_options=options)
manger = StreamingDownloadManager("ILSVRC2012",download_config=config)
input_path = "lakefs://datalab/main/imagenet/ILSVRC2012.zip"


download_files = manger.download_and_extract(input_path)

current_dir = xjoin(download_files,"ILSVRC2012/Images/ILSVRC2012_img_train")
folder_list = xlistdir(current_dir)

`
dataset cache only stores one config of the dataset in parquet dir, and uses that for all other configs resulting in showing same data in all configs.
State: closed

### Describe the bug

ds = load_dataset("ai2_arc", "ARC-Easy"), i have tried to force redownload, delete cache and changing the cache dir.

### Steps to reproduce the bug

dataset = []
dataset_name = "ai2_arc"
possible_configs = [
    'ARC-Challenge',
    'ARC-Easy'
    ]
for config in possible_configs:
    dataset_slice = load_dataset(dataset_name, config,ignore_verifications=True,cache_dir='ai2_arc_files')
    dataset.append(dataset_slice)

### Expected behavior

all configs should get saved in cache with their respective names.

### Environment info

ai2_arc
Unable to load `eli5` dataset with streaming
State: closed

### Describe the bug

Unable to load `eli5` dataset with streaming.

### Steps to reproduce the bug

This fails with FileNotFoundError: https://files.pushshift.io/reddit/submissions 
```
from datasets import load_dataset
load_dataset("eli5", streaming=True)
```

This works correctly.
```
from datasets import load_dataset
load_dataset("eli5")
```

### Expected behavior

- Loading `eli5` dataset should not raise an error under the streaming mode.
- Or at the very least, show a warning that streaming mode is not supported with `eli5` dataset.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.2.0-39-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.6.0
document page 404 not found after redirection
State: closed

### Describe the bug

The redirected page encountered 404 not found.

### Steps to reproduce the bug

1. In this tutorial: https://huggingface.co/learn/nlp-course/chapter5/4?fw=pt 
original md: https://github.com/huggingface/course/blob/2c733c2246b8b7e0e6f19a9e5d15bb12df43b2a3/chapters/en/chapter5/4.mdx#L49

```
By default, 🤗 Datasets will decompress the files needed to load a dataset. If you want to preserve hard drive space, you can pass `DownloadConfig(delete_extracted=True)` to the `download_config` argument of `load_dataset()`. See the [documentation](https://huggingface.co/docs/datasets/package_reference/builder_classes.html?#datasets.utils.DownloadConfig) for more details.
```
The documentation points to `https://huggingface.co/docs/datasets/package_reference/builder_classes.html?#datasets.utils.DownloadConfig`

it shows `The documentation page PACKAGE_REFERENCE/BUILDER_CLASSES.HTML doesn’t exist in v2.16.1, but exists on the main version. Click [here](https://huggingface.co/docs/datasets/main/en/package_reference/builder_classes.html) to redirect to the main version of the documentation.`

But the redirected website `https://huggingface.co/docs/datasets/main/en/package_reference/builder_classes.html` is 404 not found.

### Expected behavior

I Guess the redirected webisite should be 
`https://huggingface.co/docs/datasets/main/en/package_reference/builder_classes`    (without `.html`)
or `https://huggingface.co/docs/datasets/main/en/package_reference/builder_classes#datasets.DownloadConfig`.

### Environment info

Datasets main
Adding option for multipart achive download
State: closed

Right now we can only download multiple separate archives or a single file archive, but not multipart archives, such as those produced by `tar --multi-volume`. This PR allows for downloading and extraction of archives split into multiple parts.

With the new `multi_part` field of the `DownloadConfig` set, the downloader will first retrieve all the files and attempt to concatenate them before starting extraction. This will obviously fail if files retrieved are actually multiple separate archives, so the option is set to `False` by default.

Tests and docs incoming.
No online docs for 2.16 release
State: closed
Labels: name: bug, name: documentation

We do not have the online docs for the latest minor release 2.16 (2.16.0 nor 2.16.1).

In the online docs, the latest version appearing is 2.15.0: https://huggingface.co/docs/datasets/index

![Screenshot from 2024-01-09 08-43-08](https://github.com/huggingface/datasets/assets/8515462/83613222-867f-41f4-8833-7a4a76582f44)
AttributeError: 'str' object has no attribute 'to'
State: closed

### Describe the bug

```
--------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-6-80c6086794e8>](https://localhost:8080/#) in <cell line: 10>()
      8 		  report_to="wandb")
      9 
---> 10 trainer = Trainer(
     11     model=model,
     12     args=training_args,

1 frames
[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _move_model_to_device(self, model, device)
    688 
    689     def _move_model_to_device(self, model, device):
--> 690         model = model.to(device)
    691         # Moving a model to an XLA device disconnects the tied weights, so we have to retie them.
    692         if self.args.parallel_mode == ParallelMode.TPU and hasattr(model, "tie_weights"):

AttributeError: 'str' object has no attribute 'to'
```

### Steps to reproduce the bug

here is the notebook:
```
https://colab.research.google.com/drive/10JDBNsLlYrQdnI2FWfDK3F5M8wvVUDXG?usp=sharing

```


### Expected behavior

run the Training 

### Environment info

Colab Notebook , T4
I train controlnet_sdxl in bf16 datatype, got unsupported ERROR in datasets
State: closed
Labels: name: bug

### Describe the bug

```
Traceback (most recent call last):
  File "train_controlnet_sdxl.py", line 1252, in <module>
    main(args)
  File "train_controlnet_sdxl.py", line 1013, in main
    train_dataset = train_dataset.map(compute_embeddings_fn, batched=True, new_fingerprint=new_fingerprint)
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 592, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3093, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3489, in _map_single
    writer.write_batch(batch)
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/arrow_writer.py", line 557, in write_batch
    arrays.append(pa.array(typed_sequence))
  File "pyarrow/array.pxi", line 248, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 113, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/arrow_writer.py", line 191, in __arrow_array__
    out = pa.array(cast_to_python_objects(data, only_1d_for_numpy=True))
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/features/features.py", line 447, in cast_to_python_objects
    return _cast_to_python_objects(
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/features/features.py", line 324, in _cast_to_python_objects
    for x in obj.detach().cpu().numpy()
TypeError: Got unsupported ScalarType BFloat16
```

### Steps to reproduce the bug

Here is my train script I use BF16 type，I use diffusers train my model
```
export MODEL_DIR="/home/mhh/sd_models/stable-diffusion-xl-base-1.0"
export OUTPUT_DIR="./control_net"
export VAE_NAME="/home/mhh/sd_models/sdxl-vae-fp16-fix"

accelerate launch train_controlnet_sdxl.py \
 --pretrained_model_name_or_path=$MODEL_DIR \
 --output_dir=$OUTPUT_DIR \
 --pretrained_vae_model_name_or_path=$VAE_NAME \
 --dataset_name=/home/mhh/sd_datasets/fusing/fill50k \
 --mixed_precision="bf16" \
 --resolution=1024 \
 --learning_rate=1e-5 \
 --max_train_steps=200 \
 --validation_image "/home/mhh/sd_datasets/controlnet_image/conditioning_image_1.png" "/home/mhh/sd_datasets/controlnet_image/conditioning_image_2.png" \
 --validation_prompt "red circle with blue background" "cyan circle with brown floral background" \
 --validation_steps=50 \
 --train_batch_size=1 \
 --gradient_accumulation_steps=4 \
 --report_to="wandb" \
 --seed=42 \
```

### Expected behavior

When I changed the data type to fp16, it worked.

### Environment info

datasets                2.16.1
numpy                   1.24.4
`drop_last_batch=True` for IterableDataset map function is ignored with multiprocessing DataLoader 
State: closed

### Describe the bug

Scenario:
- Interleaving two iterable datasets of unequal lengths (`all_exhausted`), followed by a batch mapping with batch size 2 to effectively merge the two datasets and get a sample from each dataset in a single batch, with `drop_last_batch=True` to skip the last batch in case it doesn't have two samples.

What works:
- Using DataLoader with `num_workers=0`

What does not work:
- Using DataLoader with `num_workers=1`, errors in the last batch.

Basically, `drop_last_batch=True` is ignored when using multiple dataloading workers.

Please take a look at the minimal repro script below.

### Steps to reproduce the bug

```python
from datasets import Dataset, interleave_datasets
from torch.utils.data import DataLoader


def merge_samples(batch):
    assert len(batch['a']) == 2, "Batch size must be 2"
    batch['c'] = [batch['a'][0]]
    batch['d'] = [batch['a'][1]]
    return batch


def gen1():
    for ii in range(1, 8385):
        yield {"a": ii}


def gen2():
    for ii in range(1, 5302):
        yield {"a": ii}


if __name__ == '__main__':

    dataset1 = Dataset.from_generator(gen1).to_iterable_dataset(num_shards=1024)
    dataset2 = Dataset.from_generator(gen2).to_iterable_dataset(num_shards=1024)

    interleaved = interleave_datasets([dataset1, dataset2], stopping_strategy="all_exhausted")
    mapped = interleaved.map(merge_samples, batched=True, batch_size=2, remove_columns=interleaved.column_names,
                             drop_last_batch=True)

    # Works
    loader = DataLoader(mapped, batch_size=32, num_workers=0)
    i = 0
    for b in loader:
        print(i, b['c'].shape, b['d'].shape)
        i += 1

    print("DataLoader with num_workers=0 works")

    # Doesn't work
    loader = DataLoader(mapped, batch_size=32, num_workers=1)
    i = 0
    for b in loader:
        print(i, b['c'].shape, b['d'].shape)
        i += 1


```

### Expected behavior

 `drop_last_batch=True` should have same behaviour for `num_workers=0` and `num_workers>=1`

### Environment info

- `datasets` version: 2.16.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.2
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.6.0

I have also tested on Linux and got the same behavior.
`Dataset.filter` missing `with_rank` parameter
State: closed

### Describe the bug

The issue shall be open: https://github.com/huggingface/datasets/issues/6435

When i try to pass `with_rank` to `Dataset.filter()`, i get this:

`Dataset.filter() got an unexpected keyword argument 'with_rank'`

### Steps to reproduce the bug

Run notebook:

https://colab.research.google.com/drive/1WUNKph8BdP0on5ve3gQnh_PE0cFLQqTn?usp=sharing

### Expected behavior

Should work?

### Environment info

NVIDIA RTX 4090
datasets.DownloadMode.FORCE_REDOWNLOAD use cache to download dataset features with load_dataset function
State: open

### Describe the bug

I have updated my dataset by adding a new feature, and push it to the hub. When I want to download it on my machine which contain the old version by using `datasets.load_dataset("your_dataset_name", download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)` I get an error (paste bellow).

Seems that the load_dataset function still use the old features schema instead of downloading everything new from the HUB.

I find a way to go around this issue by manually deleting the old dataset cache. But from my understanding of `datasets.DownloadMode.FORCE_REDOWNLOAD` option, the dataset cache should be ignored.

### Steps to reproduce the bug

1. Download your dataset in your machine using `datasets.load_dataset`
2. Create a new feature in your dataset and push it to the hub
3. On the same machine redownload your dataset using `datasets.load_dataset("your_dataset_name", download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)`

### Expected behavior

`
ValueError: Couldn't cast
id: string
level: string
context: list<element: string>
  child 0, element: string
type: string
answer: string
question: string
supporting_facts: list<element: string>
  child 0, element: string
fra_answer: string
fra_question: string
-- schema metadata --
huggingface: '{"info": {"features": {"id": {"dtype": "string", "_type": "' + 490
to
{'id': Value(dtype='string', id=None), 'level': Value(dtype='string', id=None), 'context': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'type': Value(dtype='string', id=None), 'answer': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'supporting_facts': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
because column names don't match

The above exception was the direct cause of the following exception:

DatasetGenerationError
...
DatasetGenerationError: An error occurred while generating the dataset`

### Environment info

datasets-2.16.1 huggingface-hub-0.20.2
Latest version 2.16.1, when load dataset error occurs. ValueError: BuilderConfig 'allenai--c4' not found. Available: ['default']
State: closed

### Describe the bug

python script is:
```

from datasets import load_dataset
cache_dir = 'path/to/your/cache/directory'
dataset = load_dataset('allenai/c4','allenai--c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train',  use_auth_token=False, cache_dir=cache_dir)
```


the script success when datasets version is 2.14.7. 
when using 2.16.1, error occurs
`
ValueError: BuilderConfig 'allenai--c4' not found. Available: ['default']`

### Steps to reproduce the bug

1. pip install datasets==2.16.1
2. run python script:
```

from datasets import load_dataset
cache_dir = 'path/to/your/cache/directory'
dataset = load_dataset('allenai/c4','allenai--c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train',  use_auth_token=False, cache_dir=cache_dir)
```

### Expected behavior

the dataset should be loaded successful in the latest version. 

### Environment info

datasets 2.16.1
Fix imagefolder with one image
State: closed

A dataset repository with one image and one metadata file was considered a JSON dataset instead of an ImageFolder dataset. This is because we pick the dataset type with the most compatible data file extensions present in the repository and it results in a tie in this case.

e.g. for https://huggingface.co/datasets/multimodalart/repro_1_image

I fixed this by deprioritizing metadata files in the count.

fix https://github.com/huggingface/datasets/issues/6545
Loading a dataset from Google Colab hangs at "Resolving data files".
State: closed

### Describe the bug

Hello,

I'm trying to load a dataset from Google Colab but the process hangs at `Resolving data files`:

![image](https://github.com/huggingface/datasets/assets/99779/7175ad85-e571-46ed-9f87-92653985777d)

It is happening when the `_get_origin_metadata` definition is invoked:

```python
def _get_origin_metadata(
    data_files: List[str],
    max_workers=64,
    download_config: Optional[DownloadConfig] = None,
) -> Tuple[str]:
    return thread_map(
        partial(_get_single_origin_metadata, download_config=download_config),
        data_files,
        max_workers=max_workers,
        tqdm_class=hf_tqdm,
        desc="Resolving data files",
        disable=len(data_files) <= 16,
```

The thread is then stuck at `waiter.acquire()` in the builtin `threading.py` file.

I can load the dataset just fine on my machine.

Cheers,

Thomas


### Steps to reproduce the bug

In Google Colab:

```python
!pip install datasets
from datasets import load_dataset

dataset = load_dataset("colour-science/color-checker-detection-dataset")
```

### Expected behavior

The dataset should be loaded.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.1
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0
Fix parallel downloads for datasets without scripts
State: closed

Enable parallel downloads using multiprocessing when `num_proc` is passed to `load_dataset`.

It was enabled for datasets with scripts already (if they passed lists to `dl_manager.download`) but not for no-script datasets (we pass dicts {split: [list of files]} to `dl_manager.download` for those ones).

I fixed this by parallelising on the lists contained in the data files dicts when possible.

I also added a context manager `stack_multiprocessing_download_progress_bars` in `DownloadManager` to stack the progress bard of the downloads (from `cached_path(...)` calls). Otherwise the progress bars overlap each other with an annoying flickering effect.
Loading from hf hub with clearer error message
State: open
Labels: name: enhancement

### Feature request

Shouldn't this kinda work ?
```
Dataset.from_json("hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json")
```
I got an error
```
File ~/miniconda3/envs/datatrove/lib/python3.10/site-packages/datasets/data_files.py:380, in resolve_pattern(pattern, base_path, allowed_extensions, download_config)
    378     if allowed_extensions is not None:
    379         error_msg += f" with any supported extension {list(allowed_extensions)}"
--> 380     raise FileNotFoundError(error_msg)
    381 return out

FileNotFoundError: Unable to find 'hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json'
(I'm logged in)
```

Fix: the correct path is
```
hf://datasets/HuggingFaceTB/eval_data/eval_data_context_and_answers.json
```

Proposal: raise a clearer error

### Motivation

Clearer error message

### Your contribution

Can open a PR
Skip if a dataset has issues
State: open

### Describe the bug

Hello everyone,
I'm using **load_datasets** from **huggingface** to download the datasets and I'm facing an issue, the download starts but it reaches some state and then  fails with the following error:
Couldn't reach https://huggingface.co/datasets/wikimedia/wikipedia/resolve/4cb9b0d719291f1a10f96f67d609c5d442980dc9/20231101.ext/train-00000-of-00001.parquet

Failed to resolve \'huggingface.co\' ([Errno -3] Temporary failure in name resolution)"))')))


![image](https://github.com/huggingface/datasets/assets/143214684/8847d9cb-529e-4eda-9c76-282713dfa3af)

so I was wondering is there a parameter to be passed to load_dataset() to skip files that can't be downloaded??

### Steps to reproduce the bug

Parameter to be passed to load_dataset() of huggingface to skip files that can't be downloaded??

### Expected behavior

load_dataset() finishes without error

### Environment info

None
`image` column not automatically inferred if image dataset only contains 1 image
State: closed

### Describe the bug

By default, the standard Image Dataset maps out `file_name` to `image` when loading an Image Dataset. 

However, if the dataset contains only 1 image, this does not take place 

### Steps to reproduce the bug

Input
(dataset with one image `multimodalart/repro_1_image`) 
```py
from datasets import load_dataset
dataset = load_dataset("multimodalart/repro_1_image")
dataset
```

Output: 
```py
DatasetDict({
    train: Dataset({
        features: ['file_name', 'prompt'],
        num_rows: 1
    })
})
```

Input 
(dataset with 2+ images `multimodalart/repro_2_image`) 
```py
from datasets import load_dataset
dataset = load_dataset("multimodalart/repro_2_image")
dataset
```

Output:
```py
DatasetDict({
    train: Dataset({
        features: ['image', 'prompt'],
        num_rows: 2
    })
})
```

### Expected behavior

Expected to map `file_name`  → `image` for all dataset sizes, including 1. 

### Environment info

Both latest main and 2.16.0
Datasets : wikipedia 20220301.en error 
State: closed

### Describe the bug

When I used load_dataset to download this data set, the following error occurred. The main problem was that the target data did not exist.

### Steps to reproduce the bug

1.I tried downloading directly.
```python
wiki_dataset = load_dataset("wikipedia", "20220301.en")
```
An exception occurred
```
MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). 
Example of usage: 
	`load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')`
```
2.I modified the code as prompted.
```python
wiki_dataset = load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')
```
An exception occurred:
```
FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/enwiki/20220301/dumpstatus.json
```


### Expected behavior

I searched in the parent directory of the corresponding URL, but there was no corresponding "20220301" directory.
I really need this data set and hope to provide a download method.

### Environment info

python 3.8
datasets 2.16.0
apache-beam 2.52.0
dill 0.3.7
Extreme inefficiency for `save_to_disk` when merging datasets
State: open

### Describe the bug

Hi, I tried to merge in total 22M sequences of data, where each sequence is of maximum length 2000. I found that merging these datasets and then `save_to_disk` is extremely slow because of flattening the indices. Wondering if you have any suggestions or guidance on this. Thank you very much! 

### Steps to reproduce the bug

The source data is too big to demonstrate

### Expected behavior

The source data is too big to demonstrate

### Environment info

python                     3.9.0
datasets                  2.7.0
pytorch                   2.0.0 
tokenizers                0.13.1
transformers              4.31.0
'Repo card metadata block was not found' when loading a pragmeval dataset
State: open

### Describe the bug

I can't load dataset subsets of 'pragmeval'.

The funny thing is I ran the dataset author's [colab notebook](https://colab.research.google.com/drive/1sg--LF4z7XR1wxAOfp0-3d4J6kQ9nj_A?usp=sharing) and it works just fine. I tried to install exactly the same packages that are installed on colab using poetry, so my environment info only differs from the one from colab in linux version - I still get the same bug outside colab.

### Steps to reproduce the bug

Install dependencies with poetry

pyproject.toml
```
[tool.poetry]
name = "project"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "^3.10"
datasets = "2.16.0"
pandas = "1.5.3"
pyarrow = "10.0.1"
huggingface-hub = "0.19.4"
fsspec = "2023.6.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
```

`poetry run python -c "import datasets; print(datasets.get_dataset_config_names('pragmeval'))`

prints ['default']

### Expected behavior

The command should print

```
['emergent',
 'emobank-arousal',
 'emobank-dominance',
 'emobank-valence',
 'gum',
 'mrda',
 'pdtb',
 'persuasiveness-claimtype',
 'persuasiveness-eloquence',
 'persuasiveness-premisetype',
 'persuasiveness-relevance',
 'persuasiveness-specificity',
 'persuasiveness-strength',
 'sarcasm',
 'squinky-formality',
 'squinky-implicature',
 'squinky-informativeness',
 'stac',
 'switchboard',
 'verifiability']
```

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0
Adding support for netCDF (*.nc) files
State: open
Labels: name: enhancement

### Feature request

netCDF (*.nc) is a file format for storing multidimensional scientific data, which is used by packages like `xarray` (labelled multi-dimensional arrays in Python). It would be nice to have native support for netCDF in `datasets`.

### Motivation

When uploading *.nc files onto Huggingface Hub through the `datasets` API, I would like to be able to preview the dataset without converting it to another format.

### Your contribution

I can submit a PR, provided I have the time.
ted_talks_iwslt | Error: Config name is missing
State: closed

### Describe the bug

Running load_dataset using the newest `datasets` library like below on the ted_talks_iwslt using year pair data will throw an error "Config name is missing"

see also:
https://huggingface.co/datasets/ted_talks_iwslt/discussions/3

likely caused by #6493, where the `and not config_kwargs` part in the if logic was removed 
 https://github.com/huggingface/datasets/blob/ef3b5dd3633995c95d77f35fb17f89ff44990bc4/src/datasets/builder.py#L512

### Steps to reproduce the bug

run: 
```python
load_dataset("ted_talks_iwslt", language_pair=("ja", "en"), year="2015")
```

### Expected behavior

Load the data without error

### Environment info

datasets 2.16.0
[Feature request] Indexing datasets by a customly-defined id field to enable random access dataset items via the id
State: open
Labels: name: enhancement

### Feature request

Some datasets may contain an id-like field, for example the `id` field in [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) and the `_id` field in [BeIR/dbpedia-entity](https://huggingface.co/datasets/BeIR/dbpedia-entity). HF datasets support efficient random access via row, but not via this kinds of id fields. I wonder if it is possible to add support for indexing by a custom "id-like" field to enable random access via such ids. The ids may be numbers or strings.

### Motivation

In some cases, especially during inference/evaluation, I may want to find out the item that has a specified id, defined by the dataset itself.

For example, in a typical re-ranking setting in information retrieval, the user may want to re-rank the set of candidate documents of each query. The input is usually presented in a TREC-style run file, with the following format:

```
<qid> Q0 <docno> <rank> <score> <tag>
```

The re-ranking program should be able to fetch the queries and documents according to the `<qid>` and `<docno>`, which are the original id defined in the query/document datasets. To accomplish this, I have to iterate over the whole HF dataset to get the mapping from real ids to row ids every time I start the program, which is time-consuming. Thus I want HF dataset to provide options for users to index by a custom id column, not by row.

### Your contribution

I'm not an expert in this project and I'm afraid that I'm not able to make contributions on the code.
Add polars compatibility
State: closed

Hey there,

I've just finished adding support to convert and format to `polars.DataFrame`. This was in response to the open issue about integrating Polars [#3334](https://github.com/huggingface/datasets/issues/3334). Datasets can be switched to Polars format via `Dataset.set_format("polars")`. I've also included `to_polars` and `from_polars`. All polars functions are checked via config.POLARS_AVAILABLE. 

A few notes:
This only supports `DataFrames` and not `LazyFrames`. This probably could be integrated fairly easily via `is_lazy` args in `set_format`, and `to_polars`.

Let me know your feedbacks.
Impossible to save a mapped dataset to disk
State: open

### Describe the bug

I want to play around with different hyperparameters when training but don't want to re-map my dataset with 3 million samples each time for tens of hours when I [fully fine-tune SDXL](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py).

After I do the mapping like this:
```
train_dataset = train_dataset.map(compute_embeddings_fn, batched=True)
train_dataset = train_dataset.map(
    compute_vae_encodings_fn,
    batched=True,
    batch_size=16,
)
```
and try to save it like this:
`train_dataset.save_to_disk("test")`
i get this error ([full traceback](https://pastebin.com/kq3vt739)):
```
TypeError: Object of type function is not JSON serializable
The format kwargs must be JSON serializable, but key 'transform' isn't.
```

But what is interesting is that pushing to hub works like that:
`train_dataset.push_to_hub("kopyl/mapped-833-icons-sdxl-1024-dataset", token=True)`
Here is the link of the pushed dataset: https://huggingface.co/datasets/kopyl/mapped-833-icons-sdxl-1024-dataset

### Steps to reproduce the bug

Here is the self-contained notebook:

https://colab.research.google.com/drive/1RtCsEMVcwWcMwlWURk_cj_9xUBHz065M?usp=sharing

### Expected behavior

It should be easily saved to disk

### Environment info

NVIDIA A100, Linux (NC24ads A100 v4 from Azure), CUDA 12.2.

[pip freeze](https://pastebin.com/QTNb6iru)