KRadim/github-issues-minimal huggingface dataset:
Abusurdly slow on iteration
State: closed

### Describe the bug

I am attempting to iterate through an image dataset, but I am encountering a significant slowdown in the iteration speed. In order to investigate this issue, I conducted the following experiment:


```python
a=torch.randn(100,224)
a=torch.stack([a] * 10000)
a.shape

# %%
ds=Dataset.from_dict({"tensor":a})
for i in tqdm(ds.with_format("numpy")):
    pass

for i in tqdm(ds.with_format("torch")):
    pass
```
I noticed that the dataset in numpy format performs significantly faster than the one in torch format. My hypothesis is that the dataset undergoes a transformation process of torch->python->numpy(torch) in the background, which might be causing the slowdown. Is there any way to expedite the process by bypassing such transformations?

Furthermore, if I increase the size of a to an image shape, like:
```python
a=torch.randn(3,224,224)
```
the iteration speed becomes absurdly slow, around 100 iterations per second, whereas the speed with numpy format is approximately 250 iterations per second. This level of speed would be unacceptable for large image datasets, as it could take several hours just to iterate through a single epoch.

### Steps to reproduce the bug

 ```python
a=torch.randn(100,224)
a=torch.stack([a] * 10000)
a.shape

# %%
ds=Dataset.from_dict({"tensor":a})
for i in tqdm(ds.with_format("numpy")):
    pass

for i in tqdm(ds.with_format("torch")):
    pass
```

### Expected behavior

iteration faster

### Environment info

 - `datasets` version: 2.11.0
- Platform: Linux-5.4.0-148-generic-x86_64-with-glibc2.10
- Python version: 3.8.16
- Huggingface_hub version: 0.13.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.0
load model error.
State: closed

### Describe the bug

I had trained one model use deepspeed, when I load the final load I get the follow error:
OSError: Can't load tokenizer for '/XXX/DeepSpeedExamples/applications/DeepSpeed-Chat/output/step3-models/1.3b/actor'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '/home/fm001/hzl/Project/DeepSpeedExamples/applications/DeepSpeed-Chat/output/step3-models/1.3b/actor' is the correct path to a directory containing all relevant files for a BloomTokenizerFast tokenizer.


my load code is : python chat.py --path /XXX/DeepSpeedExamples/applications/DeepSpeed-Chat/output/step3-models/1.3b/actor/



### Steps to reproduce the bug

。。。

### Expected behavior

。。。

### Environment info

。。。
Can't add iterable datasets to a Dataset Dict.
State: closed

### System Info

standard env

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Get the following error:

TypeError: Values in `DatasetDict` should be of type `Dataset` but got type '<class 'datasets.iterable_dataset.IterableDataset'>'


### Expected behavior

should be able to add iterable datasets to a dataset dict
Make models/functions optimized with `torch.compile` hashable
State: closed
Labels: name: enhancement

As reported in https://github.com/huggingface/datasets/issues/5819, hashing functions/transforms that reference a model, or a function, optimized with `torch.compile` currently fails due to them not being picklable (the concrete error can be found in the linked issue).

The solutions to consider:
1. hashing/pickling the original, uncompiled version of a compiled model/function (attributes `_orig_mod`/`_torchdynamo_orig_callable`) (less precise than the 2nd option as it ignores the other params of `torch.compute`)
2. wait for https://github.com/pytorch/pytorch/issues/101107 to be resolved
Streaming support for `load_from_disk`
State: closed
Labels: name: enhancement

### Feature request

Support for streaming datasets stored in object stores in `load_from_disk`. 

### Motivation

The `load_from_disk` function supports fetching datasets stored in object stores such as `s3`. In many cases, the datasets that are stored in object stores are very large and being able to stream the data from the buckets becomes essential.

### Your contribution

I'd be happy to contribute this feature if I could get the guidance on how to do so.
Use DeepSpeed load myself " .csv " dataset.
State: open

### Describe the bug

When I use DeepSpeed train a model with my own " XXX.csv" dataset I got the follow question:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/fm001/.conda/envs/hzl/lib/python3.8/site-packages/datasets/load.py", line 1767, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/fm001/.conda/envs/hzl/lib/python3.8/site-packages/datasets/load.py", line 1498, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/fm001/.conda/envs/hzl/lib/python3.8/site-packages/datasets/load.py", line 1217, in dataset_module_factory
    raise FileNotFoundError(
FileNotFoundError: Couldn't find a dataset script at /home/fm001/hzl/Data/qa.csv/qa.csv.py or any data file in the same directory.




### Steps to reproduce the bug

my code is :
from datasets import load_dataset
mydata = load_dataset("/home/fm001/hzl/Data/qa.csv")

### Expected behavior

。。。

### Environment info

。。。
Is uint8 supported?
State: closed

### Describe the bug

I expect the dataset to store the data in the `uint8` data type, but it's returning `int64` instead.
While I've found that `datasets` doesn't yet support float16 (https://github.com/huggingface/datasets/issues/4981), I'm wondering if this is the case for other data types as well.
Is there a way to store vector data as `uint8` and then upload it to the hub?

### Steps to reproduce the bug

```python
from datasets import Features, Dataset, Sequence, Value
import numpy as np

dataset = Dataset.from_dict(
    {"vector": [np.array([0, 1, 2], dtype=np.uint8)]}, features=Features({"vector": Sequence(Value("uint8"))})
).with_format("numpy")

print(dataset[0]["vector"].dtype)
```

### Expected behavior

Expected: `uint8`
Actual: `int64`

### Environment info

- `datasets` version: 2.12.0
- Platform: macOS-12.1-x86_64-i386-64bit
- Python version: 3.8.12
- Huggingface_hub version: 0.12.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
[Bug]504 Server Error when loading dataset which was already cached
State: open

### Describe the bug

I have already cached the dataset using:
```
dataset = load_dataset("databricks/databricks-dolly-15k",
                        cache_dir="/mnt/data/llm/datasets/databricks-dolly-15k")
```
After that, I tried to load it again using the same machine, I got this error:
```
Traceback (most recent call last):
  File "/mnt/home/llm/pythia/train.py", line 16, in <module>
    dataset = load_dataset("databricks/databricks-dolly-15k",
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/datasets/load.py", line 1773, in load_dataset
    builder_instance = load_dataset_builder(
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/datasets/load.py", line 1502, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/datasets/load.py", line 1219, in dataset_module_factory
    raise e1 from None
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/datasets/load.py", line 1186, in dataset_module_factory
    raise e
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/datasets/load.py", line 1160, in dataset_module_factory
    dataset_info = hf_api.dataset_info(
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 120, in _inner_fn
    return fn(*args, **kwargs)
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 1667, in dataset_info
    hf_raise_for_status(r)
  File "/mnt/data/conda/envs/pythia_ft/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py", line 301, in hf_raise_for_status
    raise HfHubHTTPError(str(e), response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError: 504 Server Error: Gateway Time-out for url: https://huggingface.co/api/datasets/databricks/databricks-dolly-15k
```

### Steps to reproduce the bug

1. cache the databrick-dolly-15k dataset using load_dataset, setting a cache_dir
2. use load_dataset again, setting the same cache_dir

### Expected behavior

Dataset loaded succuessfully.

### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-4.18.0-372.16.1.el8_6.x86_64-x86_64-with-glibc2.27
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
(mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))
State: closed

### Describe the bug

M2 MBP can't run
```python
from datasets import load_dataset

jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
```

### Steps to reproduce the bug

1. Use M2 MBP
2. Python 3.10.10 from pyenv
3. Run 
```
from datasets import load_dataset

jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
```

### Expected behavior

Be able to run normally

### Environment info

```
from datasets import load_dataset

jazzy = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision='v1.2-jazzy')
```
OSX: 13.2
CPU: M2
Stream data concatenation issue
State: closed

### Describe the bug

I am not able to concatenate the augmentation of the stream data. I am using the latest version of dataset.

ValueError: The features can't be aligned because the key audio of features {'audio_id': Value(dtype='string', 
id=None), 'audio': {'array': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), 'path': 
Value(dtype='null', id=None), 'sampling_rate': Value(dtype='int64', id=None)}, 'transcript': Value(dtype='string', 
id=None)} has unexpected type - {'array': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), 
'path': Value(dtype='null', id=None), 'sampling_rate': Value(dtype='int64', id=None)} (expected either 
Audio(sampling_rate=16000, mono=True, decode=True, id=None) or Value("null").

### Steps to reproduce the bug

dataset = load_dataset("tobiolatunji/afrispeech-200", "all", streaming=True).shuffle(seed=42)
dataset_cln = dataset.remove_columns(['speaker_id', 'path', 'age_group', 'gender', 'accent', 'domain', 'country', 'duration'])
dataset_cln = dataset_cln.cast_column("audio", Audio(sampling_rate=16000))
from audiomentations import AddGaussianNoise,Compose,Gain,OneOf,PitchShift,PolarityInversion,TimeStretch

augmentation = Compose([
    AddGaussianNoise(min_amplitude=0.005, max_amplitude=0.015, p=0.2)
])

def augment_dataset(batch):
    audio = batch["audio"]
    audio["array"] = augmentation(audio["array"], sample_rate=audio["sampling_rate"])    
    return batch

augmented_dataset_cln = dataset_cln['train'].map(augment_dataset)
dataset_cln['train'] = interleave_datasets([dataset_cln['train'], augmented_dataset_cln])
dataset_cln['train'] = dataset_cln['train'].shuffle(seed=42)

### Expected behavior

I should be able to merge as sampling rate is same.

### Environment info

import datasets
import transformers
import accelerate
print(datasets.__version__)
print(transformers.__version__)
print(torch.__version__)
print(evaluate.__version__)
print(accelerate.__version__)

2.12.0
4.28.1
2.0.0
0.4.0
0.18.0
load json dataset interrupt when dtype cast problem occured
State: open

### Describe the bug

i have a json like this:
[
    {"id": 1, "name": 1},
    {"id": 2, "name": "Nan"},
    {"id": 3, "name": 3}, 
    ....
]
，which have several problematic rows data like row 2, then i load it with datasets.load_dataset('json', data_files=['xx.json'], split='train'), it will report like this:
Generating train split: 0 examples [00:00, ? examples/s]Failed to read file 'C:\Users\gawinjunwu\Downloads\test\data\a.json' with error <class 'pyarrow.lib.ArrowInvalid'>: Could not convert '2' with type str: tried to convert to int64
Traceback (most recent call last):
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1858, in _prepare_split_single
    for _, table in generator:
  File "D:\Python3.9\lib\site-packages\datasets\packaged_modules\json\json.py", line 146, in _generate_tables
    raise ValueError(f"Not able to read records in the JSON file at {file}.") from None
ValueError: Not able to read records in the JSON file at C:\Users\gawinjunwu\Downloads\test\data\a.json.     

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\gawinjunwu\Downloads\test\scripts\a.py", line 4, in <module>
    ds = load_dataset('json', data_dir='data', split='train')
  File "D:\Python3.9\lib\site-packages\datasets\load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 985, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1746, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1891, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset.
Could datasets skip those problematic data row? 

### Steps to reproduce the bug

prepare a json file like this:
[
    {"id": 1, "name": 1},
    {"id": 2, "name": "Nan"},
    {"id": 3, "name": 3}
]
then use datasets.load_dataset('json', dir_files=['xxx.json']) to load the json file

### Expected behavior

skip the problematic data row and load row1 and row3

### Environment info

python3.9
FileNotFound even though exists
State: closed

### Describe the bug

I'm trying to download https://huggingface.co/datasets/bigscience/xP3/resolve/main/ur/xp3_facebook_flores_spa_Latn-urd_Arab_devtest_ab-spa_Latn-urd_Arab.jsonl which works fine in my webbrowser, but somehow not with datasets. Am I doing sth wrong?

```
Downloading builder script: 100%
2.82k/2.82k [00:00<00:00, 64.2kB/s]
Downloading readme: 100%
12.6k/12.6k [00:00<00:00, 585kB/s]
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
[<ipython-input-2-4b45446a91d5>](https://localhost:8080/#) in <cell line: 4>()
      2 lang = "ur"
      3 fname = "xp3_facebook_flores_spa_Latn-urd_Arab_devtest_ab-spa_Latn-urd_Arab.jsonl"
----> 4 dataset = load_dataset("bigscience/xP3", data_files=f"{lang}/{fname}")

6 frames
[/usr/local/lib/python3.10/dist-packages/datasets/data_files.py](https://localhost:8080/#) in _resolve_single_pattern_locally(base_path, pattern, allowed_extensions)
    291         if allowed_extensions is not None:
    292             error_msg += f" with any supported extension {list(allowed_extensions)}"
--> 293         raise FileNotFoundError(error_msg)
    294     return sorted(out)
    295 

FileNotFoundError: Unable to find 'https://huggingface.co/datasets/bigscience/xP3/resolve/main/ur/xp3_facebook_flores_spa_Latn-urd_Arab_devtest_ab-spa_Latn-urd_Arab.jsonl' at /content/https:/huggingface.co/datasets/bigscience/xP3/resolve/main
```

### Steps to reproduce the bug

```
!pip install -q datasets
from datasets import load_dataset
lang = "ur"
fname = "xp3_facebook_flores_spa_Latn-urd_Arab_devtest_ab-spa_Latn-urd_Arab.jsonl"
dataset = load_dataset("bigscience/xP3", data_files=f"{lang}/{fname}")
```

### Expected behavior

Correctly downloads

### Environment info

latest versions
[2.12.0] DatasetDict.save_to_disk not saving to S3
State: closed

### Describe the bug

When trying to save a `DatasetDict` to a private S3 bucket using `save_to_disk`, the artifacts are instead saved locally, and not in the S3 bucket.

I have tried using the deprecated `fs` as well as the `storage_options` arguments and I get the same results.

### Steps to reproduce the bug

1. Create a DatsetDict `dataset`
2. Create a S3FileSystem object
`s3 = datasets.filesystems.S3FileSystem(key=aws_access_key_id, secret=aws_secret_access_key)`
3. Save using `dataset_dict.save_to_disk(f"{s3_bucket}/{s3_dir}/{dataset_name}", storage_options=s3.storage_options)` or `dataset_dict.save_to_disk(f"{s3_bucket}/{s3_dir}/{dataset_name}", fs=s3)`
4. Check the corresponding S3 bucket and verify nothing has been uploaded
5. Check the path at f"{s3_bucket}/{s3_dir}/{dataset_name}" and verify that files have been saved there

### Expected behavior

Artifacts are uploaded at the f"{s3_bucket}/{s3_dir}/{dataset_name}" S3 location.

### Environment info

- `datasets` version: 2.12.0
- Platform: macOS-13.3.1-x86_64-i386-64bit
- Python version: 3.11.2
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1
IterableDataset Arrow formatting
State: closed

Adding an optional `.iter_arrow` to examples iterable. This allows to use Arrow formatting in map/filter.

This will also be useful for torch formatting, since we can reuse the TorchFormatter that converts Arrow data to torch tensors

Related to https://github.com/huggingface/datasets/issues/5793 and https://github.com/huggingface/datasets/issues/3444

Required for https://github.com/huggingface/datasets/pull/5852

### Example: 

Speed x10 in map

```python
from datasets import Dataset
import pyarrow.compute as pc
import time


ds = Dataset.from_dict({"a": range(100_000)})


ids = ds.to_iterable_dataset()
ids = ids.map(lambda x: {"a": [a + 10 for a in x["a"]]}, batched=True)

_start = time.time()
print(f"Python ({sum(1 for _ in ids)} items):\t{(time.time() - _start) * 1000:.1f}ms")
# Python (100000 items):  695.7ms

ids = ds.to_iterable_dataset().with_format("arrow")
ids = ids.map(lambda t: t.set_column(0, "a", pc.add(t[0], 10)), batched=True)
ids = ids.with_format(None)

_start = time.time()
print(f"Arrow ({sum(1 for _ in ids)} items):\t{(time.time() - _start) * 1000:.1f}ms)")
# Arrow (100000 items):   81.0ms)
```


### Implementation details

I added an optional `iter_arrow` method to examples iterable. If an example iterable has this method, then it can be used to iterate on the examples by batch of arrow tables.
Incomplete docstring for `BuilderConfig`
State: closed
Labels: name: good first issue

Hi guys !
I stumbled upon this docstring while working on a project.
Some of the attributes have missing descriptions.

https://github.com/huggingface/datasets/blob/bc5fef5b6d91f009e4101684adcb374df2c170f6/src/datasets/builder.py#L104-L117
Ability to update a dataset
State: open
Labels: name: enhancement

### Feature request

The ability to load a dataset, add or change something, and save it back to disk.

Maybe it's possible, but I can't work out how to do it, e.g. this fails:

```py
import datasets

dataset = datasets.load_from_disk("data/test1")
dataset = dataset.add_item({"text": "A new item"})
dataset.save_to_disk("data/test1")
```

With the error:
```
PermissionError: Tried to overwrite /mnt/c/Users/david/py/learning/mini_projects/data_sorting_and_filtering/data/test1 but a dataset can't overwrite itself.
```


### Motivation

My use case is that I want to process a dataset in a particular way but it doesn't fit in memory if I do it in one go. So I want to perform a loop and at each step in the loop, process one shard and append it to an ever-growing dataset. The code in the loop will load a dataset, add some rows, then save it again.

Maybe I'm just thinking about things incorrectly and there's a better approach. FWIW I can't use `dataset.map()` to do the task because that doesn't work with `num_proc` when adding rows, so is confined to a single process which is too slow.

The only other way I can think of is to create a new file each time, but surely that's not how people do this sort of thing.

### Your contribution

na
Easy way to create a Kaggle dataset from a Huggingface dataset?
State: open

I'm not sure whether this is more appropriately addressed with HuggingFace or Kaggle.  I would like to somehow directly create a Kaggle dataset from a HuggingFace Dataset.

While Kaggle does provide the option to create a dataset from a URI, that URI must point to a single file.  For example:

![image](https://user-images.githubusercontent.com/5355286/235792394-7c559d07-4aff-45b7-ad2b-9c5280c88415.png)


Is there some mechanism from huggingface to represent a dataset (such as that from `load_dataset('wmt14', 'de-en', split='train')` as a single file?  Or, some other way to get that into a Kaggle dataset so that I can use the huggingface `datasets` module to process and consume it inside of a Kaggle notebook?

Thanks in advance!
Cannot shuffle interleaved IterableDataset with "all_exhausted" stopping strategy
State: closed
Labels: name: bug, name: streaming

### Describe the bug

Shuffling interleaved `IterableDataset` with "all_exhausted" strategy yields non-exhaustive sampling.

### Steps to reproduce the bug

```py
from datasets import IterableDataset, interleave_datasets

def gen(bias, length):
  for i in range(length):
    yield dict(a=bias+i)

seed = 42
probabilities = [0.2, 0.6, 0.2]
d1 = IterableDataset.from_generator(lambda: gen(0, 3))
d2 = IterableDataset.from_generator(lambda: gen(10, 4))
d3 = IterableDataset.from_generator(lambda: gen(20, 3))
ds = interleave_datasets([d1, d2, d3], probabilities=probabilities, seed=seed, stopping_strategy='all_exhausted')
ds = ds.shuffle(buffer_size=1000)
for x in ds:
  print(x)
```

This code produces 
```
{'a': 0}
{'a': 22}
{'a': 20}
{'a': 21}
{'a': 10}
{'a': 1}
```

### Expected behavior

It should produce a longer list of examples to exhaust all the datasets.
If you comment out the shuffle line, it will exhaust all the datasets properly.
Here is the output if you comment out shuffling:
```
{'a': 10}
{'a': 11}
{'a': 20}
{'a': 12}
{'a': 0}
{'a': 21}
{'a': 13}
{'a': 10}
{'a': 1}
{'a': 11}
{'a': 12}
{'a': 22}
{'a': 13}
{'a': 20}
{'a': 10}
{'a': 11}
{'a': 12}
{'a': 2}
```

### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

This was run on Google Colab.
load_dataset: TypeError: 'NoneType' object is not callable, on local dataset filename changes
State: open

### Describe the bug

I've adapted Databrick's [train_dolly.py](/databrickslabs/dolly/blob/master/train_dolly.py) to train using a local dataset, which has been working. Upon changing the filenames of the `.json` & `.py` files in my local dataset directory, `dataset = load_dataset(path_or_dataset)["train"]` throws the error:

```python
2023-04-30 09:10:52 INFO [training.trainer] Loading dataset from dushowxa-characters
Traceback (most recent call last):
  File "/data/dushowxa-dolly/train_dushowxa.py", line 26, in <module>
    load_training_dataset()
  File "/data/dushowxa-dolly/training/trainer.py", line 89, in load_training_dataset
    dataset = load_dataset(path_or_dataset)["train"]
  File "/data/dushowxa-dolly/.venv/lib/python3.10/site-packages/datasets/load.py", line 1773, in load_dataset
    builder_instance = load_dataset_builder(
  File "/data/dushowxa-dolly/.venv/lib/python3.10/site-packages/datasets/load.py", line 1528, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
TypeError: 'NoneType' object is not callable
```
The local dataset filenames were of the form `dushowxa-characters/expanse-dushowxa-characters.json` and are now of the form `dushowxa-characters/dushowxa-characters.json` (the word `expanse-` was removed from the filenames). Is this perhaps a dataset caching issue?

I have attempted to manually clear caches, but to no effect:

```sh
rm -rfv ~/.cache/huggingface/datasets/*
rm -rfv ~/.cache/huggingface/modules/* 
```

### Steps to reproduce the bug

Run `python3 train_dushowxa.py` (adapted from Databrick's [train_dolly.py](/databrickslabs/dolly/blob/master/train_dolly.py)).

### Expected behavior

Training succeeds as before local dataset filenames were changed.

### Environment info

Ubuntu 22.04, Python 3.10.6, venv
```python
accelerate>=0.16.0,<1
click>=8.0.4,<9
datasets>=2.10.0,<3
deepspeed>=0.9.0,<1
transformers[torch]>=4.28.1,<5
langchain>=0.0.139
```
Add `fn_kwargs` to `map` and `filter` of `IterableDataset` and `IterableDatasetDict`
State: closed

# Overview
I've added an argument`fn_kwargs` for map and filter methods of `IterableDataset` and `IterableDatasetDict` classes.

# Details
Currently, the map and filter methods of some classes related to `IterableDataset` do not allow specifing the arguments passed to the function. This pull request adds `fn_kwargs` to pass arguments to the mapping function. This allows users to preprocess data more flexibly.

Added `fn_kwargs` to the following classes and methods (description of the argument is also added).
1. class `FilteredExamplesIterable`
2. method `filter` of class `IterableDataset`
3. method `map` of class `IterableDatasetDict`
4. method `filter` of class `IterableDatasetDict`

# Example of changes
Here's an example of how to use the new functionality:
```python
from datasets import IterableDatasetDict

def preprocess_function(example, a=None, b=None):
    # do something
    return example

dataset = IterableDatasetDict(...)
dataset = dataset.map(preprocess_function, fn_kwargs={"a": 1, "b": 2})
```
# Related Issues
This pull request is related to the following issue:
https://github.com/huggingface/datasets/issues/3444 .

# Testing
I have added unit tests to test the new functionality.

In test_iterable_dataset.py  
- Added `test_filtered_examples_iterable_with_fn_kwargs` for [1](#details).
- Added `test_iterable_dataset_filter` for [2](#details).
- Added `test_iterable_dataset_map_with_fn_kwargs`. This is not a newly added feature, but was added because it was not tested.

In test_dataset_dict.py  
- Added `_create_dummy_iterable_dataset` for [3](#details) and [4](#details).
- Added `_create_dummy_iterable_dataset_dict` for [3](#details) and [4](#details).
- Added `test_iterable_map` for [3](#details).
- Added `test_iterable_filter` for [4](#details).

Note that, there is no test for `IterableDatasetDict` at the current main branch. I thought about writing tests for `IterableDatasetDict` in a new file, but I decided to add them in the test file for `DatasetDict` (test_dataset_dict.py).

# Checklist
- [x] Format the code.
- [x] Added tests.
- [x] Passed tests locally.
wiki_dpr details for Open Domain Question Answering tasks
State: closed

Hey guys!

Thanks for creating the wiki_dpr dataset!

I am currently trying to combine wiki_dpr and my own datasets. but I don't know how to make the embedding value the same way as wiki_dpr.

As an experiment, I embeds the text of id="7" of wiki_dpr, but this result was very different from wiki_dpr.
Return the name of the currently loaded file in the load_dataset function.
State: open
Labels: name: enhancement, name: good first issue

### Feature request

Add an optional parameter return_file_name in the load_dataset function. When it is set to True, the function will include the name of the file corresponding to the current line as a feature in the returned output.

### Motivation

When training large language models, machine problems may interrupt the training process. In such cases, it is common to load a previously saved checkpoint to resume training. I would like to be able to obtain the names of the previously trained data shards, so that I can skip these parts of the data during continued training to avoid overfitting and redundant training time.

### Your contribution

I currently use a dataset in jsonl format, so I am primarily interested in the json format. I suggest adding the file name to the returned table here https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/json/json.py#L92.
Improve `Create a dataset` tutorial 
State: open
Labels: name: documentation

Our [tutorial on how to create a dataset](https://huggingface.co/docs/datasets/create_dataset) is a bit misleading. 
1. In **Folder-based builders** section it says that we have two folder-based builders as standard builders, but we also have similar builders (that can be created from directory with data of required format) for `csv`, `json/jsonl`, `parquet` and `txt` files. We have info about these loaders in separate [guide for loading](https://huggingface.co/docs/datasets/loading#local-and-remote-files) but it's worth briefly mentioning them in the beginning tutorial because they are more common and for consistency. Would be helpful to add the link to the full guide.
2. **From local files** section lists methods for creating a dataset from in-memory data which are also described in [loading guide](https://huggingface.co/docs/datasets/loading#inmemory-data).  

Maybe we should actually rethink and restructure this tutorial somehow.
Files downloaded to cache do not respect umask
State: closed
Labels: name: bug

As reported by @stas00, files downloaded to the cache do not respect umask:
```bash
$ ls -l /path/to/cache/datasets/downloads/
-rw------- 1 uername    username    150M Apr 25 16:41 5e646c1d600f065adaeb134e536f6f2f296a6d804bd1f0e1fdcd20ee28c185c6 
```

Related to:
- #2065
Support parallelized downloading and processing in load_dataset with Spark
State: open
Labels: name: enhancement

### Feature request

When calling `load_dataset` for datasets that have multiple files, support using Spark to distribute the downloading and processing job to worker nodes when `cache_dir` is a cloud file system shared among nodes.

```python
load_dataset(..., use_spark=True)
```

### Motivation

Further speed up `dl_manager.download` and `_prepare_split` by distributing the workloads to worker nodes. 

### Your contribution

I can submit a PR to support this.
load_dataset is case sentitive?
State: open

### Describe the bug

load_dataset() function is case sensitive?

### Steps to reproduce the bug

The following two code, get totally different behavior.

1. load_dataset('mbzuai/bactrian-x','en')

2. load_dataset('MBZUAI/Bactrian-X','en')

### Expected behavior

Compare 1 and 2.
1 will download all 52 subsets, shell output:
```Downloading and preparing dataset json/MBZUAI--bactrian-X to xxx```
2 will only download single subset, shell output
```Downloading and preparing dataset bactrian-x/en to xxx```


### Environment info

Python 3.10.11
datasets Version: 2.11.0
CI ZeroDivisionError
State: closed
Labels: name: bug

Sometimes when running our CI on Windows, we get a ZeroDivisionError:
```
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_frugalscore - ZeroDivisionError: float division by zero
```

See for example:
- https://github.com/huggingface/datasets/actions/runs/4809358266/jobs/8560513110
- https://github.com/huggingface/datasets/actions/runs/4798359836/jobs/8536573688

```
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

split = 'test', start_time = 1682516718.8236516, num_samples = 2, num_steps = 1

    def speed_metrics(split, start_time, num_samples=None, num_steps=None):
        """
        Measure and return speed performance metrics.
    
        This function requires a time snapshot `start_time` before the operation to be measured starts and this function
        should be run immediately after the operation to be measured has completed.
    
        Args:
        - split: name to prefix metric (like train, eval, test...)
        - start_time: operation start time
        - num_samples: number of samples processed
        """
        runtime = time.time() - start_time
        result = {f"{split}_runtime": round(runtime, 4)}
        if num_samples is not None:
>           samples_per_second = num_samples / runtime
E           ZeroDivisionError: float division by zero

C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\site-packages\transformers\trainer_utils.py:354: ZeroDivisionError
```
IterableDataset.with_format("torch") not working
State: closed
Labels: name: bug, name: enhancement, name: streaming

### Describe the bug

After calling the with_format("torch") method on an IterableDataset instance, the data format is unchanged.

### Steps to reproduce the bug

```python
from datasets import IterableDataset
def gen():
    for i in range(4):
        yield {"a": [i] * 4}
dataset = IterableDataset.from_generator(gen).with_format("torch")
next(iter(dataset))
```

### Expected behavior

`{"a": torch.tensor([0, 0, 0, 0])}` is expected, but `{"a": [0, 0, 0, 0]}` is observed.

### Environment info

```bash
platform==ubuntu 22.04.01
python==3.10.9
datasets==2.11.0
```
TIFF/TIF support 
State: closed
Labels: name: enhancement

### Feature request

I currently have a dataset (with tiff and json files) where I have to do this:

`wget path_to_data/images.zip && unzip images.zip`

`wget path_to_data/annotations.zip && unzip annotations.zip`
 
 Would it make sense a contribution that supports these type of files? 

### Motivation

instead of using `load_dataset` have to use wget as these files are not supported for annotations with JSON and images with TIFF files. 


Additionally to this, the PIL formatting from datasets does not read correctly the image channels with TIFF format, besides multichannel adaptation might be necessary as well (as my data e.g has more than 3 channels)

### Your contribution

1. Support TIFF images over multi channel format
2. Support JSON annotations
Allow to run CI on push to ci-branch
State: closed

This PR allows to run the CI on push to a branch named "ci-*", without needing to open a PR.
- This will allow to make CI tests without opening a PR, e.g., for future `huggingface-hub` releases, future dependency releases (like `fsspec`, `pandas`,...)

Note that to build the documentation, we already allow it on push to a branch named "doc-builder*".

See:
- #5788

CC: @Wauplin
Support streaming datasets that use jsonlines
State: open
Labels: name: enhancement

Extend support for streaming datasets that use `jsonlines.open`.

Currently, if `jsonlines` is installed, `datasets` raises a `FileNotFoundError`:
```
FileNotFoundError: [Errno 2] No such file or directory: 'https://...'
```

See:
- https://huggingface.co/datasets/masakhane/afriqa/discussions/1
Prepare tests for hfh 0.14
State: closed

Related to the coming release of `huggingface_hub==0.14.0`. It will break some internal tests. The PR fixes these tests. Let's double-check the CI but I expect the fixed tests to be running fine with both `hfh<=0.13.4` and `hfh==0.14`. Worth case scenario, existing PRs will have to be rebased once this fix is merged.

See related [discussion](https://huggingface.slack.com/archives/C02V5EA0A95/p1682337463368609?thread_ts=1681994202.635609&cid=C02V5EA0A95) (private slack).

cc @lhoestq
Multiprocessing in a `filter` or `map` function with a Pytorch model
State: closed

### Describe the bug

I am trying to use a Pytorch model loaded on CPUs with multiple processes with a `.map` or a `.filter` method.

Usually, when dealing with models that are non-pickable, creating a class such that the `map` function is the method `__call__`, and adding `reduce` helps to solve the problem.

However, here, the command hangs without throwing an error.


### Steps to reproduce the bug

```
from datasets import Dataset
import torch
from torch import nn
from torchvision import models
​
​
class FilterFunction:
    #__slots__ = ("path_model", "model")  # Doesn't change anything uncommented
    def __init__(self, path_model):
        self.path_model = path_model
        model = models.resnet50()
        model.fc = nn.Sequential(
            nn.Linear(2048, 512),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(512, 10),
            nn.LogSoftmax(dim=1)
        )
        model.load_state_dict(torch.load(path_model, map_location=torch.device("cpu")))
        model.eval()
        self.model = model
    def __call__(self, batch):
        return [True] * len(batch["id"])
    # Comment this to have an error
    def __reduce__(self):
        return (self.__class__, (self.path_model,))
​
​
dataset = Dataset.from_dict({"id": [0, 1, 2, 4]})
​
# Download (100 MB) at https://github.com/emiliantolo/pytorch_nsfw_model/raw/master/ResNet50_nsfw_model.pth
path_model = "/fsx/hugo/nsfw_image/ResNet50_nsfw_model.pth"
​
filter_function = FilterFunction(path_model=path_model)
​
# Works
filtered_dataset = dataset.filter(filter_function, num_proc=1, batched=True, batch_size=2)
# Doesn't work
filtered_dataset = dataset.filter(filter_function, num_proc=2, batched=True, batch_size=2)
```

### Expected behavior

The command `filtered_dataset = dataset.filter(filter_function, num_proc=2, batched=True, batch_size=2)` should work and not hang.

### Environment info

Datasets: 2.11.0
Pyarrow: 11.0.0
Ubuntu
Offset overflow while doing regex on a text column
State: open

### Describe the bug

`ArrowInvalid: offset overflow while concatenating arrays`

Same error as [here](https://github.com/huggingface/datasets/issues/615)

### Steps to reproduce the bug


Steps to reproduce: (dataset is a few GB big so try in colab maybe)

```
import datasets
import re

ds = datasets.load_dataset('nishanthc/dnd_map_dataset_v0.1', split = 'train')

def get_text_caption(example):
    regex_pattern = r'\s\d+x\d+|,\sLQ|,\sgrid|\.\w+$'
    example['text_caption'] = re.sub(regex_pattern, '', example['picture_text'])
    return example

ds = ds.map(get_text_caption)
```

I am trying to apply a regex to remove certain patterns from a text column. Not sure why this error is showing up.

### Expected behavior

Dataset should have a new column with processed text

### Environment info

Datasets version - 2.11.0
Error using `load_datasets`
State: closed

### Describe the bug

I tried to load a dataset using the `datasets` library in a conda jupyter notebook and got the below error. 

```
ImportError: dlopen(/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-38-darwin.so, 0x0002): Library not loaded: @rpath/liblapack.3.dylib
  Referenced from: <65B094A2-59D7-31AC-A966-4DB9E11D2A15> /Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-38-darwin.so
  Reason: tried: '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/../../../../../../liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/lib/python3.8/site-packages/scipy/sparse/linalg/_isolve/../../../../../../liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/bin/../lib/liblapack.3.dylib' (no such file), '/Users/gilbertyoung/miniforge3/envs/review_sense/bin/../lib/liblapack.3.dylib' (no such file), '/usr/local/lib/liblapack.3.dylib' (no such file), '/usr/lib/liblapack.3.dylib' (no such file, not in dyld cache)
```

### Steps to reproduce the bug

Run the `load_datasets` function

### Expected behavior

I expected the dataset to be loaded into my notebook.

### Environment info

name: review_sense
channels:
  - apple
  - conda-forge
dependencies:
  - python=3.8
  - pip>=19.0
  - jupyter
  - tensorflow-deps
  #- scikit-learn
  #- scipy
  - pandas
  - pandas-datareader
  - matplotlib
  - pillow
  - tqdm
  - requests
  - h5py
  - pyyaml
  - flask
  - boto3
  - ipykernel
  - seaborn
  - pip:
      - tensorflow-macos==2.9
      - tensorflow-metal==0.5.0
      - bayesian-optimization
      - gym
      - kaggle
      - huggingface_hub
      - datasets
      - numpy
      - huggingface
TypeError: 'NoneType' object does not support item assignment
State: closed

command：
```
def load_datasets(formats, data_dir=datadir, data_files=datafile）：
      dataset = load_dataset(formats, data_dir=datadir, data_files=datafile, split=split, streaming=True, **kwargs)
return dataset
raw_datasets = DatasetDict()
raw_datasets["train"] = load_datasets(“csv”,  args.datadir, "train.csv", split=train_split)
raw_datasets["test"] = load_datasets(“csv”, args.datadir, "dev.csv", split=test_split)
raw_datasets = raw_datasets.cast_column("audio", Audio(sampling_rate=16000))
```

error：
```
    main()
  File "peft_adalora_whisper_large_training.py", line 502, in main
    raw_datasets = raw_datasets.cast_column("audio", Audio(sampling_rate=16000))
  File "/home/ybZhang/miniconda3/envs/whister/lib/python3.8/site-packages/datasets/dataset_dict.py", line 2015, in cast_column
    info.features[column] = feature
TypeError: 'NoneType' object does not support item assignment
```
Schrödinger's dataset_dict
State: closed

### Describe the bug

If you use load_dataset('json', data_files="path/test.json"), it will return DatasetDict({train:...}).

And if you use load_dataset("path"), it will return DatasetDict({test:...}).

Why can't the output behavior be unified?

### Steps to reproduce the bug

as description above.

### Expected behavior

consistent predictable output.

### Environment info

'2.11.0'
datasets.load_dataset("code_search_net", "python") : NotADirectoryError: [Errno 20] Not a directory
State: closed

### Describe the bug

While checking out the [tokenizer tutorial](https://huggingface.co/course/chapter6/2?fw=pt), i noticed getting an error while initially downloading the python dataset used in the examples.

The [collab with the error is here](https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/en/chapter6/section2.ipynb#scrollTo=hGb69Yo3eV8S)

```
from datasets import load_dataset
import os

os.environ["HF_DATASETS_CACHE"] = "/workspace"

# This can take a few minutes to load, so grab a coffee or tea while you wait!
raw_datasets = load_dataset("code_search_net", "python")
```

yeilds:

```
ile /opt/conda/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py:524, in xlistdir(path, use_auth_token)
    522 main_hop, *rest_hops = _as_str(path).split("::")
    523 if is_local_path(main_hop):
--> 524     return os.listdir(path)
    525 else:
    526     # globbing inside a zip in a private repo requires authentication
    527     if not rest_hops and (main_hop.startswith("http://") or main_hop.startswith("https://")):

NotADirectoryError: [Errno 20] Not a directory: '/workspace/downloads/25ceeb4c25ab737d688bd56ea92bfbb1f199fe572470456cf2d675479f342ac7/python/final/jsonl/train'
```

I was able to reproduce this erro both in the collab and on my own pytorch/pytorch container pulled from the dockerhub official pytorch image, so i think it may be a server side thing. 

### Steps to reproduce the bug

Steps to reproduce the issue:

1. run `raw_datasets = load_dataset("code_search_net", "python")`

### Expected behavior

expect the code to not exception during dataset pull.

### Environment info

i tried both the default HF_DATASETS_CACHE on Collab, and on my local container.  i then pointed to the HF_DATASETS_CACHE to a  large capacity local storage and the problem was consisten across all 3 scenarios.
Use Pandas' `read_json` in the JSON builder
State: open
Labels: name: enhancement

Instead of PyArrow's `read_json`, we should use `pd.read_json` in the JSON builder for consistency with the CSV and SQL builders (e.g., to address https://github.com/huggingface/datasets/issues/5725).

In Pandas2.0, to get the same performance, we can set the `engine` to "pyarrow". The issue is that Colab still doesn't install Pandas 2.0 by default, so I think it's best to wait for this to be resolved on their side to avoid downgrading decoding performance in scenarios when Pandas 2.0 is not installed.
ArrowDataset.save_to_disk lost some logic of remote
State: closed

### Describe the bug

https://github.com/huggingface/datasets/blob/e7ce0ac60c7efc10886471932854903a7c19f172/src/datasets/arrow_dataset.py#L1371

Here is the bug point, when I want to save from a `DatasetDict` class and the items of the instance is like `[('train', Dataset({features: ..., num_rows: ...}))]` , there is no guarantee that there exists a directory name `train` under `dataset_dict_path`.

### Steps to reproduce the bug

1. Mock a DatasetDict with items like what I said.
2. using save_to_disk with storage_options, u can use local sftp. code may like below

```python
from datasets import load_dataset
dataset = load_dataset(...)
dataset.save_to_disk('sftp:///tmp', storage_options={'host': 'localhost', 'username': 'admin'})
```
I suppose u can reproduce the bug by these steps. 

### Expected behavior

Should create the folder if it does not exists, just like we do locally.

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-6.2.10-arch1-1-x86_64-with-glibc2.35
- Python version: 3.10.9
- Huggingface_hub version: 0.13.2
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
Fix JSON builder when missing keys in first row 
State: closed

Until now, the JSON builder only considered the keys present in the first element of the list:
- Either explicitly: by passing index 0 in `dataset[0].keys()`
- Or implicitly: `pa.Table.from_pylist(dataset)`, where "schema (default None): If not passed, will be inferred from the first row of the mapping values"

This PR fixes the bug by considering the union of the keys present in all the rows.


Fix #5726.
Tiktoken tokenizers are not pickable 
State: closed

### Describe the bug

Since tiktoken tokenizer is not pickable, it is not possible to use it inside `dataset.map()` with multiprocessing enabled. However, you [made](https://github.com/huggingface/datasets/issues/5536) tiktoken's tokenizers pickable in `datasets==2.10.0` for caching. For some reason, this logic does not work in dataset processing and raises `TypeError: cannot pickle 'builtins.CoreBPE' object`


### Steps to reproduce the bug

```
from datasets import load_dataset
import tiktoken

dataset = load_dataset("stas/openwebtext-10k")

enc = tiktoken.get_encoding("gpt2")

tokenized = dataset.map(
    process,
    remove_columns=['text'],
    desc="tokenizing the OWT splits",
    num_proc=2,
)

def process(example):
        ids = enc.encode(example['text'])
        ids.append(enc.eot_token)
        out = {'ids': ids, 'len': len(ids)}
        return out
```

### Expected behavior

starts processing dataset 

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.0-1021-oracle-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.13.4
- PyArrow version: 9.0.0
- Pandas version: 2.0.0