konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
HfHubHTTPError - exceeded our hourly quotas for action: commit

### Describe the bug

I try to upload a very large dataset of images, and get the following error:
```
File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/hf_api.py:2712, in HfApi.create_commit(self, repo_id, operations, commit_message, commit_description, token, repo_type, revision, create_pr, num_threads, parent_commit, run_as_future)
   2710 try:
   2711     commit_resp = get_session().post(url=commit_url, headers=headers, data=data, params=params)
-> 2712     hf_raise_for_status(commit_resp, endpoint_name="commit")
   2713 except RepositoryNotFoundError as e:
   2714     e.append_to_message(_CREATE_COMMIT_NO_REPO_ERROR_MESSAGE)

File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py:301, in hf_raise_for_status(response, endpoint_name)
    297     raise BadRequestError(message, response=response) from e
    299 # Convert `HTTPError` into a `HfHubHTTPError` to display request information
    300 # as well (request id and/or server error message)
--> 301 raise HfHubHTTPError(str(e), response=response) from e

HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/yuvalkirstain/pickapic_v2/commit/main (Request ID: Root=1-65112399-12d63f7d7f28bfa40a36a0fd)

You have exceeded our hourly quotas for action: commit. We invite you to retry later.
```

this makes it much less convenient to host large datasets on HF hub.

### Steps to reproduce the bug

Upload a very large dataset of images

### Expected behavior

the upload to work well

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.0-1033-aws-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 1.5.3

how is your dataset structured? (file types, how many commits and files are you trying to push, etc)

I succeeded in uploading it after several attempts with an hour gap between each attempt (inconvenient but worked). The final dataset is [here](https://huggingface.co/datasets/yuvalkirstain/pickapic_v2), code and context to the dataset can be found [here](https://github.com/yuvalkirstain/PickScore/).
I can close the issue if this behavior is intended, as most users probably do not need to upload large-scale datasets.

We could fix this by creating a single commit for all the (Parquet) shards in `push_to_hub` instead of one commit per shard, as we currently do. 

@Wauplin Any updates on the 2-step commit process suggested by you that we need to implement this?

> Any updates on the 2-step commit process suggested by you that we need to implement this?

Re-prioritizing this, sorry. Will let you know but probably can be done this week.
load_dataset() function's cache_dir does not seems to work

### Describe the bug

datasets version: 2.14.5

when trying to run the following command
trec = load_dataset('trec', split='train[:1000]', cache_dir='/path/to/my/dir')

I keep getting error saying the command does not have permission to the default cache directory on my macbook pro machine. 

It seems the cache_dir parameter cannot change the dataset saving directory from the default 

what ever explained in the https://huggingface.co/docs/datasets/cache does not seem to work


### Steps to reproduce the bug

datasets version: 2.14.5

when trying to run the following command
trec = load_dataset('trec', split='train[:1000]', cache_dir='/path/to/my/dir')

I keep getting error saying the command does not have permission to the default cache directory on my macbook pro machine. 

It seems the cache_dir parameter cannot change the dataset saving directory from the default 

what ever explained in the https://huggingface.co/docs/datasets/cache does not seem to work

### Expected behavior

the dataset should be saved to the cache_dir points to 

### Environment info

datasets version: 2.14.5
macos X: Ventura 13.4.1 (c)

Can you share the error message?

Also, it would help if you could check whether `huggingface_hub`'s download behaves the same:
```python
from huggingface_hub import snapshot_download
snapshot_download("trec", repo_type="dataset", cache_dir='/path/to/my/dir)
```

In the next major release, we aim to switch to `huggingface_hub` for file download/caching, but we could align the `cache_dir`'s `umask` behavior earlier than this if their solution works for your use case.

@mariosasko, Hey , is there any update on this? 

Yes, I am still facing the same issue.

```python
from huggingface_hub import snapshot_download
snapshot_download("trec", repo_type="dataset", cache_dir='/path/to/my/dir)
```

But this is working for me.

Thanks.

Could you clarify a bit? What if I wanted only 100 samples from train split, how would i change the above? Thanks

I have exactly same issue

@lhoestq Can this be open again? Reading #7200 this doesn't even address not being able to specify `cache_dir`, it just updates the documentation? Their own comment states this:

> In my testing, I could not specify the cache directory even by load_dataset("dataset_name" cache_dir="..."). It might be another issue. I also welcome any advice to solve this issue.

Nothing else in the PR seems to indicate that this is actually resolved.

I've noticed the same issue. Looks like #7499 was fixing it, but from what I see in the comments this setting is for different caches?

Hi ! you can find the information about the `datasets` and `huggingface_hub` cache directories here: https://huggingface.co/docs/datasets/v3.6.0/en/cache#cache-directory
Dataset.from_generator() cost much more time in vscode debugging mode then running mode

### Describe the bug

Hey there,
I’m using Dataset.from_generator() to convert a torch_dataset to the Huggingface Dataset.
However, when I debug my code on vscode, I find that it runs really slow on Dataset.from_generator() which may even 20 times longer then run the script on terminal.

### Steps to reproduce the bug

I write a simple test code :

```python
import os
from functools import partial
from typing import Callable

import torch
import time
from torch.utils.data import Dataset as TorchDataset

from datasets import load_from_disk, Dataset as HFDataset
  
import torch  
from torch.utils.data import Dataset  
  
class SimpleDataset(Dataset):  
    def __init__(self, data):  
        self.data = data  
        self.keys = list(data[0].keys())
      
    def __len__(self):  
        return len(self.data)  
      
    def __getitem__(self, index):  
        sample = self.data[index]  
        return {key: sample[key] for key in self.keys}  
  

def TorchDataset2HuggingfaceDataset(torch_dataset: TorchDataset, cache_dir: str = None
) -> HFDataset:
    
    """
        convert torch dataset to huggingface dataset
    """
    generator : Callable[[], TorchDataset] = lambda: (sample for sample in torch_dataset)   

    return HFDataset.from_generator(generator, cache_dir=cache_dir)

if __name__ == '__main__':
    data = [  
        {'id': 1, 'name': 'Alice'},  
        {'id': 2, 'name': 'Bob'},  
        {'id': 3, 'name': 'Charlie'}  
    ]
    
    torch_dataset = SimpleDataset(data)
    start_time = time.time() 
    huggingface_dataset = TorchDataset2HuggingfaceDataset(torch_dataset)
    end_time = time.time()
    print("time: ", end_time - start_time)
    print(huggingface_dataset)
```

### Expected behavior

this test on my machine report that the running time on terminal is 0.086,
however the running time in debugging mode on vscode is 0.25, which I think is much longer than expected.

I’d like to know is the anything wrong in the code or just because of debugging?
I have traced the code and I find is this func which I get stuck.

```python
def create_config_id(
        self,
        config_kwargs: dict,
        custom_features: Optional[Features] = None,
    ) -> str:
...
# stuck in this line
suffix = Hasher.hash(config_kwargs_to_add_to_suffix)
```



### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.11.0-27-generic-x86_64-with-glibc2.31
- Python version: 3.11.3
- Huggingface_hub version: 0.17.2
- PyArrow version: 11.0.0
- Pandas version: 2.0.1

Answered on the forum: https://discuss.huggingface.co/t/dataset-from-generator-cost-much-more-time-in-vscode-debugging-mode-then-running-mode/56005/2
exif_transpose not done to Image (PIL problem)

### Feature request

I noticed that some of my images loaded using PIL have some metadata related to exif that can rotate them when loading.
Since the dataset.features.Image uses PIL for loading, the loaded image may be rotated (width and height will be inverted) thus for tasks as object detection and layoutLM this can create some inconsistencies (between input bboxes and input images). 

For now there is no option in datasets.features.Image to specify that. We need to do the following when preparing examples (when preparing images for training, test or inference): 
```
from PIL import Image, ImageOps 
pil = ImageOps.exif_transpose(pil)
```

reference: https://stackoverflow.com/a/63950647/5720150 

Is it possible to add this by default to the datasets.feature.Image ? or to add the option to do the ImageOps.exif_transpose?

Thank you

### Motivation

Prevent having inverted data related to exif metadata that may affect object detection tasks

### Your contribution

Changing in datasets.featrues.Image I can help with that. 

Indeed, it makes sense to do this by default. 

In the meantime, you can use `.with_transform` to transpose the images when accessing them:

```python
import PIL.ImageOps

def exif_transpose_transform(batch):
  batch["image"] = [PIL.ImageOps.exif_transpose(image) for image in batch["image"]]
  return batch

dataset = dataset.with_transform(exif_transpose_transform)
```

This operation sets some `Image` attributes to `None` (`.format`, `.filename`, etc.), causing our tests to fail, so I think we should wait for Datasets 3.0 to make this change. In version 3.0, storing image paths will be replaced by embedding image bytes, so there will be fewer instances where we use the `.filename` attribute.
Data alteration when loading dataset with unspecified inner sequence length

### Describe the bug

When a dataset saved with a specified inner sequence length is loaded without specifying that length, the original data is altered and becomes inconsistent.

### Steps to reproduce the bug

```python
from datasets import Dataset, Features, Value, Sequence, load_dataset

# Repository ID
repo_id = "my_repo_id"

# Define features with a specific length of 3 for each inner sequence
specified_features = Features({"key": Sequence(Sequence(Value("float32"), length=3))})

# Create a dataset with the specified features
data = [
    [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]],
    [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]],
]
dataset = Dataset.from_dict({"key": data}, features=specified_features)

# Push the dataset to the hub
dataset.push_to_hub(repo_id)

# Define features without specifying the length
unspecified_features = Features({"key": Sequence(Sequence(Value("float32")))})

# Load the dataset from the hub with this new feature definition
dataset = load_dataset(f"qgallouedec/{repo_id}", split="train", features=unspecified_features)

# The obtained data is altered
print(dataset.to_dict())  # {'key': [[[1.0], [2.0]], [[3.0], [4.0]]]}
```


### Expected behavior

```python
print(dataset.to_dict())  # {'key': [[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]], [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]]]}
```

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-6.2.0-32-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3

While this issue may seem specific, it led to a silent problem in my workflow that took days to diagnose. If this feature is not intended to be supported, an error should be raised when encountering this configuration to prevent such issues.

Thanks for reporting! This is a MRE:

```python
import pyarrow as pa
from datasets.table import cast_array_to_feature
from datasets import Sequence, Value
data = [
    [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]],
    [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]],
]
arr = pa.array(data, pa.list_(pa.list_(pa.float32(), 3)))
cast_array_to_feature(arr, Sequence(Sequence(Value("float32"))))
```

I've opened a PR with a fix.
Dataloader stuck on multiple GPUs

### Describe the bug

I am trying to get CLIP to fine-tuning with my code.
When I tried to run it on multiple GPUs using accelerate, I encountered the following phenomenon.

- Validation dataloader stuck in 2nd epoch only on multi-GPU
Specifically, when the "for inputs in valid_loader:" process is finished, it does not proceed to the next step. train_loader process is completed. Also, both train and valid are working correctly in the first epoch.
The accelerate command at that time is as follows.
`accelerate launch --multi_gpu --num_processes=2 {script_name.py} {--arg1} {--arg2} ...`

- This will not happen when single GPU is used.
`CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...`
- Setting num_workers=0 in dataloader did not change the result.

### Steps to reproduce the bug

1. The codes for fine-tuning the regular CLIP were updated for accelerate.
2. Run the code with the accelerate command as `accelerate launch --multi_gpu --num_processes=2 {script_name.py} {--arg1} {--arg2} ...` and the above problem will occur.
3. CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...` , it works fine.

### Expected behavior

It Should end normally as if it was run on a single GPU.

### Environment info

Since `datasets-cli env` did not work, the environment is described below.

- OS: Ubuntu 22.04 with Docker
- Docker: 24.0.5, build ced0996
- Python: 3.10.12
- torch==2.0.1
- accelerate==0.21.0
- transformers==4.33.1

What type of dataset are you using in this script? `torch.utils.data.Dataset` or `datasets.Dataset`? Please share the `datasets` package version if it's the latter. Otherwise, it's better to move this issue to the `accelerate` repo.

Very sorry, I thought I had a repo in `accelerate!`
I will close this issue and repo the issue in the appropriate place.
`dataset.filter` ALWAYS removes the first item from the dataset when using batched=True

### Describe the bug

If you call batched=True when calling `filter`, the first item is _always_ filtered out, regardless of the filter condition.



### Steps to reproduce the bug

Here's a minimal example:
```python
def filter_batch_always_true(batch, indices):
    print("First index being passed into this filter function: ", indices[0])
    return indices  # Keep all indices

data = {"value": list(range(10))}
dataset = Dataset.from_dict(data)

filtered_dataset = dataset.filter(filter_batch_always_true, with_indices=True, batched=True)
print("Length of original dataset: ", len(dataset))
print("Length of filtered_dataset: ", len(filtered_dataset))
print("Is equal to original? ", len(filtered_dataset) == len(dataset))
print("First item of filtered dataset: ", filtered_dataset[0])
print("Last item of filtered dataset: ", filtered_dataset[-1])
```
prints:
```
First index being passed into this filter function:  0
Length of original dataset:  10
Length of filtered_dataset:  9
Is equal to original?  False
First item of filtered dataset:  {'value': 1}
Last item of filtered dataset:  {'value': 9}
```

### Expected behavior

Filter should respect the filter condition.

### Environment info


- `datasets` version: 2.14.4
- Platform: macOS-13.5-arm64-arm-64bit
- Python version: 3.9.18
- Huggingface_hub version: 0.17.1
- PyArrow version: 10.0.1
- Pandas version: 2.0.2


`filter` treats the function's output as a (selection) mask - `True` keeps the sample, and `False` drops it. In your case, `bool(0)` evaluates to `False`, so dropping the first sample is the correct behavior.

Oh gosh! 🤦  I totally misunderstood the API! My apologies!
Tokenization with multiple workers is too slow

I am trying to tokenize a few million documents with multiple workers but the tokenization process is taking forever. 

Code snippet:

```
raw_datasets.map(
            encode_function,
            batched=False,
            num_proc=args.preprocessing_num_workers,
            load_from_cache_file=not args.overwrite_cache,
            remove_columns=[name for name in raw_datasets["train"].column_names if name not in ["input_ids", "labels", "attention_mask"]],
            desc="Tokenizing data",
        )
```

Details:

```
transformers==4.28.0.dev0
datasets==4.28.0.dev0
preprocessing_num_workers==48
```
tokenizer == decapoda-research/llama-7b-hf




[This](https://huggingface.co/docs/datasets/nlp_process#map) is the most performant way to tokenize a dataset (`batched=True, num_proc=None, return_tensors="np"`) 

If`tokenizer.is_fast` returns `True`, `num_proc` must be `None/1` to benefit from the fast tokenizers' parallelism (the fast tokenizers are implemented in Rust, and Rust multi-threading doesn't work well with Python multi-processing)
Support buffer shuffle for to_tf_dataset

### Feature request

I'm using to_tf_dataset to convert a large dataset to tf.data.Dataset and use Keras fit to train model. 
Currently, to_tf_dataset only supports full size shuffle, which can be very slow on large dataset. 
tf.data.Dataset support buffer shuffle by default. 
shuffle(
    buffer_size, seed=None, reshuffle_each_iteration=None, name=None
)

### Motivation

I'm very frustrated to find the loading with shuffling large dataset is very slow. It seems impossible to shuffle before training Keras with big dataset. 

### Your contribution

NA

cc @Rocketknight1 

Hey! You can implement this yourself, just:

1) Create the dataset with `to_tf_dataset()` with `shuffle=False`
2) Add an `unbatch()` at the end (or use batch_size=1)
3) Add a `shuffle()` to the resulting dataset with your desired buffer size
4) Add a `batch()` at the end again to re-batch your dataset.

Note that the way we construct datasets in `to_tf_dataset()`, we don't actually shuffle the entire dataset in-memory, using `tf.data.Dataset.shuffle()`! Instead, we shuffle an index array and then load from the dataset with that. This means that shuffling with `tf.data.Dataset.shuffle()` will probably be slower and use more memory than our approach - I don't think adding the option for smaller shuffle buffers will actually save you memory on this!

Thanks for your reply! @Rocketknight1 
"We don't actually shuffle the entire dataset in-memory, using tf.data.Dataset.shuffle()! Instead, we shuffle an index array and then load from the dataset with that."
In such case, there will be random access to dataset data during shuffling. When the dataset is large, the performance can be X10 times slow. I have tried many ways with to_tf_dataset() trying to achieve comparable performance with tf.data.Dataset().shuffle(buffer_size).batch(). But the performance with to_tf_dataset() is still slow.
Support multiprocessing for download/extract nestedly

### Feature request

Current multiprocessing for download/extract is not done nestedly. For example, when processing SlimPajama, there is only 3 processes (for train/test/val), while there are many files inside these 3 folders

```
Downloading data files #0:   0%|          | 0/1 [00:00<?, ?obj/s]
Downloading data files #1:   0%|          | 0/1 [00:00<?, ?obj/s]
Downloading data files #2:   0%|          | 0/1 [00:00<?, ?obj/s]

Extracting data files #0:   0%|          | 0/1 [00:00<?, ?obj/s]
Extracting data files #1:   0%|          | 0/1 [00:00<?, ?obj/s][A
Extracting data files #2:   0%|          | 0/1 [00:00<?, ?obj/s][A[A
```

### Motivation

speedup dataset loading

### Your contribution

I can help test the feature
Overwrite legacy default config name in `dataset_infos.json` in packaged datasets

Currently if we push data as default config with `.push_to_hub` to a repo that has a legacy `dataset_infos.json` file containing a legacy default config name like `{username}--{dataset_name}`, new key `"default"` is added to `dataset_infos.json` along with the legacy one. I think the legacy one should be dropped in this case.

Also, in `load.py` I suggest to check if a legacy config name is indeed a legacy config name because after this fix it might not be the case (this check was first introduced in https://github.com/huggingface/datasets/pull/6218)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6231). All of your documentation changes will be reflected on that endpoint.

realized that this pr is still not merged, @lhoestq maybe you can take a look at it? 

I think https://github.com/huggingface/datasets/pull/6218 fixed the issue (a bit differently though)

ah actually nope, let me check

@lhoestq yeah the pr you're referencing doesn't fix the problem when two semantically analogous configs occur in datasets_info.json, i suggest to rewrite the legacy one if it exists during .push_to_hub

Only the old versions of `datasets` use the JSON file over the README and they can only load one config so the name doesn't really matter.

That's why I chose to load the info from the JSON no matter the name (no check to see if it's "username--dataset_name") in my previous PR.

I think you can remove the old info without even checking the name. In this case maybe no need to update load.py 

(also minor: not checking the name makes it more robust to dataset renaming)

@lhoestq okay makes sense... so you think it's not a problem that in some cases we might end up with `dataset_infos.json` having two keys in it?

> @lhoestq okay makes sense... so you think it's not a problem that in some cases we might end up with dataset_infos.json having two keys in it?

Ideally they should have only one config no ? Since old versions of `datasets` simply load the first config in the JSON.
We can overwrite it with the new default one (and no matter the name of the outdated config in the JSON)
Conversion from RGB to BGR in Object Detection tutorial

The [tutorial](https://huggingface.co/docs/datasets/main/en/object_detection) mentions the necessity of conversion the input image from BGR to RGB

> albumentations expects the image to be in BGR format, not RGB, so you’ll have to convert the image before applying the transform.

[Link to tutorial](https://github.com/huggingface/datasets/blob/0a068dbf3b446417ffd89d32857608394ec699e6/docs/source/object_detection.mdx#L77)

However, relevant albumentations' tutorials [on channels conversion](https://albumentations.ai/docs/examples/example/#read-the-image-from-the-disk-and-convert-it-from-the-bgr-color-space-to-the-rgb-color-space) and  [on boxes](https://albumentations.ai/docs/examples/example_bboxes/) imply that it's not really true no more.

I suggest removing this outdated conversion from the tutorial.

Good catch!
Support saving datasets with custom formatting

Requested in https://discuss.huggingface.co/t/using-set-transform-on-a-dataset-leads-to-an-exception/53036.

I am not sure if supporting this is the best idea for the following reasons:

>For this to work, we would have to pickle a custom transform, which means the transform and the objects it references need to be serializable. Also, deserializing these bytes would make `load_from_disk` unsafe, so I'm not sure this is a good idea. 

@lhoestq WDYT?


Not a fan of pickling this sort of stuff either.
Note that users can also share the code in their dataset documentation.
`Dataset.to_dict()` ignore `decode=True` with Image feature

### Describe the bug

`Dataset.to_dict` seems to ignore the decoding instruction passed in features.

### Steps to reproduce the bug

```python
import datasets
import numpy as np
from PIL import Image

img = np.random.randint(0, 256, (5, 5, 3), dtype=np.uint8)
img = Image.fromarray(img)

features = datasets.Features({"image": datasets.Image(decode=True)})
dataset = datasets.Dataset.from_dict({"image": [img]}, features=features)
print({key: dataset[key] for key in dataset.column_names})
# {'image': [<PIL.PngImagePlugin.PngImageFile image mode=RGB size=5x5 at 0x7EFBC80E15B0>]}
print(dataset.to_dict())
# {'image': [{'bytes': b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x05\x00\x00\x00\x05\x08\x02\x00\x00\x00\x02\r\xb1\xb2\x00\x00\x00[IDATx\x9c\x01P\x00\xaf\xff\x01\x13\x1b<7\xe7\xe0\xdc^6\xed\x04\xc7M\xd2\x9f\x00X\x1b\xb0?\x1ba\x15\xc5 o\xd0\x80\xbe\x19/\x01\xec\x95\x1f\x9f\xffj\xfa1\xa7\xc4X\xea\xbe\xa4g\x00\xc4\x15\xdeC\xc7 \xbbaqe\xc8\xb9\xa9q\xe7\x00,?M\xc0)\xdaD`}\xb1\xdci\x1e\xafC\xa9]%.@\xa6\xf0\xb3\x00\x00\x00\x00IEND\xaeB`\x82', 'path': None}]}
```




### Expected behavior

I would expect `{key: dataset[key] for key in dataset.column_names}` and `dataset.to_dict()` to be equivalent. If the previous behavior is expected, then it should be stated [in the doc](https://huggingface.co/docs/datasets/v2.14.4/en/package_reference/main_classes#datasets.Dataset.to_dict).

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-6.2.0-31-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- Pillow 9.5.0
- numpy 1.25.2

We need to implement the `Image` type as a PyArrow extension type (to allow us to override the Python conversion) for this to work as expected. For now, it's best to use your approach indeed.
No-script datasets with ZIP files do not load

While investigating an issue on a Hub dataset, I have discovered the no-script datasets containing ZIP files do not load.

For example, that no-script dataset containing ZIP files, raises NonMatchingSplitsSizesError:
```python
In [2]: ds = load_dataset("sidovic/LearningQ-qg")

NonMatchingSplitsSizesError: [
  {
    'expected': SplitInfo(name='train', num_bytes=0, num_examples=188660, shard_lengths=None, dataset_name=None), 
    'recorded': SplitInfo(name='train', num_bytes=0, num_examples=0, shard_lengths=None, dataset_name='learning_q-qg')
  }, {
    'expected': SplitInfo(name='validation', num_bytes=0, num_examples=20630, shard_lengths=None, dataset_name=None), 
    'recorded': SplitInfo(name='validation', num_bytes=0, num_examples=0, shard_lengths=None, dataset_name='learning_q-qg')
  }, {
    'expected': SplitInfo(name='test', num_bytes=0, num_examples=18227, shard_lengths=None, dataset_name=None), 
    'recorded': SplitInfo(name='test', num_bytes=0, num_examples=0, shard_lengths=None, dataset_name='learning_q-qg')
  }
]
```

As another example, a no-script dataset containing just a (CSV)-ZIP file, raises a DatasetGenerationError:
```
>               num_examples, num_bytes = writer.finalize()

src/datasets/builder.py:1949: 

>           raise SchemaInferenceError("Please pass `features` or at least one example when writing data")
E           datasets.arrow_writer.SchemaInferenceError: Please pass `features` or at least one example when writing data

src/datasets/arrow_writer.py:598: SchemaInferenceError

The above exception was the direct cause of the following exception:

src/datasets/load.py:2143: in load_dataset
    builder_instance.download_and_prepare(
src/datasets/builder.py:954: in download_and_prepare
    self._download_and_prepare(
src/datasets/builder.py:1049: in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
src/datasets/builder.py:1813: in _prepare_split
    for job_id, done, content in self._prepare_split_single(

>           raise DatasetGenerationError("An error occurred while generating the dataset") from e
E           datasets.builder.DatasetGenerationError: An error occurred while generating the dataset

src/datasets/builder.py:1958: DatasetGenerationError
```

After investigating, I think this bug was introduced in this PR:
- #5972

Related to:
- https://huggingface.co/datasets/sidovic/LearningQ-qg/discussions/1

CC: @lhoestq
avoid downgrading jax version

### Feature request

Whenever I `pip install datasets[jax]` it downgrades jax to version 0.3.25. I seem to be able to install this library first then upgrade jax back to version 0.4.13.

### Motivation

It would be nice to not overwrite currently installed version of jax if possible.

### Your contribution

I would be willing to beta test. Or maybe write some code if I could get pointed in the right direction, I'm not super familiar with this codebase.

https://github.com/huggingface/datasets/blob/main/setup.py#L236
Currently has the highest version at 0.3.25; Not sure if there is any reason for this, other than that was the tested version?
Use load_dataset for local json files, but it not works

### Describe the bug

when I use load_dataset to load my local datasets，it always goes to Hugging Face to download the data instead of loading the local dataset.

### Steps to reproduce the bug

`raw_datasets = load_dataset(
        ‘json’,
        data_files=data_files)`


### Expected behavior

![image](https://github.com/huggingface/datasets/assets/50519434/add3747f-6481-4da7-b374-8f81c5a6472c)


### Environment info

python version 3.8.5
datasets version 2.12
os version unbuntu 18.04

Hugging Face's datasets library may prioritize remote configurations. Make sure there are no conflicting configurations causing the library to prefer downloading data
May  be try debugging
raw_datasets = load_dataset('json', data_files=data_files)
print(raw_datasets)


It doesn't download them but writes them to the local HF cache. The logging could indeed be better. Does loading the dataset succeed? If it doesn't, can you share the error stack trace?
ValueError: 'index=True' is only valid when 'orient' is 'split', 'table', 'index', or 'columns'

### Describe the bug

Saving a dataset `.to_json()` fails with a `ValueError` since the latest `pandas` [release](https://pandas.pydata.org/docs/dev/whatsnew/v2.1.0.html) (`2.1.0`)

In their latest release we have:

> Improved error handling when using [DataFrame.to_json()](https://pandas.pydata.org/docs/dev/reference/api/pandas.DataFrame.to_json.html#pandas.DataFrame.to_json) with incompatible index and orient arguments ([GH 52143](https://github.com/pandas-dev/pandas/issues/52143))

i.e. an error is now raised for invalid combinations of `index` and `orient`.

This means that unfortunately the custom logic at this line might sometimes lead to contradictions:

https://github.com/huggingface/datasets/blob/029227a116c14720afca71b9b22e78eb2a1c09a6/src/datasets/io/json.py#L96

e.g. for the default case `orient=records` leads to `index=True`, which now raises a `ValueError`

### Steps to reproduce the bug

```python
import datasets


if __name__ == '__main__':

    dataset = datasets.Dataset.from_dict({"A": [1, 2, 3], "B": [4, 5, 6]})
    dataset.to_json("dataset.json")
```

```shell
>>>
ValueError: 'index=True' is only valid when 'orient' is 'split', 'table', 'index', or 'columns'.
```

### Expected behavior

The dataset is successfully saved as `.json`

### Environment info

`python >= 3.9`
`pandas >= 2.1.0`

Thanks for reporting. We are investigating it.

This issue is caused by latest `pandas` release 2.1.0 (released yesterday Aug 30).

See: https://github.com/huggingface/datasets/actions/runs/6035484010/job/16375932085?pr=6198


People using previous releases of `datasets` should pin `pandas` in their local environment:
```
python -m pip install 'pandas<2.1.0'
```
Split order is not preserved

I have noticed that in some cases the split order is not preserved.

For example, consider a no-script dataset with configs:
```yaml
configs:
- config_name: default
  data_files:
  - split: train
    path: train.csv
  - split: test
    path: test.csv
```
- Note the defined split order is [train, test]

Once the dataset is loaded, the split order is not preserved:
```python
In [16]: ds
Out[16]: 
DatasetDict({
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 1
    })
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 2
    })
})
```
- Note the obtained split order is [test, train]
Dataset loading script method does not work with .pyc file

### Describe the bug

The huggingface dataset library specifically looks for ‘.py’ file while loading the dataset using loading script approach and it does not work with ‘.pyc’ file.
While deploying in production, it becomes an issue when we are restricted to use only .pyc files. Is there any work around for this ?

### Steps to reproduce the bug

1. Create a dataset  loading script to read the custom data.
2. compile the code to make sure that .pyc file is created 
3. Delete the loading script and re-run the code. Usually, python should make use of complied .pyc files. However, in this case, the dataset library errors out with the message that it's unable to find the data loader loading script.

### Expected behavior

The code should make use of .pyc file and run without any error.

### Environment info

NA

Before dynamically loading `.py` scripts with `importlib.import_module`, we also parse their contents to check imports, which is tricky to implement for binary `.pyc` files (requires parsing bytecode), so I don't think this is something we want to support (unless more users request it ofc) as this use case is a bit too specific.

@lhoestq What's your opinion on this?

> Before dynamically loading .py scripts with importlib.import_module, we also parse their contents to check imports, which is tricky to implement for binary .pyc files (requires parsing bytecode), so I don't think this is something we want to support (unless more users request it ofc) as this use case is a bit too specific.

Yes indeed. Though you can use a .py that imports a package that contains your .pyc code and that you previously installed

Hi @lhoestq ,
Could you share some example code related to the approach that you are suggesting?
`Invalid user token` even when correct user token is passed!

### Describe the bug

I'm working on a dataset which comprises other datasets on the hub.
URL: https://huggingface.co/datasets/open-asr-leaderboard/datasets-test-only

Note: Some of the sub-datasets in this metadataset require explicit access.

All the other datasets work fine, except, `common_voice`.

### Steps to reproduce the bug

https://github.com/Vaibhavs10/scratchpad/blob/main/cv_datasets_bug_repro.ipynb

### Expected behavior

It should work if the provided access token is valid (as it does for all the other datasets)

### Environment info

datasets version -> 2.14.4

This is because `download_config.use_auth_token` is deprecated - you should use `download_config.token` instead

Works! Thanks for the quick fix! <3
[Feature Request] Check the length of batch before writing so that empty batch is allowed

### Use Case
I use `dataset.map(process_fn, batched=True)` to process the dataset, with data **augmentations or filtering**. However, when all examples within a batch is filtered out, i.e. **an empty batch is returned**, the following error will be thrown:
  ```
  ValueError: Schema and number of arrays unequal
  ```
This is because the empty batch does not comply with the schema of other batches. I think an empty batch should be allowed to facilitate coding (one does not need to assign an empty list manually for all keys.)

A simple fix is to check the length of `batch` before writing:
```
if len(batch):
  writer.write_batch(batch)
```
instead of 

https://github.com/huggingface/datasets/blob/74d60213dcbd7c99484c62ce1d3dfd90a1df0770/src/datasets/arrow_dataset.py#L3493


I think this error means you filter all examples within an (input) batch by deleting its columns. In that case, to avoid the error, you can set the column value to an empty list (`input_batch["col"] = []`) instead.
Error in saving the PIL image into *.arrow files using datasets.arrow_writer

### Describe the bug

I am using the ArrowWriter from datasets.arrow_writer to save a json-style file as arrow files. Within the dictionary, it contains a feature called "image" which is a list of PIL.Image objects.
I am saving the json using the following script:
```
def save_to_arrow(path,temp):
    with ArrowWriter(path=path,writer_batch_size=20) as writer: 
        writer.write_batch(temp) 
        writer.finalize() 
```
However, when I attempt to restore the dataset and use the ```Dataset.from_file(path)``` function to load the arrow file, there seems to be an issue with the PIL.Image object in the dataset. The list of PIL.Images appears as follows rather than a normal PIL.Image object:
![1693051705440](https://github.com/huggingface/datasets/assets/14247682/03b204c2-d0fa-4d19-beff-6f4d7b83c848)


### Steps to reproduce the bug

1. Storing the data json into arrow files:
```
def save_to_arrow(path,temp):
    with ArrowWriter(path=path,writer_batch_size=20) as writer: 
        writer.write_batch(temp) 
        writer.finalize() 
save_to_arrow( path, json_file )
```
2.  try to load the arrow file into the Dataset object using the ```Dataset.from_file(path)```


### Expected behavior

Except to saving the contained "image" feature as a list PIL.Image objects as the arrow file. And I can restore the dataset from the file.

### Environment info


- `datasets` version: 2.12.0
- Platform: Linux-5.4.0-150-generic-x86_64-with-glibc2.17
- Python version: 3.8.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.4.4

You can cast the `input_image` column to the `Image` type to fix the issue:
```python
ds.cast_column("input_image", datasets.Image())
```
Map cache does not detect function changes in another module

```python
# dataset.py
import os
import datasets

if not os.path.exists('/tmp/test.json'):
  with open('/tmp/test.json', 'w') as file:
    file.write('[{"text": "hello"}]')

def transform(example):
  text = example['text']
  # text += ' world'
  return {'text': text}

data = datasets.load_dataset('json', data_files=['/tmp/test.json'], split='train')
data = data.map(transform)
```

```python
# test.py
import dataset
print(next(iter(dataset.data)))
```

Initialize cache
```
python3 test.py
# {'text': 'hello'}
```

Edit dataset.py and uncomment the commented line, run again
```
python3 test.py
# {'text': 'hello'}
# expected: {'text': 'hello world'}
```

Clear cache and run again
```
rm -rf ~/.cache/huggingface/datasets/*
python3 test.py
# {'text': 'hello world'}
```


If instead the two files are combined, then changes to the function are detected correctly. But it's expected when working on any realistic codebase that things will be modularized into separate files.

This issue is a duplicate of https://github.com/huggingface/datasets/issues/3297.  This is a limitation of `dill`, a package we use for caching (non-`__main__` module objects are serialized by reference). You can find more info about it here: https://github.com/uqfoundation/dill/issues/424.

In your case, moving 
```
data = datasets.load_dataset('json', data_files=['/tmp/test.json'], split='train')
data = data.map(transform)
``` 
to `test.py`  and setting `transform.__module__ = None` at the end of `dataset.py` should fix the issue.

I understand this may be a limitation of an upstream tool, but for a user for datasets this is very annoying, as when you have dozens of different datasets with different preprocessing functions you can't really move them all into the same file. It may be worth seeing if there is a way to specialize the dependency (eg. subclass it) and enforce behaviors that makes sense for your product.

I was able to work around this for now by setting `__module__ = None`. If such workarounds are required for now it may be better to document it somewhere than a single obscure issue from a long time ago.

As this is a duplicate issue I'm closing it.

I have another issue with the cache https://github.com/huggingface/datasets/issues/6179 can you take a look?
Loading Meteor metric in HF evaluate module crashes due to datasets import issue 

### Describe the bug

When using python3.9 and ```evaluate``` module loading Meteor metric crashes at a non-existent import from ```datasets.config``` in ```datasets v2.14```

### Steps to reproduce the bug

```
from evaluate import load
meteor = load("meteor")
```

produces the following error:
  ```
    from datasets.config import importlib_metadata, version
ImportError: cannot import name 'importlib_metadata' from 'datasets.config' (<path_to_project>/venv/lib/python3.9/site-packages/datasets/config.py)
```

### Expected behavior

```datasets``` of v2.10 has the following workaround in ```config.py```:

```
if PY_VERSION < version.parse("3.8"):
    import importlib_metadata
else:
    import importlib.metadata as importlib_metadata
```

However, it's absent in v2.14 which might be the cause of the issue.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.5-arm64-arm-64bit
- Python version: 3.9.6
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- Evaluate version: 0.4.0

Our minimal Python version requirement is 3.8, so we dropped `importlib_metadata`. 

Feel free to open a PR in the `evaluate` repo to replace the problematic import with
```python
if PY_VERSION < version.parse("3.8"):
    import importlib_metadata
else:
    import importlib.metadata as importlib_metadata
```

Any idea when you guys will release the next version which deals with this problem?
I'm still having the same issue with py 3.10 when I install the lib with pip.
I'm assuming that it has not yet been updated since the merge was 3 days ago.

Yes, this requires a new `evaluate` release (cc @lvwerra for this). 

In the meantime, you can get the fixed version by installing `evaluate` from `main`: `pip install git+https://github.com/huggingface/evaluate.git`

I'll aim for a release this week!
'import datasets' throws "invalid syntax error"

### Describe the bug

Hi,

I have been trying to import the datasets library but I keep gtting this error.

`Traceback (most recent call last):

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/IPython/core/interactiveshell.py:3508 in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  Cell In[2], line 1
    import datasets

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/__init__.py:22
    from .arrow_dataset import Dataset

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/arrow_dataset.py:67
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/arrow_writer.py:27
    from .features import Features, Image, Value

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/features/__init__.py:17
    from .audio import Audio

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/features/audio.py:11
    from ..download.streaming_download_manager import xopen, xsplitext

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/download/__init__.py:10
    from .streaming_download_manager import StreamingDownloadManager

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/download/streaming_download_manager.py:18
    from aiohttp.client_exceptions import ClientError

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/aiohttp/__init__.py:7
    from .connector import *  # noqa

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/aiohttp/connector.py:12
    from .client import ClientRequest

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/aiohttp/client.py:144
    yield from asyncio.async(resp.release(), loop=loop)
                       ^
SyntaxError: invalid syntax`

I have simply used these commands:
`import datasets`
and
`from datasets import load_dataset`

### Environment info

The library has been installed a virtual machine on JupyterHub. Although I have used this library multiple times (on the same VM) before, to train/test an ASR or other ML models, I had never encountered this error. 

This seems to be related to your environment and not the `datasets` code (e.g., this could happen when exposing the  Python 3.9 site packages to a lower Python version (interpreter))
feat: Return the name of the currently loaded file

Added an optional parameter return_file_name in the load_dataset function. When it is set to True, the function will include the name of the file corresponding to the current line as a feature in the returned output.

I added this here https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/json/json.py#L92.

fixes #5806

Your change adds a new element in the key used to avoid duplicates when generating the examples of a dataset. I don't think it fixes the issue you're trying to solve.
Configurations in yaml not working

### Dataset configurations cannot be created in YAML/README

Hello! I'm trying to follow the docs here in order to create structure in my dataset as added from here (#5331): https://github.com/huggingface/datasets/blob/8b8e6ee067eb74e7965ca2a6768f15f9398cb7c8/docs/source/repository_structure.mdx#L110-L118

I have the exact example in my config file for [my data repo](https://huggingface.co/datasets/tsor13/test):
```
configs:
- config_name: main_data
  data_files: "main_data.csv"
- config_name: additional_data
  data_files: "additional_data.csv"
```

Yet, I'm unable to load different configurations:
```
from datasets import get_dataset_config_names
get_dataset_config_names('tsor13/test', use_auth_token=True)
```
returns a single split, `['tsor13--test']`

Does anyone have any insights?

@polinaeterna thank you for adding this feature, it is super useful. Do you happen to have any ideas?

### Steps to reproduce the bug

from datasets import get_dataset_config_names
get_dataset_config_names('tsor13/test')

### Expected behavior

I would expect there to be two splits, `main_data` and `additional_data`. However, only `['tsor13--test']` test is returned.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.1

Unfortunately, I cannot reproduce this behavior on my machine or Colab - the reproducer returns `['main_data', 'additional_data']` as expected.

Thank you for looking into this, Mario. Is this on [my repository](https://huggingface.co/datasets/tsor13/test), or on another one that you have reproduced? Would you mind pointing me to it if so?

Whoa, in colab I received the correct behavior using my dataset. It must have something to do with my local copy of `datasets` (which again just failed).

I've tried uninstalling/reinstnalling to no avail

hi @tsor13 , I haven't been able to reproduce your issue on `tsor13/test` dataset locally either. reinstalling doesn't help?
Error type: ArrowInvalid Details: Failed to parse string: '[254,254]' as a scalar of type int32

### Describe the bug

I am getting the following error while I am trying to upload the CSV sheet to train a model. My CSV sheet content is exactly same as shown in the example CSV file in the Auto Train page. Attaching screenshot of error for reference. I have also tried converting the index of the answer that are integer into string by placing inverted commas and also without inverted commas. 
Can anyone please help me out? 
FYI : I am using Chrome browser.

Error type: ArrowInvalid
Details: Failed to parse string: '[254,254]' as a scalar of type int32

![Screenshot 2023-08-19 165827](https://github.com/huggingface/datasets/assets/90616801/95fad96e-7dce-4bb5-9f83-9f1659a32891)


### Steps to reproduce the bug

Kindly let me know how to fix this?

### Expected behavior

Kindly let me know how to fix this?

### Environment info

Kindly let me know how to fix this?

Answered on the forum [here](https://discuss.huggingface.co/t/error-type-arrowinvalid-details-failed-to-parse-string-254-254-as-a-scalar-of-type-int32/51323).