KRadim/github-issues-minimal huggingface dataset:
Duplicate `data_files` when named `<split>/<split>.parquet`
State: closed
Labels: name: bug

e.g. with `u23429/stock_1_minute_ticker`

```ipython
In [1]: from datasets import *

In [2]: b = load_dataset_builder("u23429/stock_1_minute_ticker")
Downloading readme: 100%|██████████████████████████| 627/627 [00:00<00:00, 246kB/s]

In [3]: b.config.data_files
Out[3]: 
{NamedSplit('train'): ['hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/train/train.parquet',
  'hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/train/train.parquet'],
 NamedSplit('validation'): ['hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/validation/validation.parquet',
  'hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/validation/validation.parquet'],
 NamedSplit('test'): ['hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/test/test.parquet',
  'hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/test/test.parquet']}

```

This bug issue is present in the current `datasets` 2.14.5 and also on `main` even after https://github.com/huggingface/datasets/pull/6244 cc @mariosasko
Overwriting Split overwrites data but not metadata, corrupting dataset
State: closed

### Describe the bug

I want to be able to overwrite/update/delete splits in my dataset. Currently the only way to do is to manually go into the dataset and delete the split. If I try to overwrite programmatically I end up in an error state and (somewhat) corrupting the dataset. Read below.

**Current Behavior**
When I push to an existing split I get this error:
`ValueError: Split complexRoofLocation_01Apr2023_to_31May2023test already present` 
This seems to suggest that the library doesn't support overwriting splits. 

**Potential Bug**
What’s strange is that datasets, despite the operation erroring out with the ValueError above, does, in fact, overwrite the split:

`Pushing dataset shards to the dataset hub: 100% [.....................] 1/1 [00:00<00:00, 55.04it/s]` 
Even though you got an error message and your code fails, your dataset is now changed. That seems like a bug. Either don't change the dataset, or don't throw the error and allow the script to proceed. 

Additional Bug
While it overwrites the split, it doesn’t overwrite the split’s information. Because of this when you pull down the dataset you may end up getting a `NonMatchingSplitsSizesError` if the size of the dataset during the overwrite is different. For example, my original split had 5 rows, but on my overwrite, I only had 4. Then when I try to download the dataset, I get a `NonMatchingSplitsSizesError` because the dataset's data.json states there’s 5 but only 4 exist in the split.

Expected Behavior
This corrupts the dataset rendering it unusable (until you take manual intervention). Either the library should let the overwrite happen (which it does but should also update the metadata) or it shouldn’t do anything.

### Steps to reproduce the bug

[Colab Notebook](https://colab.research.google.com/drive/1bqVkD06Ngs9MQNdSk_ygCG6y1UqXA4pC?usp=sharing)


### Expected behavior

The split should be overwritten and I should be able to use the new version of the dataset without issue. 

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
Dataset.from_generator raises with sharded gen_args
State: closed

### Describe the bug

According to the docs of Datasets.from_generator:
```
        gen_kwargs(`dict`, *optional*):
            Keyword arguments to be passed to the `generator` callable.
            You can define a sharded dataset by passing the list of shards in `gen_kwargs`.
```
So I'd expect that if gen_kwargs was a list, then my generator would be called once for each element in the list with the dict in the list for that element.
It doesn't work that way though.

### Steps to reproduce the bug

```python
#!/usr/bin/python

from pathlib import Path
import datasets

def process_yaml(file):
    yield dict(example=42)


if __name__ == '__main__':
    import sys
    dir = Path(sys.argv[0]).parent
    ds = datasets.Dataset.from_generator(process_yaml, gen_kwargs=[{'file':f} for f in dir.glob('*.yml')],
        )
    ds.to_json('training.jsonl')
    

```
```
Generating train split: 0 examples [00:00, ? examples/s]
Traceback (most recent call last):
  File "/tmp/dataset_bug.py", line 13, in <module>
    ds = datasets.Dataset.from_generator(process_yaml, gen_kwargs=[{'file':f} for f in dir.glob('*.yml')],
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hartmans/ai/venv/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 1072, in from_generator
    ).read()
      ^^^^^^
  File "/home/hartmans/ai/venv/lib/python3.11/site-packages/datasets/io/generator.py", line 47, in read
    self.builder.download_and_prepare(
  File "/home/hartmans/ai/venv/lib/python3.11/site-packages/datasets/builder.py", line 954, in download_and_prepare
    self._download_and_prepare(
  File "/home/hartmans/ai/venv/lib/python3.11/site-packages/datasets/builder.py", line 1717, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/hartmans/ai/venv/lib/python3.11/site-packages/datasets/builder.py", line 1049, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/hartmans/ai/venv/lib/python3.11/site-packages/datasets/builder.py", line 1555, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/home/hartmans/ai/venv/lib/python3.11/site-packages/datasets/builder.py", line 1656, in _prepare_split_single
    generator = self._generate_examples(**gen_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: datasets.packaged_modules.generator.generator.Generator._generate_examples() argument after ** must be a ```
mapping, not list


### Expected behavior

I would expect that process_yaml would be called once for each yaml file in the directory where the script is run.
I also tried with the list being in gen_kwargs, but in that case process_yaml gets called with a list.


### Environment info

- `datasets` version: 2.14.6.dev0 (git commit 0cc77d7f45c7369; also tested with 2.14.0)
- Platform: Linux-6.1.0-10-amd64-x86_64-with-glibc2.36
- Python version: 3.11.2
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
Reduce the number of commits in `push_to_hub`
State: closed

Reduces the number of commits in `push_to_hub` by using the `preupload` API from https://github.com/huggingface/huggingface_hub/pull/1699. Each commit contains a maximum of 50 uploaded files.

A shard's fingerprint no longer needs to be added as a suffix to support resuming an upload, meaning the shards' naming scheme is the same as the initial one.

Also, it adds support for the following params: `create_pr`, `commit_message` and `revision` (`branch` deprecated; unlike the previous implementation, this one creates a branch if the branch does not exist to be consistent with `transformers`). 

(Nit) This implementation keeps the markdown section of the generated README.md empty to enable importing the card template (when the card is accessed on the Hub).

Fixes https://github.com/huggingface/datasets/issues/5492, fixes https://github.com/huggingface/datasets/issues/6257, fixes https://github.com/huggingface/datasets/issues/5045, fixes https://github.com/huggingface/datasets/issues/6271

TODO:
- [x] set the minimal version to the next `hfh` release (once it's published)
Add repo_id to DatasetInfo
State: open

```python
from datasets import load_dataset

ds = load_dataset("lhoestq/demo1", split="train")
ds = ds.map(lambda x: {}, num_proc=2).filter(lambda x: True).remove_columns(["id"])
print(ds.repo_id)
# lhoestq/demo1
```

- repo_id is None when the dataset doesn't come from the Hub, e.g. from Dataset.from_dict
- repo_id is set to None when concatenating datasets with different repo ids

related to https://github.com/huggingface/datasets/issues/4129

TODO:
- [ ] discuss if it's ok for now
- [ ] tests
Multi label class encoding
State: open
Labels: name: enhancement

### Feature request

I have a multi label dataset and I'd like to be able to class encode the column and store the mapping directly in the features just as I can with a single label column.  `class_encode_column` currently does not support multi labels.

Here's an example of what I'd like to encode:

```
data = {
    'text': ['one', 'two', 'three', 'four'],
    'labels': [['a', 'b'], ['b'], ['b', 'c'], ['a', 'd']]
}

dataset = Dataset.from_dict(data)
dataset = dataset.class_encode_column('labels')
```

I did some digging into the code base to evaluate the feasibility of this (note I'm very new to this code base) and from what I noticed the `ClassLabel` feature is still stored as an underlying raw data type of int so I thought a `MultiLabel` feature could similarly be stored as a Sequence of ints, thus not requiring significant serialization / conversion work to / from arrow.

I did a POC of this [here](https://github.com/huggingface/datasets/commit/15443098e9ce053943172f7ec6fce3769d7dff6e) and included a simple test case (please excuse all the commented out tests, going for speed of POC here and didn't want to fight IDE to debug a single test).   In the test I just assert that `num_classes` is the same to show that things are properly serializing, but if you break after loading from disk you'll see the dataset correct and the dataset feature is as expected.

After digging more I did notice a few issues
- After loading from disk I noticed type of the `labels` class is `Sequence` not `MultiLabel` (though the added `feature` attribute came through).  This doesn't happen for `ClassLabel` but I couldn't find the encode / decode code paths that handle this.
- I subclass `Sequence` in `MultiLabel` to leverage existing serialization, but this does miss the custom encode logic that `ClassLabel` has.  I'm not sure of the best way to approach this as I haven't fully understood the encode / decode flow for datasets.  I suspect my simple implementation will need some improvement as it'll require a significant amount of repeated logic to mimic `ClassLabel` behavior.



### Motivation

See above - would like to support multi label class encodings.

### Your contribution

This would be a big help for us and we're open to contributing but I'll likely need some guidance on how to implement to fit the encode / decode flow.  Some suggestions on tests / would be great too, I'm guessing in addition to the class encode tests (that I'll need to expand) we'll need encode / decode tests.
Use LibYAML with PyYAML if available
State: open

PyYAML, the YAML framework used in this library, allows the use of LibYAML to accelerate the methods `load` and `dump`. To use it, a user would need to first install a PyYAML version that uses LibYAML (not available in PyPI; needs to be manually installed). Then, to actually use them, PyYAML suggests importing the LibYAML version of the `Loader` and `Dumper` and falling back to the default ones. This PR implements this change. See [PyYAML docs](https://pyyaml.org/wiki/PyYAMLDocumentation) for more info.

This change was motivated after trying to use any of [the SugarCREPE datasets in the Hub](https://huggingface.co/datasets?search=sugarcrepe) provided by [the org HuggingFaceM4](https://huggingface.co/datasets/HuggingFaceM4). Such datasets save a lot of information (~1MB) in the YAML metadata from the `README.md` file and I noticed this slowed down the data loading process. BTW, I also noticed cache files for it is also slow because it tries to hash an instance of `DatasetInfo`, which in turn has all this metadata.

Also, I changed two list comprehensions into generator expressions to avoid allocating extra memory unnecessarily.

And BTW, there's [an issue in PyYAML suggesting to make this automatic](https://github.com/yaml/pyyaml/issues/437).
CI is broken: ImportError: cannot import name 'context' from 'tensorflow.python'
State: closed
Labels: name: bug

Python 3.10 CI is broken for `test_py310`.

See: https://github.com/huggingface/datasets/actions/runs/6322990957/job/17169678812?pr=6262

```
FAILED tests/test_py_utils.py::TempSeedTest::test_tensorflow - ImportError: cannot import name 'context' from 'tensorflow.python' (/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/tensorflow/python/__init__.py)
```

```
_________________________ TempSeedTest.test_tensorflow _________________________
[gw1] linux -- Python 3.10.13 /opt/hostedtoolcache/Python/3.10.13/x64/bin/python

self = <tests.test_py_utils.TempSeedTest testMethod=test_tensorflow>

    @require_tf
    def test_tensorflow(self):
        import tensorflow as tf
        from tensorflow.keras import layers
    
        model = layers.Dense(2)
    
        def gen_random_output():
            x = tf.random.uniform((1, 3))
            return model(x).numpy()
    
>       with temp_seed(42, set_tensorflow=True):

tests/test_py_utils.py:155: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/contextlib.py:135: in __enter__
    return next(self.gen)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

seed = 42, set_pytorch = False, set_tensorflow = True

    @contextmanager
    def temp_seed(seed: int, set_pytorch=False, set_tensorflow=False):
        """Temporarily set the random seed. This works for python numpy, pytorch and tensorflow."""
        np_state = np.random.get_state()
        np.random.seed(seed)
    
        if set_pytorch and config.TORCH_AVAILABLE:
            import torch
    
            torch_state = torch.random.get_rng_state()
            torch.random.manual_seed(seed)
    
            if torch.cuda.is_available():
                torch_cuda_states = torch.cuda.get_rng_state_all()
                torch.cuda.manual_seed_all(seed)
    
        if set_tensorflow and config.TF_AVAILABLE:
            import tensorflow as tf
>           from tensorflow.python import context as tfpycontext
E           ImportError: cannot import name 'context' from 'tensorflow.python' (/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/tensorflow/python/__init__.py)

/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/datasets/utils/py_utils.py:257: ImportError
```
Fix CI 404 errors
State: closed

Currently our CI usually raises 404 errors when trying to delete temporary repositories. See, e.g.: https://github.com/huggingface/datasets/actions/runs/6314980985/job/17146507884
```
FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_multiple_files_with_max_shard_size - huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6512fb99-4a52c561752ece3d77eb6d57;2b61cae4-613d-4a73-bbb1-2faf9e32b02d)
Repository Not Found for url: https://hub-ci.huggingface.co/api/repos/delete.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.

FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_to_hub_custom_features_audio - huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6512fbb2-0333dd666d42f0e173c2bb68;dfdc4271-b49b-4008-8c49-f05cf7c1d53d)
Repository Not Found for url: https://hub-ci.huggingface.co/api/repos/delete.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.

FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_custom_splits - huggingface_hub.utils._errors.RepositoryNotFoundError: 404 Client Error. (Request ID: Root=1-6512fbca-167690694f39770a5b3a444e;baeaa905-0a57-4585-ac97-9aaae12dd47d)
Repository Not Found for url: https://hub-ci.huggingface.co/api/repos/delete.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
```

I think this can be caused by collisions in temporary repository IDs because we create them in multiprocessing:
```python
with temporary_repo(f"{CI_HUB_USER}/test-{int(time.time() * 10e3)}") as ds_name:
```
This can also be caused when there is another issue that does not allow the creation of the repository, thus making it impossible to delete it.

This PR tries to fix this issue by increasing the precision of the number on the repository ID: `10e6` instead of `10e3`.
Additionally, this PR catches RepositoryNotFoundError.
REUSE_DATASET_IF_EXISTS don't work 
State: closed

### Describe the bug

I use the following code  to download natural_question dataset. Even though I have completely download it, the next time I run this code, the new download procedure will start and cover the original /data/lxy/NQ

config=datasets.DownloadConfig(resume_download=True,max_retries=100,cache_dir=r'/data/lxy/NQ',download_desc='NQ')

data=datasets.load_dataset('natural_questions',cache_dir=r'/data/lxy/NQ',download_config=config,download_mode=DownloadMode.REUSE_DATASET_IF_EXISTS)

---
Since I don't have apache_beam installed, it throw a exception. After I pip install apache_beam ,the download restart..

![image](https://github.com/huggingface/datasets/assets/88258534/f28ce7fe-29ea-4348-b87f-e69182a8bd41)


### Steps to reproduce the bug

run this two line code

config=datasets.DownloadConfig(resume_download=True,max_retries=100,cache_dir=r'/data/lxy/NQ',download_desc='NQ')

data=datasets.load_dataset('natural_questions',cache_dir=r'/data/lxy/NQ',download_config=config,download_mode=DownloadMode.REUSE_DATASET_IF_EXISTS)

### Expected behavior

Download behavior can be correctly follow DownloadMode

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-3.10.0-1160.88.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.3
Duplicated Rows When Loading Parquet Files from Root Directory with Subdirectories
State: closed

### Describe the bug

When parquet files are saved in "train" and "val" subdirectories under a root directory, and datasets are then loaded using `load_dataset("parquet", data_dir="root_directory")`, the resulting dataset has duplicated rows for both the training and validation sets.

### Steps to reproduce the bug

1. Create a root directory, e.g., "testing123".
2. Under "testing123", create two subdirectories: "train" and "val".
3. Create and save a parquet file with 3 unique rows in the "train" subdirectory.
4. Create and save a parquet file with 4 unique rows in the "val" subdirectory.
5. Load the datasets from the root directory using `load_dataset("parquet", data_dir="testing123")`
6. Iterate through the datasets and print the rows

Here's a collab reproducing these steps:

https://colab.research.google.com/drive/11NEdImnQ3OqJlwKSHRMhr7jCBesNdLY4?usp=sharing

### Expected behavior

- Training set should contain 3 unique rows.
- Validation set should contain 4 unique rows.

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.2
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
HfHubHTTPError - exceeded our hourly quotas for action: commit
State: closed

### Describe the bug

I try to upload a very large dataset of images, and get the following error:
```
File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/hf_api.py:2712, in HfApi.create_commit(self, repo_id, operations, commit_message, commit_description, token, repo_type, revision, create_pr, num_threads, parent_commit, run_as_future)
   2710 try:
   2711     commit_resp = get_session().post(url=commit_url, headers=headers, data=data, params=params)
-> 2712     hf_raise_for_status(commit_resp, endpoint_name="commit")
   2713 except RepositoryNotFoundError as e:
   2714     e.append_to_message(_CREATE_COMMIT_NO_REPO_ERROR_MESSAGE)

File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py:301, in hf_raise_for_status(response, endpoint_name)
    297     raise BadRequestError(message, response=response) from e
    299 # Convert `HTTPError` into a `HfHubHTTPError` to display request information
    300 # as well (request id and/or server error message)
--> 301 raise HfHubHTTPError(str(e), response=response) from e

HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/yuvalkirstain/pickapic_v2/commit/main (Request ID: Root=1-65112399-12d63f7d7f28bfa40a36a0fd)

You have exceeded our hourly quotas for action: commit. We invite you to retry later.
```

this makes it much less convenient to host large datasets on HF hub.

### Steps to reproduce the bug

Upload a very large dataset of images

### Expected behavior

the upload to work well

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.0-1033-aws-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 1.5.3
load_dataset() function's cache_dir does not seems to work
State: closed

### Describe the bug

datasets version: 2.14.5

when trying to run the following command
trec = load_dataset('trec', split='train[:1000]', cache_dir='/path/to/my/dir')

I keep getting error saying the command does not have permission to the default cache directory on my macbook pro machine. 

It seems the cache_dir parameter cannot change the dataset saving directory from the default 

what ever explained in the https://huggingface.co/docs/datasets/cache does not seem to work


### Steps to reproduce the bug

datasets version: 2.14.5

when trying to run the following command
trec = load_dataset('trec', split='train[:1000]', cache_dir='/path/to/my/dir')

I keep getting error saying the command does not have permission to the default cache directory on my macbook pro machine. 

It seems the cache_dir parameter cannot change the dataset saving directory from the default 

what ever explained in the https://huggingface.co/docs/datasets/cache does not seem to work

### Expected behavior

the dataset should be saved to the cache_dir points to 

### Environment info

datasets version: 2.14.5
macos X: Ventura 13.4.1 (c)
Dataset.from_generator() cost much more time in vscode debugging mode then running mode
State: closed

### Describe the bug

Hey there,
I’m using Dataset.from_generator() to convert a torch_dataset to the Huggingface Dataset.
However, when I debug my code on vscode, I find that it runs really slow on Dataset.from_generator() which may even 20 times longer then run the script on terminal.

### Steps to reproduce the bug

I write a simple test code :

```python
import os
from functools import partial
from typing import Callable

import torch
import time
from torch.utils.data import Dataset as TorchDataset

from datasets import load_from_disk, Dataset as HFDataset
  
import torch  
from torch.utils.data import Dataset  
  
class SimpleDataset(Dataset):  
    def __init__(self, data):  
        self.data = data  
        self.keys = list(data[0].keys())
      
    def __len__(self):  
        return len(self.data)  
      
    def __getitem__(self, index):  
        sample = self.data[index]  
        return {key: sample[key] for key in self.keys}  
  

def TorchDataset2HuggingfaceDataset(torch_dataset: TorchDataset, cache_dir: str = None
) -> HFDataset:
    
    """
        convert torch dataset to huggingface dataset
    """
    generator : Callable[[], TorchDataset] = lambda: (sample for sample in torch_dataset)   

    return HFDataset.from_generator(generator, cache_dir=cache_dir)

if __name__ == '__main__':
    data = [  
        {'id': 1, 'name': 'Alice'},  
        {'id': 2, 'name': 'Bob'},  
        {'id': 3, 'name': 'Charlie'}  
    ]
    
    torch_dataset = SimpleDataset(data)
    start_time = time.time() 
    huggingface_dataset = TorchDataset2HuggingfaceDataset(torch_dataset)
    end_time = time.time()
    print("time: ", end_time - start_time)
    print(huggingface_dataset)
```

### Expected behavior

this test on my machine report that the running time on terminal is 0.086,
however the running time in debugging mode on vscode is 0.25, which I think is much longer than expected.

I’d like to know is the anything wrong in the code or just because of debugging?
I have traced the code and I find is this func which I get stuck.

```python
def create_config_id(
        self,
        config_kwargs: dict,
        custom_features: Optional[Features] = None,
    ) -> str:
...
# stuck in this line
suffix = Hasher.hash(config_kwargs_to_add_to_suffix)
```



### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.11.0-27-generic-x86_64-with-glibc2.31
- Python version: 3.11.3
- Huggingface_hub version: 0.17.2
- PyArrow version: 11.0.0
- Pandas version: 2.0.1
Check builder cls default config name in inspect
State: closed

Fix https://github.com/huggingface/datasets-server/issues/1812

this was causing this issue:

```ipython
In [1]: from datasets import *

In [2]: inspect.get_dataset_config_names("aakanksha/udpos")
Out[2]: ['default']

In [3]: load_dataset_builder("aakanksha/udpos").config.name
Out[3]: 'en'
```
exif_transpose not done to Image (PIL problem)
State: closed
Labels: name: enhancement

### Feature request

I noticed that some of my images loaded using PIL have some metadata related to exif that can rotate them when loading.
Since the dataset.features.Image uses PIL for loading, the loaded image may be rotated (width and height will be inverted) thus for tasks as object detection and layoutLM this can create some inconsistencies (between input bboxes and input images). 

For now there is no option in datasets.features.Image to specify that. We need to do the following when preparing examples (when preparing images for training, test or inference): 
```
from PIL import Image, ImageOps 
pil = ImageOps.exif_transpose(pil)
```

reference: https://stackoverflow.com/a/63950647/5720150 

Is it possible to add this by default to the datasets.feature.Image ? or to add the option to do the ImageOps.exif_transpose?

Thank you

### Motivation

Prevent having inverted data related to exif metadata that may affect object detection tasks

### Your contribution

Changing in datasets.featrues.Image I can help with that.
Update create_dataset.mdx
State: closed

modified , as AudioFolder and ImageFolder not in Dataset Library. 

``` from datasets import AudioFolder ```  and ```from datasets import ImageFolder```  to ```from datasets import load_dataset```

``` 
cannot import name 'AudioFolder' from 'datasets' (/home/eswardivi/miniconda3/envs/Hugformers/lib/python3.10/site-packages/datasets/__init__.py)

```
Add new column to dataset
State: closed

### Describe the bug

```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
[<ipython-input-9-bd197b36b6a0>](https://localhost:8080/#) in <cell line: 1>()
----> 1 dataset['train']['/workspace/data']

3 frames
[/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py](https://localhost:8080/#) in _check_valid_column_key(key, columns)
    518 def _check_valid_column_key(key: str, columns: List[str]) -> None:
    519     if key not in columns:
--> 520         raise KeyError(f"Column {key} not in the dataset. Current columns in the dataset: {columns}")
    521 
    522 

KeyError: "Column train not in the dataset. Current columns in the dataset: ['image', '/workspace/data']"
```

### Steps to reproduce the bug

please find the notebook for reference: https://colab.research.google.com/drive/10lZ_zLtU4itYVmIVTvIEVbjfOtCZaAZy?usp=sharing

### Expected behavior

add column to the dataset

### Environment info

colab pro
Data alteration when loading dataset with unspecified inner sequence length
State: closed

### Describe the bug

When a dataset saved with a specified inner sequence length is loaded without specifying that length, the original data is altered and becomes inconsistent.

### Steps to reproduce the bug

```python
from datasets import Dataset, Features, Value, Sequence, load_dataset

# Repository ID
repo_id = "my_repo_id"

# Define features with a specific length of 3 for each inner sequence
specified_features = Features({"key": Sequence(Sequence(Value("float32"), length=3))})

# Create a dataset with the specified features
data = [
    [[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]],
    [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]],
]
dataset = Dataset.from_dict({"key": data}, features=specified_features)

# Push the dataset to the hub
dataset.push_to_hub(repo_id)

# Define features without specifying the length
unspecified_features = Features({"key": Sequence(Sequence(Value("float32")))})

# Load the dataset from the hub with this new feature definition
dataset = load_dataset(f"qgallouedec/{repo_id}", split="train", features=unspecified_features)

# The obtained data is altered
print(dataset.to_dict())  # {'key': [[[1.0], [2.0]], [[3.0], [4.0]]]}
```


### Expected behavior

```python
print(dataset.to_dict())  # {'key': [[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]], [[7.0, 8.0, 9.0], [10.0, 11.0, 12.0]]]}
```

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-6.2.0-32-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
Dataloader stuck on multiple GPUs
State: closed

### Describe the bug

I am trying to get CLIP to fine-tuning with my code.
When I tried to run it on multiple GPUs using accelerate, I encountered the following phenomenon.

- Validation dataloader stuck in 2nd epoch only on multi-GPU
Specifically, when the "for inputs in valid_loader:" process is finished, it does not proceed to the next step. train_loader process is completed. Also, both train and valid are working correctly in the first epoch.
The accelerate command at that time is as follows.
`accelerate launch --multi_gpu --num_processes=2 {script_name.py} {--arg1} {--arg2} ...`

- This will not happen when single GPU is used.
`CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...`
- Setting num_workers=0 in dataloader did not change the result.

### Steps to reproduce the bug

1. The codes for fine-tuning the regular CLIP were updated for accelerate.
2. Run the code with the accelerate command as `accelerate launch --multi_gpu --num_processes=2 {script_name.py} {--arg1} {--arg2} ...` and the above problem will occur.
3. CUDA_VISIBLE_DEVICES="0" accelerate launch {script_name.py} --arg1 --arg2 ...` , it works fine.

### Expected behavior

It Should end normally as if it was run on a single GPU.

### Environment info

Since `datasets-cli env` did not work, the environment is described below.

- OS: Ubuntu 22.04 with Docker
- Docker: 24.0.5, build ced0996
- Python: 3.10.12
- torch==2.0.1
- accelerate==0.21.0
- transformers==4.33.1
`dataset.filter` ALWAYS removes the first item from the dataset when using batched=True
State: closed

### Describe the bug

If you call batched=True when calling `filter`, the first item is _always_ filtered out, regardless of the filter condition.



### Steps to reproduce the bug

Here's a minimal example:
```python
def filter_batch_always_true(batch, indices):
    print("First index being passed into this filter function: ", indices[0])
    return indices  # Keep all indices

data = {"value": list(range(10))}
dataset = Dataset.from_dict(data)

filtered_dataset = dataset.filter(filter_batch_always_true, with_indices=True, batched=True)
print("Length of original dataset: ", len(dataset))
print("Length of filtered_dataset: ", len(filtered_dataset))
print("Is equal to original? ", len(filtered_dataset) == len(dataset))
print("First item of filtered dataset: ", filtered_dataset[0])
print("Last item of filtered dataset: ", filtered_dataset[-1])
```
prints:
```
First index being passed into this filter function:  0
Length of original dataset:  10
Length of filtered_dataset:  9
Is equal to original?  False
First item of filtered dataset:  {'value': 1}
Last item of filtered dataset:  {'value': 9}
```

### Expected behavior

Filter should respect the filter condition.

### Environment info


- `datasets` version: 2.14.4
- Platform: macOS-13.5-arm64-arm-64bit
- Python version: 3.9.18
- Huggingface_hub version: 0.17.1
- PyArrow version: 10.0.1
- Pandas version: 2.0.2
Tokenization with multiple workers is too slow
State: closed

I am trying to tokenize a few million documents with multiple workers but the tokenization process is taking forever. 

Code snippet:

```
raw_datasets.map(
            encode_function,
            batched=False,
            num_proc=args.preprocessing_num_workers,
            load_from_cache_file=not args.overwrite_cache,
            remove_columns=[name for name in raw_datasets["train"].column_names if name not in ["input_ids", "labels", "attention_mask"]],
            desc="Tokenizing data",
        )
```

Details:

```
transformers==4.28.0.dev0
datasets==4.28.0.dev0
preprocessing_num_workers==48
```
tokenizer == decapoda-research/llama-7b-hf
Support buffer shuffle for to_tf_dataset
State: open
Labels: name: enhancement

### Feature request

I'm using to_tf_dataset to convert a large dataset to tf.data.Dataset and use Keras fit to train model. 
Currently, to_tf_dataset only supports full size shuffle, which can be very slow on large dataset. 
tf.data.Dataset support buffer shuffle by default. 
shuffle(
    buffer_size, seed=None, reshuffle_each_iteration=None, name=None
)

### Motivation

I'm very frustrated to find the loading with shuffling large dataset is very slow. It seems impossible to shuffle before training Keras with big dataset. 

### Your contribution

NA
Support multiprocessing for download/extract nestedly
State: open
Labels: name: enhancement

### Feature request

Current multiprocessing for download/extract is not done nestedly. For example, when processing SlimPajama, there is only 3 processes (for train/test/val), while there are many files inside these 3 folders

```
Downloading data files #0:   0%|          | 0/1 [00:00<?, ?obj/s]
Downloading data files #1:   0%|          | 0/1 [00:00<?, ?obj/s]
Downloading data files #2:   0%|          | 0/1 [00:00<?, ?obj/s]

Extracting data files #0:   0%|          | 0/1 [00:00<?, ?obj/s]
Extracting data files #1:   0%|          | 0/1 [00:00<?, ?obj/s][A
Extracting data files #2:   0%|          | 0/1 [00:00<?, ?obj/s][A[A
```

### Motivation

speedup dataset loading

### Your contribution

I can help test the feature
Overwrite legacy default config name in `dataset_infos.json` in packaged datasets
State: open

Currently if we push data as default config with `.push_to_hub` to a repo that has a legacy `dataset_infos.json` file containing a legacy default config name like `{username}--{dataset_name}`, new key `"default"` is added to `dataset_infos.json` along with the legacy one. I think the legacy one should be dropped in this case.

Also, in `load.py` I suggest to check if a legacy config name is indeed a legacy config name because after this fix it might not be the case (this check was first introduced in https://github.com/huggingface/datasets/pull/6218)
Conversion from RGB to BGR in Object Detection tutorial
State: closed

The [tutorial](https://huggingface.co/docs/datasets/main/en/object_detection) mentions the necessity of conversion the input image from BGR to RGB

> albumentations expects the image to be in BGR format, not RGB, so you’ll have to convert the image before applying the transform.

[Link to tutorial](https://github.com/huggingface/datasets/blob/0a068dbf3b446417ffd89d32857608394ec699e6/docs/source/object_detection.mdx#L77)

However, relevant albumentations' tutorials [on channels conversion](https://albumentations.ai/docs/examples/example/#read-the-image-from-the-disk-and-convert-it-from-the-bgr-color-space-to-the-rgb-color-space) and  [on boxes](https://albumentations.ai/docs/examples/example_bboxes/) imply that it's not really true no more.

I suggest removing this outdated conversion from the tutorial.
Ignore `dataset_info.json` in data files resolution
State: closed

`save_to_disk` creates this file, but also [`HugginFaceDatasetSever`](https://github.com/gradio-app/gradio/blob/26fef8c7f85a006c7e25cdbed1792df19c512d02/gradio/flagging.py#L214), so this is needed to avoid issues such as [this one](https://discord.com/channels/879548962464493619/1149295819938349107/1149295819938349107).
Support saving datasets with custom formatting
State: open

Requested in https://discuss.huggingface.co/t/using-set-transform-on-a-dataset-leads-to-an-exception/53036.

I am not sure if supporting this is the best idea for the following reasons:

>For this to work, we would have to pickle a custom transform, which means the transform and the objects it references need to be serializable. Also, deserializing these bytes would make `load_from_disk` unsafe, so I'm not sure this is a good idea. 

@lhoestq WDYT?
Rename old push_to_hub configs to "default" in dataset_infos
State: closed

Fix

```python
from datasets import load_dataset_builder

b = load_dataset_builder("lambdalabs/pokemon-blip-captions", "default")
print(b.info)
```

which should return

```
DatasetInfo(
  features={'image': Image(decode=True, id=None), 'text': Value(dtype='string', id=None)}, 
  dataset_name='pokemon-blip-captions',
  config_name='default', 
  version=0.0.0,
  splits={'train': SplitInfo(name='train', num_bytes=119417410.0, num_examples=833, shard_lengths=None, dataset_name='pokemon-blip-captions')},
  download_checksums=None,
  download_size=99672355,
  dataset_size=119417410.0,
  size_in_bytes=219089765.0,
  ...
)
```

instead of and empty dataset info.

The dataset has a dataset_infos.json file with a deprecated config name "lambdalabs--pokemon-blip-captions". We switched those config names to "default" in 2.14, so the builder.info should take this into account.
`Dataset.to_dict()` ignore `decode=True` with Image feature
State: open

### Describe the bug

`Dataset.to_dict` seems to ignore the decoding instruction passed in features.

### Steps to reproduce the bug

```python
import datasets
import numpy as np
from PIL import Image

img = np.random.randint(0, 256, (5, 5, 3), dtype=np.uint8)
img = Image.fromarray(img)

features = datasets.Features({"image": datasets.Image(decode=True)})
dataset = datasets.Dataset.from_dict({"image": [img]}, features=features)
print({key: dataset[key] for key in dataset.column_names})
# {'image': [<PIL.PngImagePlugin.PngImageFile image mode=RGB size=5x5 at 0x7EFBC80E15B0>]}
print(dataset.to_dict())
# {'image': [{'bytes': b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x05\x00\x00\x00\x05\x08\x02\x00\x00\x00\x02\r\xb1\xb2\x00\x00\x00[IDATx\x9c\x01P\x00\xaf\xff\x01\x13\x1b<7\xe7\xe0\xdc^6\xed\x04\xc7M\xd2\x9f\x00X\x1b\xb0?\x1ba\x15\xc5 o\xd0\x80\xbe\x19/\x01\xec\x95\x1f\x9f\xffj\xfa1\xa7\xc4X\xea\xbe\xa4g\x00\xc4\x15\xdeC\xc7 \xbbaqe\xc8\xb9\xa9q\xe7\x00,?M\xc0)\xdaD`}\xb1\xdci\x1e\xafC\xa9]%.@\xa6\xf0\xb3\x00\x00\x00\x00IEND\xaeB`\x82', 'path': None}]}
```




### Expected behavior

I would expect `{key: dataset[key] for key in dataset.column_names}` and `dataset.to_dict()` to be equivalent. If the previous behavior is expected, then it should be stated [in the doc](https://huggingface.co/docs/datasets/v2.14.4/en/package_reference/main_classes#datasets.Dataset.to_dict).

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-6.2.0-31-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- Pillow 9.5.0
- numpy 1.25.2
Unpin fsspec < 2023.9.0
State: closed
Labels: name: enhancement

Once root issue is fixed, remove temporary pin of fsspec < 2023.9.0 introduced by:
- #6210

Related to issue:
- #6209

After investigation, I think the root issue is related to the new glob behavior with double asterisk `**` they have introduced in:
- https://github.com/fsspec/filesystem_spec/pull/1329
No-script datasets with ZIP files do not load
State: closed
Labels: name: bug

While investigating an issue on a Hub dataset, I have discovered the no-script datasets containing ZIP files do not load.

For example, that no-script dataset containing ZIP files, raises NonMatchingSplitsSizesError:
```python
In [2]: ds = load_dataset("sidovic/LearningQ-qg")

NonMatchingSplitsSizesError: [
  {
    'expected': SplitInfo(name='train', num_bytes=0, num_examples=188660, shard_lengths=None, dataset_name=None), 
    'recorded': SplitInfo(name='train', num_bytes=0, num_examples=0, shard_lengths=None, dataset_name='learning_q-qg')
  }, {
    'expected': SplitInfo(name='validation', num_bytes=0, num_examples=20630, shard_lengths=None, dataset_name=None), 
    'recorded': SplitInfo(name='validation', num_bytes=0, num_examples=0, shard_lengths=None, dataset_name='learning_q-qg')
  }, {
    'expected': SplitInfo(name='test', num_bytes=0, num_examples=18227, shard_lengths=None, dataset_name=None), 
    'recorded': SplitInfo(name='test', num_bytes=0, num_examples=0, shard_lengths=None, dataset_name='learning_q-qg')
  }
]
```

As another example, a no-script dataset containing just a (CSV)-ZIP file, raises a DatasetGenerationError:
```
>               num_examples, num_bytes = writer.finalize()

src/datasets/builder.py:1949: 

>           raise SchemaInferenceError("Please pass `features` or at least one example when writing data")
E           datasets.arrow_writer.SchemaInferenceError: Please pass `features` or at least one example when writing data

src/datasets/arrow_writer.py:598: SchemaInferenceError

The above exception was the direct cause of the following exception:

src/datasets/load.py:2143: in load_dataset
    builder_instance.download_and_prepare(
src/datasets/builder.py:954: in download_and_prepare
    self._download_and_prepare(
src/datasets/builder.py:1049: in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
src/datasets/builder.py:1813: in _prepare_split
    for job_id, done, content in self._prepare_split_single(

>           raise DatasetGenerationError("An error occurred while generating the dataset") from e
E           datasets.builder.DatasetGenerationError: An error occurred while generating the dataset

src/datasets/builder.py:1958: DatasetGenerationError
```

After investigating, I think this bug was introduced in this PR:
- #5972

Related to:
- https://huggingface.co/datasets/sidovic/LearningQ-qg/discussions/1

CC: @lhoestq
Support loading from a DVC remote repository
State: closed
Labels: name: enhancement

### Feature request

Adding support for loading a file from a DVC repository, tracked remotely on a SCM.

### Motivation

DVC is a popular version control system to version and manage datasets. The files are stored on a remote object storage platform, but they are tracked using Git. Integration with DVC is possible through the `DVCFileSystem`.

I have a Gitlab repository where multiple files are tracked using DVC and stored in a GCP bucket. I would like to be able to load these files using `datasets` directly using an URL. My goal is to write a generic code that abstracts the storage layer, such that my users will only have to pass in an `fsspec`-compliant URL and the corresponding files will be loaded.

### Your contribution

I managed to instantiate a `DVCFileSystem` pointing to a Gitlab repo from a `fsspec` chained URL in [this pull request](https://github.com/iterative/dvc/pull/9903) to DVC. 

```python
from fsspec.core import url_to_fs

fs, _ = url_to_fs("dvc::https://gitlab.com/repository/group/my-repo")
```

From now I'm not sure how to continue, it seems that `datasets` expects the URL to be fully qualified like so: `dvc::https://gitlab.com/repository/group/my-repo/my-folder/my-file.json` but this fails because `DVCFileSystem` expects the URL to point to the root of an SCM repo. Is there a way to make this work with `datasets`?
avoid downgrading jax version
State: closed
Labels: name: enhancement

### Feature request

Whenever I `pip install datasets[jax]` it downgrades jax to version 0.3.25. I seem to be able to install this library first then upgrade jax back to version 0.4.13.

### Motivation

It would be nice to not overwrite currently installed version of jax if possible.

### Your contribution

I would be willing to beta test. Or maybe write some code if I could get pointed in the right direction, I'm not super familiar with this codebase.
Fix to_json ValueError and remove pandas pin
State: closed

This PR fixes the root cause of the issue:
- #6197

This PR also removes the temporary pin of `pandas` introduced by:
- #6200


Note that for orient in ['records', 'values'], index value is ignored but
- in `pandas` < 2.1.0, a ValueError is raised if not index and orient not in ['split', 'table']
  - for orient = 'records', we need index = True
  - default index value is True
- in `pandas` = 2.1.0, a ValueError is raised if index is True and orient in ['records', 'values']
  - for orient = 'records', we need index = False or None
  - default index value is None

This PR fixes the issue by not passing index and thus using default index value (valid for all pandas versions), unless orient is 'split' or 'table' (where we pass index = False, as it was done before this fix).
Use load_dataset for local json files, but it not works
State: open

### Describe the bug

when I use load_dataset to load my local datasets，it always goes to Hugging Face to download the data instead of loading the local dataset.

### Steps to reproduce the bug

`raw_datasets = load_dataset(
        ‘json’,
        data_files=data_files)`


### Expected behavior

![image](https://github.com/huggingface/datasets/assets/50519434/add3747f-6481-4da7-b374-8f81c5a6472c)


### Environment info

python version 3.8.5
datasets version 2.12
os version unbuntu 18.04
Preserve split order in DataFilesDict
State: closed

After investigation, I have found that this copy forces the splits to be sorted alphabetically: https://github.com/huggingface/datasets/blob/029227a116c14720afca71b9b22e78eb2a1c09a6/src/datasets/builder.py#L556

This PR removes the alphabetically sort of `DataFilesDict` keys.
- Note that for a `dict`, the order of keys is relevant when hashing:
  ```python
  hash1 = Hasher.hash({'train': 'train.csv', 'test': 'test.csv'})
  hash2 = Hasher.hash({'test': 'test.csv', 'train': 'train.csv'})
  assert hash1 != hash2
  ```
- The `DataFilesDict` is a subclass of `dict`, thus the order should be relevant as well
  ```python
  hash1 = Hasher.hash(DataFilesDict({'train': 'train.csv', 'test': 'test.csv'}))
  hash2 = Hasher.hash(DataFilesDict({'test': 'test.csv', 'train': 'train.csv'}))
  assert hash1 != hash2
  ```

Fix #6196.