Minimartzz/github-issues huggingface dataset:
Duplicate `data_files` when named `<split>/<split>.parquet`
State: closed
Labels: description: Something isn't working
name: bug

e.g. with `u23429/stock_1_minute_ticker`

```ipython
In [1]: from datasets import *

In [2]: b = load_dataset_builder("u23429/stock_1_minute_ticker")
Downloading readme: 100%|██████████████████████████| 627/627 [00:00<00:00, 246kB/s]

In [3]: b.config.data_files
Out[3]: 
{NamedSplit('train'): ['hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/train/train.parquet',
  'hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/train/train.parquet'],
 NamedSplit('validation'): ['hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/validation/validation.parquet',
  'hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/validation/validation.parquet'],
 NamedSplit('test'): ['hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/test/test.parquet',
  'hf://datasets/u23429/stock_1_minute_ticker@65c973cf4ec061f01a363b40da4c1bb128ba4166/test/test.parquet']}

```

This bug issue is present in the current `datasets` 2.14.5 and also on `main` even after https://github.com/huggingface/datasets/pull/6244 cc @mariosasko 

Comments:
Also reported in https://github.com/huggingface/datasets/issues/6259

I think it's best to drop duplicates with a `set`  (as a temporary fix) and improve the patterns when/if https://github.com/fsspec/filesystem_spec/pull/1382 gets merged. @lhoestq Do you have some other ideas?

Alternatively we could just use this no ?

```python
if config.FSSPEC_VERSION < version.parse("2023.9.0"):
    KEYWORDS_IN_PATH_NAME_BASE_PATTERNS = [
        "{keyword}[{sep}/]**",
        "**[{sep}]{keyword}[{sep}/]**",
        "**/{keyword}[{sep}/]**",
    ]
else:
    KEYWORDS_IN_PATH_NAME_BASE_PATTERNS = [
        "{keyword}[{sep}/]**",
        "**/*[{sep}]{keyword}[{sep}/]**",
        "**/*/{keyword}[{sep}/]**",
    ]
```

This way no need to implement sets, which would require a bit of work since we've always considered a list of pattern to be resolved as the concatenated list of resolved files for each pattern (including duplicates)


Arf `"**/*/{keyword}[{sep}/]**"` does return `data/keyword.txt` in latest `fsspec` but not in `glob.glob`

EDIT: actually forgot to set `recursive=True`

Actually `glob.glob` does return it with `recursive=True` ! my bad

Pff just tested and my idea sucks, pattern 1 and 3 obviously give duplicates 

> I think it's best to drop duplicates with a set (as a temporary fix)

I started https://github.com/huggingface/datasets/pull/6278 to use DataFilesSet objects instead of DataFilesList
Overwriting Split overwrites data but not metadata, corrupting dataset
State: closed

### Describe the bug

I want to be able to overwrite/update/delete splits in my dataset. Currently the only way to do is to manually go into the dataset and delete the split. If I try to overwrite programmatically I end up in an error state and (somewhat) corrupting the dataset. Read below.

**Current Behavior**
When I push to an existing split I get this error:
`ValueError: Split complexRoofLocation_01Apr2023_to_31May2023test already present` 
This seems to suggest that the library doesn't support overwriting splits. 

**Potential Bug**
What’s strange is that datasets, despite the operation erroring out with the ValueError above, does, in fact, overwrite the split:

`Pushing dataset shards to the dataset hub: 100% [.....................] 1/1 [00:00<00:00, 55.04it/s]` 
Even though you got an error message and your code fails, your dataset is now changed. That seems like a bug. Either don't change the dataset, or don't throw the error and allow the script to proceed. 

Additional Bug
While it overwrites the split, it doesn’t overwrite the split’s information. Because of this when you pull down the dataset you may end up getting a `NonMatchingSplitsSizesError` if the size of the dataset during the overwrite is different. For example, my original split had 5 rows, but on my overwrite, I only had 4. Then when I try to download the dataset, I get a `NonMatchingSplitsSizesError` because the dataset's data.json states there’s 5 but only 4 exist in the split.

Expected Behavior
This corrupts the dataset rendering it unusable (until you take manual intervention). Either the library should let the overwrite happen (which it does but should also update the metadata) or it shouldn’t do anything.

### Steps to reproduce the bug

[Colab Notebook](https://colab.research.google.com/drive/1bqVkD06Ngs9MQNdSk_ygCG6y1UqXA4pC?usp=sharing)


### Expected behavior

The split should be overwritten and I should be able to use the new version of the dataset without issue. 

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- PyArrow version: 9.0.0
- Pandas version: 1.5.3


Comments:
Use LibYAML with PyYAML if available
State: open

PyYAML, the YAML framework used in this library, allows the use of LibYAML to accelerate the methods `load` and `dump`. To use it, a user would need to first install a PyYAML version that uses LibYAML (not available in PyPI; needs to be manually installed). Then, to actually use them, PyYAML suggests importing the LibYAML version of the `Loader` and `Dumper` and falling back to the default ones. This PR implements this change. See [PyYAML docs](https://pyyaml.org/wiki/PyYAMLDocumentation) for more info.

This change was motivated after trying to use any of [the SugarCREPE datasets in the Hub](https://huggingface.co/datasets?search=sugarcrepe) provided by [the org HuggingFaceM4](https://huggingface.co/datasets/HuggingFaceM4). Such datasets save a lot of information (~1MB) in the YAML metadata from the `README.md` file and I noticed this slowed down the data loading process. BTW, I also noticed cache files for it is also slow because it tries to hash an instance of `DatasetInfo`, which in turn has all this metadata.

Also, I changed two list comprehensions into generator expressions to avoid allocating extra memory unnecessarily.

And BTW, there's [an issue in PyYAML suggesting to make this automatic](https://github.com/yaml/pyyaml/issues/437).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6266). All of your documentation changes will be reflected on that endpoint.

On Ubuntu, if `libyaml-dev` is installed, you can install PyYAML 6.0.1 with LibYAML with the following command (as it's automatically detected):

```bash
pip install git+https://github.com/yaml/pyyaml.git@6.0.1
```

Are the failing tests flaky?

We use `huggingface_hub`'s RepoCard API instead of these modules to parse the YAML block (notice the deprecations), so the `huggingface_hub` repo is the right place to suggest these changes.

Personally, I'm not a fan of these changes, as a single non-standard usage of the `ClassLabel` type is not a sufficient reason to merge them. Also, the dataset in question stores data in a single Parquet file, with the features info embedded in its (schema) metadata, which means the YAML parsing can be skipped while preserving the features by directly loading the Parquet file:
```python
from datasets import load_dataset
ds = load_dataset("parquet", data_files="https://huggingface.co/datasets/HuggingFaceM4/SugarCrepe_swap_obj/resolve/main/data/test-00000-of-00001-ca2ae6017a2336d7.parquet")
```

PS: Yes, these tests are flaky. We are working on fixing them.

Oh, I didn't realize they were deprecated. Thanks for the tip on how to work around this issue!

For future reference, the places to change the code in `huggingface_hub` would be:

https://github.com/huggingface/huggingface_hub/blob/89cc69105074f1d071e0471144605f3cdfe1dab3/src/huggingface_hub/repocard.py#L506

https://github.com/huggingface/huggingface_hub/blob/89cc69105074f1d071e0471144605f3cdfe1dab3/src/huggingface_hub/utils/_fixes.py#L34
CI is broken: ImportError: cannot import name 'context' from 'tensorflow.python'
State: closed
Labels: description: Something isn't working
name: bug

Python 3.10 CI is broken for `test_py310`.

See: https://github.com/huggingface/datasets/actions/runs/6322990957/job/17169678812?pr=6262

```
FAILED tests/test_py_utils.py::TempSeedTest::test_tensorflow - ImportError: cannot import name 'context' from 'tensorflow.python' (/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/tensorflow/python/__init__.py)
```

```
_________________________ TempSeedTest.test_tensorflow _________________________
[gw1] linux -- Python 3.10.13 /opt/hostedtoolcache/Python/3.10.13/x64/bin/python

self = <tests.test_py_utils.TempSeedTest testMethod=test_tensorflow>

    @require_tf
    def test_tensorflow(self):
        import tensorflow as tf
        from tensorflow.keras import layers
    
        model = layers.Dense(2)
    
        def gen_random_output():
            x = tf.random.uniform((1, 3))
            return model(x).numpy()
    
>       with temp_seed(42, set_tensorflow=True):

tests/test_py_utils.py:155: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/contextlib.py:135: in __enter__
    return next(self.gen)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

seed = 42, set_pytorch = False, set_tensorflow = True

    @contextmanager
    def temp_seed(seed: int, set_pytorch=False, set_tensorflow=False):
        """Temporarily set the random seed. This works for python numpy, pytorch and tensorflow."""
        np_state = np.random.get_state()
        np.random.seed(seed)
    
        if set_pytorch and config.TORCH_AVAILABLE:
            import torch
    
            torch_state = torch.random.get_rng_state()
            torch.random.manual_seed(seed)
    
            if torch.cuda.is_available():
                torch_cuda_states = torch.cuda.get_rng_state_all()
                torch.cuda.manual_seed_all(seed)
    
        if set_tensorflow and config.TF_AVAILABLE:
            import tensorflow as tf
>           from tensorflow.python import context as tfpycontext
E           ImportError: cannot import name 'context' from 'tensorflow.python' (/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/tensorflow/python/__init__.py)

/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/datasets/utils/py_utils.py:257: ImportError
```

Comments:
REUSE_DATASET_IF_EXISTS don't work 
State: closed

### Describe the bug

I use the following code  to download natural_question dataset. Even though I have completely download it, the next time I run this code, the new download procedure will start and cover the original /data/lxy/NQ

config=datasets.DownloadConfig(resume_download=True,max_retries=100,cache_dir=r'/data/lxy/NQ',download_desc='NQ')

data=datasets.load_dataset('natural_questions',cache_dir=r'/data/lxy/NQ',download_config=config,download_mode=DownloadMode.REUSE_DATASET_IF_EXISTS)

---
Since I don't have apache_beam installed, it throw a exception. After I pip install apache_beam ,the download restart..

![image](https://github.com/huggingface/datasets/assets/88258534/f28ce7fe-29ea-4348-b87f-e69182a8bd41)


### Steps to reproduce the bug

run this two line code

config=datasets.DownloadConfig(resume_download=True,max_retries=100,cache_dir=r'/data/lxy/NQ',download_desc='NQ')

data=datasets.load_dataset('natural_questions',cache_dir=r'/data/lxy/NQ',download_config=config,download_mode=DownloadMode.REUSE_DATASET_IF_EXISTS)

### Expected behavior

Download behavior can be correctly follow DownloadMode

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-3.10.0-1160.88.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.3

Comments:
Hi! Unfortunately, the current behavior is to delete the downloaded data when this error happens. So, I've opened a PR that removes the problematic import to avoid losing data due to `apache_beam` not being installed (we host the preprocessed version of `natual_questions` on the HF GCS, so requiring `apache_beam` in that case doesn't make sense)

Thanks for your reply. I met another question that I set `export HF_DATASETS_CACHE=/data/lxy/.cache` , but each time I run load_datasets, the datasets module still looking for NQ in the wrong default cache dir '/home/lxy/.cache' 。How to avoid this incorrect behavior. I am sure HF_DATASETS_CACHE was set correctly since I use echo & to check it.
![image](https://github.com/huggingface/datasets/assets/88258534/e7029f27-b9f9-496c-8948-6234ef695646)
by the way I delete the file in '/home/lxy/.cache' since I found there has some kb size file seems useless.

You need to set this variable before the `datasets` import. Then, you can use `import datasets; datasets.config.HF_DATASETS_CACHE` to verify the cache location.
Duplicated Rows When Loading Parquet Files from Root Directory with Subdirectories
State: closed

### Describe the bug

When parquet files are saved in "train" and "val" subdirectories under a root directory, and datasets are then loaded using `load_dataset("parquet", data_dir="root_directory")`, the resulting dataset has duplicated rows for both the training and validation sets.

### Steps to reproduce the bug

1. Create a root directory, e.g., "testing123".
2. Under "testing123", create two subdirectories: "train" and "val".
3. Create and save a parquet file with 3 unique rows in the "train" subdirectory.
4. Create and save a parquet file with 4 unique rows in the "val" subdirectory.
5. Load the datasets from the root directory using `load_dataset("parquet", data_dir="testing123")`
6. Iterate through the datasets and print the rows

Here's a collab reproducing these steps:

https://colab.research.google.com/drive/11NEdImnQ3OqJlwKSHRMhr7jCBesNdLY4?usp=sharing

### Expected behavior

- Training set should contain 3 unique rows.
- Validation set should contain 4 unique rows.

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.2
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

Comments:
Thanks for reporting this issue! We should be able to avoid this by making our `glob` patterns more precise. In the meantime, you can load the dataset by directly assigning splits to the data files: 
```python
from datasets import load_dataset
ds = load_dataset("parquet", data_files={"train": "testing123/train/output_train.parquet", "validation": "testing123/val/output_val.parquet"})
```
HfHubHTTPError - exceeded our hourly quotas for action: commit
State: closed

### Describe the bug

I try to upload a very large dataset of images, and get the following error:
```
File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/hf_api.py:2712, in HfApi.create_commit(self, repo_id, operations, commit_message, commit_description, token, repo_type, revision, create_pr, num_threads, parent_commit, run_as_future)
   2710 try:
   2711     commit_resp = get_session().post(url=commit_url, headers=headers, data=data, params=params)
-> 2712     hf_raise_for_status(commit_resp, endpoint_name="commit")
   2713 except RepositoryNotFoundError as e:
   2714     e.append_to_message(_CREATE_COMMIT_NO_REPO_ERROR_MESSAGE)

File /fsx-multigen/yuvalkirstain/miniconda/envs/pickapic/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py:301, in hf_raise_for_status(response, endpoint_name)
    297     raise BadRequestError(message, response=response) from e
    299 # Convert `HTTPError` into a `HfHubHTTPError` to display request information
    300 # as well (request id and/or server error message)
--> 301 raise HfHubHTTPError(str(e), response=response) from e

HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/yuvalkirstain/pickapic_v2/commit/main (Request ID: Root=1-65112399-12d63f7d7f28bfa40a36a0fd)

You have exceeded our hourly quotas for action: commit. We invite you to retry later.
```

this makes it much less convenient to host large datasets on HF hub.

### Steps to reproduce the bug

Upload a very large dataset of images

### Expected behavior

the upload to work well

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.0-1033-aws-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 1.5.3

Comments:
how is your dataset structured? (file types, how many commits and files are you trying to push, etc)

I succeeded in uploading it after several attempts with an hour gap between each attempt (inconvenient but worked). The final dataset is [here](https://huggingface.co/datasets/yuvalkirstain/pickapic_v2), code and context to the dataset can be found [here](https://github.com/yuvalkirstain/PickScore/).
I can close the issue if this behavior is intended, as most users probably do not need to upload large-scale datasets.

We could fix this by creating a single commit for all the (Parquet) shards in `push_to_hub` instead of one commit per shard, as we currently do. 

@Wauplin Any updates on the 2-step commit process suggested by you that we need to implement this?

> Any updates on the 2-step commit process suggested by you that we need to implement this?

Re-prioritizing this, sorry. Will let you know but probably can be done this week.
load_dataset() function's cache_dir does not seems to work
State: closed

### Describe the bug

datasets version: 2.14.5

when trying to run the following command
trec = load_dataset('trec', split='train[:1000]', cache_dir='/path/to/my/dir')

I keep getting error saying the command does not have permission to the default cache directory on my macbook pro machine. 

It seems the cache_dir parameter cannot change the dataset saving directory from the default 

what ever explained in the https://huggingface.co/docs/datasets/cache does not seem to work


### Steps to reproduce the bug

datasets version: 2.14.5

when trying to run the following command
trec = load_dataset('trec', split='train[:1000]', cache_dir='/path/to/my/dir')

I keep getting error saying the command does not have permission to the default cache directory on my macbook pro machine. 

It seems the cache_dir parameter cannot change the dataset saving directory from the default 

what ever explained in the https://huggingface.co/docs/datasets/cache does not seem to work

### Expected behavior

the dataset should be saved to the cache_dir points to 

### Environment info

datasets version: 2.14.5
macos X: Ventura 13.4.1 (c)

Comments:
Can you share the error message?

Also, it would help if you could check whether `huggingface_hub`'s download behaves the same:
```python
from huggingface_hub import snapshot_download
snapshot_download("trec", repo_type="dataset", cache_dir='/path/to/my/dir)
```

In the next major release, we aim to switch to `huggingface_hub` for file download/caching, but we could align the `cache_dir`'s `umask` behavior earlier than this if their solution works for your use case.

@mariosasko, Hey , is there any update on this? 

Yes, I am still facing the same issue.

```python
from huggingface_hub import snapshot_download
snapshot_download("trec", repo_type="dataset", cache_dir='/path/to/my/dir)
```

But this is working for me.

Thanks.

Could you clarify a bit? What if I wanted only 100 samples from train split, how would i change the above? Thanks

I have exactly same issue

@lhoestq Can this be open again? Reading #7200 this doesn't even address not being able to specify `cache_dir`, it just updates the documentation? Their own comment states this:

> In my testing, I could not specify the cache directory even by load_dataset("dataset_name" cache_dir="..."). It might be another issue. I also welcome any advice to solve this issue.

Nothing else in the PR seems to indicate that this is actually resolved.

I've noticed the same issue. Looks like #7499 was fixing it, but from what I see in the comments this setting is for different caches?

Hi ! you can find the information about the `datasets` and `huggingface_hub` cache directories here: https://huggingface.co/docs/datasets/v3.6.0/en/cache#cache-directory
Dataset.from_generator() cost much more time in vscode debugging mode then running mode
State: closed

### Describe the bug

Hey there,
I’m using Dataset.from_generator() to convert a torch_dataset to the Huggingface Dataset.
However, when I debug my code on vscode, I find that it runs really slow on Dataset.from_generator() which may even 20 times longer then run the script on terminal.

### Steps to reproduce the bug

I write a simple test code :

```python
import os
from functools import partial
from typing import Callable

import torch
import time
from torch.utils.data import Dataset as TorchDataset

from datasets import load_from_disk, Dataset as HFDataset
  
import torch  
from torch.utils.data import Dataset  
  
class SimpleDataset(Dataset):  
    def __init__(self, data):  
        self.data = data  
        self.keys = list(data[0].keys())
      
    def __len__(self):  
        return len(self.data)  
      
    def __getitem__(self, index):  
        sample = self.data[index]  
        return {key: sample[key] for key in self.keys}  
  

def TorchDataset2HuggingfaceDataset(torch_dataset: TorchDataset, cache_dir: str = None
) -> HFDataset:
    
    """
        convert torch dataset to huggingface dataset
    """
    generator : Callable[[], TorchDataset] = lambda: (sample for sample in torch_dataset)   

    return HFDataset.from_generator(generator, cache_dir=cache_dir)

if __name__ == '__main__':
    data = [  
        {'id': 1, 'name': 'Alice'},  
        {'id': 2, 'name': 'Bob'},  
        {'id': 3, 'name': 'Charlie'}  
    ]
    
    torch_dataset = SimpleDataset(data)
    start_time = time.time() 
    huggingface_dataset = TorchDataset2HuggingfaceDataset(torch_dataset)
    end_time = time.time()
    print("time: ", end_time - start_time)
    print(huggingface_dataset)
```

### Expected behavior

this test on my machine report that the running time on terminal is 0.086,
however the running time in debugging mode on vscode is 0.25, which I think is much longer than expected.

I’d like to know is the anything wrong in the code or just because of debugging?
I have traced the code and I find is this func which I get stuck.

```python
def create_config_id(
        self,
        config_kwargs: dict,
        custom_features: Optional[Features] = None,
    ) -> str:
...
# stuck in this line
suffix = Hasher.hash(config_kwargs_to_add_to_suffix)
```



### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.11.0-27-generic-x86_64-with-glibc2.31
- Python version: 3.11.3
- Huggingface_hub version: 0.17.2
- PyArrow version: 11.0.0
- Pandas version: 2.0.1

Comments:
Answered on the forum: https://discuss.huggingface.co/t/dataset-from-generator-cost-much-more-time-in-vscode-debugging-mode-then-running-mode/56005/2