liuqingwen/github-issues huggingface dataset:
Allow setting the environment variable for the lock file path
Labels: name: enhancement
description: New feature or request

### Feature request

Add an environment variable to replace the default lock file path.

### Motivation

Usually, dataset path is a read-only path while the lock file needs to be modified each time. It would be convenient if the path can be reset individually.

### Your contribution

```/src/datasets/utils/filelock.py
class UnixFileLock(BaseFileLock):
    def __init__(self, lock_file, timeout=-1, max_filename_length=None):
        #-------------------
        if os.getenv('DS_TMP_PATH'):
            file_name = str(lock_file).split('/')[-1]
            dataset_tmp_path = os.getenv('DS_TMP_PATH')
            lock_file = os.path.join(dataset_tmp_path, file_name)
        #-------------------
        max_filename_length = os.statvfs(os.path.dirname(lock_file)).f_namemax
        super().__init__(lock_file, timeout=timeout, max_filename_length=max_filename_length)
```
A simple demo is as upper. Thanks.

Comments:
data configuration hash suffix depends on uncanonicalized data_dir
Labels: name: good first issue
description: Good for newcomers

### Describe the bug

I am working with the `recipe_nlg` dataset, which requires manual download. Once it's downloaded, I've noticed that the hash in the custom data configuration is different if I add a trailing `/` to my `data_dir`. It took me a while to notice that the hashes were different, and to understand that that was the cause of my dataset being processed anew instead of the cached version being used.

### Steps to reproduce the bug

1. Follow the steps to manually download the `recipe_nlg` dataset to `/data/recipenlg`.
2. Load it using `load_dataset`, once without a trailing slash and once with one:

    ```python
    >>> ds = load_dataset("recipe_nlg", data_dir="/data/recipenlg")
    Using custom data configuration default-082278caeea85765
    Downloading and preparing dataset recipe_nlg/default to /home/kyle/.cache/huggingface/datasets/recipe_nlg/default-082278caeea85765/1.0.0/aa4f120223637bedf7360cecb70a9bd108acfd64e38207ca90c9f385d21e5e74...
    Dataset recipe_nlg downloaded and prepared to /home/kyle/.cache/huggingface/datasets/recipe_nlg/default-082278caeea85765/1.0.0/aa4f120223637bedf7360cecb70a9bd108acfd64e38207ca90c9f385d21e5e74. Subsequent calls will reuse this data.
    100%|███████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.10s/it]
    DatasetDict({
        train: Dataset({
            features: ['id', 'title', 'ingredients', 'directions', 'link', 'source', 'ner'],
            num_rows: 2231142
        })
    })
    >>> ds = load_dataset("recipe_nlg", data_dir="/data/recipenlg/")
    Using custom data configuration default-83e87680785d0493
    Downloading and preparing dataset recipe_nlg/default to /home/user/.cache/huggingface/datasets/recipe_nlg/default-83e87680785d0493/1.0.0/aa4f120223637bedf7360cecb70a9bd108acfd64e38207ca90c9f385d21e5e74...
    Generating train split:   1%|          | 12701/2231142 [00:04<13:15, 2790.25 examples/s
    ^C
    ```

3. Observe that the hash suffix in the custom data configuration changes due to the altered string.

### Expected behavior

I think I would expect the hash to remain constant if it actually points to the same location on disk. I would expect the use of `os.path.normpath` to canonicalize the paths.

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-5.4.0-147-generic-x86_64-with-glibc2.31
- Python version: 3.10.8
- PyArrow version: 10.0.1
- Pandas version: 1.5.2

Comments:
It could even use `os.path.realpath` to resolve symlinks.

Indeed, it makes sense to normalize `data_dir`. Feel free to submit a PR (this can be "fixed" [here](https://github.com/huggingface/datasets/blob/89f775226321ba94e5bf4670a323c0fb44f5f65c/src/datasets/builder.py#L173))

#self-assign
Is it possible to change a cached file and 're-cache' it instead of re-generating?
Labels: name: enhancement
description: New feature or request

### Feature request

Hi,
I have a huge cached file using `map`(over 500GB), and I want to change an attribution of each element, is there possible to do it using some method instead of re-generating, because `map` takes over 24 hours

### Motivation

For large datasets, I think it is very important because we always face the problem which is changing something in the original cache without re-generating it.

### Your contribution

For now, I can't help, sorry.

Comments:
Arrow files/primitives (tables and arrays) are immutable, so re-generating them is the only option, I'm afraid.

> 

Got it, thanks for your reply
Deprecate task api

The task API is not well adopted in the ecosystem, so this PR deprecates it. The `train_eval_index` is a newer, more flexible solution that should be used instead (I think?).

These are the projects that still use the task API :
* the image classification example in Transformers: [here](https://github.com/huggingface/transformers/blob/8f76dc8e5aaad58f2df7748b6d6970376f315a9a/examples/pytorch/image-classification/run_image_classification_no_trainer.py#L262) and [here](https://github.com/huggingface/transformers/blob/8f76dc8e5aaad58f2df7748b6d6970376f315a9a/examples/tensorflow/image-classification/run_image_classification.py#L277)
* autotrain: [here](https://github.com/huggingface/autotrain-backend/blob/455e274004b56f9377d64db4ab03671508fcc4cd/zeus/zeus/run/utils.py#L666)
* api-inference-community: [here](https://github.com/huggingface/api-inference-community/blob/fb8fb29d577a5bf01c82944db745489a6d6ed3d4/manage.py#L64) (but the rest of the code does not call the `resolve_dataset` function)

So we need to update these files after the merge.

cc @lewtun 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5865). All of your documentation changes will be reflected on that endpoint.

If it's easy to keep supporting it we can keep it no ? There are many datasets on the hub that implement the tasks templates in dataset scripts and it's maybe easier to keep task templates than opening PRs to those datasets.

do we know if people use the tasks api?

edit: i mean, i'm fine with removing it if it's not used much, especially considering that it's not documented well.
IndexError: list index out of range with data hosted on Zenodo
Labels: name: bug
description: Something isn't working

The dataset viewer sometimes raises an `IndexError`:
```
IndexError: list index out of range
```
See:
- huggingface/datasets-server#1151
  - https://huggingface.co/datasets/reddit/discussions/5
- huggingface/datasets-server#1118
  - https://huggingface.co/datasets/krr-oxford/OntoLAMA/discussions/1
- https://huggingface.co/datasets/hyperpartisan_news_detection/discussions/3
- https://huggingface.co/datasets/um005/discussions/2
- https://huggingface.co/datasets/tapaco/discussions/2
- https://huggingface.co/datasets/common_language/discussions/3
- https://huggingface.co/datasets/pass/discussions/1

After investigation:
- This happens with data files hosted on Zenodo
- Indeed, there is an underlying 429 HTTP error: Too Many Requests

Note that some time ago, it also happened with data files hosted on Google Drive. See:
  - #4581
  - #4580 

The reason then was that there was a 403 HTTP error: Forbidden



Comments:
This error is also raised when data is hosted on Google Drive:
- https://huggingface.co/datasets/docred/discussions/5
- https://huggingface.co/datasets/linnaeus/discussions/3
- https://huggingface.co/datasets/poleval2019_mt/discussions/3
- https://huggingface.co/datasets/reddit_tifu/discussions/2
- https://huggingface.co/datasets/species_800/discussions/3
- https://huggingface.co/datasets/wiki_lingua/discussions/1
- https://huggingface.co/datasets/yoruba_text_c3/discussions/1
Adding chemistry dataset/models in huggingface
Labels: name: enhancement
description: New feature or request

### Feature request

Huggingface is really amazing platform for open science.

In addition to computer vision, video and NLP, would it be of interest to add chemistry/materials science dataset/models in Huggingface? Or, if its already done, can you provide some pointers.

We have been working on a comprehensive benchmark on this topic: [JARVIS-Leaderboard](https://pages.nist.gov/jarvis_leaderboard/) and I am wondering if we could contribute/integrate this project as a part of huggingface. 

### Motivation

Similar to the main stream AI field, there is need of large scale benchmarks/models/infrastructure for chemistry/materials data.

### Your contribution

We can start adding datasets as our [benchmarks](https://github.com/usnistgov/jarvis_leaderboard/tree/main/jarvis_leaderboard/benchmarks) should be easily convertible to the dataset format.

Comments:
Hi! 

This would be a nice addition to the Hub! You can find the existing chemistry datasets/models on the Hub (using the `chemistry` tag) [here](https://huggingface.co/search/full-text?q=chemistry&type=model&type=dataset).

Feel free to ping us here on the Hub if you need help adding the datasets.
Can not load audiofolder dataset on kaggle

### Describe the bug
It's crash log:
FileNotFoundError: Couldn't find a dataset script at /kaggle/working/audiofolder/audiofolder.py or any data file in the same directory. Couldn't find 'audiofolder' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/audiofolder/audiofolder.py


### Steps to reproduce the bug

![image](https://github.com/huggingface/datasets/assets/93691919/a2829d27-d15c-4acc-86fb-d1987c760468)
common_voice = load_dataset("audiofolder", data_dir="/kaggle/working/data")

### Expected behavior

load dataset without error. It works ok on colab, but on kaggle it happends.

### Environment info

- `datasets` version: 2.1.0
- Platform: Linux-5.15.109+-x86_64-with-glibc2.31
- Python version: 3.10.10
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

Comments:
Hi! `audiofolder` requires `datasets>=2.5.0`, so please update the `datasets`' installation (`pip install -U datasets`) in the environment to resolve the issue.

> Hi! `audiofolder` requires `datasets>=2.5.0`, so please update the `datasets`' installation (`pip install -U datasets`) in the environment to resolve the issue.

I don't think it is a problem of the version. It runs ok on colab or local machine. Only on kaggle will has this bug.
CSV datasets should only read the CSV data files in the repo
Labels: name: bug
description: Something isn't working

When a no-script dataset has many CSV files and a JPG file, the library infers to use the Csv builder, but tries to read as CSV all files in the repo, also the JPG file.

I think the Csv builder should filter out non-CSV files when reading.

An analogue solution should be implemented for other packaged builders.

Related to:
- https://huggingface.co/datasets/abidlabs/img2text/discussions/1
- https://github.com/gradio-app/gradio/pull/3973#issuecomment-1545409061

CC: @abidlabs @severo 

Comments:
load_dataset('bigcode/the-stack-dedup', streaming=True) very slow!

### Describe the bug

Running

```
import datasets
ds = datasets.load_dataset('bigcode/the-stack-dedup', streaming=True)
```

takes about 2.5 minutes! 

I would expect this to be near instantaneous. With other datasets, the runtime is one or two seconds.


### Environment info

- `datasets` version: 2.11.0
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.10.10
- Huggingface_hub version: 0.13.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.0

Comments:
This is due to the slow resolution of the data files: https://github.com/huggingface/datasets/issues/5537.

We plan to switch to `huggingface_hub`'s `HfFileSystem` soon to make the resolution faster (will be up to 20x faster once we merge https://github.com/huggingface/huggingface_hub/pull/1443)



You're right, when I try to parse more than 50GB of text data, I also get very slow, usually taking hours or even tens of hours.

> You're right, when I try to parse more than 50GB of text data, I also get very slow, usually taking hours or even tens of hours.

That's unrelated to the problem discussed in this issue. 

> > You're right, when I try to parse more than 50GB of text data, I also get very slow, usually taking hours or even tens of hours.
> 
> That's unrelated to the problem discussed in this issue.

Sorry, I misunderstood it.
Make models/functions optimized with `torch.compile` hashable
Labels: name: enhancement
description: New feature or request

As reported in https://github.com/huggingface/datasets/issues/5819, hashing functions/transforms that reference a model, or a function, optimized with `torch.compile` currently fails due to them not being picklable (the concrete error can be found in the linked issue).

The solutions to consider:
1. hashing/pickling the original, uncompiled version of a compiled model/function (attributes `_orig_mod`/`_torchdynamo_orig_callable`) (less precise than the 2nd option as it ignores the other params of `torch.compute`)
2. wait for https://github.com/pytorch/pytorch/issues/101107 to be resolved


Comments:
load json dataset interrupt when dtype cast problem occured

### Describe the bug

i have a json like this:
[
    {"id": 1, "name": 1},
    {"id": 2, "name": "Nan"},
    {"id": 3, "name": 3}, 
    ....
]
，which have several problematic rows data like row 2, then i load it with datasets.load_dataset('json', data_files=['xx.json'], split='train'), it will report like this:
Generating train split: 0 examples [00:00, ? examples/s]Failed to read file 'C:\Users\gawinjunwu\Downloads\test\data\a.json' with error <class 'pyarrow.lib.ArrowInvalid'>: Could not convert '2' with type str: tried to convert to int64
Traceback (most recent call last):
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1858, in _prepare_split_single
    for _, table in generator:
  File "D:\Python3.9\lib\site-packages\datasets\packaged_modules\json\json.py", line 146, in _generate_tables
    raise ValueError(f"Not able to read records in the JSON file at {file}.") from None
ValueError: Not able to read records in the JSON file at C:\Users\gawinjunwu\Downloads\test\data\a.json.     

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\gawinjunwu\Downloads\test\scripts\a.py", line 4, in <module>
    ds = load_dataset('json', data_dir='data', split='train')
  File "D:\Python3.9\lib\site-packages\datasets\load.py", line 1797, in load_dataset
    builder_instance.download_and_prepare(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 890, in download_and_prepare
    self._download_and_prepare(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 985, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1746, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "D:\Python3.9\lib\site-packages\datasets\builder.py", line 1891, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset.
Could datasets skip those problematic data row? 

### Steps to reproduce the bug

prepare a json file like this:
[
    {"id": 1, "name": 1},
    {"id": 2, "name": "Nan"},
    {"id": 3, "name": 3}
]
then use datasets.load_dataset('json', dir_files=['xxx.json']) to load the json file

### Expected behavior

skip the problematic data row and load row1 and row3

### Environment info

python3.9

Comments:
Indeed the JSON dataset builder raises an error when it encounters an unexpected type.

There's an old PR open to add away to ignore such elements though, if it can help: https://github.com/huggingface/datasets/pull/2838
[2.12.0] DatasetDict.save_to_disk not saving to S3

### Describe the bug

When trying to save a `DatasetDict` to a private S3 bucket using `save_to_disk`, the artifacts are instead saved locally, and not in the S3 bucket.

I have tried using the deprecated `fs` as well as the `storage_options` arguments and I get the same results.

### Steps to reproduce the bug

1. Create a DatsetDict `dataset`
2. Create a S3FileSystem object
`s3 = datasets.filesystems.S3FileSystem(key=aws_access_key_id, secret=aws_secret_access_key)`
3. Save using `dataset_dict.save_to_disk(f"{s3_bucket}/{s3_dir}/{dataset_name}", storage_options=s3.storage_options)` or `dataset_dict.save_to_disk(f"{s3_bucket}/{s3_dir}/{dataset_name}", fs=s3)`
4. Check the corresponding S3 bucket and verify nothing has been uploaded
5. Check the path at f"{s3_bucket}/{s3_dir}/{dataset_name}" and verify that files have been saved there

### Expected behavior

Artifacts are uploaded at the f"{s3_bucket}/{s3_dir}/{dataset_name}" S3 location.

### Environment info

- `datasets` version: 2.12.0
- Platform: macOS-13.3.1-x86_64-i386-64bit
- Python version: 3.11.2
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1

Comments:
Hi ! Can you try adding the `s3://` prefix ?
```python
f"s3://{s3_bucket}/{s3_dir}/{dataset_name}"
```

Ugh, yeah that was it. Thank you!
Incomplete docstring for `BuilderConfig`
Labels: name: good first issue
description: Good for newcomers

Hi guys !
I stumbled upon this docstring while working on a project.
Some of the attributes have missing descriptions.

https://github.com/huggingface/datasets/blob/bc5fef5b6d91f009e4101684adcb374df2c170f6/src/datasets/builder.py#L104-L117

Comments:
Thanks for reporting! You are more than welcome to improve `BuilderConfig`'s docstring.

This class serves an identical purpose as `tensorflow_datasets`'s `BuilderConfig`, and its docstring is [here](https://github.com/tensorflow/datasets/blob/a95e38b5bb018312c3d3720619c2a8ef83ebf57f/tensorflow_datasets/core/dataset_builder.py#L81), so feel free to re-use parts of it.
Ability to update a dataset
Labels: name: enhancement
description: New feature or request

### Feature request

The ability to load a dataset, add or change something, and save it back to disk.

Maybe it's possible, but I can't work out how to do it, e.g. this fails:

```py
import datasets

dataset = datasets.load_from_disk("data/test1")
dataset = dataset.add_item({"text": "A new item"})
dataset.save_to_disk("data/test1")
```

With the error:
```
PermissionError: Tried to overwrite /mnt/c/Users/david/py/learning/mini_projects/data_sorting_and_filtering/data/test1 but a dataset can't overwrite itself.
```


### Motivation

My use case is that I want to process a dataset in a particular way but it doesn't fit in memory if I do it in one go. So I want to perform a loop and at each step in the loop, process one shard and append it to an ever-growing dataset. The code in the loop will load a dataset, add some rows, then save it again.

Maybe I'm just thinking about things incorrectly and there's a better approach. FWIW I can't use `dataset.map()` to do the task because that doesn't work with `num_proc` when adding rows, so is confined to a single process which is too slow.

The only other way I can think of is to create a new file each time, but surely that's not how people do this sort of thing.

### Your contribution

na

Comments:
This [reply](https://discuss.huggingface.co/t/how-do-i-add-things-rows-to-an-already-saved-dataset/27423) from @mariosasko on the forums may be useful :)

In this case, I think we can avoid the `PermissionError` by unpacking the underlying `ConcatenationTable` and saving only the newly added data blocks (in new files).

Thanks @stevhliu and @mariosasko , so saving to individual files then loading them later, concatenating again and saving again is the recommended way. Good to know.

Question that I hope doesn't sound rude: is this sort of thing (processing a dataset that doesn't fit in memory) outside of `datasets`'s core area of focus? Are there other tools you would recommend to do this sort of thing that play nice with `datasets`? Or is it just that I've found myself in a niche situation that hasn't specifically been catered for?
Easy way to create a Kaggle dataset from a Huggingface dataset?

I'm not sure whether this is more appropriately addressed with HuggingFace or Kaggle.  I would like to somehow directly create a Kaggle dataset from a HuggingFace Dataset.

While Kaggle does provide the option to create a dataset from a URI, that URI must point to a single file.  For example:

![image](https://user-images.githubusercontent.com/5355286/235792394-7c559d07-4aff-45b7-ad2b-9c5280c88415.png)


Is there some mechanism from huggingface to represent a dataset (such as that from `load_dataset('wmt14', 'de-en', split='train')` as a single file?  Or, some other way to get that into a Kaggle dataset so that I can use the huggingface `datasets` module to process and consume it inside of a Kaggle notebook?

Thanks in advance!


Comments:
Hi @hrbigelow , I'm no expert for such a question so I'll ping @lhoestq from the `datasets` library (also this issue could be moved there if someone with permission can do it :) )

Hi ! Many datasets are made of several files, and how they are parsed often requires a python script. Because of that, datasets like wmt14 are not available as a single file on HF. Though you can create this file using `datasets`:

```python
from datasets import load_dataset

ds = load_dataset("wmt14", "de-en", split="train")

ds.to_json("wmt14-train.json")
# OR to parquet, which is compressed:
# ds.to_parquet("wmt14-train.parquet")
```

We are also working on providing parquet exports for all datasets, but wmt14 is not supported yet (we're rolling it out for datasets <1GB first). They're usually available in the `refs/convert/parquet` branch (empty for wmt14):

<img width="267" alt="image" src="https://user-images.githubusercontent.com/42851186/235878909-7339f5a4-be19-4ada-85d8-8a50d23acf35.png">


also cc @nateraw for visibility on this (and cc @osanseviero too)
Cannot shuffle interleaved IterableDataset with "all_exhausted" stopping strategy
Labels: name: bug
description: Something isn't working

name: streaming

### Describe the bug

Shuffling interleaved `IterableDataset` with "all_exhausted" strategy yields non-exhaustive sampling.

### Steps to reproduce the bug

```py
from datasets import IterableDataset, interleave_datasets

def gen(bias, length):
  for i in range(length):
    yield dict(a=bias+i)

seed = 42
probabilities = [0.2, 0.6, 0.2]
d1 = IterableDataset.from_generator(lambda: gen(0, 3))
d2 = IterableDataset.from_generator(lambda: gen(10, 4))
d3 = IterableDataset.from_generator(lambda: gen(20, 3))
ds = interleave_datasets([d1, d2, d3], probabilities=probabilities, seed=seed, stopping_strategy='all_exhausted')
ds = ds.shuffle(buffer_size=1000)
for x in ds:
  print(x)
```

This code produces 
```
{'a': 0}
{'a': 22}
{'a': 20}
{'a': 21}
{'a': 10}
{'a': 1}
```

### Expected behavior

It should produce a longer list of examples to exhaust all the datasets.
If you comment out the shuffle line, it will exhaust all the datasets properly.
Here is the output if you comment out shuffling:
```
{'a': 10}
{'a': 11}
{'a': 20}
{'a': 12}
{'a': 0}
{'a': 21}
{'a': 13}
{'a': 10}
{'a': 1}
{'a': 11}
{'a': 12}
{'a': 22}
{'a': 13}
{'a': 20}
{'a': 10}
{'a': 11}
{'a': 12}
{'a': 2}
```

### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

This was run on Google Colab.

Comments: