Jhonatan321/Datasets_Jhona huggingface dataset:
Datasets Unknown Keyword Argument Error - task_templates

### Describe the bug

Issue

```python
from datasets import load_dataset
examples = load_dataset('facebook/winoground', use_auth_token=<YOUR USER ACCESS TOKEN>)
```

Gives error

```
TypeError: DatasetInfo.__init__() got an unexpected keyword argument 'task_templates'
```

A simple downgrade to lower `datasets v 2.21.0` solves it.




### Steps to reproduce the bug

1. `pip install datsets`
2.
```python
from datasets import load_dataset
examples = load_dataset('facebook/winoground', use_auth_token=<YOUR USER ACCESS TOKEN>)
```



### Expected behavior

Should load the dataset correctly.

### Environment info

- Datasets version  `3.0.0`
- `transformers` version: 4.45.0.dev0
- Platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.35
- Python version: 3.12.4
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.5
- Accelerate version: 0.35.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes


Thanks, for reporting.

We have been fixing most Hub datasets to remove the deprecated (and now non-supported) task templates, but we missed the "facebook/winoground".

It is fixed now: https://huggingface.co/datasets/facebook/winoground/discussions/8



Hello @albertvillanova 

I got the same error while loading this dataset: https://huggingface.co/datasets/alaleye/aloresb...

How can I fix it ? 
Thanks
Cannot use cached dataset without Internet connection (or when servers are down)

### Describe the bug

I want to be able to use cached dataset from HuggingFace even when I have no Internet connection (or when HuggingFace servers are down, or my company has network issues).  
The problem why I can't use it:
`data_files` argument from `datasets.load_dataset()` function get it updates from the server before calculating hash for caching. As a result, when I run the same code with and without Internet I get different dataset configuration directory name.

### Steps to reproduce the bug

```
import datasets

c4_dataset = datasets.load_dataset(
    path="allenai/c4",
    data_files={"train": "en/c4-train.00000-of-01024.json.gz"},
    split="train",
    cache_dir="/datesets/cache",
    download_mode="reuse_cache_if_exists",
    token=False,
)
```
1. Run this code with the Internet.
2. Run the same code without the Internet.

### Expected behavior

When running without the Internet connection, the loader should be able to get dataset from cache

### Environment info

- `datasets` version: 2.19.0
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.10.13
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.12.2

There are 2 workarounds, tho:
1. Download datasets from web and just load them locally
2. Use metadata directly (temporal solution, since metadata can change)
```
import datasets
from datasets.data_files import DataFilesDict, DataFilesList

data_files_list = DataFilesList(
    [
        "hf://datasets/allenai/c4@1588ec454efa1a09f29cd18ddd04fe05fc8653a2/en/c4-train.00000-of-01024.json.gz"
    ],
    [("allenai/c4", "1588ec454efa1a09f29cd18ddd04fe05fc8653a2")],
)
data_files = DataFilesDict({"train": data_files_list})
c4_dataset = datasets.load_dataset(
    path="allenai/c4",
    data_files=data_files,
    split="train",
    cache_dir="/datesets/cache",
    download_mode="reuse_cache_if_exists",
    token=False,
)
```
Second solution also shows where to find the bug. I suggest that the hashing functions should always use only original parameter `data_files`, and not the one they get after connecting to the server and creating `DataFilesDict`

Hi! You need to set the `HF_DATASETS_OFFLINE` env variable to `1` to load cached datasets offline, as explained in the docs [here](https://huggingface.co/docs/datasets/v2.19.0/en/loading#offline).

Just tested. It doesn't work, because of the exact problem I described above: hash of dataset config is different.
The only error difference is the reason why it cannot connect to HuggingFace (now it's 'offline mode is enabled')
![image](https://github.com/huggingface/datasets/assets/112088378/1a7e1720-d711-46e3-9c90-53d52c441e68)


Met a pretty similar issue here, as I manually load the dataset into ~/.cache and try to let `load_dataset` detect it automatically, but it will always try reach hub even I set `HF_DATASETS_OFFLINE` to 1. Have you solved it? 

same here!
Allow deleting a subset/config from a no-script dataset

As proposed by @BramVanroy, it would be neat to have this functionality through the API.

Probably best to implement this as a CLI command?

Thanks for your comment, @mariosasko. Or maybe both (in Python and as CLI command)? The Python command would be just the reverse of `push_to_hub`...

I am working on a draft implementation, so we can discuss about the API and UX.
[Feature request] Indexing datasets by a customly-defined id field to enable random access dataset items via the id

### Feature request

Some datasets may contain an id-like field, for example the `id` field in [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) and the `_id` field in [BeIR/dbpedia-entity](https://huggingface.co/datasets/BeIR/dbpedia-entity). HF datasets support efficient random access via row, but not via this kinds of id fields. I wonder if it is possible to add support for indexing by a custom "id-like" field to enable random access via such ids. The ids may be numbers or strings.

### Motivation

In some cases, especially during inference/evaluation, I may want to find out the item that has a specified id, defined by the dataset itself.

For example, in a typical re-ranking setting in information retrieval, the user may want to re-rank the set of candidate documents of each query. The input is usually presented in a TREC-style run file, with the following format:

```
<qid> Q0 <docno> <rank> <score> <tag>
```

The re-ranking program should be able to fetch the queries and documents according to the `<qid>` and `<docno>`, which are the original id defined in the query/document datasets. To accomplish this, I have to iterate over the whole HF dataset to get the mapping from real ids to row ids every time I start the program, which is time-consuming. Thus I want HF dataset to provide options for users to index by a custom id column, not by row.

### Your contribution

I'm not an expert in this project and I'm afraid that I'm not able to make contributions on the code.

You can simply use a python dict as index:

```python
>>> from datasets import load_dataset
>>> ds = load_dataset("BeIR/dbpedia-entity", "corpus", split="corpus")
>>> index = {key: idx for idx, key in enumerate(ds["_id"])}
>>> ds[index["<dbpedia:Pikachu>"]]
{'_id': '<dbpedia:Pikachu>',
 'title': 'Pikachu',
 'text': 'Pikachu (Japanese: ピカチュウ) are a fictional species of Pokémon.  Pokémon are fictional creatures that appear in an assortment of comic books, animated movies and television shows, video games, and trading card games licensed by The Pokémon Company, a Japanese corporation. The Pikachu design was conceived by Ken Sugimori.'}
```

Thanks for your reply. Yes, I can do that, but it is time-consuming to do that every time I launch the program (some datasets are extremely big). HF Datasets has a nice feature to support instant data loading and efficient random access via row ids. I'm curious if this beneficial feature could be further extended to custom data columns.


+1 on the issue I think it would be extremely useful

+1. This could be very useful.

+1 - currently having to manually implement this

If anyone has an idea how to do this in the right way (perhaps @albertvillanova ?) I would be happy to implement it

This would be very helpful to implement aspect ratio bucketing for image and video datasets
Incremental dataset (e.g. `.push_to_hub(..., append=True)`)

### Feature request

Have the possibility to do `ds.push_to_hub(..., append=True)`.

### Motivation

Requested in this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/3#65252597c4edc168202a5eaa) and 
this [comment](https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/4#6524f675c9607bdffb208d8f). Discussed internally on [slack](https://huggingface.slack.com/archives/C02EMARJ65P/p1696950642610639?thread_ts=1690554266.830949&cid=C02EMARJ65P).


### Your contribution

What I suggest to do for parquet datasets is to use `CommitOperationCopy` + `CommitOperationDelete` from `huggingface_hub`:
1. list files
2. copy files from parquet-0001-of-0004 to parquet-0001-of-0005
3. delete files like parquet-0001-of-0004
4. generate + add last parquet file parquet-0005-of-0005

=> make a single commit with all commit operations at once


I think it should be quite straightforward to implement. Happy to review a PR (maybe conflicting with the ongoing "1 commit push_to_hub" PR https://github.com/huggingface/datasets/pull/6269)

Yea I think waiting for #6269 would be best, or branching from it. For reference, this [PR](https://github.com/LAION-AI/Discord-Scrapers/pull/2) is progressing pretty well which will do similar using the hf hub for our LAION dataset bot https://github.com/LAION-AI/Discord-Scrapers/pull/2. 

Is there any update on this?

Is there any update on this?

No update so far on this feature request but for broader context, this announce will help with incremental datasets https://huggingface.co/blog/xethub-joins-hf :)