noamaanMulla-03/datasets-issues huggingface dataset:
Easy way to create a Kaggle dataset from a Huggingface dataset?

I'm not sure whether this is more appropriately addressed with HuggingFace or Kaggle.  I would like to somehow directly create a Kaggle dataset from a HuggingFace Dataset.

While Kaggle does provide the option to create a dataset from a URI, that URI must point to a single file.  For example:

![image](https://user-images.githubusercontent.com/5355286/235792394-7c559d07-4aff-45b7-ad2b-9c5280c88415.png)


Is there some mechanism from huggingface to represent a dataset (such as that from `load_dataset('wmt14', 'de-en', split='train')` as a single file?  Or, some other way to get that into a Kaggle dataset so that I can use the huggingface `datasets` module to process and consume it inside of a Kaggle notebook?

Thanks in advance!



Hi @hrbigelow , I'm no expert for such a question so I'll ping @lhoestq from the `datasets` library (also this issue could be moved there if someone with permission can do it :) )

Hi ! Many datasets are made of several files, and how they are parsed often requires a python script. Because of that, datasets like wmt14 are not available as a single file on HF. Though you can create this file using `datasets`:

```python
from datasets import load_dataset

ds = load_dataset("wmt14", "de-en", split="train")

ds.to_json("wmt14-train.json")
# OR to parquet, which is compressed:
# ds.to_parquet("wmt14-train.parquet")
```

We are also working on providing parquet exports for all datasets, but wmt14 is not supported yet (we're rolling it out for datasets <1GB first). They're usually available in the `refs/convert/parquet` branch (empty for wmt14):

<img width="267" alt="image" src="https://user-images.githubusercontent.com/42851186/235878909-7339f5a4-be19-4ada-85d8-8a50d23acf35.png">


also cc @nateraw for visibility on this (and cc @osanseviero too)

I've requested support for creating a Kaggle dataset from an imported HF dataset repo on their "forum" here: https://www.kaggle.com/discussions/product-feedback/427142 (upvotes appreciated 🙂)
Adds CodeClippy dataset [WIP]

CodeClippy is an opensource code dataset scrapped from github during flax-jax-community-week
https://the-eye.eu/public/AI/training_data/code_clippy_data/

Labels: name: dataset contribution

Thanks for your contribution, @arampacha. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.

Sorry to resurrect a dead issue, but any chance the dataset will make it to HuggingFace? I would love to use it to finetune Llama 2 and HF makes this a breeze. Also happy to submit a PR prepping it for HF if that is needed.
Changing pixel values of images in the Winoground dataset

Hi, as I followed the instructions, with lasted "datasets" version:
"
from datasets import load_dataset
examples = load_dataset('facebook/winoground', use_auth_token=<YOUR USER ACCESS TOKEN>)
"
I got slightly different datasets in colab and in my hpc environment. Specifically, the pixel values of images are slightly different. 
I thought it was due to the package version difference, but today's morning I found out that my winoground dataset in colab became the same with the one in my hpc environment. The dataset in colab can produce the correct result but now it is gone as well. 

Can you help me with this? What causes the datasets to have the wrong pixel values?
Loading local data files initiates web requests

As documented in the [official docs](https://huggingface.co/docs/datasets/v2.14.0/en/package_reference/loading_methods#datasets.load_dataset.example-2), I tried to load datasets from local files by
```python
# Load a JSON file
from datasets import load_dataset
ds = load_dataset('json', data_files='path/to/local/my_dataset.json')
```
But this failed on a web request because I'm executing the script on a machine without Internet access. Stacktrace shows
```
in PackagedDatasetModuleFactory.__init__(self, name, data_dir, data_files, download_config, download_mode)
    940 self.download_config = download_config
    941 self.download_mode = download_mode
--> 942 increase_load_count(name, resource_type="dataset")
```
I've read from the source code that this can be fixed by setting environment variable to run in offline mode. I'm just wondering that is this an expected behaviour that even loading a LOCAL JSON file requires Internet access by default? And what's the point of requesting to `increase_load_count` on some server when loading just LOCAL data files?
fsspec dependency is set too low

### Describe the bug

fsspec.callbacks.TqdmCallback (used in https://github.com/huggingface/datasets/blob/73bed12ecda17d1573fd3bf73ed5db24d3622f86/src/datasets/utils/file_utils.py#L338) was first released in fsspec [2022.3.0](https://github.com/fsspec/filesystem_spec/releases/tag/2022.3.0, commit where it was added: https://github.com/fsspec/filesystem_spec/commit/9577c8a482eb0a69092913b81580942a68d66a76#diff-906155c7e926a9ff58b9f23369bb513b09b445f5b0f41fa2a84015d0b471c68cR180), however the dependency is set to 2021.11.1 https://github.com/huggingface/datasets/blob/main/setup.py#L129



### Steps to reproduce the bug

1. Install fsspec==2021.11.1
2. Install latest datasets==2.14.1
3. Import datasets, import fails due to lack of `fsspec.callbacks.TqdmCallback`

### Expected behavior

No import issue

### Environment info

N/A


Thanks for reporting! A PR with a fix has just been merged.
FilesIterable skips all the files after a hidden file

### Describe the bug

When initializing `FilesIterable` with a list of file paths using `FilesIterable.from_paths`, it will discard all the files after a hidden file.
The problem is in [this line](https://github.com/huggingface/datasets/blob/88896a7b28610ace95e444b94f9a4bc332cc1ee3/src/datasets/download/download_manager.py#L233C26-L233C26) where `return` should be replaced by `continue`. 

### Steps to reproduce the bug

https://colab.research.google.com/drive/1SQlxs4y_LSo1Q89KnFoYDSyyKEISun_J#scrollTo=93K4_blkW-8-

### Expected behavior

The script should print all the files except the hidden one.

### Environment info

- `datasets` version: 2.14.1
- Platform: Linux-5.15.109+-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.4
- PyArrow version: 9.0.0
- Pandas version: 1.5.3


Thanks for reporting. We've merged a PR with a fix.
resume_download with streaming=True

### Describe the bug

I used:
```
dataset = load_dataset(
    "oscar-corpus/OSCAR-2201",
    token=True,
    language="fr",
    streaming=True,
    split="train"
)
```
Unfortunately, the server had a problem during the training process. I saved the step my training stopped at.
But how can I resume download from step 1_000_´000 without re-streaming all the first 1 million docs of the dataset?

`download_config=DownloadConfig(resume_download=True)` seems to not work with streaming=True.

### Steps to reproduce the bug

```
from datasets import load_dataset, DownloadConfig
dataset = load_dataset(
    "oscar-corpus/OSCAR-2201",
    token=True,
    language="fr",
    streaming=True,  # optional
    split="train",
    download_config=DownloadConfig(resume_download=True)
)
# interupt the run and try to relaunch it => this restart from scratch
```

### Expected behavior

I would expect a parameter to start streaming from a given index in the dataset.

### Environment info

- `datasets` version: 2.14.0
- Platform: Linux-5.19.0-45-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.0


Currently, it's not possible to efficiently resume streaming after an error. Eventually, we plan to support this for Parquet (see https://github.com/huggingface/datasets/issues/5380). 

Ok thank you for your answer

I'm closing this as a duplicate of #5380
Dataset.get_nearest_examples does not return all feature values for the k most similar datapoints - side effect of Dataset.set_format

### Describe the bug

Hi team!

I observe that there seems to be a side effect of `Dataset.set_format`: after setting a format and creating a FAISS index, the method `get_nearest_examples` from the `Dataset` class, fails to retrieve anything else but the embeddings themselves - not super useful. This is not the case if not using the `set_format` method: you can also retrieve any other feature value, such as an index/id/etc.

Are you able to reproduce what I observe?

### Steps to reproduce the bug

```python

from datasets import Dataset
import numpy as np

foo = {'vectors': np.random.random((100,1024)), 'ids': [str(u) for u in range(100)]}
foo = Dataset.from_dict(foo)
foo.set_format('numpy', ['vectors'])
foo.add_faiss_index('vectors')
new_vector = np.random.random(1024)
scores, res = foo.get_nearest_examples('vectors', new_vector, k=3)
```
This will return, for the resulting most similar vectors to `new_vector` - in particular it will not return the `ids` feature:
```
{'vectors': array([[random values ...]])}
```

### Expected behavior

The expected behavior happens when the `set_format` method is not called:
```python

from datasets import Dataset
import numpy as np

foo = {'vectors': np.random.random((100,1024)), 'ids': [str(u) for u in range(100)]}
foo = Dataset.from_dict(foo)
# foo.set_format('numpy', ['vectors'])
foo.add_faiss_index('vectors')
new_vector = np.random.random(1024)
scores, res = foo.get_nearest_examples('vectors', new_vector, k=3)
```
This *will*  return the `ids` of the similar vectors - with unfortunately a list of lists in lieu of the array I think for caching reasons - read it elsewhere
```
{'vectors': [[random values on multiple lines...]], 'ids': ['x', 'y', 'z']}
```

### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.4.0-155-generic-x86_64-with-glibc2.31
- Python version: 3.10.6
- Huggingface_hub version: 0.15.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3



Actually, my bad -- specifying
```python
foo.set_format('numpy', ['vectors'], output_all_columns=True)
```
fixes it.
HF Datasets data access is extremely slow even when in memory

### Describe the bug

Doing a simple `some_dataset[:10]` can take more than a minute.

Profiling it:
<img width="1280" alt="image" src="https://github.com/huggingface/datasets/assets/36224762/e641fb95-ff02-4072-9016-5416a65f75ab">

`some_dataset` is completely in memory with no disk cache.

This is proving fatal to my usage of HF Datasets. Is there a way I can forgo the arrow format and store the dataset as PyTorch tensors so that `_tensorize` is not needed? And is `_consolidate` supposed to take this long?

It's faster to produce the dataset from scratch than to access it from HF Datasets!

### Steps to reproduce the bug

I have uploaded the dataset that causes this problem [here](https://huggingface.co/datasets/NightMachinery/hf_datasets_bug1).

```python
#!/usr/bin/env python3
import sys
import time
import torch
from datasets import load_dataset


def main(dataset_name):
    # Start the timer
    start_time = time.time()

    # Load the dataset from Hugging Face Hub
    dataset = load_dataset(dataset_name)

    # Set the dataset format as torch
    dataset.set_format(type="torch")

    # Perform an identity map
    dataset = dataset.map(lambda example: example, batched=True, batch_size=20)

    # End the timer
    end_time = time.time()

    # Print the time taken
    print(f"Time taken: {end_time - start_time:.2f} seconds")


if __name__ == "__main__":
    dataset_name = "NightMachinery/hf_datasets_bug1"
    print(f"dataset_name: {dataset_name}")
    main(dataset_name)
```

### Expected behavior

_

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3


Possibly related:
- https://github.com/pytorch/pytorch/issues/22462
TypeError when loading from GCP bucket

### Describe the bug

Loading a dataset from a GCP bucket raises a type error. This bug was introduced recently (either in 2.14 or 2.14.1), and appeared during a migration from 2.13.1.

### Steps to reproduce the bug

Load any file from a GCP bucket:

```python
import datasets

datasets.load_dataset("json", data_files=["gs://..."])
```

The following exception is raised:

```python
Traceback (most recent call last):
...
packages/datasets/data_files.py", line 335, in resolve_pattern
    protocol_prefix = fs.protocol + "://" if fs.protocol != "file" else ""
TypeError: can only concatenate tuple (not "str") to tuple
```

With a `GoogleFileSystem`, the attribute `fs.protocol` is a tuple `('gs', 'gcs')` and hence cannot be concatenated with a string.


### Expected behavior

The file should be loaded without exception.

### Environment info

- `datasets` version: 2.14.1
- Platform: macOS-13.2.1-x86_64-i386-64bit
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3



Thanks for reporting, @bilelomrani1.

We are fixing it. 

We have fixed it. We are planning to do a patch release today.
WikiText 103 tokenizer hangs

See issue here: https://github.com/huggingface/transformers/issues/19702 

Labels: name: bug

any updates on this? It happens to me on [OpenWikiText-20%](https://huggingface.co/datasets/Bingsu/openwebtext_20p) dataset, but not on [OpenWebText-10k](https://huggingface.co/datasets/stas/openwebtext-10k). This is really strange because I don't change anything else in my running script.

transformers version 4.18.0.dev0
datasets version 1.18.0
Dataset is slower after calling `to_iterable_dataset`

### Describe the bug

Can anyone explain why looping over a dataset becomes slower after calling `to_iterable_dataset` to convert to `IterableDataset`

### Steps to reproduce the bug

Any dataset after converting to `IterableDataset`

### Expected behavior

Maybe it should be faster on big dataset? I only test on small dataset

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3


@lhoestq 

It's roughly the same code between the two so we can expected roughly the same speed, could you share a benchmark ?
Merge DatasetDict and Dataset

As discussed in #2424 and #2437 (please see there for detailed conversation):
- It would be desirable to improve UX with respect the confusion between DatasetDict and Dataset.
- The difference between Dataset and DatasetDict is an additional abstraction complexity that confuses "typical" end users. 
- A user expects a "Dataset" (whatever it contains multiple or a single split) and maybe it could be interesting to try to simplify the user-facing API as much as possible to hide this complexity from the end user.

Here is a proposal for discussion and refined (and potential abandon if it's not good enough):
- let's consider that a DatasetDict is also a Dataset with the various split concatenated one after the other
- let's disallow the use of integers in split names (probably not a very big breaking change)
- when you index with integers you access the examples progressively in split after the other is finished (in a deterministic order)
- when you index with strings/split name you have the same behavior as now (full backward compat)
- let's then also have all the methods of a Dataset on the DatasetDict

The end goal would be to merge both Dataset and DatasetDict object in a single object that would be (pretty much totally) backward compatible with both.


There are a few things that we could discuss if we want to merge Dataset and DatasetDict:

1. what happens if you index by a string ? Does it return the column or the split ? We could disallow conflicts between column names and split names to avoid ambiguities. It can be surprising to be able to get a column or a split using the same indexing feature
   ```
   from datasets import load_dataset

   dataset = load_dataset(...)
   dataset["train"]
   dataset["input_ids"]
   ```
2. what happens when you iterate over the object ? I guess it should iterate over the examples as a Dataset object, but a DatasetDict used to iterate over the splits as they are the dictionary keys. This is a breaking change that we can discuss.

Moreover regarding your points:

- integers are not allowed as split names already
- it's definitely doable to have all the methods. Maybe some of them like train_test_split that is currently only available for Dataset can be tweaked to work for a split dataset


cc: @thomwolf @lhoestq 

Labels: name: enhancement

name: generic discussion

Any update on this? @lhoestq 

Unless there is high demande I don't think we will end up implementing this. This is a lot of work with very few advantages
Force JSON format regardless of file naming on S3

I have a file on S3 created by Data Version Control, it looks like `s3://dvc/ac/badff5b134382a0f25248f1b45d7b2` but contains a json file. If I run 
```python
dataset = load_dataset(
    "json",
    data_files='s3://dvc/ac/badff5b134382a0f25248f1b45d7b2' 
)
```
It gives me 
```
InvalidSchema: No connection adapters were found for 's3://dvc/ac/badff5b134382a0f25248f1b45d7b2'
```
However, I cannot go ahead and change the names of the s3 file. Is there a way to "force" load a S3 url with certain decoder (JSON, CSV, etc.) regardless of s3 URL naming?

Labels: name: enhancement

Hi ! Support for URIs like `s3://...` is not implemented yet in `data_files=`. You can use the HTTP URL instead if your data is public in the meantime

Hi,
I want to make sure I understand this response. I have a set of files on S3 that are private for security reasons. Because they are not public files  I cannot read those files (many are parquet) into my hf notebooks in Kaggle? That can't be correct, can it? 

Hi ! There is a discussion at https://github.com/huggingface/datasets/issues/5281

Using the latest `datasets` 2.11 you can try passing fsspec URLs to private buckets to `data_files` in `load_dataset()`. Though this is still experimental and undocumented, so feedback is welcome. You may not have the best experience though, since anything related to performance and caching hasn't been tested properly yet.

closing this one since data_files supports fsspec (still experimental/untested/undocumented for s3 though)
[Docs] The "Process" how-to guide lacks description of `select_columns` function

### Feature request

The [how to process dataset guide](https://huggingface.co/docs/datasets/main/en/process) currently does not mention the [`select_columns`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.select_columns) function. It would be nice to include it in the guide.

### Motivation

This function is a commonly requested feature (see this [forum thread](https://discuss.huggingface.co/t/how-to-create-a-new-dataset-from-another-dataset-and-select-specific-columns-and-the-data-along-with-the-column/15120) and #5468 #5474). However, it has not been included in the guide since its implementation by PR #5480.

Mentioning it in the guide would help future users discover this added feature.

### Your contribution

I could submit a PR to add a brief description of the function to said guide.

Labels: name: enhancement

Great idea, feel free to open a PR! :)
Can not load audiofolder dataset on kaggle

### Describe the bug
It's crash log:
FileNotFoundError: Couldn't find a dataset script at /kaggle/working/audiofolder/audiofolder.py or any data file in the same directory. Couldn't find 'audiofolder' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/audiofolder/audiofolder.py


### Steps to reproduce the bug

![image](https://github.com/huggingface/datasets/assets/93691919/a2829d27-d15c-4acc-86fb-d1987c760468)
common_voice = load_dataset("audiofolder", data_dir="/kaggle/working/data")

### Expected behavior

load dataset without error. It works ok on colab, but on kaggle it happends.

### Environment info

- `datasets` version: 2.1.0
- Platform: Linux-5.15.109+-x86_64-with-glibc2.31
- Python version: 3.10.10
- PyArrow version: 9.0.0
- Pandas version: 1.5.3


Hi! `audiofolder` requires `datasets>=2.5.0`, so please update the `datasets`' installation (`pip install -U datasets`) in the environment (and restart the env for the update to take effect) to resolve the issue.

> Hi! `audiofolder` requires `datasets>=2.5.0`, so please update the `datasets`' installation (`pip install -U datasets`) in the environment to resolve the issue.

I don't think it is a problem of the version. It runs ok on colab or local machine. Only on kaggle will has this bug.

Based on your dataset info, the installed version is `2.1.0`, which does not include `audiofolder`.

By default, Kaggle preinstalls `datasets` into a new env, but the version it installs is outdated and does not contain newer features such as `audiofolder`
Allow dataset implement .take

### Feature request

I want to do:
```
dataset.take(512)
```

but it only works with streaming = True

### Motivation

uniform interface to data sets. Really surprising the above only works with streaming = True. 

### Your contribution

Should be trivial to copy paste the IterableDataset .take to use the local path in the data (when streaming = False)

Labels: name: enhancement

```
    dataset = IterableDataset(dataset) if type(dataset) != IterableDataset else dataset  # to force dataset.take(batch_size) to work in non-streaming mode
   ```


hf discuss: https://discuss.huggingface.co/t/how-does-one-make-dataset-take-512-work-with-streaming-false-with-hugging-face-data-set/50770

so: https://stackoverflow.com/questions/76902824/how-does-one-make-dataset-take512-work-with-streaming-false-with-hugging-fac

Feel free to work on this. In addition, `IterableDataset` supports `skip`, so we should also add this method to `Dataset`.
[BUG] Dataset initialized from in-memory data does not create cache.

### Describe the bug

`Dataset` initialized from in-memory data (dictionary in my case, haven't tested with other types) does not create cache when processed with the `map` method, unlike `Dataset` initialized by other methods such as `load_dataset`.

### Steps to reproduce the bug

```python
# below code was run the second time so the map function can be loaded from cache if exists

from datasets import load_dataset, Dataset

dataset = load_dataset("tatsu-lab/alpaca")['train']

dataset = dataset.map(lambda x: {'input': x['input'] + 'hi'}) # some random map
print(len(dataset.cache_files))
# 1

# copy the exact same data but initialize from a dictionary
memory_dataset = Dataset.from_dict({
    'instruction': dataset['instruction'],
    'input': dataset['input'],
    'output': dataset['output'],
    'text': dataset['text']})

memory_dataset = memory_dataset.map(lambda x: {'input': x['input'] + 'hi'}) # exact same map
print(len(memory_dataset.cache_files))
# Map: 100%|██████████| 52002[/52002]
# 0
```

### Expected behavior

The `map` function should create cache regardless of the method the `Dataset` was created.

### Environment info

- `datasets` version: 2.14.2
- Platform: Linux-5.15.0-41-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3


This is expected behavior. You must provide `cache_file_name` when performing `.map` on an in-memory dataset for the result to be cached.