Sulak2020/github-issues-multirepo-datasets-Sulakshana huggingface dataset:
Title: `Dataset.from_generator` should not try to download from HF GCS
State: closed

### Describe the bug

When using [`Dataset.from_generator`](https://github.com/huggingface/datasets/blob/c9c1166e1cf81d38534020f9c167b326585339e5/src/datasets/arrow_dataset.py#L1072) with `streaming=False`, the internal logic will call [`download_and_prepare`](https://github.com/huggingface/datasets/blob/main/src/datasets/io/generator.py#L47) which will attempt to download from HF GCS which is redundant, because user has already provided the generator from which the data should be drawn. 

If someone attempts to call `Dataset.from_generator` from an environment that doesn't have external internet access (for example internal production machine) and doesn't set `HF_DATASETS_OFFLINE=1`, this will result in process being stuck at building connection.

### Steps to reproduce the bug

```python
import datasets
def gen():
    for _ in range(100):
        yield {"text": "dummy text"}

dataset = datasets.Dataset.from_generator(gen)
```

A minimum example executed on any environment that doesn't have access to HF GCS can result in the error

### Expected behavior

`try_from_hf_gcs` should be set to False here https://github.com/huggingface/datasets/blob/c9c1166e1cf81d38534020f9c167b326585339e5/src/datasets/io/generator.py#L51

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-3.10.0-1160.90.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.12
- Huggingface_hub version: 0.17.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3

Comments: Indeed, setting `try_from_gcs` to `False` makes sense for `from_generator`.

We plan to deprecate and remove `try_from_hf_gcs`  soon, as we can use Hub for file hosting now, but this is a good temporary fix.
Title: TensorDataset format does not work with Trainer from transformers
State: closed

### Describe the bug

The model was built to do fine tunning on BERT model for relation extraction.

trainer.train() returns an error message ```TypeError: vars() argument must have __dict__ attribute``` when it has `train_dataset` generated from `torch.utils.data.TensorDataset`

However, in the document, the required data format is `torch.utils.data.TensorDataset`. 
![image](https://github.com/huggingface/datasets/assets/49014051/36fa34ac-3127-4c64-9580-9ab736136d83)

Transformers trainer is supposed to accept the train_dataset in the format of torch.utils.data.TensorDataset, but it returns error message *"TypeError: vars() argument must have __dict__ attribute"*

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-30-5df728c929a2> in <cell line: 1>()
----> 1 trainer.train()
      2 trainer.evaluate(test_dataset)

9 frames
/usr/local/lib/python3.10/dist-packages/transformers/data/data_collator.py in <listcomp>(.0)
    107 
    108     if not isinstance(features[0], Mapping):
--> 109         features = [vars(f) for f in features]
    110     first = features[0]
    111     batch = {}

TypeError: vars() argument must have __dict__ attribute
```

### Steps to reproduce the bug

Create train_dataset using `torch.utils.data.TensorDataset`, for instance,

```train_dataset = torch.utils.data.TensorDataset(train_input_ids, train_attention_masks, train_labels)```

Feed this `train_dataset` to your trainer and run trainer.train

```
trainer = Trainer(model,
                  training_args,
                  train_dataset=train_dataset,
                  eval_dataset=dev_dataset,
                  compute_metrics=compute_metrics,
                  )
```

### Expected behavior

Trainer should start training

### Environment info

It is running on Google Colab

- `datasets` version: 2.14.6
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

Comments: I figured it out. I found that `Trainer` does not work with TensorDataset even though the document says it uses it. Instead, I ended up creating a dictionary and converting it to a dataset using `dataset.Dataset.from_dict()`.

I will leave this post open for a while. If someone knows a better approach, please leave a comment.Only issues directly related to the HF datasets library should be reported here. ~So, I'm transferring this issue to the `transformers` repo.~ I'm not a `transformers` maintainer, so GitHub doesn't let me transfer it there :(. This means you need to do it manually.
Title: with_format() function returns bytes instead of PIL images even when image column is not part of "columns"
State: closed

### Describe the bug

When using the with_format() function on a dataset containing images, even if the image column is not part of the columns provided in the function, its type will be changed to bytes.
Here is a minimal reproduction of the bug:
https://colab.research.google.com/drive/1hyaOspgyhB41oiR1-tXE3k_gJCdJUQCf?usp=sharing

### Steps to reproduce the bug

1. Load the image dataset
2. apply with_format(columns=["text"])
3. Check the type of images in the "image" column before and after applying with_format

### Expected behavior

The type should stay the same, but it does not

### Environment info

datasets==2.14.6


Comments: Thanks for reporting! I've opened a PR with a fix.
Title: Parquet size grows exponential for categorical data
State: closed

### Describe the bug

It seems that when saving a data frame with a categorical column inside the size can grow exponentially.

This seems to happen because when we save the categorical data to parquet, we are saving the data + all the categories existing in the original data. This happens even when the categories are not present in the original data.


### Steps to reproduce the bug

To reproduce the bug, it is enough to run this script:

```
import pandas as pd
import os

if __name__ == "__main__":
    for n in [10, 1e2, 1e3, 1e4, 1e5]:
        for n_col in [1, 10, 100, 1000, 10000]:
            input = pd.DataFrame([{"{i}": f"{i}_cat" for col in range(n_col)} for i in range(int(n))])
            input.iloc[0:100].to_parquet("a.parquet")
            for col in input.columns:
                input[col] = input[col].astype("category")
            input.iloc[0:100].to_parquet("b.parquet")
            a_size_mb = os.stat("a.parquet").st_size / (1024 * 1024)
            b_size_mb = os.stat("b.parquet").st_size / (1024 * 1024)
            print(f"{n} {n_col} {a_size_mb} {b_size_mb} {100*b_size_mb/a_size_mb:.2f}")
```

That produces this output:

<img width="464" alt="Screenshot 2023-10-31 at 11 25 25" src="https://github.com/huggingface/datasets/assets/82567957/2b8a9284-7f9e-4c10-a006-0a27236ebd15">



### Expected behavior

In my opinion either:

1. The two file should have (almost) the same size
2. There should be warning telling the user that such difference in size is possible

### Environment info

Python 3.8.18
pandas==2.0.3
numpy==1.24.4

Comments: Wrong repo.
Title:  Add support for `Sequence(Audio/Image)` feature in `push_to_hub`
State: closed
Labels: name: enhancement
description: New feature or request

### Feature request

Allow for `Sequence` of `Image` (or `Audio`) to be embedded inside the shards.

### Motivation

Currently, thanks to #3685, when `embed_external_files` is set to True (which is the default) in `push_to_hub`, features of type `Image` and `Audio` are embedded inside the arrow/parquet shards, instead of only storing paths to the files.

I've noticed that this behavior does not extend to `Sequence` of `Image`, when working with a [dataset of timelapse images](https://huggingface.co/datasets/1aurent/Human-Embryo-Timelapse).

### Your contribution

I'll submit a PR if I find a way to add this feature

Comments: This issue stems from https://github.com/huggingface/datasets/blob/6d2f2a5e0fea3827eccfd1717d8021c15fc4292a/src/datasets/table.py#L2203-L2205

I'll address it as part of https://github.com/huggingface/datasets/pull/6283.

In the meantime, this should work

```python
import pyarrow as pa
from datasets import Image

dataset = dataset.with_format("arrow")

def embed_images(pa_table):
    images_arr = pa.chunked_array(
        [
            pa.ListArray.from_arrays(chunk.offsets, Image().embed_storage(chunk.values), mask=chunk.is_null())
            for chunk in pa_table["images"].chunks
        ]
    )
    return pa_table.set_column(pa_table.schema.get_field_index("images"), "images", images_arr)

dataset = dataset.map(embed_images, batched=True)

dataset = dataset.with_format("python")

dataset.push_to_hub(...)
```
Title: Stuck in "Resolving data files..."
State: open

### Describe the bug

I have an image dataset with 300k images, the size of image is 768 * 768.

When I run `dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')` in second time, it takes 50 minutes to finish "Resolving data files" part, what's going on in this part?

From my understand, after Arrow files been created in the first run, the second run should not take time longer than one or two minutes.

### Steps to reproduce the bug

# Run following code two times
dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')

### Expected behavior

Fast dataset building

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- PyArrow version: 10.0.1
- Pandas version: 1.5.3

Comments: Most likely, the data file inference logic is the problem here.

You can run the following code to verify this:
```python
import time
from datasets.data_files import get_data_patterns
start_time = time.time()
get_data_patterns("/path/to/img_dir")
end_time = time.time()
print(f"Elapsed time: {end_time - start_time:.2f}s")
```
 
We plan to optimize this for the next version (or version after that). In the meantime, specifying the split patterns manually should give better performance:
```python
ds = load_dataset("imagefolder", data_files={"train": "path/to/img_dir/train/**", ...}, split="train")
```Hi, @mariosasko, you are right; data file inference logic is extremely slow.

I have done a similar test, that is I modify the source code of datasets/load.py to measure the cost of two suspicious operations:
```python
def get_module(self) -> DatasetModule:
        base_path = Path(self.data_dir or "").expanduser().resolve().as_posix()
        start = time.time()
        patterns = sanitize_patterns(self.data_files) if self.data_files is not None else get_data_patterns(base_path)
        print(f"patterns: {time.time() - start}")
        start = time.time()
        data_files = DataFilesDict.from_patterns(
            patterns,
            download_config=self.download_config,
            base_path=base_path,
        )
        print(f"data_files: {time.time() - start}")
```
It gaves:
patterns: 3062.2050700187683
data_files: 413.9576675891876

Thus, these two operations contribute to almost all of load time. What's going on in them?Furthermore, what's my current workaround about this problem? Should I save it by `save_to_disk()` and load dataset through `load_from_disk`?were you able to solve this issue?, I am facing the same issueStill suffering from this issue. For me, I cannot download Emilia dataset, and stucked at 'Resolving data files' forever.
``` e = load_dataset('amphion/Emilia-Dataset', token=my_token)
Resolving data files:   0%|                                                                                  | 1/4343 [00:00<11:27,  6.32it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████| 4343/4343 [00:01<00:00, 3844.15it/s]
```
Title: Allow passing a multiprocessing context to functions that support `num_proc`
State: open
Labels: name: enhancement
description: New feature or request

### Feature request

Allow specifying [a multiprocessing context](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) to functions that support `num_proc` or use multiprocessing pools. For example, the following could be done:

```python
dataset = dataset.map(_func, num_proc=2, mp_context=multiprocess.get_context("spawn"))
```

Or at least the multiprocessing start method ("fork", "spawn", "fork_server" or `None`):

```python
dataset = dataset.map(_func, num_proc=2, mp_start_method="spawn")
```

Another option could be passing the `pool` as an argument.

### Motivation

By default, `multiprocess` (the `multiprocessing`-fork library that this repo uses) uses the "fork" start method for multiprocessing pools (for the default context). It could be changed by using `set_start_method`. However, this conditions the multiprocessing start method from all processing in a Python program that uses the default context, because [you can't call that function more than once](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods:~:text=set_start_method()%20should%20not%20be%20used%20more%20than%20once%20in%20the%20program.). My proposal is to allow using a different multiprocessing context, not to condition the whole Python program.

One reason to change the start method is that "fork" (the default) makes child processes likely deadlock if thread pools were created before (and also this is not supported by POSIX). For example, this happens when using PyTorch because OpenMP threads are used for CPU intra-op parallelism, which is enabled by default (e.g., for context see [`torch.set_num_threads`](https://pytorch.org/docs/stable/generated/torch.set_num_threads.html)). This can also be fixed by setting `torch.set_num_threads(1)` (or similarly by other methods) but this conditions the whole Python program as it can only be set once to guarantee its behavior (similarly to). There are noticeable performance differences when setting this number to 1 even when using GPU(s). Using, e.g., a "spawn" start method would solve this issue.

For more context, see:

* https://discuss.huggingface.co/t/dataset-map-stuck-with-torch-set-num-threads-set-to-2-or-larger/37984
* https://discuss.huggingface.co/t/using-num-proc-1-in-dataset-map-hangs/44310

### Your contribution

I'd be happy to review a PR that makes such a change. And if you really don't have the bandwidth for it, I'd consider creating one.
Title: Different objects are returned from calls that should be returning the same kind of object.
State: open

### Describe the bug

        1.  dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=training_args.cache_dir, split='train[:1%]')
        2.  dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=training_args.cache_dir)

The only difference I would expect these calls to have is the size of the dataset.

But, while 2. returns a dictionary with "train" key in it, 1. returns a dataset WITHOUT any initial "train" keyword.

Both calls are to be used within exactly the same context. They should return identically structured datasets of different size.

### Steps to reproduce the bug

See above.

### Expected behavior

Expect both calls to return the same structured Dataset structure but with different number of elements, i.e. call 1. should have 1% of the data of the call 2.0

### Environment info

Ubuntu 20.04
gcc 9.x.x.

It is really irrelevant.

Comments: `load_dataset` returns a `DatasetDict` object unless `split`  is defined, in which case it returns a `Dataset` (or a list of datasets if `split` is a list). We've discussed dropping `DatasetDict` from the API in https://github.com/huggingface/datasets/issues/5189 to always return the same type in `load_dataset` and support datasets without (explicit) splits. IIRC the main discussion point is deciding what to return when loading a dataset with multiple splits, but `split` is not specified. What would you expect as a return value in that scenario?> `load_dataset` returns a `DatasetDict` object unless `split` is defined, in which case it returns a `Dataset` (or a list of datasets if `split` is a list). We've discussed dropping `DatasetDict` from the API in #5189 to always return the same type in `load_dataset` and support datasets without (explicit) splits. IIRC the main discussion point is deciding what to return when loading a dataset with multiple splits, but `split` is not specified. What would you expect as a return value in that scenario?

Wouldn't a dataset with multiple splits already have keys and their related data arrays?

Lets say the dataset has "train" : trainset, "valid": validset and "test": testset

So a dictionary can be returned,, i.e.

{ 
"train": trainset,
"valid": validset,
"test": testset
}

if a split is provided split=['train[:80%]', 'valid[80%:90%]', 'test[90%:100%]']

would also return the same dictionary as above.

split='train[:10%]' should return the same value as split=['train[:10%]']

{
"train": trainset
}
Title: Parquet stream-conversion fails to embed images/audio files from gated repos
State: open
Labels: name: bug
description: Something isn't working

it seems to be an issue with datasets not passing the token to embed_table_storage when generating a dataset

See https://github.com/huggingface/datasets-server/issues/2010

Comments: I have created a project to stream audio in the datasets viewer on Hugging Face using Parquet.

https://github.com/pr0mila/ParquetToHuggingFace
Title: Incorrect example code in 'Create a dataset' docs
State: closed

### Describe the bug

On [this](https://huggingface.co/docs/datasets/create_dataset) page, the example code for loading in images and audio is incorrect.

Currently, examples are:

``` python
from datasets import ImageFolder
dataset = load_dataset("imagefolder", data_dir="/path/to/pokemon")
```
and 
``` python
from datasets import AudioFolder
dataset = load_dataset("audiofolder", data_dir="/path/to/folder")
```

I'm pretty sure the imports are wrong and should be:

``` python 
from datasets import load_dataset
dataset = load_dataset("audiofolder", data_dir="/path/to/folder")
```

I am happy to update this if this is right but just wanted to check before making any changes.

### Steps to reproduce the bug

Go to https://huggingface.co/docs/datasets/create_dataset

### Expected behavior

N/A

### Environment info

N/A

Comments: This was fixed in https://github.com/huggingface/datasets/pull/6247. You can find the fix in the `main` version of the docsAh great, thanks :)
Title: support squad structure datasets using a YAML parameter
State: open
Labels: name: enhancement
description: New feature or request

### Feature request

Since the squad structure is widely used, I think it could be beneficial to support it using a YAML parameter.
could you implement automatic data loading of squad-like data using squad JSON format, to read it from JSON files and view it in the correct squad structure.
The dataset structure should be like this:
https://huggingface.co/datasets/squad
Columns:id,title,context,question,answers

### Motivation

Dataset repo requires arbitrary Python code execution

### Your contribution

The dataset structure should be like this:
https://huggingface.co/datasets/squad
Columns:id,title,context,question,answers
train and dev sets in squad structure JSON files
Title: Support fsspec 2023.10.0
State: closed

Once root issue is fixed, remove temporary pin of fsspec < 2023.10.0 introduced by:
- #6331

Related to issue:
- #6330

As @ZachNagengast suggested, the issue might be related to:
- https://github.com/fsspec/filesystem_spec/pull/1381

Comments: Hi @albertvillanova @lhoestq 

I believe the pull request that pins the fsspec version (https://github.com/huggingface/datasets/pull/6331) was merged by mistake. Another fix for the issue was merged on the same day an hour apart. See https://github.com/huggingface/datasets/pull/6334

I'm now having an issue in my project where I can't use newer versions of fsspec.

Can we remove the pin?

Have a nice day! :)Hi @tomscholz,

Thanks for pointing this out. I think you are right.

I am doing some cross-checks and fixing it. Hi again, @tomscholz.

After a more cautious investigation, I think the pin is OK because there are other reasons for it. Chronologically:
- #6331 
- #6334
- #6336 
- #6337 

The reason is that after version 2023.10.0, they changed again the behavior of their `glob` function. See: https://github.com/huggingface/datasets/pull/6337#issuecomment-1774930135
We are working on our side to support both previous and new glob behavior.

Note:
- First pin was < 2023.10.0
- Last pin is <= 2023.10.0Fixed by #6334 and #6336.
Title: FileNotFoundError when trying to load the downloaded dataset with `load_dataset(..., streaming=True)`
State: closed

### Describe the bug

Hi, I'm trying to load the dataset `togethercomputer/RedPajama-Data-1T-Sample` with `load_dataset` in streaming mode, i.e., `streaming=True`, but `FileNotFoundError` occurs.

### Steps to reproduce the bug

I've downloaded the dataset and save it to the cache dir in advance. My hope is loading the files in offline environment and without taking too much hours to prepross the entire data before running into the training process.
So I try the following code to load the files streamingly
```py
dataset = load_dataset('togethercomputer/RedPajama-Data-1T-Sample', streaming=True)
print(next(iter(dataset['train'])))
```
Sadly, it raises the following:
```
FileNotFoundError: [Errno 2] No such file or directory: 'CURRENT_CODE_PATH/arxiv_sample.jsonl'
```
I've noticed that the dataset can be properly found in the begining
```
Using the latest cached version of the module from /root/.cache/huggingface/modules/datasets_modules/datasets/togethercomputer--RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 (last modified on Sat Oct 21 20:12:57 2023) since it couldn't be found locally at togethercomputer/RedPajama-Data-1T-Sample., or remotely on the Hugging Face Hub.
```
But it seems that the paths couldn't be properly parsed when loading iteratively.
 
How should I fix this error. I've tried specifying `data_files` or `data_dir` as `.../arxiv_sample.jsonl` but none of them works.

Thanks.

### Expected behavior

Properly load the dataset.

### Environment info

`datasets==2.14.5`

Comments: You can clone the `togethercomputer/RedPajama-Data-1T-Sample` repo and load the dataset with `load_dataset("path/to/cloned_repo")` to use it offline.@mariosasko Thank you for your kind reply! I'll try it as a workaround.
Does that mean that currently it's not supported to simply load with a short name?It is, but manually downloading repo files to the cache can easily lead to failure (the HF cache is not meant to be modified by a user besides deleting the files 🙂), as in your case. Hence, the clone + `load_dataset("path/to/cloned_repo")` workflow should be used instead.