Minimartzz/github-issues huggingface dataset:
Formatting overhead
State: closed

### Describe the bug

Hi! I very recently noticed that my training time is dominated by batch formatting. Using Lightning's profilers, I located the bottleneck within `datasets.formatting.formatting` and then narrowed it down with `line-profiler`. It turns out that almost all of the overhead is due to creating new instances of `self.python_arrow_extractor`. I admit I'm confused why that could be the case - as far as I can tell there's no complex `__init__` logic to execute.

![image](https://github.com/huggingface/datasets/assets/320321/5e022e0b-0d21-43d0-8e6f-9e641142e96b)

### Steps to reproduce the bug

1. Set up a dataset `ds` with potentially several (4+) columns (not sure if this is necessary, but it did at one point of the investigation make overhead worse)
2. Process it using a custom transform, `ds = ds.with_transform(transform_func)`
3. Decorate this function https://github.com/huggingface/datasets/blob/main/src/datasets/formatting/formatting.py#L512 with `@profile` from https://pypi.org/project/line-profiler/
4. Profile with `$ kernprof -l script_to_profile.py`

### Expected behavior

Batch formatting should have acceptable overhead.

### Environment info

```
datasets=2.14.6
pyarrow=14.0.0
```

Comments:
Ah I think the `line-profiler` log is off-by-one and it is in fact the `extract_batch` method that's taking forever. Will investigate further.

I tracked it down to a quirk of my setup. Apologies.
imagenet-1k downloads over and over
State: closed

### Describe the bug

What could be causing this?
```
$ python3
Python 3.8.13 (default, Mar 28 2022, 11:38:47) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> load_dataset("imagenet-1k")
Downloading builder script: 100%|██████████| 4.72k/4.72k [00:00<00:00, 7.51MB/s]
Downloading readme: 100%|███████████████████| 85.4k/85.4k [00:00<00:00, 510kB/s]
Downloading extra modules: 100%|████████████| 46.4k/46.4k [00:00<00:00, 300kB/s]
Downloading data: 100%|████████████████████| 29.1G/29.1G [19:36<00:00, 24.8MB/s]
Downloading data: 100%|████████████████████| 29.3G/29.3G [08:38<00:00, 56.5MB/s]
Downloading data: 100%|████████████████████| 29.0G/29.0G [09:26<00:00, 51.2MB/s]
Downloading data: 100%|████████████████████| 29.2G/29.2G [09:38<00:00, 50.6MB/s]
Downloading data: 100%|███████████████████▉| 29.2G/29.2G [09:37<00:00, 44.1MB/s^Downloading data:   0%|                     | 106M/29.1G [00:05<23:49, 20.3MB/s]
```

### Steps to reproduce the bug

See above commands/code

### Expected behavior

imagenet-1k is downloaded

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-6.2.0-34-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.15.1
- PyArrow version: 14.0.0
- Pandas version: 1.5.2

Comments:
Have you solved this problem?
Add CheXpert dataset for vision
State: open
Labels: description: New feature or request
name: enhancement

description: Requesting to add a new dataset
name: dataset request

### Feature request

### Name 

**CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison**

### Paper

https://arxiv.org/abs/1901.07031

### Data

https://stanfordaimi.azurewebsites.net/datasets/8cbd9ed4-2eb9-4565-affc-111cf4f7ebe2

### Motivation

CheXpert is one of the fundamental models in medical image classification and can serve as a viable pre-training dataset for radiology classification or low-scale ablation / exploratory studies.

This could also serve as a good pre-training dataset for Kaggle competitions.

### Your contribution

Would love to make a PR and pre-process / get this into 🤗

Comments:
Hey @SauravMaheshkar ! Just responded to your email.

_For transparency, copying part of my response here:_
I agree, it would be really great to have this and other BenchMD datasets easily accessible on the hub.

I think the main limiting factor is that the ChexPert dataset is currently hosted on the Stanford AIMI Shared Datasets website, with a license that does not permit redistribution IIRC. Thus, I believe we would need to create a [dataset loading script](https://huggingface.co/docs/datasets/image_dataset#loading-script) that would check authentication with the Stanford AIMI site before downloading and extracting the data. 

I've started a HF dataset repo [here](https://huggingface.co/datasets/katielink/CheXpert), in case you want to collaborate on writing up this loading script! I'm also happy to take a stab when I have some more time next week.

Hey @katielink I would love to try this out. Please guide me.

Hi @katielink , I would also love to be on board and contribute to this loading script/project if it is still being developed. I'm interested because I personally would like to gain access to the CheXpert dataset and am facing some weird issues, so I'd like to sort it out for me, and potentially others. Please keep me updated and guide me on this as well!!!
Add my dataset
State: closed

## medical data

**Description:**
This dataset, named "medical data," is a collection of text data from various sources, carefully curated and cleaned for use in natural language processing (NLP) tasks. It consists of a diverse range of text, including articles, books, and online content, covering topics from science to literature.

**Citation:**
If applicable, please include a citation for this dataset to give credit to the original sources or contributors.

**Key Features:**
- Language: The text is primarily in English, but it may include content in other languages as well.
- Use Cases: This dataset is suitable for text classification, language modeling, sentiment analysis, and other NLP tasks.

**Usage:**
To access this dataset, use the `load_your_dataset` function provided in the `your_dataset.py` script within this repository. You can specify the dataset split you need, such as "train," "test," or "validation," to get the data for your specific task.

**Contributors:**
- [Keyur Chaudhari]

**Contact:**
If you have any questions or need assistance regarding this dataset, please feel free to contact [keyurchaudhari536@gmail.com].

Please note that this dataset is shared under a specific license, which can be found in the [LICENSE](link to your dataset's license) file. Make sure to review and adhere to the terms of the license when using this dataset for your projects.


Comments:
Hi! We do not host datasets in this repo. Instead, you should use `dataset.push_to_hub` to upload the dataset to the HF Hub.

@mariosasko could you provide me proper guide to push data on HF hub 

You can find this info here: https://huggingface.co/docs/datasets/upload_dataset. Also, check https://huggingface.co/docs/datasets/loading for how to load a local dataset (before pushing it to the Hub).
Fix for continuation behaviour on broken dataset archives due to starving download connections via HTTP-GET
State: open

This PR proposes a (slightly hacky) fix for an Issue that can occur when downloading large dataset parts over unstable connections.
The underlying issue is also being discussed in https://github.com/huggingface/datasets/issues/5594.

Issue Symptoms & Behaviour:
- Download of a large archive file during dataset download via HTTP-GET fails.
- An silent net exception (which I was unable to identify) is thrown within the `tqdm` download progress.
- Due to missing exception catch code, the above process just continues processing, assuming `http_get` completed successfully.
- Pending Archive file gets renamed to remove the `.incomplete` extension, despite not all data has been downloaded. 
- Also, for reasons I did not investigate, there seems to be no real integrity check for the downloaded files; or it does not detect this problem. This is especially problematic, since the downloader script won't retry downloading this archive after CRC-Checking, even if it is being manually restarted / executed again after running into errors on extraction.

Fix proposal: Adding a retry mechanic for HTTP-GET downloads, which adds the following behaviour:
- Download Progress Thread checks for download size validity in case the HTTP connection starves mid download. If the check fails, a RuntimeError is thrown
- Cache Downloader code with retry mechanic monitors for an exception thrown by the download progress thread, and retries download with updated `resume_size`.
- Cache Downloader will not mark incomplete files which have thrown an exception during download, and exceeded retries, as complete.

Comments:
Caching problem when deleting a dataset 
State: closed

### Describe the bug

Pushing a dataset with n + m features to a repo which was deleted, but contained n features, will fail.

### Steps to reproduce the bug

1. Create a dataset with n features per row
2. `dataset.push_to_hub(YOUR_PATH, SPLIT, token=TOKEN)`
3. Go on the hub, delete the repo at `YOUR_PATH`
4. Update your local dataset to have n + m features per row
5. `dataset.push_to_hub(YOUR_PATH, SPLIT, token=TOKEN)` will fail because of a mismatch in features number


### Expected behavior

Step 5 should work or display a message to indicate the cache has not been cleared

### Environment info

- `datasets` version: 2.12.0
- Platform: Linux-5.15.0-88-generic-x86_64-with-glibc2.31
- Python version: 3.10.10
- Huggingface_hub version: 0.16.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.0


Comments:
Thanks for reporting! Can you also share the error message printed in step 5?

I did not store it at the time but I'll try to re-do a mwe next week to get it again

I haven't managed to reproduce this issue using a [notebook](https://colab.research.google.com/drive/1m6eduYun7pFTkigrCJAFgw0BghlbvXIL?usp=sharing) that follows the steps to reproduce the bug. So, I'm closing it.

But feel free to re-open it if you have a better reproducer.
CI is broken: TypeError: Couldn't cast array
State: closed

See: https://github.com/huggingface/datasets/actions/runs/6730567226/job/18293518039
```
FAILED tests/test_table.py::test_cast_sliced_fixed_size_array_to_features - TypeError: Couldn't cast array of type
fixed_size_list<item: int32>[3]
to
Sequence(feature=Value(dtype='int64', id=None), length=3, id=None)
```

Comments:
`Dataset.from_generator` should not try to download from HF GCS
State: closed

### Describe the bug

When using [`Dataset.from_generator`](https://github.com/huggingface/datasets/blob/c9c1166e1cf81d38534020f9c167b326585339e5/src/datasets/arrow_dataset.py#L1072) with `streaming=False`, the internal logic will call [`download_and_prepare`](https://github.com/huggingface/datasets/blob/main/src/datasets/io/generator.py#L47) which will attempt to download from HF GCS which is redundant, because user has already provided the generator from which the data should be drawn. 

If someone attempts to call `Dataset.from_generator` from an environment that doesn't have external internet access (for example internal production machine) and doesn't set `HF_DATASETS_OFFLINE=1`, this will result in process being stuck at building connection.

### Steps to reproduce the bug

```python
import datasets
def gen():
    for _ in range(100):
        yield {"text": "dummy text"}

dataset = datasets.Dataset.from_generator(gen)
```

A minimum example executed on any environment that doesn't have access to HF GCS can result in the error

### Expected behavior

`try_from_hf_gcs` should be set to False here https://github.com/huggingface/datasets/blob/c9c1166e1cf81d38534020f9c167b326585339e5/src/datasets/io/generator.py#L51

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-3.10.0-1160.90.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.12
- Huggingface_hub version: 0.17.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3

Comments:
Indeed, setting `try_from_gcs` to `False` makes sense for `from_generator`.

We plan to deprecate and remove `try_from_hf_gcs`  soon, as we can use Hub for file hosting now, but this is a good temporary fix.
TensorDataset format does not work with Trainer from transformers
State: closed

### Describe the bug

The model was built to do fine tunning on BERT model for relation extraction.

trainer.train() returns an error message ```TypeError: vars() argument must have __dict__ attribute``` when it has `train_dataset` generated from `torch.utils.data.TensorDataset`

However, in the document, the required data format is `torch.utils.data.TensorDataset`. 
![image](https://github.com/huggingface/datasets/assets/49014051/36fa34ac-3127-4c64-9580-9ab736136d83)

Transformers trainer is supposed to accept the train_dataset in the format of torch.utils.data.TensorDataset, but it returns error message *"TypeError: vars() argument must have __dict__ attribute"*

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-30-5df728c929a2> in <cell line: 1>()
----> 1 trainer.train()
      2 trainer.evaluate(test_dataset)

9 frames
/usr/local/lib/python3.10/dist-packages/transformers/data/data_collator.py in <listcomp>(.0)
    107 
    108     if not isinstance(features[0], Mapping):
--> 109         features = [vars(f) for f in features]
    110     first = features[0]
    111     batch = {}

TypeError: vars() argument must have __dict__ attribute
```

### Steps to reproduce the bug

Create train_dataset using `torch.utils.data.TensorDataset`, for instance,

```train_dataset = torch.utils.data.TensorDataset(train_input_ids, train_attention_masks, train_labels)```

Feed this `train_dataset` to your trainer and run trainer.train

```
trainer = Trainer(model,
                  training_args,
                  train_dataset=train_dataset,
                  eval_dataset=dev_dataset,
                  compute_metrics=compute_metrics,
                  )
```

### Expected behavior

Trainer should start training

### Environment info

It is running on Google Colab

- `datasets` version: 2.14.6
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.17.3
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

Comments:
I figured it out. I found that `Trainer` does not work with TensorDataset even though the document says it uses it. Instead, I ended up creating a dictionary and converting it to a dataset using `dataset.Dataset.from_dict()`.

I will leave this post open for a while. If someone knows a better approach, please leave a comment.

Only issues directly related to the HF datasets library should be reported here. ~So, I'm transferring this issue to the `transformers` repo.~ I'm not a `transformers` maintainer, so GitHub doesn't let me transfer it there :(. This means you need to do it manually.
with_format() function returns bytes instead of PIL images even when image column is not part of "columns"
State: closed

### Describe the bug

When using the with_format() function on a dataset containing images, even if the image column is not part of the columns provided in the function, its type will be changed to bytes.
Here is a minimal reproduction of the bug:
https://colab.research.google.com/drive/1hyaOspgyhB41oiR1-tXE3k_gJCdJUQCf?usp=sharing

### Steps to reproduce the bug

1. Load the image dataset
2. apply with_format(columns=["text"])
3. Check the type of images in the "image" column before and after applying with_format

### Expected behavior

The type should stay the same, but it does not

### Environment info

datasets==2.14.6


Comments:
Thanks for reporting! I've opened a PR with a fix.
Parquet size grows exponential for categorical data
State: closed

### Describe the bug

It seems that when saving a data frame with a categorical column inside the size can grow exponentially.

This seems to happen because when we save the categorical data to parquet, we are saving the data + all the categories existing in the original data. This happens even when the categories are not present in the original data.


### Steps to reproduce the bug

To reproduce the bug, it is enough to run this script:

```
import pandas as pd
import os

if __name__ == "__main__":
    for n in [10, 1e2, 1e3, 1e4, 1e5]:
        for n_col in [1, 10, 100, 1000, 10000]:
            input = pd.DataFrame([{"{i}": f"{i}_cat" for col in range(n_col)} for i in range(int(n))])
            input.iloc[0:100].to_parquet("a.parquet")
            for col in input.columns:
                input[col] = input[col].astype("category")
            input.iloc[0:100].to_parquet("b.parquet")
            a_size_mb = os.stat("a.parquet").st_size / (1024 * 1024)
            b_size_mb = os.stat("b.parquet").st_size / (1024 * 1024)
            print(f"{n} {n_col} {a_size_mb} {b_size_mb} {100*b_size_mb/a_size_mb:.2f}")
```

That produces this output:

<img width="464" alt="Screenshot 2023-10-31 at 11 25 25" src="https://github.com/huggingface/datasets/assets/82567957/2b8a9284-7f9e-4c10-a006-0a27236ebd15">



### Expected behavior

In my opinion either:

1. The two file should have (almost) the same size
2. There should be warning telling the user that such difference in size is possible

### Environment info

Python 3.8.18
pandas==2.0.3
numpy==1.24.4

Comments:
Wrong repo.
Add support for `Sequence(Audio/Image)` feature in `push_to_hub`
State: closed
Labels: description: New feature or request
name: enhancement

### Feature request

Allow for `Sequence` of `Image` (or `Audio`) to be embedded inside the shards.

### Motivation

Currently, thanks to #3685, when `embed_external_files` is set to True (which is the default) in `push_to_hub`, features of type `Image` and `Audio` are embedded inside the arrow/parquet shards, instead of only storing paths to the files.

I've noticed that this behavior does not extend to `Sequence` of `Image`, when working with a [dataset of timelapse images](https://huggingface.co/datasets/1aurent/Human-Embryo-Timelapse).

### Your contribution

I'll submit a PR if I find a way to add this feature

Comments:
This issue stems from https://github.com/huggingface/datasets/blob/6d2f2a5e0fea3827eccfd1717d8021c15fc4292a/src/datasets/table.py#L2203-L2205

I'll address it as part of https://github.com/huggingface/datasets/pull/6283.

In the meantime, this should work

```python
import pyarrow as pa
from datasets import Image

dataset = dataset.with_format("arrow")

def embed_images(pa_table):
    images_arr = pa.chunked_array(
        [
            pa.ListArray.from_arrays(chunk.offsets, Image().embed_storage(chunk.values), mask=chunk.is_null())
            for chunk in pa_table["images"].chunks
        ]
    )
    return pa_table.set_column(pa_table.schema.get_field_index("images"), "images", images_arr)

dataset = dataset.map(embed_images, batched=True)

dataset = dataset.with_format("python")

dataset.push_to_hub(...)
```
Stuck in "Resolving data files..."
State: open

### Describe the bug

I have an image dataset with 300k images, the size of image is 768 * 768.

When I run `dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')` in second time, it takes 50 minutes to finish "Resolving data files" part, what's going on in this part?

From my understand, after Arrow files been created in the first run, the second run should not take time longer than one or two minutes.

### Steps to reproduce the bug

# Run following code two times
dataset = load_dataset("imagefolder", data_dir="/path/to/img_dir", split='train')

### Expected behavior

Fast dataset building

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- Huggingface_hub version: 0.17.3
- PyArrow version: 10.0.1
- Pandas version: 1.5.3

Comments:
Most likely, the data file inference logic is the problem here.

You can run the following code to verify this:
```python
import time
from datasets.data_files import get_data_patterns
start_time = time.time()
get_data_patterns("/path/to/img_dir")
end_time = time.time()
print(f"Elapsed time: {end_time - start_time:.2f}s")
```
 
We plan to optimize this for the next version (or version after that). In the meantime, specifying the split patterns manually should give better performance:
```python
ds = load_dataset("imagefolder", data_files={"train": "path/to/img_dir/train/**", ...}, split="train")
```

Hi, @mariosasko, you are right; data file inference logic is extremely slow.

I have done a similar test, that is I modify the source code of datasets/load.py to measure the cost of two suspicious operations:
```python
def get_module(self) -> DatasetModule:
        base_path = Path(self.data_dir or "").expanduser().resolve().as_posix()
        start = time.time()
        patterns = sanitize_patterns(self.data_files) if self.data_files is not None else get_data_patterns(base_path)
        print(f"patterns: {time.time() - start}")
        start = time.time()
        data_files = DataFilesDict.from_patterns(
            patterns,
            download_config=self.download_config,
            base_path=base_path,
        )
        print(f"data_files: {time.time() - start}")
```
It gaves:
patterns: 3062.2050700187683
data_files: 413.9576675891876

Thus, these two operations contribute to almost all of load time. What's going on in them?

Furthermore, what's my current workaround about this problem? Should I save it by `save_to_disk()` and load dataset through `load_from_disk`?

were you able to solve this issue?, I am facing the same issue

Still suffering from this issue. For me, I cannot download Emilia dataset, and stucked at 'Resolving data files' forever.
``` e = load_dataset('amphion/Emilia-Dataset', token=my_token)
Resolving data files:   0%|                                                                                  | 1/4343 [00:00<11:27,  6.32it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████| 4343/4343 [00:01<00:00, 3844.15it/s]
```