Almost identical datasets, huge performance difference
Opened by eli-osherovich (open)

### Describe the bug

I am struggling to understand (huge) performance difference between two datasets that are almost identical. 

### Steps to reproduce the bug

# Fast (normal) dataset speed:
```python
import cv2
from datasets import load_dataset
from torch.utils.data import DataLoader

dataset = load_dataset("beans", split="train")

for x in DataLoader(dataset.with_format("torch"), batch_size=16, shuffle=True, num_workers=8):
    pass
```

The above pass over the dataset takes about 1.5 seconds on my computer. 
However, if I re-create (almost) the same dataset, the sweep takes HUGE amount of time: 15 minutes. Steps to reproduce:

```python
def transform(example):
    example["image2"] = cv2.imread(example["image_file_path"])
    return example

dataset2 = dataset.map(transform, remove_columns=["image"])

for x in DataLoader(dataset2.with_format("torch"), batch_size=16, shuffle=True, num_workers=8):
    pass

```

### Expected behavior

Same timings

### Environment info

python==3.10.9
datasets==2.10.1

Comments:
Do I miss something here?

Hi! 

The first dataset stores images as bytes (the "image" column type is `datasets.Image()`) and decodes them as `PIL.Image` objects and the second dataset stores them as variable-length lists (the "image" column type is `datasets.Sequence(...)`)), so I guess going from `arrow bytes -> NumPy -> decoding as PIL.Image -> PyTorch` is faster than going from `arrow list -> NumPy -> PyTorch`. 

To store image bytes in the second example, you can do the following:

```python
def transform(example):
    example["image2"] = cv2.imread(example["image_file_path"])
    return example

features = dataset.features.copy()
del features["image"]
features["image2"] = datasets.Image()
dataset2 = dataset.map(transform, remove_columns=["image"], features=features)

for x in DataLoader(dataset2.with_format("torch"), batch_size=16, shuffle=True, num_workers=8):
    pass
```

Thanks, @mariosasko. I could not understand why a (decoded) sequence should be MUCH slower than an encoded image (that must be decoded every time). At any rate, I tried you suggestion. It made the `map` step to run extremely slow (consumes all the 16GB of memory and starts swapping)

I tried also the easiest (as I see it) scenario, where images are kept as bytes, but it made things even worse: not only it was extremely slow, but also crashes

```python

def transform(example):
    example["image2"] = cv2.imread(example["image_file_path"]).tobytes()
    return example

dataset2 = dataset.map(transform, remove_columns=["image"])

for x in DataLoader(dataset2.with_format("torch"), batch_size=16, shuffle=True, num_workers=8):
    pass


Resource temporarily unavailable (src/thread.cpp:269)
Output exceeds the size limit. Open the full output data in a text editor
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
File ~/virtenvs/py310/lib/python3.10/site-packages/torch/utils/data/dataloader.py:1133, in _MultiProcessingDataLoaderIter._try_get_data(self, timeout)
   1132 try:
-> 1133     data = self._data_queue.get(timeout=timeout)
   1134     return (True, data)

File ~/virtenvs/py310/lib/python3.10/multiprocessing/queues.py:113, in Queue.get(self, block, timeout)
    112 timeout = deadline - time.monotonic()
--> 113 if not self._poll(timeout):
    114     raise Empty

File ~/virtenvs/py310/lib/python3.10/multiprocessing/connection.py:257, in _ConnectionBase.poll(self, timeout)
    256 self._check_readable()
--> 257 return self._poll(timeout)

File ~/virtenvs/py310/lib/python3.10/multiprocessing/connection.py:424, in Connection._poll(self, timeout)
    423 def _poll(self, timeout):
--> 424     r = wait([self], timeout)
    425     return bool(r)

File ~/virtenvs/py310/lib/python3.10/multiprocessing/connection.py:931, in wait(object_list, timeout)
    930 while True:
--> 931     ready = selector.select(timeout)
    932     if ready:
...
-> 1146     raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
   1147 if isinstance(e, queue.Empty):
   1148     return (False, None)

RuntimeError: DataLoader worker (pid(s) 195393) exited unexpectedly
Resource temporarily unavailable (src/thread.cpp:269)
Resource temporarily unavailable (src/thread.cpp:269)
Resource temporarily unavailable (src/thread.cpp:269)
Resource temporarily unavailable (src/thread.cpp:269)
Resource temporarily unavailable (src/thread.cpp:269)
```


Correction: the `beans` dataset stores the image file paths, not the bytes.

For your use case, I think it makes  more sense to use `with_tranform` than `map` and lazily decode images with `cv2.imread` when indexing an example/batch:
```python
import cv2

def transform(batch):
    batch["image2"] = np.stack([cv2.imread(image_file_path) for image_file_path in batch["image_file_path"]])
    return batch

dataset = dataset.with_transform(transform)
```


This is incorrect.

Did you try to run it?  dataset[0] returns a tensor of numbers. dataset2[0]
returns the same tensor, but after a few long seconds.  Looping over a
thousand of images cannot take 15 minutes.

On Fri, 24 Mar 2023 at 19:28 Mario Šaško ***@***.***> wrote:

> Correction: the beans dataset stores the image file paths, not the bytes.
>
> For your use case, I think it makes more sense to use with_tranform than
> map and lazily decode images with cv2.imread when accessing an
> example/batch:
>
> import cv2
> def transform(batch):
>     batch["image2"] = np.stack([cv2.imread(image_file_path) for image_file_path in batch["image_file_path"]])
>     return batch
> dataset = dataset.with_transform(transform)
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/5669#issuecomment-1483084347>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AASS73SHRWXIQX6SCYCJ7ITW5XDUDANCNFSM6AAAAAAWFSHWEM>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


I updated the transform with the NumPy -> PyTorch conversion.

I'm sharing the entire code:
```python
import cv2
import numpy as np
import datasets
import torch
from datasets import load_dataset
from torch.utils.data import DataLoader

dataset = load_dataset("beans", split="train")

def transform(batch):
    # # Pillow decodes as RGB
    # batch["image"] = torch.stack([torch.from_numpy(cv2.cvtColor(cv2.imread(image_file_path), cv2.COLOR_BGR2RGB)) for image_file_path in batch["image_file_path"]])
    batch["image"] = torch.stack([torch.from_numpy(cv2.imread(image_file_path)) for image_file_path in batch["image_file_path"]])
    batch["labels"] = torch.tensor(batch["labels"])
    return batch

dataset2 = dataset.cast_column("image", datasets.Image(decode=False)).with_transform(transform)

for x in DataLoader(dataset2, batch_size=16, shuffle=True, num_workers=8):
    pass
```

This code is ≈ 10% faster on my machine than the default decoding with Pillow and `.with_format("torch")`.

Thanks, @mariosasko 
My question remain unanswered though. Why is the `map`ed dataset  so slow? My understanding is that a dataset of numpy arrays should be must faster than a dataset that has to decode images into numpy arrays every time one accesses an item.