Speed up in dataset, but not in dataloader for contiguous tensors

korotaS: <p>I am trying to speed up a training process for some model, it includes pretty default image loading and converting to tensor. After profiling I’ve noticed that big portion of the preprocessing takes (obviously) converting image in numpy array to a contiguous pytorch tensor and I’ve seen some alternatives how I can achieve that (for example <a href="https://github.com/open-mmlab/mmpretrain/blob/17a886cb5825cd8c26df4e65f7112d404b99fe12/mmpretrain/datasets/transforms/formatting.py#L106" rel="noopener nofollow ugc">here</a>). So for testing I’ve created 2 converting methods and a simple dataset (all the images are between 100 and 300 pixels width/height so resizing takes not much time):</p>
<pre data-code-wrap="python"><code class="lang-python">from time import time 

import cv2
import numpy as np
import torch
from torch.utils.data import DataLoader, Dataset


def cv2_to_tensor_1(image):
    return torch.from_numpy(np.ascontiguousarray(image.transpose(2, 0, 1)))

def cv2_to_tensor_2(image):
    return torch.from_numpy(image).permute(2, 0, 1).contiguous()

class MyDataset(Dataset):
    def __init__(self, paths, transform):
        super().__init__()
        self.paths = paths
        self.transform = transform

    def __len__(self):
        return len(self.paths)

    def __getitem__(self, i):
        image = cv2.imread(self.paths[i])
        image = cv2.resize(image, (224, 224))
        return self.transform(image)
</code></pre>
<p>And I tested this 2 methods inside dataset and inside dataloader (with multiple workers and with warmup to set up persistent workers) with different batch sizes using this piece of code:</p>
<pre data-code-wrap="python"><code class="lang-python">n_images = 1000
for bs in [100, 1000]:
    for method in [cv2_to_tensor_1, cv2_to_tensor_2]:
        dataset = MyDataset(all_paths[:n_images], method)
        dataloader = DataLoader(
            dataset, batch_size=bs, num_workers=6, persistent_workers=True, pin_memory=True
        )
        # warmup
        for _ in range(3):
            for item in dataset:
                pass
            for batch in dataloader:
                pass
                
        start = time()
        for item in dataset:
            pass
        dataset_time = time() - start
        start = time()
        for batch in dataloader:
            pass
        dataloader_time = time() - start
        print(f'BS: {bs}, DS: {dataset_time:.3f}, DL: {dataloader_time:.3f}')
</code></pre>
<p>And got some strange results:</p>
<pre><code class="lang-plaintext">BS: 100, DS: 0.517, DL: 0.190
BS: 100, DS: 4.413, DL: 0.476
BS: 1000, DS: 0.538, DL: 0.788
BS: 1000, DS: 3.705, DL: 1.009
</code></pre>
<p>I can clearly see that the second method is much slower in dataset (about 7-9 times slower), however there is no such speed up when using dataloader, especially with large batch size which I am using during training.</p>
<p>Perhaps the second method is performing better with multiprocessing because it is using only pytorch methods? Or maybe the overhead in multiprocess dataloader (with collate_fn, process initialization etc) is so big so the speed up is not that noticable? Or is it expected behavior?</p>