DataLoader num_workers does not help

username: jfryprc
cooked: <p>Increasing <code>num_workers</code> for <code>DataLoader</code> appears to be of no help, and is always worse than <code>num_workers = 0</code> across batch sizes 32, 64, 128, 256 (<code>pin_memory</code> has no effect). Architecture is MacBook Pro, M3 Max, 36GB. Does this make sense?</p>
<p>Here’s the test code (using built-in dataset and DataLoader, where FashionMNIST was previously downloaded on prior run). I saw similar results - actually much worse -  on my own dataset (which was backed by a pandas DataFrame of image file paths).</p>
<pre><code class="lang-auto">from time import time
import multiprocessing as mp
from torch.utils.data import DataLoader
import torchvision
import torchvision.transforms.v2 as v2

epochs_per_inner_loop = 3

dataset = torchvision.datasets.FashionMNIST(
    root='data',
    train=True,
    download=True,
    transform=v2.ToImage()
)


for batch_size in [32,64,128,256]:
    print(f'batch_size = {batch_size}')

    for num_workers in range(0, mp.cpu_count()+2, 2):
        
        dataloader = DataLoader(dataset, batch_size=batch_size, num_workers=num_workers)

        t_start = time()
        for _ in range(epochs_per_inner_loop):
            for _, _ in enumerate(dataloader, 0):
                pass
        
        t_end = time()
        t_elapsed = t_end - t_start
        t_per_epoch = t_elapsed / epochs_per_inner_loop

        print(f'  {t_per_epoch:.1f} sec, {num_workers} workers')
        
    print('--------------------------')
</code></pre>
<p>The results are as follows:</p>
<p>batch_size = 32<br>
0.9 sec, 0 workers<br>
1.9 sec, 2 workers<br>
1.8 sec, 4 workers<br>
1.8 sec, 6 workers<br>
1.9 sec, 8 workers<br>
2.1 sec, 10 workers<br>
2.4 sec, 12 workers<br>
2.9 sec, 14 workers</p>
<p>batch_size = 64<br>
1.0 sec, 0 workers<br>
1.8 sec, 2 workers<br>
1.7 sec, 4 workers<br>
1.7 sec, 6 workers<br>
1.9 sec, 8 workers<br>
2.0 sec, 10 workers<br>
2.4 sec, 12 workers<br>
2.9 sec, 14 workers</p>
<p>batch_size = 128<br>
1.0 sec, 0 workers<br>
1.7 sec, 2 workers<br>
1.7 sec, 4 workers<br>
1.7 sec, 6 workers<br>
1.8 sec, 8 workers<br>
2.1 sec, 10 workers<br>
2.4 sec, 12 workers<br>
2.8 sec, 14 workers</p>
<p>batch_size = 256<br>
1.0 sec, 0 workers<br>
1.6 sec, 2 workers<br>
1.6 sec, 4 workers<br>
1.7 sec, 6 workers<br>
1.8 sec, 8 workers<br>
2.0 sec, 10 workers<br>
2.3 sec, 12 workers<br>
2.9 sec, 14 workers</p>

username: jfryprc
cooked: <p>I think I understand the issue here (basically user error): FashionMNIST comprises 60k samples in a single file so we are I/O limited in reading all samples from same file. This is further compounded by the fact that Dataset transform is the trivial <code>v2.ToImage()</code> so there is no real benefit to parallelization.</p>
<p>Using my own (small 1k dataset) with non-trivial transform (<code>Compose</code> of RandomRotation, RandomPerspective, ColorJitter, RandomEqualize, RandomResizedCrop, Normalize) on images of varying sizes I see results I would expect:</p>
<p>22.5 sec, 0 workers<br>
16.6 sec, 2 workers<br>
9.8 sec, 4 workers<br>
7.9 sec, 6 workers<br>
6.5 sec, 8 workers<br>
6.4 sec, 10 workers<br>
6.2 sec, 12 workers<br>
6.7 sec, 14 workers</p>