DataLoader wrap-up time kills performance on both Intel and M1 Macs, but not on Colab

username: kbwiley
cooked: <p>On two very different Mac architectures (i7 and M1), both running torch 2.1.2, I’m seeing a 5s/worker delay in wrapping up a DataLoader batch loop (which costs 40-50s per epoch of completely wasted time). But on Colab, running torch 2.1.0+cu121, I don’t see this effect.</p>
<p>This absolutely kills training/testing because it can add 5s/worker to each epoch for both the training loop and the testing loop. That is time that has nothing to do with the GPU (MPS on Mac), nothing even to do with NN training at all. And it also isn’t time spent actually loading data from disk. That went quickly during the iteration over the DataLoader. This problem occurs after the entire iteration is complete and the for loop is just trying to “get out”. It sits there for a huge amount of time, presumably deconstructing some massive amount of state – but it doesn’t do that on Colab. This makes PyTorch all but useless on the Mac. 5-epoch trainings that take 10s on Colab end up taking over 200s on both Macs. I don’t know if the problem is the slight difference in torch version or something to do with the Mac. It has nothing to do with the recent MPS Metal GPU support though, since I am getting the same result on an i7 Intel iMac (plus the code clearly makes no use of the MPS).</p>
<p>See attached plots.</p>
<p>Any ideas?</p>
<p>Here’s sample output from a single experiment, just to show what the output looks like:</p>
<pre><code class="lang-auto">Batch size: 32    Num workers: 10
Creating DataLoader with batch size 32 and 10 workers.
Iterating over DataLoader

Batch  1 time:               3.01242
Batch  2 time:               0.00620
Batch  3 time:               0.00006
Batch  4 time:               0.00003
Batch  5 time:               0.00003
Batch  6 time:               0.00005
Batch  7 time:               0.00005
Batch  8 time:               0.00004
Enumeration wrap-up time:   40.02002
Total time:                 43.03889
</code></pre>
<p>The code to reproduce the experiments is below. It uses a dataset from a popular Udemy course.</p>
<pre><code class="lang-auto">import os
from timeit import default_timer
from pathlib import Path
import pandas as pd
import numpy as np

import torch
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

def download_data(image_path):
    if image_path.is_dir():
        print(f"{image_path} directory exists.")
    else:
        print(f"Did not find {image_path} directory, creating one...")
        image_path.mkdir(parents=True, exist_ok=True)
        
        # Download pizza, steak, sushi data
        # 16.2 MB dataset comprising 225 and 75 train/test images
        with open(data_path / "pizza_steak_sushi.zip", "wb") as f:
            request = requests.get("https://github.com/mrdbourke/pytorch-deep-learning/raw/main/data/pizza_steak_sushi.zip")
            print("Downloading pizza, steak, sushi data...")
            f.write(request.content)
    
        # Unzip pizza, steak, sushi data
        with zipfile.ZipFile(data_path / "pizza_steak_sushi.zip", "r") as zip_ref:
            print("Unzipping pizza, steak, sushi data...") 
            zip_ref.extractall(image_path)

# data_path = Path("data/")
image_path = data_path / "pizza_steak_sushi"
train_dir = image_path / "train"

download_data(image_path)

# Create simple transform
simple_transform = transforms.Compose([ 
    transforms.Resize((64, 64)),
    transforms.ToTensor(),
])

# Load and transform data
from torchvision import datasets
train_data_simple = datasets.ImageFolder(root=train_dir, transform=simple_transform)

# Setup batch size and number of workers 
BATCH_SIZE = 32
NUM_WORKERS = os.cpu_count()

rows = []
for batch_size in [8, 16, 32]:
    for num_workers in range(1, NUM_WORKERS + 1):
        print("_" * 100)
        print(f"Batch size: {batch_size:&gt;2}    Num workers: {num_workers:&gt;2}")
        
        # Create DataLoader
        print(f"Creating DataLoader with batch size {batch_size} and {num_workers} workers.")
        train_dataloader_simple = DataLoader(train_data_simple, 
                                             batch_size=batch_size, 
                                             shuffle=True, 
                                             num_workers=num_workers)
    
        # Iterate over the batch without doing anything with the images
        print("Iterating over DataLoader")
        times = [default_timer()]
        for batch, (X, y) in enumerate(train_dataloader_simple):
            times.append(default_timer())
            # print(f"Batch {batch + 1}")
        times.append(default_timer())
        
        print()
        for ti, t in enumerate(times):
            if ti &gt; 0:
                et = t - times[ti - 1]
        
                if ti &lt; len(times) - 1:
                    print(f"Batch {ti:&gt;2} time:             {et:9.5f}")
                else:
                    print(f"Enumeration wrap-up time:  {et:9.5f}")
        et = times[-1] - times[0]
        print(f"Total time:                {et:9.5f}")

        mean_batch_time_wo_first_batch = np.mean([times[i] - times[i-1] for i in range(2, len(times)-1)])

        row = [batch_size, num_workers, times[1] - times[0], mean_batch_time_wo_first_batch, times[-1] - times[-2], times[-1] - times[0]]
        rows.append(row)

df = pd.DataFrame(rows, columns=["Batch_size", "Num_workers", "1st_batch_secs", "mean_2nd_onward_batches_secs", "Wrap-up_secs", "Total_secs"])
display(df)

print()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4b488bed2ff82ae790ed58c41281463e42f54cd.png" data-download-href="https://discuss.pytorch.org/uploads/default/e4b488bed2ff82ae790ed58c41281463e42f54cd" title="dataloader_speed_test_m1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4b488bed2ff82ae790ed58c41281463e42f54cd_2_492x500.png" alt="dataloader_speed_test_m1" data-base62-sha1="wDdEFm79th5Y6zplyYi5pCEOA1f" width="492" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4b488bed2ff82ae790ed58c41281463e42f54cd_2_492x500.png, https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4b488bed2ff82ae790ed58c41281463e42f54cd.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4b488bed2ff82ae790ed58c41281463e42f54cd.png 2x" data-dominant-color="F0F1EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">dataloader_speed_test_m1</span><span class="informations">697×708 94.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f4620a179c6c9ed43f7ffa9ab269051955d2684.png" data-download-href="https://discuss.pytorch.org/uploads/default/3f4620a179c6c9ed43f7ffa9ab269051955d2684" title="dataloader_speed_test_imac"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f4620a179c6c9ed43f7ffa9ab269051955d2684_2_490x500.png" alt="dataloader_speed_test_imac" data-base62-sha1="91KkkYmal3Q69SMWjxP0MSNLMkk" width="490" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3f4620a179c6c9ed43f7ffa9ab269051955d2684_2_490x500.png, https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f4620a179c6c9ed43f7ffa9ab269051955d2684.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f4620a179c6c9ed43f7ffa9ab269051955d2684.png 2x" data-dominant-color="F3F3F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">dataloader_speed_test_imac</span><span class="informations">690×704 79.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cda193694de51bc0a234ed619b6228570f25fe32.png" data-download-href="https://discuss.pytorch.org/uploads/default/cda193694de51bc0a234ed619b6228570f25fe32" title="dataloader_speed_test_colab"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cda193694de51bc0a234ed619b6228570f25fe32_2_536x500.png" alt="dataloader_speed_test_colab" data-base62-sha1="tl63tsI5nF3gkhGUWhaug2td18m" width="536" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cda193694de51bc0a234ed619b6228570f25fe32_2_536x500.png, https://discuss.pytorch.org/uploads/default/original/3X/c/d/cda193694de51bc0a234ed619b6228570f25fe32.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/d/cda193694de51bc0a234ed619b6228570f25fe32.png 2x" data-dominant-color="F7F8F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">dataloader_speed_test_colab</span><span class="informations">699×651 46.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: kbwiley
cooked: <p>I found the solution in this past discussion:</p>
<aside class="quote quote-modified" data-post="1" data-topic="131204">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/adamjstewart/48/19125_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/data-loader-multiprocessing-slow-on-macos/131204">Data loader multiprocessing slow on macOS</a> 
  </div>
  <blockquote>
    I’ve noticed a significant slowdown when using num_workers &gt; 0 on macOS for Python 3.8+. With the following benchmarking script: 
import sys
import time

from torch.utils.data import DataLoader, Dataset


class MyDataset(Dataset):
    def __getitem__(self, i):
        return i

    def __len__(self):
        return 5


if __name__ == '__main__':
    ds = MyDataset()
    dl = DataLoader(ds, num_workers=int(sys.argv[1]))

    start = time.time()
    for i in dl:
        continue
    end = time.tim…
  </blockquote>
</aside>

<p>In short:</p>
<pre><code class="lang-auto">DataLoader(...
    multiprocessing_context="forkserver",
    persistent_workers=True)
</code></pre>
<p><code>multiprocessing_context="forkserver"</code> eliminates the 5s-per-worker hangup when exiting (tearing down) the DataLoader iteration. This is only needed on a Mac, not Linux, because the system level mp context differs on those platforms and the default PyTorch value uses the system value.</p>
<p><code>persistent_workers</code> offers further gains, mostly unrelated to my original post above but relevant in multi-epoch runs, on all epochs after the first one.</p>
<p>I realize the discussion of <code>multiprocessing_context</code> indicates that this problem occurs because Linux and Mac use different multiprocessing contexts (spawn and fork, and forkserver, I guess) but I’m still curious why PyTorch doesn’t use the “correct” value of <code>multiprocessing_context</code> on a per-architecture basis. Why doesn’t it default to “forkserver” when running on a Mac?</p>
<p>Likewise, I’m curious why the default value of <code>persistent_workers</code> is the less performant value. I presume there is a good reason to default it to False and require the user to set to True if and only if they want it to be True, but I’m unclear when I would ever want it to be False. Why is that the better default value instead of the other way around (default to the more performant value and require the user to override it if they see reason to do so)? I’m new to PyTorch though, so I’m sure I’m just not understanding the parameter well enough to realize there is a perfectly good explanation.</p>
<p>Cheers!</p>