Loss.item() Causes the wrong time consumption

username: aonrdbit
cooked: <p>Hello everyone!</p>
<p>I am using NCCL as the backend for the distributed training of PyTorch. I used four compute nodes. But there was a strange problem in calculating the time consumption.</p>
<p>Part of the code is as follows.</p>
<pre><code class="lang-auto">def average_gradients(model):
    size = float(dist.get_world_size())
    for param in model.parameters():
        dist.all_reduce(param.grad.data, op=dist.ReduceOp.SUM)
        param.grad /= size

 for epoch in range(args.epoch):
        begin_time = time.time()
        print('epoch', str(epoch))
        model.train()
        loss_sum = 0
        time_sum = 0
        for batch_idx,(images,labels) in enumerate(trainloader):
            batch_1 = time.time()
            images = images.to(device)
            labels = labels.to(device)
            optimizer.zero_grad()
            output = model(images)
            loss = criterion(output,labels)
            loss_sum += loss.item()
            loss.backward()
            average_gradients(model)
            optimizer.step()
            batch_2 = time.time()
            print("batch:{}\tloss:{}".format(batch_idx, loss.item()))
            time_sum += (batch_2 - batch_1)
            print(time_sum)
        end_time = time.time()
        print("epoch:{}\tloss:{}\ttime:{}".format(epoch, loss_sum / len(trainloader), end_time - begin_time))
</code></pre>
<p>In each epoch, the time_sum is 6.3s, but the (end_time-begin_time) is 18.8s，which is a big time difference. I guess that loss.item() is causing this problem, but I’m still confused. I would be very grateful if you could give me some advice.</p>

username: ptrblck
cooked: <p><code>loss.item()</code> will synchronize the device for you, as the value stored on the GPU has to be moved to the host.<br>
Since CUDA operations are executed asynchronously you would need to synchronize the code via <code>torch.cuda.synchronize()</code> before starting and stopping the timers. In your current code the profiling is thus invalid.</p>

username: aonrdbit
cooked: <p>Thank you for your reply. But I still have a question.<br>
Why does this time mismatch phenomenon not occur at a single node when training is performed? When I run similar code on a single node, the <code>time_sum</code> approximately equals to <code>(end_time-begin_time)</code>.<br>
Does <code>torch.cuda.synchronize</code> relates to distributed training?<br>
Looking forward to your reply.</p>

username: ptrblck
cooked: <p>It’s hard to tell as it depends if and when the CPU run run ahead. I.e. your CPU might be blocked by something in the distributed run which would slow it down and could cause the times to approx. match.</p>

username: aonrdbit
cooked: <p>I see. Thank you for your help!</p>

username: aonrdbit
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I’m sorry to bother you again.<br>
I found another phenomenon under this topic.<br>
If I add <code>multiprocessing.set_start_method('spawn')</code> at the code beginning, there will still be a time difference between <code>time_sum</code> and <code>(end_time-begin_time)</code> even if I used torch.cuda.synchronize().<br>
I’m very confused. Could you give me some advice? Thank you!</p>

username: ptrblck
cooked: <p>I don’t know the reason for it. Could you post a minimal, executable code snippet which would show this behavior?</p>

username: aonrdbit
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your quick reply!<br>
I used three nodes to do the experiment of distributed training.</p>
<pre><code class="lang-auto">#!/usr/bin/env python
import argparse
import os
import torch
import torch.distributed as dist
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torch.backends.cudnn as cudnn
from math import ceil
from random import Random
from torch.multiprocessing import Process
from torch.autograd import Variable
from torchvision import datasets, transforms
# from torchvision import models
from models import resnet
import time
from multiprocessing import Pool
import multiprocessing

def average_gradients(model):
    size = float(dist.get_world_size())
    for param in model.parameters():
        dist.all_reduce(param.grad.data, op=dist.ReduceOp.SUM)
        param.grad /= size

def test(model):
    correct_sum = 0
    model.eval()
    with torch.no_grad():
        for batch_idx,(images,labels) in enumerate(testloader):
            images, labels = images.to(device),labels.to(device)
            output = model(images)
            pred = output.argmax(dim=1, keepdim=True)
            pred = pred.to(device)
            correct = pred.eq(labels.view_as(pred)).sum().item()
            correct_sum += correct
    print("Acc:{}".format(correct_sum/len(testloader.dataset)))

if __name__ == '__main__':

    parser = argparse.ArgumentParser()
    parser.add_argument('--rank', type=int)
    parser.add_argument('--world_size', type=int)
    parser.add_argument('--batch_size', type=int, default=256)
    parser.add_argument('--epoch', type=int, default=100)
    parser.add_argument('--lr', type=float, default=0.1)
    args = parser.parse_args()
    multiprocessing.set_start_method("spawn")
    os.environ["MASTER_ADDR"] = "master_ip"
    os.environ["MASTER_PORT"] = "master_port"
    dist.init_process_group('nccl',rank=args.rank, world_size=args.world_size)
    print("init end...")
    cudnn.benchmark = True
    torch.manual_seed(1000)
    torch.cuda.manual_seed(2000)
    model = resnet.ResNet18()
    device = torch.device(f"cuda:{0}" if torch.cuda.is_available() else "cpu")
    model.to(device)
    print("model end...")
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.SGD(model.parameters(), args.lr, momentum=0.9, weight_decay=1e-4)
    transform_train = transforms.Compose([
        transforms.RandomCrop(32, padding=4),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
    ])
    transform_test = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
    ])
    trainset = datasets.CIFAR10(
        root='./data', train=True, download=True, transform=transform_train)
    sampler = torch.utils.data.DistributedSampler(trainset,num_replicas=args.world_size,rank=args.rank)
    trainloader = torch.utils.data.DataLoader(
        trainset, batch_size=args.batch_size, shuffle=(sampler is None), sampler=sampler, num_workers=2)
    testset = datasets.CIFAR10(
        root='./data', train=False, download=True, transform=transform_test)
    testloader = torch.utils.data.DataLoader(
        testset, batch_size=args.batch_size, shuffle=True, num_workers=2)

    for epoch in range(args.epoch):
        begin_time = time.time()
        print('epoch', str(epoch))
        model.train()
        loss_sum = 0
        time_sum = 0
        for batch_idx,(images,labels) in enumerate(trainloader):
            torch.cuda.synchronize()
            batch_1 = time.time()
            images = images.to(device)
            labels = labels.to(device)
            optimizer.zero_grad()
            output = model(images)
            loss = criterion(output,labels)
            loss_sum += loss.item()
            loss.backward()
            average_gradients(model)
            optimizer.step()
            torch.cuda.synchronize()
            batch_2 = time.time()
            time_sum += (batch_2 - batch_1)
        end_time = time.time()
        print(f"time_sum:{time_sum}")
        print(f"end_time-begin_time:{end_time-begin_time}")
        test(model)
        time.sleep(2)
</code></pre>
<p>The experimental results are as follows:<br>
node1:  python3 main.py --rank 0 --world_size 3<br>
node2:  python3 main.py --rank 1 --world_size 3<br>
node3:  python3 main.py --rank 2 --world_size 3<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d152603c5e7021b21e7e8d6cf32a65adc8bbdc65.png" data-download-href="https://discuss.pytorch.org/uploads/default/d152603c5e7021b21e7e8d6cf32a65adc8bbdc65" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/1/d152603c5e7021b21e7e8d6cf32a65adc8bbdc65.png" alt="image" data-base62-sha1="tRKhpRkC520TM9THwRxIHIO1CTz" width="365" height="500" data-dominant-color="1C1C1C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">555×759 18.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If I delete the <code>multiprocessing.set_start_method</code>，the <code>time_sum</code> and <code>(end_time-begin_time)</code> are similar.</p>
<p>Looking forward to your reply.</p>

username: ptrblck
cooked: <p>Thanks for the code. I guess the teardown of the processes of the <code>DataLoader</code>s seems to be different, but that’s just my guess and you could try to create a profile to see which operations are executed between these timers.</p>

username: aonrdbit
cooked: <p>Thank you for your reply.<br>
When I set <code>num_workers</code> in <code>DataLoader</code> parameters to 1, the time difference still exists.<br>
By the way, how to create a profile to see which operations are executed between these timers?  I’ve never done this before.</p>

username: ptrblck
cooked: <p><code>num_workers=1</code> would still spawn a new process while <code>num_workers=0</code> would use the main process for the data loading.<br>
You could use the <a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html">PyTorch profiler</a> and check the timeline or e.g. Nsight Systems.</p>