PyTorch backward is too slow

111513: <p>Hi, I’m doing a profile of the pytorch training process (with 1*V100, training ResNet18 on ImageNet), and I find that  backward is really slow (as shown in the figure, 12ms with forward, but 300ms with backward, other batchs are the same)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/469b62abad8f0a09a3ceab3954f98d7ac0d7568d.png" data-download-href="https://discuss.pytorch.org/uploads/default/469b62abad8f0a09a3ceab3954f98d7ac0d7568d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/469b62abad8f0a09a3ceab3954f98d7ac0d7568d_2_690x216.png" alt="image" data-base62-sha1="a4CkKwE7N6bU0oUCWXbXnpfIVRz" width="690" height="216" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/469b62abad8f0a09a3ceab3954f98d7ac0d7568d_2_690x216.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/469b62abad8f0a09a3ceab3954f98d7ac0d7568d_2_1035x324.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/469b62abad8f0a09a3ceab3954f98d7ac0d7568d_2_1380x432.png 2x" data-dominant-color="E1E0E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1884×591 61.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>this is my code</p>
<pre><code class="lang-auto">    nvtx.range_push("Epoch: " + str(epoch))
    nvtx.range_push("Batch 0")
    nvtx.range_push("Load Data")
    for i, (input_data, target) in enumerate(train_loader):
        if(profile and i==300):
            exit(0)
        # measure data loading time
        data_time.update(time.time() - end)

        target = target.cuda(non_blocking=True)
        input_data = input_data.cuda(non_blocking=True)
        nvtx.range_pop()

        # compute output
        nvtx.range_push("Forward")
        output = model(input_data)
        nvtx.range_pop()

        # compute gradient and do SGD step
        nvtx.range_push("Backward")
        optimizer.zero_grad()
        loss = criterion(output, target)
        # measure accuracy and record loss
        prec1, prec5 = accuracy(output, target, topk=(1, 5))
        losses.update(loss.item(), input_data.size(0))
        top1.update(prec1[0], input_data.size(0))
        top5.update(prec1[0], input_data.size(0))
        loss.backward()
        nvtx.range_pop()

        nvtx.range_push("SGD")
        optimizer.step()
        nvtx.range_pop()

        # measure elapsed time
        batch_time.update(time.time() - end)
        end = time.time()
        nvtx.range_pop()
        if i % print_freq == 0:
            print('Epoch: [{0}][{1}/{2}]\t'
                  'Time {batch_time.val:.3f} ({batch_time.avg:.3f})\t'
                  'Data {data_time.val:.3f} ({data_time.avg:.3f})\t'
                  'Loss {loss.val:.4f} ({loss.avg:.4f})\t'
                  'Prec@1 {top1.val:.3f} ({top1.avg:.3f})\t'
                  'Prec@5 {top5.val:.3f} ({top5.avg:.3f})\t'.format(
                epoch, i, len(train_loader), batch_time=batch_time,
                data_time=data_time, loss=losses, top1=top1, top5=top5))
        nvtx.range_push("Batch " + str(i+1))
        nvtx.range_push("Load Data")
    nvtx.range_pop()
    nvtx.range_pop()
    nvtx.range_pop()
</code></pre>

111513: <p>Other batchs have the same slow backward<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/6/567e083c14fe3f21e6e312f79d0155c4c68cd3c3.png" data-download-href="https://discuss.pytorch.org/uploads/default/567e083c14fe3f21e6e312f79d0155c4c68cd3c3" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/6/567e083c14fe3f21e6e312f79d0155c4c68cd3c3.png" alt="image" data-base62-sha1="cl95hf0vlHZjvfMlQ8Q79Tp1jFx" width="690" height="256" data-dominant-color="DCD9DE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">910×338 26.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You could try to use <code>torch.backends.cudnn.benchmark = True</code> in case you are not already using it.<br>
Also, to avoid synchronizations you could remove all synchronizing <code>item()</code> calls.</p>