Why loss.backward and optimizer.step take longer with larger batch size?

username: raven
cooked: <p>I was trying to measure the training time with different batch size. It shows that backward() and step() increase with larger batch size (forward time also increases but that’s expected). The following times are all averaged over all iterations in the epoch. (I believe I measured time correctly with torch.cuda.synchronize()</p>
<p>batch size = 96<br>
Train Epoch <a href="https://github.com/pytorch/pytorch/issues/1" rel="noopener nofollow ugc">#1</a>: 100%|██████████| 3336/3336 [19:59&lt;00:00, 2.78it/s, loss=5.85, top1=5.28, top5=15, data_time=0.00287, forward_time=0.0669, backward_time=0.197, cuda_time=0.00804, step_time=0.0417<br>
batch size = 64<br>
Train Epoch <a href="https://github.com/pytorch/pytorch/issues/1" rel="noopener nofollow ugc">#1</a>: 100%|█████████▉| 4995/5004 [19:58&lt;00:02, 3.90it/s, loss=6.16, top1=2.9, top5=9.46, data_time=0.00134, forward_time=0.0444, backward_time=0.135, cuda_time=0.00387, step_time=0.0225<br>
batch size = 32<br>
Train Epoch <a href="https://github.com/pytorch/pytorch/issues/1" rel="noopener nofollow ugc">#1</a>: 100%|██████████| 10008/10008 [26:11&lt;00:00, 6.37it/s, loss=6.84, top1=0.266, top5=1.22, data_time=0.000503, forward_time=0.0267, backward_time=0.0792, cuda_time=0.00204, step_time=0.0193]</p>
<p>Thanks!</p>

username: albanD
cooked: <p>Hi,</p>
<p>Could you show how you use synchronize? Your forward is suspiciously fast compared to the backward <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
Also the forward/backward being slower for larger batch sizes is expected as there is more work to do!</p>

username: raven
cooked: <p>Thank you very much for the reply! Yes, I was expecting forward time increases with larger batch size, but backward and step time keeps the same. Here is the code</p>
<pre><code class="lang-auto">...
            torch.cuda.synchronize()
            time0 = time.time()
            images, labels = images.cuda(), labels.cuda()
            target = labels
            torch.cuda.synchronize()
            cuda_time.update(time.time() - time0)
            
            # soft target
            if args.kd_ratio &gt; 0:
                args.teacher_model.train()
                with torch.no_grad():
                    soft_logits = args.teacher_model(images).detach()
                    soft_label = F.softmax(soft_logits, dim=1)

            # clear gradients
            optimizer.zero_grad()
            
            torch.cuda.synchronize()
            time1 = time.time()
            output = run_manager.net(images)
            loss = run_manager.train_criterion(output, labels)
            torch.cuda.synchronize()
            forward_time.update(time.time() - time1)

            torch.cuda.synchronize()
            time2 = time.time()
            loss.backward()
            torch.cuda.synchronize()
            backward_time.update(time.time() - time2)

            torch.cuda.synchronize()
            time3 = time.time()
            optimizer.step()
            torch.cuda.synchronize()
            step_time.update(time.time() - time3)
...
</code></pre>

username: albanD
cooked: <p>Ok the timing looks good.</p>
<p>And the backward time will increase the same as the forward time. THe more things you used in the forward, the more things needs to be done in the backward <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: raven
cooked: <p>Thanks! Is it right that forward() calculates the gradients for parameters with require_grad=True, and backward() will accumulate those? If so, it makes sense that backward() increases with batch size. But it still doesn’t make sense that step() also takes longer… Any idea?</p>

username: Toby
cooked: <p>As I understand, in forward(), they compute the function and store the input and keep the place holder for the gradient from next node (by chain rule).</p>
<p>In backward(), from gradient of loss function, it compute all the gradient to be ready to optimize.</p>
<p>In step(), they update all gradient depend on your optimization method (SGD, Adam, RMSProp, etc).</p>
<p>So when you increase batch size, all computation cost  will raise which leads to an increase in time.</p>

username: albanD
cooked: <p>You are right: the forward only evaluates the forward (and save the necessary info for the backward)<br>
The backward compute the gradients and accumulate them.<br>
The step only performs the gradient step (this does not depend on the batch size as the gradient accumulation is done during the backward).</p>