Accumulate gradients leads to dramatic performance degrade

N_Hunter: <p>Hi, everyone, I’ve implemented the accumulate gradients using  Pytorch, and trained on cifar100 dataset, here is my code snippet:</p>
<pre data-code-wrap="python"><code class="lang-python">    net.train()
    loss = 0
    acc_batches = args.acc_b
    for batch_index, (images, labels) in enumerate(cifar100_training_loader):
        if epoch &lt;= args.warm:
            warmup_scheduler.step()

        if args.gpu:
            labels = labels.cuda()
            images = images.cuda()

        outputs = net(images)
        scaled_loss = loss_function(outputs, labels) / args.acc_b
        scaled_loss.backward()
        loss += scaled_loss

        print(1)
        if acc_batches &gt; 1:
            acc_batches -= 1
            continue

        optimizer.step()
        optimizer.zero_grad()

        n_iter = (epoch - 1) * len(cifar100_training_loader) + batch_index + 1

        last_layer = list(net.children())[-1]
        for name, para in last_layer.named_parameters():
            if 'weight' in name:
                writer.add_scalar('LastLayerGradients/grad_norm2_weights', para.grad.norm(), n_iter)
            if 'bias' in name:
                writer.add_scalar('LastLayerGradients/grad_norm2_bias', para.grad.norm(), n_iter)

        print('Training Epoch: {epoch} [{trained_samples}/{total_samples}]\tLoss: {:0.4f}\tLR: {:0.6f}'.format(
            loss.item(),
            optimizer.param_groups[0]['lr'],
            epoch=epoch,
            trained_samples=batch_index * args.b + len(images),
            total_samples=len(cifar100_training_loader.dataset)
        ))

        #update training loss for each iteration
        writer.add_scalar('Train/loss', loss.item(), n_iter)

        if acc_batches == 1:
            acc_batches = args.acc_b
            loss = 0

    for name, param in net.named_parameters():
        layer, attr = os.path.splitext(name)
        attr = attr[1:]
        writer.add_histogram("{}/{}".format(layer, attr), param, epoch)

</code></pre>
<p>then I run the following command to train my resnet50:</p>
<pre data-code-wrap="shell"><code class="lang-shell">python train.py -net resnet50 -gpu -b 256
python train.py -net resnet50 -gpu -b 256 -acc_b 2
python train.py -net resnet50 -gpu -b 256 -acc_b 4
python train.py -net resnet50 -gpu -b 256 -acc_b 8
</code></pre>
<p>I’ve plotted the test acc using tensorboard, and here is the result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edca071a9aa862ea733ec9615fab80936e3afca7.png" data-download-href="https://discuss.pytorch.org/uploads/default/edca071a9aa862ea733ec9615fab80936e3afca7" title="2020-11-24 22-27-27屏幕截图"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edca071a9aa862ea733ec9615fab80936e3afca7_2_690x280.png" alt="2020-11-24 22-27-27屏幕截图" data-base62-sha1="xVA0blZRwOPVSnRheLLVDKAyaZ9" width="690" height="280" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edca071a9aa862ea733ec9615fab80936e3afca7_2_690x280.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edca071a9aa862ea733ec9615fab80936e3afca7_2_1035x420.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/edca071a9aa862ea733ec9615fab80936e3afca7_2_1380x560.png 2x" data-dominant-color="FBFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2020-11-24 22-27-27屏幕截图</span><span class="informations">1439×586 80.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>orange line</strong>:  <code>args.acc_b = 1</code><br>
<strong>dark blue line</strong>: <code>args.acc_b = 2</code> (loss becomes nan at 37th epoch)</p>
<pre><code class="lang-auto">Training Epoch: 37 [3072/50000] Loss: 1.2027    LR: 0.100000
1
1
Training Epoch: 37 [3584/50000] Loss: 5.3613    LR: 0.100000
1
1
Training Epoch: 37 [4096/50000] Loss: 5.7180    LR: 0.100000
1
1
Training Epoch: 37 [4608/50000] Loss: nan       LR: 0.100000
1
1
Training Epoch: 37 [5120/50000] Loss: nan       LR: 0.100000
</code></pre>
<p><strong>red line</strong>: <code>args.acc_b = 4</code> (loss increases suddenly from 0.7734 to 6.5302 at 38 epoch)</p>
<pre><code class="lang-auto">Training Epoch: 38 [43008/50000]        Loss: 0.7734    LR: 0.100000
1
1
1
1
Training Epoch: 38 [44032/50000]        Loss: 6.5302    LR: 0.100000
1
1
1
1
Training Epoch: 38 [45056/50000]        Loss: 5.4722    LR: 0.100000
1
1
1
1
</code></pre>
<p><strong>light blue line</strong>: <code>args.acc_b =  8</code>  (loss increases suddenly from 0.0202 to 6.8125 at 105 epoch):</p>
<pre><code class="lang-auto">Training Epoch: 105 [6144/50000]        Loss: 0.0202    LR: 0.020000
1
1
1
1
1
1
1
1
Training Epoch: 105 [8192/50000]        Loss: 6.8125    LR: 0.020000
</code></pre>
<p>The whole project is at <a href="https://github.com/weiaicunzai/pytorch-cifar100/tree/feat/accumulate_grad_batches" class="inline-onebox" rel="noopener nofollow ugc">GitHub - weiaicunzai/pytorch-cifar100 at feat/accumulate_grad_batches</a></p>
<p>I just do not understand why my loss keeps exploding when I set arg.acc_b &gt; 1.<br>
Thanks in advance.</p>