Accumulate Gradient

duddal: <p>Hello,<br>
Because of memory constraints, I can only use batch_size of 1. But then I came across a trick called “Gradient Accumulation”. I have implemented two versions of it and would like to which one is correct and why, if possible.</p>
<p>Version 1:</p>
<pre><code class="lang-auto"># Backpropogation step
            total_loss.backward()

            # Accumulate Gradients and Update/Back-propagate at every 8th iteration.
            if (iter_i+1) % 32 == 0:
                optimizer.step()
                optimizer.zero_grad()
</code></pre>
<p>Version 2:</p>
<pre><code class="lang-auto">            # Accumulate Gradients and Update/Back-propagate at every 8th iteration.
            if (iter_i+1) % 32 == 0:
                total_loss.backward()
                optimizer.step()
                optimizer.zero_grad()
</code></pre>
<p>The total_loss consists of the following loss functions:</p>
<pre><code class="lang-auto">cls_loss_function = HeatmapLoss(reduction='mean')  # Custom Loss
txty_loss_function = nn.BCEWithLogitsLoss(reduction='none')
twth_loss_function = nn.SmoothL1Loss(reduction='none')
</code></pre>
<p>Should the <code>total_loss</code> be divided by 32 before using the <code>backward</code> function? I have several <code>BatchNorm2d</code> layers in the architecture.</p>
<p>So the question here is whether the <code>total_loss.backward()</code> should be inside or outside the if statement.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">This post</a> gives some examples with detailed explanations of advantages and shortcomings for each one. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

duddal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for replying. I have checked that post but there seem to be some discrepancies between that post and the steps mentioned <a href="https://spell.ml/blog/pytorch-training-tricks-YAnJqBEAACkARhgD" rel="noopener nofollow ugc">here</a>. I will post the code here:</p>
<pre><code class="lang-auto">model.zero_grad()                                   # Reset gradients tensors
for i, (inputs, labels) in enumerate(training_set):
    predictions = model(inputs)                     # Forward pass
    loss = loss_function(predictions, labels)       # Compute loss function
    loss = loss / accumulation_steps                # Normalize our loss (if averaged)
    loss.backward()                                 # Backward pass
    if (i+1) % accumulation_steps == 0:             # Wait for several backward steps
        optimizer.step()                            # Now we can do an optimizer step
        model.zero_grad()                           # Reset gradients tensors
        if (i+1) % evaluation_steps == 0:           # Evaluate the model when we...
            evaluate_model()                        # ...have no gradients accumulated
</code></pre>
<p>Above, the loss is divided by the accumulation steps before using the backward function. But it is also mentioned in the comment “Normalize our loss (if averaged)”. I don’t really know what that means.</p>
<p>Also, should I still use gradient accumulation if I have BatchNorm2D Layers in my network?</p>
<p>Thank You.</p>

ptrblck: <p>Often you would divide the loss by the <code>accumulation_steps</code> so this point seems to be indeed missing in the linked post.</p>
<aside class="quote no-group" data-username="duddal" data-post="3" data-topic="129309">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/duddal/48/39820_2.png" class="avatar"> duddal:</div>
<blockquote>
<p>Also, should I still use gradient accumulation if I have BatchNorm2D Layers in my network?</p>
</blockquote>
</aside>
<p>You could try to use it, but note that the smaller the actual batch size is the more noise the running stats in batchnorm layers could get, which is not specific to gradient accumulation but a general limitation of small batch sizes in combination with batchnorm layers.</p>

duddal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I have another system in which I can use a batch size of max 8. But I read on a blog that BatchNorm works best with batch_size 32. Would you recommend using <code>InstanceNorm2D</code> or <code>GroupNorm</code> instead of BatchNorm2D?</p>
<p>Edit:<br>
After reading the GroupNorm paper it is clear that GorupNorm combined with Weight Standardization should be able to theoretically outperform BatchNorm for smaller batch sizes. What do you think?</p>

ptrblck: <p>I also wanted to suggest to take a look at the GroupNorm paper, which you’ve already done.<br>
However, I don’t have any stats to share when one or the other approach would work better, so in case you are comparing both, it would be great to hear back which experiment outperformed the other one.</p>

duddal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi,</p>
<p>I will definitely post the results here. However, my next hurdle is using a pre-trained model, i.e. Can I Use the pretrained model provided by PyTorch and just change the BatchNorm layers? Or with GroupNorm (GN) + Weight Standardization (WS), the pretrained weights become redundant? If so, do you where I can find the pre-trained ResNet models with GN+WS?</p>
<p>The code for GN+WS can be found <a href="https://nbviewer.jupyter.org/github/amaarora/amaarora.github.io/blob/master/nbs/Group%20Normalization%20WS.ipynb" rel="noopener nofollow ugc">here</a>.</p>
<p>I really don’t have the time or system capabilities to train the network from scratch.</p>

ptrblck: <p>I’m not sure, but wouldn’t expect that replacing batchnorm layers with GroupNorm once would work out of the box (I might be wrong, as I haven’t looked into it deeply). Also, I’m not aware of any (official) pretrained ImageNet models using GroupNorm so you might indeed need to train them from scratch.</p>

duddal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Ahh no. That’s what I was afraid of. I will try training them from scratch. Are there any set hyperparameters (Learning rate, No.of Epochs, Optimizer(SGD, Adam)) one should use for training the ResNet Model from scratch. The problem I have is that I have only 2750 Training images.</p>