Does number of gradient accumulation steps affect model's performance?

username: mathcircler
cooked: <p>Hi, I wanted to imitate training with a large batch size using gradient accumulation approach as per this <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255" rel="nofollow noopener">article</a>, due to a lack of GPU memory for a larger batch. A snippet of the code is below:</p>
<pre><code class="lang-auto">model.zero_grad()                                   # Reset gradients tensors
for i, (inputs, labels) in enumerate(training_set):
    predictions = model(inputs)                     # Forward pass
    loss = loss_function(predictions, labels)       # Compute loss function
    loss = loss / accumulation_steps                # Normalize our loss (if averaged)
    loss.backward()                                 # Backward pass
    if (i+1) % accumulation_steps == 0:             # Wait for several backward steps
        optimizer.step()                            # Now we can do an optimizer step
        model.zero_grad()                           # Reset gradients tensors
        if (i+1) % evaluation_steps == 0:           # Evaluate the model when we...
            evaluate_model()                        # ...have no gradients accumulated
</code></pre>
<p>For some reason, I find that the results of training vary for different values of gradient_accumulation_steps. I also see that others also report different results for different values of gradient_accumulation_steps (for example, <a href="https://github.com/llamazing/numnet_plus/issues/5#issuecomment-546937949" rel="nofollow noopener">this issue</a> is from the project I was trying to replicate).</p>
<p>I was wondering if the number of steps can indeed in any way influence the performance of the model (for a fixed batch size)? My understanding was that this parameter should only has effect on the speed of training, but not the results, as long as the batch size is the same.</p>

username: ptrblck
cooked: <p>Your gradient accumulation approach might change the model performance, if you are using batch-size-dependent layers such as batchnorm layers.<br>
Batchnorm layers will use the current batch statistic to update the running stats. The smaller the batch size the more noise these stats updates will have. You could try to counter this effect by changing the <code>momentum</code> of these layers.</p>
<p>Let me know, if that might be the case or if you are not using batchnorm layers.</p>

username: mathcircler
cooked: <p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a>. Yes, this seems to be the probable reason, thank you! I use Roberta as an encoder, so there are some BatchNorm modules in the encoder, but, as far as I can see, there are no batch-size dependent modules in the rest of the model (by the way, I’m curious now, if there any other popular batch-size dependent modules like batch norm).</p>
<p>But I didn’t quite understand, how momentum can help here? Because the momentum term is used for accelerating gradient using gradients from previous batches, but in the scope of several accumulation steps that we have in one batch it doesn’t influence on gradient computation in any way.</p>

username: ptrblck
cooked: <p>Normalization layers might be batch-dependent and of course some custom layers.<br>
I’m not sure if there are any other “core layers”. I guess RNN training would also be different, if you pass the samples sequentially or in a batched way, but this doesn’t seem to be your use case.</p>
<aside class="quote no-group" data-username="mathcircler" data-post="3" data-topic="85859">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/45deac/48.png" class="avatar"> mathcircler:</div>
<blockquote>
<p>But I didn’t quite understand, how momentum can help here? Because the momentum term is used for accelerating gradient using gradients from previous batches, but in the scope of several accumulation steps that we have in one batch it doesn’t influence on gradient computation in any way.</p>
</blockquote>
</aside>
<p>Sorry, I meant the batchnorm <code>momentum</code>, which is used to update the running stats.<br>
If your current batch statistics are noisy, you might change the momentum to decrease the current updates.</p>

username: charl
cooked: <p>Hi,<br>
I’m also very curious about this question and I do some experiments.</p>
<p>Set-ups:<br>
RTX3090, Ubuntu 20.04, Nvidia Driver 460.32.03, Cuda 11.1<br>
PyTorch 1.7.1, transformers 4.3.2 (huggingface)<br>
Model: Roberta-large (RobertaForMultipleChoice)<br>
Goal: real batch size (16) = effective batch size (bs_gpu=8, gradient accumulation steps=2)?<br>
Fix all random seeds and all hyperparameters</p>
<p>To my knowledge, the Roberta model uses <strong>LayerNorm</strong> instead of <strong>BatchNorm</strong>,<br>
which does not have <strong>running statistics</strong>.<br>
However, to make sure everything is all right,<br>
I used <strong>model.eval()</strong> even in training and set the weight and bias in each LayerNorm layer to <strong>requires_grad=False</strong>. I also <strong>disabled gradient norm clipping</strong>.</p>
<p>==============================================================================<br>
Results:<br>
Orange line: real batch size; Blue line: effective batch size (using gradient_accumulation)<br>
I only show the plot before the 1st epoch ends in case that the last batch matters<br>
The x-axis is the <span class="hashtag-raw">#global</span> steps (= effective steps)</p>
<p>Fig1: torch.abs(classifier.weight.grad).sum()</p>
<p>Fig2: torch.abs(output).sum() given the same input</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/3/03048c738ac91c173545aeed7948b9c1f1ec57fd.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/03048c738ac91c173545aeed7948b9c1f1ec57fd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/03048c738ac91c173545aeed7948b9c1f1ec57fd_2_344x499.jpeg" alt="image" data-base62-sha1="qHb1yi7BsbfxmNIc7psenjhHRP" width="344" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/03048c738ac91c173545aeed7948b9c1f1ec57fd_2_344x499.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/0/3/03048c738ac91c173545aeed7948b9c1f1ec57fd.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/3/03048c738ac91c173545aeed7948b9c1f1ec57fd.jpeg 2x" data-dominant-color="AFB2B2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">512×744 96.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>There might have some bugs in my code,<br>
if not, my observation is that effective batch size != real batch size,<br>
and the difference becomes very significant after a few steps.</p>
<p>Could it be gradient_accumulation causes some numerical issues and leads to error propagation?</p>
<p>BTW I make sure the plots are the same given the same run using real_batch_size (my code is reproducible)</p>
<p>I wonder if my experiments and suggestion are right?<br>
Thanks</p>

username: Miguel_C
cooked: <p>Have you found a solution to this? In my case I experimented with batch size of 8 and gradient acc. steps of 8 vs batch size of 4 and gradient acc. steps of 16 for an effective batch size of 64…</p>
<p>The first converges correctly, the second starts are an even lower loss for the first few steps of training and then explodes…</p>