What is the running mean of BatchNorm if gradients are accumulated?

Zhang_Chi: <p>hi<br>
due to limited gpu memory , i want to accumulate gradients in some iterations and back propagate to work as large batch.  for example,now the batch size is 2, and i backward after five iterations, which is the same with a batch of 10. However, what is running mean of BN layer in this process?  Will pytorch average the 10 data or only take the average of the last mini-batch (2 in this case ) as the running mean?</p>

crcrpar: <p>Hi, <a class="mention" href="/u/zhang_chi">@Zhang_Chi</a></p>
<p>Batch Normalization updates its running mean and variance every call of <code>forward</code> method.<br>
Also, by default, BatchNorm updates its running mean by <code>running_mean = alpha * mean + (1 - alpha) * running_mean</code> (the details are <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/THNN/generic/BatchNormalization.c#L50-L62" rel="noopener nofollow ugc">here</a>).</p>
<p>As to accumulating gradients, this thread “<a href="https://discuss.pytorch.org/t/how-to-implement-accumulated-gradient/3822/8" class="inline-onebox">How to implement accumulated gradient？ - #8 by Gopal_Sharma</a>” might help you.</p>
<aside class="quote no-group" data-username="Zhang_Chi" data-post="1" data-topic="18870">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhang_chi/48/1582_2.png" class="avatar"> Zhang_Chi:</div>
<blockquote>
<p>or example,now the batch size is 2, and i backward after five iterations, which is the same with a batch of 10.</p>
</blockquote>
</aside>
<p>As a side note, I don’t think the accumulated gradients and the gradient will be the same in the example.</p>

Zhang_Chi: <p>thank you very much for your reply. <a class="mention" href="/u/crcrpar">@crcrpar</a><br>
1.so you mean running mean is updated during forward process not backward process ?<br>
2.why do you think they are different?  do you mean that the accumulated gradients should be divided by 10? any other difference ?</p>

crcrpar: <ol>
<li>Yes.</li>
<li>Accumulated gradients will be the same if you divide them by the number of iterations. I referred below.</li>
</ol>
<aside class="quote no-group" data-username="apaszke" data-post="3" data-topic="884" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/apaszke/48/21_2.png" class="avatar"><a href="https://discuss.pytorch.org/t/pytorch-gradients/884/3">PyTorch Gradients</a></div>
<blockquote>
<p>Not that loss functions average over the batch size, so if you do multiple backprops you might need to average over the number of the for loop iterations.</p>
</blockquote>
</aside>

Zhang_Chi: <p>yeah. it there a quick way to do that? what i can think it to loop all the parameters’ gradients and divide each of them by the number of iterations. Or is it the same if I divide the loss to be backpropogated by the number of iterations?</p>

crcrpar: <aside class="quote no-group" data-username="Zhang_Chi" data-post="5" data-topic="18870">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhang_chi/48/1582_2.png" class="avatar"> Zhang_Chi:</div>
<blockquote>
<p>Or is it the same if I divide the loss to be backpropogated by the number of iterations?</p>
</blockquote>
</aside>
<p>yes it’ll be the same.</p>

Zhang_Chi: <aside class="quote no-group" data-username="crcrpar" data-post="6" data-topic="18870">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/crcrpar/48/24766_2.png" class="avatar"> crcrpar:</div>
<blockquote>
<p>it’ll be the same.</p>
</blockquote>
</aside>
<p>thank you so much  <a class="mention" href="/u/crcrpar">@crcrpar</a></p>

crcrpar: <p>Glad to hear that!<img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=5" title=":grin:" class="emoji" alt=":grin:"></p>

du_s: <p><a class="mention" href="/u/crcrpar">@crcrpar</a><br>
hi,why calculate running_mean as:</p>
<pre><code class="lang-auto">running_mean = alpha * mean + (1 - alpha) * running_mean
</code></pre>
<p>not just use:</p>
<pre><code class="lang-auto">running_mean = sum(mean of every batch)/batch counts
</code></pre>
<p>what is the difference or what is the benifit of <code> running_mean = alpha * mean + (1 - alpha) * running_mean</code></p>