Use running averages of batch normalization during training as well?

TriKri: <p>In <a href="https://www.youtube.com/watch?v=HR0lt1hlR6U&amp;t=2h05m44s" rel="noopener nofollow ugc">this video</a>, running batch normalization is discussed as an alternative to regular batch normalization, to eliminate the training–inference disparity and improve model performance. This works simply by using the running averages, not only during inference, but during training as well. Is there some way to do this using the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html" rel="noopener nofollow ugc"><code>BatchNorm1d</code></a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html" rel="noopener nofollow ugc"><code>BatchNorm2d</code></a> layers in PyTorch, or do I need to roll my own module to do that?</p>

ptrblck: <p>If I understand your use case correctly, you would like to:</p>
<ul>
<li>calculate the batch stats</li>
<li>update the running stats with the current batch stats using the running average</li>
<li>normalize the input using the updated running stats instead of the batch stats</li>
</ul>
<p>If so, then I don’t think you can use the <code>nn.BatchNormXd</code> layers to do so and would need to implement a custom layer.<br>
You could use <a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py">my manual implementation</a> as a template and change the logic in the <code>forward</code> method.</p>

JaneChen: <p>May I ask if I want to</p>
<ul>
<li>given a pretrained model and a batch of training data</li>
<li>ignore the batch stats but use the pretrained running stats for data normalization</li>
<li>do NOT update the running stats using this batch of data</li>
<li>update the model<br>
Then, can I achieve this by just using the eval mode of BatchNorm2d for training?<br>
I tried this but ended up with bad results (gradients become nan after one iteration of update.) May I ask whether my trial achieves my goal and why my goal gives me bad results?</li>
</ul>
<p>Thanks a lot!</p>

ptrblck: <aside class="quote no-group" data-username="JaneChen" data-post="3" data-topic="170122">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/ebca7d/48.png" class="avatar"> JaneChen:</div>
<blockquote>
<p>Then, can I achieve this by just using the eval mode of BatchNorm2d for training?</p>
</blockquote>
</aside>
<p>Yes, calling <code>.eval()</code> on batchnorm layers only will use the running stats to normalize the inputs and will will not update the running stats anymore. However, the affine parameters (<code>weight</code> and <code>bias</code>) are still trainable.</p>
<aside class="quote no-group" data-username="JaneChen" data-post="3" data-topic="170122">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/ebca7d/48.png" class="avatar"> JaneChen:</div>
<blockquote>
<p>I tried this but ended up with bad results (gradients become nan after one iteration of update.)</p>
</blockquote>
</aside>
<p>I’m not familiar with your use case but would guess that the running stats are not properly normalizing the training data and the range might thus be large causing the forward pass or gradients to blow up.</p>