Train BatchNorm2D with Training and Evaluation Modes

wwaayyaaww: <p>Hi. I am wondering if it is possible to train the batchnorm2d layer using a very low learning rate?</p>
<p>Assuming that we are to train an CNN with a batchnorm2d layer, i.e. bn_layer.</p>
<p>To freeze the bn_layer from learning in the training stage, are the following two settings equivalent?</p>
<p><strong>Setting 1</strong> : let the bn_layer in evaluation mode</p>
<blockquote>
<p>bn_layer.eval()</p>
</blockquote>
<p><strong>Setting 2</strong> : let the bn_layer in training mode</p>
<blockquote>
<p>bn_layer.train()<br>
bn_layer.momentum = 0<br>
bn_layer.weight.requires_grad = False<br>
bn_layer.biast.requires_grad = False</p>
</blockquote>
<p>Based on my understanding, both settings should be the same, thereby returning the similar results. Unfortunately, the <strong>setting 2</strong> doesn’t freeze the bn_layer from learning at all.</p>
<p>However, I observe all parameters and running statistics including running_mean, running_var, weight, and bias are remained unchanged.</p>
<p>Any reason why the two settings are different?</p>
<p>Any code snippets if only weight and bias are trained? but fixing both running statistics unchanged?</p>
<p>Thank you.</p>

ptrblck: <p>No, these settings are not equivalent, as the first one will not freeze the trainable parameters (<code>weight</code> and <code>bias</code>), but will only use the internal <code>running_mean</code> and <code>running_var</code>.</p>
<aside class="quote no-group" data-username="wwaayyaaww" data-post="1" data-topic="109213">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wwaayyaaww/48/22536_2.png" class="avatar"> wwaayyaaww:</div>
<blockquote>
<p>Unfortunately, the <strong>setting 2</strong> doesn’t freeze the bn_layer from learning at all.</p>
<p>However, I observe all parameters and running statistics including running_mean, running_var, weight, and bias are remained unchanged.</p>
</blockquote>
</aside>
<p>I don’t understand this statement. If all parameters and the running stats remain the same, what do you consider is still “learning”?</p>

wwaayyaaww: <p>Thank you for your response.</p>
<p>How if the following two settings? are they equivalent?</p>
<p><strong>Setting 1</strong> :</p>
<blockquote>
<p>bn_layer.eval()<br>
bn_layer.weight.requires_grad = False<br>
bn_layer.biast.requires_grad = False</p>
</blockquote>
<p><strong>Setting 2</strong> :</p>
<pre><code class="lang-auto">bn_layer.train()
bn_layer.momentum = 0
bn_layer.weight.requires_grad = False
bn_layer.biast.requires_grad = False
</code></pre>
<p>My goal is to explore the momentum such that the bn_layer only peforms very minimal learning.</p>
<p>However, following <strong>Setting 2</strong>, the performance drops drastiscally even setting momentum = 0.00001.</p>
<p>Thank you.</p>

ptrblck: <p>No, these approaches still wouldn’t be equivalent, since the batchnorm layer in setting 1 would normalize the input data using the running statistics (zero mean, unit variance by default if they weren’t updated) while the layer in setting 2 would use the batch statistics, but would not update the internal stats with them.</p>

wwaayyaaww: <p>Assuming that I am fine-tuning a pre-learned model trained with running statistics.</p>
<p>Based on <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html" class="inline-onebox" rel="noopener nofollow ugc">BatchNorm2d — PyTorch 2.1 documentation</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/9/69586ae4378e974b8616a495052b6743156f51b1.png" data-download-href="https://discuss.pytorch.org/uploads/default/69586ae4378e974b8616a495052b6743156f51b1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/69586ae4378e974b8616a495052b6743156f51b1_2_690x176.png" alt="image" data-base62-sha1="f1VzdPiorpvtKy3gPdHSGW2Gwtr" width="690" height="176" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/69586ae4378e974b8616a495052b6743156f51b1_2_690x176.png, https://discuss.pytorch.org/uploads/default/original/3X/6/9/69586ae4378e974b8616a495052b6743156f51b1.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/9/69586ae4378e974b8616a495052b6743156f51b1.png 2x" data-dominant-color="D5E5ED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1012×259 21.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If momentum = 0,  the running statisics X_new should be equal to the pre-trained statistics X_hat, and therefore we should be able to obtain the same results for both Setting 1 and 2?</p>

ptrblck: <p>The running stats will be equal, since the <code>momentum</code> was set to zero and they will thus not be updated. However, the layer is still in <code>training</code> mode and will thus use the batch stats to normalize the input not the running stats.</p>

wwaayyaaww: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="109213">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>will thus use the batch stats to normalize the input not the running stats.</p>
</blockquote>
</aside>
<p>I think I got your point…!</p>
<p>In the training mode, we always use the batch statistics for normalization, but in the meantime the running statistics are updated w.r.t. the batch statistics.</p>
<p>However, it the CNN is set to the evaluation mode, only the running statistics are used for both training and testing stages.</p>
<p>Hmmmmmm may I know if my understand is correct?</p>
<p>Thank you.</p>

ptrblck: <p>Yes, your understanding is correct.<br>
In the default setup:</p>
<ul>
<li>training: normalize input with <code>input_stats</code> and update <code>running_stats</code> with <code>input_stats</code> using <code>momentum</code> formula</li>
<li>evaluation: use <code>running_stats</code> to normalize input</li>
</ul>
<p>You can of course change this behavior by setting <code>track_running_stats=False</code>, which would also use the <code>input_stats</code> during evaluation.<br>
The affine parameters are independent from this behavior.</p>

wwaayyaaww: <p>Thank you, and I truly appreciate your very clear explanation.</p>
<p>Would it be possible to use <code>running_stats</code> instead of <code>input_stats</code> for normalization during the training stage?</p>
<p>Because I am dealing with the training data of differrent domains, and therefore the direct use of <code>input_stats</code> harms the performance directly.</p>
<p>Thank you.</p>

ptrblck: <aside class="quote no-group" data-username="wwaayyaaww" data-post="9" data-topic="109213">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wwaayyaaww/48/22536_2.png" class="avatar"> wwaayyaaww:</div>
<blockquote>
<p>Would it be possible to use <code>running_stats</code> instead of <code>input_stats</code> for normalization during the training stage?</p>
</blockquote>
</aside>
<p>Yes, you could call <code>.eval()</code> on the batchnorm layers to use the running stats.<br>
Note however, that these stats won’t normalize the input if they were not updated, since the <code>running_mean</code> is initialized with zeros, while the <code>running_var</code> with ones.</p>

wwaayyaaww: <p>For the pre-learned BN layers, would it be possible to force the model to use the <code>running_stats</code> for normalization during fine-tuning (in the training mode), instead of <code>input_stats</code>? Specifically,</p>
<p><strong>training</strong> - normalize input with <code>running_stats</code> and update <code>running_stats</code> with <code>input_stats</code> using <code>momentum</code> formula</p>
<p>I am sorry for my confused sentences.</p>

ptrblck: <p>Yes, you would have to call <code>.eval()</code> on these batchnorm layers and the running (pretrained) stats will be used.</p>

wwaayyaaww: <p>Ah i got it completely…!</p>
<p>Thank you very much for your patient. :"))</p>

wwaayyaaww: <p>Hi. I just completed some experiments that if <code>.evalI()</code> is set during training, the <code>running_stats</code> will no longer be updated based on the <code>input_stats</code> accordingly.</p>
<p>Is my observation correct?</p>

ptrblck: <p>Yes, this is described <a href="https://discuss.pytorch.org/t/train-batchnorm2d-with-training-and-evaluation-modes/109213/8">here</a>. During <code>eval()</code> only the running stats will be applied.</p>