The behavior of the BN layer in train and eval mode

username: is_Juncheng
cooked: <p>Search before you ask. I saw a lot of related questions, but none of them were complete enough, so I wanted to provide a comprehensive summary and in-depth discussion here.</p>
<ol>
<li>First we should clarify how the BN layer is calculated：<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/009bfbf389985264d65e03d57f9d387151d7f6ed.png" data-download-href="https://discuss.pytorch.org/uploads/default/009bfbf389985264d65e03d57f9d387151d7f6ed" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/009bfbf389985264d65e03d57f9d387151d7f6ed_2_517x152.png" alt="image" data-base62-sha1="5oc3WBvpTKx3DTrYt9VFGGX5aB" width="517" height="152" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/009bfbf389985264d65e03d57f9d387151d7f6ed_2_517x152.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/009bfbf389985264d65e03d57f9d387151d7f6ed_2_775x228.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/0/009bfbf389985264d65e03d57f9d387151d7f6ed.png 2x" data-dominant-color="F6F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">921×271 23.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
reference：<a href="https://pytorch.org/docs/master/generated/torch.nn.BatchNorm2d.html#torch.nn.BatchNorm2d" class="inline-onebox" rel="noopener nofollow ugc">BatchNorm2d — PyTorch master documentation</a><br>
The BN layer will calculate the mean and val in each batch of input data and then normalize them. And move through γ and β to the new data distribution where γ and β are learnable parameters.</li>
<li>According to my own experiments, setting eval mode on the BN layer does not prevent the BN layer parameters from being updated.Just different behavior！<br>
This is also reflected in many of the responses, eg.<br>
<a href="https://discuss.pytorch.org/t/what-is-the-meaning-of-the-training-attribute-model-train-and-model-eval/40259/2" class="inline-onebox">What is the meaning of the training attribute?model.train() and model.eval()? - #2 by ptrblck</a><br>
<a href="https://discuss.pytorch.org/t/does-gradients-change-in-model-eval/127732/2" class="inline-onebox">Does gradients change in Model.eval()? - #2 by ptrblck</a><br>
<a href="https://discuss.pytorch.org/t/what-does-model-eval-do-for-batchnorm-layer/7146/2" class="inline-onebox">What does model.eval() do for batchnorm layer? - #2 by smth</a></li>
<li>problem:<br>
In model.train(), we use the input data’s statistical features (mean and var) to normalize and use the training γ and β to move to the new data distribution.<br>
In model.eval(), we use the trained γ and β statistical features for normalization. Right?</li>
</ol>
<p>I’m not sure if my insights are correct, feel free to correct me!</p>

username: ptrblck
cooked: <ol>
<li>
<p>No, <code>gamma</code> and <code>beta</code> refer to the trainable affine parameters stored as the <code>.weight</code> and <code>.bias</code> attributes. These parameters will not be updated by the calculated mean and stddev but the <code>running_mean</code> and <code>running_var</code> during training.</p>
</li>
<li>
<p>That’s correct as the <code>.train()</code> and <code>.eval()</code> mode does not change the behavior of the trainable parameters, just how the input activation will be normalized:</p>
</li>
</ol>
<ul>
<li>during training: the input activation will be normalized using the calculated input activations stats (mean and var) and the running stats will be updated (<code>running_mean</code> and <code>running_var</code>)</li>
<li>during eval: the input activation will be normalized using the running stats as described also in your <a href="https://discuss.pytorch.org/t/does-gradients-change-in-model-eval/127732/5">cross-post</a>.</li>
</ul>
<p>Calling <code>model.train()</code> or <code>model.eval()</code> ooes not disable the gradient calculation of the affine parameters (<code>.weight</code> and <code>.bias</code>).</p>
<ol start="3">
<li>No, during <code>eval()</code> <code>gamma/weight</code> and <code>beta/bias</code> will still be used as before. The normalization will be done using the running stats (mean subtraction and division by the stddev).</li>
</ol>