What does requires_grad=False on BatchNorm2d perform?

Giulio: <p>Hi everyone, I have a question regarding BatchNorm2d.</p>
<p>What changes happen in the model if during training I set requires_grad=False on BatchNorm2d layers?<br>
I read that <code>running_mean</code> and <code>running_var</code> are buffers and do not require gradients. Is it true? If so, what will be the difference in BatchNorm2d if I set  requires_grad=False opposed to  requires_grad=True?</p>
<p>Thanks in advance!</p>

ptrblck: <aside class="quote no-group" data-username="Giulio" data-post="1" data-topic="137904">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/giulio/48/39056_2.png" class="avatar"> Giulio:</div>
<blockquote>
<p>I read that <code>running_mean</code> and <code>running_var</code> are buffers and do not require gradients. Is it true?</p>
</blockquote>
</aside>
<p>Yes, that’s true as the running stats will be updated in each forward pass if the module is set to training mode using the batch statistics.</p>
<p>By default batchnorm layers will contain trainable parameters (<code>weight</code> and <code>bias</code>), which will get gradients and will thus be updated. Setting their <code>requires_grad</code> attribute to <code>False</code> would freeze these parameters.</p>

Giulio: <p>Ok thanks!</p>
<p>However, do these parameters (<code>weight</code> and <code>bias</code>) influence the output of the BatchNorm2d layer or they are just there to create consistency among layers’ implementations?<br>
Because looking at the formula, BatchNorm2d requires only the running stats and the expected mean/variance and there is no weight and bias</p>
<p>Thanks in advance!</p>

mMagmer: <p>see : <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html" rel="noopener nofollow ugc">batch norm</a>.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a177ddb5e07c2ae5572032ffef2fa518c3b9f624.png" alt="image" data-base62-sha1="n2pGrxvtVj6IcCGV8n1NAZtVfFi" width="252" height="67"><br>
Beta and gamma are weights and bias.<br>
In training time with forward pass E(x) and Var(x) are estimated using batch samples.<br>
In test time, using model.eval() will change the behavior of forward to use running means instead of E(x) and Var(x).</p>

ptrblck: <p>As <a class="mention" href="/u/mmagmer">@mMagmer</a> explained, <code>gamma=weight</code> and <code>beta=bias</code> will be used in the default setup unless you are creating the batchnorm layers with <code>affine=False</code>.</p>

Giulio: <p>Ok now everything is clear, thank you both! <a class="mention" href="/u/mmagmer">@mMagmer</a> <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

Nimeshika_Udayangani: <p>Hi I fine-tuned a Wide-ResNet setting only last fully connected layer, and all other layers remain same since I set requires_grad=False for these layers. Then the output from a batchNorm layer of the original pre-trained resnet and one fine-tuned (only the last FC layer) should be same(since I set requires_grad=False for all other layers) . However I get different outputs just after batchNorm layer for the same input. Do you know why is that:?</p>

ptrblck: <p>Setting <code>requires_grad = False</code> will freeze the trainable, affine parameters, but will not change the running stats updates as explained in my previous post. Call <code>.eval()</code> on these layers to use the fixed running stats instead.</p>

Nimeshika_Udayangani: <p>Can I call .eval() for only selected layers? since I need to train last FC layer.</p>

ptrblck: <p>Yes, you can call <code>eval()</code> on any layer. Nota that it won’t freeze the trainable parameters but will change the behavior of some layers, such as batchnorm. Calling <code>eval()</code> on the last linear layer won’t have any effect.</p>

Nimeshika_Udayangani: <aside class="quote no-group" data-username="mMagmer" data-post="4" data-topic="137904">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmagmer/48/43283_2.png" class="avatar"> mMagmer:</div>
<blockquote>
<p>In test time, using model.eval() will change the behavior of forward to use running means instead of E(x) and Var(x).</p>
</blockquote>
</aside>
<p>So does this means if I set model.eval() at test time(when features are extracted) 2 models(models with same affine parameters but different batchnorm stats) should give same results since it discard E(x) and Var(x) and calculate these stats according to test set?</p>

ptrblck: <p>No, since during eval the running stats are used to normalize the input activation.</p>

svyatoslav_11: <p>hello, sorry for this but i have a question: should BatchNorm layer’s parameters get updated? i mean, we only update nn.Linear(), nn.Bilinear() etc right? BatchNorm is a regularization ‘technique’, so should we update them (set requires_grad=True) in the first place? Thanks!</p>

ptrblck: <p>Yes, by default the affine parameters are initialized and trained. However, you are free to either freeze or not create these parameters if it would fit your use case better.</p>