Check gamma and beta of BN

Ardeal: <p>Hi,</p>
<p>BN is calculated according to the following formula:<br>
<a href="https://pytorch.org/docs/master/generated/torch.nn.BatchNorm2d.html#torch.nn.BatchNorm2d" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/generated/torch.nn.BatchNorm2d.html#torch.nn.BatchNorm2d</a></p>
<p>I have 2 questions about BN:<br>
how to print the gamma and beta of BN in the code?<br>
how are running_var and running_mean calculated in code? is there a calculation formula?</p>

ptrblck: <aside class="quote no-group" data-username="Ardeal" data-post="1" data-topic="135090">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardeal/48/33257_2.png" class="avatar"> Ardeal:</div>
<blockquote>
<p>how to print the gamma and beta of BN in the code?</p>
</blockquote>
</aside>
<p><code>gamma</code> and <code>beta</code> are assigned as <code>weight</code> and <code>bias</code>, respectively, so you could use:</p>
<pre data-code-wrap="python"><code class="lang-python">print(bn.weight)
print(bn.bias)
</code></pre>
<aside class="quote no-group" data-username="Ardeal" data-post="1" data-topic="135090">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardeal/48/33257_2.png" class="avatar"> Ardeal:</div>
<blockquote>
<p>how are running_var and running_mean calculated in code? is there a calculation formula?</p>
</blockquote>
</aside>
<p>Yes, the <code>NOTE</code> section in the linked docs show the update formula.</p>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Thank you!</p>
<p>how are running_mean and running_var used in code?<br>
I mean in what case they are used?</p>
<p>I am experiencing the following issue:<br>
there are 2 bn layers in  2  models respectively. the input, weight, bias are all the same, and the only difference is running_mean and running_var.  in model.train() case, the output of those 2 bn layer are different. why?</p>

ptrblck: <p>In the default setup the running stats are updated during training (i.e. if the model is in <code>train()</code> mode) using the mentioned update formula. The input batches are normalized using the batch statistics.<br>
During validation (i.e. if the model is in <code>eval()</code> mode) the running stats will be used to normalize the input.</p>
<aside class="quote no-group" data-username="Ardeal" data-post="3" data-topic="135090">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardeal/48/33257_2.png" class="avatar"> Ardeal:</div>
<blockquote>
<p>there are 2 bn layers in 2 models respectively. the input, weight, bias are all the same, and the only difference is running_mean and running_var. in model.train() case, the output of those 2 bn layer are different. why?</p>
</blockquote>
</aside>
<p>If both layers are getting the same input and are both in <code>train()</code>, their output would also be equal. I would guess that the input might be different and you should compare it.</p>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Thank you!</p>
<p>Please check the following compaison. both are in <code>train()</code>,  input, weight, bias are all identical. running_mean and running_var are different, and the output of BN is not identical.</p>
<p>I have the same understanding as yours, that is why I asked you those above questions. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/9/c951d94d9e675198bf885ccd1378994cc278e53d.png" data-download-href="https://discuss.pytorch.org/uploads/default/c951d94d9e675198bf885ccd1378994cc278e53d" title="图片"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c951d94d9e675198bf885ccd1378994cc278e53d_2_610x500.png" alt="图片" data-base62-sha1="sIXkzNhyAcQuhdeta4UbyHqeQeV" width="610" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c951d94d9e675198bf885ccd1378994cc278e53d_2_610x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c951d94d9e675198bf885ccd1378994cc278e53d_2_915x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/9/c951d94d9e675198bf885ccd1378994cc278e53d_2_1220x1000.png 2x" data-dominant-color="E9D7CF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">图片</span><span class="informations">1743×1427 227 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you post a minimal executable code snippet to reproduce this issue? The output difference is large enough that it doesn’t seem to come from the limited precision errors.</p>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Thank you!</p>
<p>I did the exmperments like this:</p>
<ol>
<li>hook function to register_forward_hook,</li>
<li>if nn.BatchNormal2d, save data and stop saving after the BN.<br>
The exmperiments is based on resnet18, and the upper picture is for the first bn of resnet18.</li>
</ol>
<p>I tried another experiment:<br>
<strong>during model loading, I copied the right running_mean and running_var to the left model, the output of bn of the left model is identical with the right.</strong></p>