`F.batch_norm` returns different results in `train` and `eval` mode given same setup

marsggbo: <ul>
<li>
<ol>
<li>set seed</li>
</ol>
</li>
</ul>
<pre><code class="lang-python">import torch
import torch.nn as nn
import torch.nn.functional as F

from pytorch_lightning.utilities.seed import seed_everything
seed_everything(666)

num=6
x = torch.rand(4,num)
print(x)
&gt;&gt;&gt; tensor([[0.3119, 0.2701, 0.1118, 0.1012, 0.1877, 0.0181],
        [0.3317, 0.0846, 0.5732, 0.0079, 0.2520, 0.5518],
        [0.8785, 0.5281, 0.4961, 0.9791, 0.5817, 0.4875],
        [0.0650, 0.7506, 0.2634, 0.3684, 0.5035, 0.9089]])
</code></pre>
<ul>
<li>
<ol start="2">
<li>randomly init a BN</li>
</ol>
</li>
</ul>
<pre><code class="lang-auto">bn = nn.BatchNorm1d(num)

rand = lambda num: torch.rand(num)

weight = rand(num)
bias = rand(num)
mean = rand(num)
var = rand(num)

bn.weight.data = weight.data
bn.bias.data = bias.data
bn.running_mean.data = mean.data
bn.running_var.data = var.data
</code></pre>
<ul>
<li>
<ol start="3">
<li>compare different ops</li>
</ol>
</li>
</ul>
<pre><code class="lang-python">bn.eval()
y1 = bn(x)

&gt;&gt;&gt; tensor([[ 0.1871, -0.7274, -0.2401,  0.2894,  0.8241,  0.0806],
        [ 0.2042, -1.5069,  0.6280,  0.2882,  0.8298,  0.6823],
        [ 0.6761,  0.3567,  0.4828,  0.3000,  0.8590,  0.6098],   
        [-0.0260,  1.2915,  0.0450,  0.2926,  0.8521,  1.0849]],
       grad_fn=&lt;NativeBatchNormBackward&gt;)

y2 = F.batch_norm(x, mean, var, weight, bias, eps=1e-5, momentum=0.1, training=False)
&gt;&gt;&gt; tensor([[ 0.1871, -0.7274, -0.2401,  0.2894,  0.8241,  0.0806],
        [ 0.2042, -1.5069,  0.6280,  0.2882,  0.8298,  0.6823],
        [ 0.6761,  0.3567,  0.4828,  0.3000,  0.8590,  0.6098],
        [-0.0260,  1.2915,  0.0450,  0.2926,  0.8521,  1.0849]])
</code></pre>
<p>We can see that the two forms return the same results.</p>
<p>But if we try <code>F.batch_norm(..., training=True)</code>, we will get totally different results</p>
<pre><code class="lang-python">y2 = F.batch_norm(x, mean, var, weight, bias, eps=1e-5, momentum=0.1, training=True)
&gt;&gt;&gt; tensor([[ 0.1582, -0.3523, -0.2571,  0.2823,  0.8138, -0.5995],
        [ 0.1835, -0.9058,  0.7885,  0.2801,  0.8397,  0.7659],
        [ 0.8830,  0.4174,  0.6137,  0.3029,  0.9721,  0.6014],
        [-0.1577,  1.0812,  0.0864,  0.2886,  0.9407,  1.6796]])
</code></pre>
<p>My question is what is the role of <code>training</code> in <code>F.batch_norm</code>?</p>

ptrblck: <p>I get the same expected results if both approaches are using eval and training mode, respectively.</p>
<aside class="quote no-group" data-username="marsggbo" data-post="1" data-topic="140493">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marsggbo/48/13247_2.png" class="avatar"> marsggbo:</div>
<blockquote>
<p>My question is what is the role of <code>training</code> in <code>F.batch_norm</code>?</p>
</blockquote>
</aside>
<p>During training the batch stats will be used to normalize the input (and the running stats will be updated) while the running stats will be used to normalize the input during eval.<br>
The <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html?highlight=batchnorm2d#torch.nn.BatchNorm2d">docs</a> explain this too and also give the formula how the running stats are updated.</p>