Set track_running_stats=False during training，but running_mean still update

111299: <p>I want to fix the running_mean and running_var in BN during some training iteration.<br>
So I set track_running_stats=False during training，but why running_mean still update.</p>
<pre><code class="lang-auto">bn = nn.BatchNorm2d(3)
for _ in range(10):
    x = torch.randn(10, 3, 24, 24)
    out = bn(x)

print(bn.running_mean)
&gt; tensor([ 0.0031, -0.0034, -0.0066])
print(bn.running_var)
&gt; tensor([1.0027, 0.9977, 1.0081])

print(bn)
&gt; BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
bn.track_running_stats=False
print(bn)
&gt; BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False)

for _ in range(10):
    x = torch.randn(10, 3, 24, 24)
    out = bn(x)
print(bn.running_mean)
&gt; tensor([ 0.0014, -0.0021, -0.0048])
print(bn.running_var)
&gt; tensor([1.0036, 0.9980, 0.9983])
</code></pre>

ptrblck: <p><code>track_running_stats</code> will be used to initialize the running stats in the <code>__init__</code> method as seen <a href="https://github.com/pytorch/pytorch/blob/b37080d97a2188b4e6cf78b255c9399e123994a2/torch/nn/modules/batchnorm.py#L31">here</a>.<br>
If you don’t want to update these stats, you should call <code>bn.eval()</code> and reset it afterwards via <code>bn.train()</code> to train them further.</p>

GabbyChan: <p>Can you give an example code to illustrate the sequence of the process that you mentioned?</p>
<p>I am having the problem of using Batch normalization. The performance of my model drops significantly when running at inference mode (net.eval()).</p>
<p>I am looking forward to how to properly <strong>train and run the inference</strong> my model with Batch normalization with <code>track_running_stats=False</code>.  Could you give the guidance.</p>

ptrblck: <aside class="quote no-group" data-username="GabbyChan" data-post="3" data-topic="78644">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gabbychan/48/18041_2.png" class="avatar"> GabbyChan:</div>
<blockquote>
<p>I am looking forward to how to properly <strong>train and run the inference</strong> my model with Batch normalization with <code>track_running_stats=False</code>. Could you give the guidance.</p>
</blockquote>
</aside>
<p>If you set <code>track_running_stats=False</code>, the batchnorm layer will always use the batch statistics to normalize the activations, so there is no need to call <code>train()</code> or <code>eval()</code> on this layer anymore.</p>

GabbyChan: <p>I see. Thank you very much. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Sahil_Goyal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , I used the track_running_stats=False. But changing batch_size during test time changes results dramatically (say using batch_size=32 gives 99% accuracy and using batch_size=1 gives 81% accuracy). How to handle that?</p>

ptrblck: <p>This is expected since with <code>track_running_stats=False</code> the normalization will be performed based on the batch statistics and is thus dependent on the batch size, which is generally not a desired property for model inference. The best you could do is most likely trying to keep the batch size in the “optimal” setup.</p>

Sahil_Goyal: <p>Okay, Actually my network architecture is a Siamese network(Metric Learning) inspired by some paper where they have set <code>track_running_stats=False.</code> Now due to the problem of batch size during inference, I tried with <code>track_running_stats=True,</code> my same model started heavily overfitting even if I increased the Dropout and weight decay.<br>
Does that mean running mean and variance are unsuitable for my test data?<br>
Any advice? It will be a great help.</p>

ptrblck: <p>That’s hard to tell, but depending on the input data and its stats the running stats of batchnorm layers might indeed not converge to the batch statistics.</p>

Sahil_Goyal: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="78644">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>depending on the input data and its stats the running stats of batchnorm layers might indeed not converge to the batch statistics.</p>
</blockquote>
</aside>
<p>Thank you for your response. So do you suggest trying LayerNorm or InstanceNorm in such a case? I know, I can just try and see the results but wanted to know your intuition.</p>