Changing batch norm statistics doesn't change result

username: mo.dify
cooked: <p>Hi,</p>
<p>During inference, I switch the batch norm modules to be training mode (batchnorm.train())<br>
And I manually change variance in every inference step by this code.</p>
<pre><code class="lang-auto">for _, m in enumerate(encoder.modules()):
    if isinstance(m, torch.nn.modules.batchnorm._BatchNorm):
        print("Running Var", m.running_var)
        m.train()
        m.running_var = 100 * m.running_var
</code></pre>
<p>I print out the batch norm parameter’s every inference step, and variance increases (and be inf at the late inference steps).</p>
<p>However, the output of the network doesn’t change. My case is auto encoder that receives and reconstructs images. The reconstructed images wouldn’t change as the batch norm parameters change. I’m wondering why this happens.</p>
<p>I use PytorchLighting for the model, is it because of the library?</p>
<p>Update:</p>
<p>I try m.eval() instead and the results make sense now.</p>
<p>I wonder if it is because we can’t change m.running_mean, m.running_var, m.momentum during m.train().</p>
<p>But if so, why m.running_var changes as I mentioned above that m.running_var can still increase over inference steps to be inf at the end.</p>
<pre><code class="lang-auto">for _, m in enumerate(encoder.modules()):
    if isinstance(m, torch.nn.modules.batchnorm._BatchNorm):
        print("Running Var", m.running_var)
        m.eval()
        m.running_var = 100 * m.running_var
</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="mo.dify" data-post="1" data-topic="197152">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mo.dify/48/62009_2.png" class="avatar"> mo.dify:</div>
<blockquote>
<p>I try m.eval() instead and the results make sense now.</p>
<p>I wonder if it is because we can’t change m.running_mean, m.running_var, m.momentum during m.train().</p>
</blockquote>
</aside>
<p>The running stats will be updated using the batch stats in <code>train()</code> mode. In <code>eval()</code> mode they will be used to normalize the input and not updated anymore.</p>
<aside class="quote no-group" data-username="mo.dify" data-post="1" data-topic="197152">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mo.dify/48/62009_2.png" class="avatar"> mo.dify:</div>
<blockquote>
<p>But if so, why m.running_var changes as I mentioned above that m.running_var can still increase over inference steps to be inf at the end.</p>
</blockquote>
</aside>
<p>You are manually increasing it, so it’s expected that the values overflow at one point.</p>

username: mo.dify
cooked: <p>Hi,<br>
Thank you for your reply.<br>
What I mean is that when I set batch stats (running_mean, running_var, momentum) manually, the output won’t change in .train() mode.</p>
<p>For example, if I set running_var = inf in .eval() mode, the output vector is [beta, beta, beta, …, beta], where beta is the affine parameter in batch norm transformation.</p>
<p>But if I switch to .train() mode, even I set batch stats manually, running_var = inf, the results are something like [x1, x2, x3,…,xM], but it should be beta vector if the batch stats really change to what I set.</p>
<p>Does this mean we cannot change batch stats manually in .train() mode?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="mo.dify" data-post="3" data-topic="197152">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mo.dify/48/62009_2.png" class="avatar"> mo.dify:</div>
<blockquote>
<p>What I mean is that when I set batch stats (running_mean, running_var, momentum) manually, the output won’t change in .train() mode.</p>
</blockquote>
</aside>
<p>This is expected since the running stats will only be used to normalize the input during <code>eval()</code>. During <code>train()</code> the running stats will be updated but not used to normalize the input as already explained. Maybe looking at <a href="https://github.com/ptrblck/pytorch_misc/blob/31ac50c415f16cf7fec277dbdba72b9fb4d732d3/batch_norm_manual.py#L39-L78">this manual implementation</a> clarifies the behavior.</p>

username: mo.dify
cooked: <p>Thank you so much.<br>
The implementation you give me is very clear and helpful.</p>