How does one use the mean and std from training in Batch Norm?

username: Brando_Miranda
cooked: <p>I wanted to use the means, stds from training rather than batch stats since it seems if I use batch statistics my model diverges (as outline here <a href="https://stackoverflow.com/questions/69845469/when-should-one-call-eval-and-train-when-doing-maml-with-the-pytorch-highe" class="inline-onebox" rel="noopener nofollow ugc">machine learning - When should one call .eval() and .train() when doing MAML with the PyTorch higher library? - Stack Overflow</a>). How does one do that?</p>
<p>I am asking since my model seems to have them be zero despite no training having been done yet:</p>
<pre><code class="lang-auto">Out[1]: BatchNorm2d(32, eps=0.001, momentum=0.95, affine=True, track_running_stats=True)
args.base_model.model.features.norm1.running_mean
Out[2]: 
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0.])
</code></pre>
<p>are these not saved in a ckpt after training? Should they have been saved?</p>
<p>Docs say they should have:</p>
<blockquote>
<p>Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation.</p>
</blockquote>
<p>by running means are zero vectors… <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"> ?</p>
<hr>
<p>related:</p>
<ul>
<li><a href="https://stackoverflow.com/questions/69846779/how-does-one-use-the-mean-and-std-from-training-in-batch-norm" class="inline-onebox" rel="noopener nofollow ugc">machine learning - How does one use the mean and std from training in Batch Norm? - Stack Overflow</a></li>
<li><a href="https://stackoverflow.com/questions/69856207/what-is-the-proper-way-to-do-evaluation-e-g-testing-validation-of-deep-learn" class="inline-onebox" rel="noopener nofollow ugc">What is the proper way to do evaluation (e.g. testing, validation) of deep learning model in PyTorch so that checkpoints are saved correctly? - Stack Overflow</a></li>
<li><a href="https://discuss.pytorch.org/t/inconsistent-batchnorm-behavior-in-eval-and-training-modes/133025" class="inline-onebox">Inconsistent Batchnorm behavior in eval and training modes</a></li>
</ul>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Brando_Miranda" data-post="1" data-topic="136029">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar"> Brando_Miranda:</div>
<blockquote>
<p>I am asking since my model seems to have them be zero despite no training having been done yet</p>
</blockquote>
</aside>
<p>That’s expected, since the <code>running_mean</code> will be initialized with zeros and the <code>running_var</code> with ones.<br>
Both are update during the training, so if no training was performed yet, they will keep their initial values.</p>
<p>If you don’t want to use the running stats but want to normalize the input activations with their batch stats, use <code>track_running_stats=False</code>.</p>

username: Brando_Miranda
cooked: <p>are they zero even from a saved checkpoint?</p>

username: Brando_Miranda
cooked: <p>I’ve been told that someone else’s running_mean is not zero from a saved checkpoint. Is this the issue when alternating evaluation and training modes in models? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: Brando_Miranda
cooked: <p>after reading the BN code in detail and posts around here + the original paper the conclusion is here <a href="https://discuss.pytorch.org/t/inconsistent-batchnorm-behavior-in-eval-and-training-modes/133025/4" class="inline-onebox">Inconsistent Batchnorm behavior in eval and training modes - #4 by Brando_Miranda</a>  in summary:</p>
<blockquote>
<p>BN intended behaviour:</p>
<ul>
<li>Importantly, during inference (eval/testing) running_mean, running_std is used (because they want a deterministic output and to use estimates of the population statistics).</li>
<li>During training the batch statistics is used but a population statistic is estimated with running averages. I assume the reason batch_stats is used during training is to introduce noise that regularizes training (noise robustness)</li>
</ul>
</blockquote>
<p>ref: <a href="https://arxiv.org/abs/1502.03167" class="inline-onebox" rel="noopener nofollow ugc">[1502.03167] Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift</a></p>
<hr>
<p>So <strong>the main mystery is to figure out why my models were saved this way and their running averages from training removed</strong>.</p>
<hr>
<p>meta-leanring context: <a href="https://github.com/facebookresearch/higher/issues/107" class="inline-onebox" rel="noopener nofollow ugc">Is there data leakage in the maml-omniglot example? · Issue #107 · facebookresearch/higher · GitHub</a></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Brando_Miranda" data-post="3" data-topic="136029" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar"> Brando_Miranda:</div>
<blockquote>
<p>are they zero even from a saved checkpoint?</p>
</blockquote>
</aside>
<p>The running stats are restored from the checkpoint. If they were updated previously and stored with these updated values, loading the <code>state_dict</code> would restore these updated values.<br>
If they were never updated and the model was used in <code>model.eval()</code> during training, their initial values would be stored and loaded.</p>

username: Brando_Miranda
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> do you have any guess how come my running mean are set to a value of zero? My understanding is that they should never have that value. In eval mode we use the running_mean. In “not tracking running mean” they should be None. So to I am so puzzled where to even start looking into my code where the running mean could have been set to zero before saving my checkpoint. Do you have an idea how this could happen? Have you ever seen such a weird thing where the checkpoint has zero in the running means?</p>
<pre><code class="lang-auto">args.mdl1.model.features.norm1.running_mean
Out[6]: 
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0.])
args.mdl1.model.features.norm1.running_var
Out[7]: 
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]
</code></pre>
<p>I don’t think this should ever be happening according to my 3hours or so of reading the docs and all the pytorch code in some detail.</p>
<p>Doing a search over my code shows the only places where <code>reset_running_stats </code> exists is in batch_norm and <code>/Users/brando/anaconda3/envs/metalearning/lib/python3.9/site-packages/torch/nn/intrinsic/qat/modules/conv_fused.py</code> (code I didn’t write and I’m not calling).</p>
<p>also, in the construction of the BN layer <code>__init__</code> thats when it’s done (<a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/batchnorm.html#BatchNorm2d" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.modules.batchnorm — PyTorch 1.10.0 documentation</a>)…so my model somehow wasn’t tracking them during training but when I print the checkpoint the <code>track_running_stats</code> is <code>True</code> which puzzles me even more.</p>
<pre><code class="lang-auto">Out[8]: 
Learner(
  (model): ModuleDict(
    (features): Sequential(
      (conv1): Conv2d(3, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (norm1): BatchNorm2d(32, eps=0.001, momentum=0.95, affine=True, track_running_stats=True)
      (relu1): ReLU()
      (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (norm2): BatchNorm2d(32, eps=0.001, momentum=0.95, affine=True, track_running_stats=True)
      (relu2): ReLU()
      (pool2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (conv3): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (norm3): BatchNorm2d(32, eps=0.001, momentum=0.95, affine=True, track_running_stats=True)
      (relu3): ReLU()
      (pool3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (conv4): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (norm4): BatchNorm2d(32, eps=0.001, momentum=0.95, affine=True, track_running_stats=True)
      (relu4): ReLU()
      (pool4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    )
    (cls): Linear(in_features=800, out_features=5, bias=True)
  )
)
</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Brando_Miranda" data-post="7" data-topic="136029">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar"> Brando_Miranda:</div>
<blockquote>
<p>My understanding is that they should never have that value.</p>
</blockquote>
</aside>
<p>They should have these values after they are initialized. I would guess that your training might set the batchnorm layers or the entire model into <code>.eval()</code> mode so that the running stats are never updated and keep their initial values.</p>
<p>Check your code for <code>.eval()</code> calls (additionally also for <code>self.training = False</code> assignments) and see if that might be the issue.</p>