How to use have batch norm not forget batch statistics it just used?

username: Brando_Miranda
cooked: <p>I am in an unusual setting where I should not use running statistics (as that would be considered cheating e.g. meta-learning). However, I often run a forward pass on a set of points (5 in fact) and then I want to evaluate only on 1 point <strong>using the previous statistics</strong> but batch norm forgets the batch statistics it just uses. I’ve tried to hard code the value it should be but I get strange errors (even when I uncomment things like from the pytorch code itself like checking the dimension size).</p>
<p>How do I hardcode the previous batch statistics so that batch norm works on a new single data point and then reset them for a fresh new next batch?</p>
<p>note: I don’t want to change the batch norm layer type.</p>
<p>Sample code I tried:</p>
<pre><code class="lang-auto">def set_tracking_running_stats(model):
    for attr in dir(model):
        if 'bn' in attr:
            target_attr = getattr(model, attr)
            target_attr.track_running_stats = True
            target_attr.running_mean = torch.nn.Parameter(torch.zeros(target_attr.num_features, requires_grad=False))
            target_attr.running_var = torch.nn.Parameter(torch.ones(target_attr.num_features, requires_grad=False))
            target_attr.num_batches_tracked = torch.nn.Parameter(torch.tensor(0, dtype=torch.long), requires_grad=False)
            # target_attr.reset_running_stats()
    return
</code></pre>
<p>my most comment errors:</p>
<pre><code class="lang-auto">    raise ValueError('expected 2D or 3D input (got {}D input)'
ValueError: expected 2D or 3D input (got 1D input)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
</code></pre>

username: Brando_Miranda
cooked: <p>related: <a href="https://discuss.pytorch.org/t/how-does-pytorch-s-batch-norm-know-if-the-forward-pass-its-doing-is-for-inference-or-training/16857" class="inline-onebox">How does pytorch’s batch norm know if the forward pass its doing is for inference or training?</a></p>

username: Brando_Miranda
cooked: <p>related: <a href="https://stackoverflow.com/questions/69845469/when-should-one-call-eval-and-train-when-doing-maml-with-the-pytorch-highe" class="inline-onebox" rel="noopener nofollow ugc">machine learning - When should one call .eval() and .train() when doing MAML with the PyTorch higher library? - Stack Overflow</a></p>

username: Brando_Miranda
cooked: <p>Solution is to use <code>mdl.train()</code> it uses batch statistics by itself:</p>
<blockquote>
<p>Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default <code>momentum</code> of 0.1.</p>
<p>If <code>track_running_stats</code> is set to <code>False</code>, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well.</p>
</blockquote>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html</a></p>