BN layer during testing

emma_ng: <p>Hi, I am confused about the BatchNorm layer behaviour during testing:</p>
<p>In previous answer (<a href="https://discuss.pytorch.org/t/the-behavior-of-the-bn-layer-in-train-and-eval-mode/152962" class="inline-onebox">The behavior of the BN layer in train and eval mode</a>), it is mentioned that if I set model.eval(), the running stats (mean, std) will be used to do normalization.<br>
<strong>What is the running stats? Are those values fixed from the trained model? Or from the new test data?</strong></p>
<p>For example, during test, I set model to <code>model.eval()</code>, then iterate through each test data and save the predicted value. (Example code below)<br>
<strong>Is this the correct way?</strong></p>
<p><strong>In this case, the running stats (mean, std) are changed after every test sample? Would this cause any problem?</strong></p>
<p>Thank you!</p>
<pre><code class="lang-auto">import torch

def main():
    # Read trained model
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    model = MyModel()
    model = torch.nn.DataParallel(model)
    model.load_state_dict(torch.load(train_model_path, map_location=device))
    model.to(device)

    # Setting to eval mode
    model.eval() 

    # Read data
    test_data = CustomDataset(image_dirs, test_csv)
    print('Number of test samples', len(test_data))

    preds = []
    for image_input, label in test_data:
        # Evaluation
        with torch.no_grad():
            output = model(image_input)
        
        # Prediction
        _, pred = torch.max(output, 1)

        # Compute some scores
        outputs_sm = F.softmax(output, dim=1)
        pred_score = outputs_sm[:, 1].item()

        pred_dx = pred.item()

        print(img_id, pred_score)


if __name__ == "__main__":
    main()
</code></pre>

ptrblck: <aside class="quote no-group" data-username="emma_ng" data-post="1" data-topic="171251">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/b4bc9f/48.png" class="avatar"> emma_ng:</div>
<blockquote>
<p>In this case, the running stats (mean, std) are changed after every test sample?</p>
</blockquote>
</aside>
<p>No, the running stats are only updated during training as described in the linked post. During testing/evaluation (i.e. after calling <code>.eval()</code> on the batchnorm layer) the running stats will only be <em>used</em> to normalize the input activations. No updates are performed anymore.</p>

emma_ng: <p>Thanks for the quick response!</p>
<p>The running stats that you’re referring to is the running stats of the final batch of training data, is that correct?</p>

ptrblck: <p>No, the running stats are stored as the internal <code>running_mean</code> and <code>running_var</code> and updated during training (i.e. after calling <code>.train()</code> on the batchnorm layer which is the default mode after initialization) in each forward pass using the specified <code>momentum</code> as described in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html#torch.nn.BatchNorm2d">docs</a>:</p>
<blockquote>
<p>This <code>momentum</code> argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is <code>x^new​=(1−momentum)×x^+momentum×xt​</code>, where <code>x^</code> is the estimated statistic and <code>xt</code>​ is the new observed value.</p>
</blockquote>

emma_ng: <p>I understand now! Thank you so much!</p>