Model.eval() gives incorrect loss for model with batchnorm layers

zhangboknight: <p>I tried to train a model with batchnorm layers. During the training, I set model.train(). Every 100 iteration, I validate the accuracy and set model.eval(). However, the validation is not correct. I don’t think this is due to overfitting because even if I use the same image as training, the testing loss is also quite different from the training loss. <strong>Also, if I still set model.train() during testing, the testing loss is correct.</strong> But such usage does not make sense because my model contains batchnorm layers.</p>
<p>Below is my training code:</p>
<pre><code>for epoch in range(num_epochs):
    epoch_loss = 0.0
    optimizer = lr_scheduler(optimizer, epoch)

    for iteration, data in enumerate(dataloader, 0):
        iter_index += 1
        label_patch = data['label_patch']
        residue_patch = data['residue_patch']
        stacked_patch = data['stacked_patch']
        microshift_patch = data['train_patch']
        inputs, residues, labels, microshifts = Variable(stacked_patch.type(dtype)), Variable(residue_patch.type(dtype), requires_grad=False), Variable(label_patch.type(dtype), requires_grad=False), Variable(microshift_patch.type(dtype), requires_grad=False)

        # set model to train mode (before zero grad)
        model.train()

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward
        outputs = model(inputs)
        loss = criterion(outputs, residues)

        # backward + optimize only if in training phase
        loss.backward()
        optimizer.step()

        # statistics
        epoch_loss += loss.data[0]

        # test the model every 100 iterations
        if iter_index % logging_iter == 0:
            loss_test, psnr_test = test_model(model)  

    # checkpoint for each epoch
    model_out_path = "checkpoints/model_epoch_{}_residue.pth".format(epoch)
    torch.save(model, model_out_path)</code></pre>

zhangboknight: <p>The testing code which is called every 100 training iters is as following:</p>
<pre><code>def test_model(model):
    psnr_test_avg = 0
    loss_test_avg = 0
    model.eval()
    
    for iteration, test_data in enumerate(dataloader_test, 0):
        label_test = test_data['label_patch']
        residue_test = test_data['residue_patch']
        stacked_test = test_data['stacked_patch']
        microshift_test = test_data['train_patch']
        inputs_test, residues_test, labels_test, microshifts_test = Variable(stacked_test.type(dtype), requires_grad=False), Variable(residue_test.type(dtype), requires_grad=False), Variable(label_test.type(dtype), requires_grad=False), Variable(microshift_test.type(dtype), requires_grad=False)
        outputs_test = model(inputs_test)
        loss_mse_test = criterion_mse(outputs_test + microshifts_test, labels_test).data.cpu().numpy()
        loss_l1_test = criterion(outputs_test, residues_test).data.cpu().numpy()
        psnr_test = 10 * np.log10(255 * 255 / loss_mse_test)
        loss_test_avg += loss_l1_test
        psnr_test_avg += psnr_test

    loss_test_avg /= (iteration + 1)
    psnr_test_avg /= (iteration + 1)

    return loss_test_avg, psnr_test_avg</code></pre>

smth: <p>it is possible that your training in general is unstable, so BatchNorm’s <code>running_mean</code> and <code>running_var</code> dont represent true batch statistics.</p>
<p><a href="http://pytorch.org/docs/master/nn.html?highlight=batchnorm#torch.nn.BatchNorm1d" class="onebox" target="_blank">http://pytorch.org/docs/master/nn.html?highlight=batchnorm#torch.nn.BatchNorm1d</a></p>
<p>Try the following:</p>
<ul>
<li>change the <code>momentum</code> term in BatchNorm constructor to higher.</li>
<li>before you set <code>model.eval()</code>, run a few inputs through <code>model</code> (just forward pass, you dont need to backward). This will help stabilize the running_mean / running_std values.</li>
</ul>
<p>Hope this helps.</p>

zhangboknight: <p>Thanks for your reply. I tried them but still get the error. I found that using the same code, sometimes the model.eval() can be correct but sometimes incorrect. I will try further and update if I found a solution.</p>

david-leon: <p>Same problem with latest 0.3.0 release here. Have you find any solution? <a class="mention" href="/u/zhangboknight">@zhangboknight</a><br>
Change <code>momentum</code> won’t solve the problem <a class="mention" href="/u/smth">@smth</a></p>

alexbellgrande: <p>I’m having the same issue. Really a bummer to have to use train mode for validation/testing.</p>

Higang: <p>I also have the same problem and haven’t figured out the reason.</p>

falmasri: <p>Any update regarding this problem. I already posted the same question. it seems to me that many are facing the same problem. Could pytorch community react to this problem ?</p>

meetshah1995: <p>I have the same problem.</p>
<p>I’m trying to load caffe weights in a pytorch model with batchnorm layers, each time I load the weights from the caffemodel file, the result for the same input is different even in eval mode.</p>
<p>I’m actually updating the running_mean and running_var from the caffemodel weights, so there shouldn’t be any issue with bad running_means during inference.</p>

smth: <p><a class="mention" href="/u/meetshah1995">@meetshah1995</a> the meaning of Caffe’s running_mean might be different from pytorch’s running_mean.</p>

smth: <p><a class="mention" href="/u/falmasri">@falmasri</a> I wrote above in the comment here: <a href="https://discuss.pytorch.org/t/model-eval-gives-incorrect-loss-for-model-with-batchnorm-layers/7561/3?u=smth">Model.eval() gives incorrect loss for model with batchnorm layers</a> with a working answer.</p>
<p>It’s not a problem in the sense that it’s not a software bug.</p>
<p>It’s a problem in the sense that if you have a non-stationary training, you will see this behavior unless you adjust your <a href="http://pytorch.org/docs/0.3.0/nn.html#torch.nn.BatchNorm2d">momentum</a> term of the BatchNorm. We set the <code>momentum</code> to 0.1 because for most workloads that we use it was sufficient. Play around with it.</p>

meetshah1995: <p><a class="mention" href="/u/smth">@smth</a>  Agreed that running_mean may mean different things in Caffe and PyTorch. However in eval mode, I guess only these 5 things - (running_mean, running_var, weight, bias, eps)  should affect the final output.</p>
<p>I’ll try to see if I can come up with a minimal reproducible example for this.</p>

falmasri: <p><a class="mention" href="/u/smth">@smth</a> What do you mean by non stationary training ?</p>

smth: <p><a class="mention" href="/u/falmasri">@falmasri</a> means the statistics of activations change rapidly during training, such that the running_mean and running_std statistics for BatchNorm at the momentum of 0.1 are not valid anymore.</p>

rohun: <p>Is it theoretically incorrect though?</p>

rohun: <p>Nice, setting the momentum to 0.5 seems to make loss calculated using model.eval() similar to the loss computed using model.train(), but only after few epochs of differing results.</p>
<p>Does the second suggestion work only when we are loading the model for eval? It should not affect the loss calculation, correct or incorrect, if we are training the network and running validation every nth step.</p>

Xingyi_Zhou: <p>I also met this problem in my project (See my answer at <a href="https://github.com/xingyizhou/pytorch-pose-hg-3d/issues/16" rel="nofollow noopener">https://github.com/xingyizhou/pytorch-pose-hg-3d/issues/16</a> and <a href="https://github.com/bearpaw/pytorch-pose/issues/33" rel="nofollow noopener">https://github.com/bearpaw/pytorch-pose/issues/33</a>). In short, down-grading pytorch version to 0.1.12 will resolve the problem. But I really don’t know what happens to the BN implementation from 0.1.12 to the later versions …</p>

SimonW: <p>I replied on the issue, but running stats is unstable in nature with batch size only being 1.</p>

Xingyi_Zhou: <p>Thanks for the reply! The training batch size is 6 instead of 1. Actually I have also tried later batch size (32) with other architectures (upsampling on ResNet18) but the bug remains. My major question is I don’t understand why pytorch 0.1.12 works while &gt;= 0.2 does not.</p>

youkaichao: <p>I think this is not about the <code>momentum</code>. I have the same problem. when I call</p>
<pre><code class="lang-auto">model.eval()
model(input)
model.train()
model(input)
model.eval()
model(input)
model.train()
model(input)
</code></pre>
<p>every call of <code>model(input)</code> is almost the same if it is after <code>model.train()</code> , but differs with what follows <code>model.eval()</code>.</p>