What does model.eval() do for batchnorm layer?

liangstein: <p>Hi Everyone,<br>
When doing predictions using a model trained with batchnorm, we should set the model to evaluation model. I have a question that how does the evaluation model affect barchnorm operation? What does evaluation model really do for batchnorm operations? Does the model ignore batchnorm?</p>

smth: <pre><code class="lang-auto">During training, this layer keeps a running estimate of its computed mean and variance. The running sum is kept with a default momentum of 0.1.

During evaluation, this running mean/variance is used for normalization.
</code></pre>
<p>Reference: <a href="http://pytorch.org/docs/master/nn.html#torch.nn.BatchNorm1d">http://pytorch.org/docs/master/nn.html#torch.nn.BatchNorm1d</a></p>

Joe1: <p>I got the same problem when I trained a model using BN layer. If I just need to test one image,the BN layer will affect the result because of the change of batch size?</p>

SimonW: <p>When evaluating you should use eval() mode and then batch size doesnt matter.</p>

Joe1: <p>Thanks~ I have solved the problem~</p>

synchro: <p>Hey Soumith,<br>
Maybe a trivial question:</p>
<ul>
<li>Trained a model with BN on CIFAR10, training accuracy is perfect</li>
<li>Testing with model.train(True) will get 76% accuracy</li>
<li>Tesing with model.eval() will get only 10% with a 0% in pretty much every category.</li>
</ul>
<p>Why is this? It should be the opposite, right? <a class="mention" href="/u/smth">@smth</a></p>

SimonW: <p>How did you construct the BN layers?</p>

synchro: <p>Standard way:</p>
<p><code>nn.BatchNorm2d(64)</code></p>
<p>Where 64 is the num of output filters of the previous layer.</p>

SimonW: <p>That’s weird. Do you mind sharing your script?</p>

synchro: <p>Hey Simon, sorry for being late.</p>
<p>The definition of the model is as follows, ignore the fact that hyper-params shoudn’t be defined in that way (it’s an old code). Any idea?</p>
<pre><code class="lang-python">class Keras_Cifar2(nn.Module):
    def __init__(self, rank1, rank2):
        super(Keras_Cifar2, self).__init__()

        # hyperparams
        self.kern = 3  # for all layers
        self.filt_size1 = 32
        self.filt_size2 = 64
        self.filt_fc1 = 512
        self.num_classes = 10

        self.conv1 = nn.Conv2d(3, 32, 3, padding=1)
        self.conv2 = nn.Conv2d(32, 32, 3)
        self.conv3 = nn.Conv2d(32, 64, 3, padding=1)
        self.conv4 = nn.Conv2d(64, 64, 3)

        self.pool = nn.MaxPool2d(2, 2)

        self.bn_1 = nn.BatchNorm2d(1)
        self.bn_2 = nn.BatchNorm2d(rank1)
        self.bn_3 = nn.BatchNorm2d(rank2)
        self.bn_4 = nn.BatchNorm2d(self.filt_fc1)
        self.bn_5 = nn.BatchNorm2d(self.num_classes)
        self.bn_6 = nn.BatchNorm2d(32)
        self.bn_7 = nn.BatchNorm2d(64)

        # decomposition
        self.cpdfc1 = nn.Conv2d(64, rank1, 1)
        self.cpdfc2 = nn.Conv2d(rank1, rank1, (6, 1))
        self.cpdfc3 = nn.Conv2d(rank1, rank1, (1, 6))
        self.cpdfc4 = nn.Conv2d(rank1, self.filt_fc1, 1)

        # conv2fc
        #self.conv2fc1 = nn.Conv2d(64, self.filt_fc1, 5)
        self.conv2fc2 = nn.Conv2d(self.filt_fc1, self.num_classes, 1)


    def forward(self, x):

        x = F.relu(self.conv1(x))
        x = self.bn_6(x)
        x = self.pool(F.relu(self.conv2(x)))
        x = self.bn_6(x)

        x = F.relu(self.conv3(x))
        x = self.bn_7(x)
        x = self.pool(F.relu(self.conv4(x)))
        x = self.bn_7(x)

        x = self.cpdfc1(x)
        x = self.bn_2(x)
        x = self.cpdfc2(x)
        x = self.bn_2(x)
        x = self.cpdfc3(x)
        x = self.bn_2(x)
        x = F.relu(self.cpdfc4(x))
        x = self.bn_4(x)
        x = self.conv2fc2(x)
        x = self.bn_5(x)

        x = x.view(-1, self.num_classes) 
        return x
</code></pre>

SimonW: <p>You shouldn’t re-use BN layers. For example, here</p>
<aside class="quote no-group" data-username="synchro" data-post="10" data-topic="7146">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/synchro/48/764_2.png" class="avatar"> synchro:</div>
<blockquote>
<p>x = F.relu(self.conv1(x))<br>
x = self.bn_6(x)<br>
x = self.pool(F.relu(self.conv2(x)))<br>
x = self.bn_6(x)</p>
</blockquote>
</aside>
<p><code>self.bn_6</code> sees data from two different layers, but accumulating the values to the same running stats buffer. Then the running stats will be inaccurate and the performance will suffer in <code>eval()</code> mode. Make sure that each BN layer is used only at one place in the network.</p>

synchro: <p>Oh, right. Probably when I wrote it I thought to define just a layer type and then use it as many times as it needed. But lol, that’s a single member.</p>
<p>Then a second question is: what’s the best practice in making heavy use of BNs? Write just as many as one needs, or define all of them in a dictctionary of BNs?</p>

SimonW: <p>In your case, the network is pretty sequential. So I’d suggest use construct a list of layers in sequential order (<code>F.relu</code> can also be written with module <code>nn.ReLU</code>) and use the <code>nn.Sequential</code> wrapper. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

synchro: <p>Yup. I didn’t use the Sequential container since it was taken like that from the tutorial, but it’ll get definitely cleaner with that.</p>

flyingmoth: <p><a class="mention" href="/u/smth">@smth</a> I want to know the parameter <em>running_var</em>  in the batch normalization refers to the variance or the standard deviation?</p>

vfmatzkin: <p>Thanks, I had the same issue.</p>

Frida: <p>Hi<br>
Can someone help me to understand why applying model.eval is better in the testing phase?<br>
Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

lugiavn: <p>Remove your last or even 2nd last BN. The BN normalizes feature, the last output is class scores and should not be normalized.</p>
<p>train mode BN uses stat from the batch, test phase it is essentially “cheating” because it accesses to other examples in the batch (hence cannot perform if batch size = 1)</p>

Frida: <p>I mean, why is it better to use model.eval and take the running statistics and not rely on the current test image statistics?</p>

lugiavn: <ol>
<li>because the params are trained on train stats, if test stats are different, then the result might be different</li>
<li>if you compute test stat, then you are basically “train” on test set, because stat in this case is a trained param. You can do it, nobody says you can’t, just that ppl would consider it “cheating”</li>
</ol>