Question about nn.BatchNorm2d

username: simon_thom
cooked: <p>Hi</p>
<p>When testing a model using nn.BatchNorm2d between convolutional layers, I saw that the error (cross entropy loss) over the test set was lower than the error for the training set.</p>
<p>I read some topics on here about testing and I realized I hadn’t used model.eval() before feeding the test set to the model.</p>
<p>What still isn’t clear to me is that in these topics, it is said that BatchNorm2d changes behaviour, when the model is set to model.eval().</p>
<p>My question is the following:<br>
I would think that this behaviour change would imply that for</p>
<pre><code class="lang-auto">torch.nn.BatchNorm2d(num_features, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
</code></pre>
<p>then “track_running_stats” should be false after setting the model to eval()</p>
<p>However I can see in the summary of model = model.eval() that “track_running_stats” is still true. Shouldn’t this be false, as the batchnorm layer should use the statistics learned from training?</p>
<p>Thanks in advance for any inputs.<br>
Simon</p>

username: simon_thom
cooked: <p>Another way to state my question, in a code example which I hope is fairly general, as I couldn’t seem to find general examples on the PyTorch docs or on here.</p>
<p>Is this the correct idea of a way of training and testing for several epochs in PyTorch?</p>
<pre><code class="lang-auto">for epoch in range(no_epochs):
    
    running_loss = 0
    
    for image,target in training_set:        
        
        unet, loss = defnet.train( unet, image, target)
        
        running_loss += loss
    
    training_loss[epoch] = running_loss / len(training_set)
    

    running_test_loss =0
    with torch.no_grad():
        criterion = nn.CrossEntropyLoss()
        evaluation_unet = unet.eval()
        for image, target in test_set:

            running_test_loss += criterion(  evaluation_unet(image),  target)
        
        loss_validation[epoch] = cel / len(test_set)

</code></pre>
<p>My concern is that I don’t want my model to “go on training” on the test set. Is “torch.no_grad” and “model.eval()” enough to make sure this does not happen?</p>

username: ptrblck
cooked: <p>If you call <code>model.eval()</code>, the internal <code>self.training</code> flag will be set to <code>False</code> and e.g. the running estimates in batchnorm layers won’t be updated anymore but applied to the current batch.<br>
Note that <code>model.train()</code> and <code>model.eval()</code> work recursively on all modules, so that you don’t have to reassign the model.</p>
<p>The training loop looks strange, as you are calling <code>defeat.train</code> with some arguments.<br>
Are you using some high-level wrapper?</p>
<p>Also note that it looks as if you are storing the computation graph using <code>running_loss += loss</code>, which might increase your memory usage in each iteration.<br>
If you would like to add the current loss to <code>running_loss</code> to print it later, use <code>running_loss += loss.item()</code> instead.</p>

username: simon_thom
cooked: <p>Hi ptrblck</p>
<p>Thank you for the reply. All right that makes sense.</p>
<p>My batch size is 1 (i.e. single images are passed forward). My understanding of the implementation of running mean and running variance is that it remembers the statistics of the previous batches and adjusts the statistics for the normalization layers as it goes through batches (i.e. in my case the single training images). This means that a batch size of 1 is not a problem.</p>
<p>Can you briefly state if this is correctly understood?</p>
<p>“defnet” is an import of a script with my network as well as this training function:</p>
<pre><code class="lang-auto">def train(network, image, target):
    criterion = nn.CrossEntropyLoss()
    optimizer = ts.optim.Adam(network.parameters())
     
    # variable wrap
    image, target = ts.autograd.Variable(image), ts.autograd.Variable(target)
        
    # forwardpass, backprop, optimization step
    prediction = network(image)
    optimizer.zero_grad()
    loss = criterion(prediction, target)
    rloss = loss.item()
    loss.backward()
    optimizer.step()
        
    return network, rloss
</code></pre>
<p>I would think that this would work, but I don’t know if there is a more efficient way around training.</p>
<p>The training function by the way returns loss.item() to answer your last question.</p>
<p>Best regards,<br>
Simon</p>

username: simon_thom
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>If I train my model with model.train() on a training set and evaluate with model.eval() on the same training set, shouldn’t I be able to reproduce the exact same loss on the training set with model.eval()?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="simon_thom" data-post="4" data-topic="48022">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simon_thom/48/12582_2.png" class="avatar"> simon_thom:</div>
<blockquote>
<p>This means that a batch size of 1 is not a problem.</p>
</blockquote>
</aside>
<p>A small batch size might be a problem for batch norm layers, as the estimate of the mean and std might be quite off. If you can’t fit more than a single sample into a batch you might want to try other normalization layers, e.g. <code>InstanceNorm</code> (or <code>GroupNorm</code> for small batch sizes).</p>
<p>The <code>train</code> function is re-initializing the optimizer in each iteration, which will reset the internal buffers of <code>Adam</code>. I would recommend passing the criterion and optimizer to this <code>train</code> method or just call the training procedure in the <code>DataLoader</code> loop.</p>
<aside class="quote no-group" data-username="simon_thom" data-post="5" data-topic="48022">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simon_thom/48/12582_2.png" class="avatar"> simon_thom:</div>
<blockquote>
<p>If I train my model with model.train() on a training set and evaluate with model.eval() on the same training set, shouldn’t I be able to reproduce the exact same loss on the training set with model.eval()?</p>
</blockquote>
</aside>
<p>Not really, as some layers e.g. <code>Dropout</code> and <code>BatchNorm</code> will behave differently as explained before.</p>

username: simon_thom
cooked: <p>Hi</p>
<p>Thanks very much for the reply. There are some useful things to try out!</p>
<p>BR,<br>
Simon</p>