Conflict between model.eval() and .train() with multiprocess training and evaluation

Felix_Lessange: <p>Hi.</p>
<p>I’m using multiprocessing to train a model. I have two processes taking batches from queues (I’m actually using mp.Arrays that are more efficient, but a little trickier) and sharing my model weights.</p>
<p>Every ten batches seen, they make a validation iteration. In a trainBatch() method and in a valBatch() method, I placed calls to model.train() or model.eval(). They go something like this :</p>
<pre><code class="lang-auto">def trainBatch(model, input, criterion, target, optimizer):
    optimizer.zero_grad()
    model.train()
    output = model(input)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
    return loss

def valBatch(model, input, criterion, target):
    model.eval()
    with torch.no_grad():
        output = model(input)
    loss = criterion(output, target)
    return loss

</code></pre>
<p>I’ve observed that without putting those calls, and therefore letting every iteration with train(True) (be it train or validation), the metrics are “rational”. But When I enable those calls, they become really weird, as can be seen here : (out_var and out_var_val are the variance of the logits, which I like to plot for diagnosis)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4c3065fcf12a8aeb44f43241d82f8f9c972851bd.png" data-download-href="https://discuss.pytorch.org/uploads/default/4c3065fcf12a8aeb44f43241d82f8f9c972851bd" title="Capture%20du%202018-07-19%2012-55-57"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4c3065fcf12a8aeb44f43241d82f8f9c972851bd_2_690x324.png" alt="Capture%20du%202018-07-19%2012-55-57" data-base62-sha1="aRZYOiQ8AZOSfoZv1gviIwUnJSJ" width="690" height="324" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4c3065fcf12a8aeb44f43241d82f8f9c972851bd_2_690x324.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4c3065fcf12a8aeb44f43241d82f8f9c972851bd_2_1035x486.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4c3065fcf12a8aeb44f43241d82f8f9c972851bd_2_1380x648.png 2x" data-dominant-color="F9F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture%20du%202018-07-19%2012-55-57</span><span class="informations">1527×719 113 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>(orange : the curves while calling model.eval() and model.train(), and in blue without those calls)</p>
<p>I wonder if the problem doesn’t stem from the fact that, in such a framework, there can be a conflict on the layers with two opposite calls. Almost always : mode.eval() will be called while the model is doing a training iteration.</p>
<p>I wonder if this doesn’t screw the training of BatchNorm layers entirely, since the metrics in validation seem to be so poor. Or maybe, calling model.eval() while a training is happening deactivates the BatchNorm entirely without altering the learning ?</p>
<p>What do you think is happening ? Is validation during training prohibited in multiprocessing ?</p>

Felix_Lessange: <p>Turns out this is related to : <a href="https://discuss.pytorch.org/t/performance-highly-degraded-when-eval-is-activated-in-the-test-phase/3323/16">Performance highly degraded when eval() is activated in the test phase</a></p>
<p>It’s a bug in pytorch’s definition of batchnorm according to those guys : <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/batchnorm.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/batchnorm.py</a></p>
<p>Their solution only partially solved my discrepancy.</p>

Ranahanocka: <p>As <a class="mention" href="/u/simonw">@SimonW</a> replied in that thread, it’s not a bug. It’s how batch norm works. In training phase the  mean/std are learned from the training data, and loaded in the evaluation phase.<br>
If you turn track_running_stats off (as suggested in the post) you will instead use the mean and std of the batch in eval mode. This is flawed and incorrect usage, since you will get an inference result which is based on the data in your batch.<br>
As an example, if you ran one image during inference - it would get a different value (much worse) than if you ran it with a batch of other images. If you change the batchsize during eval with tracking off you will see what I mean.</p>

Felix_Lessange: <p>It’s indeed not a bug. It’s still very challenging, especially if you’re doomed, like me, to use small batch sizes.</p>

Ranahanocka: <p>Have you tried using groupnorm instead of batch norm?<br>
It works better for me (especially when there are small batch sizes)</p>

Felix_Lessange: <p>It does, by it’s slower, heavier, and there’s no transferring from pretrained models with it <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"> so a bit tedious for my work !</p>