Model (resnet34) performs better without model.eval()

username: Giuseppe_Sarno
cooked: <p>Hello,<br>
I am using a ResNet34 and retrained fully to recognize sines in a voice message. With relatively short train set I was able to achieve promising results (almost 100% accuracy - I split the signal in windows and then take the mel spectrogram images for training and inference - pretty standard approach).<br>
All this until I found out I was not calling model.eval(). I then added to my code and since then the model is not working at all or in any case the performance is really bad. (lots of false positives).</p>
<p>Now I read for inference we must set model.eval() but I don’t quite understand why not setting it actually works and so well. Is there some technique I can implement in those circumstances ? I see other people having similar problem but do not see anything that I can do using a predefined model.</p>
<p>Your help is really appreciated.</p>

username: ptrblck
cooked: <p>Based on your description it seems the running stats of the batchnorm layers might not properly capturing the mean and stddev of your validation dataset which then causes the bad results. We have a few topics where similar issues were discussed and e.g. changing the <code>momentum</code> of the batchnorm layers was suggested to try to smooth the running stats updates.</p>

username: Giuseppe_Sarno
cooked: <p>Thank you,<br>
I have read few more articles on the subject… It is possible that my training set is not big enough.<br>
However reading I have seen that other people in similar situations have not allowed the batch norm to fix the var and std during inference.<br>
<a href="https://towardsdatascience.com/batch-normalization-in-3-levels-of-understanding-14c2da90a338" rel="noopener nofollow ugc">Batch normalization in 3 levels of understanding</a></p>
<p>I tried this code</p>
<pre><code class="lang-auto">def _freeze_norm_stats(net):
    try:
        for m in net.modules():
            if isinstance(m, nn.BatchNorm2d):
                #m.track_running_stats = False
                m.train()

    except ValueError:  
        print("error")
        return
</code></pre>
<p>and just before inference setting eval() and simply changed the batchnorm.</p>
<pre><code class="lang-auto">new_model_pred.eval()
_freeze_norm_stats(new_model_pred)
</code></pre>
<p>It seems to work well on the data in my use case.</p>
<p>Is there a drawback I do not see ?</p>
<p>Obviously with time I can build a wide enough set of training data that should fix this problem.</p>

username: ptrblck
cooked: <p>The drawback of using the batch stats only (during training <em>and</em> validation) would be the dependency on the used batch size. Since now you are normalizing the activations using the batch statistics you might need to keep the batch size equal as changing it might yield different results.</p>

username: Giuseppe_Sarno
cooked: <p>Thank you, learning a lot.</p>
<p>I am now running a test where I am changing the momentum similar code but now fixing momentum to a higher value. I think fixing the batch size is currently not too bad as step 1. I am building a PoC at this stage.  Let’s see what the momentum test will tells me.</p>

username: Giuseppe_Sarno
cooked: <p>So the test still shows worse performance but when I run on the same data as training is lot better than before. Surely is about the variance on the input data from training to inference.</p>
<p>Is there a way to visualize the variance, std for pictures in a resnet34 network ? It will help me understand how the input data differs from one data set to another.</p>
<p>I generally apply the same pre-processing and using porch audio I apply same transformation so I would expect the data to be ranging in the same bulk.</p>
<p>Thanks,</p>

username: ptrblck
cooked: <p>You could use forward hook and store some stats of the intermediate activations you are interested in. <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">This post</a> explains how to use forward hooks on modules.</p>