Using BatchNorm gives a significantly low accuracy

jitesh: <p>I am working on an image classification problem on a Custom Dataset consisting of <code>1888 </code>training images and <code>800</code> validation images(8 classes). I have tried applying transfer learning using various models from the <code>torchvision.models</code> library. For each model, I am using pre-trained weights and am only training the final <code>Linear</code> layer which performs classification. I have the following results so far on my validation set(using <code>batch size 32</code> and using <code>SGD with momentum</code> as the optimizer with <code>lr 0.001</code>)</p>
<pre><code class="lang-auto">1. Alexnet - 93 
2. VGG16 - 93  
3. VGG16_bn - 57 
4. Resnet50 - 26
5. VGG19 - 91
6. VGG19_bn - 59 
</code></pre>
<p>I have repeated the experiments with both <code>model.train()</code> and <code>model.eval()</code> but the results do not seem to change much. So, from the results, I am guessing that the models having <code>BatchNorm</code> layers are performing very poorly compared to other models that don’t have them. Any ideas why this might be happening? Any help would be appreciated. Thanks!</p>
<p>I have looked at <a href="https://discuss.pytorch.org/t/model-eval-gives-incorrect-loss-for-model-with-batchnorm-layers/7561/3">this</a> post on the forums about a possible solution which involves increasing the momentum value in the BatchNorm constructor.<br>
How exactly am I supposed to make this change? Do I have to manually change the <code>batchnorm.py</code> code or is there any better way to make this happen?</p>

ptrblck: <p>Do you see an increased accuracy, if you run the evaluation in <code>model.train()</code> so that the batchnorm layers will use the batch statistics?</p>
<p>You can change the momentum by directly assigning the new value to this attribute:</p>
<pre><code class="lang-python">model = ...
model.bn.momentum = new_value
</code></pre>

jitesh: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for the help. I have summarised a list of results which I have obtained. So, basically <code>train()</code> and <code>eval()</code> makes little difference. And changing the momentum value from the default 0.1 to 0.5 also did not seem to make much of a difference. I am sharing the plots of loss and accuracy which I have obtained after training.</p>
<p><code>Normal VGG16  without batchnorm</code><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/5/556d3600a60751909ce7e4c3333d7189cc9ec58a.png" alt="vgg16_loss" data-base62-sha1="cbIzeN19GUZxwOZIvr38tMaSSeu" width="362" height="248"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/6/5/654ebe6398618f94bd3f01811a69573348eed833.png" alt="vgg16_acc" data-base62-sha1="escVN6FyF1sveuAIxZSt5a3u45J" width="368" height="251"></p>
<p><code>VGG16  with batchnorm,set to eval()</code><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/7/67de0bbd426ce75664074734cfbb3a1b908c881f.png" alt="vgg16_bn_eval_default_momentum_loss" data-base62-sha1="eOQUs01TifE5QwIxDuCuRToLJoP" width="362" height="251"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4f238eb3f4f8d5eaf478c1f6adf698688a7a8445.png" alt="vgg16_bn_eval_default_momentum_acc" data-base62-sha1="bi5TTO9LwIZYJjmuh4BLrn18hMN" width="368" height="248"></p>
<p><code>VGG16  with batchnorm, set to train()</code><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b56f21953bfbf5aa71178fceca4ef1f2e903791.png" alt="vgg16_bn_train_default_momentum_acc" data-base62-sha1="3TRaFb8CGksIi1GFihgNmEA4lxv" width="368" height="248"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a859694cab66e324b61d896a5f67aacc22d9bab9.png" alt="vgg16_bn_train_default_momentum_loss" data-base62-sha1="o1hMejGMk5VHgSDm8tuhU4ooAPf" width="362" height="250"></p>
<p><code>VGG16  with batchnorm, set to eval() and momentum changed to 0.5</code><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b0d1bf276079544c1d77db2702884d5c962885bb.png" alt="vgg16_bn_eval_5_momentum_acc" data-base62-sha1="pedpJkOeqeswmkFAVqE8OJxSC7N" width="368" height="248"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e6531780ab437c1f960d57546142c6f8f94f7253.png" alt="vgg16_bn_eval_5_momentum_loss" data-base62-sha1="wRxQb1OVOQqmFVZy5K9TSSRShd9" width="362" height="251"></p>
<p><code>VGG16  with batchnorm, set to train() and momentum changed to 0.5</code><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/7660bb1493ee4652d13e66ab5337d4ca33562856.png" alt="vgg16_bn_train_5_momentum_acc" data-base62-sha1="gTdAlzZ70neNpDZrFHzcHYX2Uvk" width="368" height="248"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed972df9aac099a62fc2c20a285f09695327cb1e.png" alt="vgg16_bn_train_5_momentum_loss" data-base62-sha1="xTP3NxNnI3LwF5x8Y8r2uNRaiiq" width="362" height="250"></p>
<p>So, from the graphs, it’s pretty clear that <code>BatchNorm</code> in general has no improvement irrespective of the momentum value and the mode. Any ideas what could be causing the problem?</p>
<p>I have also shared a link to my Google Colab Notebook which I used for training <a href="https://colab.research.google.com/drive/18fOrzaS8mZIXvaoU1gecy5V1ytUbtLCF" rel="noopener nofollow ugc">here</a> if it further helps in debugging anything</p>
<p>Thanks!</p>