Model .eval() problem

Toby: <p>Hi,<br>
I met a strange bug:</p>
<p>My model: EfficientDet-D4 (following <a href="https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch" rel="noopener nofollow ugc">this repo</a>)</p>
<p>While training the model, I use <code>model.train()</code>, then change it to <code>model.eval()</code> in validate step and it worked normally.</p>
<p>However, in the test phase, my code is:</p>
<pre><code class="lang-python">from efficientdet.model import Classifier
model = EfficientDetBackbone(num_classes=len(params.obj_list), compound_coef=4,
                                 ratios=eval(params.anchors_ratios), scales=eval(params.anchors_scales))
model.backbone_net.model._conv_stem.conv = nn.Conv2d(4, 48, kernel_size=(3, 3), stride=(2, 2), bias=False)
model.classifier.header.pointwise_conv.conv = nn.Conv2d(224, 9, kernel_size=(1, 1), stride=(1, 1))
model.classifier = Classifier(in_channels=model.fpn_num_filters[4], num_anchors=model.num_anchors,
                            num_classes=1,
                            num_layers=model.box_class_repeats[4],
                            pyramid_levels=model.pyramid_levels[4])
model.load_state_dict(torch.load(weights_path),strict=False)
model.requires_grad_(False)
model.eval()

with torch.no_grad():
    for iter, data in enumerate(test_generator):
        imgs = data['img']
        _, regression, classification, anchors = model(imgs)
</code></pre>
<p>The result was really bad, it always returned the bounding boxes in the corner of the image, but when I commented out the line <code>model.eval()</code>, the result was really good. Any idea why is it?</p>

ptrblck: <p><code>model.eval()</code> disables the dropout and uses the internal running stats in all batchnorm layers (also custom modules might change their behavior using the <code>self.training</code> flag, which is changed by calling <code>eval()</code>).<br>
If your validation loss and predictions are bad, this might point to bad running estimates in the batchnorm layers. You would see it if e.g. the training and validation data are not processed in the same way (especially the normalization) or if they are sampled from another data domain and thus have different statistics.<br>
To counter this effect, you could play around with the <code>momentum</code> of the batchnorm layers to smooth the statistics a bit.<br>
Also, if not already used, you should shuffle the training dataset to avoid adding a bias for the stats updates based on the last samples in your training dataset.</p>

Toby: <p>Thank you for your reply,</p>
<p>I am sure that the data processing is the same, and I already shuffle data while training.</p>
<p>I think you make sense of the batch norm problem. I will check the <code>running_mean and running_variance</code>, then try to do something with momentum. However, I don’t think it to have that much of an impact.</p>
<p>This is the result with <code>model.train()</code></p>
<pre><code class="lang-auto">{'rois': array([[448.55865, 435.95306, 648.1225 , 714.6619 ]], dtype=float32),
 'class_ids': array([0], dtype=int64), 
'scores': array([0.9849996], dtype=float32)} 
</code></pre>
<p>And this is the result with <code>model.train(False)</code>:</p>
<pre><code class="lang-auto">{'rois': array([[   0.      ,  921.97406 ,  152.82751 , 1023.      ],
       [ 929.1798  ,  921.1621  , 1023.      , 1023.      ],
       [   0.      ,    0.      ,  159.41663 ,   97.29772 ],
       [ 874.1246  ,    0.      , 1023.      ,   99.22618 ],
       [ 719.88806 ,  820.15857 , 1023.      , 1023.      ],
       [   0.      ,    0.      ,   28.660963,   59.322357],
       [ 991.6581  ,  965.65967 , 1023.      , 1023.      ]],
      dtype=float32),
 'class_ids': array([0, 0, 0, 0, 0, 0, 0], dtype=int64), 
'scores': array([0.22587009, 0.22511227, 0.21677688, 0.2106454 , 0.20411198,
       0.20127359, 0.20020929], dtype=float32)}
</code></pre>

PositivePony: <p>Nice collection keep up the good work<a href="https://mcdvoice.me/" rel="noopener nofollow ugc">.</a></p>

hoangphucITJP: <p>So there is still no solution to this problem?</p>

Toby: <p>The problem happened a long time ago so I don’t remember in detail. However, I ticked the solution for ptrblck so I think you could try to solve your problem with that approach.</p>