Get losses during validation torchvision FasterRCNN model

username: Chame_call
cooked: <p>Hello.<br>
I trained FasterRCNN model on my problem to a certain extent.<br>
Now I’d like to set model to evaluation mode (<code>model.eval()</code>) to disable some specific layers (like dropout) but I still want to get loss on my data.<br>
If I set model.eval() then I get detection but not losses while if I set model.train() my loss is different every run on the same sample and I dont’t get detections.<br>
Can I do it without modifying the module source code?</p>
<p>I see such training conditions in FasterRCNN submodules:</p>
<pre><code class="lang-auto">losses = {}
if self.training:
    assert targets is not None
    labels, matched_gt_boxes = self.assign_targets_to_anchors(anchors, targets)
    regression_targets = self.box_coder.encode(matched_gt_boxes, anchors)
    loss_objectness, loss_rpn_box_reg = self.compute_loss(
        objectness, pred_bbox_deltas, labels, regression_targets)
    losses = {
        "loss_objectness": loss_objectness,
        "loss_rpn_box_reg": loss_rpn_box_reg,
    }
return boxes, losses
</code></pre>
<p>[rpn.py - <code>forward()</code>]</p>

username: Chame_call
cooked: <p>Does somebody have any adivices on the subject</p>

username: ptrblck
cooked: <p>I think you could call <code>.train()</code> on all submodules via e.g. <code>model.apply()</code> or alternatively try to use:</p>
<pre><code class="lang-python">model.eval()
model.training = True
</code></pre>
<p>This would also set all submodules to the eval mode and would reset the parent model to the training mode.</p>

username: Chame_call
cooked: <p>Now I get neither losses nor predictions while I need both of them.</p>

username: ptrblck
cooked: <p>Are you getting a return value of <code>None</code>s now?<br>
I’m unsure about your initial use case, as <code>torchvision.models.detection.fasterrcnn_resnet50_fpn</code> doesn’t seem to contain any dropout layers and uses <code>FrozenBatchNorm</code> layers. Which layers would you like to set to <code>eval</code> mode specifically?</p>
<p>At least this code snippet doesn’t find any dropout layers:</p>
<pre><code class="lang-python">model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.train()
for name, m in model.named_modules():
    if isinstance(m, nn.Dropout):
        print('setting {} to eval()'.format(name))
        m.eval()
</code></pre>

username: Chame_call
cooked: <p>Sorry.<br>
I didn’t know about which specific layer effects such behaviour and only knew that it may be Dropout or BatchNormalization or like this cause with the same sample my losses fluctuated a bit every run.</p>
<p>But the main problem that I cannot get predictions and losses simultaneously.<br>
To get predictions I need to set <code>model.eval()</code>, to get losses I need to set <code>model.train()</code> and at the time I run the methods one by one with the same sample to get losses and predictions.</p>
<p>So could I get desired ouput without dual run?</p>

username: Khalid_Cawl
cooked: <p>Hi. Any update on this? I would like to get the losses during validation as well. Otherwise, I won’t be able to see if the model overfits or not. Any help would be appreciated.</p>