Compute validation loss for Faster RCNN

loicdtx: <p>Hi, I’m doing object detection on a custom dataset using transfer learning from a pretrained Faster RCNN model.<br>
I would like to compute validation loss at the end of each epoch. How can this be done?</p>
<p>If I run the code below (model in training mode) I get losses, but dropout isn’t deactivated, so I am wondering how ‘valid’ are these loss values. And running the model in eval mode only returns the predictions.</p>
<pre><code class="lang-auto">model.train()
for images, targets in data_loader_val:
    images = [image.to(device) for image in images]
    targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

    with torch.no_grad():
        val_loss_dict = model(images, targets)
        print(val_loss_dict)
</code></pre>

MLaurenceFournier: <p>I’m wondering the same thing. Did you find a solution? I was thinking of forcing the training mode on only some submodules (the ones that output losses).</p>

loicdtx: <p>I thought it through and came to the conclusion that validation loss is only meaningful when considered relatively to training loss. Training loss is computed with dropouts too, so they are comparable.</p>

MLaurenceFournier: <p>I guess for dropout I might be ok, but in general wouldn’t that screw modules like batch norm that keep running estimates?</p>

mapostig: <p>Hello,</p>
<p>Did you reach any conclusion? I am working also at object detection in my custom dataset and I would like to check validation and training losses evolution, but I’m not sure if it is a good practice to use the .train() mode during evaluation.</p>

Usama_Hasan: <p><a class="mention" href="/u/mapostig">@mapostig</a> No, I guess it’s not a good practice to use model.train() mode in evaluation. You can use the same custom dataset class to create a different dataset loader for your evaluation dataset.</p>
<pre><code class="lang-auto">for phase in ['train','val']:
    if phase == 'train':
        model.train()
        #Training Part with backprog
    else:
        model.eval()
        #Just a forward Pass.
</code></pre>
<p>Some layers like Dropout and batch Norm will behaves differently under model.eval().<br>
Further you can look to this discussion.</p><aside class="quote quote-modified" data-post="4" data-topic="5744">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/67e7ee/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/model-train-and-model-eval-vs-model-and-model-eval/5744/4">Model.train() and model.eval() vs model and model.eval()</a> 
  </div>
  <blockquote>
    Why do model.train() and model.eval() return a reference to the model.  What is the intended usage for the return value? 
I am using as follows: 
model.train() 
But this means that in a Jupyter notebook it outputs the model object repr which is unwanted: 
LeNet(
  (m): Sequential(
    (0): Sequential(
      (0): Conv2d(1, 20, kernel_size=(5, 5), stride=(1, 1))
      (1): BatchNorm2d(20, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU(inplace)
      (3): MaxPool2d(…
  </blockquote>
</aside>


Arun_Mohan: <p>Even Iam stuck at the same place. Is it possible to calculate validation loss properly?</p>

Arun_Mohan: <p><a class="mention" href="/u/loicdtx">@loicdtx</a> did u got any solution for this problem</p>

loicdtx: <p><a class="mention" href="/u/arun_mohan">@Arun_Mohan</a>, validation loss is just there to control for overfit during training; it has no analytical value. It’s therefore completely fine to compute it like I did in the original post (model in train mode and gradient deactivated).</p>

Arun_Mohan: <p><a class="mention" href="/u/loicdtx">@loicdtx</a> thanks…Even I tried the same way. I think it is not an issue.</p>

mkisantal: <p>While I agree with those above arguing that train mode validation loss calculation is fine, there still a serious efficiency problem here.</p>
<p>If you also want the model <strong>outputs</strong> (for tracking IOU, accuracy, etc., which is often the case), then you <em>need to run inference twice</em>.  Training mode for the loss, and eval mode for the outputs. Would be better to have both with a single pass!</p>

mkisantal: <p>I posted a hacky solution on StackOverflow to get both outputs and losses in a single pass: <a href="https://stackoverflow.com/questions/60339336/validation-loss-for-pytorch-faster-rcnn/65347721#65347721" rel="noopener nofollow ugc">https://stackoverflow.com/questions/60339336/validation-loss-for-pytorch-faster-rcnn/65347721#65347721</a></p>

coolcucumber94: <p>Hi <a class="mention" href="/u/usama_hasan">@Usama_Hasan</a>,</p>
<p>Thanks for your answer. I am using the pretrained Faster RCNN model, and I see that the BatchNormalization Layers are Frozen:</p>
<pre><code class="lang-auto">FasterRCNN(
  (transform): GeneralizedRCNNTransform()
  (backbone): BackboneWithFPN(
    (body): IntermediateLayerGetter(
      (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
      (bn1): FrozenBatchNorm2d()
      (relu): ReLU(inplace=True)
      (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
      (layer1): Sequential(
        (0): Bottleneck(
          (conv1): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (bn1): FrozenBatchNorm2d()
          (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn2): FrozenBatchNorm2d()
          (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (bn3): FrozenBatchNorm2d()
          (relu): ReLU(inplace=True)
          (downsample): Sequential(
            (0): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (1): FrozenBatchNorm2d()
          )
        )
</code></pre>
<p>Then it should not affect the running stats right?</p>

Usama_Hasan: <p>Hey <a class="mention" href="/u/coolcucumber94">@coolcucumber94</a> , You’re right it won’t affect the running stats.<br>
Plus you can add your code inside code section or just write them between triple comas, it helps in debugging and understanding code.</p>

coolcucumber94: <p>Thanks for the reply.</p>

SU801T: <p>Was this found to be appropriate? I’ve been told that batch norm layers and dropout needs to be in eval() mode, however, I’m only interested in calculating validation loss to save the “best” model</p>