How to get losses and predictions at the same time?

username: Rafa
cooked: <p>Hi all,</p>
<p>I’m currently working on a project based on the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">finetuning tutorial</a>. I have a model object the same as they have it in the tutorial.</p>
<p>Now, for validation purposes I need two things: first, I want the validation losses and second, I want the predictions in order to get the Mean Average Precision.</p>
<p>According to the <a href="https://pytorch.org/vision/main/_modules/torchvision/models/detection/mask_rcnn.html" rel="noopener nofollow ugc">docs</a>:<br>
"During training, the model expects both the input tensors, as well as a targets (list of dictionary), containing boxes, labels, masks. The model returns a Dict[Tensor] during training, containing the classification and regression losses for both the RPN and the R-CNN, and the mask loss.</p>
<p>During inference, the model requires only the input tensors, and returns the post-processed<br>
predictions as a List[Dict[Tensor]], one for each input image. The fields of the Dict are as<br>
follows: boxes, labels, scores, masks."</p>
<p>So, to get the losses we do:</p>
<pre><code class="lang-auto">losses = model(images, targets)
</code></pre>
<p>And to get the predictions we do:</p>
<pre><code class="lang-auto">predictions = model(images)
</code></pre>
<p>I need both things. Losses to measure the validation error and the predictions to measure MAP. I’d want something like:</p>
<pre><code class="lang-auto">losses, predictions = model(images, targets)
</code></pre>
<p>Does anyone know if it is possible with this implementation? I could get both things by just doing those two lines of code, but that means iterating through the dataset twice, something that can take a long time.</p>

username: ptrblck
cooked: <p>Based on this code snippet from the tutorial:</p>
<pre><code class="lang-python"># For Training
images,targets = next(iter(data_loader))
images = list(image for image in images)
targets = [{k: v for k, v in t.items()} for t in targets]
output = model(images,targets)   # Returns losses and detections
# For inference
model.eval()
x = [torch.rand(3, 300, 400), torch.rand(3, 500, 400)]
predictions = model(x)           # Returns predictions
</code></pre>
<p>it seems the model returns losses <em>and</em> detections during training (i.e. if the model is in <code>.train()</code> mode), so wouldn’t this work for your use case?</p>

username: Rafa
cooked: <p>Unfortunately, I don’t think that works. Even though that comment says it returns losses and detections, it only returns losses when you feed it both images and targets.</p>
<p>I did a couple of prints right after the code snippet you mentioned:´</p>
<pre><code class="lang-auto">print("Output:", output)
print("Predictions:", predictions)
</code></pre>
<p>and it yields:</p>
<pre><code class="lang-auto">Output: {'loss_classifier': tensor(0.1091, grad_fn=&lt;NllLossBackward0&gt;), 'loss_box_reg': tensor(0.0480, grad_fn=&lt;DivBackward0&gt;), 'loss_objectness': tensor(0.0275, grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward0&gt;), 'loss_rpn_box_reg': tensor(0.0034, grad_fn=&lt;DivBackward0&gt;)}

Predictions: [{'boxes': tensor([], size=(0, 4), grad_fn=&lt;StackBackward0&gt;), 'labels': tensor([], dtype=torch.int64), 'scores': tensor([], grad_fn=&lt;IndexBackward0&gt;)}, {'boxes': tensor([], size=(0, 4), grad_fn=&lt;StackBackward0&gt;), 'labels': tensor([], dtype=torch.int64), 'scores': tensor([], grad_fn=&lt;IndexBackward0&gt;)}]
</code></pre>
<p>So, feeding a model with only images gives a prediction. Feeding images and targets gives the loss.</p>