Plot training and validation accuracy and loss of Mask RCNN for instance segmentation

imran: <p>I am using Mask RCNN tutorial at</p>
<p><a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a></p>
<p>I want to plot the training and validation accuracy and loss. How can I plot it.</p>
<p>Second problem is that after fine tuning I get a lot of masks. The only valid masks are the top The number of persons in image. All other masks are errors. How can I identify the total valid masks.</p>

ptrblck: <p>You could store the loss and accuracy in a list as scalar values via e.g.:</p>
<pre data-code-wrap="python"><code class="lang-python">losses.append(loss.detach().cpu().item())
</code></pre>
<p>and use <code>matplotlib.pyplot.plot</code> to visualize the loss and accuracy curves.</p>
<aside class="quote no-group" data-username="imran" data-post="1" data-topic="84021">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/imran/48/29244_2.png" class="avatar"> imran:</div>
<blockquote>
<p>Second problem is that after fine tuning I get a lot of masks. The only valid masks are the top The number of persons in image. All other masks are errors. How can I identify the total valid masks.</p>
</blockquote>
</aside>
<p>You could probably just index the first masks given the number of people. I’m not sure I understand the problem correctly, so could you explain the issue a bit more?</p>

ham952: <p>Adding to what <a class="mention" href="/u/ptrblck">@ptrblck</a> said … u can set a threshold value on scores predicted by the model and subsequently filter the corresponding masks … e-g</p>
<pre><code>    outputs = model(image)
    outputs = [{k: v.to(cpu_device) for k, v in t.items()} for t in outputs]
    pred_score = list(outputs[0]['scores'].numpy())
    pred_t = [pred_score.index(x) for x in pred_score if x&gt;=threshold][-1]
    masks = (outputs[0]['masks']&gt;0.5).squeeze().numpy()
    masks = masks[:pred_t+1]</code></pre>

ham952: <p>i am a little confused …</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="84021">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>and use <code>matplotlib.pyplot.plot</code> to visualize the loss and accuracy curves.</p>
</blockquote>
</aside>
<p>its okay with the loss, but where is <strong>“accuracy”</strong>  during <strong>training</strong> in pytorch Mask R-CNN model ??</p>
<pre><code>  # For Training
  images,targets = next(iter(data_loader))
  images = list(image for image in images)
  targets = [{k: v for k, v in t.items()} for t in targets]
  output = model(images,targets)

 #Output 
 {'loss_box_reg': tensor(0.0100, grad_fn=&lt;DivBackward0&gt;),
 'loss_classifier': tensor(0.2265, grad_fn=&lt;NllLossBackward&gt;),
 'loss_objectness': tensor(0.0190, grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;),
'loss_rpn_box_reg': tensor(0.0068, grad_fn=&lt;DivBackward0&gt;)}
</code></pre>
<p>its just a dictionary of losses … they can be plotted but i am more concerned in <strong>accuracy</strong> plotting during <strong>training</strong> … any thoughts ?</p>

ptrblck: <p>Depending which accuracy you want you could calculate it using the model output.<br>
How are the targets defined in your use case and which accuracy are you looking for?</p>

ham952: <p>Dont we need to have <strong>predictions</strong> from the model output in order to calculate an accuracy ?? what i was trying to say  earlier (and couldnt make it clear) was that for pytorch’s Mask RCNN implementation … we need to have model in <strong>eval</strong> model in order to generate predictions whcih can be then subsequently used for accuracy calculations … the same cannot be done in model <strong>train</strong> mode …  as said in its documentation <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/detection/mask_rcnn.py" rel="nofollow noopener">here</a></p>
<pre><code>   The model returns a Dict[Tensor] during training, containing the classification and regression
   losses for both the RPN and the R-CNN, and the mask loss.
</code></pre>
<p>and this is realized via  <em><strong>eager_outputs</strong></em>  methods in generalizedRCNN class as seen <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/detection/generalized_rcnn.py" rel="nofollow noopener">here</a>.<br>
To my understanding , in order to get predictions (and subsequent acc) in train mode , the only options seems to be is modification of this eager_outputs method. i have tried it but in vain (as of now).</p>
<p>To answer your  question , i am looking for mIoU … which i am already calculting via predictions in <strong>eval</strong> mode. but since now i want to analyse model overfitting , i need to have a train_mIoU generated from the training dataset. ofcourse i can just change the dataset to training dataset in eval mode to generate train_mIoU but that adds another overhead in computations … i was looking if there is any option to get the same during model training and <strong>NOT</strong> during evaluations …</p>
<p><strong>P.S</strong> thinking out of the box , any other option to analyse model overfitting ? i feel reluctant to go to train/test loss monitoring for this purpose … but open to suggestions .</p>

filmonmehari: <p>Hi guys can you  please help me to plot the training an d validationn and loss accuracy using example code along with its output. Sorry guys i didnt get the concept of all ur suggestions as i am a beginner to use mask rcnn model for my project.</p>
<p>thanks in advance!!!</p>