Runaway Loss on Mask R-CNN finetuning on custom dataset

Udinanon: <p>I’m working on a fine tuning of the Mask R-CNN model, trying to use it on the EgoHands dataset to get hands instance segmentation.<br>
To achieve this i used <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 1.12.0+cu102 documentation</a> this tutorial as a reference point.<br>
I finally created my dataset loader, and i tried running the model on the dataset. It seems to work correctly, but the loss values are completely off, starting randomly from 600 to -900 and usually spiraling into negative infinity fast.<br>
I can’t understand why this is happening, the dataset i’m using and the PennFudan used in the example seem similar and the dataset class i built should be correct<br>
I’ve tried changing SGD to Adamw fro the optimizer and fiddling with the hyperparameters, but the result is the same and this to me hints at some fundamental error<br>
my code is here <a href="https://colab.research.google.com/drive/1vbGTYq_IAIoc9XbvEWRpGXYVbZ1vDu-D#scrollTo=tAnZZjhSnqXq&amp;uniqifier=4" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>

ptrblck: <aside class="quote no-group" data-username="Udinanon" data-post="1" data-topic="156463">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/udinanon/48/51402_2.png" class="avatar"> Udinanon:</div>
<blockquote>
<p>but the loss values are completely off, starting randomly from 600 to -900 and usually spiraling into negative infinity fast.</p>
</blockquote>
</aside>
<p>This sounds concerning as the “standard” loss functions are not creating a negative loss.<br>
Check what your outputs and targets represent and why the loss is negative first before trying to change the hyperparameters.</p>

Udinanon: <p>by running the model in eval mode and comparing the results with my targets everything seems to be in order, a dictionary with areas, bounding boxes, masks and labels all as tensors.</p>
<p>Also, by more thoroughly reading my results, it seems that the driving error is the loss_mask value, while all other losses seem to be in normal ranges.<br>
The odd thing is that this value does not appear in any way in the tutorial i am following</p>
<p>I think something here might be the cause, maybe the masks are the wrong shape/size and this causes issue to the loss computation, but i can’t really find much regarding this specific loss</p>

Udinanon: <p>No that’s not accurate, the error appears when the MaskRCNN model is used, it obviously does not appear in the parts where FasterRCNN is used, i didn’t notice the difference in model<br>
The masks generated by the model are apparently of the same size<br>
the error was mine, in building the masks for the targets.<br>
I misunderstood how the masks should have been built, so i built masks with 0 for background and 255 for the only class, instead it should have been 1<br>
Now that this error is fixed, the training seems to be going perfectly fine.</p>

Ahmad_Obeid: <p>Thank you for brining this to my attention. I was facing the same problem and it was solved by setting the mask to 1 instead of 255</p>

PremRaj_Kala: <p>Saved my day too! Thanks!</p>