Rpn_box_reg loss is nan

ahmed: <p>Hi, I am trying to run a faster r-cnn model based on the torchvision <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">example</a> for a custom dataset.</p>
<p>However, I have noticed that when training,  if xmax is smaller than xmin, the rpn_box_reg loss goes to nan. xmax and ymax represent the top left corner and xmin and ymin represent the bottom right corner. This is a snippet of the error that i get with the bounding boxes printed:</p>
<pre><code class="lang-auto">tensor([[ 44., 108.,  49., 224.],
        [ 29.,  73., 210., 230.],
        [ 31.,  58., 139., 228.],
        [ 22.,  43., 339., 222.]], device='cuda:0')
Epoch: [0]  [   0/1173]  eta: 0:09:46  lr: 0.000000  loss: 9.3683 (9.3683)  loss_classifier: 1.7522 (1.7522)  loss_box_reg: 0.0755 (0.0755)  loss_objectness: 6.1522 (6.1522)  loss_rpn_box_reg: 1.3884 (1.3884)  time: 0.4997  data: 0.1162  max mem: 5696
tensor([[  0.,   0., 640., 512.]], device='cuda:0')
tensor([[ 28.,  57., 197., 220.]], device='cuda:0')
tensor([[ 23.,  46., 281., 222.]], device='cuda:0')
tensor([[ 20.,  28., 328., 210.]], device='cuda:0')
tensor([[ 37.,  45.,  47., 161.],
        [ 31.,  39., 111., 154.]], device='cuda:0')
tensor([[  0.,   0., 640., 512.]], device='cuda:0')
tensor([[ 33.,  85., 546., 222.],
        [ 31.,  85., 527., 213.]], device='cuda:0')
tensor([[ 40.,  76.,  29., 211.],
        [ 64.,  51.,  26., 206.],
        [ 40.,  77.,   1., 221.]], device='cuda:0')
Loss is nan, stopping training
{'loss_classifier': tensor(1.78, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;), 'loss_box_reg': tensor(0., device='cuda:0', grad_fn=&lt;DivBackward0&gt;), 'loss_objectness': tensor(16.28, device='cuda:0', grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward&gt;), 'loss_rpn_box_reg': tensor(nan, device='cuda:0', grad_fn=&lt;DivBackward0&gt;)}
An exception has occurred, use %tb to see the full traceback.
</code></pre>
<p>As you can see, for each box is set as <code>[xmin, ymin, xmax, ymax]</code>.</p>
<p>Thank you in advance.</p>

Diego: <p>Hello, sometimes if your learning rate is too high the proposals will go outside the image and the <code>rpn_box_regression</code> loss will be too high, resulting in <code>nan</code> eventually. Try printing the <code>rpn_box_regression</code> loss and see if this is the case, if so, try lowering the learning rate. Remember to scale your learning rate linearly according to your batch size. Hope this helps <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ahmed: <p>Thank you for your quick reply. I have tried reducing the learning all the way down to <code>0.00001</code>, but i continue to get the same issue. This is the settings:</p>
<pre><code class="lang-auto">params = [p for p in model_ft.parameters() if p.requires_grad]

optimizer = torch.optim.SGD(params, lr=0.00001,
                            momentum=0.9, weight_decay=0.0005)

lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                               step_size=1,
                                               gamma=0.1)
</code></pre>
<p>I have noticed that this only seems to emerge when xmax value is lower than xmin value. Would you have any idea why this may be the case?</p>
<p>I will have a look at the <code>rpn_box_regression</code> as you suggested to see if the values are high.</p>

Diego: <p>You can use <a href="https://pytorch.org/docs/stable/autograd.html#anomaly-detectionl" rel="nofollow noopener">the anomaly detection tool</a> to check where the nans are produced. In my experience nans appear when you have high values in your gradients or if you are doing a mathematically undefined operation (e.g log(0)). In this case the latter is not likely, so it must be the former. Try and double check that your <code>dataloader</code> is working correctly and that it is providing you with the correct annotations.</p>

tree33: <p>Hi, a simple alternative would be to preprocess your groundtruth before training,<br>
you can discard the invalid samples in your groundtruth.<br>
several things to notice:<br>
1 xmax, ymax &gt; xmin ymin<br>
2 xmax ymax and xmin ymin are always inside your image.</p>

ahmed: <p>Thank you for you advice. I have checked the datalaoder, and it seems to be fine. I even tested the annotations that were giving me issues by plotting the images and placing the bounding boxes on them and it looks to be working as it should. I haven’t managed to get the anomaly detection tool working yet. I will continue to work on it tomorrow. Would you have an example of how it would be used?</p>

ahmed: <p>Hi, all the samples that I am using have are valid. The issue that I am facing is when the x1 (xmax) value is smaller than x2 (xmin), the loss rpn_box_reg loss becomes NaN. For example, for the image below, the bouding boxes are <code>tensor([[ 53.,  89.,   7., 226.]])</code> i.e. <code>[x2, y2, x1, y1]</code>. When the x1 value is smaller than x2, the loss goes to zero, however, it works fine when <code>x1&gt;x2</code>. In fact, it trains quite well. As you can see, the values are correct as the <code>cyclist</code> has the correct bounding box based on the values above. I hope this makes it a bit more clear as to the issue I am facing.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/752dc8d0e27c59004e63b84b17e0e873fba33191.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/752dc8d0e27c59004e63b84b17e0e873fba33191" title="test"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/752dc8d0e27c59004e63b84b17e0e873fba33191_2_500x500.jpeg" alt="test" data-base62-sha1="gIBXjJCQ6ijgnV6IVf9EIy9yU5X" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/752dc8d0e27c59004e63b84b17e0e873fba33191_2_500x500.jpeg, https://discuss.pytorch.org/uploads/default/original/2X/7/752dc8d0e27c59004e63b84b17e0e873fba33191.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/7/752dc8d0e27c59004e63b84b17e0e873fba33191.jpeg 2x" data-dominant-color="B6AAA3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">test</span><span class="informations">720×720 103 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

tree33: <p>Well I think I get your point, and why don’t you remove this annotation from the beginning?</p>

ahmed: <p>Thank you for your reply. I would preferably like to keep these annotations as possible as they’re limited annotations for cyclists. Also, would it not mean that during evaluation, the model outputs would be always be coordinates where x1&gt;x2 and therefore it would never pick up the objects like the cyclist in the lower left-hand of the image?</p>
<p>I’m hoping this is making sense. I am new to machine learning and pytorch so I may not fully understand some of the concepts.</p>

kl_divergence: <p>Even I discussed this issue <a href="https://discuss.pytorch.org/t/loss-rpn-box-reg-is-nan-torchvision-0-3/49916/2">here</a>. I tried with two different datasets. In both cases, rpn_box_reg becomes nan in first epoch itself.</p>

ahmed: <p>I will let you know if I get this working.</p>

kl_divergence: <p>Is this torchvision specific or my code has some issue ? I’ve filed an issue.  I checked my data  thoroughly and everything seems to be working fine except the value of specific loss.</p>

ahmed: <p>i’m not sure but i think it may have something to do with how the rpn box loss calculated.  again, i’m not     sure and this is only a guess. Perhaps as x1 is smaller than x2, a negative loss is calculated and that is what is causing the NaN value. I am still looking into how the rpn box loss is being calculated based on the rpn.py file.</p>

kl_divergence: <p>Let us know if you find something.</p>

ahmed: <p>I have decreased the learning rate, which is quite small. Would you think that i decrease it further?</p>
<aside class="quote no-group" data-username="ahmed" data-post="3" data-topic="50480">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/8c91f0/48.png" class="avatar"> ahmed:</div>
<blockquote>
<p>optimizer = torch.optim.SGD(params, lr=0.00001, momentum=0.9, weight_decay=0.0005)</p>
</blockquote>
</aside>

kl_divergence: <p>No, decreasing LR won’t help. In my case, fmassa says that I’ve an issue with bbox notation. I’ll work on the fix</p>

ahmed: <p>This should be the solution to our issue: <a href="https://github.com/pytorch/vision/issues/1128" rel="nofollow noopener">https://github.com/pytorch/vision/issues/1128</a></p>
<p>However, we will have to skip those images without annotations. So I have followed the solution given here: <a href="https://discuss.pytorch.org/t/ignore-images-without-annotations/50768/5" class="inline-onebox">Ignore images without annotations</a></p>
<p>Good luck!</p>

pvtien96: <p>This is gold “to scale your learning rate linearly according to your batch size”. Thank you!</p>