Negative value in my Loss function

edshkim98: <p>Hello everyone,<br>
I am currently doing a deep learning research project and have a question regarding use of loss function. Basically, for my loss function I am using Weighted cross entropy + Soft dice loss functions but recently I came across with a mean IOU loss which works, but the problem is that it purposely return negative loss. First, it seemed odd to me that it returns -loss, so i changed the function to return 1-loss, but it performed worse so I believe the negative loss is correct approach. This means though that my final loss will be sum of positive, positive and negative values which seem to me very odd and don’t really make sense but surprisingly working not badly.<br>
Hence, during training, my loss values go below 0 as the training continues. My current guess is that it is working fine because the optimization of loss function is to reduce the gradient of loss to zero not the loss itself.<br>
My question is, is it okay to use combination of positive and negative loss functions as what matters is just a gradient of my final loss function?<br>
I used the following approach for IOU loss: <a href="https://discuss.pytorch.org/t/how-to-implement-soft-iou-loss/15152" class="inline-onebox">How to implement soft-IoU loss?</a></p>
<p>Thank you and I look forward to hearing from someone to answer this question soon!</p>

KFrank: <p>Hi Edward!</p>
<aside class="quote no-group" data-username="edshkim98" data-post="1" data-topic="101776" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/edshkim98/48/29279_2.png" class="avatar"> edshkim98:</div>
<blockquote>
<p>My current guess is that it is working fine because the optimization of loss function is to reduce the gradient of loss to zero not the loss itself.<br>
My question is, is it okay to use combination of positive and negative loss functions as what matters is just a gradient of my final loss function?</p>
</blockquote>
</aside>
<p>Yes, it is perfectly fine to use a loss that can become negative.<br>
Your reasoning about this is correct.</p>
<p>To add a few words of explanation:</p>
<p>A smaller loss – algebraically less positive or algebraically more<br>
negative – means (or should mean) better predictions.  The<br>
optimization step uses some version of gradient descent to make<br>
your loss smaller.  The overall level of the loss doesn’t matter as<br>
far as the optimization goes.  The gradient tells the optimizer how<br>
to change the model parameters to reduce the loss, and it doesn’t<br>
care about the overall level of the loss.</p>
<p>When gradient descent drives the loss to a minimum, the gradient<br>
becomes zero (although it can be zero at places other than a<br>
minimum).  (Also, when the gradient is zero, plain-vanilla gradient<br>
descent stops changing the parameters.)</p>
<p>It is true that several common loss functions are non-negative, and<br>
become zero precisely when the predictions are “perfect.”  Examples<br>
include <code>MSELoss</code> and <code>CrossEntropyLoss</code>.  But this is by no means<br>
a requirement.</p>
<p>Consider, for example, optimizing with <code>lossA = MSELoss</code>.  Now<br>
imagine optimizing with <code>lossB = lossA - 17.2</code>.  The <code>17.2</code> doesn’t<br>
really change anything at all.  It is true that “perfect” predictions<br>
will yield <code>lossB = -17.2</code> rather than zero.  (<code>lossA</code> will, of course,<br>
be zero for “perfect” predictions.)  But who cares?</p>
<p>Best.</p>
<p>K. Frank</p>