What happens when loss are negative?

username: ljj7975
cooked: <p>Based on my understanding of back prop and gradient descent,<br>
Loss is multiplied to gradient when taking a step with gradient descent.<br>
So when gradient becomes negative, gradient descent takes a step in the opposite direction.</p>
<p>Such idea is well captured when implementing gradient ascent,<br>
as it can simply be implemented by multiplying -1 to the loss.</p>
<p>Then, what happens if my loss starts from positive and goes below zero?<br>
If what I said above is correct, training will go for minimization when the loss is positive and go for maximization when the loss is negative.<br>
And this means that gradient descent is going for zero instead of minimum value…</p>
<p>Can anyone help me understand this concept?</p>

username: Balamurali_M
cooked: <p>The loss functions are chosen in such a way that you minimize to 0. mse, l1 can’t be zero. The deviations from error 0 is what we are trying to minimise. Have a look at the loss functions available in pytorch. In all the cases, when you give equal value the loss reduces to zero. It is either maximizing a negative term or minimizing a positive term but it is always going to zero.</p>

username: KFrank
cooked: <p>Hello Brandon!</p>
<aside class="quote no-group" data-username="ljj7975" data-post="1" data-topic="47883" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ljj7975/48/12561_2.png" class="avatar"> ljj7975:</div>
<blockquote>
<p>Based on my understanding of back prop and gradient descent,<br>
Loss is multiplied to gradient when taking a step with gradient descent.<br>
So when gradient becomes negative, gradient descent takes a step in the opposite direction.</p>
</blockquote>
</aside>
<p>This isn’t true.  All common optimization algorithms I’m aware<br>
of – and in particular, gradient descent – only care about the<br>
gradient of the loss, and not the loss itself.</p>
<p>Plain-vanilla gradient descent takes the following optimization<br>
step:</p>
<p>new weights  =  old weights  -  learning rate * gradient</p>
<p>Could you have misread “learning rate” for “loss” at some point?</p>
<aside class="quote no-group">
<blockquote>
<p>Such idea is well captured when implementing gradient ascent,<br>
as it can simply be implemented by multiplying -1 to the loss.</p>
<p>Then, what happens if my loss starts from positive and goes below zero?<br>
If what I said above is correct, training will go for minimization when the loss is positive and go for maximization when the loss is negative.<br>
And this means that gradient descent is going for zero instead of minimum value…</p>
<p>Can anyone help me understand this concept?</p>
</blockquote>
</aside>
<p>Gradient descent (and, again, all common algorithms that I am<br>
aware of) seek to minimize the loss, and don’t care whether that<br>
minimum value is a large positive value, a value close to zero,<br>
exactly zero, or a large negative value.  It simply seeks to drive<br>
the loss to a smaller (that is, algebraically more negative) value.</p>
<p>You could replace your loss with</p>
<p>modified loss = conventional loss - 2 * Pi</p>
<p>and you should get the exact same training results and model<br>
performance (except that all values of your loss will be shifted<br>
down by 2 * Pi).</p>
<p>It is the case that we often use loss functions that become equal<br>
to zero when the fit of the model to the training data is perfect,<br>
but the optimization algorithms don’t care about this, and they<br>
drive the loss function to algebraically more negative values,<br>
and not towards zero.</p>
<p>Good luck!</p>
<p>K. Frank</p>

username: ljj7975
cooked: <p>I see, so the implicit assumption of minimum loss being zero<br>
enforces the minimization with positive loss and maximization with negative loss.</p>
<p>I am currently experimenting something and my custom loss function can be less than zero.</p>
<p>Based on what you told me, the framework will maximize the loss to zero, when loss is negative and minimize the loss to zero, when loss is positive.</p>
<p>I.e. Loss should be going up and down around 0 instead of reaching a single convergence point</p>
<p>Is my understanding correct?</p>

username: ljj7975
cooked: <p>Thank you Frank.</p>
<p>Oh I thought, loss affects the gradient descent directly.<br>
So it simply provides the search space and training a model purely depends on learning rate, not the loss</p>
<p>flipping the sign of loss turns the problem into maximization because within the search space everything is flipped (min point becoming max point).</p>
<p>Is this correct?</p>

username: KFrank
cooked: <p>Hi Brandon!</p>
<aside class="quote no-group" data-username="ljj7975" data-post="5" data-topic="47883" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ljj7975/48/12561_2.png" class="avatar"> ljj7975:</div>
<blockquote>
<p>Thank you Frank.</p>
<p>Oh I thought, loss affects the gradient descent directly.<br>
So it simply provides the search space and training a model purely depends on learning rate, not the loss</p>
<p>flipping the sign of loss turns the problem into maximization because within the search space everything is flipped (min point becoming max point).</p>
<p>Is this correct?</p>
</blockquote>
</aside>
<p>If I understand what you are saying, yes that is  correct.</p>
<p>But, just to be sure, you could try the following exercise:</p>
<p>Set up a simple neural network and train it using gradient<br>
descent on some well-behaved data set.  Make a plot of<br>
your training loss (and, while you’re at it, your training<br>
accuracy) as a function of batch number or epoch number,<br>
and make sure that it is training stably and that you’re getting<br>
reasonable results.</p>
<p>Use a standard loss function when you do this.  Let’s call this<br>
loss-original.</p>
<p>Let’s say that your loss runs from 1.0 down to 0.1 when you<br>
train.</p>
<p>Now define both:</p>
<pre><code class="lang-plaintext">   loss-shifted = loss-original - 1.5
   loss-negative = -loss-original
</code></pre>
<p>and train your neural network again using these two modified<br>
loss functions and make your loss and accuracy plot for each<br>
of these two modified training runs.  See if you get the results<br>
you expect, and, if not, post what you got and ask any questions<br>
that you might have.</p>
<p>Best.</p>
<p>K. Frank</p>

username: ljj7975
cooked: <p>Thanks Frank, I did the exercise. It behaves as I expected.<br>
Training is fine and generates exactly same accuracy for  <code>loss-shifted</code>  (even though loss is &lt; 0).<br>
For  <code>loss-negative</code> , training fails, the graph say that loss decreases but since the sign is flipped, conceptually it is increasing the loss by applying gradient ascent.</p>
<p>I actually have another question about loss.<br>
From our previous discussion, it is clear that value of loss itself does not mean anything.<br>
what actually matter is the gradient with respect to input and their direction (sign).</p>
<p>I am in the situation where I have to define a new loss function that is not differentiable.<br>
Based on my understanding, (though it is not going to be easy) it is possible to train a model with non-differentiable loss function, if I define a custom Backward function that returns “fake” gradient with appropriate direction and magnitude.</p>
<p>Is my assumption correct?</p>

username: KFrank
cooked: <p>Hello Brandon!</p>
<aside class="quote no-group" data-username="ljj7975" data-post="8" data-topic="47883" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ljj7975/48/12561_2.png" class="avatar"> ljj7975:</div>
<blockquote>
<p>I am in the situation where I have to define a new loss function that is not differentiable.<br>
Based on my understanding, (though it is not going to be easy) it is possible to train a model with non-differentiable loss function, if I define a custom Backward function that returns “fake” gradient with appropriate direction and magnitude.</p>
</blockquote>
</aside>
<p>I would not recommend doing things this way (or thinking about it<br>
this way).  The best approach – because network training relies<br>
so heavily on gradient-based optimization – is to make your loss<br>
function differentiable.</p>
<p>Consider the softmax function (which should more properly be<br>
called the “soft-argmax” function):  It can be understood as a<br>
differentiable approximation to the argmax function (which is<br>
not differentiable because it jumps around discretely).</p>
<p>(In a similar vein, the sigmoid function can be considered a<br>
differentiable version of the step function.)</p>
<p>So you should look at your loss function, and try to find a sensible<br>
differentiable replacement for it that has more or less the same<br>
structure.</p>
<p>You could view what you call the “fake gradient” as, in effect,<br>
defining your differentiable loss function, but I wouldn’t think<br>
about it this way.  Both in your code and in your mental picture<br>
you should have an explicitly differentiable loss function, and<br>
calculate its real gradient.</p>
<p>(I wouldn’t change or repost this post, but when you switch topics<br>
like this it would be better for the forum if you would start a new<br>
thread in the future.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: ljj7975
cooked: <p>I see. Thank you</p>
<p>Okay, I will see if I can create differentiable variation of the loss function I am looking for.</p>
<p>Let me rephrase your passage about softmax, just to confirm that I understood correctly.<br>
Softmax is essentially continuous version of argmax as it also takes in the same input but returns very similar outputs (1 if the greatest else 0).<br>
And while argmax is not differentiable softmax is, so it can nicely replaces argmax while enabling backprop through the framework.<br>
I think this is pretty brilliant and I feel like I kinda know what I need to do. Thanks</p>