Loss function for classifcation rewarding closer guess?

jerron: <p>The default loss function in multi class classification is cross_entropy, which treats all wrong guesses equally. If the distance between buckets are meaningful, for example, given the real bucket is 5, the guess 6 is considered 3 times better than 9, is there such function rewarding better guess (without losing the wights from probabilities as captured by cross_entropy )?</p>

jerron: <p>bump. any one can help pls?</p>

Mah_Neh: <aside class="quote no-group" data-username="jerron" data-post="1" data-topic="186851">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/48db29/48.png" class="avatar"> jerron:</div>
<blockquote>
<p>he default loss function in multi class classification is cross_entropy, which treats all wrong guesses equally. If the distance between buckets are meaningful, for</p>
</blockquote>
</aside>
<p>I haven’t used it but presume the weights argument in the cross entropy loss covers this case <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="inline-onebox" rel="noopener nofollow ugc">CrossEntropyLoss — PyTorch 2.1 documentation</a></p>
<p>I am not sure though.</p>

KFrank: <p>Hi Jerron!</p>
<aside class="quote no-group" data-username="jerron" data-post="1" data-topic="186851" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/48db29/48.png" class="avatar"> jerron:</div>
<blockquote>
<p>is there such function rewarding better guess (without losing the wights from probabilities as captured by cross_entropy )?</p>
</blockquote>
</aside>
<p>There are a couple of ways you could go about this.</p>
<p>First, you could use <code>CrossEntropyLoss</code>’s <em>probabilistic</em> (“soft”) targets.<br>
Let’s say you have five classes and the right answer is <code>2</code>.  Instead of<br>
using a “hard,” integer class label of <code>2</code>, you could use, for example, the<br>
set of probabilities <code>[0.0, 0.25, 0.5, 0.25, 0.0]</code>.  So predicting<br>
classes <code>1</code> or <code>3</code> will not be penalized as much as predicting classes <code>0</code><br>
or <code>4</code>.</p>
<p>Note, however, that predicting just class <code>2</code> (with high probability) will not<br>
be the best prediction.  Instead, you will be training your model to predict<br>
a mix of classes <code>1</code>, <code>2</code>, and <code>3</code> (with probabilities <code>[0.25, 0.5, 0.25]</code>), which<br>
might not be what you want.</p>
<p>Another approach would be to use a predicted-probability-weighted <code>MSELoss</code><br>
(or <code>L1Loss</code> or whatever).  Convert the (unnormalized) log-probabilities<br>
predicted by your model into probabilities by passing them through <code>softmax()</code>.<br>
Then, letting <code>t</code> be the correct class (for example <code>t = 2</code>), you could use as<br>
your loss function:</p>
<pre data-code-wrap="python"><code class="lang-python">p[0] * (t - 0)**2 + p[1] * (t - 1)**2 + p[2] * (t - 2)**2 + p[3] * (t - 3)**2 + p[4] * (t - 4)**2
</code></pre>
<p>For <code>t = 2</code>, your best prediction – with a loss of zero – will be to predict<br>
class <code>2</code> with probability one (<code>p[2] = 1.0</code>).  Predicting, say, class <code>3</code> will<br>
have a higher loss, and <code>4</code>, higher still.  So you do penalize different incorrect<br>
predictions differently.</p>
<p>However, when your prediction is completely wrong, <code>CrossEntropyLoss</code><br>
has a logarithmic divergence that I believe is very helpful for training.  This<br>
<code>MSELoss</code>-like loss doesn’t have such a divergence, which could be a<br>
disadvantage.</p>
<p>You could consider adding such a <code>MSELoss</code>-like loss to the conventional<br>
hard-label <code>CrossEntropyLoss</code>.  Now your loss will be at its minimum (of<br>
zero) when your prediction is completely correct, but will penalize worse<br>
incorrect predictions more than not-as-bad incorrect predictions.</p>
<p>But think carefully about your use case.  If the distance between your buckets<br>
is meaningful and your buckets are ordered sequentially, then perhaps your<br>
problem is better modelled as regression (rather than classification) and you<br>
should use something like <code>MSELoss</code> without any bells and whistles.</p>
<p>In any event, you should only use a non-standard loss, such as those<br>
described above, if you can show that they work better on your problem<br>
than a standard pure-classification with <code>CrossEntropyLoss</code> or a standard<br>
pure-regression with something like <code>MSELoss</code> or <code>L1Loss</code>.</p>
<p>Good luck!</p>
<p>K. Frank</p>

jerron: <p>Thank you Frank. Yes, I agree it can be also a regression rather than classification. If we use regression, can we still somehow get the probabilities?<br>
I’m a little surprised that there is no builtin function for it – is such use case that rare? I will try your advise on customized loss function, too. Though I assume the performance would be not as good as the builtin <code>CrossEntropyLoss</code></p>

KFrank: <p>Hi Jerron!</p>
<aside class="quote no-group" data-username="jerron" data-post="5" data-topic="186851" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/48db29/48.png" class="avatar"> jerron:</div>
<blockquote>
<p>is such use case that rare?</p>
</blockquote>
</aside>
<p>What, concretely, is this use case?  Specifically, if you train a model for your<br>
use case with loss-function A, and then train a second model with loss-function<br>
B, how would you decide which model is better?  What specific performance<br>
metrics would you use to choose between the two models.</p>
<p>The loss function you train with is, in some sense, a training-friendly proxy for<br>
the performance metrics that determine in a practical sense how well your model<br>
is working, so the performance metrics tell us what your use case is (and the<br>
loss function is a mere “implementation detail”).</p>
<p>Best.</p>
<p>K. Frank</p>

jerron: <p>I’m now using loss function like following:</p>
<pre><code class="lang-auto">def cross_entropy_mse_loss(input, target):
    return F.mse_loss(input.argmax(1),target)*a+F.cross_entropy (input,target)
</code></pre>
<p>And it seems working. I’m not sure if it mathematically legit. Also, what would be the right coefficient a? for now I set a=1.</p>

KFrank: <p>Hi Jerron!</p>
<aside class="quote no-group" data-username="jerron" data-post="7" data-topic="186851" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/48db29/48.png" class="avatar"> jerron:</div>
<blockquote>
<pre><code class="lang-auto">def cross_entropy_mse_loss(input, target):
    return F.mse_loss(input.argmax(1),target)*a+F.cross_entropy (input,target)
</code></pre>
<p>And it seems working. I’m not sure if it mathematically legit.</p>
</blockquote>
</aside>
<p>This won’t do what you want – the <code>mse_loss()</code> term won’t have any effect.<br>
This is because <code>input.argmax(1)</code> returns an integer and is therefore not<br>
(usefully) differentiable.  From memory, autograd won’t backpropagate through<br>
<code>argmax()</code> (but if it did, it would backpropagate a zero gradient).</p>
<p>To add a differentiable mse-like term to your combined loss function, consider<br>
using something like the predicted-probability-weighted <code>MSELoss</code> I suggested<br>
in my previous post.</p>
<aside class="quote no-group">
<blockquote>
<p>Also, what would be the right coefficient a? for now I set a=1.</p>
</blockquote>
</aside>
<p>I don’t know of any a priori best value for <code>a</code>.  You should treat is as a tunable<br>
hyperparameter (like a learning rate or weight-decay coefficient) and perform<br>
multiple training runs with different values and see which value works best.</p>
<p>(But, as noted above, with your specific proposal, the value of <code>a</code> won’t have<br>
any effect on your training.)</p>
<p>Best.</p>
<p>K. Frank</p>