What is the strongest loss function?

YuhskeHujisaki: <p>Sometimes the loss function is not written when implementing deep learning in a dissertation. What is the recommended loss function at this time? I would appreciate it if you could write the reason as well.</p>

KFrank: <p>Hi Yuhske!</p>
<aside class="quote no-group" data-username="YuhskeHujisaki" data-post="1" data-topic="154366" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuhskehujisaki/48/48683_2.png" class="avatar"> YuhskeHujisaki:</div>
<blockquote>
<p>What is the recommended loss function at this time?</p>
</blockquote>
</aside>
<p>I always recommend starting out with the simplest standard approach<br>
commonly recommended for the type of problem, and only suggest<br>
moving to a more elaborate or less-standard approach if a good reason<br>
to do so becomes clear.</p>
<p>For a <em>binary classification</em> problem, start with <code>BCEWithLogitsLoss</code>.  This<br>
will also apply to something like binary semantic segmentation, where you<br>
are performing a binary classification (e.g., “background” vs. “foreground”)<br>
for each pixel.  If you have unbalanced data, try using the <code>pos_weight</code><br>
constructor argument before moving to a different loss function.</p>
<p>For a <em>multi-class classification</em> problem, start with <code>CrossEntropyLoss</code>.<br>
Its <code>weight</code> constructor argument can be used to compensate for unbalanced<br>
data.</p>
<p>For a <em>multi-label, multi-class classification</em> problem, go back to using<br>
<code>BCEWithLogitsLoss</code>.  This is because the multi-label problem should<br>
be understood as a set of binary problems, as each label ca be active<br>
independently for a given sample.  (E.g., does the first label apply to the<br>
sample – yes or no?; does the second label apply – yes or no?; and so<br>
on.)</p>
<p>For a “regression” problem, start with <code>MSELoss</code>.  By regression, I mean<br>
that your model predicts a floating-point number that should match, as<br>
closely as possible, a floating-point target – and that a close match is<br>
quantitatively better than a less-close match.  Let’s say that you are trying<br>
to predict the return on a stock price from today to tomorrow:  If tomorrow’s<br>
price turns out to be 1.1% higher than today’s, and you predict 1.0%, you’ve<br>
done quite well (so don’t penalize your prediction very much).  If you predict<br>
0.9%, you’ve still done quite well.  But a prediction of -1.0% or 3.5% would<br>
be significantly worse, and should be penalized more heavily.  That’s exactly<br>
how the squared errors in <code>MSELoss</code> work.</p>
<p>(As an aside <em>don’t</em> use <code>MSELoss</code> for typical classification problems.  Let’s<br>
say you have “0 – cat,” “1 – trout,” and “2 – sparrow.”  Misclassifying a<br>
sparrow as a trout is just as bad as misclassifying it as a cat.  But <code>MSELoss</code><br>
would say that trout would be less of an error than cat, so it’s not the right<br>
fit for this kind of problem, and you won’t train as effectively.)</p>
<p>Best.</p>
<p>K. Frank</p>