A suitable way to punish mis-classification via nn.CrossEntropyLoss?

username: localh
cooked: <p>Good Afternoon,</p>
<p>I am wondering whether this is a suitable way to approach a problem or if I should consider alternatives (that I am unaware of) as well.</p>
<p>I have a relatively balanced classification problem involving 3 labels: 0, 1, and 2. I am interested, however, in paying a little more attention to getting the 0s right. My approach is, when using nn.CrossEntropyLoss, is to punish mistakes for the 0s more than 1 or 2 by:</p>
<pre><code class="lang-auto">weights = [2.0, 1.0, 1.0]
class_weights = torch.FloatTensor(weights).cuda()
criterion = nn.CrossEntropyLoss(weight=class_weights)
</code></pre>
<p>Is this is a reasonable strategy or are there other things that I might consider?</p>
<p>Thanks in advance for your time and consideration!</p>

username: KFrank
cooked: <p>Hi Andrew!</p>
<aside class="quote no-group" data-username="localh" data-post="1" data-topic="94343" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/localh/48/18991_2.png" class="avatar"> localh:</div>
<blockquote>
<pre><code class="lang-auto">weights = [2.0, 1.0, 1.0]
</code></pre>
</blockquote>
</aside>
<p>Yes, this is a reasonable approach to more accurately classify<br>
class-0 samples.</p>
<p>That is, adding these class weights to your loss function will train<br>
your network to more often correctly label class-0 samples as<br>
class-0 (fewer <em>false negatives</em>), but at the cost of more frequently<br>
incorrectly labelling class-1 and class-2 samples as class-0 (more<br>
<em>false positives</em> from the perspective of class-0).</p>
<aside class="quote no-group">
<blockquote>
<p>more attention to getting the 0s right.<br>
…<br>
punish mistakes for the 0s more than 1 or 2 by:</p>
</blockquote>
</aside>
<p>Just to emphasize what I alluded to above, this depends on what<br>
you mean by “getting the 0s right” and “punish mistakes for the 0s”.</p>
<p>If you consider mislabelling a class-1 as a class-0 (a false positive)<br>
as not “getting the 0s right” and a “mistake for the 0s,” then this<br>
likely <em>won’t</em> accomplish your goal.  But if you’re only counting<br>
mislabelling an actual class-0 (a false negative) as a mistake, this<br>
is the way to go.</p>
<p>Best.</p>
<p>K. Frank</p>

username: localh
cooked: <p>Thanks so much, this makes a lot of sense and this is precisely what I want:</p>
<blockquote>
<p>adding these class weights to your loss function will train<br>
your network to more often correctly label class-0 samples as<br>
class-0 (fewer false negatives)</p>
</blockquote>
<p>Out of curiosity, do you mind elaborating on this problem and what one might do?</p>
<blockquote>
<p>Just to emphasize what I alluded to above, this depends on what<br>
you mean by “getting the 0s right” and “punish mistakes for the 0s”.</p>
<p>If you consider mislabelling a class-1 as a class-0 (a false positive)<br>
as not “getting the 0s right” and a “mistake for the 0s,” then this<br>
likely won’t accomplish your goal.</p>
</blockquote>

username: KFrank
cooked: <p>Hi Andrew!</p>
<aside class="quote no-group quote-modified" data-username="localh" data-post="3" data-topic="94343" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/localh/48/18991_2.png" class="avatar"> localh:</div>
<blockquote>
<p>Out of curiosity, do you mind elaborating on this problem and what one might do?</p>
<blockquote>
<p>If you consider mislabelling a class-1 as a class-0 (a false positive)<br>
as not “getting the 0s right” and a “mistake for the 0s,” then this<br>
likely won’t accomplish your goal.</p>
</blockquote>
</blockquote>
</aside>
<p>Your network does what your loss function trains it to do.</p>
<p>As a general rule, if you train your network to do better on one thing,<br>
then – all else being equal – it will be likely to do worse on something<br>
else.</p>
<p>(That doesn’t mean you can’t train your network to do better on<br>
everything, perhaps by training longer, or using a more apt loss<br>
function, or using a better optimization algorithm, or training with<br>
more or better data, etc.)</p>
<p>My point is that using class weights that favor class-0, you’re telling<br>
your network training that you care less about getting classes 1 and<br>
2 right, so, in general, your network won’t perform as well on classes<br>
1 and 2, including mislabelling them as class-0 – because that’s what<br>
you trained it to do.</p>
<p>Now, if you don’t really care about mixing up classes 1 and 2, but<br>
want to get class-0 right from both a false negative and false positive<br>
perspective (that is, you’re willing, e.g.,  to mislabel class-1 as class-2,<br>
but you don’t want to mislabel class-1 as class-0), you could, at the<br>
extreme, train a binary classifier that identifies class-0 vs. everything<br>
else.  (Again, a trade-off: do better on both class-0 false positives and<br>
false negatives, at the cost of not distinguishing class 1 from class-2.)</p>
<p>Now, some speculation, because I’ve never actually tried this.  You<br>
could add to your conventional three-class loss function (class-0 vs.<br>
class-1 vs. class-2) that does distinguish between class-1 and class-2,<br>
a two-class loss function (class-0 vs other-than-class-0).  This will<br>
help train your network to do better on both class-0 false negatives<br>
and false positives, while still somewhat distinguishing class-1 and<br>
class-2, but at the cost of not distinguishing them as well.</p>
<p>This will also bias your network to do better on class-0, but in a<br>
way that’s different than overweighting class-0 in your conventional<br>
three-class loss function.  You’d still be making a trade-off, just a<br>
different one.  Which way to go depends on the details of what’s<br>
more important to you.</p>
<p>Best.</p>
<p>K. Frank</p>

username: localh
cooked: <p>Wow! Thank you so much for the insightful and easy to understand post. Your comment below actually sparked a memory from something that I read not too long ago, a journal article I believe, which I think did just that. I am going to look for it and see if I can apply the same for fun/learning.</p>
<blockquote>
<p>Now, some speculation, because I’ve never actually tried this. You<br>
could add to your conventional three-class loss function (class-0 vs.<br>
class-1 vs. class-2) that does distinguish between class-1 and class-2,<br>
a two-class loss function (class-0 vs other-than-class-0). This will<br>
help train your network to do better on both class-0 false negatives<br>
and false positives, while still somewhat distinguishing class-1 and<br>
class-2, but at the cost of not distinguishing them as well.</p>
</blockquote>
<p>If interested, I can post it here for you to skim – but otherwise, this has been quite the learning experience for me. Thank you again!</p>
<p>Edit: I found it. It was a Stanford ML final project paper. The point is located on the end of page 3: <a href="https://github.com/JRC1995/BERT-Disaster-Classification-Capsule-Routing/blob/master/Project_Report.pdf" rel="noopener nofollow ugc">https://github.com/JRC1995/BERT-Disaster-Classification-Capsule-Routing/blob/master/Project_Report.pdf</a>. I think its similar to your point.</p>