Using `BCEWithLogisLoss` for multi-label classification

username: MrRobot
cooked: <h1>Question</h1>
<p>The key difference of <code>nn.CrossEntropyLoss()</code> and <code>nn.BCEWithLogitsLoss()</code> is the former uses Softmax while the latter uses multiple Sigmoid when computing loss. I have confusion about this</p>
<ul>
<li>Does using <code>nn.BCEWithLogitsLoss()</code> and setting a threshold (say 0.5) <strong>implicitly</strong> assume we are doing multi-label classification?</li>
<li>If it is so. Then if I <strong>wrongly</strong> use <code>nn.CrossEntropyLoss()</code> when I should have used <code>nn.CrossEntropyLoss()</code>.  If discard threshold 0.5 and choose class that has highest probability, then the performance is still very good. But how this is equivalent to using <code>nn.CrossEntropyLoss()</code>?</li>
</ul>

username: KFrank
cooked: <p>Hi Mr. Robot!</p>
<aside class="quote no-group quote-modified" data-username="MrRobot" data-post="1" data-topic="67011" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/b38774/48.png" class="avatar"> MrRobot:</div>
<blockquote>
<p>The key difference of <code>nn.CrossEntropyLoss()</code> and <code>nn.BCEWithLogitsLoss()</code> is the former uses Softmax while the latter uses multiple Sigmoid when computing loss.<br>
…</p>
</blockquote>
</aside>
<p>While true, this is hardly the <em>key</em> difference between the two.</p>
<p>Let me first clear up a potential point of confusion:</p>
<p>“Multi-class” classification means that a given sample is in precisely<br>
one class.  (One of your classes can be “background” or “no class”<br>
or “unclassified” if this fits your workflow.)  (Binary classification means<br>
that you have two classes, e.g., “yes” and “no” or “0” and “1”.)</p>
<p>“Multi-label” classification means that each sample can be in any<br>
number of the specified classes, including zero.  So multi-label<br>
classification can be understood as a series of binary classifications:<br>
Is sample 1 in class A – yes or no?  Is sample 1 in class B – yes or no?<br>
And so on.</p>
<p>You can’t use <code>CrossEntropyLoss</code> to do multi-label classification.<br>
(It’s just the ticket, though, for multi-class classification.)</p>
<p>You <em>can</em> use <code>BCEWithLogitsLoss</code> for multi-label classification<br>
(as well as for ordinary, single-label binary classification).</p>
<p>Could you clarify whether you are doing multi-class or multi-label<br>
classification, and then follow up with any further questions you have?</p>
<p>Good Luck.</p>
<p>K. Frank</p>

username: MrRobot
cooked: <p>Thank you for your explanation.</p>
<p>I am using MNIST data for multi-class classification (there are ten classes, 0, 1 through 9).</p>
<p>I could use <strong>both</strong> <code>nn.CrossEntropyLoss()</code> and <code>nn.BCEWithLogtisLoss()</code> on MNIST and get good performance. However, before reading your reply, I thought <code>nn.BCEWithLogitsLoss()</code> could <strong>only</strong> be using in the multi-label classification (you said it <strong>can</strong> be used in this setting).</p>
<p>Then here comes two questions</p>
<ul>
<li>
<strong>Focusing on multi-class classification</strong>. If using two losses can both achieve good performance, I suspect they should be similar in some sense but they actually use Softmax and multiple Sigmoid respectively, which I deem very different. For example, for Softmax, it makes sense to choose the class that has highest probability. However, for Sigmoid, it likely that two or more classes have very close output (say 0.79, 0.8, 0.81), then choosing one particular class does not seem to make sense and this is <strong>main confusion</strong> for using <code>nn.BCEWithLogitsLoss()</code> for multi-class classification.</li>
<li>
<strong>Focusing on multi-label classification</strong>. Set <code>nn.CrossEntropyLoss()</code> aside and we now focus on <code>nn.BCEWithLogitsLoss()</code>.  If I am building an image recognition system that is trying to identify whether there are car, people and stop sign in the given image at the same time. Now it is clear this is a multi-label classification problem (with three labels). However, people generally set 0.5 as a decision threshold for 1/0 classification. So a output like [0.2, 0.3, 0.4] will all be regarded as [0, 0, 0] by hard thresholding. But I believe this is similar to Logistic regression and the decision threshold should be tuned and using hard thresholding does not make sense. Now comes the <strong>main question</strong>. How do I tune three thresholds at the same time or maybe I just output probability (like many object detection algorithms do).</li>
</ul>

username: KFrank
cooked: <p>Hello Mr. Robot!</p>
<aside class="quote no-group" data-username="MrRobot" data-post="3" data-topic="67011" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/b38774/48.png" class="avatar"> MrRobot:</div>
<blockquote>
<p>I could use <strong>both</strong> <code>nn.CrossEntropyLoss()</code> and <code>nn.BCEWithLogtisLoss()</code> on MNIST and get good performance.</p>
</blockquote>
</aside>
<p>I’ve haven’t done the experiment, but I very much expect that if<br>
you train a network on reasonably clean multi-class data (say,<br>
MNIST digits) with both <code>CrossEntropyLoss</code> and<br>
<code>BCEWithLogitsLoss</code> you will get <em>better</em> performance using<br>
<code>CrossEntropyLoss</code>.</p>
<p>It’s worth saying a few words about why:</p>
<p>When you train with <code>BCEWithLogitsLoss</code> (with ten outputs<br>
from your network) you really are training a multi-label,<br>
multi-class classifier.  You happen to be training it with samples<br>
that always correspond to exactly one class.  If your training<br>
data is representative and reasonably clean, your network will<br>
learn that for the kinds samples you give it, the right answer is<br>
exactly one label.  Then when applied to a test sample (assuming<br>
that your training and test data are similar in character) you<br>
should get predictions that have a high probability for exactly<br>
one of the classes, and low probabilities for the others.</p>
<p>But your network had to <em>learn</em> that you were giving it – at<br>
least with the training set you were using – a single-label<br>
(but multi-class) problem.</p>
<p>On the other hand, if you use <code>CrossEntropyLoss</code> you are,<br>
in essence, telling your network that this is a single-label<br>
problem, so your network doesn’t have to learn this.  You’ve<br>
made its job easier by telling it this, so you should expect<br>
better performance.</p>
<aside class="quote no-group">
<blockquote>
<p>However, before reading your reply, I thought <code>nn.BCEWithLogitsLoss()</code> could <strong>only</strong> be using in the multi-label classification (you said it <strong>can</strong> be used in this setting).</p>
</blockquote>
</aside>
<p>Well, as a semantic quibble, you could consider ordinary binary<br>
classification to be multi-label in the sense that you have one class<br>
(say, “cat”) and zero (nothing, so “no cat”) or one (your only class<br>
is “cat”, so “yes cat”) labels.  You can’t have more than one label<br>
because you only have one class.  That is, you can have none, any,<br>
or all of your labels (in this case only one – “cat”) turned off or on.<br>
So this is a multi-label problem.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li><strong>Focusing on multi-class classification</strong>. If using two losses can both achieve good performance, I suspect they should be similar in some sense but they actually use Softmax and multiple Sigmoid respectively, which I deem very different. For example, for Softmax, it makes sense to choose the class that has highest probability. However, for Sigmoid, it likely that two or more classes have very close output (say 0.79, 0.8, 0.81), then choosing one particular class does not seem to make sense and this is <strong>main confusion</strong> for using <code>nn.BCEWithLogitsLoss()</code> for multi-class classification.</li>
</ul>
</blockquote>
</aside>
<p>I think you’ve identified the main point.</p>
<p>First, a technical note:  In both cases the output of your network<br>
will be “raw scores” (<em>logits</em>).  They, in effect, get converted to<br>
probabilities internally to the loss functions.  I will speak in terms<br>
of probabilities.</p>
<p><code>CrossEntropyLoss</code> internally uses (in effect) <code>Softmax</code> to convert<br>
the output of your network in to probabilities for which single class<br>
is being predicted.  These probabilities sum to one.</p>
<p><code>BCEWithLogitsLoss</code> internally uses (in effect) <code>Sigmoid</code> to convert<br>
the output of your network into probabilities for each of your classes<br>
individually being active.  These probabilities individually range<br>
from zero to one (and sum anywhere from zero to your number<br>
of classes, in the MNIST example, ten).</p>
<p>So, to use your example of probabilities <code>(0.79, 0.8, 0.81)</code>,<br>
your network is saying that class A, B, and C all have a probability<br>
of about 80% of being active, so you should logically predict that<br>
your image contains an “A” and a “B” and a “C”.  (You shouldn’t<br>
say that your image only contains a “C”, but it’s a kind of squiggly<br>
“C” that looks rather like an “A” and a “B”.)</p>
<p>Of course, if your problem really is single-label (multi-class),<br>
you can’t have “A”, “B”, and “C” at the same time.  This is why<br>
<code>BCEWithLogitsLoss</code> (with its internal <code>Sigmoid</code>) isn’t a good<br>
fit for the single-label problem.</p>
<p>Now with <code>CrossEntropyLoss</code> (with its internal <code>Softmax</code>) you<br>
will never get two classes both with greater than 50% probability.<br>
You could get <code>(0.33, 0.33, 0.34, 0.0, 0.0, ...)</code>, in<br>
which case you would predict class “C”, but with low certainty,<br>
because it’s a squiggly “C” that looks a lot like an “A” and a “B”.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li><strong>Focusing on multi-label classification</strong>. Set <code>nn.CrossEntropyLoss()</code> aside and we now focus on <code>nn.BCEWithLogitsLoss()</code>.  If I am building an image recognition system that is trying to identify whether there are car, people and stop sign in the given image at the same time. Now it is clear this is a multi-label classification problem (with three labels). However, people generally set 0.5 as a decision threshold for 1/0 classification. So a output like [0.2, 0.3, 0.4] will all be regarded as [0, 0, 0] by hard thresholding.</li>
</ul>
</blockquote>
</aside>
<p>And this is reasonable.  Your network has only a 40% probability<br>
of there being a stop sign in the image.  Your best guess is that<br>
there isn’t a stop sign (but you wouldn’t be the farm on it).</p>
<p>So perhaps your network is saying that the bicycle in the image<br>
looks a little bit like a car (20%), the bags of garbage looks<br>
somewhat like people (30%), and the restaurant logo looks<br>
rather like a stop sign (40%).  But your best prediction is that<br>
none of your classes appear in the image.</p>
<aside class="quote no-group">
<blockquote>
<p>But I believe this is similar to Logistic regression and the decision threshold should be tuned and using hard thresholding does not make sense.</p>
</blockquote>
</aside>
<p>Well, as I argued above, the hard, a priori thresholding does make<br>
sense.</p>
<p>Nonetheless, it is plausible that you could develop some tuning<br>
scheme for your post-network thresholds that would give you<br>
better accuracy.  But I think such tuning would likely be highly<br>
dependent on your particular problem and network.</p>
<aside class="quote no-group">
<blockquote>
<p>Now comes the <strong>main question</strong>. How do I tune three thresholds at the same time or maybe I just output probability (like many object detection algorithms do).</p>
</blockquote>
</aside>
<p>I don’t know of any good post-network threshold-tuning scheme,<br>
but it’s believable that you could develop one.</p>
<p>Whether you output probabilities or thresholded “yes / no”<br>
predictions really depends on your use case – on how you<br>
use the output of your classifier.  There are certainly many<br>
use cases where outputting probabilities is appropriate.</p>
<p>Best.</p>
<p>K. Frank</p>

username: MrRobot
cooked: <p>It is really nice to see such a detailed and convincing reply! Thank you very much!</p>