Multi-target loss recommendations

Saswat: <p>I’m working on a classification problem. The number of classes is 5000. I have a ground truth vector that has the shape (1000) instead of 1. The values in this target vector are the possible classes and the predicted vector is of the shape (1x5000) which holds the softmax scores for all the classes. Which loss function is recommended for such a use case?</p>

srishti-git1110: <p>Hi,<br>
Since you’ve used softmax, it should be a multi-class classification problem (rather than multi-label).<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#crossentropyloss" rel="noopener nofollow ugc">Cross Entropy loss</a> is the most used for such cases.</p>
<p>However, what about your target vector?<br>
It should be the same shape as the predicted outputs. Better to have it in a one-hot encoded fashion with 1 corresponding to the index of the right class, 0s otherwise.</p>

Saswat: <p>Ground truth vector is the target vector</p>
<p>For example:</p>
<pre><code class="lang-auto">predicted_vector = tensor([0.0669, 0.1336, 0.3400, 0.3392, 0.1203]
ground_truth = tensor([3,2,5])
</code></pre>
<p>For the above illustration, a typical argmax operation would result in declaring class 3 as the predicted class (0.34) but I want the model to reward even if the argmax class is any of 3,2, or 5</p>

srishti-git1110: <p>I didn’t quite understand you on this part -</p>
<aside class="quote no-group" data-username="Saswat" data-post="3" data-topic="162768">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saswat/48/49862_2.png" class="avatar"> Saswat:</div>
<blockquote>
<p>but I want the model to reward even if the argmax class is any of 3,2, or 5</p>
</blockquote>
</aside>

KFrank: <p>Hi Saswat (and Srishti)!</p>
<p>It’s not clear what your use case is.</p>
<p>The first question, as Srishti mentioned, is whether you have a <em>multi-label,</em><br>
multi-class problem or a <em>multi-class</em> (single-label) problem.</p>
<aside class="quote no-group" data-username="Saswat" data-post="3" data-topic="162768" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saswat/48/49862_2.png" class="avatar"> Saswat:</div>
<blockquote>
<p>For example:</p>
<pre><code class="lang-auto">predicted_vector = tensor([0.0669, 0.1336, 0.3400, 0.3392, 0.1203]
ground_truth = tensor([3,2,5])
</code></pre>
</blockquote>
</aside>
<p>Conceptually, is a given sample in exactly one class (say, class 3), but for<br>
training purposes, predicting class 2 or 5 is still okay so you don’t want to<br>
penalize your model too heavily?</p>
<p>Or, conceptually, can a given sample be in multiple classes at the same<br>
time, and in your example your sample really is in classes 3 and 2 and 5,<br>
and you would like to penalize your model if it <em>doesn’t</em> predict all three (and<br>
also penalize it if it were additionally to predict other classes, for example,<br>
class 4 or class 7)?</p>
<p>In the first case you have a single-label, multi-class problem, but with<br>
probabilistic (“soft”) labels, and you should use <code>CrossEntropyLoss</code><br>
(and not use <code>softmax()</code>).  In your example your (soft) target might be<br>
a probability of 0.7 for class 3, a probability of 0.2 for class 2, and a<br>
probability of 0.1 for class 5 (and zero for everything else).</p>
<p>In the second case you have a multi-label, multi-class problem, and you<br>
should use <code>BCEWithLogitsLoss</code> (and no <code>sigmoid()</code> nor <code>softmax()</code>).<br>
In this case your multi-label target might be 1.0 for classes 3, 2, and 5<br>
(and 0.0 for all other classes).  You can also use probabilities for targets<br>
with <code>BCEWithLogitsLoss</code>, e.g., perhaps, 0.9 for class 3, 0.8 for class 2,<br>
and 0.7 for class 5 (and 0.0 or close to 0.0 for all other classes).</p>
<p>First start with <em>conceptually</em> what kind of classification problem you have<br>
and then drill down as to what loss function you should use and how you<br>
should structure / interpret your target data.</p>
<p>Best.</p>
<p>K. Frank</p>

Saswat: <p>Thanks <a class="mention" href="/u/kfrank">@KFrank</a><br>
This was really elaborate and helpful. I’m dealing with the first case that you mentioned. Could you please list any pointers as to how to implement the CE loss function with soft labels?</p>

srishti-git1110: <p>Hi Saswat,<br>
As <a class="mention" href="/u/kfrank">@KFrank</a> has mentioned, your target vector should look like:</p>
<pre><code class="lang-auto">target = torch.tensor([0.1, 0.2, 0, 0, 0, 0.7])  # Let's say we have 6 classes only
</code></pre>
<p>essentially, it should contain the probabilities corresponding to <strong>each</strong> class.</p>
<p>And your <code>output</code> tensor could then be used to calculate the Cross-entropy loss:</p>
<pre><code class="lang-auto">loss_fn = torch.nn.CrossEntropyLoss()
loss = loss_fn(output, target)
</code></pre>
<p>Note that you do not have to include in your model a softmax layer explicitly. From the docs-</p>
<blockquote>
<p>Note that this case is equivalent to the combination of <a href="https://pytorch.org/docs/stable/generated/torch.nn.LogSoftmax.html#torch.nn.LogSoftmax" rel="noopener nofollow ugc"><code>LogSoftmax</code></a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss" rel="noopener nofollow ugc"><code>NLLLoss</code></a>.</p>
</blockquote>

Saswat: <p>Got it <a class="mention" href="/u/srishti-git1110">@srishti-git1110</a>. Thanks!</p>

Saswat: <p><a class="mention" href="/u/kfrank">@KFrank</a> I have a silly follow-up question on this.</p>
<aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="162768">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>In the first case you have a single-label, multi-class problem, but with<br>
probabilistic (“soft”) labels, and you should use <code>CrossEntropyLoss</code><br>
(and not use <code>softmax()</code>).</p>
</blockquote>
</aside>
<p>You’ve asked me not to use softmax. Is it just because <code>CrossEntropyLoss</code> applies softmax inherently? Or is it because of some other reason?</p>

srishti-git1110: <aside class="quote no-group quote-modified" data-username="srishti-git1110" data-post="7" data-topic="162768">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/srishti-git1110/48/52515_2.png" class="avatar"> srishti-git1110:</div>
<blockquote>
<p>Note that you do not have to include in your model a softmax layer explicitly. From the docs-</p>
<blockquote>
<p>Note that this case is equivalent to the combination of <a href="https://pytorch.org/docs/stable/generated/torch.nn.LogSoftmax.html#torch.nn.LogSoftmax" rel="noopener nofollow ugc"><code>LogSoftmax</code></a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss" rel="noopener nofollow ugc"><code>NLLLoss</code></a>.</p>
</blockquote>
</blockquote>
</aside>
<p><a class="mention" href="/u/saswat">@Saswat</a> This is the reason.</p>

Saswat: <p><a class="mention" href="/u/kfrank">@KFrank</a> If all classes in the target are equally likely, won’t the model get penalized for predicting only one of them and not predicting others?</p>
<p>For example, if the target vector is <code>[0,1,1,1,0,1,0,0,0,0]</code> and predicting 1 of classes 1,2,3 or 5 is fine and the model is doing so at some point, won’t “not-predicting” other classes send negative feedback to readjust the weights? I tried normalizing the target class values (to make them soft, since all have equal probability) so the new target looks like this `[0,0.25,0.25,0.25,0,0.25,0,0,0,0] but to no avail</p>