Proper loss function for multilabel images with imbalanced data

ritchie46: <p>Hi,</p>
<p>I am training a modified vgg11 net on a dataset of images. I want to predict 40 labels (I use a 1D vector of length 40 with ones on the places that label is active). One image can have multiple labels (most are not exclusive, some are). Also my dataset has quite some imbalance. It is hard to balance it because there most of the times are multiple labels in the image. The final activation layer is a sigmoid function with 40 output nodes.</p>
<p>Which loss function is good for this purpose? I see that nn.NLLLoss gives me the possibility to weight the classes, where nn.MultiLabelMarginLoss don’t.</p>
<p>And when should I use MultiLabelSoftMarginLoss vs MultiLabelMarginLoss?</p>

smth: <p>because it’s a multilabel classification, you should use <code>BCEWithLogitsLoss</code>. The output of your network should come from a Linear layer and it should go into this loss.</p>
<p><a href="http://pytorch.org/docs/0.3.0/nn.html#bcewithlogitsloss" class="onebox" target="_blank">http://pytorch.org/docs/0.3.0/nn.html#bcewithlogitsloss</a></p>

ritchie46: <p>Hi, Thanks for your reply. Cool, cause that was the one I was using. I only got confused when I read people went for the other options. <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

gokriznastic: <p>Hey, I have a multilabel classification which is not binary, i.e. each class can have more than 2 states. (in my case- positive, negative, uncertain). Can I use BCEWithLogitsLoss for this and if not then what loss function do you suggest?</p>

KFrank: <p>Hi Gopal!</p>
<aside class="quote no-group" data-username="gokriznastic" data-post="4" data-topic="12440" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gokriznastic/48/16251_2.png" class="avatar"> gokriznastic:</div>
<blockquote>
<p>Hey, I have a multilabel classification which is not binary, i.e. each class can have more than 2 states. (in my case- positive, negative, uncertain). Can I use BCEWithLogitsLoss for this and if not then what loss function do you suggest?</p>
</blockquote>
</aside>
<p>Yes, you can.  The <code>target</code> for <code>BCEWithLogitsLoss</code> can be a<br>
probability (number between 0 and 1) – it is not required to be<br>
a class label (e.g., a number that is either 0 or 1).</p>
<p>So understand your <code>target</code> to be the probability of the sample<br>
being in the “positive” class.  Then “positive” → <code>P = 1.0</code>,<br>
“negative” → <code>P = 0.0</code>, and “uncertain” → <code>P = 0.5</code>.</p>
<p>(If you had your training data so annotated, you could, of course,<br>
use finer gradations, such as “leaning negative” → <code>P = 0.25</code>,<br>
and “leaning positive” → <code>P = 0.75</code>, or the like.)</p>
<p>(In the future, it would probably help for forum search and navigation<br>
if you could start a new thread with its own topic title for questions<br>
like this.)</p>
<p>Best.</p>
<p>K. Frank</p>

gokriznastic: <p>Hey K. Frank,<br>
Thank you for the quick reply.</p>
<p>I have a follow up query. I thought about the approach you suggest earlier and the problem with that in my particular case is I want my model to not learn wherever the labels are “uncertain”. The reason for this being-  once the model is trained I hope to use it to replace the “uncertain” labels with “positive” or “negative”.</p>
<p>Correct me if I’m wrong but putting 0.5 would make my model biased towards examples where it can easily predict “positive” or “negative” otherwise.</p>

KFrank: <p>Hello Gopal!</p>
<aside class="quote no-group" data-username="gokriznastic" data-post="6" data-topic="12440" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gokriznastic/48/16251_2.png" class="avatar"> gokriznastic:</div>
<blockquote>
<p>I want my model to not learn wherever the labels are “uncertain”. The reason for this being-  once the model is trained I hope to use it to replace the “uncertain” labels with “positive” or “negative”.</p>
<p>Correct me if I’m wrong but putting 0.5 would make my model biased towards examples where it can easily predict “positive” or “negative” otherwise.</p>
</blockquote>
</aside>
<p>My understanding of your situation is the following:</p>
<p>You have annotated training data.  You have samples that are<br>
“positive” and are labeled “positive”, and similarly for “negative.”<br>
You also have samples that are clearly “positive” or “negative”<br>
but are labeled “uncertain” (for some reason).</p>
<p>If you have cases where the labels don’t match reality, I would<br>
view those samples as not being (successfully) labeled, so I<br>
would simply leave them out of my training data.  If you don’t<br>
want to do that (Maybe, for a given sample, some of your<br>
multilabel labels are correctly labeled, while a few are labeled<br>
“uncertain.”), then you can pass (per-sample, per-multilabel-label)<br>
weights to the “function” version of <code>BCEWithLogitsLoss</code>, namely<br>
<a href="https://pytorch.org/docs/stable/nn.functional.html#binary-cross-entropy-with-logits" rel="noopener nofollow ugc">binary_cross_entropy_with_logits</a>, and use 0 for the weights of<br>
your “uncertain” instances.</p>
<p>Good luck.</p>
<p>K. Frank</p>