Multi-label model outputs only negative values while trained with BCEWithLogitsLoss()

amirhf: <p>Hi,</p>
<p>I am training a multi-class multi-label model with <code>torch.nn.BCEWithLogitsLoss()</code> on 8M data points for 10 epochs. I have 54 classes and a single image can have multiple labels. During training the loss decreases nice and decreasing:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d9d74cc55ce915325ee8610eb544f0f4c8cacd6.png" data-download-href="https://discuss.pytorch.org/uploads/default/5d9d74cc55ce915325ee8610eb544f0f4c8cacd6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d9d74cc55ce915325ee8610eb544f0f4c8cacd6_2_690x325.png" alt="image" data-base62-sha1="dm9KHhMrqdcH2GqCahlLLfbLSia" width="690" height="325" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d9d74cc55ce915325ee8610eb544f0f4c8cacd6_2_690x325.png, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d9d74cc55ce915325ee8610eb544f0f4c8cacd6.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d9d74cc55ce915325ee8610eb544f0f4c8cacd6.png 2x" data-dominant-color="FDFEFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">898×423 9.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However, when I look at trained my model outputs for the last epoch, I see that the model is outputing negative values only. For example, for one sample with the following label:</p>
<pre><code class="lang-auto">target = 
tensor([[0., 0., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.,
             0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
             0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 0., 1., 0., 0., 0., 0., 0.]])
</code></pre>
<p>I get the following output from the model:</p>
<pre><code class="lang-auto">output = 
tensor([[-1.2380, -2.3283, -2.3025, -2.1275, -2.1020, -2.3684, -3.4669, -3.4503,
         -2.1905, -1.8565, -3.4215, -3.5318, -3.5715, -4.3836, -4.5215, -6.2270,
         -3.8660, -3.7280, -4.6043, -4.7601, -9.5219, -9.4969, -9.4392, -8.0596,
         -6.0773, -5.7972, -4.2495, -4.4533, -4.2641, -4.1068, -4.9987, -4.9321,
         -7.9726, -7.4475, -4.8016, -5.6634, -6.3762, -6.0103, -6.7561, -3.3259,
         -3.8778, -6.7682, -6.5663, -4.0945, -3.0747, -5.5408, -5.6429, -5.9659,
         -5.8574, -7.6435, -7.8895, -6.6514, -6.5506, -5.0583]],
       device='cuda:0')
</code></pre>
<p>So if I do sigmoid on top of this, I won’t get any good prediction.</p>

KFrank: <p>Hi Amir -</p>
<aside class="quote no-group" data-username="amirhf" data-post="1" data-topic="103422" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amirhf/48/5634_2.png" class="avatar"> amirhf:</div>
<blockquote>
<pre><code class="lang-auto">target = 
tensor([[0., 0., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.,
             0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
             0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 0., 1., 0., 0., 0., 0., 0.]])
</code></pre>
</blockquote>
</aside>
<p>The short answer is that your dataset is “unbalanced,” so you<br>
should try using the <code>pos_weight</code> argument when you construct<br>
your <code>BCEWithLogitsLoss</code> loss criterion.</p>
<p>Looking at your target, and naively assuming that all positive<br>
class labels appear about equally frequently, it appear that any<br>
giving class will be labelled positive only once in about every<br>
nine images.</p>
<p>So your classifier could do a pretty good job by just predicting<br>
negative for all of your classes all of the time.</p>
<p>It is the purpose of the <code>pos_weight</code> argument to address this<br>
issue by weighting the less-frequent positive samples more heavily<br>
than the more-frequent negative samples.  Doing so will penalize<br>
a model that simply tries to predict negative all the time.</p>
<p>It’s quite likely that some classes have positive labels more often<br>
than others.  <code>pos_weight</code> takes a vector of per-class weights so<br>
that you can reweight each class separately.  A common (and<br>
reasonable) choice for the class weights is:</p>
<pre data-code-wrap="python"><code class="lang-python">weights[i] = total_samples / number_of_positive_class_i_samples[i]
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

amirhf: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a>,</p>
<p>Thank you fo analysis. However I wonder how you calculated this:</p>
<blockquote>
<p>Looking at your target, and naively assuming that all positive<br>
class labels appear about equally frequently, it appear that any<br>
giving class will be labelled positive only once in about every<br>
nine images.</p>
</blockquote>
<p>The target that I have here is only for one image, meaning that some classes are present for only one image!</p>

KFrank: <p>Hi Amir!</p>
<aside class="quote no-group" data-username="amirhf" data-post="3" data-topic="103422" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amirhf/48/5634_2.png" class="avatar"> amirhf:</div>
<blockquote>
<p>I wonder how you calculated this:</p>
</blockquote>
</aside>
<p>Yes, this was not meant to be a realistic calculation.  I was illustrating<br>
an oversimplified estimate of the class weights based on the single<br>
data point you provided.</p>
<p>The (unrealistic) details:  Out of the 54 binary labels in your target,<br>
six were positive.  If one assumes (without any evidence) that positive<br>
labels occur equally frequently for all 54 classes, and further assumes<br>
(again, without any evidence) that the single target you gave is in<br>
some sense randomly representative of all of your targets, then one<br>
would conclude that, across your ensemble of targets, any given<br>
class is labelled positive about one time in nine.</p>
<p>Under these assumptions, you would want to use the same value<br>
of <code>9.0</code> for the <code>pos_weight</code> for all of your 54 classes.</p>
<p>Of course, these assumptions are probably not correct, so you should<br>
look at a representative sample of your training data to determine the<br>
per-class <code>pos_weight</code> values.</p>
<p>Best.</p>
<p>K. Frank</p>