With BCEWithLogitsLoss, how do I deal with pos_weight if there are no positive cases in a batch?

shamoons: <p>My loss function is:</p>
<pre><code class="lang-auto">def loss_fn(inp, target):
    zeros_sum = (target == 0).sum(dim = 0).float()
    one_sum = (target == 1).sum(dim = 0).float()

    pos_weight = zeros_sum / one_sum 
    loss_fn = torch.nn.BCEWithLogitsLoss(reduction='mean', pos_weight=pos_weight)
    
    loss = loss_fn(inp, target)

    return loss
</code></pre>
<p>However, in certain batches, there are no <code>one_sum</code> examples for a particular class. How do I deal with <code>pos_weight</code> in those cases?</p>

KFrank: <p>Hi Shamoon!</p>
<aside class="quote no-group quote-modified" data-username="shamoons" data-post="1" data-topic="74788" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shamoons/48/21191_2.png" class="avatar"> shamoons:</div>
<blockquote>
<pre><code class="lang-auto">    pos_weight = zeros_sum / one_sum 
</code></pre>
<p>However, in certain batches, there are no <code>one_sum</code> examples for a particular class. How do I deal with <code>pos_weight</code> in those cases?</p>
</blockquote>
</aside>
<p>I would recommend (and would in general, not just for your case)<br>
not computing <code>pos_weight</code> on a per-batch basis, but, instead,<br>
use the <code>pos_weight</code> for your whole training set, or for a<br>
representative sample of your training set.</p>
<p>In any event, your individual batches should be representative<br>
of your whole training set.  They differ from one another randomly,<br>
of course, but shouldn’t differ from one another systematically.<br>
For example, you wouldn’t want to train on several batches of<br>
mostly positive samples, and then train on batches of mostly<br>
negative samples.</p>
<p>Best.</p>
<p>K. Frank</p>

shamoons: <p>I’m using the <code>DataLoader</code>, so I don’t really have all of the samples loaded at once - only batch per batch. And those batches are randomly loaded, so isn’t it the same to have my <code>pos_weight</code> be per batch and per dataset?</p>

KFrank: <p>Hello Shamoon!</p>
<aside class="quote no-group" data-username="shamoons" data-post="3" data-topic="74788" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shamoons/48/21191_2.png" class="avatar"> shamoons:</div>
<blockquote>
<p>I don’t really have all of the samples loaded at once - only batch per batch.</p>
</blockquote>
</aside>
<p>Pre-process your training set.  Count the number of positive vs.<br>
negative samples in your training set – or if this is too bulky or<br>
expensive, in a representative sample of your training set – and<br>
use these counts to calculate a single value of <code>pos_weight</code> that<br>
you use for all of your batches.</p>
<aside class="quote no-group">
<blockquote>
<p>And those batches are randomly loaded, so isn’t it the same to have my <code>pos_weight</code> be per batch and per dataset?</p>
</blockquote>
</aside>
<p>Well, for large enough batches, this will be approximately true.<br>
But, as you’ve discovered, with your batch size this can fail<br>
because a batch can can contain no positive sample, so you<br>
get a zero-divide when you try to calculate a per-batch<br>
<code>pos_weight</code> for such a batch.</p>
<p>Best.</p>
<p>K. Frank</p>

shamoons: <p>Part of the reason that some of the classes will not have any positive samples is because I am <code>pad_sequence</code> my data, so a bunch of <code>0</code> are added to the end. So I’ll always have some elements that have no positive case.</p>
<p>And it’s not that the batch doesn’t contain ANY positive samples. It doesn’t contain positive samples for a specific position of the <code>target</code> vector</p>

KFrank: <p>Hi Shamoon!</p>
<aside class="quote no-group quote-modified" data-username="shamoons" data-post="5" data-topic="74788" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shamoons/48/21191_2.png" class="avatar"> shamoons:</div>
<blockquote>
<p>some of the classes will not have any positive samples is because I am <code>pad_sequence</code> my data<br>
…<br>
It doesn’t contain positive samples for a specific position of the <code>target</code> vector</p>
</blockquote>
</aside>
<p>From this I understand that you are performing a <em>multi-label,<br>
multi-class</em> classification problem, where your “classes” are<br>
the positions in your sequence.</p>
<p>If it is really true that some positions (that is, some classes)<br>
are <em>never</em> positive across your entire training set, then don’t<br>
reweight such classes (no zero-divides).  Simply let your network<br>
learn that these classes are never positive, and your network<br>
will always (correctly) predict negative for them.</p>
<p>Good luck!</p>
<p>K. Frank</p>

shamoons: <p>So then are you proposing that I don’t use <code>pos_weight</code> at all? But I do have a severe imbalance since most of my value are negative most of the time.</p>