Calculating weighted BCEWithLogitsLoss

ayadav01: <p>I am slightly confused on using weighted BCEWithLogitsLoss. My input data shape is : 1 x 52 x 52 x 52 (3D Volume) and label for each volume is either 0 or 1 and i am using a batch size of 5. So, at each epoch, input is 5 x 1 x 52 x 52 x 52 and label is 1 x 5. The way i am calculating weights is:</p>
<pre><code class="lang-auto">weight_0 = count_of_lbl1 / (total_lbl_count)
weight_1 = count_of_lbl0 / (total_lbl_count)
</code></pre>
<p>My question is should i calculate weights for each batch or per dataset. Also, assuming my label per batch is [0, 0, 1, 0, 1] and weight_0 is 0.4 and weight_1 is 0.6, so does the weight tensor passed to nn.BCEWithLogitsLoss will be [0.4, 0.4, 0.6, 0.4, 0.6] ?</p>

KFrank: <p>Hello Anil!</p>
<aside class="quote no-group" data-username="ayadav01" data-post="1" data-topic="88739" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/c67d28/48.png" class="avatar"> ayadav01:</div>
<blockquote>
<p>My question is should i calculate weights for each batch or per dataset.</p>
</blockquote>
</aside>
<p>I prefer calculating the weight based on the entire dataset,<br>
rather than on a per-batch basis, although it shouldn’t matter<br>
if your batches are of reasonable size.  The point is that the<br>
weights aren’t magic numbers that have to be just right.  They<br>
are approximate numbers that are used to (partially) account<br>
for having a significantly unbalanced dataset.</p>
<aside class="quote no-group">
<blockquote>
<p>Also, assuming my label per batch is [0, 0, 1, 0, 1] and weight_0 is 0.4 and weight_1 is 0.6, so does the weight tensor passed to nn.BCEWithLogitsLoss will be [0.4, 0.4, 0.6, 0.4, 0.6] ?</p>
</blockquote>
</aside>
<p>This would be reasonable, but the disadvantage is that you would<br>
have to construct a new instance of <code>BCEWithLogitsLoss</code> for<br>
every batch, because your <code>weight</code> tensor depends on the batch.</p>
<p>I am assuming here that by “weight tensor” you mean the tensor<br>
you pass into <code>BCEWithLogitsLoss</code>’s constructor as its named<br>
<code>weight</code> argument, e.g.:</p>
<pre data-code-wrap="python"><code class="lang-python">criterion = torch.nn.BCEWithLogitsLoss (weight = my_weight_tensor)
loss = criterion (preds, targets)
</code></pre>
<p>It is more convenient to use the named <code>pos_weight</code> constructor<br>
argument:</p>
<pre><code class="lang-auto">criterion = torch.nn.BCEWithLogitsLoss (pos_weight = torch.FloatTensor ([count_of_lbl0 / count_of_lbl1]))
</code></pre>
<p>You can now use the same <code>criterion</code> loss object, constructed<br>
once, over and over again for each batch.</p>
<p>I would also note that if your relative weights are, in fact, <code>0.4</code><br>
and <code>0.6</code>, your dataset isn’t really very unbalanced, and I probably<br>
wouldn’t bother using weights in the loss function.</p>
<aside class="quote no-group">
<blockquote>
<p>My input data shape is : 1 x 52 x 52 x 52 (3D Volume) … and i am using a batch size of 5. So, at each epoch, input is 5 x 1 x 52 x 52 x 52 and label is 1 x 5.</p>
</blockquote>
</aside>
<p>As an aside, your shapes look a little confused.  I assume that<br>
“5 x 1 x 52 x 52 x 52” is the shape of the input to your <em>model</em><br>
(not your loss function).  The shape of your label is probably <code>[5]</code><br>
(although it could be <code>[5, 1]</code>), but a shape of <code>[1, 5]</code> would be<br>
wrong.</p>
<p>Best.</p>
<p>K. Frank</p>

ayadav01: <p>I see what you mean (took me a while to figure out). so rather than passing the <code>weight</code> argument (which is the rescaling weight of each bach element) to the loss function we pass the <code>pos_weight</code> argument (which is weight of positive example). In what scenarios would you want to use one or the other ? Also, if i am calculating <code>pos_weight</code> argument for each batch, don’t i still have to instantiate the loss function for each batch to pass the <code>pos_weight</code> argument. Will something like this work:</p>
<pre><code class="lang-auto">criterion = torch.nn.BCEWithLogitsLoss()
for img, label in trainloader:
    # Assuming i calculate 'pos_weights' for each batch
    weights = torch.FloatTensor ([count_of_lbl0 / count_of_lbl1])
    criterion.pos_weight = weights
    loss = criterion(output, label)
</code></pre>

KFrank: <p>Hi Anil!</p>
<aside class="quote no-group" data-username="ayadav01" data-post="3" data-topic="88739" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/c67d28/48.png" class="avatar"> ayadav01:</div>
<blockquote>
<p>rather than passing the <code>weight</code> argument (which is the rescaling weight of each bach element) to the loss function we pass the <code>pos_weight</code> argument (which is weight of positive example).</p>
</blockquote>
</aside>
<p>That is correct.</p>
<aside class="quote no-group">
<blockquote>
<p>In what scenarios would you want to use one or the other ?</p>
</blockquote>
</aside>
<p>If your sample weight only depends on the class of the sample,<br>
you can use either.  I tend to think that <code>pos_weight</code> is a little more<br>
convenient.</p>
<p>If your sample weight depends on something other than the sample’s<br>
class – for example, if you’re “hard mining” and want to weight “hard”<br>
samples more heavily – then you would have to use <code>weight</code> and<br>
provide per-sample weights.</p>
<p>Also, if you are using the different, but related loss class, <code>BCELoss</code><br>
(which you generally shouldn’t be using), you will have to use <code>weight</code><br>
because, for whatever reason, <code>BCELoss</code> doesn’t have a <code>pos_weight</code><br>
argument for its constructor.</p>
<aside class="quote no-group">
<blockquote>
<p>Also, if i am calculating <code>pos_weight</code> argument for each batch, don’t i still have to instantiate the loss function for each batch to pass the <code>pos_weight</code> argument.</p>
</blockquote>
</aside>
<p>Yes (although, as you note below, it does appear to be possible to<br>
modify  the <code>pos_weight</code> property after <code>BCEWithLogitsLoss</code> has<br>
been constructed).  But, again, my preference is to use the same<br>
<code>pos_weight</code> for the whole dataset, rather than calculate it for each<br>
batch.</p>
<aside class="quote no-group">
<blockquote>
<p>Will something like this work:</p>
<pre><code class="lang-auto">    weights = torch.FloatTensor ([count_of_lbl0 / count_of_lbl1])
    criterion.pos_weight = weights
</code></pre>
</blockquote>
</aside>
<p>I don’t see this discussed in the documentation, but it does appear to<br>
work for me (using pytorch 0.3.0, and <code>weight</code> rather than <code>pos_weight</code>).</p>
<p>I would probably avoiding doing it this way because I’m not sure that<br>
it’s officially supported.</p>
<p>Best.</p>
<p>K. Frank</p>

Jona: <p>Sorry for being late.</p>
<p>In my case, 90% of my dataset’s have negative class and 10% of positive class (both train and validation set, binary classification problem)</p>
<p>How woul I use <code>BCEWithLogitsLoss</code> or <code>BCELoss</code> with theirs paremeters <code>weights</code> and/or <code>pos_weight</code> to handle my imbalanced dataset’s?</p>
<p>Best regards</p>