Problem with setting weights for BCEWithLogitsLoss-binary problem

liran: <p>Hi,<br>
I try to solve binary problem (2 labels) so I use nn.BCEWithLogitsLoss for my criterion. I tried to set weight of 500 for the positive labels (that`s also the minority class) by this way:</p>
<pre><code class="lang-auto">torch_weights = torch.ones([1]).cuda()
torch_weights = torch_weights*500
criterion = nn.BCEWithLogitsLoss(pos_weight=torch_weights )
</code></pre>
<p>Its has been compiled, but I think that its not the right way, because my loss function is on the same value, and if I change it to:<br>
<code>criterion = nn.BCEWithLogitsLoss(pos_weight=None)</code></p>
<p>I get much smaller value on the loss function<br>
What I’m doing wrong? Thank you for your HELP!</p>

Isaac_Kargar: <p>Did you compare the accuracy of these two cases? I think when you use weights, it’s not fair to compare loss values. I think accuracy would be a better metric to compare the performance.</p>

liran: <p>I think that accuracy is not the right metric for me, since my data is very imbalanced (only 3% of data is my minority class, and raw data is 200K raws). I can use other metrics, like prec, recall and f-score.<br>
Is this way is the right way for using weights?<br>
Thank you</p>

Isaac_Kargar: <p>Yes I think it is better than loss.</p>

liran: <p>Ok, so is this code is ok?</p>
<pre><code class="lang-auto">torch_weights = torch.ones([1]).cuda()
torch_weights = torch_weights*500
criterion = nn.BCEWithLogitsLoss(pos_weight=torch_weights )
</code></pre>

Isaac_Kargar: <p>Here you set the weights to 500 and you have one class only? Shouldn’t it be 2? Check <a href="https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/12">this</a> please.</p>

liran: <p><a class="mention" href="/u/isaac_kargar">@Isaac_Kargar</a> I saw this issue but I still didnt understand how to set weights to each class. If I have 2 classes (Binary problem) so I need to set weight only to the positive label?<br>
Thank you</p>

KFrank: <p>Hi Liran and Isaac!</p>
<aside class="quote no-group" data-username="liran" data-post="1" data-topic="81961" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/e19adc/48.png" class="avatar"> liran:</div>
<blockquote>
<p>I tried to set weight of 500 for the positive labels (that`s also the minority class)</p>
</blockquote>
</aside>
<p>As an aside, You mention in a later post that your positive class is<br>
about 3% of your data.  You might start out then by setting <code>pos_weight</code><br>
to about 30, so that (after weighting) your negative and positive classes<br>
are about equally weighted.  This won’t necessarily be the best choice<br>
for <code>pos_weight</code>, but it’s a good place to start, and you should certainly<br>
compare it with other values you try (such as 500).</p>
<aside class="quote no-group">
<blockquote>
<p>if I change it to:<br>
<code>criterion = nn.BCEWithLogitsLoss(pos_weight=None)</code></p>
<p>I get much smaller value on the loss function</p>
</blockquote>
</aside>
<p>This is not necessarily a problem.  With <code>pos_weight = None</code>, your<br>
model could easily train to predict your negative class (almost) all<br>
of the time, and almost always be right.  So you get a small loss.</p>
<p>When you use <code>pos_weight</code> to increase the weight of your<br>
under-represented positive class, your model actually has to learn<br>
to tell the two classes apart, so you are likely to get a larger, but<br>
more meaningful loss.</p>
<aside class="quote no-group" data-username="Isaac_Kargar" data-post="6" data-topic="81961" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/isaac_kargar/48/9424_2.png" class="avatar"> Isaac_Kargar:</div>
<blockquote>
<p>Here you set the weights to 500 and you have one class only? Shouldn’t it be 2? Check <a href="https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/12">this </a> please.</p>
</blockquote>
</aside>
<p>I think this doesn’t apply to Liran’s use case.  It sounds like he has a<br>
<em>single-label,</em> binary classification problem.  So <code>pos_weight</code> should<br>
have a <code>size</code> of 1 rather than 2.</p>
<p>(The terminology is a little bit confusing.  When we talk about a binary<br>
problem, we do say that there are two classes.  But when we talk<br>
about a multi-label, multi-class problem, we talk about <code>nClass</code><br>
classes.  But the multi-label problem should be understood as<br>
<code>nClass</code> <em>binary</em> problems (that share the same network) – that is,<br>
each of the <code>nClass</code> labels can be absent or present.  The example<br>
you link to is a 64-class multi-label, multi-class problem.  It therefore<br>
consists of 64 binary problems, each one having its own value of<br>
<code>pos_weight</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>

Nikhil_Chhabra: <p>So for binary image classification this is how it should be done i feel. Correct me if i am wrong.</p>
<p>Input= [batch_size, channels, height,width]<br>
predicted_mask = [batch_size, 1, height, width]<br>
gt_mask = [batch_size, 1, height, width]</p>
<p>Assuming your data has 25% positive pixels and 75% negative pixels. Then a good weight value will be 75/25 = 3</p>
<p>positive_weight = torch.ones([1])*3.0<br>
loss = nn.BCEWithLogitsLoss(pos_weight=positive_weight)</p>
<p>My only confusion is do i need to have extra one dimension for class or it doesn’t matter ?<br>
predicted_mask = <strong>[batch_size, 1, height, width] OR  [batch_size, height, width]</strong><br>
gt_mask = <strong>[batch_size, 1, height, width] OR  [batch_size, height, width]</strong></p>

KFrank: <p>Hi Nikhil!</p>
<aside class="quote no-group" data-username="Nikhil_Chhabra" data-post="9" data-topic="81961" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikhil_chhabra/48/14634_2.png" class="avatar"> Nikhil_Chhabra:</div>
<blockquote>
<p>Input= [batch_size, channels, height,width]<br>
predicted_mask = [batch_size, 1, height, width]<br>
gt_mask = [batch_size, 1, height, width]</p>
</blockquote>
</aside>
<p>This is fine.</p>
<aside class="quote no-group">
<blockquote>
<p>Assuming your data has 25% positive pixels and 75% negative pixels.</p>
<p>positive_weight = torch.ones([1])*3.0<br>
loss = nn.BCEWithLogitsLoss(pos_weight=positive_weight)</p>
</blockquote>
</aside>
<p>This is fine.</p>
<aside class="quote no-group">
<blockquote>
<p>My only confusion is do i need to have extra one dimension for class or it doesn’t matter ?<br>
predicted_mask = <strong>[batch_size, 1, height, width] OR  [batch_size, height, width]</strong><br>
gt_mask = <strong>[batch_size, 1, height, width] OR  [batch_size, height, width]</strong></p>
</blockquote>
</aside>
<p>I would call the “extra one dimension” (<em>singleton</em> dimension) a “channel”<br>
dimension (rather than “class”).</p>
<p>Regardless of terminology, as long as the shapes of <code>predicted_mask</code><br>
and <code>gt_mask</code> match, both ways will work and are essentially equivalent.</p>
<p>If the output of the model and <code>gt_mask</code> (from, say, your dataloader) already<br>
match, I wouldn’t bother doing anything.  If one has the singleton dimension<br>
and the other doesn’t, I would probably <code>squeeze()</code> the extra dimension away.</p>
<p>Best.</p>
<p>K. Frank</p>