BCEWithLogitsLoss and Class Weights

username: bfeeny
cooked: <p>I am trying to apply Class Weights to BCEWithLogitsLoss.  I felt I had read every post on this and was well prepared to do so but apparently not.</p>
<p>I have two classes.</p>
<pre><code class="lang-auto">print(class_weights)
[ 0.50897302 28.36130137]
criterion = nn.BCEWithLogitsLoss(pos_weight=torch.tensor(class_weights, device=device))
</code></pre>
<p>Each batch is 136 items.</p>
<p>I get:</p>
<p><code>RuntimeError: output with shape [136, 1] doesn't match the broadcast shape [136, 2]</code></p>
<p>I am not sure what I am doing wrong</p>

username: KFrank
cooked: <p>Hello bfeeny!</p>
<aside class="quote no-group quote-modified" data-username="bfeeny" data-post="1" data-topic="88837" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bfeeny/48/26059_2.png" class="avatar"> bfeeny:</div>
<blockquote>
<pre><code class="lang-auto">print(class_weights)
[ 0.50897302 28.36130137]
criterion = nn.BCEWithLogitsLoss(pos_weight=torch.tensor(class_weights, device=device))
</code></pre>
<p><code>RuntimeError: output with shape [136, 1] doesn't match the broadcast shape [136, 2]</code></p>
</blockquote>
</aside>
<p>Because your output has shape <code>[136, 1]</code>, I conclude that your<br>
network has a single output, and you are performing a single-label<br>
(in contrast to multi-label) binary classification problem.</p>
<p>Even though you technically have two classes (the “0” or “no”<br>
class and the “1” or “yes” class), the <code>pos_weight</code> constructor<br>
argument of <code>BCEWithLogitsLoss</code> only takes a single weight,<br>
namely that for the “1” (“positive”) class.</p>
<p>So you want something like:</p>
<pre data-code-wrap="python"><code class="lang-python">criterion = nn.BCEWithLogitsLoss(pos_weight=torch.FloatTensor ([28.36 / 0.5090], device=device))
</code></pre>
<p>where I’ve chosen to pass in as <code>pos_weight</code> the relative weight<br>
given by what I presume are your weights for class “0” and<br>
class “1”.</p>
<p>Best.</p>
<p>K. Frank</p>

username: bfeeny
cooked: <p>Thanks for your reply.  I have much more of my 0 class and not much of my 1 class.  So you are saying I should pass in just the weight of my 1 class.  Would that be 28.36 / 0.5090?</p>
<p>0.5090 is the “weight” of my 0 class and 28.36 is the weight of my 1 class.</p>

username: KFrank
cooked: <p>Hello bfeeny!</p>
<aside class="quote no-group" data-username="bfeeny" data-post="3" data-topic="88837" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bfeeny/48/26059_2.png" class="avatar"> bfeeny:</div>
<blockquote>
<p>0.5090 is the “weight” of my 0 class and 28.36 is the weight of my 1 class.</p>
</blockquote>
</aside>
<p>Yes, I think that is what I am saying.</p>
<p>But, to be concrete, let’s take an extreme example:</p>
<p>Let’s say that your entire training set consists of 1,000 samples,<br>
990 class-“0” samples, and 10 class-“1” samples.  If you want<br>
to reweight the classes in your loss function to fully account for<br>
this imbalance, you would use:</p>
<pre><code class="lang-auto">criterion = nn.BCEWithLogitsLoss (pos_weight = torch.FloatTensor ([99.0]))
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: Borda
cooked: <p>Hi, how does this solution apply if I have 3 and more classes?</p>