Weights in BCEWithLogitsLoss, but with 'weight' instead of 'pos_weight'

Santosh_Gupta: <p>I’m looking how to do class weighting using BCEWithLogitsLoss.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html</a></p>
<p>The example on how to use <code>pos_weight</code> seems clear to me. If there are 3x more negative samples than positive samples, then you can set pos_weight=3</p>
<p>Does the <code>weight</code> parameter do the same thing?</p>
<p>Say that I set it <code>weight=torch.tensor([1, 3])</code>. Is that the same thing as <code>pos_weight=3</code></p>
<p>Also, is <code>weight</code> normalized? Is <code>weight=torch.tensor([1, 3])</code> the same as <code>weight=torch.tensor([3, 9])</code>, or are they different in how they affect the magnitude of the loss?</p>

ptrblck: <aside class="quote no-group" data-username="Santosh_Gupta" data-post="1" data-topic="146402">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/santosh_gupta/48/41337_2.png" class="avatar"> Santosh_Gupta:</div>
<blockquote>
<p>Does the <code>weight</code> parameter do the same thing?</p>
</blockquote>
</aside>
<p>No, the <code>weight</code> argument will apply the weight to each sample of the batch.</p>

Santosh_Gupta: <p>Thanks ptrblack!</p>
<p>I have one more question, about the example</p>
<blockquote>
<blockquote>
<blockquote>
<p>target = torch.ones([10, 64], dtype=torch.float32)  # 64 classes, batch size = 10<br>
output = torch.full([10, 64], 1.5)  # A prediction (logit)<br>
pos_weight = torch.ones([64])  # All weights are equal to 1<br>
criterion = torch.nn.BCEWithLogitsLoss(pos_weight=pos_weight)<br>
criterion(output, target)  # -log(sigmoid(1.5))<br>
tensor(0.2014)</p>
</blockquote>
</blockquote>
</blockquote>
<p>This seems like it could be one of two scenarios. Either a single binary class, with 64 examples per batch, and a batch size of 10. Or, it could be 64 binary classes, with one example per batch, and a batch size of 10.</p>
<p>How would <code>torch.nn.BCEWithLogitsLoss</code> know which scenario this is? If I put in a single value, pos_weight=torch.tensor(3), would it automatically assume it’s the former from the length of the tensor? And if I use pos_weight = torch.ones([64]*3, <code>BCEWithLogitsLoss</code> will assume it’s the former?</p>

ptrblck: <p><code>pos_weight = torch.ones([64]*3)</code> won’t work in this example and will raise:</p>
<pre data-code-wrap="python"><code class="lang-python">RuntimeError: The size of tensor a (64) must match the size of tensor b (10) at non-singleton dimension 1
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">pos_weight=torch.tensor(3)
</code></pre>
<p>will broadcast the <code>pos_weight</code> and you will get the same loss value if you use <code>torch.tensor(1.)</code>.</p>
<aside class="quote no-group" data-username="Santosh_Gupta" data-post="3" data-topic="146402">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/santosh_gupta/48/41337_2.png" class="avatar"> Santosh_Gupta:</div>
<blockquote>
<p>This seems like it could be one of two scenarios. Either a single binary class, with 64 examples per batch, and a batch size of 10. Or, it could be 64 binary classes, with one example per batch, and a batch size of 10.</p>
</blockquote>
</aside>
<p>The example uses 10 samples (<code>batch_size=10</code>) where each sample contains 64 classes which can be active or inactive and can thus be seen as a multi-label classification use case.</p>