Pos_weight in BCEWithLogitsLoss not behaving as expected when using 'mean' reduction

fgoldmann: <p>The documentation for <a href="https://pytorch.org/docs/stable/nn.html#bcewithlogitsloss" rel="noopener nofollow ugc">nn.BCEWithLogitsLoss</a> states that</p>
<blockquote>
<p>For example, if a dataset contains 100 positive and 300 negative examples of a single class, then pos_weight for the class should be equal to 300/100=3. <em>The loss would act as if the dataset contains 3×100=300 positive examples.</em></p>
</blockquote>
<p>This is not the case when the default ‘mean’ reduction is used. I found that the positive class weight is applied correctly, but the total loss is only divided by the unadjusted number of samples.</p>
<p>Example:</p>
<ul>
<li>Imbalanced Dataset with 10% positives and 90% negatives.</li>
<li>Batch size 10</li>
<li>Loss for positive sample is 2.0, loss for negative samples is 0.5</li>
</ul>
<p><code>With a batch size of 10, containing 1 positive and 9 negative examples, the mean loss will amount to (1×2.0 + 9×0.5) / (1 + 9) = 0.65</code><br>
<code>With a batch size of 10, containing 5 positive and 5 negative examples, the mean loss will amount to (5×2.0 + 5×0.5) / (5 + 5) = 1.25</code></p>
<p>From the description in the documentation, I would expect the loss to be 1.25 for any distribution of positives and negatives in a batch when pos_weight has been set correctly to num_neg/num_pos = 9. However, the denominator is not being adjusted when the pos_weight is being set, resulting in the following behaviour:<br>
<code>With a batch size of 10, containing 1 positive and 9 negative samples, the mean loss will amount to (1×9×2.0 + 9×0.5) / (1 + 9) = 2.25</code>, which is even higher than the loss would have been if the batch contained only positive samples (=2.0)</p>
<p>To get the loss to behave as expected, I had to use ‘sum’ reduction instead of ‘mean’ and manually divide the loss sum by the adjusted sample number (pos_weight × num_pos_samples + num_neg_samples):<br>
<code>With a batch size of 10, containing 1 positive and 9 negative samples, the mean loss will amount to (1×9×2.0 + 9×0.5) / (1×9 + 9) = 1.25</code></p>
<p>Is this intended behavior? If it is, should the documentation be updated for clarification?</p>

SofiaCP: <p>Any updates on this?</p>

KFrank: <p>Hi Sofia (and Florian)!</p>
<aside class="quote no-group" data-username="SofiaCP" data-post="2" data-topic="68620" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sofiacp/48/44302_2.png" class="avatar"> SofiaCP:</div>
<blockquote>
<p>Any updates on this?</p>
</blockquote>
</aside>
<p>Florian is correct that the documentation for <code>BCEWithLogitsLoss</code> that<br>
he quotes is not right for the reason he gives.</p>
<p>Perhaps this should be considered a minor documentation bug.</p>
<p>Please note, however, in general <code>BCEWithLogitsLoss</code> does not have<br>
“positive” and “negative” samples.  This is because <code>BCEWithLogitsLoss</code><br>
accepts probabilistic targets that can be any value between 0.0 and 1.0,<br>
rather that exactly 0 (negative sample) or exactly 1 (positive sample)<br>
class labels.</p>
<p><code>pos_weight</code> therefore doesn’t weight the entire sample, but, rather,<br>
weights the “positive” term in the binary-cross-entropy expression for<br>
a given sample.  So, in the general case of probabilistic targets, it is<br>
not possible to reproduce the effect of <code>pos_weight</code> by including<br>
multiple copies of certain samples in your input data.</p>
<p>Best.</p>
<p>K. Frank</p>