Weight for BCE loss

James_Lee: <p>Hi there. Is there a way for me to calculate the BCE loss for different areas of a batch with different weights? Seemed that the * <strong>weight</strong> (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noopener nofollow ugc"><em>Tensor</em></a><em>,</em> <em>optional</em>) – a manual rescaling weight if provided it’s repeated to match input tensor shape for <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.binary_cross_entropy_with_logits.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional.binary_cross_entropy_with_logits — PyTorch 1.9.1 documentation</a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html" class="inline-onebox" rel="noopener nofollow ugc">BCELoss — PyTorch 1.9.1 documentation</a> only work at a batch-level, not pixel-by-pixel? Can I adopt the weight for specific pixels? Thanks.</p>

KFrank: <p>Hi James!</p>
<aside class="quote no-group" data-username="James_Lee" data-post="1" data-topic="134221">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/james_lee/48/29487_2.png" class="avatar"> James_Lee:</div>
<blockquote>
<p>Can I adopt the weight for specific pixels?</p>
</blockquote>
</aside>
<p>The documentation is misleadingly unclear, but <code>BCEWithLogitsLoss</code><br>
(which you should use) and <code>BCELoss</code> (which you shouldn’t use)<br>
do support what you call “pixel-by-pixel” weights (as do their<br>
<code>functional</code> equivalents).</p>
<p>Let’s assume that <code>input</code> and <code>target</code> have the same shape (which<br>
they will in any reasonable use case).  Then what is required is that<br>
the <code>weight</code> argument passed into <code>BCEWithLogitsLoss</code>’s constructor<br>
have a shape that is <em>broadcastable</em> to the shape of <code>input</code> (and hence<br>
to the shape of <code>target</code>).  After broadcasting, <code>weight</code> is just used to<br>
weight each element-by-element term in the loss function on an<br>
element-by-element basis.</p>
<p>(Those elements can be pixels in an image, classes in a<br>
multi-label problem, channels, etc., or any combination thereof.<br>
<code>BCEWithLogitsLoss</code> doesn’t care how you choose to interpret<br>
them – they’re just elements of a multi-dimensional tensor.)</p>
<p>If <code>weight</code> has the same shape as <code>input</code>, then no broadcasting is done<br>
(or you could say that the broadcasting is trivial), and <code>weight</code> will be<br>
applied to “specific pixels” in the way that I think you’re asking for.</p>
<p>Best.</p>
<p>K. Frank</p>

James_Lee: <p>Many thanks for your detailed explanation, Frank. Is there a way to mask the cross-entropy loss with PyTorch BCE loss explicitly? I would like to mask out the low possibility pixels with weight=0 and keep others.</p>

KFrank: <p>Hi James!</p>
<aside class="quote no-group" data-username="James_Lee" data-post="3" data-topic="134221" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/james_lee/48/29487_2.png" class="avatar"> James_Lee:</div>
<blockquote>
<p>I would like to mask out the low possibility pixels with weight=0 and keep others.</p>
</blockquote>
</aside>
<p>If I understand your question correctly, you can just use weights of <code>0.0</code><br>
and <code>1.0</code> to do this.</p>
<p>Let’s say that the input to your model is a batch of images of shape<br>
<code>[nBatch, height, width]</code>, and that the output of your model (which<br>
will be the <code>input</code> to <code>BCEWithLogitsLoss</code>) and the <code>target</code> you pass<br>
to <code>BCEWithLogitsLoss</code> also have this shape.</p>
<p>Now let me assume that you somehow construct a<br>
<code>pixel_possibility_tensor</code>, also of the same shape, and that this<br>
tensor tells you the “possibility” of each pixel (whatever that means<br>
in your use case).  If a “possibility” of, say, less than <code>0.35</code> is considered<br>
“low possibility”, you would:</p>
<pre data-code-wrap="python"><code class="lang-python">weight = (pixel_possibility_tensor &gt; 0.35).float()
loss = (weight.numel() / weight.sum()) * torch.nn.BCEWithLogitsLoss (weight = weight) (input, target)
</code></pre>
<p>In the <code>loss</code> computation, the <code>0.0</code> values in <code>weight</code> will mask out<br>
the “low possibility” pixels (and the <code>1.0</code> values will keep unchanged<br>
the remaining pixels).  The <code>weight.numel() / weight.sum()</code><br>
pre-factor is so that <code>loss</code> will be the loss averaged only over the<br>
non-masked pixels, assuming that’s what you want – without it, all<br>
of the masked pixels would also cause a bunch of zeros to get<br>
averaged into the loss.</p>
<p>Best.</p>
<p>K. Frank</p>

James_Lee: <p>Got it. Thank you, Frank.</p>