How to apply a weighted BCE loss to an imbalanced dataset? What will the weight tensor contain?

phdproblems: <p>Hi,</p>
<p>There have been previous discussions on weighted BCELoss here but none of them give a clear answer how to actually apply the weight tensor and what will it contain?</p>
<p>I’m doing binary segmentation where the output is either foreground or background (1 and 0). But my dataset is highly imbalanced and there is way more background than foreground. (To be exact there is 95 times more background pixels than foreground). So I want to penalize the background by multiplying it with a small number.</p>
<p>I see that <code>BCELoss</code> has a weight parameter: <a href="https://pytorch.org/docs/stable/nn.html" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html</a> which according to docs is to be of the size <code>nbatch</code>.</p>
<p>So if I have a batch of size 1, would the eight tensor be of size 1 as well? wont that just be a single float value?</p>
<p>ive read the discussion here: <a href="https://discuss.pytorch.org/t/binary-cross-entropy-weights/13299" class="inline-onebox">Binary cross entropy weights</a> but that does not answer what the weight tensor would look like for a batch size of 1?</p>
<p>Sorry if I’m rambling but I can’t seem to find anywhere how to use the weight parameter properly. If my output and label tensors are of the size <code>[1, 1, 60, 40, 40]</code> how would I make my weight parameter to penalize the background class (0)?</p>

KFrank: <p>Hi phdproblems!</p>
<aside class="quote no-group quote-modified" data-username="phdproblems" data-post="1" data-topic="56823" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/phdproblems/48/15453_2.png" class="avatar"> phdproblems:</div>
<blockquote>
<p>I’m doing binary segmentation where the output is either foreground or background (1 and 0). But my dataset is highly imbalanced and there is way more background than foreground. (To be exact there is 95 times more background pixels than foreground).</p>
<p>…</p>
<p>Sorry if I’m rambling but I can’t seem to find anywhere how to use the weight parameter properly. If my output and label tensors are of the size <code>[1, 1, 60, 40, 40]</code> how would I make my weight parameter to penalize the background class (0)?</p>
</blockquote>
</aside>
<p>Let me add some hypothetical context to your question to make it<br>
more concrete:</p>
<p>As you say, you have a batch size of one.  Each batch tensor is<br>
of <code>torch.Size([1, 1, 60, 40, 40])</code>, so each sample is of<br>
<code>torch.Size([1, 60, 40, 40])</code>.</p>
<p>Let me imagine that each sample represents, say, a one-channel<br>
(e.g., black and white), three-dimensional image of 60 z-slices<br>
(or time-slices), each of x-y dimension of 40x40.  The output<br>
of your network is a tensor for which each element is the predicted<br>
probability (between 0 and 1) that the corresponding voxel (3-d<br>
pixel) in your 3-d image is a foreground voxel.  I will call<br>
this your <code>predict</code> tensor.</p>
<p>Your <code>target</code> tensor is of the same shape, and is the known<br>
training data with values of 0 or 1 indicating background and<br>
foreground voxels, respectively.</p>
<p>(That is, your network performs a binary classification of your<br>
voxels.  As far as the loss is concerned, you don’t care that<br>
the voxels happen to be arranged as a (one-channel) 60x40x40<br>
3-d image.)</p>
<p>To weight each voxel’s contribution separately, you want a weight<br>
tensor that is the same shape as your <code>predict</code> and <code>target</code> tensors,<br>
i.e., <code>torch.Size([1, 1, 60, 40, 40])</code>.  (So its first dimension is<br>
your batch size of one.)</p>
<p>The following script (for pytorch 0.3.0) illustrates this:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

torch.manual_seed (2019)

predict = torch.rand ((1, 1, 60, 40, 40))
target = torch.bernoulli (predict)
weight =  torch.rand ((1, 1, 60, 40, 40))

weight_rebal = torch.ones_like (target) / 95.0  +  (1.0 - 1.0 / 95.0) * target

predict_f = predict.clone()
target_f = target.clone()
weight_f = weight.clone()
predict_f.resize_((96000, 1))
target_f.resize_((96000, 1))
weight_f.resize_((96000, 1))

predict = torch.autograd.Variable (predict)
target = torch.autograd.Variable (target)

predict_f = torch.autograd.Variable (predict_f)
target_f = torch.autograd.Variable (target_f)

loss = torch.nn.functional.binary_cross_entropy (predict, target)
lossw = torch.nn.functional.binary_cross_entropy (predict, target, weight)
loss_rebal = torch.nn.functional.binary_cross_entropy (predict, target, weight_rebal)

loss_f = torch.nn.functional.binary_cross_entropy (predict, target)
loss_fw = torch.nn.functional.binary_cross_entropy (predict_f, target_f, weight_f)

print ('predict.shape =', predict.shape)
print ('loss =', loss)
print ('lossw =', lossw)
print ('loss_rebal =', loss_rebal)
print ('predict_f.shape =', predict_f.shape)
print ('loss_f =', loss_f)
print ('loss_fw =', loss_fw)
</code></pre>
<p>The output is:</p>
<pre><code class="lang-plaintext">0.3.0b0+591e73e
predict.shape = torch.Size([1, 1, 60, 40, 40])
loss = Variable containing:
 0.5002
[torch.FloatTensor of size 1]

lossw = Variable containing:
 0.2503
[torch.FloatTensor of size 1]

loss_rebal = Variable containing:
 0.2530
[torch.FloatTensor of size 1]

predict_f.shape = torch.Size([96000, 1])
loss_f = Variable containing:
 0.5002
[torch.FloatTensor of size 1]

loss_fw = Variable containing:
 0.2503
[torch.FloatTensor of size 1]
</code></pre>
<p><code>weight</code> is a tensor of random weights (one for each voxel).</p>
<p><code>weight_rebal</code> is a tensor of foreground-background weights<br>
(again, one for each voxel), computed from your training-data<br>
<code>target</code>, where background voxels are given a weight factor of<br>
1/95 (to account for their greater frequency of occurrence).</p>
<p>Last, the <code>_f</code> (for <em>flattened</em>) tensors and losses are just to show<br>
that the shape doesn’t affect the per-voxel loss computation.<br>
These can be understood, if you will, as consisting of a batch<br>
of 96,000 samples (batch size = 96,000) of single floating-point<br>
prediction values and single 0 or 1 class labels.</p>
<p>(As a side note, <code>CrossEntropyLoss</code> permits you to specify a<br>
<em>per-class</em> weight, e.g., one weight for foreground and another<br>
weight for background voxels, rather than having to pass in a<br>
tensor of <em>per-voxel</em> weights.  You could recast your problem<br>
as a two-class (foreground / background) multiclass classification<br>
problem and use <code>CrossEntropyLoss</code>, but I wouldn’t recommend it.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

phdproblems: <p>Thank you so much. I think this might be the first detailed response on this question here. And for once cleared up the weight tensor size and shape confusion. I really appreciate the time you took in this detailed answer, cleared up all of my confusions <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=12" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="56823">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>(As a side note, <code>CrossEntropyLoss</code> permits you to specify a<br>
<em>per-class</em> weight, e.g., one weight for foreground and another<br>
weight for background voxels, rather than having to pass in a<br>
tensor of <em>per-voxel</em> weights. You could recast your problem<br>
as a two-class (foreground / background) multiclass classification<br>
problem and use <code>CrossEntropyLoss</code> , but I wouldn’t recommend it.)</p>
</blockquote>
</aside>
<p>Can you please mention why you would not recommend this, I though BCE is just CE but for a binary problem, is it not advised to use CE for binary problems?</p>
<p>Thanks again.</p>

phdproblems: <p>hi another followup question to this.</p>
<p>How you explained in your answer is exactly what I need, i.e. calculate the weight tensor for each instance based on the target tensor. But from what i have read, pytorch does not support this, it only supports the same weight for all instances in a batch which has to be provided when the loss is declared/initialized.</p>
<p>Is this the case or can I provide a different weight for each instance?</p>
<p>Thank you</p>

KFrank: <p>Hello phdproblems!</p>
<aside class="quote no-group quote-modified" data-username="phdproblems" data-post="3" data-topic="56823" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/phdproblems/48/15453_2.png" class="avatar"> phdproblems:</div>
<blockquote>
<aside class="quote no-group quote-modified" data-username="KFrank" data-post="2" data-topic="56823">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>(You could recast your problem<br>
as a two-class (foreground / background) multiclass classification<br>
problem and use <code>CrossEntropyLoss</code> , but I wouldn’t recommend it.)</p>
</blockquote>
</aside>
<p>Can you please mention why you would not recommend this, I though BCE is just CE but for a binary problem, is it not advised to use CE for binary problems?</p>
</blockquote>
</aside>
<p>BCE takes a <em>single</em> number per sample for its prediction – the<br>
probability of the sample being in class “1”.</p>
<p>Multiclass CE takes, for N classes, N numbers for its prediction.<br>
These are the probabilities of the sample being in each of the<br>
N classes.  (Pytorch’s <code>CrossEntropyLoss</code> has a <code>softmax</code><br>
implicitly built in, so it takes logits, rather than probabilities.)</p>
<p>So if you use multiclass CE for a binary (two-class) problem,<br>
you will now have to pass two numbers to it instead of one.<br>
(If these are official probabilities, they will be redundant, in<br>
that they will sum to 1.)  So your network will have to have<br>
two (redundant) outputs instead of one.  Hardly a big deal,<br>
but it would presumably make your network ever so slightly<br>
less efficient.</p>
<p>If you do everything correctly, you will be doing exactly the<br>
same problem and will get exactly the same result (up to<br>
some round-off error), but it just seems cleaner to me to<br>
use BCE for the binary problem.</p>
<p>Best.</p>
<p>K. Frank</p>

KFrank: <p>Hi phdproblems!</p>
<aside class="quote no-group" data-username="phdproblems" data-post="4" data-topic="56823" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/phdproblems/48/15453_2.png" class="avatar"> phdproblems:</div>
<blockquote>
<p>But from what i have read, pytorch does not support this, it only supports the same weight for all instances in a batch which has to be provided when the loss is declared/initialized.</p>
<p>Is this the case or can I provide a different weight for each instance?</p>
</blockquote>
</aside>
<p>Well, the documentation for the <code>weight</code> argument in <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCELoss" rel="noopener nofollow ugc">BCELoss</a><br>
and <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.binary_cross_entropy" rel="noopener nofollow ugc">binary_cross_entropy</a> is, to put it nicely, somewhat abbreviated.</p>
<p>I purposely used <code>binary_cross_entropy</code> in my example,<br>
because you can pass in a batch of weights (together with<br>
your <code>predict</code> and <code>target</code>) every time the loss is called.</p>
<p>(As you note, with <code>BCELoss</code> you pass in the <code>weight</code> only at<br>
the beginning when you instantiate the <code>BCELoss</code> class, so<br>
you can’t give it different weights every time you call it with a<br>
new <code>predict</code> and <code>target</code>.)</p>
<p>Also in my example I showed that passing in per-voxel weights<br>
to <code>binary_cross_entropy</code> does indeed work, even if the<br>
documentation doesn’t make this clear, and showed that this<br>
gives the same result as the “flattened” version (<code>predict_f</code>,<br>
<code>target_f</code>, <code>weight_f</code>) where a batch of weights – one weight<br>
for each sample in the batch – is passed in, consistent with<br>
the simplest interpretation of the documentation.</p>
<p>Good luck.</p>
<p>K. Frank</p>

phdproblems: <p>You’re right, sorry I missed that you were using <code>binary_cross_entropy()</code> and not <code>BCELoss()</code> (which I am using).</p>
<p>Thanks for clarifying. I’ll just use <code>binary_cross_entropy()</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thanks for your help</p>