"weight" vs. "pos_weight" in nn.BCEWithLogitsLoss()

stark: <p>I’m confused reading the explanation given in the official doc i.e.,</p>
<ul>
<li><strong>pos_weight</strong> (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noopener nofollow ugc"><em>Tensor</em></a><em>,</em> <em>optional</em> ) – a weight of positive examples. Must be a vector with length equal to the number of classes.</li>
</ul>
<blockquote>
<p>For example, if a dataset contains 100 positive and 300 negative examples of a single class, then <code>pos_weight</code> for the class should be equal to 300/100=3. The loss would act as if the dataset contains 3×100=300 positive examples.</p>
</blockquote>
<ul>
<li><strong>weight</strong> (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noopener nofollow ugc"><em>Tensor</em></a><em>,</em> <em>optional</em> ) – a manual rescaling weight given to the loss of each batch element. If given, has to be a Tensor of size nbatch.</li>
</ul>
<p>In the context of multi-label binary classification (i.e., semantic segmentation):</p>
<ol>
<li>How <strong><code>weight</code></strong> is supposed to be calculated when the ground truth mask is of the shape [N, 1, H, W]?</li>
<li>In what ways both parameters work differently and contribute to the calculation of the BCE loss?</li>
</ol>

Manuel_Alejandro_Dia: <p>According to the docs, in CEL the weights you have to provide are manual rescaling weights given to each class. This is so the classes with the lowest amount of occurrences will have the same influence on the loss than the classes with the highest amount of occurrences.</p>
<p>Here is a small script that could help you get these weights:</p>
<pre><code class="lang-auto">n_classes = 13
total_pxls = np.zeros((n_classes,1))
for img_name in file_list:
    print("image: %s"%img_name)
    new_lbl = cv2.imread(img_name,-1)
    new_lbl = np.array(new_lbl, dtype=np.uint8)
    for k in range(0,n_classes):
        total_pxls[k] = np.sum(new_lbl==k)+total_pxls[k]

no_total_pxls = np.sum(total_pxls[1::])
medians = [total_pxls[k]/no_total_pxls for k in range(1,n_classes)]
median_all = np.mean(medians)
my_freqs = median_all/(np.float64(medians)+(10E-14))
print(['%4.4f'%x for x in np.round(np.float64(my_freqs),3)])#these are the weights
</code></pre>

cameron_a: <p>This answer is helpful, but does not cover the full scope of the original question (which I was hoping to receive). What is the situation when you have a binary semantic classification problem (e.g. output prediction size = [B, 1, H, W] and the same for the mask → only one output channel, not 2)?</p>

KFrank: <p>Hi Cameron!</p>
<p>The short answer is that <code>BCEWithLogitsLoss</code> secretly doesn’t really<br>
have a notion of “classes” inside of it.  The weights you pass in just<br>
match up with the elements of your predictions and targets. Whether<br>
you have one or two or more “output channels” (or no “channels”<br>
dimension) is irrelevant – it’s just one more dimension that<br>
<code>BCEWithLogitsLoss</code> processes in the same way.</p>
<p>Some more detail:</p>
<p>(A disclaimer:  I might not be entirely right about this.  It’s also<br>
conceivable that the generality of the operation of <code>BCEWithLogitsLoss</code><br>
has increased over time, so my comments might not apply to older<br>
versions of pytorch.  I am currently working with version 1.7.1.)</p>
<p>The documentation for <a href="https://pytorch.org/docs/1.8.1/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">BCEWithLogitsLoss</a> is somewhat misleading.  It<br>
talks about batches and classes and such, but it just has dimensions;<br>
it doesn’t, per se, operate on batch dimensions and class dimensions,<br>
and so on.  The <em>user</em> is allowed to think in those terms, but that<br>
doesn’t affect what <code>BCEWithLogitsLoss</code> actually does.</p>
<p>Consider this example:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

_ = torch.manual_seed (2021)

nBatch = 2

# consider a "pos_weighted" multi-label, 15-class use case:

nClass = 15

pred_class = torch.randn (nBatch, nClass)
targ_class = torch.rand (nBatch, nClass)
posw_class = torch.rand (nBatch, nClass)

loss_class = torch.nn.BCEWithLogitsLoss (pos_weight = posw_class) (pred_class, targ_class)
print ('loss_class =', loss_class)

# but is there any difference between 15 classes and 3 * 5 image pixels?

height = 3
width = 5

pred_image = pred_class.reshape (nBatch, height, width)
targ_image = targ_class.reshape (nBatch, height, width)
posw_image = posw_class.reshape (nBatch, height, width)

loss_image = torch.nn.BCEWithLogitsLoss (pos_weight = posw_image) (pred_image, targ_image)
print ('loss_image =', loss_image)

# and is the batch dimension treated specially?

loss_flat = torch.nn.BCEWithLogitsLoss (pos_weight = posw_class.flatten()) (pred_class.flatten(), targ_class.flatten())
print ('loss_flat =', loss_flat)
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">1.7.1
loss_class = tensor(0.6958)
loss_image = tensor(0.6958)
loss_flat = tensor(0.6958)
</code></pre>
<p>The documentation hints at this in its “shape” section:</p>
<blockquote>
<p>Shape:</p>
<pre><code>       Input: (N, *) where "*" means, any number of additional dimensions

       Target: (N, *), same shape as the input

       Output: scalar. If reduction is 'none', then (N, *), same shape as input.
</code></pre>
</blockquote>
<p>But <code>BCEWithLogitsLoss</code> doesn’t actually differentiate between the<br>
batch dimension and the other dimensions.  I think this part of the<br>
documentation should simply read:</p>
<blockquote>
<p>Shape:</p>
<pre><code>       Input: (*) where "*" means, any number of dimensions
</code></pre>
</blockquote>
<p>(Use cases with “batches” and “classes” and “channels” and “images”<br>
could be illustrated with examples, but the documentation should state<br>
clearly that there is no operational difference.)</p>
<p>(As far as I can tell, <code>input</code> and <code>target</code> must have the shame shape<br>
and <code>pos_weight</code> must be broadcastable over <code>input</code> and <code>target</code>.<br>
Also <code>weight</code> enters a little differently into the per-element loss<br>
function, but behaves the same as <code>pos_weight</code> in terms of shapes<br>
and dimensions.)</p>
<p>Best.</p>
<p>K. Frank</p>

cameron_a: <p>Thank you for the response!</p>
<p>Ya, I realized after some discussion on a <a href="https://discuss.pytorch.org/t/weights-in-bcewithlogitsloss/27452/21">related post</a> that the default return type for pos_weight is a single mean value of the error calculation for every pixel. So what is propagated backwards is a single value. This made a lot more sense to me after.</p>

sumona: <p><a class="mention" href="/u/manuel_alejandro_dia">@Manuel_Alejandro_Dia</a> Hi, I have a doubt about the binary classification problem. In my case, my negative class is the minority class. Would pos_weight work in that case?</p>

Manuel_Alejandro_Dia: <p>Yes!</p>
<p>In this case, your pos_weight will  be under 1, since the positive class is the majority one.</p>
<p>Basically, what you want to do in the end, is to use this parameter to balance the addition each class does to the loss.</p>

sumona: <p>Hi, I have one more doubt. How can address class imbalance using BCELoss, because it does not have pos_weight but weight? What should be the value of weight if the ratio of my labels negative: positive is 20:80?</p>

cgr71ii: <p>Hi!</p>
<p>Just in case anyone is in any doubt, I think the most important part of the code to understand what’s being discussed here is this: <a href="https://github.com/pytorch/pytorch/blob/5301710b1535c9141c7e5f22c70e59742b04ec57/aten/src/ATen/native/Loss.cpp#L350-L351" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/Loss.cpp at 5301710b1535c9141c7e5f22c70e59742b04ec57 · pytorch/pytorch · GitHub</a></p>