"pos_weight" and "weight" parameters in BCEWithLogitsLoss

username: Ali_Aghababaei
cooked: <p>Hi everyone,</p>
<p>I have gotten confused in understanding the “pos_weight” and “weight” parameters in BCEWithLogitsLoss. I’ve read all the relevant discussions in this regard, to my knowledge; however, still, I’ve not understood them completely.</p>
<p>Imagine that I have a multi-class, multi-label classification problem; my imbalanced one-hot coded dataset includes 1000 images with 4 labels with the following frequencies: class 0: 600, class 1: 550, class 2: 200, class 3: 100.<br>
As I said, the targets are in a one-hot coded structure.<br>
For instance, the target [0, 1, 1, 0] means that classes 1 and 2 are present in the corresponding image.</p>
<p>Well, in order to calculate the BCEWithLogitsLoss concerning the data imbalance, one way is that suggested in this post: <a href="https://discuss.pytorch.org/t/multi-label-multi-class-class-imbalance/37573/2">Multi-Label, Multi-Class class imbalance - #2 by ptrblck</a>. Here, we calculate the class weights by inverting the frequencies of each class, i.e., the class weight tensor in my example would be: torch.tensor ([1/600, 1/550, 1/200, 1/100]). After that, the class weight tensor will be multiplied by the unreduced loss and the final loss would be the mean of this tensor.</p>
<p>However, as far as I know, the pos_weight parameter of the BCEWithLogitsLoSS could also be used in this case. Here is my question:<br>
to my knowledge, the two tensors (class weight tensor in the previous paragraph and the pos_weight tensor) are totally different. For each class, the number of positive and negative samples should be calculated and the num_negative/ num_positive would be the pos_weight for that particular class. In my example, the pos_weight of class 0 would be (1000-600)/600 = 0.67, the pos_weight of class 1 would be (1000-550)/550 = .82, the pos_weight of class 2 would be (1000-200)/200 = 4, and the pos_weight of class 3 would be (1000-100)/100 = 9. Thus, the pos_weight tensor would be torch.tensor ([.67, .82, 4, 9]). Is this way of calculating pos_weight tensor the right one? If the answer is yes, I think that the previous method (I mean calculating the class weights and multiplying it with the unreduced loss) would be more convenient in the case of a dataset with a large number of labels since we should only invert the frequencies, am I right?</p>
<p>Also, another question is about the weight parameter of the BCEWithLogitsLoss. As represented in the formula, the weight parameter is a tensor that is multiplied by the whole loss, not merely the positive targets (as opposed to pos_weight). My question is that how the weight parameter tensor is different from the class weights tensor, considering that the class weights tensor is similarly multiplied by the whole loss. However, it is said that the weight parameter tensor is of size nbatch, and I do not understand what its function is.</p>
<p>I deeply appreciate your consideration.</p>

username: KFrank
cooked: <p>Hi Ali!</p>
<p>Let me answer a couple of your specific questions first and then explain<br>
how I look at it.</p>
<aside class="quote no-group" data-username="Ali_Aghababaei" data-post="1" data-topic="130651" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ali_aghababaei/48/61342_2.png" class="avatar"> Ali_Aghababaei:</div>
<blockquote>
<p>it is said that the weight parameter tensor is of size nbatch</p>
</blockquote>
</aside>
<p>In this case the current (1.9.0) documentation for <a href="https://pytorch.org/docs/1.9.0/generated/torch.nn.BCEWithLogitsLoss.html" rel="noopener nofollow ugc">BCEWithLogitsLoss</a><br>
is wrong (or at least misleading).  Quoting:</p>
<blockquote>
<p><strong>weight</strong> (<a href="https://pytorch.org/docs/1.9.0/tensors.html#torch.Tensor" rel="noopener nofollow ugc"><em>Tensor</em></a><em>,</em> <em>optional</em>) – a manual rescaling weight given to the loss of each batch element. If given, has to be a Tensor of size nbatch.</p>
</blockquote>
<p>On the contrary, <code>weight</code> can have other shapes – for example, it can<br>
have the same shape as <code>input</code>.  (As near as I can tell, the precise<br>
requirement is that the shape of <code>weight</code> and <code>input</code> need to be<br>
<a href="https://pytorch.org/docs/1.9.0/notes/broadcasting.html" rel="noopener nofollow ugc">broadcastable</a>.)</p>
<p>The documentation for the functional version,<br>
<a href="https://pytorch.org/docs/1.9.0/generated/torch.nn.functional.binary_cross_entropy_with_logits.html" rel="noopener nofollow ugc">binary_cross_entropy_with_logits()</a>, comes closer to being correct:</p>
<blockquote>
<p><strong>weight</strong> (<a href="https://pytorch.org/docs/1.9.0/tensors.html#torch.Tensor" rel="noopener nofollow ugc"><em>Tensor</em></a><em>,</em> <em>optional</em>) – a manual rescaling weight if provided it’s repeated to match input tensor shape</p>
</blockquote>
<aside class="quote no-group">
<blockquote>
<p>My question is that how the weight parameter tensor is different from the class weights tensor</p>
</blockquote>
</aside>
<p>I think that what you are calling the “class weights tensor” <em>is</em> the<br>
<code>weight</code> tensor.  (Any imagined difference would arise from how<br>
you interpret them based on your use case.)</p>
<p>Please note that <code>BCEWithLogitsLoss</code> takes four tensor arguments:<br>
<code>weight</code>, <code>pos_weight</code>, <code>input</code>, and <code>target</code>.  The first two are passed<br>
in when <code>BCEWithLogitsLoss</code>’s constructor is called to instantiate a<br>
loss-function object, and the second two are passed in when the<br>
resulting loss-function object is called.  There is no separate “class<br>
weights” argument.</p>
<p>Now to explain my understanding:</p>
<p>As far as I can tell, <code>weight</code>, <code>pos_weight</code>, <code>input</code>, and <code>target</code> need<br>
only be broadcastable to one another.  To simplify the discussion, let’s<br>
assume that they are all of the same shape.</p>
<p><code>BCEWithLogitsLoss</code> doesn’t make any distinction, for example,<br>
between labels / predictions for a specific class and specific samples<br>
within a batch.  It simply applies the <code>BCEWithLogitsLoss</code> formula<br>
on an element-wise basis, including the <code>weight</code> and <code>pos_weight</code><br>
weightings, also on an element-wise basis.  This produces a tensor<br>
of element-wise loss values of the same shape as <code>input</code> (and the<br>
other arguments) that is then reduced (or not) according to the value<br>
of <code>reduction</code>.</p>
<p>Consider the following:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; _ = torch.manual_seed (2021)
&gt;&gt;&gt; nBatch = 2
&gt;&gt;&gt; nClass = 3
&gt;&gt;&gt; nSomethingElse = 5
&gt;&gt;&gt; weight = torch.rand (nBatch, nClass, nSomethingElse)
&gt;&gt;&gt; pos_weight = torch.rand (nBatch, nClass, nSomethingElse)
&gt;&gt;&gt; input = torch.randn (nBatch, nClass, nSomethingElse)
&gt;&gt;&gt; target = torch.rand (nBatch, nClass, nSomethingElse)
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (weight = weight, pos_weight = pos_weight) (input, target)
tensor(0.3176)
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (weight = weight.flatten(), pos_weight = pos_weight.flatten()) (input.flatten(), target.flatten())
tensor(0.3176)
</code></pre>
<p><code>BCEWithLogitsLoss</code> doesn’t care about any particular dimensions or<br>
assign them meanings like “batch” or “class” or “height” or “width” – it<br>
simply performs the (weighted) element-wise loss computation and<br>
then reduces the result.</p>
<p>Where does the notion of a “class weights tensor” come from?<br>
Consider a use case where we are performing a multi-label,<br>
<code>nClass</code>-class loss calculation for a batch of <code>nBatch</code> samples:</p>
<p>Let <code>input</code> and <code>target</code> both have shape <code>[nBatch, nClass]</code>.  If <code>weight</code><br>
has shape <code>[nClass]</code> it will be broadcast to match the shape of <code>input</code>,<br>
and, indeed, the elements of <code>weight</code> will be class weights in the loss<br>
calculation – but not because <code>BCEWithLogitsLoss</code> knows or cares<br>
about what you might interpret as a “class” dimension.  Rather, the<br>
elements of <code>weight</code> become class weights just because that’s how<br>
the tensor elements line up in the element-wise computation after<br>
broadcasting.</p>
<p>Note, if you modify this example to try to pass in a 1d tensor of <em>sample</em><br>
weights (of shape <code>[nBatch]</code>), as suggested by the documentation quoted<br>
above, it <em>won’t</em> work.</p>
<p>Thus:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; _ = torch.manual_seed (2021)
&gt;&gt;&gt; nBatch = 2
&gt;&gt;&gt; nClass = 3
&gt;&gt;&gt; sample_weights = torch.rand (nBatch)
&gt;&gt;&gt; class_weights = torch.rand (nClass)
&gt;&gt;&gt; input = torch.randn (nBatch, nClass)
&gt;&gt;&gt; target = torch.rand (nBatch, nClass)
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (weight = class_weights) (input, target)
tensor(0.5677)
&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (weight = sample_weights) (input, target)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;path_to_pytorch&gt;\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "&lt;path_to_pytorch&gt;\torch\nn\modules\loss.py", line 716, in forward
    reduction=self.reduction)
  File "&lt;path_to_pytorch&gt;\torch\nn\functional.py", line 2960, in binary_cross_entropy_with_logits
    return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)
RuntimeError: The size of tensor a (3) must match the size of tensor b (2) at non-singleton dimension 1
</code></pre>
<p>If you <em>want</em> sample weights (rather than class weights) in this use<br>
case, you would have to <code>unsqueeze()</code> <code>sample_weights</code> so that<br>
broadcasting lines the weights up the way you want:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.nn.BCEWithLogitsLoss (weight = sample_weights.unsqueeze (1)) (input, target)
tensor(0.3120)
</code></pre>
<p>One last clarifying comment:</p>
<p>Although we often use <code>BCEWithLogitsLoss</code> with <code>target</code> values<br>
(ground-truth labels) that are binary no-yes labels (expressed as<br>
<code>0.0</code>-<code>1.0</code> floating-point numbers), <code>BCEWithLogitsLoss</code> is more<br>
general and accepts a probabilistic <code>target</code> whose elements are<br>
floating-point values that run from <code>0.0</code> to <code>1.0</code> and represent the<br>
probability that the sample in question is in class-“1”.</p>
<p>Is this more general case we don’t have samples that are purely<br>
“negative” or “positive,” so, strictly speaking, <code>pos_weight</code> doesn’t<br>
weight the “positive” samples.  Rather, it weights the “positive” part of<br>
the binary-cross-entropy formula used for each individual element-wise<br>
loss computation.</p>
<aside class="quote no-group">
<blockquote>
<p>As I said, the targets are in a one-hot coded structure.<br>
For instance, the target [0, 1, 1, 0] means that classes 1 and 2 are present in the corresponding image.</p>
</blockquote>
</aside>
<p>An aside about terminology:  This is not “one-hot” encoding (and, as a<br>
rule of thumb, there’s never really any reason to use one-hot encoding<br>
with pytorch).  You have a multi-label use case and your sample labels<br>
are “multi-hot encoded,” if you will.</p>
<p>The term “one-hot encoding” is often used imprecisely, but doing so<br>
can be quite misleading.  A one-hot encoded single-label class label<br>
is a vector where exactly one element is <code>1</code> and all the others are <code>0</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Ali_Aghababaei
cooked: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a>,</p>
<p>I am deeply grateful for your kind and clear responses and sincerely appreciate your help.</p>
<p>Your comments clarified most of my problems. I understood that as far as pos_weight and weight parameters are broadcastable to inputs and targets, no more strict consideration about their size is required. Actually, what the PyTorch documentation has mentioned about the weight parameter size seems to be wrong!</p>
<p>So, as you said, the tensor given to the weight parameter is the same as class_weights tensor in my case. Actually, final_loss1 and loss2 are the same here:</p>
<pre><code class="lang-auto">num_batch = 2
num_class = 3
targets = torch.randn (num_batch, num_class)
inputs = torch.randn (num_batch, num_class)
class_weights = torch.tensor ([1., 2., 3.])
criterion1 = torch.nn.BCEWithLogitsLoss (reduction = 'none')
criterion2 = torch.nn.BCEWithLogitsLoss (weight = class_weights)

loss1 = criterion1 (inputs, targets)
final_loss1 = (loss1 * class_weights).mean ()

loss2 = criterion2 (inputs, targets)
</code></pre>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="130651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Although we often use <code>BCEWithLogitsLoss</code> with <code>target</code> values<br>
(ground-truth labels) that are binary no-yes labels (expressed as<br>
<code>0.0</code>-<code>1.0</code> floating-point numbers), <code>BCEWithLogitsLoss</code> is more<br>
general and accepts a probabilistic <code>target</code> whose elements are<br>
floating-point values that run from <code>0.0</code> to <code>1.0</code> and represent the<br>
probability that the sample in question is in class-“1”.</p>
<p>Is this more general case we don’t have samples that are purely<br>
“negative” or “positive,” so, strictly speaking, <code>pos_weight</code> doesn’t<br>
weight the “positive” samples. Rather, it weights the “positive” part of<br>
the binary-cross-entropy formula used for each individual element-wise<br>
loss computation.</p>
</blockquote>
</aside>
<p>It was an interesting comment which I have not considered, but I have a question in this regard.<br>
In this more general case of BCEWithLogitsLoss, how can I calculate the pos_weight tensor exactly?</p>
<p>Also, regarding my first post, was the way of calculating the pos_weight tensor in the case of a binary target correct?</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="130651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>An aside about terminology: This is not “one-hot” encoding (and, as a<br>
rule of thumb, there’s never really any reason to use one-hot encoding<br>
with pytorch). You have a multi-label use case and your sample labels<br>
are “multi-hot encoded,” if you will.</p>
<p>The term “one-hot encoding” is often used imprecisely, but doing so<br>
can be quite misleading. A one-hot encoded single-label class label<br>
is a vector where exactly one element is <code>1</code> and all the others are <code>0</code>.</p>
</blockquote>
</aside>
<p>Yes. You are right and I made a mistake here.</p>

username: swathi_guptha
cooked: <aside class="quote no-group" data-username="Ali_Aghababaei" data-post="1" data-topic="130651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ali_aghababaei/48/61342_2.png" class="avatar"> Ali_Aghababaei:</div>
<blockquote>
<p>Imagine that I have a multi-class, multi-label classification problem; my imbalanced one-hot coded dataset includes 1000 images with 4 labels with the following frequencies: class 0: 600, class 1: 550, class 2: 200, class 3: 100.</p>
</blockquote>
</aside>
<p>You mentioned that you have 1000 images, per se, my training data have around 10000 images and validation data have 5000 images.  So would the calculation of pos_weights and class_weight differ for validation and training dataset?</p>
<p>Or the 1000 images refer to the number of samples present in a batch?</p>
<aside class="quote no-group" data-username="Ali_Aghababaei" data-post="3" data-topic="130651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ali_aghababaei/48/61342_2.png" class="avatar"> Ali_Aghababaei:</div>
<blockquote>
<p>Also, regarding my first post, was the way of calculating the pos_weight tensor in the case of a binary target correct?</p>
</blockquote>
</aside>
<p>Did you use any other approach for calculating the pos_weights?</p>

username: ankita.buntolia
cooked: <aside class="quote no-group" data-username="Ali_Aghababaei" data-post="3" data-topic="130651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ali_aghababaei/48/61342_2.png" class="avatar"> Ali_Aghababaei:</div>
<blockquote>
<p>It was an interesting comment which I have not considered, but I have a question in this regard.<br>
In this more general case of BCEWithLogitsLoss, how can I calculate the pos_weight tensor exactly?</p>
</blockquote>
</aside>
<p>Did you find another way to calculate pos_weight or the one you already mentioned is correct?</p>