How to assign class weights in the loss function in BCEloss

aquorio15: <p>I have a 2d tensor of the shape (batch_size, 500), 500 is no of voice frames taken at a time and each of these frames has two labels i.e. 0 and 1, 0 denoting silence while 1 denotes speech.</p>
<pre><code class="lang-auto">tensor([[0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 1., 0., 0.],
        [0., 0., 0.,  ..., 0., 0., 0.],
        [0., 0., 1.,  ..., 0., 0., 0.]], device='cuda:0')
</code></pre>
<p>After passing the input features into my BiLSTM model this is the output tensor</p>
<pre><code class="lang-auto">tensor([[0.5058, 0.5088, 0.5075,  ..., 0.5071, 0.5079, 0.5057],
        [0.5010, 0.4988, 0.4984,  ..., 0.5046, 0.5041, 0.5022],
        [0.5081, 0.5079, 0.5069,  ..., 0.4985, 0.4982, 0.4992],
        ...,
        [0.5064, 0.5104, 0.5117,  ..., 0.5039, 0.5040, 0.5041],
        [0.5049, 0.5075, 0.5079,  ..., 0.5178, 0.5174, 0.5162],
        [0.4936, 0.4948, 0.4970,  ..., 0.5033, 0.5038, 0.5041]],
       device='cuda:0', grad_fn=&lt;SqueezeBackward0&gt;)
</code></pre>
<p>Now, from the above tensor i am assigning 1 if the value is above 0.5 and 0 otherwise, so that each of the 500 frames has 1 or 0 assigned to it. After that i am calculating the BCEloss and then back propagating. But the issue is the input labels are unbalanced, so i want to assign class_weight to the labels during calculating the BCEloss.</p>
<p>After obtaining the class_weight from compute_class_weight module in sklearn, i am getting class_weights as <code>array([0.59432247, 3.15048184])</code>. But when i pass this as a tensor to the BCEloss i am getting an error.<br>
<code>RuntimeError: The size of tensor a (500) must match the size of tensor b (2) at non-singleton dimension 1</code><br>
This is probably happening because the output tensor size is  (batch_size, 500) while i have class_weights for two labels [0, 1].</p>
<p>Can anyone help me as to how i can assign the class_weights during training. Any kind of help would be greatly appreciated.</p>
<p>Thank You</p>

KFrank: <p>Hi Amartya!</p>
<aside class="quote no-group" data-username="aquorio15" data-post="1" data-topic="183209" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aquorio15/48/56449_2.png" class="avatar"> aquorio15:</div>
<blockquote>
<p>But when i pass this as a tensor to the BCEloss i am getting an error.<br>
<code>RuntimeError: The size of tensor a (500) must match the size of tensor b (2) at non-singleton dimension 1</code></p>
</blockquote>
</aside>
<p>The short story:  Use <code>BCEWithLogitsLoss</code> with <code>pos_weight</code>.</p>
<p>First, you should be using <code>BCEWithLogitsLoss</code> (without a final <code>sigmoid()</code><br>
in your model) for reasons of numerical stability.  Let me first answer your<br>
question in the context of <code>BCEWithLogitsLoss</code>.</p>
<p><code>BCEWithLogitsLoss</code> (unlike <code>BCELoss</code>) takes a <code>pos_weight</code> constructor<br>
argument that multiplies the positive-class contribution to the loss function.<br>
Typically, if you had, say, ten times as many negative samples as positive<br>
samples, your would use, approximately, <code>pos_weight = 10.0</code>.</p>
<p>For <code>BCELoss</code>, on the other hand, you pass in a <code>weight</code> constructor<br>
argument that (for your use case) is the same shape as the output of<br>
your model, with a separate weight value for each of its elements.<br>
So you would build a tensor with <code>1.0</code>s in the positions of the negative<br>
samples and <code>10.0</code>s in the positions of the positive samples.  You would<br>
have to build this <code>weight</code> tensor separately for each batch and use it<br>
to construct a new <code>BCELoss</code> object for each batch.</p>
<p>Best.</p>
<p>K. Frank</p>

aquorio15: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a> thank you for your reply, Can you kindly tell me how can i create the weight tensor in case of <code>BCEWithLogitsLoss</code>. While using BCEloss i was calculating the class_weight as you rightly mentioned for every batch using <code>sklearn.utils.class_weight.compute_class_weight</code> module. Shall i do the same for <code>BCEWithLogitsLoss</code>.</p>

KFrank: <p>Hi Amartya!</p>
<aside class="quote no-group" data-username="aquorio15" data-post="3" data-topic="183209" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aquorio15/48/56449_2.png" class="avatar"> aquorio15:</div>
<blockquote>
<p>Can you kindly tell me how can i create the weight tensor in case of <code>BCEWithLogitsLoss</code>.</p>
</blockquote>
</aside>
<p>I don’t fully understand your use case.</p>
<p>In the typical use case, <code>BCEWithLogitsLoss</code> takes (as does <code>BCELoss</code>)<br>
an <code>input</code> argument that is the predictions output by your model and a<br>
<code>target</code> argument that is your ground-truth labels.</p>
<p><code>input</code> and <code>target</code> are the same shape; for example, they could be batches<br>
of 500-frame time series.  <code>target</code> typically consists of the values <code>0.0</code> (which<br>
would be the label for the “negative”-class samples) and <code>1.0</code> (the label for<br>
the “positive” class) (but <code>target</code> can also contain “probabilistic” labels whose<br>
values range from <code>0.0</code> to <code>1.0</code> and represent the probability that the item in<br>
question belongs to the “positive” class).</p>
<p>In the typical use case, <code>pos_weight</code> is a single value (but packaged as a<br>
tensor, e.g., of shape <code>[1]</code>).  The <em>exact</em> value of <code>pos_weight</code> shouldn’t<br>
matter, but you typically want it to be inversely proportional to the number<br>
of “positive” samples.</p>
<p>I recommend counting the number of “positive”-class and “negative”-class<br>
labels in your training set (or in a representative subset of your training<br>
set) and setting <code>pos_weight = num_negative / num_positive</code>.  The<br>
idea is that if you don’t have very many “positive” samples in your training<br>
set, you want to make up for it by weighting the “positive” samples more<br>
heavily in your loss function.</p>
<p>(People sometimes compute <code>pos_weight</code> on a per-batch basis, but I think<br>
that it’s preferable to use the same <code>pos_weight</code> for all the batches.)</p>
<p>As an aside:  You posted above:</p>
<aside class="quote no-group" data-username="aquorio15" data-post="1" data-topic="183209">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aquorio15/48/56449_2.png" class="avatar"> aquorio15:</div>
<blockquote>
<p>Now, from the above tensor i am assigning 1 if the value is above 0.5 and 0 otherwise, so that each of the 500 frames has 1 or 0 assigned to it. After that i am calculating the BCEloss and then back propagating.</p>
</blockquote>
</aside>
<p>I’m not certain what you mean by this, but if you threshold the output of your<br>
model and then pass those thresholded values to your loss function, you<br>
<em>won’t</em> be able to (usefully) backpropagate through the thresholding step,<br>
your gradients will be zero, and your model won’t train.</p>
<p>Best.</p>
<p>K. Frank</p>

aquorio15: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a></p>
<p>Thanks for the swift reply, I am basically getting a distribution from the output of my model, where the last layer is a sigmoid.</p>
<pre><code class="lang-auto">tensor([[0.5058, 0.5088, 0.5075,  ..., 0.5071, 0.5079, 0.5057],
        [0.5010, 0.4988, 0.4984,  ..., 0.5046, 0.5041, 0.5022],
        [0.5081, 0.5079, 0.5069,  ..., 0.4985, 0.4982, 0.4992],
        ...,
        [0.5064, 0.5104, 0.5117,  ..., 0.5039, 0.5040, 0.5041],
        [0.5049, 0.5075, 0.5079,  ..., 0.5178, 0.5174, 0.5162],
        [0.4936, 0.4948, 0.4970,  ..., 0.5033, 0.5038, 0.5041]],
       device='cuda:0', grad_fn=&lt;SqueezeBackward0&gt;)
</code></pre>
<p>But as my original labels consists of positive (1) and negative samples (0), i wanted to calculate the accuracy between the original and predicted labels, that’s why i used a threshold to convert the distribution into 0’s and 1’s. I am not using the threshold to calculate the loss.</p>
<p>I am very sorry if I was unclear before.</p>
<p>Thank You</p>