How to use the 'weight' -- loss functions parameter

username: shirui-japina
cooked: <p>My understanding is that:</p>
<ul>
<li>
<p>In fact, when it’s the default value (None), the parameter <em><strong>weight</strong></em> is a vector which is full of (1,). I mean, <em><strong>weight</strong></em> = (1, 1, …, 1). The size of vector equals the number of batch-size.</p>
</li>
<li>
<p>Assume the size of batch is 4, and I set the <em><strong>weight</strong></em> = (w1, w2, w3, w4).<br>
Then the value of loss function:<br>
Loss = (loss_sample_1 * w1 +  loss_sample_2 * w2 + loss_sample_3 * w3 + loss_sample_4 * w4) / (w_1 + w_2 + w_3 + w_4)<br>
Here, each loss sample is also a vector, with size of number of output nodes.</p>
</li>
<li>
<p>Actually, it adjusts the size of the learning rate for each sample.</p>
</li>
<li>
<p>So the larger the value of <em><strong>weight</strong></em> parameter, the greater the impact to the Loss.</p>
</li>
<li>
<p>It is generally used when the size of various data sets is not balanced.</p>
</li>
</ul>
<p>Is this the right understanding?<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"><br>
I tried to see the source code, but I could not find it.<img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=9" title=":flushed:" class="emoji" alt=":flushed:"></p>

username: albanD
cooked: <p>Hi,</p>
<p>The <code>weight</code> parameter is usually with respect to labels, not batch samples !<br>
It will apply the weight depending on the ground truth label for the given sample.<br>
It will indeed rescale the gradients and is very useful if your dataset is unbalanced for example.</p>

username: shirui-japina
cooked: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="58206" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>The <code>weight</code> parameter is usually with respect to labels, not batch samples !</p>
</blockquote>
</aside>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=12" title=":flushed:" class="emoji" alt=":flushed:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=12" title=":flushed:" class="emoji" alt=":flushed:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=12" title=":flushed:" class="emoji" alt=":flushed:" loading="lazy" width="20" height="20">Oh~~~<br>
Like <a href="https://pytorch.org/docs/master/_modules/torch/nn/functional.html#cross_entropy" rel="noopener nofollow ugc">here (def cross_entropy)</a> said,</p>
<blockquote>
<p>weight (Tensor, optional): a manual rescaling weight given to each class. If given, has to be a Tensor of size <code>C</code></p>
</blockquote>
<p>Providing the output node is size of (100,), then I should set the parameter <em><strong>weight</strong></em> with shape of (100,).<br>
This works like:</p>
<p><em><strong>label vector = (</strong></em><br>
<em><strong>label_1_new = label_1_old * weight_1,</strong></em><br>
<em><strong>label_2_new = label_2_old * weight_2,</strong></em><br>
<em><strong>…</strong></em><br>
<em><strong>label_100_new = label_100_old * weight_100,</strong></em><br>
<em><strong>)</strong></em></p>
<p>In fact, it just adjusts the size of the label.<br>
Maybe I can adjust the size of the label directly, instead of using this parameter?<br>
Is this understanding right?<img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=12" title=":flushed:" class="emoji" alt=":flushed:" loading="lazy" width="20" height="20"></p>
<p>But <a href="https://pytorch.org/docs/master/_modules/torch/nn/functional.html#binary_cross_entropy" rel="noopener nofollow ugc">here (binary_cross_entropy)</a> said,</p>
<blockquote>
<p>weight (Tensor, optional): a manual rescaling weight if provided it’s repeated to match input tensor shape</p>
</blockquote>
<p>Why the focus here is turned into input, but not label?</p>

username: KFrank
cooked: <p>Hi Shirui!</p>
<aside class="quote no-group" data-username="shirui-japina" data-post="1" data-topic="58206" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shirui-japina/48/15285_2.png" class="avatar"> shirui-japina:</div>
<blockquote>
<p>My understanding is that:</p>
<ul>
<li>In fact, when it’s the default value (None), the parameter  <em><strong>weight</strong></em>  is a vector which is full of (1,). I mean,  <em><strong>weight</strong></em>  = (1, 1, …, 1). The size of vector equals the number of batch-size.</li>
</ul>
</blockquote>
</aside>
<p>Yes, this is correct, but <em>specifically for</em> <code>binary_cross_entropy</code><br>
(and <code>binary_cross_entropy_with_logits</code>).  As Alban pointed<br>
out it is <em>not</em> true for <code>CrossEntropyLoss</code>.</p>
<p>(As an aside, my guess is that when the <code>weight</code> parameter is<br>
<code>None</code>, there isn’t actually a weight vector of ones.  I assume<br>
that no multiplication by (default) weights occurs.  Of course,<br>
not multiplying by anything is equivalent to multiplying by one,<br>
so the effect of not multiplying by anything and multiplying by<br>
a default weight vector of ones is the same.)</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>Assume the size of batch is 4, and I set the  <em><strong>weight</strong></em>  = (w1, w2, w3, w4).<br>
Then the value of loss function:<br>
Loss = (loss_sample_1 * w1 + loss_sample_2 * w2 + loss_sample_3 * w3 + loss_sample_4 * w4) / (w_1 + w_2 + w_3 + w_4)<br>
Here, each loss sample is also a vector, with size of number of output nodes.</li>
</ul>
</blockquote>
</aside>
<p>If I understand what you’re saying, this isn’t quite right.</p>
<p>I’m assuming that by <code>loss_sample</code> you mean the (batch of<br>
four) output vectors of your network.  (These would often be<br>
called <em>predictions</em>.)  The loss is a measure of how much<br>
your predictions differ from your “ground truth,” that is, your<br>
<em>targets</em>.  I will use the term <code>predict</code> rather than <code>loss_sample</code>.</p>
<p>Then your weighted loss is:</p>
<pre><code class="lang-plaintext">loss = loss (predict_1, target_1) * w1 + loss (predict_2, target_2) * w2 + loss (predict_3, target_3) * w3 + loss (predict_4, target_4) * w4
</code></pre>
<aside class="quote no-group">
<blockquote>
<ul>
<li>Actually, it adjusts the size of the learning rate for each sample.</li>
</ul>
</blockquote>
</aside>
<p>Yes, you could say this (but I don’t think I would phrase it this way).</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>So the larger the value of  <em><strong>weight</strong></em>  parameter, the greater the impact to the Loss.</li>
</ul>
</blockquote>
</aside>
<p>Yes.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>It is generally used when the size of various data sets is not balanced.</li>
</ul>
</blockquote>
</aside>
<p>Well, this would depend on what you mean by “various data sets.”</p>
<p>If you are training a classifier, and your training dataset has<br>
many more samples for one class than another – that is, it is<br>
<em>unbalanced</em> – then yes, weights are often used to increase<br>
the contribution of underrepresented classes to the loss (and<br>
hence to the gradients used for training).</p>
<aside class="quote no-group">
<blockquote>
<p>Is this the right understanding?<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>Basically yes, <em>for weights that are applied to samples</em>, such as<br>
with <code>binary_cross_entropy</code> (but not exactly right for class<br>
weights, such as with <code>CrossEntropyLoss</code>).</p>
<p>[qoute]<br>
I tried to see the source code, but I could not find it.<img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=12" title=":flushed:" class="emoji" alt=":flushed:" loading="lazy" width="20" height="20"><br>
[/quote]</p>
<p>Here are some source links:</p>
<p><a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#BCEWithLogitsLoss" rel="noopener nofollow ugc">BCEWithLogitsLoss</a><br>
<a href="https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#binary_cross_entropy_with_logits" rel="noopener nofollow ugc">binary_cross_entropy_with_logits</a></p>
<p><a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#BCELoss" rel="noopener nofollow ugc">BCELoss</a><br>
<a href="https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#binary_cross_entropy" rel="noopener nofollow ugc">binary_cross_entropy</a></p>
<p><a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/loss.html#CrossEntropyLoss" rel="noopener nofollow ugc">CrossEntropyLoss</a><br>
<a href="https://pytorch.org/docs/stable/_modules/torch/nn/functional.html#cross_entropy" rel="noopener nofollow ugc">cross_entropy</a></p>
<aside class="quote no-group quote-modified" data-username="shirui-japina" data-post="3" data-topic="58206" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shirui-japina/48/15285_2.png" class="avatar"> shirui-japina:</div>
<blockquote>
<aside class="quote no-group" data-username="albanD" data-post="2" data-topic="58206" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>The <code>weight</code> parameter is usually with respect to labels, not batch samples !</p>
</blockquote>
</aside>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=12" title=":flushed:" class="emoji" alt=":flushed:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=12" title=":flushed:" class="emoji" alt=":flushed:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=12" title=":flushed:" class="emoji" alt=":flushed:" loading="lazy" width="20" height="20">Oh~~~<br>
Like <a href="https://pytorch.org/docs/master/_modules/torch/nn/functional.html#cross_entropy" rel="noopener nofollow ugc">here (def cross_entropy)</a> said,</p>
<blockquote>
<p>weight (Tensor, optional): a manual rescaling weight given to each class. If given, has to be a Tensor of size <code>C</code></p>
</blockquote>
</blockquote>
</aside>
<p>Yes.  <code>CrossEntropyLoss</code> (and the function form, <code>cross_entropy</code>,<br>
you linked to) takes <em>class</em> weights (not <em>sample</em> weights).</p>
<aside class="quote no-group">
<blockquote>
<p>Providing the output node is size of (100,), then I should set the parameter <em><strong>weight</strong></em> with shape of (100,).</p>
</blockquote>
</aside>
<p>Yes, <em>because having an “output node” of size 100 implies that<br>
you are building a 100-class classifier, so your class weights are<br>
then a vector of 100 weights.</em></p>
<aside class="quote no-group">
<blockquote>
<p>This works like:</p>
<p><em><strong>label vector = (</strong></em><br>
<em><strong>label_1_new = label_1_old * weight_1,</strong></em><br>
<em><strong>label_2_new = label_2_old * weight_2,</strong></em><br>
<em><strong>…</strong></em><br>
<em><strong>label_100_new = label_100_old * weight_100,</strong></em><br>
<em><strong>)</strong></em></p>
</blockquote>
</aside>
<p>No.  When you say <em><strong>label_vector</strong></em>, I assume you mean the<br>
known, ground truth class labels for your training data.  These<br>
are what are often called the <em>targets.</em>  For <code>CrossEntropyLoss</code>,<br>
they have to be integer class labels in <code>[0, nClass - 1]</code>.<br>
If you multiply them by weights, they won’t be class labels<br>
anymore.</p>
<p>Let <code>predict_1</code> be the first of four prediction vectors (output<br>
of your network) for your example with a batch size of four.<br>
<code>target_1</code> is the first class label in your batch of four.  You say<br>
that your output is a vector of length 100, so this means that<br>
you have 100 classes, so your class labels are in <code>[0, 99]</code>,<br>
and your weight vector is of length 100.</p>
<p>Then, for example, <code>weight[17]</code> is the weight for class 17.</p>
<p>Now, because these are class weights instead of sample weights,<br>
we have for your example with a batch size of four:</p>
<pre><code class="lang-auto">loss = weight[target_1] * loss (predict_1, target_1) + weight[target_2] * loss (predict_2, target_2) + weight[target_3] * loss (predict_3, target_3) + weight[target_4] * loss (predict_4, target_4)
</code></pre>
<p>Remember, your targets are class-label integers in <code>[0, 99]</code>.  So<br>
they are valid indices into the weight vector of length 100.</p>
<aside class="quote no-group">
<blockquote>
<p>In fact, it just adjusts the size of the label.</p>
</blockquote>
</aside>
<p>Yes.  I would probably phrase it as “it adjusts the contribution<br>
to the loss of those samples with the specified label.”</p>
<aside class="quote no-group">
<blockquote>
<p>Maybe I can adjust the size of the label directly, instead of using this parameter?</p>
</blockquote>
</aside>
<p>You can’t.  Again, the labels themselves are integer class labels.<br>
They don’t really have sizes, and you can’t change them (without<br>
breaking things).</p>
<p>(If, for some reason, you don’t want to use the <code>weight</code> parameter<br>
to accomplish this reweighting, you would have to write your own<br>
custom loss function.)</p>
<aside class="quote no-group">
<blockquote>
<p>But <a href="https://pytorch.org/docs/master/_modules/torch/nn/functional.html#binary_cross_entropy" rel="noopener nofollow ugc">here (binary_cross_entropy)</a> said,</p>
<blockquote>
<p>weight (Tensor, optional): a manual rescaling weight if provided it’s repeated to match input tensor shape</p>
</blockquote>
<p>Why the focus here is turned into input, but not label?</p>
</blockquote>
</aside>
<p>I don’t know why the weight parameter is used differently in<br>
<code>binary_cross_entropy</code> and <code>CrossEntropyLoss</code>.  It is<br>
somewhat inconsistent.</p>
<p>There are legitimate use cases for both sample weights and<br>
class weights.</p>
<p>I tend to think the class-weight use case is more common,<br>
even for a binary-classification problem.  You can certainly<br>
have unbalanced training data in the binary case, and it<br>
would be convenient to be able to pass a weight parameter<br>
to the binary loss function that reweights your class-0 samples<br>
relative to your class-1 samples.  (You can do this, of course,<br>
using sample weights, but for every batch, you have to construct<br>
your own sample-weight vector (of length batch-size) based on<br>
the class (“0” or “1”) of each sample.  This is straightforward,<br>
but a bit of a nuisance.)</p>
<p>Note that sample weights are more general than class weights,<br>
because, as outlined above, you can use sample weights to<br>
weight samples based on their class.  So this mechanism lets<br>
you use class weights with <code>BCELoss</code>.  In contrast, there is<br>
no way to get <em>sample</em> weights using <code>CrossEntropyLoss</code><br>
(other than writing a custom loss function).</p>
<p>Class weights (if that’s what you want) are more convenient.<br>
You only need one weight vector that you can reuse for all<br>
of your batches.</p>
<p>You can pass in a weight vector when constructing a<br>
<code>CrossEntropyLoss</code> loss-function object.  This is convenient<br>
and makes sense because you will typically use the same<br>
class-weight weight vector for all of your batches.</p>
<p>You can also pass in a weight vector when constructing a<br>
<code>BCELoss</code> (or <code>BCEWithLogitsLoss</code>).  This is less likely to<br>
make sense, because you are not so likely to want to use<br>
the same <em>sample-weight</em> weight vector for each batch.<br>
(And you can’t if the batch sizes are not the same.)  So<br>
if you want to use weights for binary classification, you<br>
will generally have to use the function form of the loss,<br>
<code>binary_cross_entropy</code> (or<br>
<code>binary_cross_entropy_with_logits</code>).</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: shirui-japina
cooked: <p>Thank you so much…even words cannot express how thankful I am…<img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=9" title=":flushed:" class="emoji" alt=":flushed:"><img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>