How weights are being used in Cross Entropy Loss

shakeel608: <p>Since I checked the doc and the explanation from <a href="https://discuss.pytorch.org/t/weight-in-cross-entropy-loss/78256">weights in CE</a><br>
But When I was checking it for more than two samples, it is showing different results as below<br>
For below snippet</p>
<pre><code class="lang-auto">inp = tensor([[0.9860, 0.1934],
        [0.9590, 0.3538],
        [0.1502, 0.9544],
        [0.7666, 0.0535],
        [0.1600, 0.3133],
        [0.1827, 0.8578],
        [0.2727, 0.7105],
        [0.3965, 0.0156]])

target = tensor([1, 1, 1, 0, 0, 0, 1, 1])

cl_wts = 1./torch.tensor([5., 3.])

loss = nn.CrossEntropyLoss()
loss_weighted = loss = nn.CrossEntropyLoss(weight = cl_wts)

l1 = loss(inp, target)
print(l1)---&gt; tensor(0.7793)

l_wt = loss_weighted(inp, target)
print(l_wt) ---&gt; tensor(0.7839)

When I was checking it manually as 

logits = Softmax(inp)
:point_down:
logits = tensor([[0.6884, 0.3116],
        [0.6469, 0.3531],
        [0.3091, 0.6909],
        [0.6711, 0.3289],
        [0.4617, 0.5383],
        [0.3374, 0.6626],
        [0.3923, 0.6077],
        [0.5941, 0.4059]])
 
manual_loss1 =  -(np.log(0.3116) + np.log(0.3531) + np.log(0.6909) + np.log(0.6711) + np.log(0.4617) + np.log(0.3374) + np.log(0.6077) + np.log(0.4059))
manual_loss = manual_loss/8  ---&gt;8 is because of mini batch size
print(manual_loss) ---&gt; 0.7793355874570308, which is equivalent to l1

However for weighted 
man_loss_weighted = -(np.log(0.3116)*0.2 + np.log(0.3531)*0.2 + np.log(0.6909)*0.2 + np.log(0.6711)*0.33 + np.log(0.4617)*0.33 + np.log(0.3374)*0.33 + np.log(0.6077)*0.2 + np.log(0.4059)*0.2)/(0.2+0.33)
man_loss_weighted /=8
print(man_loss_weighted)---&gt; 0.3633250361678566   
Which is not equivalent to l2 weighted loss, 

How is it being computed. Any help would be appreciated
Thank you 

</code></pre>

KFrank: <p>Hi Shakeel!</p>
<aside class="quote no-group" data-username="shakeel608" data-post="1" data-topic="122627" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a587f6/48.png" class="avatar"> shakeel608:</div>
<blockquote>
<pre><code class="lang-auto">man_loss_weighted = -(np.log(0.3116)*0.2 + np.log(0.3531)*0.2 + np.log(0.6909)*0.2 + np.log(0.6711)*0.33 + np.log(0.4617)*0.33 + np.log(0.3374)*0.33 + np.log(0.6077)*0.2 + np.log(0.4059)*0.2)/(0.2+0.33)
man_loss_weighted /=8
</code></pre>
</blockquote>
</aside>
<p>You have two errors in your computation of <code>man_loss_weighted</code>:</p>
<p>First you need to divide by the sum of the weights used for each<br>
individual sample.*</p>
<p>Second, you have mixed up your class-0 and class-1 weights.</p>
<p>Here is the correct manual computation:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; -(np.log(0.3116)*0.333333 + np.log(0.3531)*0.333333 + np.log(0.6909)*0.333333 + np.log(0.6711)*0.2 + np.log(0.4617)*0.2 + np.log(0.3374)*0.2 + np.log(0.6077)*0.333333 + np.log(0.4059)*0.333333)/(0.333333 + 0.333333 + 0.333333 + 0.2 + 0.2 + 0.2 + 0.333333 + 0.333333)
0.784032260475451
</code></pre>
<p>*)  You have divided first by <code>cl_wts[0] + cl_wts[1]</code>.  But you need<br>
to divide by the actual weights used for each sample in the batch.<br>
Suppose your batch contained only class-0 samples. In such a case<br>
it wouldn’t wouldn’t make sense to use <code>cl_wts[1]</code> in the computation<br>
at all.  Then you divide by <code>8</code>.  But suppose that all of your weights<br>
were <code>1</code>.  You would first divide by the sum of those weights, which<br>
would be <code>8</code>, and then you would divide by <code>8</code> again, which would be<br>
wrong.</p>
<p>Best.</p>
<p>K. Frank</p>

shakeel608: <p>Thank you <a class="mention" href="/u/kfrank">@KFrank</a><br>
but since the weight tensor is</p>
<pre><code class="lang-auto">cl_wts = 1./torch.tensor([5., 3.]) = tensor([0.2, 0.333])

</code></pre>
<p>Why are you multiplying samples with label =1 by 0.333 (Doesn’t this mean we are giving it more importance , since it is already higher than other class.  Shouldn’t we multiply the samples with lower class with more weights [in our case 0.333 is more and it should be multiplied to lower class and 0.2 for higher class] )</p>
<p>Could you please clarify it</p>
<p>One more thing when i pass the weights</p>
<pre><code class="lang-auto">tensor([0.2, 0.333])

</code></pre>
<p>does it work like the class_0 will get weight 0.2 and class_1 will get 0.333 and so on ??</p>

KFrank: <p>Hi Shakeel!</p>
<aside class="quote no-group" data-username="shakeel608" data-post="3" data-topic="122627" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a587f6/48.png" class="avatar"> shakeel608:</div>
<blockquote>
<p>weight tensor is</p>
<pre><code class="lang-auto">cl_wts = 1./torch.tensor([5., 3.]) = tensor([0.2, 0.333])

</code></pre>
<p>Why are you multiplying samples with label =1 by 0.333</p>
</blockquote>
</aside>
<p>I’m just using the weight tensor you specified.</p>
<aside class="quote no-group">
<blockquote>
<p>(Doesn’t this mean we are giving it more importance , since it is already higher than other class.  Shouldn’t we multiply the samples with lower class with more weights [in our case 0.333 is more and it should be multiplied to lower class and 0.2 for higher class] )</p>
</blockquote>
</aside>
<p>Yes, this would be the typical approach.  Note, however, I wouldn’t<br>
reweight a batch of size 8 with the counts of the classes in that batch.<br>
I would typically weight my classes based on the (approximate) class<br>
counts in my whole training set (and I wouldn’t bother reweighting<br>
unless the classes were much more imbalanced than 5-to-3).</p>
<aside class="quote no-group">
<blockquote>
<p>One more thing when i pass the weights</p>
<pre><code class="lang-auto">tensor([0.2, 0.333])
</code></pre>
<p>does it work like the class_0 will get weight 0.2 and class_1 will get 0.333 and so on ??</p>
</blockquote>
</aside>
<p>Yes, that is correct.  That is what I meant when I said that “you mixed<br>
up your class-0 and class-1 weights.”</p>
<p>Best.</p>
<p>K. Frank</p>

shakeel608: <p><a class="mention" href="/u/kfrank">@KFrank</a><br>
Thank you this makes sense now</p>

shakeel608: <p><a class="mention" href="/u/kfrank">@KFrank</a><br>
How does weights effect in propagation so that we can be sure model is focusing on the minority class</p>

shakeel608: <p>How does weights effect in back propagation so that we can be sure model is focusing on the minority class<br>
Does it boosts the gradient or the it increases the number of updates. Could you please clarify this</p>

KFrank: <p>Hi Shakeel!</p>
<aside class="quote no-group" data-username="shakeel608" data-post="7" data-topic="122627" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/a587f6/48.png" class="avatar"> shakeel608:</div>
<blockquote>
<p>Does it boosts the gradient or the it increases the number of updates.</p>
</blockquote>
</aside>
<p>I suggest that you try a quick test.</p>
<p>Don’t use a model.  Just create <code>pred</code> with <code>requires_grad = True</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">pred = torch.randn (10, 2, requires_grad = True)
</code></pre>
<p>Then create some target class labels:</p>
<pre data-code-wrap="python"><code class="lang-python">targ = torch.randint (2, (10, ))
</code></pre>
<p>Then calculate and backpropagate <code>CrossEntropyLoss</code> with<br>
various choices for <code>weight</code> and see what happens:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.nn.CrossEntropyLoss (weight = my_class_weights) (pred, targ).backward()
print (pred.grad)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>