Passing custom weights to cross entropy loss

olisp: <p>I am trying to assign different weights to different classes, so I have modified my loss criterion as such:</p>
<p>I had to convert the weight tensor to double <code>torch.DoubleTensor(weight)</code> since my <code>model</code> is already moved to <code>double()</code>.</p>
<p>Am I doing this correctly ?</p>
<pre><code>weights = [0.4,0.8,1.0]

class_weights = torch.DoubleTensor(weights).cuda()

criterion = nn.CrossEntropyLoss(weight=class_weights)</code></pre>

ptrblck: <p>This should be correct, yes. Are you seeing any issues using this code?</p>

olisp: <p>thanks, I just wanted to confirm, the code works just fine.</p>

jrepo: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, related to this question:</p>
<p>How can we be sure that weights are mapped to the proper class? During training, the dataset will be in batch (and shuffled). Should I map weights according to how the model is seeing each label in each batch?</p>
<p>I’m asking that because I would be much more comfortable passing a dictionary of weights to the loss function, instead of a list</p>

InnovArul: <p>In my understanding, classes are assigned with integers (from 0 to <code>num_classes</code>).<br>
Following this, weights will be a list of <code>num_classes</code>, with <code>weights[i]</code> corresponding to weight of <code>i</code>th class.</p>

jrepo: <p>Thanks for your time.</p>
<p>So if I understood correctly, if I have a situation like this:</p>
<pre><code class="lang-auto">labels = [2, 2, 1, 3]
</code></pre>
<p>I calculate weights</p>
<pre><code class="lang-auto">weights = [0.2, 0.2, 0.4, 0.4]
</code></pre>
<p>but to the loss functions I should pass them as</p>
<pre><code class="lang-auto">weights = [0.4, 0.2, 0.4, 0.4]
</code></pre>
<p>because classes are assigned from 0 to num_classes, so  <code>[1, 2, 2, 3]</code></p>
<p>Am I right?</p>

InnovArul: <aside class="quote no-group" data-username="jrepo" data-post="6" data-topic="81393">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jrepo/48/29813_2.png" class="avatar"> jrepo:</div>
<blockquote>
<p><code>weights = [0.2, 0.2, 0.4, 0.4]</code></p>
</blockquote>
</aside>
<p>Can you elaborate a bit on how you calculate the weights?</p>
<aside class="quote no-group" data-username="jrepo" data-post="6" data-topic="81393">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jrepo/48/29813_2.png" class="avatar"> jrepo:</div>
<blockquote>
<pre><code class="lang-auto">weights = [0.2, 0.2, 0.4, 0.4]
</code></pre>
<p>but to the loss functions I should pass them as</p>
<pre><code class="lang-auto">weights = [0.4, 0.2, 0.4, 0.4]
</code></pre>
</blockquote>
</aside>
<p>Also, both the <code>weights</code> lists are different here.</p>
<p>Basically, the <code>weights</code> list has length = number of total classes. (Not the number of samples)<br>
<code>weights[i]</code> = weight calculated for class <code>i</code>.<br>
I am not sure how your example relates to this definition. Maybe I am a bit confused with your example.</p>

jrepo: <p>You are totally right, let me explain better now (and correct my mistakes)</p>
<p>For example, image I have a dataframe <code>df</code> like this</p>
<pre><code class="lang-auto">| var | label
|  a  | 2	
|  b  | 2	
|  c  | 1	
|  d  | 3	
</code></pre>
<p>In the example before, I calculated the class weights as</p>
<pre><code class="lang-auto">w = 1 / (df.label.value_counts()/df.label.value_counts().sum())
w = w/w.sum()
</code></pre>
<p>which leads to this class weight (you are right, number total class not number of samples)</p>
<pre><code class="lang-auto">2    0.2
3    0.4
1    0.4
</code></pre>
<p>Now, when I pass <code>w</code> to my loss function through <code>torch.from_numpy(w.values)</code>, should I first order it so that it reflects the order of the classes (1, 2, 3), therefore <code>[0.4, 0.2, 0.4]</code>? Or should I pass it as-is, therefore <code>[0.2, 0.4, 0.4]</code>?</p>
<p>Hope it’s clearer now, sorry about before!</p>

InnovArul: <aside class="quote no-group" data-username="jrepo" data-post="8" data-topic="81393">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jrepo/48/29813_2.png" class="avatar"> jrepo:</div>
<blockquote>
<p>should I first order it so that it reflects the order of the classes (1, 2, 3), therefore <code>[0.4, 0.2, 0.4]</code>?</p>
</blockquote>
</aside>
<p>Thanks for explaining!<br>
Assuming that you are mapping the classes [1,2,3] into pytorch equivalent 0-based classes of [0,1,2], you have to pass <code>[0.4, 0.2, 0.4]</code> as you mentioned.</p>