Multi-Classification and CrossEntropyLoss - interplay of *weight* and *ignore_index*?

Bendrien: <p>Hello there.<br>
First of all, im having a great time playing around with PyTorch - thanks for that <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Now to my question.<br>
Im building a CNN for sequential data using <code>CrossEntropyLoss</code>. I have 3 classes as output of which two are of interest and the last is used if one of the two wasnt a fitt:</p>
<ul>
<li>
<strong>Class 0</strong>: no fitt</li>
<li>
<strong>Class 1</strong>: interest 1</li>
<li>
<strong>Class 2</strong>: interest 2</li>
</ul>
<p>All three classes are exclusive and my labeld training data is highly unbalanced in this order of occurences:  <strong>Class 0</strong> &gt;&gt; <strong>Class 1</strong> &gt; <strong>Class 2</strong>. To compensate for the occurences im computing weights for the <em>weight</em> setting of the <code>CrossEntropyLoss</code>  like this:</p>
<pre><code class="lang-auto">def weights(labels, classes):  # classes = 3
    hist = torch.histc(labels, classes)
    m = hist.min()
    w = torch.tensor([v if v == 0 else m / v for v in hist])
    return w
</code></pre>
<p>After reading the documentation of <code>CrossEntropyLoss</code> again i stumbled over the <em>ignore_index</em> parameter which i havent noticed before. So i was wondering if this would help me but im not quite sure how to use this parameter in conjunction with the weights or to use it at all.</p>
<p>Would it make sense to set the <em>ignore_index</em> parameter to my <strong>Class 0</strong>?<br>
If this is the case do i have to adapt my weight computation to ignore <strong>Class 0</strong>?</p>

KFrank: <p>Hello Anyere!</p>
<aside class="quote no-group quote-modified" data-username="Bendrien" data-post="1" data-topic="68030" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/c67d28/48.png" class="avatar"> Bendrien:</div>
<blockquote>
<p>Im building a CNN for sequential data using <code>CrossEntropyLoss</code>. I have 3 classes as output of which two are of interest and the last is used if one of the two wasnt a fitt:</p>
<ul>
<li><strong>Class 0</strong>: no fitt</li>
<li><strong>Class 1</strong>: interest 1</li>
<li><strong>Class 2</strong>: interest 2</li>
</ul>
<p>…</p>
<p>Would it make sense to set the <em>ignore_index</em> parameter to my <strong>Class 0</strong>?</p>
</blockquote>
</aside>
<p>Unless there is something unusual about your classification<br>
problem, I think you should not use <code>ignore_index</code> for Class 0.</p>
<p>Suppose you feed a Class-0 sample to your model (so <code>target</code><br>
is 0), and your model incorrectly classifies it as Class 2.  You<br>
do want to tell your model not to do this, so you do want this<br>
incorrect classification to add a penalty to your loss function.</p>
<p>If you set <code>ignore_index = 0</code>, this sample (and all other Class-0<br>
samples) will not contribute to the loss function, and so Class-0<br>
misclassifications will not be penalized.</p>
<p>(Because your training data is unbalanced, you may indeed want<br>
to deweight the Class-0 samples in your training set – either by<br>
using <code>CrossEntropyLoss</code>’s <code>weight</code> argument, or by sampling<br>
fewer of your Class-0 samples.  But ignoring them altogether goes<br>
too far and would be hiding import information from your model’s<br>
training.)</p>
<p>Good luck!</p>
<p>K. Frank</p>

Bendrien: <p>Thank you for your answer <a class="mention" href="/u/kfrank">@KFrank</a> ! this was actually my gut feeling but wanted some other opinions.</p>
<p>One further question:<br>
I am recomputing the weights for each minibatch. Does the shown computation of the weights represent the occurences of my labels properly for the <code>CrossEntropyLoss</code>?</p>

KFrank: <p>Hi Anyere!</p>
<aside class="quote no-group" data-username="Bendrien" data-post="3" data-topic="68030" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/c67d28/48.png" class="avatar"> Bendrien:</div>
<blockquote>
<p>I am recomputing the weights for each minibatch. Does the shown computation of the weights represent the occurences of my labels properly for the <code>CrossEntropyLoss</code>?</p>
</blockquote>
</aside>
<p>With one correction, your weight computation looks correct.</p>
<p>Note, that in the case that <code>hist.min() == 0</code> (perfectly possible), all<br>
of your weights will be zero.  I would eliminate <code>m = hist.min()</code>, and<br>
go with <code>1.0 / v</code> (in place of <code>m  / v</code>).</p>
<p>A couple of comments:</p>
<p>I don’t <em>think</em> that the overall normalization of the weights matters.  I<br>
<em>think</em> that if you use <code>reduction = 'mean'</code> in your <code>CrossEntropyLoss</code><br>
(the default), the overall scale factor of your weights will drop out.  (I.e.,<br>
something like <code>weight = 123.4 * weight</code> won’t have any effect.)</p>
<p>I would probably just calculate the weights for the entire training set,<br>
and not bother doing it on a per-batch basis.  (I think both approaches<br>
are reasonable.)  As you take multiple steps in your gradient-descent<br>
optimization algorithm, you are, in a sense, averaging over multiple<br>
batches, so there is nothing magic about reweighting individual batches.</p>
<p>Last, using <code>weight-of-class = 1 / number-of-samples-in-class</code><br>
is perfectly reasonable, and I think somewhat standard.  But other<br>
choices are okay.  You might experiment with only partially reweighting<br>
your classes.</p>
<p>Good luck!</p>
<p>K. Frank</p>

Bendrien: <p>Hey Frank</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="68030">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Note, that in the case that <code>hist.min() == 0</code> (perfectly possible), all<br>
of your weights will be zero. I would eliminate <code>m = hist.min()</code> , and<br>
go with <code>1.0 / v</code> (in place of <code>m  / v</code> ).</p>
</blockquote>
</aside>
<p>I think <code>hist.min()</code> cant be zero since my labels are exclusive. There has to be always exactly one true of all three classes. And for the normalisation I just chose 1 as it is the most common value in those situations.</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="68030">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I would probably just calculate the weights for the entire training set,<br>
and not bother doing it on a per-batch basis.</p>
</blockquote>
</aside>
<p>I was wondering about that too and decided to recalculate on a per batch basis for those cases in which my sequence does not have one class to appear at all. As far as I understand this will make the true meaning of this sequence more clear and granular for the backpropagation.</p>
<p>As a side note, when I’m doing my evalutation on test data I reset all weights to 1, since this would be the normal usecase of the model in the wild - this makes sense, right?</p>
<p>Here is one followup question <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
During training I have sequences of variable length. For not having unnecessary zero-padding within my CNN I want to feed one sequence as a whole to my model. But for backpropagation I thought it would make sense to compute the losses on a fixed window to make the loss computation independent of the incoming sequence length. So i came up with this routine (its a simplified version ignoring the weight computation):</p>
<pre><code class="lang-auto">inputs, labels = sequence_data
outputs = model(inputs)

for l in range(0, len(labels), window_size):
    r = l + window_size
    loss = criterion(outputs[l:r], labels[l:r])
    optimizer.zero_grad()
    retain_graph = r &lt; len(labels)
    loss.backward(retain_graph=retain_graph)
    optimizer.step()
</code></pre>
<p>For this to work I have had to use the <code>retain_graph</code> parameter of the backpropagation and I found the documentation on this feature a bit sparse. Do I use this correct and could someone explain to me whats happing there in more detail and what data will be retained?</p>

Bendrien: <aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="68030">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Note, that in the case that <code>hist.min() == 0</code> (perfectly possible), all<br>
of your weights will be zero. I would eliminate <code>m = hist.min()</code> , and<br>
go with <code>1.0 / v</code> (in place of <code>m  / v</code> ).</p>
</blockquote>
</aside>
<p>I checked the code again and realised this situation could happen which I have not foreseen. Thanks for the headsup <a class="mention" href="/u/kfrank">@KFrank</a>  But since the following is still true</p>
<aside class="quote no-group" data-username="Bendrien" data-post="5" data-topic="68030">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/c67d28/48.png" class="avatar"> Bendrien:</div>
<blockquote>
<p>my labels are exclusive. There has to be always exactly one true of all three classes.</p>
</blockquote>
</aside>
<p>I fixed this issue by removing all zeros before taking the minimum: <code>m = hist[hist != 0].min()</code></p>
<aside class="quote no-group" data-username="Bendrien" data-post="5" data-topic="68030">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/c67d28/48.png" class="avatar"> Bendrien:</div>
<blockquote>
<p>I have had to use the <code>retain_graph</code> parameter of the backpropagation and I found the documentation on this feature a bit sparse. Do I use this correct and could someone explain to me whats happing there in more detail and what data will be retained?</p>
</blockquote>
</aside>
<p>This article got me covered:<br>
<a href="https://towardsdatascience.com/pytorch-autograd-understanding-the-heart-of-pytorchs-magic-2686cd94ec95" class="onebox" target="_blank" rel="noopener nofollow ugc">https://towardsdatascience.com/pytorch-autograd-understanding-the-heart-of-pytorchs-magic-2686cd94ec95</a></p>