Label smoothing for only a subset of classes

username: macazinc
cooked: <p>In the standard label smoothing regime, label smoothing is applied to every training example. I have a real world dataset where there is some label noise between 2 of the 30 classes I’m training the classifier on. Basically, two of the classes may be mislabeled as one another. I was hoping to try to use label smoothing as a way to compensate for some of the noise in these labels.</p>
<p>Does anyone have an idea of how to implement this efficiently?</p>

username: KFrank
cooked: <p>Hi macazinc!</p>
<aside class="quote no-group" data-username="macazinc" data-post="1" data-topic="114208" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/bbe5ce/48.png" class="avatar"> macazinc:</div>
<blockquote>
<p>Basically, two of the classes may be mislabeled as one another. I was hoping to try to use label smoothing as a way to compensate for some of the noise in these labels.</p>
</blockquote>
</aside>
<p>Here’s what I think you’re asking:</p>
<p>You have a multi-class (30 classes) classification problem.  You know<br>
that for most of your classes, your ground-truth <code>target</code> labels are<br>
correct, but your labels sometimes mix up two of your classes, say <code>4</code><br>
and <code>9</code>,  Let’s say that a sample labelled <code>4</code> is actually a <code>9</code> 25% of the<br>
time, and that a sample labelled <code>9</code> is actually a <code>4</code> 10% of the time.</p>
<p>You will (most likely) want to use cross-entropy loss, but pytorch only<br>
provides a version that takes integer categorical class labels for its<br>
<code>target</code>.  In your case, you want what I call <em>soft labels,</em> and will have<br>
to write your own soft-label version of cross-entropy.  See this post<br>
for an implementation:</p>
<aside class="quote quote-modified" data-post="2" data-topic="69501">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/soft-cross-entropy-loss-tf-has-it-does-pytorch-have-it/69501/2">Soft Cross Entropy Loss (TF has it does Pytorch have it)</a> 
  </div>
  <blockquote>
    Hello Raaj! 

I do not believe that pytorch has a “soft” cross-entropy function built in. 
But you can implement it using pytorch tensor operations, so you should 
get the full benefit of autograd and gpu acceleration. 
See this (pytorch version 0.3.0) script: 
import torch
torch.__version__

# define "soft" cross-entropy with pytorch tensor operations
def softXEnt (input, target):
    logprobs = torch.nn.functional.log_softmax (input, dim = 1)
    return  -(target * logprobs).sum() / input.shap…
  </blockquote>
</aside>

<p>Now let me assume that your (sometimes incorrect) <code>target</code> labels<br>
are given as integer categorical labels.  First use <a href="https://pytorch.org/docs/1.7.1/nn.functional.html#torch.nn.functional.one_hot" rel="noopener nofollow ugc">one_hot()</a> (followed<br>
by <a href="https://pytorch.org/docs/1.7.1/tensors.html#torch.Tensor.float" rel="noopener nofollow ugc">float()</a>) to convert your categorical labels into soft labels (that all<br>
happen to be zero or one).  Then whenever a sample is labelled <code>4</code><br>
(<code>target[i, 4] == 1.0</code>), set <code>target[i, 4] = 0.75</code> and<br>
<code>target[i, 9] = 0.25</code>.  Similarly, when <code>target[i, 9] == 1.0</code>, set<br>
<code>target[i, 9] = 0.90</code>, and <code>target[i, 4] = 0.10</code>.</p>
<p>You then feed the soft-label <code>target</code> you constructed into the soft-label<br>
cross-entropy you implemented yourself.</p>
<p>Best.</p>
<p>K. Frank</p>

username: macazinc
cooked: <p>Hi Frank,</p>
<p>That is exactly what I was asking. Thank you!</p>