Labels smoothing and categorical loss functions - alternatives?

Dawid_S: <p>Labels smoothing seems to be important regularization technique now and important component of Sequence-to-sequence networks.</p>
<p>Implementing labels smoothing is fairly simple. It requires, however, one-hot encoded labels to be passed to the cost function (smoothing is changing one and zero to slightly different values).</p>
<p>Is there any way to implement it in PyTorch? Could I use maybe some different loss function, that accepts one-hot vectors, or rewrite nn.functional.cross_entropy so that the gradient can be derived?</p>
<p>Or maybe any different ideas how to use labels smoothing not going into Tensorflow?</p>

smth: <p>you could use <a href="http://pytorch.org/docs/0.3.0/nn.html#torch.nn.BCEWithLogitsLoss">http://pytorch.org/docs/0.3.0/nn.html#torch.nn.BCEWithLogitsLoss</a> for this purpose</p>

Dawid_S: <p>Thank you. So I should just use this function? As prediction I should pass softmax output and as labels, and I should pass smoothed vector like [0.05, 0.05, 0.9], is that correct?</p>

Dawid_S: <p>Ok so I understand that I shouldn’t use softmax function, instead my predictions should be raw output of the last linear layer, and I should pass smoothed vector like [0.05, 0.05, 0.9] as a target. Moreover, it is ok to use BCELoss for multi-class classification.</p>
<p>I am looking for the confirmation that my ideas are ok.</p>

smth: <p>you are pretty much on spot with your last comment. raw preds go into BCELossWithLogits and you can use it for multi-class classification (target is one-hot encoding)</p>

sharifza: <p>Hi, I got a bit confused here so sorry for asking again. So do we need to to label smoothing of target values or there is no need anymore if we use BCELossWithLogits ? Because in the last question by Dawid he mentioned that he should pass smoothed vector as a target, but from your answer it seems like we need to just pass the one-hot encoded (and not smoothed) target vector.<br>
Thanks !</p>

chenyangh: <p>I think based on the paper, we need to convert the one-hot vector to the smoothed vector and use the original loss. But changing the loss criterion is doing the same math I think.</p>

sidSingla: <p>I have a question here, maybe naive but how does it work for multiclass?<br>
I believe BCE loss and Sigmoid activation is for binary classes only. I Will be grateful to know about it in detail.</p>
<p>-Sidharth</p>

ptrblck: <p>Multi-label classification use cases, where zero, one or multiple classes can be active in each sample, can use <code>nn.BCEWithLogitsLoss</code> as the loss function.<br>
The model output in this case should be <code>[batch_size, nb_classes]</code>.</p>
<p>Multi-class classification use cases, where only a single class is active for each sample, would use <code>nn.CrossEntropyLoss</code>.</p>

Mrig: <p>So how to use label smoothing in case of multi class classification with also class weights?</p>

ptrblck: <p>You could use e.g. <code>nn.KLDivLoss</code> and add the weighting to the unreduced loss or use a manual implementation for label smoothing (you should be able to find some posts in this forum).</p>

Brando_Miranda: <p>is the answer from here good? <a class="mention" href="/u/ptrblck">@ptrblck</a>  I am sort of confused what I should be using, to many options nothing that seems official pytorch.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/OpenNMT/OpenNMT-py/blob/e8622eb5c6117269bb3accd8eb6f66282b5e67d9/onmt/utils/loss.py#L186" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/OpenNMT/OpenNMT-py/blob/e8622eb5c6117269bb3accd8eb6f66282b5e67d9/onmt/utils/loss.py#L186" target="_blank" rel="noopener nofollow ugc">OpenNMT/OpenNMT-py/blob/e8622eb5c6117269bb3accd8eb6f66282b5e67d9/onmt/utils/loss.py#L186</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="176" style="counter-reset: li-counter 175 ;">
<li>        num_non_padding = non_padding.sum().item()</li>
<li>        return onmt.utils.Statistics(loss.item(), num_non_padding, num_correct)</li>
<li>
<li>    def _bottle(self, _v):</li>
<li>        return _v.view(-1, _v.size(2))</li>
<li>
<li>    def _unbottle(self, _v, batch_size):</li>
<li>        return _v.view(-1, batch_size, _v.size(1))</li>
<li>
<li>
<li class="selected">class LabelSmoothingLoss(nn.Module):</li>
<li>    """</li>
<li>    With label smoothing,</li>
<li>    KL-divergence between q_{smoothed ground truth prob.}(w)</li>
<li>    and p_{prob. computed by model}(w) is minimized.</li>
<li>    """</li>
<li>    def __init__(self, label_smoothing, tgt_vocab_size, ignore_index=-100):</li>
<li>        assert 0.0 &lt; label_smoothing &lt;= 1.0</li>
<li>        self.ignore_index = ignore_index</li>
<li>        super(LabelSmoothingLoss, self).__init__()</li>
<li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">class LabelSmoothingLoss(nn.Module):
    """
    With label smoothing,
    KL-divergence between q_{smoothed ground truth prob.}(w)
    and p_{prob. computed by model}(w) is minimized.
    """
    def __init__(self, label_smoothing, tgt_vocab_size, ignore_index=-100):
        assert 0.0 &lt; label_smoothing &lt;= 1.0
        self.ignore_index = ignore_index
        super(LabelSmoothingLoss, self).__init__()

        smoothing_value = label_smoothing / (tgt_vocab_size - 2)
        one_hot = torch.full((tgt_vocab_size,), smoothing_value)
        one_hot[self.ignore_index] = 0
        self.register_buffer('one_hot', one_hot.unsqueeze(0))

        self.confidence = 1.0 - label_smoothing

    def forward(self, output, target):
        """
        output (FloatTensor): batch_size x n_classes
        target (LongTensor): batch_size
        """
        model_prob = self.one_hot.repeat(target.size(0), 1)
        model_prob.scatter_(1, target.unsqueeze(1), self.confidence)
        model_prob.masked_fill_((target == self.ignore_index).unsqueeze(1), 0)

        return F.kl_div(output, model_prob, reduction='sum')
</code></pre>
<p>or is something else recommended?</p>
<p>obtained from:</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/55681502/label-smoothing-in-pytorch" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/2096883/jared-nielsen" target="_blank" rel="noopener nofollow ugc">
    <img alt="Jared Nielsen" src="https://i.stack.imgur.com/fesnr.png?s=128&amp;g=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/55681502/label-smoothing-in-pytorch" target="_blank" rel="noopener nofollow ugc">Label Smoothing in PyTorch</a>
</h4>

<div class="tags">
  <strong>python, tensorflow, machine-learning, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/2096883/jared-nielsen" target="_blank" rel="noopener nofollow ugc">
    Jared Nielsen
  </a>
  on <a href="https://stackoverflow.com/questions/55681502/label-smoothing-in-pytorch" target="_blank" rel="noopener nofollow ugc">01:14AM - 15 Apr 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<hr>
<p>references:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/cross-entropy-with-one-hot-targets/13580" class="inline-onebox">Cross-entropy with one-hot targets</a></li>
<li><a href="https://github.com/pytorch/pytorch/issues/7455" class="inline-onebox" rel="noopener nofollow ugc">[PyTorch][Feature Request] Label Smoothing for CrossEntropyLoss · Issue #7455 · pytorch/pytorch · GitHub</a></li>
</ul>