Cross entropy for soft label

username: xzhu
cooked: <p>The cross entropy in pythorch can’t be used for the case when the target is soft label, a value between 0 and 1 instead of 0 or 1.</p>
<p>I code my own cross entropy, but i found the classification accuracy is always worse than the nn.CrossEntropyLoss() when i test on the dataset with hard labels, here is my loss:</p>
<pre><code class="lang-auto">class softCrossEntropy(nn.Module):
    def __init__(self):
        super(softCrossEntropy, self).__init__()
        return

    def forward(self, inputs, target):
        """
        :param inputs: predictions
        :param target: target labels
        :return: loss
        """
        log_likelihood = - F.log_softmax(inputs, dim=1)
        sample_num, class_num = target.shape
        loss = torch.sum(torch.mul(log_likelihood, target))/sample_num

        return loss
</code></pre>
<p>could anyone help me to check if there is problem in my code?</p>

username: rasbt
cooked: <p>Note that cross-entropy for non 0/1 labels is not symmetric, which could be an explanation for the poor performance.</p>
<p>E.g., consider the scenario for the binary cross entropy:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a592a21a8e083ae35a8133afea0bf39dcf11738e.png" alt="42%20PM" data-base62-sha1="nCIWsi9oMMTRU3v4hKY0NhybKNg" width="428" height="360"></p>
<p>Or consider the following, where the ground truth and the predicted labels are shown on the x axis. I.e., you can see that if both are 0, the cost is zero. However, if both are 0.5, the cost is almost 0.7, although prediction=true label</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/32944f6c7bd5034436e9f01b198884c5d62bf62f.png" alt="34%20PM" data-base62-sha1="7drCRhNL5x0czoRoItySU0qYDV5" width="607" height="437"></p>

username: xzhu
cooked: <p>similar with the label smoothing:<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/7455" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/KaiyuYue" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/19852297?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/7455" target="_blank" rel="nofollow noopener">Issue: [PyTorch][Feature Request] Label Smoothing for CrossEntropyLoss</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/KaiyuYue" target="_blank" rel="nofollow noopener">KaiyuYue</a>
	on <a href="https://github.com/pytorch/pytorch/issues/7455" target="_blank" rel="nofollow noopener">2018-05-10</a>
	</div>
	<div class="user">
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">Hi, guys. The type torch.LongTensor of target will hinder the implementation like some methods in reference. So is there a possible...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">todo</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: sadikneipp
cooked: <p>I believe this method from the new Snorkel lib is what you want:<br>
<a href="https://snorkel.readthedocs.io/en/master/packages/_autosummary/classification/snorkel.classification.cross_entropy_with_probs.html" class="onebox" target="_blank" rel="nofollow noopener">https://snorkel.readthedocs.io/en/master/packages/_autosummary/classification/snorkel.classification.cross_entropy_with_probs.html</a></p>

username: gabrieldernbach
cooked: <p>You can directly incorporate soft labels in a two class classification setting.<br>
Try a sigmoid activation on the scalar output of your network together with the Binary Cross Entropy Loss Function ( BCELoss() )</p>
<p>As you noted the multi class Cross Entropy Loss provided by pytorch does not support soft labels.<br>
You can however substitute the Cross Entropy Loss by taking the Kullback-Leibler Divergence (they are similar up to a constant offset which does not affect optimization).<br>
The KLDivLoss() of pytorch supports soft targets.</p>
<p>A quick recap on the relationship of KL Divergence and CE Loss can be found here: <a href="https://adventuresinmachinelearning.com/cross-entropy-kl-divergence/" rel="nofollow noopener">https://adventuresinmachinelearning.com/cross-entropy-kl-divergence/</a></p>