How to use Soft-label for Cross-Entropy loss?

oasjd7: <p>As far as I know, Cross-entropy Loss for Hard-label is:</p>
<pre><code class="lang-auto">def hard_label(input, target):
    log_softmax = torch.nn.LogSoftmax(dim=1)
    nll = torch.nn.NLLLoss(reduction='none')
return nll(log_softmax(input), target) 
</code></pre>
<p>And then, How to implement Cross-entropy Loss for soft-label?</p>
<ol>
<li>What kind of Softmax should I use ?<br>
<code> nn.Softmax()</code> or <code>nn.LogSoftmax()</code> ?</li>
<li>How to make target labels?<br>
Just add random noise values to the zeros.?<br>
<code>[0,0,1] -&gt; [0.1,0.2,0.7]</code>
</li>
</ol>

KFrank: <p>Hello oasjd7!</p>
<aside class="quote no-group quote-modified" data-username="oasjd7" data-post="1" data-topic="72844" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oasjd7/48/5053_2.png" class="avatar"> oasjd7:</div>
<blockquote>
<p>As far as I know, Cross-entropy Loss for Hard-label is:</p>
<pre><code class="lang-auto">def hard_label(input, target):
...
</code></pre>
</blockquote>
</aside>
<p>(Or you can just use <code>torch.nn.CrossEntropyLoss</code>.)</p>
<aside class="quote no-group">
<blockquote>
<p>How to implement Cross-entropy Loss for soft-label?</p>
</blockquote>
</aside>
<p>Please see the following thread for an implementation:</p>
<aside class="quote" data-post="1" data-topic="69501">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/soulslicer/48/9454_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/soft-cross-entropy-loss-tf-has-it-does-pytorch-have-it/69501">Soft Cross Entropy Loss (TF has it does Pytorch have it)</a> 
  </div>
  <blockquote>
    <a href="https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits" rel="noopener nofollow ugc">softmax_cross_entropy_with_logits</a> 
TF supports not needing to have hard labels for cross entropy loss: 
logits = [[4.0, 2.0, 1.0], [0.0, 5.0, 1.0]] labels = [[1.0, 0.0, 0.0], [0.0, 0.8, 0.2]] tf.nn.softmax_cross_entropy_with_logits(labels=labels, logits=logits) 
Can we do the same thing in Pytorch?
  </blockquote>
</aside>

<aside class="quote no-group">
<blockquote>
<ol>
<li>What kind of Softmax should I use ?<br>
<code> nn.Softmax()</code> or <code>nn.LogSoftmax()</code> ?</li>
</ol>
</blockquote>
</aside>
<p>You should use <code>LogSoftmax</code>.  You have to pass the output of <code>Softmax</code><br>
through <code>log()</code> anyway to calculate the cross entropy, and the<br>
implementation of <code>LogSoftmax</code> is numerically  more stable than (the<br>
mathematically, but not numerically equivalent) <code>log (Softmax)</code>.</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>How to make target labels?<br>
Just add random noise values to the zeros.?<br>
<code>[0,0,1] -&gt; [0.1,0.2,0.7]</code></li>
</ol>
</blockquote>
</aside>
<p>If you don’t naturally have soft target labels (probabilities across the<br>
classes), I don’t see any value in ginning up soft labels by adding<br>
noise to your <code>0, 1</code> (<em>one-hot</em>) labels.  Just use <code>CrossEntropyLoss</code><br>
with your hard labels.</p>
<p>(If your hard labels are encoded as <code>0, 1</code>-style one-hot labels you will<br>
have to convert them to integer categorical class labels, as those are<br>
what <code>CrossEntropyLoss</code> requires.)</p>
<p>Best.</p>
<p>K. Frank</p>

saba: <p>HI There,</p>
<p>I want to use smooth labeling with the criterion=nn.CrossEntropyLoss() with batch size of 64. The labels are random number between 0.8 to 0.9 and the outputs are from sigmoid. The code is</p>
<pre><code class="lang-auto">

label=(0.9-0.8)* torch.rand(b_size) + 0.8
    label=label.to(device).type(torch.LongTensor)

    # Forward pass real batch through D
    
    netD=netD.float()
    output = netD(real_cpu).view(-1)
    # Calculate loss on all-real batch
    output1=torch.zeros(64,64)
    for ii in range(64):
        output1[:,ii]=ii
    for ii in range(64):
        output1[ii,:]= output[ii].type(torch.LongTensor)
        
    errD_real = criterion(output1, label)
</code></pre>
<p>and the error is:<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
<p>By applying (torch.LongTensor) all the labels and output become 0! and without (torch.LongTensor)  it gave me error.</p>

sailist: <p>My solution</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/46218566/pytorch-equivalence-for-softmax-cross-entropy-with-logits/63658068#63658068" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/11185460/sailist" target="_blank" rel="nofollow noopener">
    <img alt="Sailist" src="https://lh5.googleusercontent.com/-AEufb5EMpGA/AAAAAAAAAAI/AAAAAAAAAAs/BR92uKOiuXM/photo.jpg?sz=128" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/46218566/pytorch-equivalence-for-softmax-cross-entropy-with-logits/63658068#63658068" target="_blank" rel="nofollow noopener">PyTorch equivalence for softmax_cross_entropy_with_logits</a>
</h4>

<div class="tags">
  <strong>tensorflow, pytorch</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/11185460/sailist" target="_blank" rel="nofollow noopener">
    Sailist
  </a>
  on <a href="https://stackoverflow.com/questions/46218566/pytorch-equivalence-for-softmax-cross-entropy-with-logits/63658068#63658068" target="_blank" rel="nofollow noopener">01:44PM - 30 Aug 20 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ube: <p>This is probably late to answer this. I am also not sure if it would work, but what if you try inserting a manual cross-entropy function inside the forward pass…<br>
soft loss= -softlabel * log(hard label)</p>
<p>then apply hard loss on the soft loss the<br>
which will be loss = -sum of (hard label * soft loss)<br>
…but then you will have to make the softloss exp(loss)…to counteract repetitively log function.<br>
I wonder how it would turn out.</p>

ube: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html</a><br>
This is actually way better</p>

zabboud: <p>Has anyone tested the recent implementation of passing class probabilities as opposed to class as target for the cross entropy loss (ie. soft labels)? I’ve generated soft labels as target images for my application which works well with the binary cross entropy - I’ve changed the criterion to the CrossEntropyLoss and pass a soft target image (with values [0,1] as required per the documentation), however the loss doesn’t seem to be propagating well, it reduces to 0 very quickly (despite having regularization) and the probability map outputs are not changing from the initial epoch.</p>
<p>if anyone has used this function successfully - any input would be helpful.</p>
<p>please note I’m using this for semantic segmentation - which is initially a binary classification task but changing the target input as soft labels instead of hard binary labels.</p>
<p>Any inputs would be helpful as to what is going on.</p>