Pytorch equivalence to sparse softmax cross entropy with logits in TensorFlow

deJQK: <p>Is there pytorch equivalence to sparse_softmax_cross_entropy_with_logits available in tensorflow?</p>
<p>I found CrossEntropyLoss and BCEWithLogitsLoss, but both seem to be not what I want. I ran the same simple cnn architecture with the same optimization algorithm and settings, tensorflow gives 99% accuracy in no more than 10 epochs, but pytorch converges to 90% accuracy (with 100 epochs simulation). Another thing is that BCEWithLogitsLoss requires one-hot form of labels (CrossEntropyLoss accepts integer valued labels).</p>
<p>If there is no such equivalent, is that possible to implement it manually?</p>

rasbt: <p>Note that <code>BCELoss</code> and <code>BCEWithLogitsLoss</code> is for binary labels.</p>
<p>There’s <code>NLLLoss</code>, i.e., negative log-likelihood, <a href="https://pytorch.org/docs/stable/nn.html#nllloss" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#nllloss</a> (operates on softmax output) and <code>CrossEntropyLoss</code>, which already combines the softmax plus the negative-likelihood for you:<br>
<a href="https://pytorch.org/docs/stable/nn.html#crossentropyloss" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#crossentropyloss</a></p>
<p>Also note that <code>CrossEntropyLoss</code> uses <code>nn.LogSoftmax</code>, not <code>nn.Softmax</code> – the log version is numerically more stable (not sure how TensorFlow implements their negative log likelihood cost func though, i.e., whether they use log softmax or softmax on the logits). You could combine <code>nn.Softmax</code> and <code>nn.LLLOSS</code> if you like and see if that replicates the TensorFlow outputs.</p>
<p>The convergence difference you mentioned can have many different reasons including the random seed for the weight initialization and the optimizer parameterization.</p>
<pre><code>import torch.nn.functional as F
loss = F.nll_loss(F.softmax(input), target)
</code></pre>
<p>The disadvantage of using softmax and the NLL loss separately is that it’s numerical less stable than using the derivative of the NLL loss with respect to the activation function directly.</p>
<p>I am not sure about the “sparse” part though, but it shouldn’t affect the results.</p>

ptrblck: <p>One side note:<br>
<code>nn.NLLLoss</code> should be used with <code>nn.LogSoftmax</code>, not <code>nn.Softmax</code> directly.<br>
So basically a logic output and <code>nn.CrossEntropyLoss</code> or a <code>nn.LogSoftmax</code> output with <code>nn.NLLLoss</code> yield identical losses:</p>
<pre><code class="lang-auto">m = nn.LogSoftmax(dim=1)

criterion1 = nn.CrossEntropyLoss()
criterion2 = nn.NLLLoss()

x = torch.randn(1, 5)
y = torch.empty(1, dtype=torch.long).random_(5)

loss1 = criterion1(x, y)
loss2 = criterion2(m(x), y)
print(loss1)
print(loss2)
</code></pre>

deJQK: <p><a class="mention" href="/u/rasbt">@rasbt</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for the explanation. The code I am modifying is from the MNIST example on <a href="https://github.com/ageron/handson-ml/blob/master/13_convolutional_neural_networks.ipynb" rel="nofollow noopener">this website</a>.</p>
<p>I also tried <a href="https://github.com/pytorch/examples/tree/master/mnist" rel="nofollow noopener">the example</a> from the official website of PyTorch, it is fast and converge to good performance. Even if I remove the dropout part, and modify the neural network as the tensorflow example, I get better results (around 95%).</p>
<p>I will focus on the official example from PyTorch. Thanks for your explanations.</p>

rasbt: <p>If helpful, I have a collection of implementations in Jupyter Notebooks where most of the multi-layer perceptrons and convnets are based on MNIST. For most, there’s a TensorFlow and a PyTorch implementation if you’d like to compare the two: <a href="https://github.com/rasbt/deep-learning-book/tree/master/code/model_zoo/" rel="nofollow noopener">https://github.com/rasbt/deep-learning-book/tree/master/code/model_zoo/</a></p>
<p>I haven’t particularly fine-tuned any of the networks, but they seem to perform decently on MNIST. E.g., ~98% test accuracy via the multi-layer perceptron with batch norm and ~99% via a super simple ResNet.</p>

deJQK: <p>Thanks <a class="mention" href="/u/rasbt">@rasbt</a>. I will learn them when I need. BTW, I like your book Python Machine Learning and learned a great many from it two years ago.</p>

ISMAX: <p>As it might be helpful to others, I posted an example of usage of nn.CrossEntropyLoss for image segmentation here :</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/63403485/is-there-a-version-of-sparse-categorical-cross-entropy-in-pytorch/66824605#66824605" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/3965888/ismael-el-atifi" target="_blank" rel="noopener nofollow ugc">
    <img alt="Ismael EL ATIFI" src="https://www.gravatar.com/avatar/b60b4f7faf22a43e8fe027b6bc022c1d?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/63403485/is-there-a-version-of-sparse-categorical-cross-entropy-in-pytorch/66824605#66824605" target="_blank" rel="noopener nofollow ugc">Is there a version of sparse categorical cross entropy in pytorch?</a>
</h4>

<div class="tags">
  <strong>pytorch, loss-function, cnn</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/3965888/ismael-el-atifi" target="_blank" rel="noopener nofollow ugc">
    Ismael EL ATIFI
  </a>
  on <a href="https://stackoverflow.com/questions/63403485/is-there-a-version-of-sparse-categorical-cross-entropy-in-pytorch/66824605#66824605" target="_blank" rel="noopener nofollow ugc">09:37PM - 26 Mar 21 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>