Categorical cross entropy loss function equivalent in PyTorch

ahmadreza9: <p>Hi,<br>
I found Categorical cross-entropy loss in Theano and Keras. Is <a href="https://pytorch.org/docs/master/generated/torch.nn.CrossEntropyLoss.html" rel="nofollow noopener">nn.CrossEntropyLoss()</a> equivalent of this loss function?<br>
I saw <a href="https://discuss.pytorch.org/t/pytorch-equivalent-of-keras-categorical-crossentropy-loss-function-and-fixing-my-pytorch-model-code/21771">this topic</a> but three is not a solution for that.</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> is used for a multi-class classification or segmentation using categorical labels.<br>
I’m not completely sure, what use cases Keras’ categorical cross-entropy includes, but based on the name I would assume, it’s the same.</p>
<p>Maybe let’s start from your use case and chose the corresponding loss function, so could you explain a bit what you are working on?</p>

ahmadreza9: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, I want something like below image. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/2/c2faf60883ee1991504ddfb1755aab744b592c05.png" data-download-href="https://discuss.pytorch.org/uploads/default/c2faf60883ee1991504ddfb1755aab744b592c05" title="IMG_20200612_153021"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c2faf60883ee1991504ddfb1755aab744b592c05_2_365x500.png" alt="IMG_20200612_153021" data-base62-sha1="rOSjh03AFPfRKe9btdx4TysCY8l" width="365" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c2faf60883ee1991504ddfb1755aab744b592c05_2_365x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c2faf60883ee1991504ddfb1755aab744b592c05_2_547x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/2/c2faf60883ee1991504ddfb1755aab744b592c05_2_730x1000.png 2x" data-dominant-color="706D6B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20200612_153021</span><span class="informations">1080×1478 154 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/8292551200b556cb0c87da85ec40c1451967c19f.png" data-download-href="https://discuss.pytorch.org/uploads/default/8292551200b556cb0c87da85ec40c1451967c19f" title="2020-06-12_19h22_35"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/2/8292551200b556cb0c87da85ec40c1451967c19f.png" alt="2020-06-12_19h22_35" data-base62-sha1="iD5AnbMCLs6LNYlVNXQLU5EbYnl" width="467" height="500" data-dominant-color="F1F1F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2020-06-12_19h22_35</span><span class="informations">611×653 42.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Bjorn_Lindqvist: <p>No. Categorical crossentropy (cce) loss in TF is not equivalent to cce loss in PyTorch.</p>
<p>The problem is that there are multiple ways to define cce and TF and PyTorch does it differently.</p>
<p>I haven’t found any builtin PyTorch function that does cce in the way TF does it, but you can easily piece it together yourself:</p>
<pre><code>&gt;&gt;&gt; y_pred = torch.tensor([[0.05, 0.95, 0], [0.1, 0.8, 0.1]])
&gt;&gt;&gt; y_true = torch.tensor([1, 2])                                                                                                                                         
&gt;&gt;&gt; nn.NLLLoss()(torch.log(y_pred), y_true)                                                                                                                               
tensor(1.1769)
</code></pre>
<p>The labels in <code>y_true</code> corresponds to TF’s one-hot encoding.</p>

ptrblck: <p>What is the difference between these implementations besides the target shape (one-hot vs. class index), i.e. do you get different losses for the same inputs?</p>

Bihy: <p>a bit late but I was trying to understand how Pytorch loss work and came across this post, on the other hand the difference  is Simply:</p>
<ul>
<li>
<code>categorical_crossentropy</code> (<code>cce</code>) produces a one-hot array containing the probable match for each category,</li>
<li>
<code>sparse_categorical_crossentropy</code> (<code>scce</code>) produces a category index of the <em>most likely</em> matching category. I think this is the one used by Pytroch</li>
</ul>
<p>Consider a classification problem with 5 categories (or classes).</p>
<ul>
<li>In the case of <code>cce</code>, the one-hot target may be <code>[0, 1, 0, 0, 0]</code> and the model may predict <code>[.2, .5, .1, .1, .1]</code> (probably right)</li>
<li>In the case of <code>scce</code>, the target index may be [1] and the model may predict: [.5].</li>
</ul>
<p>Consider now a classification problem with 3 classes.</p>
<ul>
<li>In the case of <code>cce</code>, the one-hot target might be <code>[0, 0, 1]</code> and the model may predict <code>[.5, .1, .4]</code> (probably inaccurate, given that it gives more probability to the first class)</li>
<li>In the case of <code>scce</code>, the target index might be <code>[0]</code>, and the model may predict <code>[.5]</code>
</li>
</ul>
<p>Many categorical models produce <code>scce</code> output because you save space, but lose A LOT of information (for example, in the 2nd example, index 2 was also very close.) I generally prefer <code>cce</code> output for model reliability.</p>
<p>There are a number of situations to use <code>scce</code>, including:</p>
<ul>
<li>when your classes are mutually exclusive, i.e. you don’t care at all about other close-enough predictions,</li>
<li>the number of categories is large to the prediction output becomes overwhelming.</li>
</ul>
<p>from <a href="https://stackoverflow.com/a/58566065" rel="noopener nofollow ugc">https://stackoverflow.com/a/58566065</a></p>

Arash_Vahabpour: <p><code>(-pred_label.log() * target_label).sum(dim=1).mean()</code></p>

100jy: <p>(-(pred_label+1e-5).log() * target_label).sum(dim=1).mean()</p>

Plagon: <p>This has also been adressed in the commens on stackoverflow but this answer is not correct.  The behavioral difference of cce and scce in tensorflow is that cce expectes the target labels as one-hot encoded and scce as class label (single integer).</p>

jamesbraza: <p>After a few hours of playing around, please see the following:</p>
<pre><code class="lang-python">import tensorflow as tf
import torch

# Shape: (B, C)
y_true: torch.Tensor
y_pred: torch.Tensor

# All the below are equivalent
# Assumption: model outputs scores (not a Softmax or LogSoftmax)

# PyTorch 1.13.1
-(torch.nn.functional.log_softmax(y_pred, dim=1) * y_true).sum(dim=1).mean()
torch.nn.functional.cross_entropy(input=y_pred, target=y_true)

# TensorFlow 2.11.0
tf.reduce_mean(
    tf.keras.metrics.categorical_crossentropy(
        y_true, torch.nn.functional.softmax(y_pred, dim=1).detach().numpy()
    )
)
tf.reduce_mean(
    tf.keras.metrics.categorical_crossentropy(
        y_true,
        torch.nn.functional.log_softmax(y_pred, dim=1).detach().numpy(),
        from_logits=True,
    )
)
</code></pre>