Cant replicate keras CategoricalCrossentropy with pytorch

jiwidi: <p>I’m trying to port some code from keras to pytorch and I’m having some trouble achieving the same loss logic. I want to perform a simlar loss to <code>tf.keras.losses.CategoricalCrossentropy </code>. Torch <code>CrossEntropyLoss</code> handles targets with an integer with <code>nclass</code> while keras does it with a onehot encoder, to overcome this I tried using <code>BCEWithLogtisLoss</code> but having still not getting it.</p>
<p>I also went with the</p>
<p>I have this code where I test the three loss functions (<code>BCEWithLogtisLoss</code> , <code>tf.keras.losses.CategoricalCrossentropy </code> and <code>CrossEntropyLoss</code>) but cant get the same value:</p>
<pre><code class="lang-python">import torch
import tensorflow as tf
import numpy as np

query = np.array([[2.0,2,],[4,4]])

query_torch = torch.tensor(query)
eye_torch = torch.eye(2,2)
torch_loss = torch.nn.BCEWithLogitsLoss(reduction="sum")
print(f"Loss with torch BCE {torch_loss(query_torch, eye_torch)}")



query_tf = tf.convert_to_tensor(query)
eye_tf = tf.eye(2,2)
tf_loss = tf.keras.losses.CategoricalCrossentropy(
        from_logits=True, reduction=tf.keras.losses.Reduction.SUM)


print(f"Loss with tf {tf_loss(query_tf, eye_tf)}")


query_torch = torch.tensor(query)
eye_torch = torch.tensor([0,1])
torch_loss = torch.nn.CrossEntropyLoss(reduction="sum")
print(f"Loss with torch crossEntropyLoss {torch_loss(query_torch, eye_torch)}")
</code></pre>
<p>That prints:</p>
<pre><code class="lang-auto">Loss with torch BCE 6.29015588760376
Loss with tf 9.759140014648438
Loss with torch crossEntropyLoss 1.3862943611198906
</code></pre>
<p>Any idea how to achieve it?</p>

paganpasta: <p>Just pass the target instead of one-hot vector. For example, with torch.CrossEntropy, instead of eye(3,3) for target, pass <code>[0, 1, 2]</code>.</p>
<p>From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">documentation</a>:</p>
<pre><code class="lang-auto"># Example of target with class indices
loss = nn.CrossEntropyLoss()
input = torch.randn(3, 5, requires_grad=True)
target = torch.empty(3, dtype=torch.long).random_(5)
output = loss(input, target)
output.backward()
# Example of target with class probabilities
input = torch.randn(3, 5, requires_grad=True)
target = torch.randn(3, 5).softmax(dim=1)
output = loss(input, target)
output.backward()</code></pre>

jiwidi: <p>Hi! Thanks for the suggestion. I tried doing that but still wont get the same values with the example. I updated the main post with the new code to reproduce.</p>

KFrank: <p>Hi Jaime!</p>
<aside class="quote no-group quote-modified" data-username="jiwidi" data-post="1" data-topic="146747" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jiwidi/48/27512_2.png" class="avatar"> jiwidi:</div>
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">print(f"Loss with tf {tf_loss(query_tf, eye_tf)}")
...
print(f"Loss with torch crossEntropyLoss {torch_loss(query_torch, eye_torch)}")
</code></pre>
</blockquote>
</aside>
<p>Keras / tensorflow’s <code>CategoricalCrossentropy</code> and pytorch’s<br>
<code>CrossEntropyLoss</code> (more precisely, the function objects obtained<br>
by instantiating these classes) take their <code>prediction</code> and <code>target</code><br>
arguments in the opposite order.</p>
<p>I believe for keras / tensorflow you want:</p>
<pre data-code-wrap="python"><code class="lang-python">print(f"Loss with tf {tf_loss(eye_tf, query_tf,)}")
</code></pre>
<p>(Your pytorch <code>CrossEntropyLoss</code> usage looks correct.)</p>
<p>Best.</p>
<p>K. Frank</p>

jiwidi: <p>You are right! It works then and is similar to the crossentropyLoss. What a stupid mistake!</p>
<p>Thanks</p>