Regarding different loss values with cross entropy loss during single tensors vs probability tensors

akshay_23: <p>So I was going through the documentation of the cross entropy loss and I noticed that while taking the probabilities they have performed softmax , but softmax is internally performed again in cross entropy loss so confused why its is mentioned again in the Example of target with class probabilities</p>
<pre><code class="lang-auto">&gt;&gt;&gt; # Example of target with class indices
&gt;&gt;&gt; loss = nn.CrossEntropyLoss()
&gt;&gt;&gt; input = torch.randn(3, 5, requires_grad=True)
&gt;&gt;&gt; target = torch.empty(3, dtype=torch.long).random_(5)
&gt;&gt;&gt; output = loss(input, target)
&gt;&gt;&gt; output.backward()
&gt;&gt;&gt;
&gt;&gt;&gt; # Example of target with class probabilities
&gt;&gt;&gt; input = torch.randn(3, 5, requires_grad=True)
&gt;&gt;&gt; target = torch.randn(3, 5).softmax(dim=1)
&gt;&gt;&gt; output = loss(input, target)
&gt;&gt;&gt; output.backward()
</code></pre>
<p>Moreover in this following example shouldnt the loss be the same in the first two cases</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch
from torch.nn import Softmax as softmax
target_tensor = torch.tensor([1, 1, 0, 1])
target_tensor_prob = torch.tensor([[1.0,  0.0],
        [1.0, 0.0],
        [0.0,  1.0],
        [1.0, 0.0]])
input_tensor = torch.randn(4, 2)

loss = nn.CrossEntropyLoss(reduction='mean')
with torch.no_grad():
    output = loss(input_tensor, target_tensor )
    print(output)
    output2 = loss(input_tensor, target_tensor_prob)
    print(output2)
    output3 = loss(input_tensor, target_tensor_prob.softmax(dim=1))
    print(output3)
</code></pre>
<p>It gives the following ouput<br>
tensor(0.7072)<br>
tensor(0.9613)<br>
tensor(0.8930)</p>
<p>According to my understanding the first 2 should be the same.</p>

KFrank: <p>Hi Akshay!</p>
<aside class="quote no-group quote-modified" data-username="akshay_23" data-post="1" data-topic="171505" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/akshay_23/48/55608_2.png" class="avatar"> akshay_23:</div>
<blockquote>
<p>I noticed that while taking the probabilities they have performed softmax , but softmax is internally performed again in cross entropy loss</p>
<pre><code class="lang-auto">&gt;&gt;&gt; # Example of target with class probabilities
&gt;&gt;&gt; input = torch.randn(3, 5, requires_grad=True)
&gt;&gt;&gt; target = torch.randn(3, 5).softmax(dim=1)
</code></pre>
</blockquote>
</aside>
<p>When using <code>CrossEntropyLoss</code> with probabilistic (sometimes called “soft”)<br>
targets, the <code>input</code> (the predictions) are unnormalized log-probabilities, but<br>
the <code>target</code> are probabilities.  Generating <code>target</code> by applying <code>softmax()</code> to<br>
a tensor is a convenient way to get legitimate probabilities for the sake of<br>
the example.</p>
<p><code>CrossEntropyLoss</code> does apply <code>log_softmax()</code> internally to its <code>input</code>, but<br>
<em>not</em> to its <code>target</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>Moreover in this following example shouldnt the loss be the same in the first two cases</p>
<pre><code class="lang-auto">target_tensor = torch.tensor([1, 1, 0, 1])
target_tensor_prob = torch.tensor([[1.0,  0.0],
        [1.0, 0.0],
        [0.0,  1.0],
        [1.0, 0.0]])
</code></pre>
</blockquote>
</aside>
<p>No, you have your <code>target_tensor_prob</code> backward.  Its first row, for example,<br>
should be <code>[0.0, 1.0]</code>, as this would mean a probability of zero of being in<br>
your “0-class” and a probability of one of being in your “1-class.”  This is what<br>
properly corresponds to the integer class label of <code>1</code> that appears as the first<br>
element of your <code>target_tensor</code>.</p>
<p>Best.</p>
<p>K. Frank</p>