Cross entropy between two softmax outputs

S_M: <p>Hi all, I want to compute the cross-entropy between two 2D tensors that are the outputs of the softmax function.</p>
<pre><code class="lang-auto">        P=nn.CrossEntropyLoss(softmax_out1,softmax_out2)
</code></pre>
<p>softmax_out1 and softmax_out2 are 2D tensors with shapes (128,10) that 128 refers to the batch size and 10 is the number of classes.<br>
the following error occurs:</p>
<pre><code class="lang-auto">RuntimeError: 1D target tensor expected, multi-target not supported
</code></pre>
<p>any example code to handle this error would be appreciated.</p>

Aman_Singh: <p>In Pytorch, <code>nn.CrossEntropyLoss</code> combines <code>LogSoftmax</code> and <code>NLLLoss</code>. Your input to <code>nn.CrossEntropyLoss</code> should be logits and the original targets and not the softmax probabilities themselves .</p>
<p>Also, it should not be used as<br>
<code> loss=nn.CrossEntropyLoss(output, target)</code></p>
<p>but instead as below:<br>
<code> loss = nn.CrossEntropyLoss()(output, target)</code></p>
<p>Do share your training loop code if possible.</p>

KFrank: <p>Hi S.!</p>
<aside class="quote no-group" data-username="S_M" data-post="1" data-topic="115807" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/s_m/48/19733_2.png" class="avatar"> S_M:</div>
<blockquote>
<pre><code class="lang-auto">        P=nn.CrossEntropyLoss(softmax_out1,softmax_out2)
</code></pre>
</blockquote>
</aside>
<p>The short answer is that you have to write your own cross-entropy<br>
function to do what you want – see below.</p>
<p>There are two things going on here:</p>
<p>First, as Aman noted, the <code>input</code> to <code>CrossEntropyLoss</code> (your<br>
<code>softmax_out1</code>) should be raw-score logits that range from <code>-inf</code> to<br>
<code>+inf</code>, rather than probabilities that range from <code>0.0</code> to <code>1.0</code>.  So you<br>
want to pass logits in as the <code>input</code> <em>without</em> converting them to<br>
probabilities by running them through <code>softmax()</code>.</p>
<p>Second, <code>CrossEntropyLoss</code> expects its <code>target</code> (your <code>softmax_out2</code>)<br>
to be integer class labels (with shape <code>[nBatch]</code>, rather than<br>
<code>[nBatch, nClass]</code>).  So <code>CategoricalCrossEntropyWithLogitsLoss</code><br>
might be a better (if lengthier) name for this loss function.</p>
<p>Now, how to do what you want:</p>
<p>Even if you write your own cross-entropy loss function, you do <em>not</em><br>
want to pass in probabilities for your <code>input</code> as doing so  will be less<br>
numerically stable than passing in logits.</p>
<p>It does, however, make sense to use probabilities (rather than integer<br>
class labels) for your <code>target</code>.  (These are sometimes called <em>soft</em> labels<br>
or <em>soft</em> targets.)  It’s just that pytorch doesn’t offer such a version of<br>
cross entropy.</p>
<p>The following post shows how to implement such a “soft cross-entropy”<br>
loss.  It takes logits for its <code>input</code> (for numerical stability) and takes<br>
probabilities for its “soft-label” <code>target</code>:</p>
<aside class="quote quote-modified" data-post="2" data-topic="69501">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/soft-cross-entropy-loss-tf-has-it-does-pytorch-have-it/69501/2">Soft Cross Entropy Loss (TF has it does Pytorch have it)</a> 
  </div>
  <blockquote>
    Hello Raaj! 

I do not believe that pytorch has a “soft” cross-entropy function built in. 
But you can implement it using pytorch tensor operations, so you should 
get the full benefit of autograd and gpu acceleration. 
See this (pytorch version 0.3.0) script: 
import torch
torch.__version__

# define "soft" cross-entropy with pytorch tensor operations
def softXEnt (input, target):
    logprobs = torch.nn.functional.log_softmax (input, dim = 1)
    return  -(target * logprobs).sum() / input.shap…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>