How should I implement cross-entropy loss with continuous target outputs?

garytho: <p>The current version of cross-entropy loss only accepts one-hot vectors for target outputs.<br>
I need to implement a version of cross-entropy loss that supports continuous target distributions. What I don’t know is how to implement a version of cross-entropy loss that is numerically stable.</p>
<p>For example, would the following implementation work well?</p>
<blockquote>
<p>output = model(input)     <span class="hashtag-raw">#model</span> output is a softmax distribution over 3 categories<br>
target = Variable(torch.FloatTensor([0.1, 0.7, 0.2]))   <span class="hashtag-raw">#target</span> distribution is continuous – not one-hot<br>
loss = -1 * torch.sum(target * torch.log(output))   <span class="hashtag-raw">#compute</span> the cross-entropy<br>
loss.backward()</p>
</blockquote>
<p>Is this stable? Is there a built-in version I can use?</p>
<p>Thank you</p>

sparseinference: <p>I don’t know if it’s possible, but I’m interested in why you would want to use CrossEntropyLoss?</p>
<p>I have used MSELoss for similar things with good results.</p>
<p>Also, unless I’m very mistaken, the targets for nn.CrossEntropyLoss are not one-hot.</p>

SimonW: <p>Change <code>softmax + log</code> to <code>nn.LogSoftmax</code> and you are golden <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:">.</p>

SimonW: <p>Unfortunately, in current PyTorch’s CrossEntropyLoss, they are one-hot in the sense that target contains only one ground-truth class with “probability” 1.</p>

Brando_Miranda: <aside class="quote no-group" data-username="garytho" data-post="1" data-topic="10720">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/b77776/48.png" class="avatar"> garytho:</div>
<blockquote>
<p>is how to implement a version of cross-entropy loss that is numerically stable.</p>
</blockquote>
</aside>
<p>sorry can you detail a little more what you mean by your answer?</p>
<p><code>Change softmax + log to nn.LogSoftmax and you are golden</code></p>

garytho: <p>instead of doing torch.log(&lt;model_softmax_output&gt;),</p>
<p>change the last layer of the neural network to LogSoftmax and remove the torch.log() from the loss equation.</p>

Brando_Miranda: <p>may I see it in code?</p>

Brando_Miranda: <p>I’m not doing</p>
<p><code>torch.log(&lt;model_softmax_output&gt;)</code></p>
<p>I’m doing</p>
<pre><code>loss = criterion(y_pred, batch_ys)
</code></pre>
<p>with:</p>
<pre><code>criterion = torch.nn.CrossEntropyLoss()</code></pre>

garytho: <p>If you are using torch.nn.CrossEntropyLoss() then you don’t need a softmax output layer on your model.</p>
<p>So it would just be</p>
<blockquote>
<p>output = model(input) <span class="hashtag-raw">#logit</span> output<br>
criterion =torch.nn.CrossEntropyLoss()<br>
loss = criterion(output, target)</p>
</blockquote>

Brando_Miranda: <p>if the output of the model is a probability distribution the right thing is to use cross entropy as its equivalent to MLE. Using square loss is something else (usually assumes the noise is Gaussian).</p>

Brando_Miranda: <aside class="quote no-group" data-username="garytho" data-post="10" data-topic="10720">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/b77776/48.png" class="avatar"> garytho:</div>
<blockquote>
<p>torch.nn.CrossEntropyLoss()</p>
</blockquote>
</aside>
<p>Honestly I’d rather not use <code>torch.nn.CrossEntropyLoss()</code>, and thats why I was asking to look at the actual code you used.</p>

garytho: <p>My code is specific for target distributions that are not one-hot, I don’t know if that’s what you want, but does this help?</p>
<blockquote>
<p>output = model(input) <span class="hashtag-raw">#final</span> layer of model is LogSoftmax(), so the output is the log-probability distribution<br>
target = Variable(torch.FloatTensor([0.1, 0.7, 0.2])) <span class="hashtag-raw">#target</span> probability distribution<br>
loss = -1 * torch.sum(target * output) <span class="hashtag-raw">#the</span> crossentropy formula is -1 * sum( log(output_dist) * target_dist)<br>
loss.backward()</p>
</blockquote>

Brando_Miranda: <aside class="quote no-group" data-username="Brando_Miranda" data-post="5" data-topic="10720">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar"> Brando_Miranda:</div>
<blockquote>
<p>nn.LogSoftmax</p>
</blockquote>
</aside>
<p>yes I don’t have hot vectors either I’m learning a distribution or continuous target values as a well.</p>
<p>Though, I thought that wasn’t right (due to numerical issues), hence your question…am I right?</p>

garytho: <p>I’m confused what you are asking is correct, but the code I wrote above works. The numerical problem arises when taking torch.log of the softmax distribution because it could potentially output nan.</p>

imosafi: <p>Hi<br>
I have the same problem and tried this solution but seems like it’s not working very well.<br>
Is there a way to achieve this while keeping the model output as a regular softmax?</p>

garytho: <p>What do you mean by “it’s not working very well”?</p>
<p>If you make the output of your neural network softmax, and then take the log of it, it will be slower than logsoftmax and sometimes the output will be nan.</p>
<p>There are other loss functions, but cross-entropy loss is arguably the best one for probability distributions.</p>

imosafi: <p>I meant I don’t get good results (of course there might be a different cause for this)<br>
Anyway I would feel more comfortable if the model could just output a softmax and this way I could make sure there is no problem there.</p>

Hongyi_Zhang: <p>The following code should work in PyTorch 0.2:</p>
<pre><code class="lang-auto">def cross_entropy(pred, soft_targets):
    logsoftmax = nn.LogSoftmax()
    return torch.mean(torch.sum(- soft_targets * logsoftmax(pred), 1))
</code></pre>
<p>assuming pred and soft_targets are both Variables with shape (batchsize, num_of_classes), each row of pred is predicted logits and each row of soft_targets is a discrete distribution.</p>

Diego999: <p>Is there now an “official” pytorch function to do it or should we still do it by hand ?</p>

smk508: <p>I believe you can use <a href="http://pytorch.org/docs/master/nn.html#torch.nn.BCELoss" rel="nofollow noopener">BCELoss</a>, as long as your label and outputs are represented as normalized vectors. For example,</p>
<pre><code class="lang-auto">loss_fn = nn.BCELoss()
softmax = nn.Softmax()
input = Variable(torch.randn(3))
output = softmax(input)
target = Variable(torch.FloatTensor([.1, .7, .2]))
loss = loss_fn(output, target)
</code></pre>