Cross entropy with logit targets

Haziq: <p>I am training a model for classification where each ground truth has some uncertainty and is thus a vector of probability scores e.g. <code>[0.1, 0, 0.7, 0, 0.2, 0, 0]</code> instead of <code>[0,0,1,0,0,0,0]</code>. The cross-entropy loss in PyTorch however, accepts only an integer target so I was hoping if someone could recommend a solution or an alternative loss function that is suitable for my classification problem.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html</a></p>

KFrank: <p>Hi Haziq!</p>
<aside class="quote no-group" data-username="Haziq" data-post="1" data-topic="134068" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/9d8465/48.png" class="avatar"> Haziq:</div>
<blockquote>
<p>each ground truth has some uncertainty and is thus a vector of probability scores e.g. <code>[0.1, 0, 0.7, 0, 0.2, 0, 0]</code> instead of <code>[0,0,1,0,0,0,0]</code>.</p>
</blockquote>
</aside>
<p>(As an aside, your title for this thread, “Cross entropy with logit<br>
targets” should probably be “Cross entropy with probability targets,”<br>
since you ask about “probability scores” and use probabilities in the<br>
example you give.)</p>
<p>Your ground-truth <code>target</code> probabilities are what are sometimes called<br>
“soft labels.”  Pytorch’s <code>CrossEntropyLoss</code> does not support this kind<br>
of <code>target</code>, but you can write a version that does.  For details, please<br>
see this post:</p>
<aside class="quote quote-modified" data-post="2" data-topic="69501">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/soft-cross-entropy-loss-tf-has-it-does-pytorch-have-it/69501/2">Soft Cross Entropy Loss (TF has it does Pytorch have it)</a> 
  </div>
  <blockquote>
    Hello Raaj! 

I do not believe that pytorch has a “soft” cross-entropy function built in. 
But you can implement it using pytorch tensor operations, so you should 
get the full benefit of autograd and gpu acceleration. 
See this (pytorch version 0.3.0) script: 
import torch
torch.__version__

# define "soft" cross-entropy with pytorch tensor operations
def softXEnt (input, target):
    logprobs = torch.nn.functional.log_softmax (input, dim = 1)
    return  -(target * logprobs).sum() / input.shap…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

ylsvik: <p><strong>Update:</strong> from version 1.10, Pytorch supports class probability targets in CrossEntropyLoss, so you can now simply use:</p>
<pre><code>criterion = torch.nn.CrossEntropyLoss()
loss = criterion(x, y)
</code></pre>
<p>where <code>x</code> is the input, <code>y</code> is the target. When <code>y</code> has the same shape as <code>x</code>, it’s gonna be treated as class probabilities. Note that <code>x</code> is expected to contain raw, unnormalized scores for each class, while <code>y</code> is expected to contain probabilities for each class (typically the output of the softmax layer). You can find details in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">docs</a>.</p>