Soft Cross Entropy Loss (TF has it does Pytorch have it)

username: soulslicer
cooked: <p><a href="https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits" rel="nofollow noopener">softmax_cross_entropy_with_logits</a></p>
<p>TF supports not needing to have hard labels for cross entropy loss:</p>
<p>logits = [[4.0, 2.0, 1.0], [0.0, 5.0, 1.0]] labels = [[1.0, 0.0, 0.0], [0.0, 0.8, 0.2]] tf.nn.softmax_cross_entropy_with_logits(labels=labels, logits=logits)</p>
<p>Can we do the same thing in Pytorch?</p>

username: KFrank
cooked: <p>Hello Raaj!</p>
<aside class="quote no-group quote-modified" data-username="soulslicer" data-post="1" data-topic="69501" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/soulslicer/48/9454_2.png" class="avatar"> soulslicer:</div>
<blockquote>
<p>TF supports not needing to have hard labels for cross entropy loss:</p>
<p>Can we do the same thing in Pytorch?</p>
</blockquote>
</aside>
<p>I do not believe that pytorch has a “soft” cross-entropy function built in.<br>
But you can implement it using pytorch tensor operations, so you should<br>
get the full benefit of autograd and gpu acceleration.</p>
<p>See this (pytorch version 0.3.0) script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

# define "soft" cross-entropy with pytorch tensor operations
def softXEnt (input, target):
    logprobs = torch.nn.functional.log_softmax (input, dim = 1)
    return  -(target * logprobs).sum() / input.shape[0]

torch.manual_seed (2020)

# input values are logits
input  = torch.autograd.Variable (torch.randn ((2, 5)))
# target values are "soft" probabilities that sum to one (for each sample in batch)
target = torch.nn.functional.softmax (torch.autograd.Variable (torch.randn ((2, 5))), dim = 1)

input
target
softXEnt (input, target)

# make "hard" categorical target
dummy, target_cat = target.max (dim = 1)
# make "hard" one-hot target
target_onehot = torch.zeros_like (target).scatter (1, target_cat.unsqueeze (1), 1)

target_cat
target_onehot
# check that softXEnt agrees with pytorch's cross_entropy for "hard" case
torch.nn.functional.cross_entropy (input, target_cat)
softXEnt (input, target_onehot)
</code></pre>
<p>Here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt;
&gt;&gt;&gt; # define "soft" cross-entropy with pytorch tensor operations
... def softXEnt (input, target):
...     logprobs = torch.nn.functional.log_softmax (input, dim = 1)
...     return  -(target * logprobs).sum() / input.shape[0]
...
&gt;&gt;&gt; torch.manual_seed (2020)
&lt;torch._C.Generator object at 0x000001F948906630&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; # input values are logits
... input  = torch.autograd.Variable (torch.randn ((2, 5)))
&gt;&gt;&gt; # target values are "soft" probabilities that sum to one (for each sample in batch)
... target = torch.nn.functional.softmax (torch.autograd.Variable (torch.randn ((2, 5))), dim = 1)
&gt;&gt;&gt;
&gt;&gt;&gt; input
Variable containing:
 1.2372 -0.9604  1.5415 -0.4079  0.8806
 0.0529  0.0751  0.4777 -0.6759 -2.1489
[torch.FloatTensor of size 2x5]

&gt;&gt;&gt; target
Variable containing:
 0.0629  0.1508  0.5417  0.1899  0.0547
 0.0867  0.0389  0.0408  0.0659  0.7677
[torch.FloatTensor of size 2x5]

&gt;&gt;&gt; softXEnt (input, target)
Variable containing:
 2.4262
[torch.FloatTensor of size 1]

&gt;&gt;&gt;
&gt;&gt;&gt; # make "hard" categorical target
... dummy, target_cat = target.max (dim = 1)
&gt;&gt;&gt; # make "hard" one-hot target
... target_onehot = torch.zeros_like (target).scatter (1, target_cat.unsqueeze (1), 1)
&gt;&gt;&gt;
&gt;&gt;&gt; target_cat
Variable containing:
 2
 4
[torch.LongTensor of size 2]

&gt;&gt;&gt; target_onehot
Variable containing:
 0  0  1  0  0
 0  0  0  0  1
[torch.FloatTensor of size 2x5]

&gt;&gt;&gt; # check that softXEnt agrees with pytorch's cross_entropy for "hard" case
... torch.nn.functional.cross_entropy (input, target_cat)
Variable containing:
 2.2656
[torch.FloatTensor of size 1]

&gt;&gt;&gt; softXEnt (input, target_onehot)
Variable containing:
 2.2656
[torch.FloatTensor of size 1]
</code></pre>
<p>Good luck!</p>
<p>K. Frank</p>

username: JuanFMontesinos
cooked: <p>You can pass soft labels, no problem</p>

username: KFrank
cooked: <p>Hi Juan!</p>
<aside class="quote no-group" data-username="JuanFMontesinos" data-post="3" data-topic="69501" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>You can pass soft labels, no problem</p>
</blockquote>
</aside>
<p>No, this simply isn’t true.</p>
<p>From the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">CrossEntropyLoss</a> documentation:</p>
<pre><code class="lang-plaintext">Shape:

   Input: (N, C) where C = number of classes, ...

   Target: (N) where each value is 0 &lt;= targets[i] &lt;= C-1, ...
</code></pre>
<p>Raaj’s example target (“labels”):</p>
<pre data-code-wrap="python"><code class="lang-python">labels = [[1.0, 0.0, 0.0], [0.0, 0.8, 0.2]]
</code></pre>
<p>is neither the right shape (should be <code>(2,)</code> rather than <code>(2, 3)</code>),<br>
nor of the right values (should be one of the integers <code>0, 1, 2</code>,<br>
rather than a float, e.g., <code>0.8</code>).  As such, it won’t be accepted as<br>
the <code>target</code> argument by <code>CrossEntropyLoss</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: soulslicer
cooked: <p>works correctly thanks</p>

username: JuanFMontesinos
cooked: <p>Oh, I was thinking of Binary Cross Entropy. My bad.</p>

username: KFrank
cooked: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="69501" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I do not believe that pytorch has a “soft” cross-entropy function built in.</p>
</blockquote>
</aside>
<p>As of the current stable version, pytorch 1.10.0, “soft” cross-entropy<br>
labels are now supported.  See:</p>
<p><a href="https://pytorch.org/docs/1.10.0/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">CrossEntropyLoss – 1.1010</a>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: ShengweiAn
cooked: <p>From the linked document, I think the current CrossEntropyLoss in PyTorch only supports the linear combination between the original target and a vector of uniform distribution, and the coefficient is denoted by <strong>label_smoothing</strong> (<a href="https://docs.python.org/3/library/functions.html#float" rel="noopener nofollow ugc"><em>float</em></a><em>,</em> <em>optional</em> ) – A float in [0.0, 1.0]</p>
<p>Assume the ground truth target is 1. The smoothed label will be [1,0,0,0] * (1-label_smoothing) + label_smoothing*[0.25,0.25,0.25,0.25].</p>
<p>To support arbitrary targets, Frank’s previous code can work.</p>

username: David_Toscano
cooked: <p>The  function would be:<br>
cls_score → logits<br>
class_weight → if weighted classes , for example<br>
list = [1/10]*number of clases<br>
list[4] = 1<br>
class_weight = torch.tensor(list)<br>
label → softlabels, for example [0, 0.2, 0.8]</p>
<p>soft entropy loss wolud be:<br>
lsm = F.log_softmax(cls_score, 1)<br>
lsm = lsm * class_weight.unsqueeze(0)<br>
loss_cls = -(label * lsm).sum(1)<br>
loss_cls = loss_cls.sum() / torch.sum( class_weight.unsqueeze(0) * label)<br>
loss_cls = loss_cls.mean()</p>
<p>and then you can loss_cls.backward()</p>