Needing clarity for equivalent of Categoricalcrossentropy as CrossEntropyLoss

someshfengde: <p>I wanted to have CategoricalCrossentropyLoss as performed in TensorFlow but while<br>
I’ve read answers from discuss but it’s I am not getting clearly what to do</p>
<p>here is what I tried</p>
<p>my outputs =</p>
<pre><code class="lang-auto">output_truth = np.ones((1,1024,15))

if we tried to find loss for above shape using 
tf.keras.losses.CategoricalCrossentropy()(output_from_model, output_truth) 
# we get answer with no error 

but if we same tried with pytorch 
torch.nn.CrossEntropyLoss()(torch.tensor(output_from_model) torch.tensor(output_truth))
</code></pre>
<p>we get following error</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_33/1032697456.py in &lt;module&gt;
----&gt; 1 torch.nn.CrossEntropyLoss()(torch.tensor(tf_outs.numpy()), torch.tensor(x[2].numpy()))

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
   1119     def forward(self, input: Tensor, target: Tensor) -&gt; Tensor:
   1120         return F.cross_entropy(input, target, weight=self.weight,
-&gt; 1121                                ignore_index=self.ignore_index, reduction=self.reduction)
   1122 
   1123 

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2822     if size_average is not None or reduce is not None:
   2823         reduction = _Reduction.legacy_get_string(size_average, reduce)
-&gt; 2824     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2825 
   2826 

RuntimeError: Expected target size [1, 15], got [1, 1024, 15]
</code></pre>

mMagmer: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a9f7bfe7cbfd603e1d430357f103d96dd1aaa0d.png" data-download-href="https://discuss.pytorch.org/uploads/default/3a9f7bfe7cbfd603e1d430357f103d96dd1aaa0d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a9f7bfe7cbfd603e1d430357f103d96dd1aaa0d_2_690x202.png" alt="image" data-base62-sha1="8mBnZnf5vQ8zOqmuMgYHhTZb32J" width="690" height="202" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a9f7bfe7cbfd603e1d430357f103d96dd1aaa0d_2_690x202.png, https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a9f7bfe7cbfd603e1d430357f103d96dd1aaa0d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a9f7bfe7cbfd603e1d430357f103d96dd1aaa0d.png 2x" data-dominant-color="F2F0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">698×205 25.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>read <a href="https://pytorch.org/docs/1.10.1/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">CrossEntropyLoss</a>.<br>
if you want target of shape 1x15x1024, then your model output should have 1xCx15x1024 shape.</p>

someshfengde: <p>15 were the number of classes in my case</p>

mMagmer: <p>then reshape target to be of size 1x15x1024<br>
and model output of size 1x15x1024</p>
<blockquote>
<p>If target containing class probabilities, same shape as the input.</p>
</blockquote>

someshfengde: <p>the output and target are already having same shape</p>

someshfengde: <p>I am just not getting why it’s throwing me an error</p>

mMagmer: <pre><code class="lang-auto">import torch

import torch.nn as nn
t = torch.ones(1,1024,15)
o = torch.randn(1,1024,15)
loss_fn = nn.CrossEntropyLoss()
loss_fn(o,t)
</code></pre>
<p>i see no error.</p>

someshfengde: <p>while I was trying I got this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de53dc22829183ee9dd6e0ea862629fbfc90f5b4.png" data-download-href="https://discuss.pytorch.org/uploads/default/de53dc22829183ee9dd6e0ea862629fbfc90f5b4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de53dc22829183ee9dd6e0ea862629fbfc90f5b4_2_690x243.png" alt="image" data-base62-sha1="vINFnG4sNoEUPvupVedT15njIFu" width="690" height="243" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de53dc22829183ee9dd6e0ea862629fbfc90f5b4_2_690x243.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de53dc22829183ee9dd6e0ea862629fbfc90f5b4_2_1035x364.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/e/de53dc22829183ee9dd6e0ea862629fbfc90f5b4.png 2x" data-dominant-color="F6D9D9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1243×438 88.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

someshfengde: <p>I even tried on my local environment still getting that error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5069afbef0070ef5ade3df316a5edd25dc5a8ce.png" data-download-href="https://discuss.pytorch.org/uploads/default/f5069afbef0070ef5ade3df316a5edd25dc5a8ce" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5069afbef0070ef5ade3df316a5edd25dc5a8ce_2_690x316.png" alt="image" data-base62-sha1="yXB8ctH9DoTVyG9umL50BvW0MgK" width="690" height="316" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5069afbef0070ef5ade3df316a5edd25dc5a8ce_2_690x316.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/5/f5069afbef0070ef5ade3df316a5edd25dc5a8ce_2_1035x474.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/5/f5069afbef0070ef5ade3df316a5edd25dc5a8ce.png 2x" data-dominant-color="3D1932"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1072×491 85.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
what’s the pytorch version you are using ?</p>

mMagmer: <p>check torch version. 1.10</p>
<pre><code class="lang-auto">torch.__version__
</code></pre>

someshfengde: <p>1.9.0+cu111<br>
it’s my version</p>

mMagmer: <p>it seems that only crossentorpy in version 1.10 can get <em>Probabilities for each class as target</em>.</p>
<aside class="quote" data-post="7" data-topic="69501">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/soft-cross-entropy-loss-tf-has-it-does-pytorch-have-it/69501/7">Soft Cross Entropy Loss (TF has it does Pytorch have it)</a> 
  </div>
  <blockquote>
    As of the current stable version, pytorch 1.10.0, “soft” cross-entropy 
labels are now supported.  See: 
<a href="https://pytorch.org/docs/1.10.0/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">CrossEntropyLoss – 1.1010</a>. 
Best. 
K. Frank
  </blockquote>
</aside>


someshfengde: <p>ohh I see thanks for help I think I can get that running<br>
the problem was with my version</p>
<p>again there is no space left on my device so have to install that after freeing up some more space</p>

mMagmer: <pre><code class="lang-auto">
from torch.nn.modules.loss import _WeightedLoss
import torch.nn.functional as F

class SmoothCrossEntropyLoss(_WeightedLoss):
    def __init__(self, weight=None, reduction='mean'):
        super().__init__(weight=weight, reduction=reduction)
        self.weight = weight
        self.reduction = reduction

    def reduce_loss(self, loss):
        return loss.mean() if self.reduction == 'mean' else loss.sum() \
        if self.reduction == 'sum' else loss

    def forward(self, inputs, targets):

        log_preds = F.log_softmax(inputs, -1)

        if self.weight is not None:
            log_preds = log_preds * self.weight.unsqueeze(0)

        return self.reduce_loss(-(targets * log_preds).sum(dim=-1))
</code></pre>
<p>maybe work. ¯\_(ツ)_/¯<br>
class <em>Probabilities</em> shoud be at last dim.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
t = torch.ones(1,1024,15)
o = torch.randn(1,1024,15)
loss_fn = SmoothCrossEntropyLoss()
loss_fn(o,t) #-&gt; 47.3582
loss_fn1 = nn.CrossEntropyLoss()
loss_fn1(o.transpose(1,2),t.transpose(1,2)) #-&gt; 47.3582
</code></pre>