Cross Entropy Loss between 3d tensors

mfcs: <p>Hi everyone!</p>
<p>Could you help me with the Cross Entropy Loss in PyTorch?</p>
<p>I was looking here on the forum if someone had already made a post with the same question as mine but I didn’t find it, that’s why I’m posting here about Cross Entropy Loss, although there are already many topics about it.</p>
<p>My problem is the following: first the images I’m working with are in the CIE Lab color space. So my CNN accepts the L channel of the image as input and at the end, the CNN’s output is a tensor with the same dimensions as the image of the <em>L</em> channel, however, with 313 layers (313, w, h). For each pixel on the L channel, there will be 313 layers. The target is generated from channels <em>a</em> and <em>b</em> of the image. A probability distribution is also generated for each pair (a, b) of each pixel, as shown in the attached figure.<br>
So I would use Cross Entropy Loss to calculate the loss between these two probability distributions. In this case, each of the 313 color bins, as shown in the image, would be a class that would have a probability for each pixel.<br>
I’ve seen in other posts that you can’t use a probability distribution on the target for Cross Entropy Loss in PyTorch. So how would be this error is calculated? Because the error must be calculated for each pixel in the image and each pixel has its probability distribution. How the target would be represented?<br>
Would be the target a single matrix in which each position of the matrix would have the number of the color bin(the class) with highest value or probability for each pixel in the image? How will Cross Entropy Loss compare a probability distribution with a color bin value that represents a class? The Cross Entropy Loss accepts as input a 3D tensor and perform the loss for a probability distribution for each pixel?</p>
<p>I tried to represent a little of what I wrote in the drawing to try to facilitate understanding.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/191e99a90839d7286987ec6eee72f0f8f3b39155.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/191e99a90839d7286987ec6eee72f0f8f3b39155" title="IMG-3862"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/191e99a90839d7286987ec6eee72f0f8f3b39155_2_690x338.jpeg" alt="IMG-3862" data-base62-sha1="3AduQfrU9NIKMNRD0qm6fQJVgAl" width="690" height="338" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/191e99a90839d7286987ec6eee72f0f8f3b39155_2_690x338.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/191e99a90839d7286987ec6eee72f0f8f3b39155_2_1035x507.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/191e99a90839d7286987ec6eee72f0f8f3b39155_2_1380x676.jpeg 2x" data-dominant-color="B6B0A5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG-3862</span><span class="informations">3141×1540 769 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks for your time and help.</p>
<p>Best regards,</p>
<p>Matheus Santos.</p>

ptrblck: <p>If I understand your use case correctly, <a href="https://pytorch.org/docs/stable/nn.html?highlight=kldivloss#torch.nn.KLDivLoss">KLDivLoss</a> might work.</p>
<p>Let me know, it it’ll work.</p>

mfcs: <p>Thanks! I will read about this Loss Function and I will let you know if it workd <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>But, in the case of Cross Entropy Loss…does it make sense for the target to be a matrix, in which the elements are the values ​​of the color bins (classes) that have the highest probability, that for each pixel and the CNN’s output to be a tensor with 313 layers , in which each pixel has a probability distribution? Would Cross Entropy Loss calculate the error correctly?</p>
<p>I saw in other posts, some people talking about <em>soft target or probabilities</em>…I understood that this soft target or probabilities was a probability distribution as target but the probabilities was not 0 or 1, instead was values between 0 and 1. This loss that you suggested for me, could be used in this other cases of soft target or probabilities?</p>

ptrblck: <aside class="quote no-group" data-username="mfcs" data-post="3" data-topic="79131">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mfcs/48/17149_2.png" class="avatar"> mfcs:</div>
<blockquote>
<p>But, in the case of Cross Entropy Loss…does it make sense for the target to be a matrix, in which the elements are the values ​​of the color bins (classes) that have the highest probability, that for each pixel and the CNN’s output to be a tensor with 313 layers , in which each pixel has a probability distribution? Would Cross Entropy Loss calculate the error correctly?</p>
</blockquote>
</aside>
<p>This sounds like a classical segmentation use case and should be possible.</p>
<p>Soft targets should also be possible to create using <code>nn.KLDivLoss</code>, and you should also be able to use one’s and zeros as the target values.</p>

mfcs: <p>I see! Thanks!</p>
<aside class="quote no-group" data-username="mfcs" data-post="3" data-topic="79131">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mfcs/48/17149_2.png" class="avatar"> mfcs:</div>
<blockquote>
<p>But, in the case of Cross Entropy Loss…does it make sense for the target to be a matrix, in which the elements are the values ​​of the color bins (classes) that have the highest probability, that for each pixel and the CNN’s output to be a tensor with 313 layers , in which each pixel has a probability distribution? Would Cross Entropy Loss calculate the error correctly?</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="79131">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>This sounds like a classical segmentation use case and should be possible.</p>
</blockquote>
</aside>
<p>But how the Cross entropy Loss perform the computation between a probability distribution and a class value instead of the probability of the class?</p>

KFrank: <p>Hello Matheus!</p>
<aside class="quote no-group" data-username="mfcs" data-post="5" data-topic="79131" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mfcs/48/17149_2.png" class="avatar"> mfcs:</div>
<blockquote>
<p>But how the Cross entropy Loss perform the computation between a probability distribution and a class value instead of the probability of the class?</p>
</blockquote>
</aside>
<p>I won’t comment on whether KL-divergence or cross-entropy is<br>
likely to be the better loss function for your use case.</p>
<p>But (assuming I understand what you are asking), no, you can’t<br>
use pytorch’s built-in <code>CrossEntropyLoss</code> with probabilities for<br>
targets (sometimes called <em>soft labels,</em> a term I don’t much like).<br>
It requires integer class labels (even though cross-entropy makes<br>
perfect sense for targets that are probabilities).</p>
<p>However, you can write your own without much difficulty (or loss<br>
of performance).  See this post:</p>
<aside class="quote quote-modified" data-post="2" data-topic="69501">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/soft-cross-entropy-loss-tf-has-it-does-pytorch-have-it/69501/2">Soft Cross Entropy Loss (TF has it does Pytorch have it)</a> 
  </div>
  <blockquote>
    Hello Raaj! 

I do not believe that pytorch has a “soft” cross-entropy function built in. 
But you can implement it using pytorch tensor operations, so you should 
get the full benefit of autograd and gpu acceleration. 
See this (pytorch version 0.3.0) script: 
import torch
torch.__version__

# define "soft" cross-entropy with pytorch tensor operations
def softXEnt (input, target):
    logprobs = torch.nn.functional.log_softmax (input, dim = 1)
    return  -(target * logprobs).sum() / input.shap…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

mfcs: <p>I see! Thanks for replying! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
I will read about all that suggestions!</p>
<p>Just trying some tests, I am trying to convert a tensor like this:<br>
‘’’</p>
<pre><code class="lang-auto">1)
tensor([[[ 1., 20.],
         [ 2., 21.],
         [ 3., 22.],
         [ 4., 23.]],

        [[ 5., 24.],
         [ 6., 25.],
         [ 7., 26.],
         [ 8., 27.]],

        [[ 9., 28.],
         [10., 29.],
         [11., 30.],
         [12., 31.]]])
</code></pre>
<p>To something like this:</p>
<pre><code class="lang-auto">2)
tensor([[ 1., 5., 9.],
        [ 2., 6., 10.],
        [ 3., 7., 11.],
	    [ 4., 8., 12.],
	    [ 20., 24., 28.],
	    [ 21., 25., 29.],
	    [ 22., 26., 30.],
	    [ 23., 27., 31.]])
</code></pre>
<p>Each line in the second tensor corresponds to (x,y) element of the first tensor.<br>
The list of values of each line in the second tensor corresponds to the values in the first tensor through layers direction.</p>
<p>Could you help me with this? I am trying using transpose() and reshape() but I am not doing it correctly.</p>