Cross entropy loss clarification

username: krioux5
cooked: <p>Hello,</p>
<p>I read the documentation for cross entropy loss, but could someone possibly give an alternative explanation? Or even walk through a small example of a 2x2 prediction and a 2x2 target. ie if target = [0,1;2,2] pred = [0,0;1,1] what would be the step by step be to calculate the cross entropy loss?</p>
<p>I’ve been having trouble finding an explanation relevant to semantic segmentation.</p>

username: ptrblck
cooked: <p>Here is a code snippet showing the PyTorch implementation and a manual approach.<br>
Note that I’ve used for loops to show how this loss can be calculated and that the difference between a standard multi-class classification and a multi-class segmentation is just the usage of the loss calculation on each pixel. You should not use this code, as it’s way slower than the internal implementation (also the numerical stability is worse, as I haven’t used <code>log_softmax</code>):</p>
<pre><code class="lang-python"># setup
criterion = nn.CrossEntropyLoss()

batch_size = 2
nb_classes = 4

output = torch.randn(batch_size, nb_classes)
target = torch.randint(0, nb_classes, (batch_size,))
loss = criterion(output, target)

# manual calculation
loss_manual = 0.
for idx in range(output.size(0)):
    # get current logit from the batch
    logit = output[idx]
    
    # get target from the batch
    t = target[idx]
    
    loss_manual += -1. * logit[t] + torch.log(torch.sum(torch.exp(logit)))

# calculate mean loss
loss_elements = output.size(0)
loss_manual = loss_manual / loss_elements
print(torch.allclose(loss, loss_manual))
&gt; True

# for segmentation
h, w = 4, 4
output = torch.randn(batch_size, nb_classes, h, w)
target = torch.randint(0, nb_classes, (batch_size, h, w,))
loss = criterion(output, target)

# manual calculation
loss_manual = 0.
for idx in range(output.size(0)):
    for h_ in range(h):
        for w_ in range(w):
            # get current logit from the batch
            logit = output[idx, :, h_, w_]
            
            # get target from the batch
            t = target[idx, h_, w_]
            
            loss_manual += -1. * logit[t] + torch.log(torch.sum(torch.exp(logit)))

# calculate mean loss
loss_elements = (output.size(0) * output.size(2) * output.size(3))
loss_manual = loss_manual / loss_elements
print(torch.allclose(loss, loss_manual))
&gt; True
</code></pre>

username: krioux5
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="103830">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>print(torch.allclose(loss, loss_manual))</code></p>
</blockquote>
</aside>
<p>Thank you for the thorough response. A few follow up questions:</p>
<ol>
<li>
<p>What is the mean loss used for? Just to print out as the accuracy metric? The more specific loss values (not averaged) are used in backpropagation, correct?</p>
</li>
<li>
<p>For some reason my network is always preferring whatever is the first label in a muli-label segmentation… For example, If I were to be segmenting pixels as dog, cat, background, if the dox pixels were label ‘1’ in training, all would be classified as dog, but the same is true if I assigned cat to 1. I realize this is very unusual. Have you ever heard of something like this before?</p>
</li>
</ol>

username: ptrblck
cooked: <ol>
<li>
<p>No, the mean loss is used in the <code>backward</code> pass and thus to calculate the gradients. You are usually using the mean, since the sum of the loss would e.g. depend on the batch size and you would have to adapt the learning rate based on the batch size</p>
</li>
<li>
<p>No, haven’t heard it before and I would assume your model overfits to the majority class. Is this behavior reproducible, i.e. are you seeing the same “label preference” using different seeds?</p>
</li>
</ol>

username: krioux5
cooked: <p>Oh wow, for some reason I thought individual loss values (pixel/voxel-wise) were used, I thought the network would need more info about where specifically/on which labels things went wrong in the prediction which wouldn’t be encapsulated in the average. That is very good to know.</p>
<p>I also thought it was overfitting to majority class, but I swapped the labels every which way and no matter what it preferred label 1. I used <a href="https://github.com/wolny/pytorch-3dunet" rel="noopener nofollow ugc">https://github.com/wolny/pytorch-3dunet</a> which is not my repo, and none of the examples include multi-class segmentation but it does support cross entropy loss so I’m not too sure.</p>

username: ptrblck
cooked: <p>The second issue is still interesting, as it smells a bit like a code bug.<br>
Would you be able to write a code snippet to reproduce this issue (either with random data or with a <code>torchvision</code> dataset)?</p>

username: krioux5
cooked: <p>In the code, is<br>
<code> loss_manual += -1. * logit[t] + torch.log(torch.sum(torch.exp(logit)))</code><br>
the equivalent to :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/5/458bde40caad863ec820b0f89d9ba4e059643a3c.png" data-download-href="https://discuss.pytorch.org/uploads/default/458bde40caad863ec820b0f89d9ba4e059643a3c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/5/458bde40caad863ec820b0f89d9ba4e059643a3c.png" alt="image" data-base62-sha1="9VeBVarCmeUwcW8UVAmLjCu8gVe" width="690" height="161" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">869×203 15.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Which works for when the outputs are probabilities summing to 1 (after a softmax layer?)</p>
<p>I will try to reproduce the issue as well.</p>

username: ptrblck
cooked: <p>Not necessarily, as the posted formula looks like the “positive” part of the binary cross-entropy loss.<br>
Let me know, if you were able to create a code snippet to reproduce this issue.</p>