Please explain the code in the following Loss Class

SubhankarHalder: <p>Hello, I am a beginner in Deep Learning and PyTorch. If my question is not relevant or does not follow the community guidelines, please pardon me.</p>
<p>I am studying the following Kaggle kernel and trying to replicate it:<br>
<a href="https://www.kaggle.com/piantic/plant-pathology-2020-pytorch-for-beginner" class="onebox" target="_blank" rel="nofollow noopener">https://www.kaggle.com/piantic/plant-pathology-2020-pytorch-for-beginner</a></p>
<p>In the section titled, “PreTrainedModels”, the author uses a resnet34 model and writes the following loss function class:</p>
<pre><code class="lang-auto">class DenseCrossEntropy(nn.Module):

    def __init__(self):
        super(DenseCrossEntropy, self).__init__()
        
        
    def forward(self, logits, labels):
        logits = logits.float()
        labels = labels.float()
        
        logprobs = F.log_softmax(logits, dim=-1)
        
        loss = -labels * logprobs
        loss = loss.sum(-1)

        return loss.mean()
</code></pre>
<p>I’ve read about Cross Entropy Loss function and I think I get the basic gist of it. But I do not understand what’s going on in this class. Can someone please explain what’s going on?</p>
<p>I can see its calculating the log softmax but then why the multiplication with the labels? And why loss.sum(-1)?</p>
<p>This is a Kaggle competition where given a plant image, we have to predict 4 classes (healthy or 3 types of diseases).</p>
<p>Thank you so much for your help!</p>

KFrank: <p>Hi  Subhankar!</p>
<aside class="quote no-group quote-modified" data-username="SubhankarHalder" data-post="1" data-topic="76295" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/subhankarhalder/48/22648_2.png" class="avatar"> SubhankarHalder:</div>
<blockquote>
<pre><code class="lang-auto">class DenseCrossEntropy(nn.Module):
...
    def forward(self, logits, labels):
        logits = logits.float()
        labels = labels.float()
        
        logprobs = F.log_softmax(logits, dim=-1)
        
        loss = -labels * logprobs
        loss = loss.sum(-1)

        return loss.mean()
</code></pre>
<p>But I do not understand what’s going on in this class.<br>
…<br>
I can see its calculating the log softmax but then why the multiplication with the labels? And why loss.sum(-1)?</p>
</blockquote>
</aside>
<p>Pytorch’s <code>CrossEntropyLoss</code> takes as its <code>target</code> (<code>labels</code>) a single<br>
integer class label for each sample in the batch (i.e., a tensor of shape<br>
<code>[nBatch]</code>).  That is, <code>target == 2</code> means that class “2” is the right answer<br>
with 100% certainty.</p>
<p>The notion of cross-entropy is more general, however, in that it compares<br>
two probability distributions.</p>
<p>So you might wish to work with probabilistic targets. e.g. for <code>nBatch = 1</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">target = torch.FloatTensor ([[0.1, 0.2, 0.7]])
</code></pre>
<p>This means that this sample is in class “0” with probability 10%, in<br>
class “1” with probability 20%, and in class “2” with probability 70%.</p>
<p>This more general version of cross-entropy (which Kaggle is calling<br>
<code>DenseCrossEntropy</code>) is not supported directly in pytorch, hence the<br>
need to implement it explicitly.  (But it’s not very hard, and you can<br>
do it using pytorch tensor operations, so you get the full benefits of<br>
autograd and gpu support.)</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">        loss = -labels * logprobs
        loss = loss.sum(-1)
</code></pre>
</blockquote>
</aside>
<p>The “multiplication with the labels” is multiplying the vector of<br>
probabilistic <code>labels</code> (element-wise) with the vector of <code>logprobs</code><br>
obtained from the vector of <code>logits</code> (raw-score predictions).  The<br>
sum is just summing these terms in the formula for cross-entropy<br>
across classes (within a single sample, hence <code>.sum (dim = -1)</code>).</p>
<p>To recap one key point:  for <code>DenseCrossEntropy</code>, <code>labels</code> has<br>
shape <code>[nBatch, nClass]</code>, while for <code>torch.nn.CrossEntropyLoss</code>,<br>
<code>labels</code> (<code>target</code>) has shape <code>[nBatch]</code>.</p>
<p>You can see the formula for this general cross-entropy in Wikipedia’s<br>
<a href="https://en.wikipedia.org/wiki/Cross_entropy" rel="noopener nofollow ugc">Cross entropy</a> entry.</p>
<p>Compare this to the formula for pytorch’s <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">CrossEntropyLoss</a> that uses<br>
categorical class labels for its <code>target</code>.</p>
<p>(As an aside, you can use <em>one-hot</em>-encoded class labels as a special<br>
case of probabilistic labels with <code>DenseCrossEntropy</code>.  These are<br>
just labels that consist of all <code>0</code>s (0% probability) for all of the classes<br>
except for one <code>1</code> (100% probability) for the class that is being labelled<br>
as “correct.”)</p>
<p>Best.</p>
<p>K. Frank</p>

SubhankarHalder: <p>Thank you so much!! I understood this now.</p>