Does nn.CrossEntropyLoss internally build a one-hot encoding of the target?

wjeliot: <p>I’m trying to build up an understanding of how nn.CrossEntropyLoss works. I see that it combines <code>log_softmax</code> and <code>NLLLoss</code> but am trying to work out exactly how.</p>
<p>Suppose a model outputs logits of [1.761, 0.590, 3.157] and the ‘target’ is the third index (2). So if I manually assembled a one-hot encoding vector, the target would look like [0, 0, 1].</p>
<p>When you use nn.CrossEntropyLoss, according to <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">the docs</a>, you can pass ‘class indices in the range [0,C) where C is the number of classes’ as the target. So in the above example, I would pass 2.</p>
<p>According to <a href="https://stackoverflow.com/a/62456801" rel="noopener nofollow ugc">this Stack Overflow answer</a> the nn.CrossEntropyLoss will simply ‘index into the output probability’ with the target (i.e. 2 indexes into the logits and selects 3.157) and then that number alone is used to compute the loss.</p>
<p>Is that true? Does this loss function simply index into the logits and work with that number? Or does it internally construct a one-hot encoding vector with a 1 in the target index position, and 0s elsewhere?</p>
<p>It seems to me that the latter is a much better option for computing loss because not only are you comparing the correct target to its predicted score, but you’re also comparing all the incorrect scores to 0. Surely that helps the model to learn faster? Or is my intuition incorrect here?</p>
<p>Thanks in advance.</p>

ptrblck: <aside class="quote no-group" data-username="wjeliot" data-post="1" data-topic="159010">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/8e8cbc/48.png" class="avatar"> wjeliot:</div>
<blockquote>
<p>Is that true? Does this loss function simply index into the logits and work with that number?</p>
</blockquote>
</aside>
<p>Yes, you can directly index the logit as seen in <a href="https://discuss.pytorch.org/t/cross-entropy-loss-clarification/103830/2">this small example</a> which calculates the loss manually using an explicit but slow approach.</p>
<aside class="quote no-group" data-username="wjeliot" data-post="1" data-topic="159010">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/8e8cbc/48.png" class="avatar"> wjeliot:</div>
<blockquote>
<p>Or does it internally construct a one-hot encoding vector with a 1 in the target index position, and 0s elsewhere?</p>
</blockquote>
</aside>
<p>No, as it would be wasteful to create the one-hot encoded target first and multiply it with the full logit tensor. The result would be equal to indexing just the logit where the one-hot encoded target would have its <code>1</code> value. The limitation is of course that indexing won’t work with “soft-targets”, i.e. target tensors which contain probabilities and not class indices. For this use case <code>nn.CrossEntropyLoss</code> accepts a <code>FloatTensor</code> with probabilities in newer PyTorch releases.</p>
<aside class="quote no-group" data-username="wjeliot" data-post="1" data-topic="159010">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/8e8cbc/48.png" class="avatar"> wjeliot:</div>
<blockquote>
<p>It seems to me that the latter is a much better option for computing loss because not only are you comparing the correct target to its predicted score, but you’re also comparing all the incorrect scores to 0. Surely that helps the model to learn faster? Or is my intuition incorrect here?</p>
</blockquote>
</aside>
<p>That’s not true, as you would multiply the logits of the uninteresting classes with a zero, so these values are not changing the loss at all.</p>

wjeliot: <p>Of course, there I was thinking my intuition was dodgy but actually I just wasn’t thinking through the process of <code>0 * any probability = 0</code>. Indexing in makes sense; as you say it would be pointless to build this one-hot encoding anyway. Thanks!</p>