Loss nll - formula in documentation

PhysicsIsFun: <p>Greetings,</p>
<p>I am a bit confused by the documented formula for the negative log-likelihood loss:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/1/8164442971f146c5d6e3894faa940c65c92ec0b3.png" data-download-href="https://discuss.pytorch.org/uploads/default/8164442971f146c5d6e3894faa940c65c92ec0b3" title="NLLL"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/8164442971f146c5d6e3894faa940c65c92ec0b3_2_690x217.png" alt="NLLL" data-base62-sha1="isEpCa5vUzbMjN8Q89VWar9f7Td" width="690" height="217" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/8164442971f146c5d6e3894faa940c65c92ec0b3_2_690x217.png, https://discuss.pytorch.org/uploads/default/original/3X/8/1/8164442971f146c5d6e3894faa940c65c92ec0b3.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/1/8164442971f146c5d6e3894faa940c65c92ec0b3.png 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">NLLL</span><span class="informations">766×242 35.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What are those x_(n,y_n) ? They say x is the input, but the loss is not calculated from the input. A loss should be calculated from the output and the target, should it not?</p>
<p>Best,<br>
PiF</p>
<p>edit: is the following assumption correct?<br>
x_(n,y_n) is the entry of the network output vector that corresponds to the probability of the target y_n, where n denotes the sample.</p>

KFrank: <p>Hi Physics!</p>
<aside class="quote no-group quote-modified" data-username="PhysicsIsFun" data-post="1" data-topic="109728" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>I am a bit confused by the documented formula for the negative log-likelihood loss:<br>
…<br>
What are those x_(n,y_n) ? They say x is the input, but the loss is not calculated from the input. A loss should be calculated from the output and the target, should it not?</p>
</blockquote>
</aside>
<p>Yes, the terminology in the documentation is somewhat unfortunate.</p>
<p>The argument to the loss functions that pytorch refers to as <code>input</code> is<br>
indeed the “output” of the network (or derived from it).  I prefer to call<br>
this the “prediction.”  The loss function then compares the “prediction”<br>
with the “ground truth” label, which is the argument that pytorch calls<br>
the <code>target</code>.</p>
<p>i guess one can think of the “prediction” being the “input” to the loss<br>
function (but so is the <code>target</code>).</p>
<p>So “output” of model = “prediction” = “input” to loss function.  But using<br>
the term <code>input</code> in the loss-function documentation has always seemed<br>
a bit confusing to me, as it suggests to me the input to the model, rather<br>
than the output of the model.  But I’ve made my peace with the pytorch<br>
terminology.</p>
<aside class="quote no-group">
<blockquote>
<p>edit: is the following assumption correct?<br>
x_(n,y_n) is the entry of the network output vector that corresponds to the probability of the target y_n, where n denotes the sample.</p>
</blockquote>
</aside>
<p>Not quite.  What you denote “x_(,y_n)” is the predicted <em>log</em>-probability<br>
of the sample corresponding to target y_n (rather than the probability).</p>
<p>So, typically, the output of the final <code>Linear</code> layer in the model are the<br>
predicted raw-score <em>logits.</em>  When passed through <code>LogSoftmax</code>, you<br>
get the predicted log-probabilities that you then pass into <code>NLLLoss</code>.<br>
(If you pass the logits through <code>Softmax</code>, you get probabilities, but it’s<br>
numerically more stable to work with the log-probabilities.)</p>
<p>(For convenience, pytorch’s <code>CrossEntropyLoss</code> combines <code>LogSoftmax</code><br>
together with <code>NLLLoss</code> so you can pass in the logits directly without<br>
passing them through a separate <code>LogSoftmax</code> function.)</p>
<p>Best.</p>
<p>K. Frank</p>

PhysicsIsFun: <p>Thank you very much K. Frank, for this detailed explanation!<br>
I guess calling the network output “input” makes sense from a programmer’s perspective, since it is an input to a function.</p>
<p>It is all clear now.<br>
PiF</p>