nn.NLLLoss() gives negative result - what it's mean?

username: laro
cooked: <p>I saw code which use <code>nn.NLLLoss()</code> (negative log likelihood loss).</p>
<p>I looked on the results and some loss results (result of <code>nn.NLLLoss()</code>) return negative values.</p>
<p>What is the meaning of negative values of this loss?</p>

username: ptrblck
cooked: <p>I don’t know what a negative loss would represent and guess your inputs are in the wrong range as seen in this example:</p>
<pre><code class="lang-python">criterion = nn.NLLLoss()

output = F.log_softmax(torch.randn(10, 10), dim=1)
target = torch.randint(0, 10, (10,))
loss = criterion(output, target)
print(loss)
# tensor(2.6895)

# wrong inputs
output = F.softmax(torch.randn(10, 10), dim=1) 
loss = criterion(output, target)
print(loss)
# tensor(-0.1101)
</code></pre>
<p><code>nn.NLLLoss</code> expects log probabilities as the model output so make sure <code>F.log_softmax</code> was applied on the model output.</p>

username: WhatWouldKantDo
cooked: <p>Hi all, sorry to revive this very old thread but I have a very similar issue using the GaussianNLLLoss function. My NN creates two tensors (expectation and variance) of the same size as the target tensor. To avoid negative variances, I apply the exponential function to the variance tensor which basically means that the immediate output of the NN is considered the log variance. During the training process, I often observe negative losses making it hard to judge if the optimization is actually converging. What am I missing? Thanks in advance for any hint <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: KFrank
cooked: <p>Hi Meister!</p>
<aside class="quote no-group quote-modified" data-username="WhatWouldKantDo" data-post="3" data-topic="155627" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/f14d63/48.png" class="avatar"> WhatWouldKantDo:</div>
<blockquote>
<p>I have a very similar issue using the GaussianNLLLoss function.<br>
…<br>
I often observe negative losses<br>
…<br>
What am I missing?</p>
</blockquote>
</aside>
<p>It is perfectly reasonable for <code>GaussianNLLLoss</code> to return negative loss<br>
values.</p>
<p>It is true that many loss criteria are never negative and become zero only<br>
for perfect predictions (for example, <code>MSELoss</code>), but there is no logical<br>
requirement for this to be the case.</p>
<p><code>GaussianNLLLoss</code> computes “likelihoods” internally – these are values<br>
of a probability density function.  A probability density function must be<br>
non-negative, but can range up to positive infinity.  (For example, the<br>
maximum value of the probability density function for a Gaussian diverges<br>
to infinity as its variance goes to zero.  So the log of a probability density<br>
function – the log-likelihood – can range from <code>-inf</code> to <code>inf</code>.  Accordingly,<br>
<code>GaussianNLLLoss</code> can return loss values that also range from <code>-inf</code> to <code>inf</code>.</p>
<p>(In contrast, a probability ranges from zero to one, so a log-probability<br>
ranges from <code>-inf</code> to <code>0.0</code>.  <code>NLLLoss</code> takes log-probabilities as input, not<br>
likelihoods nor log-likelihoods.  When passed valid inputs, <code>NLLLoss</code><br>
returns values that range from <code>0.0</code> to <code>inf</code>.)</p>
<p>As an aside, it would have been better to post this a a new question, rather<br>
than resurrecting a zombie thread.</p>
<p>Best.</p>
<p>K. Frank</p>