Logits vs. log-softmax

Zhihan_Yang: <p>In a classification task where the input can only belong to one class, the softmax function is naturally used as the final activation function, taking in “logits” (often from a preceeding linear layer) and outputting proper probabilities.</p>
<p>I am confused about the exact meaning of “logits” because many call them “unnormalized log-probabilities”. Yet they are different from applying log directly to the output of softmax, which are actual probabilities.</p>
<p>For example, consider the following experiment:</p>
<pre><code class="lang-python">import torch
import torch.nn.functional as F

logits = torch.tensor([51., 50., 49., 48.])

print('Probas from logits:\n', F.softmax(logits, dim=0))

print('Log-softmax:\n', F.log_softmax(logits, dim=0))
print('Difference between logits and log-softmax:\n', logits - F.log_softmax(logits, dim=0))

print('Probas from log-softmax:\n', F.softmax(F.log_softmax(logits, dim=0), dim=0))
</code></pre>
<p>and its output:</p>
<pre><code class="lang-auto">Probas from logits:
 tensor([0.6439, 0.2369, 0.0871, 0.0321])
Log-softmax:
 tensor([-0.4402, -1.4402, -2.4402, -3.4402])
Difference between logits and log-softmax:
 tensor([51.4402, 51.4402, 51.4402, 51.4402])
Probas from log-softmax:
 tensor([0.6439, 0.2369, 0.0871, 0.0321])
</code></pre>
<p>We can see that 1) the difference between the logits and the result of log-softmax is a constant and 2) the logits and the result of log-softmax yield the same probabilities after applying softmax.</p>
<p>So, my question is, why do we have a designated function for log-softmax? Why would we ever need the log-softmax of logits?</p>

KFrank: <p>Hi Zhihan!</p>
<aside class="quote no-group" data-username="Zhihan_Yang" data-post="1" data-topic="95979" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zhihan_yang/48/12773_2.png" class="avatar"> Zhihan_Yang:</div>
<blockquote>
<p>So, my question is, why do we have a designated function for log-softmax? Why would we ever need the log-softmax of logits?</p>
</blockquote>
</aside>
<p>The short, practical answer is because of what you typically do with<br>
the log-softmax of the logits.  You pass them into a loss function such<br>
as <code>nll_loss()</code>.  (Doing this gives you, in effect, the cross-entropy loss.)</p>
<p>If you were to pass the raw logits into <code>nll_loss()</code> you would get an<br>
ill-behaved loss function that is unbounded below.  That is, by, for<br>
example, making the biases of your last linear layer (that produces the<br>
logits) arbitrarily large, the logits will become arbitrarily large, and the<br>
loss function will become arbitrarily “good,” that is large and negative.</p>
<p>But why is this?</p>
<aside class="quote no-group">
<blockquote>
<p>I am confused about the exact meaning of “logits” because many call them “unnormalized log-probabilities”. Yet they are different from applying log directly to the output of softmax, which are actual probabilities.</p>
<p>…</p>
<p>We can see that 1) the difference between the logits and the result of log-softmax is a constant and 2) the logits and the result of log-softmax yield the same probabilities after applying softmax.</p>
</blockquote>
</aside>
<p>As you have noticed, the <code>log()</code> function is almost, but <em>not quite</em> the<br>
inverse of the <code>softmax()</code> function – the difference being a constant<br>
(across classes for a given set of logits).</p>
<p>This constant is the difference between proper log-probabilities and<br>
the “unnormalized log-probabilities” we call logits, and this is the<br>
constant that becomes arbitrarily large when the <code>nll_loss()</code> function<br>
diverges to <code>-inf</code>.  Calculating <code>log_softmax (logits)</code> normalizes this<br>
constant away.  (And, in some sense, that’s all it does, because<br>
<code>log_softmax (log_softmax (logits)) = log_softmax (logits)</code>.)</p>
<p>This constant is the log of the denominator in the formula for<br>
<code>softmax()</code>, namely <code>log (sum_i {exp (logit_i)})</code>.</p>
<p><code>log_softmax()</code> has the further technical advantage:  Calculating<br>
<code>log()</code> of <code>exp()</code> in the normalization constant can become numerically<br>
unstable.  Pytorch’s <code>log_softmax()</code> uses the <a href="https://en.wikipedia.org/wiki/LogSumExp#log-sum-exp_trick_for_log-domain_calculations" rel="noopener nofollow ugc">“log-sum-exp trick”</a> to<br>
avoid this numerical instability.</p>
<p>From this perspective, the purpose of pytorch’s <code>log_softmax()</code><br>
function is to remove this normalization constant – in a numerically<br>
stable way – from the raw, unnormalized logits we get from a linear<br>
layer so we can pass them into a useful loss function.</p>
<p>Best.</p>
<p>K. Frank</p>