What is the difference between log_softmax and softmax?

username: yenming
cooked: <p>What is the difference between log_softmax and softmax?<br>
How to explain them in mathematics?<br>
Thank you!</p>

username: KaiyangZhou
cooked: <p>log_softmax applies logarithm after softmax.</p>
<p>softmax:</p>
<pre><code class="lang-auto">exp(x_i) / exp(x).sum()
</code></pre>
<p>log_softmax:</p>
<pre><code class="lang-auto">log( exp(x_i) / exp(x).sum() )
</code></pre>
<p>log_softmax essential does log(softmax(x)), but the practical implementation is different and more efficient while doing the same operation. You might want to have a look at <a href="http://pytorch.org/docs/master/nn.html?highlight=log_softmax#torch.nn.LogSoftmax" rel="nofollow noopener">http://pytorch.org/docs/master/nn.html?highlight=log_softmax#torch.nn.LogSoftmax</a> and the source code.</p>

username: dchatterjee172
cooked: <p>Can you please link that implementation?<br>
Is it calculated by, <code>x_i - log( exp(x).sum() )</code> ?</p>

username: KaiyangZhou
cooked: <p>The implementation is done in <code>torch.nn.functional</code> where the function is called from c code: <a href="http://pytorch.org/docs/master/_modules/torch/nn/functional.html#log_softmax" rel="nofollow noopener">http://pytorch.org/docs/master/_modules/torch/nn/functional.html#log_softmax</a>.</p>

username: dchatterjee172
cooked: <p>Is there any way to see the c code?</p>

username: KaiyangZhou
cooked: <aside class="quote" data-post="1" data-topic="2015">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/acgtyrant/48/4971_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/where-does-torch--c-come-from/2015">Where does `torch._C` come from?</a> 
  </div>
  <blockquote>
    I am read the code of batch normlization, and I find <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py#L454" rel="noopener nofollow ugc">this line</a>: 
f = torch._C._functions.BatchNorm(running_mean, running_var, training, momentum, eps, torch.backends.cudnn.enabled) 
But I do not find any library called the _C. I do not know where does torch._C.functions.BatchNorm come from.
  </blockquote>
</aside>


username: drscotthawley
cooked: <p><a class="mention" href="/u/kaiyangzhou">@KaiyangZhou</a>’s answer may have been correct once, but does not match the current documentation, which reads:</p>
<blockquote>
<p>“While mathematically equivalent to log(softmax(x)), doing these two<br>
operations separately is slower, and numerically unstable. This function<br>
uses an alternative formulation to compute the output and gradient correctly.”</p>
</blockquote>
<p>And unfortunately the linked-to  source for <code>log_softmax</code> merely includes a call to another <code>.log_softmax()</code> method which is defined somewhere else, but I have been unable to find it, even after running <code>grep -r 'def log_softmax *</code> on the pytorch directory.</p>
<p><strong>EDIT:</strong> Regarding the source, Similar post: “<a href="https://discuss.pytorch.org/t/understanding-code-organization-where-is-log-softmax-really-implemented/55299" class="inline-onebox">Understanding code organization: where is `log_softmax` really implemented?</a>”, answered by <a class="mention" href="/u/ptrblck">@ptrblck</a> as pointing to the source code here: <a href="https://github.com/pytorch/pytorch/blob/420b37f3c67950ed93cd8aa7a12e673fcfc5567b/aten/src/ATen/native/SoftMax.cpp#L146" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/420b37f3c67950ed93cd8aa7a12e673fcfc5567b/aten/src/ATen/native/SoftMax.cpp#L146</a>     …And yet all that does is call still-other functions <code>log_softmax_lastdim_kernel()</code> or <code>host_softmax</code>.   Still trying to find where the actual implementation is, not just calls-to-calls-to-calls.</p>

username: ptrblck
cooked: <p>You are right. There are two more dispatches involved and eventually <a href="https://github.com/pytorch/pytorch/blob/378881e903999342c84ad6f20224f712148e0359/aten/src/ATen/native/cpu/SoftMaxKernel.cpp#L25-L93"><code>_vec_log_softmax_lastdim</code></a> is called for the <code>log_softmax</code> with a non-scalar input.</p>

username: granth_jain
cooked: <p>Hi,</p>
<p>Is taking F.softmax and then applying torch.log same as F.log_softmax??</p>

username: ptrblck
cooked: <p>In theory these methods are equal, in practice <code>F.log_softmax</code> is numerically more stable, as it uses the log-sum-exp trick internally.</p>