Error occurs when using "softmax and log"  replace "log_softmax"

lidaiyu: <p>When I using</p>
<blockquote>
<p>Blockquote ```<br>
x = F.softmax(x, dim=1)<br>
x = torch.log(x)</p>
</blockquote>
<pre><code class="lang-auto">to replace

&gt; Blockquote ```
x = F.log_softmax(x, dim=1)
</code></pre>
<p>something wrong happend. But if I use " F.log_softmax" directly, everything went well. Then I subtracted the former and the latter, and found that they were not exactly the same.  I want to know how to divide “F.log_softmax” into “F.softmax” and “torch.log” correctly, I need the result of “F.softmax”. Thank you very much!<br>
(PS:I had subtracted the maximum to prevent the softmax overflow, but the training still reported an error)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f15e870cad445a050fc1e59c2c39af647ff2a90c.png" data-download-href="https://discuss.pytorch.org/uploads/default/f15e870cad445a050fc1e59c2c39af647ff2a90c" title="QQ图片20200330210157_看图王"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f15e870cad445a050fc1e59c2c39af647ff2a90c.png" alt="QQ图片20200330210157_看图王" data-base62-sha1="yrfARZTJD9hj2b9f0JeOLYZHmNu" width="690" height="235" data-dominant-color="121211"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">QQ图片20200330210157_看图王</span><span class="informations">929×317 16.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

vainaijr: <p>it might have to do with logsumexp trick applied in F.log_softmax,<br>
I carried following experiment,</p>
<pre><code class="lang-auto">import torch.nn as nn, torch, torch.nn.functional as F
from math import exp, log
torch.set_printoptions(precision=15)
</code></pre>
<pre><code class="lang-auto">x = torch.randn(5); x
</code></pre>
<pre><code class="lang-auto">tensor([ 2.229876756668091,  0.264560282230377, -0.100190632045269,
         0.228291451931000, -0.119905993342400])
</code></pre>
<pre><code class="lang-auto">a = torch.softmax(x, dim=0); a
</code></pre>
<pre><code class="lang-auto">tensor([0.681239962577820, 0.095449574291706, 0.066277280449867,
        0.092049762606621, 0.064983405172825])
</code></pre>
<pre><code class="lang-auto">torch.log(a)
</code></pre>
<pre><code class="lang-auto">tensor([-0.383840680122375, -2.349157094955444, -2.713908195495605,
        -2.385426044464111, -2.733623266220093])
</code></pre>
<p>while</p>
<pre><code class="lang-auto">F.log_softmax(x, dim=0)
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">tensor([-0.383840620517731, -2.349157094955444, -2.713907957077026,
        -2.385425806045532, -2.733623266220093])
</code></pre>
<p>we see a difference in values obtained,</p>
<pre><code class="lang-auto">torch.log(a) - F.log_softmax(x, dim=0)
</code></pre>
<pre><code class="lang-auto">tensor([-5.960464477539062e-08,  0.000000000000000e+00, -2.384185791015625e-07,
        -2.384185791015625e-07,  0.000000000000000e+00])
</code></pre>
<p>the first case (for 3rd value) is equivalent to,</p>
<pre><code class="lang-auto">log(exp(x[2])/(exp(x[0]) + exp(x[1]) + exp(x[2]) + exp(x[3]) + exp(x[4])))
</code></pre>
<pre><code class="lang-auto">-2.7139080429149542
</code></pre>
<p>while the second case is equivalent to,</p>
<pre><code class="lang-auto">x[2] - x[0] - log((exp(0) + exp(x[1] - x[0]) + exp(x[2] - x[0]) + exp(x[3] - x[0]) + exp(x[4] - x[0])))
</code></pre>
<pre><code class="lang-auto">-2.713907957077026
</code></pre>
<p>I think it done to avoid exponential of a large number</p>

KFrank: <p>Hello lidaiyu!</p>
<p>You most likely have two separate issues here.</p>
<aside class="quote no-group quote-modified" data-username="lidaiyu" data-post="1" data-topic="74794" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/e8c25b/48.png" class="avatar"> lidaiyu:</div>
<blockquote>
<p>When I using</p>
<blockquote>
<p>Blockquote ```<br>
x = F.softmax(x, dim=1)<br>
x = torch.log(x)</p>
</blockquote>
<pre><code class="lang-auto">to replace

&gt; Blockquote ```
x = F.log_softmax(x, dim=1)
</code></pre>
<p>something wrong happend.</p>
</blockquote>
</aside>
<p><code>log (softmax())</code> can have numerical issues.  That’s why<br>
you are better off using <code>log_softmax()</code>.  It is <em>mathematically</em><br>
the same, but <em>numerically</em> different, and, in fact, numerically<br>
more stable.</p>
<aside class="quote no-group">
<blockquote>
<p>Then I subtracted the former and the latter, and found that they were not exactly the same.</p>
</blockquote>
</aside>
<p>However, the result of the subtraction that you posted in your<br>
screenshot (Note, please post text instead of screenshots for<br>
textual information.) shows that the two differ only by a small<br>
<em>round-off</em> error.  This is to be expected.</p>
<p>Try converting your tensor <code>a</code> to <code>DoubleTensor</code><br>
(<code>dtype = torch.float64</code>) before performing the rest of<br>
the computation, and the discrepancy will get smaller (although<br>
not necessarily go to zero).</p>
<aside class="quote no-group">
<blockquote>
<p>I need the result of “F.softmax”.</p>
</blockquote>
</aside>
<p>My assumption is that you are using <code>log_softmax()</code> for training,<br>
but need <code>softmax()</code> for some other purpose.  If this is the case,<br>
keep using <code>log_softmax()</code> for training, and compute <code>softmax()</code><br>
separately.  (Or you can re-exponentiate <code>log_softmax()</code> to<br>
recover <code>softmax()</code>.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

lidaiyu: <p>Thank you for your reply！It is really helpful and gives me a deeper understanding of the details of “F.log_softmax”. I print the last linear layer’s parameters when error occurs, and the parameters are ‘nan’. It may have overflowed when executing ‘F.softmax’ or ‘log’, I will add ‘assert’ to debug the code.</p>

lidaiyu: <p>Hello KFrank!<br>
Thank you for your comprehensive reply! It is really helpful. I am really sorry for my unclear and unfriendly expression.  I think the small <em>round-off</em>  error is acceptable and not the cause of the error. It may have overflowed or some other error when executing ‘F.softmax’ or ‘log’ (But I have subtracted the maximum to prevent the softmax overflow). I will continue to debug the code for the real reason. In fact, I need to apply an attention map to the result of ‘F.softmax’ to calculate a new probability distribution. And then use the new probability distribution to calculate the loss. I willl try to re-exponentiate  <code>log_softmax()</code>  to recover  <code>softmax()</code>. Thank you again for your reply, it helps me a lot.</p>