A weird softmax problem

username: Lechao_Cheng
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a46b532babbc83699b7ed3fd642482cd61d1f2ff.png" data-download-href="https://discuss.pytorch.org/uploads/default/a46b532babbc83699b7ed3fd642482cd61d1f2ff" title="微信截图_20200901154055"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a46b532babbc83699b7ed3fd642482cd61d1f2ff.png" alt="微信截图_20200901154055" data-base62-sha1="nswfm0cfYRCYCE2opuOpLFBSkgf" width="689" height="258" data-dominant-color="143A43"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信截图_20200901154055</span><span class="informations">1293×484 20.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>dst_out  is the output.<br>
After I got the NaN loss, I look through the intermediate status and found the negative values in softmax output.<br>
Could anyone explain this?</p>

username: ptrblck
cooked: <p>There is no negative output after applying <code>F.softmax</code>, since <code>Y[Y&lt;0]</code> returns an empty tensor.<br>
<code>dst_out</code> would be the logits, which are not bounded to a specific range.</p>

username: Lechao_Cheng
cooked: <p>Thanks. But X is inconsistent with Y. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

username: ptrblck
cooked: <p>I don’t quite understand it. Could you explain what exactly is inconsistent?<br>
Note that the softmax function will map values from <code>[-inf, inf]</code> to <code>[0, 1]</code>.</p>

username: Lechao_Cheng
cooked: <p>When I perform X = torch.softmax(dst_out,dim=1), X contains negative values shown as line 2.</p>

username: ptrblck
cooked: <p>Ah, thanks for the pointer. I didn’t see the <code>torch.softmax</code> in the first line.<br>
Which PyTorch version are you using, as I cannot reproduce this issue with the latest nightly binary:</p>
<pre><code class="lang-python">x = torch.randn(10, 10)
out1 = torch.softmax(x, dim=1)
out2 = F.softmax(x, dim=1)
print((out1 == out2).all())
&gt; True
print(out1[out1&lt;0], out2[out2&lt;0])
&gt; tensor([]) tensor([])
</code></pre>

username: Lechao_Cheng
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/096378ba156b392e7924b3e586e45b72b1cf08c0.png" data-download-href="https://discuss.pytorch.org/uploads/default/096378ba156b392e7924b3e586e45b72b1cf08c0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/096378ba156b392e7924b3e586e45b72b1cf08c0.png" alt="image" data-base62-sha1="1l3pFV7LeAzwtZktqdwx7bQRDQk" width="690" height="258" data-dominant-color="123942"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">717×269 7.98 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>That’s quite weird!  I test the codes and got the same results. But it crashed in my framework. <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>That’s really strange indeed.<br>
Could you try to add <code>assert</code> statements to your original code and check for negative outputs again?</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Lechao_Cheng
cooked: <p>The torch vision is 1.5.0. By adding the <code>assert</code> statement,  it hits the point.<br>
By the way, it works when I change “torch.softmax(…)” to “torch.log_softmax(…).exp()” . <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:"></p>

username: ptrblck
cooked: <p>Could you update to <code>1.6</code> and rerun the code?</p>