Why is my grad_fn changing when i index into the variable?

umangkaushik: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/5/45c4dc8956eb074bca9a54f04ae0ccdb87f3df9f.png" data-download-href="https://discuss.pytorch.org/uploads/default/45c4dc8956eb074bca9a54f04ae0ccdb87f3df9f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45c4dc8956eb074bca9a54f04ae0ccdb87f3df9f_2_690x305.png" alt="image" data-base62-sha1="9XcIAFsqXTcvFyufSt454FRj8Pd" width="690" height="305" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45c4dc8956eb074bca9a54f04ae0ccdb87f3df9f_2_690x305.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45c4dc8956eb074bca9a54f04ae0ccdb87f3df9f_2_1035x457.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45c4dc8956eb074bca9a54f04ae0ccdb87f3df9f_2_1380x610.png 2x" data-dominant-color="1F1F1F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1601×709 85.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
in the above image, the top cell is showing my softmax predictions, but when i index into it (to get the preds for the first input) i am not getting the exact same numbers (as shown in the cell below).</p>
<p>i want to use integer array indexing for calculating the neg. log likelihood but since the preds are not same, the NLL is coming ‘inf’.</p>

ptrblck: <p>The <code>grad_fn</code> changes since indexing is a differentiable operation.</p>
<aside class="quote no-group" data-username="umangkaushik" data-post="1" data-topic="159191">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/umangkaushik/48/52444_2.png" class="avatar"> umangkaushik:</div>
<blockquote>
<p>but when i index into it (to get the preds for the first input) i am not getting the exact same numbers (as shown in the cell below).</p>
</blockquote>
</aside>
<p>You are getting the same values, but the printing uses another format (e.g. <code>-98.1084</code> instead of <code>-9.8108e+01</code>) since the range of the slice is smaller and wouldn’t benefit from the “scientific” format.</p>
<aside class="quote no-group" data-username="umangkaushik" data-post="1" data-topic="159191">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/umangkaushik/48/52444_2.png" class="avatar"> umangkaushik:</div>
<blockquote>
<p>i want to use integer array indexing for calculating the neg. log likelihood but since the preds are not same, the NLL is coming ‘inf’.</p>
</blockquote>
</aside>
<p>That’s not true and the <code>-inf</code> value is also in the original <code>sm_pred</code> tensor but just not displayed.</p>

umangkaushik: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/9/59f0b09c638f1131eafaa7948e7f7776b0a99f02.png" data-download-href="https://discuss.pytorch.org/uploads/default/59f0b09c638f1131eafaa7948e7f7776b0a99f02" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/59f0b09c638f1131eafaa7948e7f7776b0a99f02_2_690x302.png" alt="image" data-base62-sha1="cPEazGluKxMTVsbYFhE4N6r8dZo" width="690" height="302" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/59f0b09c638f1131eafaa7948e7f7776b0a99f02_2_690x302.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/59f0b09c638f1131eafaa7948e7f7776b0a99f02_2_1035x453.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/59f0b09c638f1131eafaa7948e7f7776b0a99f02_2_1380x604.png 2x" data-dominant-color="202020"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1761×773 81.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
i am re-implementing a notebook from 2019…this is the <code>log_softmax</code> function and the <code>NLL</code> function… i am just using the exact same code from the original notebook but still my <code>preds</code> are not the same… has any functionality used in these functions changed during these years???</p>

ptrblck: <p>I’m sure a lot of things were moved around between 2019 and now, but I don’t know the exact changes in your used operations.<br>
However, your current manual log_softmax implementation is numerically not stable and I would recommend to use <code>F.log_softmax</code> which might be able to avoid the invalid outputs assuming the input does not contain any invalid values (it did before).</p>

umangkaushik: <p>Yeah i understand. But i am implementing it from scratch just to deepen my understanding of the pytorch library. At the end, i use the <code>F.log_softmax</code> function for all my projects.</p>

ptrblck: <p>I understand the idea, but in this case it would be your responsibility to check the numerical stability as your current implementation can create invalid outputs more easily:</p>
<pre><code class="lang-python">def log_softmax(x):
    return (x.exp() / (x.exp().sum(-1,keepdim=True))).log()

x = torch.tensor([[-100., 100.]])
out = log_softmax(x)
print(out)
# tensor([[-inf, nan]])

out = F.log_softmax(x, dim=1)
print(out)
# tensor([[-200.,    0.]])
</code></pre>