CustomLoss Function: Outputting Inf as a loss in one iteration and NaNs in subsequent iterations of training

username: Rio1210
cooked: <p>Hello all, I am quite new to Pytorch.<br>
I am trying out my custom loss function in a simple single layer Neural network (x = 2, h_1 = 2, output =2). However, during a particular epoch’s iteration my loss function is first outputting “Inf” and "NaN"s in subsequent iterations. My Loss Function has a log, but it basically calculates distance and uses those as arguments for the log parameter. There might be subtle numerical instability. An Inf arises when there is a zero distance between columns. I can fix that by adding a little epsilon inside the log argument (which I didn’t).</p>
<p>But, what I found the issue at is that, the batch X at one iteration when goes through the loss function, outputs “inf”. But the same matrix if I just copy and pass it through the loss function as a simple torch.tensor(X), then it outputs a value, not an “inf”. I concluded that it must be because of the torch.tensor(X, grad_fn = &lt; permute_backwards&gt; ) the backward gradient might be the reason. I do not know how that works, but without the permute backward generated inside training, the matrix alone doesn’t generate Inf.</p>
<p>I would be glad if someone could take a look. I’ve linked the .ipynb here.<br>
<a href="https://drive.google.com/file/d/1JFKeRG7CWCTBK_qTuhsR6iX_vgY-1Vtk/view?usp=sharing" rel="nofollow noopener">iPy NoteBook Link</a></p>

username: ptrblck
cooked: <p>You might have skipped the problematic batch somehow?<br>
If I add <code>torch.autograd.set_detect_anomaly(True)</code> in cell12:</p>
<pre><code class="lang-python">xArray = []
iter = 0
torch.autograd.set_detect_anomaly(True)
</code></pre>
<p>the code will raise an exception, pointing to a detected invalid value. If I run the next cells afterwards manually, I’ll get an <code>Inf</code> for <code>customLoss(X.T)</code>.</p>

username: Rio1210
cooked: <p>Thanks for taking the time to go through the notebook. I really appreciate it!<br>
I am still a beginner to Pytorch, could you please elaborate a bit on what you mean, specially regarding the.<code>torch.autograd.set_detect_anomaly(True)</code> ?</p>
<p>Regarding getting the Inf at last cell, please select and copy the numerical values of the tensor from cell-17 output, and paste it inside the cell-21, assigning it to the variable <code>X_new = torch.tensor( paste here )</code>. Afterwards, if I use the Loss function on this copy-pasted variable:  X_new in cell 22, I get a valid number as output. But this same, matrix was giving Loss = inf in cell-18 and inside the main training code-block.</p>
<p>If it again gives Inf in the last cell, changing the two random.seed()s at the start only 1 or 2 times will surely generate this scenario.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/02c766a798b3d2f6bfab1b2ce006f26326318864.png" data-download-href="https://discuss.pytorch.org/uploads/default/02c766a798b3d2f6bfab1b2ce006f26326318864" title="27%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/02c766a798b3d2f6bfab1b2ce006f26326318864_2_690x388.png" alt="27%20PM" data-base62-sha1="oAavWUeaQP5gqqSyfVJtm9jtJi" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/02c766a798b3d2f6bfab1b2ce006f26326318864_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/2X/0/02c766a798b3d2f6bfab1b2ce006f26326318864_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/0/02c766a798b3d2f6bfab1b2ce006f26326318864_2_1380x776.png 2x" data-dominant-color="F1F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">27%20PM</span><span class="informations">2276×1282 293 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>After copying and pasting this matrix to the cell21, passing this numbers doesn’t generate Inf from the loss function, as it generated in Cell-18.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/304cafe7148aaaffff8d17dcceb1c7421f50fe8c.png" data-download-href="https://discuss.pytorch.org/uploads/default/304cafe7148aaaffff8d17dcceb1c7421f50fe8c" title="19%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/304cafe7148aaaffff8d17dcceb1c7421f50fe8c_2_690x315.png" alt="19%20PM" data-base62-sha1="6ThdDAT3eTOl8nTVP8X1fQAyqlu" width="690" height="315" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/304cafe7148aaaffff8d17dcceb1c7421f50fe8c_2_690x315.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/304cafe7148aaaffff8d17dcceb1c7421f50fe8c_2_1035x472.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/3/304cafe7148aaaffff8d17dcceb1c7421f50fe8c_2_1380x630.png 2x" data-dominant-color="EEF1EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">19%20PM</span><span class="informations">2270×1038 321 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What effect is grad_fn having on these numbers?</p>
<p>Thanks again!</p>