Function 'LinalgEighBackward0' returned nan

dongtiange: <p>File “/lustre/home/tgdong/ML-QMMM/pytorch_heat_ref_new_onlyR_nucdependNN_moreelement/AM1_model_force_AM1dm_NN_ri_encode_smoothri.py”, line 769, in forward<br>
e, c = torch.linalg.eigh(F_new+1.0e-9)<br>
(Triggered internally at  …/torch/csrc/autograd/python_anomaly_mode.cpp:104.)<br>
inputs, allow_unused, accumulate_grad=False)<br>
Traceback (most recent call last):<br>
File “pytorch_lossin_force_allmolineachbatch_bigmol_readmask_nucparm_am_smoothri_diffweightri_w100_sigma.py”, line 560, in <br>
train_loss1= train_loop_RMSE(training_data,mol_ind_all,model, loss_fn, optimizer,lambda_AM1inti,penaltyAM1,batch_size,w_E,w_chg,w_dip,w_F)<br>
File “pytorch_lossin_force_allmolineachbatch_bigmol_readmask_nucparm_am_smoothri_diffweightri_w100_sigma.py”, line 185, in train_loop_RMSE<br>
pred_E,pred_E_HEAT,out_correction,pred_F = model(x)<br>
File “/lustre/software/anaconda/anaconda3-2019.10-py37/envs/pytorch-gpu-1.3.1-py37/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 1102, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/lustre/home/tgdong/ML-QMMM/pytorch_heat_ref_new_onlyR_nucdependNN_moreelement/AM1_model_force_AM1dm_NN_ri_encode_smoothri.py”, line 843, in forward<br>
grad_e = -torch.autograd.grad((e_heat_formation).sum(), input_atomcoord, create_graph=True)[0]<br>
File “/lustre/software/anaconda/anaconda3-2019.10-py37/envs/pytorch-gpu-1.3.1-py37/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 236, in grad<br>
inputs, allow_unused, accumulate_grad=False)<br>
RuntimeError: Function ‘LinalgEighBackward0’ returned nan values in its 0th output.</p>
<pre><code class="lang-auto">type or paste code here
</code></pre>

KFrank: <p>Hi Dong!</p>
<aside class="quote no-group" data-username="dongtiange" data-post="1" data-topic="185813" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/bc79bd/48.png" class="avatar"> dongtiange:</div>
<blockquote>
<p>RuntimeError: Function ‘LinalgEighBackward0’ returned nan values in its 0th output.</p>
</blockquote>
</aside>
<p>It would appear that you are calling <code>torch.linalg.eigh()</code> in your forward<br>
pass.  Note that when an otherwise well-behaved eigenvector / eigenvalue<br>
problem has eigenvalues that are (nearly) degenerate, gradients with<br>
respect to eigenvectors become ill defined.</p>
<p>(In this case, the term <em>degenerate eigenvalues</em> means eigenvalues that are<br>
equal to one another.)</p>
<p>You might try printing out the eigenvalues during the forward pass and see<br>
whether any of them are close to being degenerate.</p>
<p>Quoting from the <a href="https://pytorch.org/docs/2.0/generated/torch.linalg.eigh.html" rel="noopener nofollow ugc">eigh() documentation:</a></p>
<blockquote>
<p>Warning</p>
<p>Gradients computed using the eigenvectors tensor will only be finite when A has distinct eigenvalues. Furthermore, if the distance between any two eigenvalues is close to zero, the gradient will be numerically unstable, as it depends on the eigenvalues λi through the computation of 1 / min ⁡i≠j λi−λj ​.</p>
</blockquote>
<p>(The core issue is that when eigenvalues are degenerate, individual<br>
eigenvectors are no longer uniquely defined.  Instead, eigen-subspaces<br>
are uniquely defined, but the choice of which eigenvectors within a given<br>
eigen-subspace to use as its basis becomes arbitrary.)</p>
<p>If you have degenerate eigenvalues and you take the gradient of something<br>
(for example, a loss function) that depends on the <em>eigenvectors,</em> you will<br>
inevitably meet this problem.  It is not merely a “numerical issue” nor a<br>
problem with pytorch’s <code>eigh()</code> implementation.</p>
<p>You <em>can</em> take the gradient of something that depends on the <em>eigenvalues</em><br>
(and not the eigenvectors themselves), but if this is your use case, you<br>
might want to make this explicit by using <a href="https://pytorch.org/docs/2.0/generated/torch.linalg.eigvalsh.html" rel="noopener nofollow ugc">eigvalsh()</a> to compute just the<br>
eigenvalues.</p>
<p>Best.</p>
<p>K. Frank</p>

dongtiange: <p>Thank you very much, the reason is what you said, but this problem seems to be no way to solve, can only skip the sample for training.</p>

xianggkl: <p>Hi Frank! I’m currently experiencing a problem similar to this one: i’m using a truncated svd decomposition on the input variable Z. The loss function is the mse of the reconstructed Z and the groundtruth, which leads to a RuntimeError: function ‘LinalgSvdBackward0’ returned nan values in its 0th output. could you explain the reason for this in more detail for me？<br>
I would be very grateful if you could help me with the explanation.</p>

xianggkl: <p>def PSVT(X, tau, r=50):<br>
# X_1 = X + (torch.eye(X.size(0)) * 1e-5).cuda()<br>
[U, S, V] = torch.svd(X)<br>
V = torch.t(V)<br>
Xd = torch.mm(torch.mm(U[:, 0:r], torch.diag(torch.complex(S[0:r], torch.tensor(0.0).cuda()))), torch.conj(torch.t(torch.t(V)[:, 0:r])))<br>
diagS = nn.functional.relu(S[r:] - tau)<br>
diagS = torch.squeeze(diagS[torch.nonzero(diagS)])<br>
svp = np.prod(list(diagS.shape))<br>
if svp &gt;= 1:<br>
diagS = torch.complex(diagS, torch.zeros_like(diagS))<br>
Xd = Xd + torch.mm(torch.mm(U[:, r:r + svp], torch.diag(diagS)), torch.conj(torch.t(torch.t(V)[:, r:r + svp])))<br>
return Xd</p>
<pre><code>def forward(self, t1, tau):
    output = torch.zeros(self.shape).cuda()
    P_fft_svt = torch.zeros(self.shape, dtype=torch.complex64).cuda()
    for b in range(self.batchsize):
        P = t1[b]
        P_fft = torch.fft.fft(P, dim=0)
        for c in range(self.channel):
            P_fft_svt[b,c,:,:] = self.PSVT(P_fft[c,:,:], tau)
        output[b,:,:,:] = torch.fft.irfft(P_fft_svt[b,:,:,:], n=3, dim=0)
    return output
</code></pre>
<p>some codes here</p>

KFrank: <p>Hi Aya!</p>
<aside class="quote no-group" data-username="xianggkl" data-post="4" data-topic="185813" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xianggkl/48/64829_2.png" class="avatar"> xianggkl:</div>
<blockquote>
<p>eads to a RuntimeError: function ‘LinalgSvdBackward0’ returned nan values in its 0th output.</p>
</blockquote>
</aside>
<p>This is essentially the same issue as in the <code>linalg.eigh()</code> case.</p>
<p>Singular-value decomposition is a a generalization of eigendecomposition and<br>
when singular values (the analogs of eigenvalues) become degenerate, gradients<br>
with respect to <code>U</code> and <code>V</code> (or <code>Vh</code>) will diverge. The only real solution is to modify<br>
your approach so that you do not take gradients with respect to <code>U</code> or <code>V</code> in cases<br>
where your singular values might become degenerate.</p>
<p>(See the “Warning” about this in the <a href="https://pytorch.org/docs/2.0/generated/torch.linalg.svd.html" rel="noopener nofollow ugc">torch.linalg.svd() documentation.</a>)</p>
<p>Note that <code>torch.svd()</code> has been deprecated in favor of <code>torch.linalg.svd()</code>,<br>
which has a slightly different syntax.</p>
<p>As an aside, it probably would have been better to start a new thread about<br>
this rather than adding a post to the <code>linalg.eigh()</code> thread.  (You can always<br>
link to an old thread if you think it provides helpful context.)</p>
<p>Best.</p>
<p>K. Frank</p>

xianggkl: <p>Hi Frank!</p>
<p>Thank you for your guidance! I appreciate your patience and I’m committed to learning and abiding by the rules in the future.<br>
I have start a new thread about this and I still have some questions, could you please take a look at my question and provide me with an answer in my new thread?</p>
<p>Best.<br>
Aya</p>