Function ‘LinalgSvdBackward0’ returned nan values in its 0th output

xianggkl: <pre><code class="lang-auto">    @staticmethod
    def PSVT(X, tau, r=50):
        # X_1 = X + (torch.eye(X.size(0)) * 1e-5).cuda()
        [U, S, V] = torch.svd(X)
        V = torch.t(V)
        Xd = torch.mm(torch.mm(U[:, 0:r], torch.diag(torch.complex(S[0:r], torch.tensor(0.0).cuda()))), torch.conj(torch.t(torch.t(V)[:, 0:r])))
        diagS = nn.functional.relu(S[r:] - tau)
        diagS = torch.squeeze(diagS[torch.nonzero(diagS)])
        svp = np.prod(list(diagS.shape))
        if svp &gt;= 1:
            diagS = torch.complex(diagS, torch.zeros_like(diagS))
            Xd = Xd + torch.mm(torch.mm(U[:, r:r + svp], torch.diag(diagS)), torch.conj(torch.t(torch.t(V)[:, r:r + svp])))
        return Xd
    
    def forward(self, t1, tau):
        output = torch.zeros(self.shape).cuda()
        P_fft_svt = torch.zeros(self.shape, dtype=torch.complex64).cuda()
        for b in range(self.batchsize):
            P = t1[b]
            P_fft = torch.fft.fft(P, dim=0)
            for c in range(self.channel):
                P_fft_svt[b,c,:,:] = self.PSVT(P_fft[c,:,:], tau)
            output[b,:,:,:] = torch.fft.irfft(P_fft_svt[b,:,:,:], n=3, dim=0)
        return output
</code></pre>
<p>I’m using a truncated svd decomposition on the input variable Z. The loss function is the mse of the reconstructed Z and the groundtruth, which leads to a RuntimeError: function ‘LinalgSvdBackward0’ returned nan values in its 0th output.<br>
<strong>This problem has been explained by KFrank, but due to my fault that I ask this question in the other topic, so I create a new one.</strong><br>
<strong>I felt bad for the mistake I made due to my lack of understanding of the community rules.<br>
Below is the old topic.</strong><br>
<a href="https://discuss.pytorch.org/t/function-linalgeighbackward0-returned-nan/185813">Function ‘LinalgEighBackward0’ returned nan</a></p>
<p>But I still have some problems about this: in my network, the input t1 equals to W - Q/u, and tau equals to 1/u, and only the u is the parameter which needs to learn, W and Q are inited by torch.zeros, so Pytorch seems not to take gradients with respect to U or V.<br>
I would like to ask you for guidance on this issue. Thank you.</p>

KFrank: <p>Hi Aya!</p>
<aside class="quote no-group quote-modified" data-username="xianggkl" data-post="1" data-topic="190410" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xianggkl/48/64829_2.png" class="avatar"> xianggkl:</div>
<blockquote>
<pre><code class="lang-auto">    def PSVT(X, tau, r=50):
        [U, S, V] = torch.svd(X)
        V = torch.t(V)
        Xd = torch.mm(torch.mm(U[:, 0:r], torch.diag(torch.complex(S[0:r], torch.tensor(0.0).cuda()))), torch.conj(torch.t(torch.t(V)[:, 0:r])))
        ...
        return Xd
</code></pre>
</blockquote>
</aside>
<p>The return value of <code>PSVT()</code> depends on <code>svd()</code>'s <code>U</code> and <code>V</code>.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    def forward(self, t1, tau):
        ...
            P = t1[b]
            P_fft = torch.fft.fft(P, dim=0)
            for c in range(self.channel):
                P_fft_svt[b,c,:,:] = self.PSVT(P_fft[c,:,:], tau)
            output[b,:,:,:] = torch.fft.irfft(P_fft_svt[b,:,:,:], n=3, dim=0)
        return output
</code></pre>
</blockquote>
</aside>
<p><code>P</code> and therefore <code>P_fft</code> depend on <code>t1</code> (which you say depends on <code>u</code>), so<br>
when you pass <code>P_fft</code> to <code>PSVT()</code>, <code>U</code> and <code>V</code> inside of <code>PSVT()</code> depend on<br>
<code>t1</code> (and hence <code>u</code>).</p>
<aside class="quote no-group">
<blockquote>
<p>RuntimeError: function ‘LinalgSvdBackward0’ returned nan values in its 0th output.</p>
</blockquote>
</aside>
<p>As noted in my post in the linked thread, this can happen if the matrix passed<br>
to <code>svd()</code> has degenerate singular values.</p>
<aside class="quote no-group">
<blockquote>
<p>the input t1 equals to W - Q/u, and tau equals to 1/u, and only the u is the parameter which needs to learn<br>
…<br>
Pytorch seems not to take gradients with respect to U or V.</p>
</blockquote>
</aside>
<p>As explained above, the <code>output</code> of your <code>forward()</code> function depends<br>
on its input <code>t1</code> (and hence <code>u</code>) through the call to <code>PSVT()</code> and, internal to<br>
<code>PSVT()</code>, on <code>svd()</code>'s <code>U</code> and <code>V</code> matrices.</p>
<p>So when autograd uses the chain rule to compute the gradient of <code>some_loss</code><br>
with respect to <code>u</code> when you call <code>some_loss.backward()</code>, autograd does take<br>
gradients with respect to <code>U</code> and <code>V</code>.</p>
<p>You have to reformulate your problem somehow so that (near) degenerate<br>
singular values don’t appear in your argument to <code>svd()</code> or so that the result<br>
doesn’t depend  on <code>U</code> nor <code>V</code>. (Your result may depend on the singular values,<br>
<code>S</code>, themselves, because those gradients don’t diverge, but not on <code>U</code> nor <code>V</code>.)</p>
<p>(As an aside, you should upgrade your code to use <code>torch.linalg.svd()</code><br>
because <code>torch.svd()</code> is deprecated.)</p>
<p>Best.</p>
<p>K. Frank</p>