PCA using torch.svd() not converging error

Harsha_1412: <p>I was trying to do PCA using PyTorch, to reduce dimensionality.<br>
I am getting the following error</p>
<p><em>RuntimeError: svd_cuda: the updating process of SBDSDC did not converge (error: 17)</em></p>
<p>This is occurring even if convert my tensor to cpu().<br>
Also when I try removing NaNs in the input to torch.svd(), there is no error but loss(CrossEntropy) is NaN.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/5/95334df8b5851504b7bc48466a2f967f1df7b37c.png" data-download-href="https://discuss.pytorch.org/uploads/default/95334df8b5851504b7bc48466a2f967f1df7b37c" title="Screenshot 2020-07-15 at 3.11.37 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95334df8b5851504b7bc48466a2f967f1df7b37c_2_690x184.png" alt="Screenshot 2020-07-15 at 3.11.37 PM" data-base62-sha1="lhT3SNXlEjBelQRP3QDhWhhi1JO" width="690" height="184" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95334df8b5851504b7bc48466a2f967f1df7b37c_2_690x184.png, https://discuss.pytorch.org/uploads/default/original/3X/9/5/95334df8b5851504b7bc48466a2f967f1df7b37c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/5/95334df8b5851504b7bc48466a2f967f1df7b37c.png 2x" data-dominant-color="E6E6E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-07-15 at 3.11.37 PM</span><span class="informations">960×256 44 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I would recommend to first try to fix the NaN loss.<br>
Do your model outputs or targets contain any NaNs and are you expecting it during training?</p>

Harsha_1412: <p>No neither my model outputs nor targets contain any NaNs.I am not expecting it during training.I also tried gradient clipping but that made no difference.</p>

ptrblck: <p>Where do the NaNs in the input to <code>svd</code> come from, if neither from the output nor from the target?<br>
How is <code>node_embeddings</code> calculated?</p>

Harsha_1412: <p>‘‘node_embeddings’’ is the output of a graph neural network, of shape (n x f) where n is the no. of nodes in a particular graph and f is the no.of features.n is around 30 and f is around 250 and I want to reduce f to 100.<br>
The experiment runs successfully if I don’t do svd. That is if I directly pass all features through the subsequent fully connected layers, no errors occur, no NaNs are encountered.Only when I try to reduce f using PCA and then process further, this issue is occurring.</p>

ptrblck: <p>I might have misunderstood the issue, but in the first post you claim, that you tried to "remove the NaNs in the input to <code>torch.svd()</code>".<br>
Based on your description now it seems that you are suddently getting NaNs in the model output, if you are trying to apply <code>svd</code>?</p>

Harsha_1412: <p>I “suspected” there might be NaNs and so zeroed them out just to be safe.<br>
But then figured out that I am suddenly getting NaNs in the model output, when I am trying to apply svd.</p>