Nan in U@torch.diag(S)@V.t() for huge data

username: Ahmad_Khan
cooked: <p>I am trying to get back my input using SVD, getting nan values in U@torch.diag(S)<span class="mention">@V.t</span>()</p>

username: AlphaBetaGamma96
cooked: <p>Do you mind sharing a bit more about your data?</p>
<p>Also, check whether you’re using <code>torch.svd</code> or <code>torch.linalg.svd</code> they return different matrices.</p>

username: Ahmad_Khan
cooked: <p>A set of 1000 224*224 images of imagenet, using torch.svd</p>

username: AlphaBetaGamma96
cooked: <p>Would you mind sharing some code so it’s easier for me understand where bug appears?</p>
<p>Also, check if there are any <code>NaNs</code> in your data (via: <code>torch.isnan(x).any()</code>) before applying <code>torch.svd</code>. If you’re normalising your data check the variance is non-zero, because that’ll introduce a divide by zero error. And, I assume you’re doing all 1000 images in one go? So, you need to use <code>torch.diag_embed(S)</code> rather than <code>torch.diag(S)</code> instead!</p>

username: Ahmad_Khan
cooked: <p>Normalising the image by subtracting the mean image (of all the image), from each image.<br>
X = images.reshape(nImage, h*w)<br>
Norm_image = X - mean_image<br>
U, S, V = torch.svd(torch.from_numpy(Norm_image.astype(np.double)).cuda(), some=False)<br>
Checked that variance is non zero.</p>

username: Ahmad_Khan
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> any solution for this.</p>

username: AlphaBetaGamma96
cooked: <p>Sorry, I didn’t get a ping for this. Could you check if any of the images are full of zeros? (after you apply the mean-centering?). And, check there are no <code>infs</code> or <code>NaNs</code> in your image before preconditioning.</p>
<pre><code class="lang-auto">images = torch.from_numpy(Norm_image.astype(np.double)).cuda() #shape [1000,255,255]

print(torch.isnan(images).any())
print(torch.isinf(images).any())
</code></pre>
<p>Also, could you try the following code and see if the NaN issue goes away?</p>
<pre><code class="lang-auto">#use torch.linalg.svd instead of torch.svd
#this returns V already transposed (so more efficient)
U, S, VT = torch.linalg.svd(images) 
images_from_svd = U @ S.diag_embed() @ VT 
</code></pre>
<aside class="quote no-group" data-username="Ahmad_Khan" data-post="5" data-topic="132827">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ahmad_khan/48/11986_2.png" class="avatar"> Ahmad_Khan:</div>
<blockquote>
<p>X = images.reshape(nImage, h*w)</p>
</blockquote>
</aside>
<p>Is this reshape right, perhaps you could be introducing the NaN when mean-centering your images? If Images is shape [B,W,H] you could just do <code>image_mean = images.mean(dim=(-1,-2))</code> then <code>image_norm = image - image_mean</code> (which is then passed to <code>torch.linalg.svd(image_norm)</code>)</p>
<p>Also, remember to write code wrapped around 3 backticks ``` so it’s shown nicely! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: Ahmad_Khan
cooked: <p>The reason why I moved to torch.svd is because of gpu memory limit, torch.linalg.svd was not able to load the data . Is it possible to run it in data parallel mode.</p>

username: AlphaBetaGamma96
cooked: <p>Did you check the other things I mentioned?</p>
<p>You can just do <code>images_from_svd</code> comment I had above but just remember to transpose <code>V</code></p>