"Training" variables to do SVD

batlike: <p>In a bid to get familiar with PyTorch syntax, I thought I’d try and see if I can use gradient descent to do SVD - but not just the standard SVD routine, instead multidimensional scaling (MDS) which requires SVD.</p>
<p>Essentially, I generated a random <code>n x n</code> matrix <code>U</code>, a random diagonal <code>n x n</code> matrix <code>s</code>, and a random <code>n x n</code> matrix <code>Vh</code>, just as a starting point. The goal is for <code>U s Vh.T</code> to approximate some matrix <code>B</code> ie. <code>U s Vh.T ~ SVD(B)</code>.</p>
<p>I guess I’m running into two rookie pitfalls: (1) the loss is not updating after the first iteration (why?) (2) is it possible to “combine” two loss functions? Below, I “combine” the loss of the SVD approximation to the loss of the MDS approximation:</p>
<pre><code class="lang-auto"># load into pytorch
D = torch.rand(N, N).float()
H = torch.eye(N).float() - \
    1/N * torch.ones(D.shape).float()
B = 0.5 * torch.matmul(torch.matmul(H, D**2), H)
pts = torch.rand(N, 3).float()

# declare constants
stop_loss = 1e-2
step_size = stop_loss / 3

# emulate simulate SVD
U = torch.autograd.Variable(torch.rand(N, N), requires_grad = True)
s = torch.autograd.Variable(torch.diag(torch.sort(torch.rand(N), descending = True).values), requires_grad = True)
Vh = torch.autograd.Variable(torch.rand(N, N), requires_grad = True)

# find embedding
embed = torch.matmul(torch.sqrt(s), Vh)
X_hat = embed.T[:, :3] # select the first 3 coordinates --&gt; x, y, z

for i in range(100000):

    # calculate loss1: how close is our SVD function?
    delta1 = (torch.matmul(U.T, U) - torch.eye(N)) + \
            (torch.matmul(V.T, V) - torch.eye(N)) + \
            (torch.matmul(torch.matmul(U, s), Vh.T) - B)

    L1 = torch.norm(delta1, p=2)

    # calculate loss2: how close is our MDS approximation?
    delta2 = torch.nan_to_num(X_hat - pts, nan=0.0)    

    L2 = torch.norm(delta2, p=2)

    # Backprop
    loss = L1 + L2
    loss.backward()
        
    # update
    U.data -= step_size * U.grad.data
    s.data -= step_size * s.grad.data
    V.data -= step_size * V.grad.data
    
    U.data.data.zero_()
    s.data.data.zero_()
    V.data.data.zero_()
    
    if i % 1000 == 0: 
        print('Loss is %s at iteration %i' % (loss, i))
    
    if abs(loss) &lt; stop_loss:
        break
</code></pre>

JuanFMontesinos: <p>Soo some hints.<br>
Variable was deprecated couple of years ago <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=10" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"><br>
You don’t need to acess <code>.data</code> That just computes in-place modifications of the tensor.<br>
You can define an optimizer and to pass tensors:</p>
<p>Some corrections would be</p>
<pre><code class="lang-auto">import torch
N=15
# load into pytorch
D = torch.rand(N, N).float()
H = torch.eye(N).float() - \
    1 / N * torch.ones(D.shape).float()
B = 0.5 * torch.matmul(torch.matmul(H, D ** 2), H)
pts = torch.rand(N, 3).float()

# declare constants
stop_loss = 1e-2
step_size = stop_loss / 3

# emulate simulate SVD
U = torch.rand(N, N, requires_grad=True)
s = torch.diag(torch.sort(torch.rand(N), descending=True).values).requires_grad_()
Vh = torch.rand(N, N, requires_grad=True)

optimizer = torch.optim.SGD([U,s,Vh], lr=step_size)
# find embedding
embed = torch.matmul(torch.sqrt(s), Vh)
X_hat = embed.T[:, :3]  # select the first 3 coordinates --&gt; x, y, z

for i in range(100000):

    # calculate loss1: how close is our SVD function?
    delta1 = (torch.matmul(U.T, U) - torch.eye(N)) + \
             (torch.matmul(V.T, V) - torch.eye(N)) + \
             (torch.matmul(torch.matmul(U, s), Vh.T) - B)

    L1 = torch.norm(delta1, p=2)

    # calculate loss2: how close is our MDS approximation?
    delta2 = torch.nan_to_num(X_hat - pts, nan=0.0)

    L2 = torch.norm(delta2, p=2)

    # Backprop
    loss = L1 + L2
    loss.backward()

    # update
    optimizer.zero_grad()
    optimizer.step()

    if i % 1000 == 0:
        print('Loss is %s at iteration %i' % (loss, i))

    if abs(loss) &lt; stop_loss:
        break
</code></pre>
<p>If you want to do your own optimization you should zeroe the gradients each step.<br>
lastly, I don’t know whether this function is backpropagable or not <code>torch.nan_to_num(X_hat - pts, nan=0.0)   </code> But I cannot imagine why you would have NaNs</p>
<p>Just to answer your questions:</p>
<pre><code class="lang-auto">    U.data.data.zero_()
    s.data.data.zero_()
    V.data.data.zero_()
</code></pre>
<p>This is making zero the values of U,s and V each iteration. Thus the result will be static cos you are “resetting” the tensor. And yes, you can combine L=L1+L2</p>

batlike: <p>I see! Thank you so much for pointing this out!</p>
<p>Even when I declare an optimizer to step through like your suggestion above, the loss does not change. Is there a common step I am forgetting here?</p>
<pre><code class="lang-auto">import torch
torch.autograd.set_detect_anomaly(True)

N=15
# load into pytorch
D = torch.rand(N, N).float()
H = torch.eye(N).float() - \
    1 / N * torch.ones(D.shape).float()
B = 0.5 * torch.matmul(torch.matmul(H, D ** 2), H)
# pts = torch.rand(N, 3).float()

# declare constants
stop_loss = 1e-2
step_size = stop_loss / 3

# # emulate simulate SVD
# U = torch.rand(N, N, requires_grad=True)
# s = torch.diag(torch.sort(torch.rand(N), descending=True).values).requires_grad_()
# Vh = torch.rand(N, N, requires_grad=True)

optimizer = torch.optim.SGD([U,s,Vh], lr=step_size)
# find embedding
# embed = torch.matmul(torch.sqrt(s), Vh)
# X_hat = embed.T[:, :3]  # select the first 3 coordinates --&gt; x, y, z

for i in range(100000):

    # calculate loss1: how close is our SVD function?
    delta1 = (torch.matmul(U.T, U) - torch.eye(N)) + \
             (torch.matmul(Vh.T, Vh) - torch.eye(N)) + \
             (torch.matmul(torch.matmul(U, s), Vh.T) - B)

    L1 = torch.norm(delta1, p=2)

    # calculate loss2: how close is our MDS approximation?
    # # delta2 = torch.nan_to_num(X_hat - pts, nan=0.0)

    # # L2 = torch.norm(delta2, p=2)

    # Backprop
    loss = L1
    loss.backward()

    # update
    optimizer.zero_grad()
    optimizer.step()

    if i % 1000 == 0:
        print('Loss is %s at iteration %i' % (loss, i))

    if abs(loss) &lt; stop_loss:
        break
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">Loss is tensor(141.6570, grad_fn=&lt;NormBackward1&gt;) at iteration 0
Loss is tensor(141.6570, grad_fn=&lt;NormBackward1&gt;) at iteration 1000
Loss is tensor(141.6570, grad_fn=&lt;NormBackward1&gt;) at iteration 2000
Loss is tensor(141.6570, grad_fn=&lt;NormBackward1&gt;) at iteration 3000
Loss is tensor(141.6570, grad_fn=&lt;NormBackward1&gt;) at iteration 4000
...
...
etc.
</code></pre>

batlike: <aside class="quote no-group" data-username="JuanFMontesinos" data-post="2" data-topic="136441">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juanfmontesinos/48/76115_2.png" class="avatar"> JuanFMontesinos:</div>
<blockquote>
<p>If you want to do your own optimization you should zeroe the gradients each step.<br>
lastly, I don’t know whether this function is backpropagable or not <code>torch.nan_to_num(X_hat - pts, nan=0.0)   </code> But I cannot imagine why you would have NaNs</p>
</blockquote>
</aside>
<p>I see your point. Sometimes my numpy array <code>pts</code> have missing rows; my toy exercise is to try and recover those points. It looks something like:</p>
<pre><code class="lang-auto">pts 
&gt;&gt;&gt;array([[1327.769     ,  922.555     ,   86.56067961],
       [          nan,           nan,           nan],
       [          nan,           nan,           nan],
       [          nan,           nan,           nan],
       [          nan,           nan,           nan],
       [1327.34660274,  921.29142466,   78.19481833],
       [          nan,           nan,           nan],
       [          nan,           nan,           nan],
       [          nan,           nan,           nan],
       [          nan,           nan,           nan],
       [          nan,           nan,           nan],
       [1316.07260274,  920.69542466,   85.07831347]])


</code></pre>
<p>I’m trying to set the loss on those missing rows to zero (ie. <code>torch.nan_to_num(X_hat - pts, nan=0.0)</code>, while taking the real L2 norm on those non-missing values. Is there a function that allows PyTorch to consider <code>nan</code> values in its loss as 0?</p>

JuanFMontesinos: <p>So if the gradient is not NaN or None I guess it works for any value but those ones.<br>
Could you manage to get the tensors updated?</p>

batlike: <p>Doesn’t seem like it. In 2 comments above, I commented out the second loss function, and only calculated the first to avoid this problem. But still, the loss remains constant!</p>

JuanFMontesinos: <p>Sorry, my bad <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=10" title=":confused:" class="emoji" alt=":confused:"><br>
Step should be called before zeroing the grad<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=10" title=":sweat_smile:" class="emoji only-emoji" alt=":sweat_smile:"></p>
<pre><code class="lang-auto">    optimizer.zero_grad()
    optimizer.step()
</code></pre>
<pre><code class="lang-auto">import torch
import matplotlib.pyplot as plt
torch.autograd.set_detect_anomaly(True)

N=5
# load into pytorch
D = torch.rand(N, N).float()
H = torch.eye(N).float() - \
    1 / N * torch.ones(D.shape).float()
B = 0.5 * torch.matmul(torch.matmul(H, D ** 2), H)
pts = torch.rand(N, 3).float()

# declare constants
stop_loss = 1e-2
step_size = stop_loss / 3

# # emulate simulate SVD
U = torch.rand(N, N, requires_grad=True)
s = torch.diag(torch.sort(torch.rand(N), descending=True).values).requires_grad_()
Vh = torch.rand(N, N, requires_grad=True)

optimizer = torch.optim.SGD([U,s,Vh], lr=step_size)
# find embedding
embed = torch.matmul(torch.sqrt(s), Vh)
print(embed)
X_hat = embed.T[:, :3]  # select the first 3 coordinates --&gt; x, y, z
loss_hist = []
for i in range(10):
    print(f'Iteration {i}:')
    print(f'\t U_s:{U.sum()}, s:{s.sum()}, Vh_s:{Vh.sum()}')
    # calculate loss1: how close is our SVD function?
    delta1 = (torch.matmul(U.T, U) - torch.eye(N)) + \
             (torch.matmul(Vh.T, Vh) - torch.eye(N)) + \
             (torch.matmul(torch.matmul(U, s), Vh.T) - B)

    L1 = torch.norm(delta1, p=2)

    # calculate loss2: how close is our MDS approximation?
    delta2 = X_hat - pts
    # 
    L2 = torch.norm(delta2, p=2)
    print(f'\t L1: {L1}, L2: {L2}')
    # Backprop
    loss = L1
    loss.backward()
    loss_hist.append(loss.item())
    # update
</code></pre>
<p>Still if I add L2 there are NaNs but guess that’s about the maths (which i didn’t check)</p>

batlike: <p>I see…! The order matters a lot. Thank you for this feedback, I learned a lot.</p>