Nullspace of a tensor

username: Abdelpakey
cooked: <p>Hi</p>
<p>I’m looking for a method to calculate the nullspace of a tensor and has a gradient (.backward()).</p>
<p>For example, Pytorch has torch.symeig method to calculate eigenvalues and eigenvectors and I can backprop.  However, I can’t find something similar for Nullspace.</p>
<p>Any suggestions would be appreciated!</p>

username: albanD
cooked: <p>Hi,</p>
<p>I don’t think we have such think.<br>
You would be looking for the vectors that form a base of the nullspace?</p>

username: Abdelpakey
cooked: <p>Yes.</p>
<p>I already wrote my own function in a differential way “hopefully”<br>
Here is:</p>
<p>def my_nullspace(At, rcond=None):</p>
<pre><code>  ut, st, vht = torch.Tensor.svd(At, some=False,compute_uv=True)
vht=vht.T        
Mt, Nt = ut.shape[0], vht.shape[1] 
if rcond is None:
    rcondt = torch.finfo(st.dtype).eps * max(Mt, Nt)
tolt = torch.max(st) * rcondt
numt= torch.sum(st &gt; tolt, dtype=int)
nullspace = vht[numt:,:].T.cpu().conj() ===&gt; problem here
# nullspace.backward(torch.ones_like(nullspace),retain_graph=True)
return nullspace
</code></pre>
<p>Everything works fine even when I use .backword() from nullspace (in case  nullspace = vht.T.cpu().conj() ) , however when I take just a part from the tensor vht (i.e., vht[numt:,:] ) the gradients become zero I dont know why shoudld I call it like this  vht[numt:,:].clone() or there is smoothing wrong?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Here is my test script, it seems to work fine no?</p>
<pre><code class="lang-python">import torch

a = torch.rand(10, 2)

A = torch.mm(a, a.t()) # rank 2 input to have a nullspace of size 8


def my_nullspace(At, rcond=None):

    ut, st, vht = torch.Tensor.svd(At, some=False,compute_uv=True)
    vht=vht.T        
    Mt, Nt = ut.shape[0], vht.shape[1] 
    if rcond is None:
        rcondt = torch.finfo(st.dtype).eps * max(Mt, Nt)
    tolt = torch.max(st) * rcondt
    numt= torch.sum(st &gt; tolt, dtype=int)
    nullspace = vht[numt:,:].T.cpu().conj()
    # nullspace.backward(torch.ones_like(nullspace),retain_graph=True)
    return nullspace

A.requires_grad_()
out = my_nullspace(A)

print("out size, should be 10x8: ", out.size())

out.sum().backward()
print("A.grad", A.grad)
</code></pre>

username: Abdelpakey
cooked: <p>It works fine, thank you.</p>
<p>Just realised that the problem wasn’t pytorch-wise.<br>
I had a saved matrix-variable with the same name in the environment of python which was (6*6)  so the null space of it was always Zero.</p>

username: Leockl
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a> and <a class="mention" href="/u/abdelpakey">@Abdelpakey</a>,</p>
<p>I am relatively new to PyTorch and deep learning in general and was wondering what are these 2 pieces of code that <a class="mention" href="/u/alband">@albanD</a> had is doing?</p>
<pre><code class="lang-auto">A.requires_grad_()   # what is this doing?
out = my_nullspace(A)
</code></pre>
<pre><code class="lang-auto">out.sum().backward()   # what is this doing?
print("A.grad", A.grad)
</code></pre>
<p>Would really appreciate if you could help. Many thanks in advance!</p>

username: Abdelpakey
cooked: <aside class="quote no-group" data-username="Leockl" data-post="6" data-topic="69980">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/e9a140/48.png" class="avatar"> Leockl:</div>
<blockquote>
<pre><code class="lang-auto">A.requires_grad_()   # what is this doing?
out = my_nullspace(A)
</code></pre>
</blockquote>
</aside>
<p>It tells Pytorch that the “A” variable can be part of the optimisation process. In other words, makes it differentiable and updates it in the gradient descend.</p>
<aside class="quote no-group" data-username="Leockl" data-post="6" data-topic="69980">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/e9a140/48.png" class="avatar"> Leockl:</div>
<blockquote>
<p><code>out.sum().backward()   # what is this doing?</code></p>
</blockquote>
</aside>
<p>It is used as loss function, in general to back prop from the loss function to network weights your loss whatever it is should be differentiatabke and a SCORE measure. For example, loss= value, where value here is just a tensor number.</p>
<p>Hope this helps.</p>

username: Leockl
cooked: <p>Many thanks <a class="mention" href="/u/abdelpakey">@Abdelpakey</a>! Your explanation for these 2 lines of code makes sense.</p>
<p>However, I am just confused with your function <code>my_nullspace()</code> where it is just calculating the null space of a matrix, so why would calculating the null space of a matrix involve things like making the “A” variable/matrix differentiable and also things like back prop the loss function?</p>
<p>Many thanks again <a class="mention" href="/u/abdelpakey">@Abdelpakey</a>.</p>