Non differentiability of complete QR decomposition for case where rows > columns

username: l5rk
cooked: <p>Hi,</p>
<p>I have an application where I am required to use the complete QR decomposition of a tall matrix (more specifically, what I am actually trying to do is find a basis of the null space for a matrix representing an underdetermined system of equations). However I see that the QR decomposition in torch does not support gradient for tall matrices. I do see some information in the literature regarding this, but before I reinvent the wheel, I was wondering for what reason, if any, torch does not support gradient for tall complete QR, so that I might avoid wasting any time.</p>

username: KFrank
cooked: <p>Hi Lucas!</p>
<aside class="quote no-group" data-username="l5rk" data-post="1" data-topic="200312" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/848f3c/48.png" class="avatar"> l5rk:</div>
<blockquote>
<p>what I am actually trying to do is find a basis of the null space for a matrix representing an underdetermined system of equations</p>
</blockquote>
</aside>
<p>While the null space of a tall-matrix QR decomposition is well defined,<br>
the individual basis vectors spanning that null space are not.  This is<br>
because any (typically orthogonal) transformation of that set of basis<br>
vectors produces an equally-valid basis.</p>
<aside class="quote no-group">
<blockquote>
<p>However I see that the QR decomposition in torch does not support gradient for tall matrices.</p>
</blockquote>
</aside>
<p>This is to be expected, as due to the indeterminacy of the null-space<br>
vectors, gradients of quantities that depend on the null-space vectors<br>
are ill defined.</p>
<p>(This is analogous to pytorch giving <code>nan</code>s for gradients of diagonalizable<br>
matrices that have degenerate eigenvalues, but are otherwise well<br>
behaved.)</p>
<p>Conceptually, if you were differentiating something (some kind of loss<br>
function, say) that depended solely on the null space in its entirety, but<br>
not on any specific basis for the null space, that could be differentiable.<br>
But I don’t see any way to do this with pytorch because it would seem<br>
that any such gradient computation would have to flow back through<br>
some (arbitrary) set of null-space basis vectors.</p>
<p>Without knowing more about your use case and what object you wish to<br>
differentiate, I can’t say whether your problem is ill posed, or whether it<br>
is potentially well posed (but maybe not something that autograd can<br>
handle).</p>
<p>Best.</p>
<p>K. Frank</p>

username: l5rk
cooked: <p>Hi Frank,</p>
<p>I really appreciate your response, it has been incredibly helpful. The problem I am trying to solve, generally speaking, is to make sure that my neural network will always output a valid solution. More specifically, suppose that what I am ultimately aiming to predict is a vector x. x is constrained by theory to satisfy Ax = 0. In my current implementation, I express x as a linear combination of v1, v2, … vn, where form a basis for the null space of A, and use my neural network to predict the scale factors for each of those vectors. This ensures that the solution that my neural network puts out satisfies the analytic constraints. The reason why I am moving away from this, and encountering this issue, is because I am trying to make a more generalisable model, but in order to do this, the analytic constraint becomes f(y)x = 0, where f: R^m → R^{m x n}, m &lt; n. Conceptually therefore, it does not depend on any particular basis for the null space, but the null space formed should be consistent and continuous with respect to a small perturbation in input to whatever the function is that spits out the basis set.</p>
<p>I have also been considering that perhaps forming a dummy matrix A = (f(y) | D) such that the matrix is square could enforce a degree of ‘sameness’ on the basis for the null set. I believe that I can do this because (at least under a householder based implementation) the ith column of Q should not depend on the jth column of A given i &lt; j, and I do not care about R. However, I am not sure whether this is also true for the implementation in the linalg package, although from preliminary playing around I believe it to be true. More important, and something that I am not necessarily able to intuit, is the question as to whether by keeping the dummy matrix D constant enforces continuity on the n+1th through mth columns of Q with respect to the elements of f(y).</p>
<p>Any further insight would be greatly appreciated,</p>
<p>Regards,</p>
<p>L5RK.</p>
<p>P.S - My apologies for the poor formatting - I am new to pytorch forums and have not figured out how to use LaTeX on here yet.</p>

username: KFrank
cooked: <p>Hi Lucas!</p>
<p>I sort of think I understand what you are trying to do.</p>
<aside class="quote no-group" data-username="l5rk" data-post="3" data-topic="200312" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/848f3c/48.png" class="avatar"> l5rk:</div>
<blockquote>
<p>suppose that what I am ultimately aiming to predict is a vector x. x is constrained by theory to satisfy Ax = 0.</p>
</blockquote>
</aside>
<p>In this case, as I understand it, <code>A</code> is “external” to your network and<br>
doesn’t depend on the input data (nor on any “ground-truth” target<br>
data).</p>
<aside class="quote no-group">
<blockquote>
<p>I express x as a linear combination of v1, v2, … vn, where form a basis for the null space of A, and use my neural network to predict the scale factors for each of those vectors.</p>
</blockquote>
</aside>
<p>(I believe that your null space has dimensionality <code>n - m</code> so the <code>v</code>s should<br>
run up to <code>v_(n - m)</code>.)</p>
<p>Because <code>A</code> is an external constant, you can pre-compute some (arbitrary)<br>
basis of <code>A</code>’s (well-defined) null space, <code>v1, v2, ...</code>.  You then use your<br>
network to predict the coordinates of your output vector, <code>x</code>, with respect<br>
to that basis.  Because the specific choice of basis is a pre-computed<br>
constant, those predicted coordinates always have the same meaning,<br>
so it makes sense to train your network to predict them.</p>
<p>Based on your comments, I assume that this version of your scheme is<br>
working for you.</p>
<aside class="quote no-group">
<blockquote>
<p>I am trying to make a more generalisable model, but in order to do this, the analytic constraint becomes f(y)x = 0, where f: R^m → R^{m x n}, m &lt; n.</p>
</blockquote>
</aside>
<p>I assume that the function <code>f</code> is “external” and doesn’t depend on the model<br>
or any of the data.</p>
<p>Where does <code>y</code> come from?  Is it part of the input data?  Is it part of the<br>
ground-truth target data?  Or is it also predicted by some part of your<br>
trained network (or itself a trained parameter)?</p>
<p>Unless <code>y</code> (and hence <code>f (y)</code>) depends on some trained parameters<br>
(and is used in the training of those parameters), I don’t see why the<br>
differentiability of the QR decomposition would enter into your problem.</p>
<aside class="quote no-group">
<blockquote>
<p>it does not depend on any particular basis for the null space, but the null space formed should be consistent and continuous with respect to a small perturbation in input to whatever the function is that spits out the basis set.</p>
</blockquote>
</aside>
<p>So, to confirm my understanding, you have a null space that is determined<br>
by <code>f (y)</code> (formerly the matrix <code>A</code>), which can vary (but “continuously”).  Your<br>
network predicts the coordinates of your output vector, <code>x</code>, with respect to<br>
some conceptually-arbitrary basis of that null space.  In order for those<br>
predicted coordinates to have some stable (and, hence, predictable)<br>
meaning, you need your null-space basis to depend “continuously” on <code>f (y)</code>.</p>
<p>Here’s a scheme that I think might suffice to break the arbitrariness of the<br>
null-space basis “by hand”:</p>
<p>Your predicted vector, <code>x</code>, conceptually has dimension <code>n</code>.  But it satisfies<br>
the <code>m</code> constraints given by <code>f (y) @ x = 0</code>, where <code>f (y)</code> is a matrix of<br>
dimension <code>[m, n]</code>, so the overall constraint is that <code>x</code> lie in the null space<br>
of <code>f (y)</code>.  Therefore <code>x</code> only has <code>n - m</code> actual degrees of freedom (and<br>
<code>n - m</code> is the dimension of the null space).</p>
<p>Let <code>k = n - m</code> denote the dimension of the null space.  Generate <code>k</code> random<br>
vectors of dimension <code>n</code>.  (Concretely I have in mind generating gaussian<br>
vectors and normalizing them to have unit length.  I would probably use<br>
low-discrepancy “random” numbers to get more uniform coverage of the<br>
<code>n</code>-dimensional space.)</p>
<p>It is <em>highly unlikely</em> that any of these vectors will be orthogonal to the null<br>
space.</p>
<p>Call these vectors <code>c1, c2, ..., ck</code>.  They are pre-computed and held<br>
constant, as they do not depend on <code>f (y)</code>.  Think of them as the “seeds”<br>
for a smoothly-varying basis of the smoothly-varying null space of <code>f (y)</code>.</p>
<p>For a given <code>f (y)</code>, compute its null space from its QR decomposition.<br>
(I am ignoring differentiability here.)  Project the <code>c1, c2, ...</code> (the seed<br>
basis) to obtain <code>b1, b2, ...</code>, the desired smoothly-varying null-space<br>
basis.</p>
<p>(I imagine that it could be helpful to orthogonalize the <code>bi</code>.  If you iteratively<br>
orthogonalize the <code>bi</code> – perhaps purposefully with a small step size – the<br>
resulting orthogonal <code>bi</code> should remain smoothly varying as a function of<br>
<code>f (y)</code>.)</p>
<p>If you require autograd-differentiability of the <code>bi</code> with respect to <code>f (y)</code>,<br>
you could use the following modified scheme.  Compute the reduced<br>
QR decomposition, which is differentiable.  This gives you a (well-defined)<br>
basis for the “non-null space” of <code>f (y)</code>.  Obtain the <code>bi</code> by projecting the<br>
<code>ci</code> <em>out of</em> the non-null space, and hence into the null space, by using<br>
the non-null-space basis given by the reduced QR decomposition.</p>
<p>My apologies if I’ve misunderstood somewhere along the way what you<br>
are trying to do.</p>
<p>It would be helpful if you could describe the conceptual details of your<br>
use case.  What do the input data to your network actually mean?  What<br>
do the target data mean?  How much data do you have?  What does<br>
the matrix <code>A</code> actually mean, where does it come from, and what is its<br>
dimensionality?  What kind of structure does your constraint function, <code>f</code>,<br>
have, and where does it come from? What is the architecture of your<br>
network and what optimizer are you using?</p>
<p>Good luck!</p>
<p>K. Frank</p>

username: l5rk
cooked: <p>Hi K,</p>
<p>I believe you’ve understood the problem quite well. I can add some clarifications for completeness, but I think that you’ve given me a pretty good starting point for me to implement an early solution!</p>
<blockquote>
<p>I assume that the function <code>f</code> is “external” and doesn’t depend on the model<br>
or any of the data.</p>
</blockquote>
<p>This is true - my problem more specifically is as follows. Given a set of data points (prices) associated with an upper and lower bounds (ask and bid prices respectively), I am trying to find a set of C2 continuous cubic splines that match the data well. The cubic splines are joined at knots at <code>(x_0, x_1, ... x_l)</code> and the original matrix A enforces C2 continuity between the splines at a predetermined set of <code>xs</code>. My current goal is to make the placement of the knots a learnable property, based on the data, hence y is an output of the trainable part of the net. This should mean that I should get <code>f(predicted knot location) = (a system that enforces continuity)</code>. Right now the structure of trainable part of the net is very basic (A stack of LeakyReLU and fully connected linear layers), and I am using SGD for simplicity as well. Because I want the errors of my loss function to propagate <em>through</em> f(y) such that the placement of the splines can be learned, I do require autograd compatability.</p>
<blockquote>
<p>If you require autograd-differentiability of the <code>bi</code> with respect to <code>f (y)</code> ,<br>
you could use the following modified scheme. Compute the reduced<br>
QR decomposition, which is differentiable. This gives you a (well-defined)<br>
basis for the “non-null space” of <code>f (y)</code> . Obtain the <code>bi</code> by projecting the<br>
<code>ci</code> <em>out of</em> the non-null space, and hence into the null space, by using<br>
the non-null-space basis given by the reduced QR decomposition.</p>
</blockquote>
<p>This solution I think addresses my problem perfectly. The only thing that I remain mildly concerned about is the numerical stability of these projections, although I suppose that it is not something easily avoidable in a general sense.</p>
<p>Thank you so much for your time,</p>
<p>Regards,<br>
Lucas.</p>

username: KFrank
cooked: <p>Hi Lukas!</p>
<aside class="quote no-group" data-username="l5rk" data-post="5" data-topic="200312" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/848f3c/48.png" class="avatar"> l5rk:</div>
<blockquote>
<p>the original matrix A enforces C2 continuity between the splines at a predetermined set of <code>xs</code>. My current goal is to make the placement of the knots a learnable property</p>
</blockquote>
</aside>
<p>If the knot locations encoded in <code>A</code> are in roughly similar spots as the<br>
learnable knots, you might get some benefit by using an orthonormal<br>
basis of <code>A</code> 's null space as your “seed” basis.</p>
<aside class="quote no-group">
<blockquote>
<p>The only thing that I remain mildly concerned about is the numerical stability of these projections</p>
</blockquote>
</aside>
<p>I don’t have a clear understanding of the C2 continuity conditions, but I<br>
would imagine that they and the matrix <code>f (y)</code> would become degenerate<br>
and / or singular when knots coincide.  You could perhaps add a regulator<br>
(as a loss term for <code>y</code>) that causes the knots to repel one another.</p>
<p>Again, based on whether or not <code>A</code> is somewhat similar to <code>f (y)</code>, you could<br>
even add a regulator that causes the predicted knot locations (with some<br>
specific labelling scheme) to be attracted to the corresponding knot locations<br>
of <code>A</code>.  This could also help stabilize the knots to remain “sensible” and not<br>
become singular.</p>
<p>Best.</p>
<p>K. Frank</p>

username: spapanic
cooked: <p>While the null-space vectors are not uniquely defined, a given algorithm will generate them in a specific way that can actually be differentiable. I have just presented some results on this in <a href="https://arxiv.org/abs/2409.13374" class="inline-onebox" rel="noopener nofollow ugc">[2409.13374] Derivatives of the Full QR Factorisation and of the Compact WY Representation</a>, which I hope can eventually be implementedin PyTorch.</p>