Dealing with matrix inversion efficiently

kl_divergence: <p>I have two matrix <code>X</code> and <code>Y</code>. Both are of dimension <code>[bs, hidden_size</code>].<br>
I want to take inverse of matrix obtained by <code>torch.cat([X, Y], dim=1)</code>.</p>
<p>Pseudo inverse is not stable (docs say the same). I obtain inverse to deal with this. So I used a <code>Linear</code> layer of dimension <code>(hidden_size, batch_size//2)</code> to convert X and Y of dimensions <code>[bs, bs//2</code>]. After concat, I get a matrix of form <code>[bs, bs]</code>. The downside of this approach is that it requires me to use a <code>batch size = hidden_size/2</code> to perform well, otherwise the network doesn’t learn anything well.</p>
<p>Is there a mathematical elegant way you can suggest as a workaround ?</p>

KFrank: <p>Hello kl!</p>
<aside class="quote no-group" data-username="kl_divergence" data-post="1" data-topic="94045" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kl_divergence/48/5069_2.png" class="avatar"> kl_divergence:</div>
<blockquote>
<p>I have two matrix <code>X</code> and <code>Y</code>. Both are of dimension <code>[bs, hidden_size</code>].<br>
I want to take inverse of matrix obtained by <code>torch.cat([X, Y], dim=1)</code>.</p>
</blockquote>
</aside>
<p>Could you explain conceptually what meaning this matrix inverse<br>
is supposed to have?  Your construction seems odd.</p>
<p>As you have recognized, a matrix must be square to be invertible<br>
(although not all square matrices are invertible, of course).  But<br>
your batch size (<code>bs</code>) is something of a 'technical" parameter that<br>
doesn’t really have anything to do with your data or the structure of<br>
your network. As you point out, you have to adjust your batch size<br>
to match <code>hidden_size / 2</code>.  But how does your construction make<br>
sense if it breaks when you change your batch size?</p>
<p>Best.</p>
<p>K. Frank</p>

kl_divergence: <p>Thanks for taking interest. The matrix inversion is part of a formula that is supposed to project features orthogonally.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f0973eac2b058026a976b5429fe30be2950c0ae.png" alt="Screenshot_20200826_083836" data-base62-sha1="dyJBXopnM1OXZAEWLuCuRvNT1sG" width="301" height="46"></p>
<p><code>$F_{G}$</code> (comes from concatenation of two vectors) is not a square matrix, that’s why I make this modification.</p>
<p>The issue is, that my network performs much better if I reduce second dimensions of X and Y to half of what it was (768-&gt;384), in all other cases, it doesn’t learn anything. There might be better ways of projection, which I’ve not explored. (For instance, if bs=512, 768-&gt;512, then I don’t notice any improvement.)</p>
<p>This is an unstable method. The main issue is to perform inversion. That implies I need square matrices. That’s why I’m asking if there’s a better mathematical way to perform this.<br>
Yes, I realize that my implementation is not batch size agnostic once model is trained. As a result, I am unable to train bigger models (because batch_size = hidden_size/2 won’t fit in memory).</p>
<p>So in general sense, what is a better way to perform inversion when the matrix is not square (<code>pinverse</code> isn’t stable) ?</p>

sprakashdash: <p><code>torch.linalg.solve()</code> is a good alternative for stability purposes.</p>