Using Torch to optimize a string of matrix products

Albert_Akhriev: <p>Dear All,<br>
I would like to employ PyTorch to optimize a string of matrix products:</p>
<p>min Frobenius_norm( M_1(p_1) * M_2(p_2) * M_3(p_3) * … * M_n(p_n)  -  T )</p>
<p>where:<br>
M_i(p_i) is the i-th matrix of known structure parameterized by the vector of parameters p_i (unique for every matrix),<br>
T is some target matrix we want to approximate by the product,<br>
n is the “depth” of the product,<br>
and the goal is to minimize Frobenius norm of difference between the target and approximating string of matrices.</p>
<p>It is not difficult to write down a formal procedure, which includes back-propagation algorithm. The major problem is computational efficiency, considering all matrices are sparse.</p>
<p>I would appreciate any suggestion. Could anyone point me out an implementation example?</p>

KFrank: <p>Hi Albert!</p>
<p>The short story is that I recommend exploring a standard non-linear<br>
least-squares optimization routine for your problem, rather than using<br>
pytorch’s more general autograd / optimizer tools.</p>
<p>I don’t have much to say about using pytorch for this, but I do have some<br>
comments, below:</p>
<aside class="quote no-group" data-username="Albert_Akhriev" data-post="1" data-topic="109719" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/albert_akhriev/48/13258_2.png" class="avatar"> Albert_Akhriev:</div>
<blockquote>
<p>min Frobenius_norm( M_1(p_1) * M_2(p_2) * M_3(p_3) * … * M_n(p_n)  -  T )</p>
<p>where:<br>
M_i(p_i) is the i-th matrix of known structure parameterized by the vector of parameters p_i (unique for every matrix),</p>
</blockquote>
</aside>
<p>The basic suggestion will be to perform the optimization iteratively,<br>
optimizing one <code>M_i (p_i)</code> at a time, looping over the <code>M_i (p_i)</code><br>
repeatedly.</p>
<p>First, I assume that the matrix <code>M (p)</code> is a deterministic function of the<br>
parameters, <code>p</code>, and that you are optimizing the <code>p</code> (rather than optimizing<br>
<code>M</code> directly).  My comments apply both to optimizing <code>p</code> and to optimizing<br>
<code>M</code>, noting that optimizing <code>M</code> directly is easier, should that be your use<br>
case.</p>
<p>If you <em>fix</em> <code>M_2, M_3, ..., M_n</code> (by fixing <code>p_2, p_3, ..., p_n</code>), then<br>
the elements of your matrix product are <em>linear</em> functions of the elements<br>
of <code>M_1</code>. So (the square of) your Frobenius-norm loss function is a sum<br>
of squares.  If you are optimizing <code>M</code> directly, this can be done in one<br>
step using (numerical) linear algebra.  But even if you are optimizing<br>
the <code>p</code>, the optimization of any give <code>p_i</code> is still a sum of squares (that<br>
depend, potentially non-linearly, on the parameters).  This is a substantive<br>
special structure that makes iterative optimization of this non-linear<br>
sum-of-squares problem significantly cheaper and more robust (see,<br>
for example, Gauss-Newton), and there are many pre-packaged tools<br>
available for doing this.</p>
<p>(Note, that even optimizing all of the <code>p_i</code> at once is still a non-linear<br>
sum-of-squares problem to which these tools should be applicable,<br>
and performing the joint optimization may turn out to be more efficient.<br>
But my intuition tells me that iterating over optimizing each <code>p_i</code> one at<br>
a time could well work better.)</p>
<aside class="quote no-group">
<blockquote>
<p>The major problem is computational efficiency, considering all matrices are sparse.</p>
</blockquote>
</aside>
<p>Please note that the product of two sparse matrices is not, in general,<br>
as sparse as the two factors, so an algorithm that has you explicitly<br>
calculating the product of many sparse matrices is likely to lose any<br>
efficiency benefit of the sparsity.</p>
<p>You haven’t told us the sparsity structure of your matrices.  Are they<br>
sparse, but the locations of the non-zero elements are random?  Or<br>
do they have structure, such as being banded or block-diagonal?</p>
<p>Are the locations of the non-zero elements of <code>M_i (p_i)</code> constant, or<br>
do these locations depend on the values of <code>p_i</code>?</p>
<p>Can you use the sparsity structure to dramatically cheapen multiplying<br>
the matrices together?  Is the structure such that the final product of the<br>
matrices is, itself, sparse?</p>
<p>You can certainly explore pytorch’s sparse-tensor support for these<br>
manipulations, but be forewarned that sparse-tensor calculations often<br>
don’t map well to gpu vectorization, so any time savings may prove<br>
negligible, or even non-negligibly negative.</p>
<p>Good luck.</p>
<p>K. Frank</p>

Albert_Akhriev: <p>Thank you very much for insightful comment. I totally agree that the sparsity structure becomes an issue as the length of the sequence gets longer.</p>