Matrix exponential algorithm implementation

Sam_Galanakis: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/7/07a29c6a4869ed9be17dd9797bc0bd681eb764b7.png" data-download-href="https://discuss.pytorch.org/uploads/default/07a29c6a4869ed9be17dd9797bc0bd681eb764b7" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/07a29c6a4869ed9be17dd9797bc0bd681eb764b7_2_496x500.png" alt="image" data-base62-sha1="15xJymLaAhUVdbmvQzsLDm8Yndl" width="496" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/7/07a29c6a4869ed9be17dd9797bc0bd681eb764b7_2_496x500.png, https://discuss.pytorch.org/uploads/default/original/3X/0/7/07a29c6a4869ed9be17dd9797bc0bd681eb764b7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/7/07a29c6a4869ed9be17dd9797bc0bd681eb764b7.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">693×698 59.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am trying to implement the above algorithm in pytorch since the provided matrix_exp is too slow for my use case. I was able to implement all but the loop at step 11-12 in a fairly efficient way (no loops). Is there any way to implement that last part? I couldn’t find a way to apply matrix power based on integers (the powers) stored in a separate tensor without looping over each which would be too slow. Thanks in advance.</p>

KFrank: <p>Hi Sam!</p>
<p>The short answer is that trying to write your own version of standard<br>
numerical-analysis algorithms is almost always a bad idea.</p>
<aside class="quote no-group quote-modified" data-username="Sam_Galanakis" data-post="1" data-topic="111961" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_galanakis/48/30002_2.png" class="avatar"> Sam_Galanakis:</div>
<blockquote>
<p>I am trying to implement the above algorithm in pytorch since the provided matrix_exp is too slow for my use case.<br>
…<br>
looping over each which would be too slow.</p>
</blockquote>
</aside>
<p>Your issue is that pytorch’s <code>matrix_exp</code> is “too slow,” but then you<br>
note that your approach would also be “too slow.”  Please consider<br>
the possibility that what you want to calculate – the exponential of a<br>
matrix – represents a substantive piece of computation, and is likely<br>
to be “slow,” whoever implements the algorithm.  While it’s not<br>
impossible that pytorch is doing a bad job of this, it’s much more likely<br>
that it’s doing a better job than you would be able to.</p>
<p>Some comments:</p>
<p>Pytorch’s implementation uses an algorithm quite similar to your<br>
reference algorithm.  (Your reference algorithm is, to be sure, a<br>
solid, legitimate algorithm for matrix exponentiation.)</p>
<p>People have been doing this kind of computerized numerical analysis<br>
for fifty years now – and, in fact, quite a bit longer.  The experts really<br>
do know what they’re doing.</p>
<p>Lastly, pytorch’s <code>matrix_exp</code> supports autograd and backpropagation<br>
out of the box.  Your version will too if you make sure to use pytorch<br>
tensor operations throughout, including inside the loops.  But doing<br>
so will create a large (one “layer” for each loop iteration) computation<br>
graph, and will likely require excessively more memory and perform<br>
backpropagation significantly more slowly than pytorch’s.</p>
<p>(As an aside, please don’t post screenshots of textual material.  Doing<br>
so breaks accessibility, searchability, and copy-paste.)</p>
<p>Best.</p>
<p>K. Frank</p>

Sam_Galanakis: <p>Hi Frank,</p>
<p>Thanks for the detailed response, I don’t expect to make a faster implementation of the exact matrix exponential, but I need a more efficient approximate algorithm such as this one as it allows for specifying the precision. I am currently using a naive truncation with ~ 15 first terms and it is several seconds faster per batch and than matrix_exp with results being within ~1e-8 of it’s output in most cases. I wanted to improve on that with the above algorithm which guarantees a certain error margin and from what I have found the algorithm I shared here is supposed to work well for small dimensions (my matrices are &lt; 7x7).</p>