Raising a tensor to a fractional power

username: mmutic
cooked: <p>Hi, I need to raise a matrix to the power of -1/2, but I believe doing so using just **(-1/2) makes the computation element-wise.</p>
<p>For example, if I do</p>
<pre><code class="lang-auto">m = torch.tensor([[.5,.5],[.7,.9]])
print(m**(-1/2))
</code></pre>
<p>I get</p>
<pre><code class="lang-auto">tensor([[1.4142, 1.4142],
        [1.1952, 1.0541]])
</code></pre>
<p>But what I want is</p>
<pre><code class="lang-auto">tensor([[2.6978, -1.1091],
   [-1.5527,1.8105]])
</code></pre>
<p>which is what Matlab provides when I run m^(-1/2). Unfortunately torch.matrix_power() seems to only take integers, and torch.pow() also seems to do the computation element-wise.</p>
<p>Thanks!</p>

username: mathematics
cooked: <p>Although it is not best solution, but it will work.<br>
<code>scipy</code> will work without changing torch tensor to numpy as eg.</p>
<pre><code class="lang-auto">import scipy.linalg
m = torch.tensor([[.5,.5],[.7,.9]])
print(scipy.linalg.fractional_matrix_power(m, (-1/2)))

array([[ 2.69776664, -1.10907208],
       [-1.55270001,  1.81051025]])
</code></pre>

username: KFrank
cooked: <p>Hi mmutic!</p>
<aside class="quote no-group" data-username="mmutic" data-post="1" data-topic="93655" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/58f4c7/48.png" class="avatar"> mmutic:</div>
<blockquote>
<p>I need to raise a matrix to the power of -1/2, but I believe doing so using just **(-1/2) makes the computation element-wise.</p>
</blockquote>
</aside>
<p>A standard approach to exponentiating a matrix is to calculate its<br>
eigendecomposition and then exponentiate the eigenvalues.</p>
<p>Here is a pytorch version 0.3.0 script that illustrates this:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

m = torch.FloatTensor([[.5,.5],[.7,.9]])           # original matrix

# desired result
mres = torch.FloatTensor ([[ 2.69776664, -1.10907208], [-1.55270001,  1.81051025]])

evals, evecs = torch.eig (m, eigenvectors = True)  # get eigendecomposition
evals = evals[:, 0]                                # get real part of (real) eigenvalues

# rebuild original matrix
mchk = torch.matmul (evecs, torch.matmul (torch.diag (evals), torch.inverse (evecs)))

mchk - m                                           # check decomposition

evpow = evals**(-1/2)                              # raise eigenvalues to fractional power

# build exponentiated matrix from exponentiated eigenvalues
mpow = torch.matmul (evecs, torch.matmul (torch.diag (evpow), torch.inverse (evecs)))

mpow - mres                                        # check result
</code></pre>
<p>Here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
0.3.0b0+591e73e
&gt;&gt;&gt;
&gt;&gt;&gt; m = torch.FloatTensor([[.5,.5],[.7,.9]])           # original matrix
&gt;&gt;&gt;
&gt;&gt;&gt; # desired result
... mres = torch.FloatTensor ([[ 2.69776664, -1.10907208], [-1.55270001,  1.81051025]])
&gt;&gt;&gt;
&gt;&gt;&gt; evals, evecs = torch.eig (m, eigenvectors = True)  # get eigendecomposition
&gt;&gt;&gt; evals = evals[:, 0]                                # get real part of (real) eigenvalues
&gt;&gt;&gt;
&gt;&gt;&gt; # rebuild original matrix
... mchk = torch.matmul (evecs, torch.matmul (torch.diag (evals), torch.inverse (evecs)))
&gt;&gt;&gt;
&gt;&gt;&gt; mchk - m                                           # check decomposition

1.00000e-07 *
 -0.5960  0.0000
 -0.5960  1.1921
[torch.FloatTensor of size 2x2]

&gt;&gt;&gt;
&gt;&gt;&gt; evpow = evals**(-1/2)                              # raise eigenvalues to fractional power
&gt;&gt;&gt;
&gt;&gt;&gt; # build exponentiated matrix from exponentiated eigenvalues
... mpow = torch.matmul (evecs, torch.matmul (torch.diag (evpow), torch.inverse (evecs)))
&gt;&gt;&gt;
&gt;&gt;&gt; mpow - mres                                        # check result

1.00000e-07 *
  4.7684  7.1526
 -2.3842 -7.1526
[torch.FloatTensor of size 2x2]
</code></pre>
<p>You can see that this scheme recovers your Matlab result (actually<br>
Mr. mathematics’s scipy result, because he printed it out with greater<br>
precision).</p>
<p>This will work mathematically for positive semi-definite (square)<br>
matrices (although numerically you will probably want your smallest<br>
eigenvalue to be enough larger than zero that its computation doesn’t<br>
yield a negative value due to numerical error).</p>
<aside class="quote no-group" data-username="mathematics" data-post="2" data-topic="93655">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mathematics/48/40082_2.png" class="avatar"> mathematics:</div>
<blockquote>
<p><code>scipy</code> will work without changing torch tensor to numpy</p>
</blockquote>
</aside>
<p>However, if you want to use autograd to calculate gradients (for, e.g.,<br>
backpropagation), performing the calculation in scipy won’t work<br>
(unless you write your own <code>.backward()</code> function).</p>
<p>Because the <code>torch.eig()</code> approach works entirely with pytorch tensor<br>
functions, you get autograd / gradients “for free.”</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: byteSamurai
cooked: <p>This is an old one, so sorry, if my question might be naive, but isn’t this valid for matrix calculation as well?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/3/3350645909c4c700c4293e79e0fd13868fc83584.png" alt="image" data-base62-sha1="7jWAA36z3DgHIMSZD0qlzuW4edC" width="260" height="96"></p>
<p>So, if I run this, I get the same result as with scipy:</p>
<pre><code class="lang-auto">X.inverse().sqrt()
</code></pre>
<p>I assume, this also maintains the gradients?</p>

username: KFrank
cooked: <p>Hi Alfred!</p>
<aside class="quote no-group" data-username="byteSamurai" data-post="4" data-topic="93655" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bytesamurai/48/38897_2.png" class="avatar"> byteSamurai:</div>
<blockquote>
<pre><code class="lang-auto">X.inverse().sqrt()
</code></pre>
</blockquote>
</aside>
<p>This is incorrect.  <a href="https://pytorch.org/docs/1.8.1/generated/torch.sqrt.html#torch.sqrt" rel="noopener nofollow ugc">torch.sqrt()</a> computes the square-roots of the<br>
individual elements of the tensor (not the matrix square-root).</p>
<p>(<code>torch.inverse()</code>, however, does compute the matrix inverse,<br>
rather than the reciprocals of the individual elements.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: byteSamurai
cooked: <p>Oh sorry, I see. Thanks for the hint <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>