Getting different eigenvalues between using numpy.linalg.eigh() and torch.symeig()

Leockl: <p>I am trying to understand why am I getting different eigenvalues between using <code>numpy.linalg.eigh()</code> and <code>torch.symeig()</code>.</p>
<p>An example is as below:</p>
<p>Code:</p>
<pre><code class="lang-auto">import numpy as np
import torch

arr_symmetric = np.array([[1.,2,3], [2,5,6], [3,6,9]])
arr_symmetric, arr_symmetric.dtype
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">(array([[1., 2., 3.],
        [2., 5., 6.],
        [3., 6., 9.]]), dtype('float64'))
</code></pre>
<p>Code:</p>
<pre><code class="lang-auto">tsr_symmetric = torch.tensor(arr_symmetric)
tsr_symmetric
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">tensor([[1., 2., 3.],
        [2., 5., 6.],
        [3., 6., 9.]], dtype=torch.float64)
</code></pre>
<p>Code:</p>
<pre><code class="lang-auto">w, v = np.linalg.eigh(arr_symmetric)
w, v
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">(array([4.05517871e-16, 6.99264746e-01, 1.43007353e+01]),
 array([[-9.48683298e-01,  1.77819106e-01, -2.61496397e-01],
        [ 2.22044605e-16, -8.26924214e-01, -5.62313386e-01],
        [ 3.16227766e-01,  5.33457318e-01, -7.84489190e-01]]))
</code></pre>
<p>Code:</p>
<pre><code class="lang-auto">e, v = torch.symeig(tsr_symmetric, eigenvectors=True)
e, v
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">(tensor([-2.6047e-16,  6.9926e-01,  1.4301e+01], dtype=torch.float64),
 tensor([[ 9.4868e-01, -1.7782e-01,  2.6150e-01],
         [ 8.6389e-16,  8.2692e-01,  5.6231e-01],
         [-3.1623e-01, -5.3346e-01,  7.8449e-01]], dtype=torch.float64))
</code></pre>
<p>As you can see one of the eigenvalues is different, ie. <code>4.05517871e-16</code> vs. <code>-2.6047e-16</code></p>
<p>Why is this happening?</p>

KFrank: <p>Hi Leo!</p>
<aside class="quote no-group quote-modified" data-username="Leockl" data-post="1" data-topic="89928" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/e9a140/48.png" class="avatar"> Leockl:</div>
<blockquote>
<pre><code class="lang-auto">tensor([[1., 2., 3.],
        [2., 5., 6.],
        [3., 6., 9.]], dtype=torch.float64)
</code></pre>
<p>…</p>
<p>As you can see one of the eigenvalues is different, ie. <code>4.05517871e-16</code> vs. <code>-2.6047e-16</code></p>
<p>Why is this happening?</p>
</blockquote>
</aside>
<p>In short, this is due to (double-precision) floating-point round-off, and<br>
is to be expected.</p>
<p>The rows of your matrix are linearly dependent – specifically,<br>
<code>tsr_symmetric[2] = 3 * tsr_symmetric[0]</code> – so your matrix<br>
has zero determinant and (at least) one zero eigenvalue.</p>
<p>The two eigenvalues you quote are within double-precision round-off<br>
of zero (and of one another).  That is, within round-off error, they are<br>
equal to one another (and to zero), and not actually different.</p>
<p>Best.</p>
<p>K. Frank</p>

Leockl: <p>Hi K. Frank, many thanks for your reply and it makes sense.</p>
<p>But what concerns me is that, that eigenvalue which is different, one is a +ve value and the other is a -ve value. This affects an algorithm that I am working on which groups the corresponding eigenvectors based on the +ve or -ve eigenvalues.</p>

KFrank: <p>Hi Leo!</p>
<aside class="quote no-group" data-username="Leockl" data-post="3" data-topic="89928" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/e9a140/48.png" class="avatar"> Leockl:</div>
<blockquote>
<p>one is a +ve value and the other is a -ve value.</p>
</blockquote>
</aside>
<p>Yes, but they’re both actually (within round-off error of being) zero.</p>
<aside class="quote no-group">
<blockquote>
<p>This affects an algorithm that I am working on which groups the corresponding eigenvectors based on the +ve or -ve eigenvalues.</p>
</blockquote>
</aside>
<p>You have to think through what it means for your algorithm if an<br>
eigenvalue is mathematically zero, even if the numerical computation<br>
gives you a non-zero, but small value.</p>
<p>If your matrices are of order one and there is no reason that they<br>
should have especially small eigenvalues, then it might be as simple<br>
as deeming any eigenvalue within some multiple of round-off error<br>
of zero to be zero and including it in neither your +ve-eigenvalue nor<br>
-ve-eigenvalue group.</p>
<p>Welcome to the world of floating-point programming!</p>
<p>Best.</p>
<p>K. Frank</p>

Leockl: <p>Yeah, I definitely understand and agree with your point about coming from the world of floating-point programming!</p>
<p>The +ve/-ve sign discrepancy doesn’t seem to happen with <code>numpy.linalg.eig()</code> and <code>torch.eig()</code>, ie. the +ve/-ve eigenvalue signs are the same/consistent between <code>numpy.linalg.eigh()</code> and <code>numpy.linalg.eig()</code> and <code>torch.eig()</code>. Would be great if we could change <code>torch.symeig()</code> to be the same too, since <code>torch.symeig()</code> is written specifically for symmetric matrices, mirroring <code>numpy.linalg.eigh()</code>. More importantly, would be nice to have some consistency across the board</p>