Linalg.eig breaks for torch >= 1.11

souryadey: <p>I have this line in the forward pass of my code:</p>
<pre><code class="lang-auto">Lambda, Wtilde = torch.linalg.eig(Ktilde)
</code></pre>
<p>where <code>Ktilde</code> is a real square matrix, and <code>Lambda</code> and <code>Wtilde</code> are complex.</p>
<p>This used to work absolutely fine up to torch &lt;= 1.10.</p>
<p>However, torch &gt;= 1.11 introduced the following check:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff3215e9a956c471e4dbb66a3ddd1f95d6a759b6.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff3215e9a956c471e4dbb66a3ddd1f95d6a759b6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff3215e9a956c471e4dbb66a3ddd1f95d6a759b6_2_690x253.png" alt="image" data-base62-sha1="Apz3TDoAKZJCYFiSp5GwQIOfZHM" width="690" height="253" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff3215e9a956c471e4dbb66a3ddd1f95d6a759b6_2_690x253.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff3215e9a956c471e4dbb66a3ddd1f95d6a759b6_2_1035x379.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff3215e9a956c471e4dbb66a3ddd1f95d6a759b6_2_1380x506.png 2x" data-dominant-color="E0D7D8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1618×594 107 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As a result, I now get:</p>
<pre><code class="lang-auto">RuntimeError: linalg_eig_backward: The eigenvectors in the complex case are specified up to multiplication by e^{i phi}. The specified loss function depends on this quantity, so it is ill-defined.
</code></pre>
<p>which crashes my code.</p>
<p>I don’t want to keep using torch &lt;= 1.10. Is there a way to fix this?</p>

AlphaBetaGamma96: <p>You could try defining a custom <code>torch.autograd.Function</code> perhaps and define the <code>backward</code> method manually? That would allow you get around this issue, although, you’ll have to derive the derivative yourself which can get quite messy depending on the function.</p>

souryadey: <p>I was more looking for a way to skip the check that autograd does for e^i*phi</p>

AlphaBetaGamma96: <p>You’d have to check with a dev but it’s most likely been added in for numerical stability as there’s a degeneracy on the phase of your eigenvectors, so your loss function isn’t well-defined. Perhaps might be best to get a dev to answer (e.g. <a class="mention" href="/u/ptrblck">@ptrblck</a>, apologizes for the tag!)</p>
<p>I do think you could get around this issue by defining your own <code>torch.autograd.Function</code> object but you’d have to derive the <code>backward</code> formula yourself which you could possibly find via the matrix cookbook or something similar.</p>

ptrblck: <p>Thanks for pinging <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a>!<br>
I think you are right and it seems the check was added in <a href="https://github.com/pytorch/pytorch/pull/70527">this PR</a> explaining:</p>
<blockquote>
<p>It also corrects the forward AD formula for it to be correct. Now all<br>
the tests pass for <code>linalg.eig</code> and <code>linalg.eigvals</code>.</p>
</blockquote>
<p>with a follow-up <a href="https://github.com/pytorch/pytorch/pull/70528">here</a>.</p>

souryadey: <p>Seems to me that the check in <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/FunctionsManual.cpp#L3615" rel="noopener nofollow ugc">this line</a> is checking whether the imaginary part of the eigenvectors is 0 (up to a specified tolerance). If it’s not, gradient computation will not occur.</p>
<p>Which brings me back to my original question – is there a way around this? I can try defining my own derivative as <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> suggested, but that can get quite messy. Is there a way to make Pytorch compute uniquely-phase-defined eigenvectors so that autograd can proceed? For example, any multiple of an eigenvector is also an eigenvector, but Pytorch gets around this magnitude non-uniqueness by normalizing the eigenvectors to have norm 1. Is there such an option for phase as well?</p>
<p>I am hoping there can be a solution, because otherwise it seems to me that backward passes for <code>torch.linalg.eig</code> in torch &gt;= 1.11 will frequently make the whole program crash.</p>

KFrank: <p>Hi Sourya!</p>
<aside class="quote no-group" data-username="souryadey" data-post="6" data-topic="157453" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/souryadey/48/44490_2.png" class="avatar"> souryadey:</div>
<blockquote>
<p>Seems to me that the check in <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/FunctionsManual.cpp#L3615" rel="noopener nofollow ugc">this line</a> is checking whether the imaginary part of the eigenvectors is 0 (up to a specified tolerance). If it’s not, gradient computation will not occur.</p>
</blockquote>
</aside>
<p>This is not correct.  Backpropagation / gradient computation can, <em>in<br>
appropriate cases,</em> occur through <code>torch.linalg.eig()</code> where the<br>
eigenvectors / eigenvalues have non-zero imaginary parts.  (The code<br>
you linked to that checks for an imaginary part tests an intermediate<br>
result, not the  eigenvectors of your <code>Ktilde</code>.)</p>
<aside class="quote no-group">
<blockquote>
<p>is there a way around this?</p>
</blockquote>
</aside>
<p>The short answer is don’t try to compute gradients of quantities that depend<br>
on the phases of your eigenvectors.  (Pytorch does support computing<br>
gradients of phase-independent quantities.)</p>
<p>There is not a <em>good</em> way around this.  At issue is that the warning  in the<br>
documentation and the <code>RuntimeError</code> you cited in your original post<br>
are fully legitimate.  If you get that <code>RuntimeError</code> you are almost certainly<br>
doing something that doesn’t make sense.</p>
<p>The problem – as mentioned in the documentation warning – is that the<br>
phases of the eigenvectors  are not mathematically uniquely defined.  So<br>
a loss function that depends on those phases, as well as the gradient of<br>
such a loss function, is also not uniquely defined.</p>
<p>Pytorch could, hypothetically, impose an ad hoc set of rules that uniquely<br>
determines those phases, but that would just give you ad hoc uniqueness<br>
that would still be mathematically arbitrary, so you’d be sweeping your<br>
issue under the rug, rather than actually fixing it.  (Also, if you look at my<br>
example, below, you can see that setting up those ad hoc rules could be<br>
tricky, because there are multiple, mathematically-equivalent paths to<br>
computing the same eigenvectors.)</p>
<p>The following example illustrates what is going on.  As a device to generate<br>
eigenvectors with differing phases, I apply an orthogonal transformation<br>
to the original real square matrix, compute its eigenvectors, and then<br>
rotate those eigenvectors back to the basis of the original matrix using<br>
the same orthogonal transformation.  You can understand this as a second,<br>
fully mathematically legitimate algorithm for computing the eigenvectors.</p>
<p>To make clear that pytorch version 1.12 <em>does</em> permit you to backpropagate<br>
through <code>eig()</code> <em>provided</em> you are computing the gradient of a<br>
phase-independent quantity, I post the example as run in both versions<br>
1.10 and 1.12.  (These two example runs are almost identical duplicates<br>
of one another, so there’s no need to compare them line by line.  The<br>
only difference is at the end where the 1.12 version flags the illegitimate<br>
backpropagation.)</p>
<p>Here is the 1.10 version:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.10.2
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; tA = torch.randn (3, 3)
&gt;&gt;&gt; tB = tA.clone()
&gt;&gt;&gt; tA.requires_grad = True
&gt;&gt;&gt; tB.requires_grad = True
&gt;&gt;&gt;
&gt;&gt;&gt; valA, vecA = torch.linalg.eig (tA)
&gt;&gt;&gt;
&gt;&gt;&gt; orth = torch.linalg.svd (torch.randn (3, 3))[0]   # orthogonal matrix to rotate tB
&gt;&gt;&gt; tBr = orth @ tB @ orth.T
&gt;&gt;&gt; valBr, vecBr = torch.linalg.eig (tBr)   # eigenvectors of rotated tB
&gt;&gt;&gt;
&gt;&gt;&gt; tAc = tA.to (dtype = torch.complex64)
&gt;&gt;&gt; orthc = orth.to (dtype = torch.complex64)
&gt;&gt;&gt;
&gt;&gt;&gt; vecB = orthc.T @ vecBr   # rotate eigenvectors back to basis of tB
&gt;&gt;&gt; torch.allclose (tAc @ vecB, valA * vecB)   # check that vecB is indeed a set of eigenvectors of tA
True
&gt;&gt;&gt;
&gt;&gt;&gt; vecB / vecA   # each eigenvector is changed by a complex phase
tensor([[ 0.9991+0.0414j,  0.9991-0.0414j, -1.0000-0.0000j],
        [ 0.9991+0.0414j,  0.9991-0.0414j, -1.0000-0.0000j],
        [ 0.9991+0.0414j,  0.9991-0.0414j, -1.0000+0.0000j]],
       grad_fn=&lt;DivBackward0&gt;)
&gt;&gt;&gt;
&gt;&gt;&gt; lossGoodA = (vecA * vecA.conj())[0, 0]
&gt;&gt;&gt; lossGoodB = (vecB * vecB.conj())[0, 0]
&gt;&gt;&gt;
&gt;&gt;&gt; lossGoodA
tensor(0.0332+0.j, grad_fn=&lt;SelectBackward0&gt;)
&gt;&gt;&gt; lossGoodB
tensor(0.0332+0.j, grad_fn=&lt;SelectBackward0&gt;)
&gt;&gt;&gt; torch.allclose (lossGoodA, lossGoodB)   # phase-independent loss is the same for both sets of eigenvectors
True
&gt;&gt;&gt;
&gt;&gt;&gt; lossGoodA.backward()   # both versions 1.10 and 1.12 permit legitimate backward pass
&gt;&gt;&gt; tA.grad
tensor([[-0.0096,  0.4561, -0.3340],
        [ 0.0084, -0.0616,  0.0662],
        [-0.0009, -0.1106,  0.0712]])
&gt;&gt;&gt;
&gt;&gt;&gt; lossBadA = (vecA * vecA).real[0, 0]
&gt;&gt;&gt; lossBadB = (vecB * vecB).real[0, 0]
&gt;&gt;&gt;
&gt;&gt;&gt; lossBadA
tensor(0.0198, grad_fn=&lt;SelectBackward0&gt;)
&gt;&gt;&gt; lossBadB
tensor(0.0176, grad_fn=&lt;SelectBackward0&gt;)
&gt;&gt;&gt; torch.allclose (lossBadA, lossBadB)   # this loss is not phase-independent
False
&gt;&gt;&gt;
&gt;&gt;&gt; lossBadB.backward()   # version 1.12 flags this backward pass as ill-defined
&gt;&gt;&gt; tB.grad   # this gradient is phase-dependent (in version 1.10 and not computed in version 1.12)
tensor([[ 0.0844,  0.2666,  0.0732],
        [-0.0095, -0.0822,  0.0267],
        [-0.0213, -0.0918, -0.0022]])
</code></pre>
<p>And here is the 1.12 version:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.12.0
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; tA = torch.randn (3, 3)
&gt;&gt;&gt; tB = tA.clone()
&gt;&gt;&gt; tA.requires_grad = True
&gt;&gt;&gt; tB.requires_grad = True
&gt;&gt;&gt;
&gt;&gt;&gt; valA, vecA = torch.linalg.eig (tA)
&gt;&gt;&gt;
&gt;&gt;&gt; orth = torch.linalg.svd (torch.randn (3, 3))[0]   # orthogonal matrix to rotate tB
&gt;&gt;&gt; tBr = orth @ tB @ orth.T
&gt;&gt;&gt; valBr, vecBr = torch.linalg.eig (tBr)   # eigenvectors of rotated tB
&gt;&gt;&gt;
&gt;&gt;&gt; tAc = tA.to (dtype = torch.complex64)
&gt;&gt;&gt; orthc = orth.to (dtype = torch.complex64)
&gt;&gt;&gt;
&gt;&gt;&gt; vecB = orthc.T @ vecBr   # rotate eigenvectors back to basis of tB
&gt;&gt;&gt; torch.allclose (tAc @ vecB, valA * vecB)   # check that vecB is indeed a set of eigenvectors of tA
True
&gt;&gt;&gt;
&gt;&gt;&gt; vecB / vecA   # each eigenvector is changed by a complex phase
tensor([[ 0.9991+0.0414j,  0.9991-0.0414j, -1.0000-0.0000j],
        [ 0.9991+0.0414j,  0.9991-0.0414j, -1.0000-0.0000j],
        [ 0.9991+0.0414j,  0.9991-0.0414j, -1.0000+0.0000j]],
       grad_fn=&lt;DivBackward0&gt;)
&gt;&gt;&gt;
&gt;&gt;&gt; lossGoodA = (vecA * vecA.conj())[0, 0]
&gt;&gt;&gt; lossGoodB = (vecB * vecB.conj())[0, 0]
&gt;&gt;&gt;
&gt;&gt;&gt; lossGoodA
tensor(0.0332+0.j, grad_fn=&lt;SelectBackward0&gt;)
&gt;&gt;&gt; lossGoodB
tensor(0.0332+0.j, grad_fn=&lt;SelectBackward0&gt;)
&gt;&gt;&gt; torch.allclose (lossGoodA, lossGoodB)   # phase-independent loss is the same for both sets of eigenvectors
True
&gt;&gt;&gt;
&gt;&gt;&gt; lossGoodA.backward()   # both versions 1.10 and 1.12 permit legitimate backward pass
&gt;&gt;&gt; tA.grad
tensor([[-0.0096,  0.4561, -0.3340],
        [ 0.0084, -0.0616,  0.0662],
        [-0.0009, -0.1106,  0.0712]])
&gt;&gt;&gt;
&gt;&gt;&gt; lossBadA = (vecA * vecA).real[0, 0]
&gt;&gt;&gt; lossBadB = (vecB * vecB).real[0, 0]
&gt;&gt;&gt;
&gt;&gt;&gt; lossBadA
tensor(0.0198, grad_fn=&lt;SelectBackward0&gt;)
&gt;&gt;&gt; lossBadB
tensor(0.0176, grad_fn=&lt;SelectBackward0&gt;)
&gt;&gt;&gt; torch.allclose (lossBadA, lossBadB)   # this loss is not phase-independent
False
&gt;&gt;&gt;
&gt;&gt;&gt; lossBadB.backward()   # version 1.12 flags this backward pass as ill-defined
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;path_to_pytorch_install&gt;\torch\_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "&lt;path_to_pytorch_install&gt;\torch\autograd\__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: linalg_eig_backward: The eigenvectors in the complex case are specified up to multiplication by e^{i phi}. The specified loss function depends on this quantity, so it is ill-defined.
&gt;&gt;&gt; tB.grad   # this gradient is phase-dependent (in version 1.10 and not computed in version 1.12)
&gt;&gt;&gt;
</code></pre>
<aside class="quote no-group">
<blockquote>
<p>it seems to me that backward passes for <code>torch.linalg.eig</code> in torch &gt;= 1.11 will frequently make the whole program crash.</p>
</blockquote>
</aside>
<p>Just to be clear, the <code>RuntimeError</code> raised in version &gt;= 1.11 isn’t <em>breaking</em><br>
your code – it’s helpfully warning you that your code is already broken.</p>
<p>Best.</p>
<p>K. Frank</p>

souryadey: <aside class="quote no-group quote-modified" data-username="KFrank" data-post="7" data-topic="157453">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<pre><code class="lang-auto">
</code></pre>
</blockquote>
</aside>
<p>Thank you for the great explanation.</p>
<p>Unfortunately, due to numerical imprecisions, the tensors in my code end up having very small imaginary parts (e.g. <code>2e-08j</code> when using float precision, or <code>2e-17j</code> when using double precision). I cannot keep these complex values because these values will then pass through <code>torch.nn</code> layers, and so should be real. So I tried to remove the imaginary portions using <code>torch.abs()</code>. I think this is causing the RunTime Error; since when I do <code>torch.abs(x)</code>, I am essentially doing <code>torch.sqrt(x*x.conj()).real</code>, and this causes the autograd graph to depend on the phase.</p>
<p>I don’t know what to do about this issue. I need to remove those tiny imaginary parts, but in a way so that the autograd graph is not affected.</p>

AlphaBetaGamma96: <p>Can’t you just do something like <code>x = x.real()</code> ? Or if you need it to remain complex, you could cast it back to complex with the imaginary components like <code>torch.zeros_like(x)</code> so you have a complex part that’s purely real and zero in the imaginary componet?</p>
<p>That shouldn’t affect backprop with these small imaginary components?</p>

souryadey: <p>Ya I tried that, it doesn’t work unfortunately. <code>x.real</code> throws away the imaginary part, but that operation becomes a part of the computational graph, so the gradient is traced back to the imaginary components.</p>
<p>I also cannot do <code>with torch.no_grad(); x.real</code> since I need the downstream gradients.</p>

KFrank: <p>Hi Sourya!</p>
<aside class="quote no-group" data-username="souryadey" data-post="8" data-topic="157453" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/souryadey/48/44490_2.png" class="avatar"> souryadey:</div>
<blockquote>
<p>Unfortunately, due to numerical imprecisions, the tensors in my code end up having very small imaginary parts (e.g. <code>2e-08j</code></p>
</blockquote>
</aside>
<p>Could you explain how these “numerical imprecisions” are related to<br>
<code>torch.linalg.eig()</code>?</p>
<aside class="quote no-group">
<blockquote>
<p>So I tried to remove the imaginary portions using <code>torch.abs()</code>. I think this is causing the RunTime Error; since when I do <code>torch.abs(x)</code>, I am essentially doing <code>torch.sqrt(x*x.conj()).real</code>, and this causes the autograd graph to depend on the phase.</p>
</blockquote>
</aside>
<p>On the contrary, <code>torch.abs (x) = torch.sqrt (x * x.conj()).real</code><br>
is independent of the phase.  A simple test shows that backpropagating<br>
an eigenvector (with a non-trivial phase) through <code>abs()</code> works just fine:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.12.0
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; t = torch.randn (3, 3, requires_grad = True)
&gt;&gt;&gt; val, vec = torch.linalg.eig (t)
&gt;&gt;&gt; loss = vec[0, 0].abs()
&gt;&gt;&gt; loss.backward()
&gt;&gt;&gt; t.grad
tensor([[-0.0265,  1.2525, -0.9171],
        [ 0.0231, -0.1691,  0.1819],
        [-0.0026, -0.3037,  0.1956]])
</code></pre>
<aside class="quote no-group">
<blockquote>
<p>I don’t know what to do about this issue.</p>
</blockquote>
</aside>
<p>Could you post a truly minimal, complete, runnable script that reproduces<br>
the issue you think you are having with <code>torch.linalg.eig()</code>?</p>
<p>It should be a simple as:  create some sample data; call <code>torch.linalg.eig()</code>;<br>
compute a phase-independent loss function; and call <code>loss.backward()</code>.<br>
It should work.</p>
<p>If you call <code>loss.backward()</code> on a phase-dependent loss function, we do<br>
expect it to fail, in which case we have to figure out what you are trying to<br>
accomplish with your phase-dependent loss function and figure out how<br>
you might actually do it correctly – that is, with a phase-independent loss.</p>
<p>Best.</p>
<p>K. Frank</p>

souryadey: <p>Here’s a <a href="https://drive.google.com/drive/folders/1Hp_938P90kpAa--IPWb95BJCnkPqZrVv?usp=sharing" rel="noopener nofollow ugc">folder</a> containing <code>main.py</code> (and its dependencies <code>helper.py</code> and data <code>Xtr.pt</code>) that reproduces the issue.</p>
<p>I could not make it simpler without disrupting functionality, hopefully this is straightforward enough for you to take a look. Just run <code>main.py</code>. The <code>linalg.eig</code> phase error will occur when computing <code>loss.backward()</code> in epoch 3.</p>
<p>FYI my torch version is 1.12.0 and numpy version is 1.23.0.</p>
<p>Your help is much appreciated, thanks a lot in advance!</p>

souryadey: <p>I have figured out the error, it was actually in a different portion of the code where I was using <code>torch.exp</code> instead of <code>torch.linalg.matrix_exp</code>. Consequently I have also deleted the folder I linked above.</p>
<p>Thanks so much for this conversation, particularly <a class="mention" href="/u/kfrank">@KFrank</a> . All of your answers helped me to re-examine my code in detail and figure out the bug. Much appreciated!!</p>

Javad: <p>Related to this question, is there a way to force pytorch to skip the check that autograd does for e^{i phi}?</p>
<p>In my code, I have used a prescription to fix the phase, but pytorch still complains that the loss-function is  ill-defined.</p>

souryadey: <p>I don’t get any errors, but I still get the e^{i*phi} warning in torch 1.12. Maybe you can use the Python <code>warnings</code> package.</p>

Javad: <p>Thanks for your suggestion, but I have to fix RuntimeError.</p>

Javad: <p>Does any one know how to turn off this error “RuntimeError: linalg_eigh_backward: The eigenvectors in the complex case are specified up to multiplication by e^{i phi}. The specified loss function depends on this quantity, so it is ill-defined.” The redundant phase gets cancelled in my code and the loss function should be independent of it, but torch still complains.</p>

bnucgd: <p>Hello, have you managed to disable this error? I am facing the same issue and I would appreciate your help if possible</p>

lcnature: <p>I am bothered by the same issue. My loss function really does not depend on the phase of the eigenvectors but I cannot find a way to skip this check. Is there any suggestion for defining the gradient function?</p>
<p>One example where this checking really hinders our work is that we want to get a fractional power of a rotation matrix. This can be done by finding the eigenvalues and eigenvectors and replacing eigenvalues with a power of a fractional number of themselves and reconstruct a rotation matrix using the eigenvectors and their inverse on the two sides. Even if my eigenvectors are multiplied by an arbitrary e^(i phi), it does not change the outcome. Since torch.linalg.matrix_power only allows integer power, there is no way around this.</p>