How to discard the eigenvectors with a 0 eigenvalues while calculating loss?

wasabi_linguist: <p>Hi, in my loss function I am calculating the eigen decomposition of the predicted and true hermitian matrices using <code>torch.linalg.eigh</code>. Because of the nature of the matrices, all of their eigenvalues are positive or zero and I need to discard the eigenvectors that have an eigenvalue of zero. Is there any way to discard them without making the loss function non differentiable? Thanks!</p>

KFrank: <p>Hi Wasabi!</p>
<aside class="quote no-group" data-username="wasabi_linguist" data-post="1" data-topic="217324">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wasabi_linguist/48/73592_2.png" class="avatar"> wasabi_linguist:</div>
<blockquote>
<p>I need to discard the eigenvectors that have an eigenvalue of zero. Is there any way to discard them without making the loss function non differentiable?</p>
</blockquote>
</aside>
<p>As an aside, without knowing how you use the eigenvectors in your loss function, it’s<br>
hard to know whether such a loss function would become non-differentiable.</p>
<p>More to the point, many pytorch operations are usefully differentiable (in the sense<br>
of working in practice for gradient-descent optimization) even though they are not<br>
technically differentiable.  Specifically they are differentiable “almost everywhere” and<br>
fail to be differentiable “on a set of measure zero.”  <code>abs (x)</code> is a concrete example: it<br>
is differentiable for all <code>x</code> except <code>x = 0</code> and is routinely used in models that are trained<br>
with backpropagation.</p>
<p>In your case, as you vary the elements of your hermitian matrices, your eigenvalues<br>
will generically change from non-zero to zero at isolates points (or, more correctly, on<br>
a set of measure zero within the space of the elements of the hermitian matrices).  So<br>
this “isolated” non-differentiabilty won’t be an issue.</p>
<p>Note, however (independent of the issue of discarding eigenvectors), if your loss function<br>
depends on the eigenvectors (as compared to just the eigenvalues), the phases of the<br>
eigenvectors are arbitrary, so the loss has to be independent of those arbitrary phases.<br>
Also, if you have degenerate eigenvalues (that is, multiple eigenvalues of the same<br>
value), eigenvectors that have the same eigenvalue lie in a well-defined “eigensubspace,”<br>
but the specific eigenvectors (within that subspace) chosen to span that subspace are<br>
arbitrary.  Therefore your loss can depend on the eigensubspace, but not on the specific<br>
eigenvectors chosen to span the subspace.</p>
<p>Differentiability and discarding eigenvectors won’t be your problem, but, more generally,<br>
a loss function that depends on eigenvectors can often lead to problems.</p>
<p>Best.</p>
<p>K. Frank</p>

wasabi_linguist: <p>Thanks!</p>
<p>Because of the nature of the matrix I am using, the only degenerete eigenvalues are the eigenvalues with an eigenvalue of 0 and I take the square of the eigenvectors to remove the effect of ambiguous phases of eigenvectors.</p>
<p>So if i create a mask like <code>mask = eigenvalues &lt; 10**-7</code> and that to get eigenvectors with non-zero eigenvalues like <code>eigenvectors[mask]</code> would it be alright as it is pseudo-differentiable?</p>

KFrank: <p>Hi Wasabi!</p>
<aside class="quote no-group" data-username="wasabi_linguist" data-post="3" data-topic="217324">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wasabi_linguist/48/73592_2.png" class="avatar"> wasabi_linguist:</div>
<blockquote>
<p>I take the square of the eigenvectors to remove the effect of ambiguous phases of eigenvectors.</p>
</blockquote>
</aside>
<p>As an aside, if you literally take the square of your eigenvectors, you won’t be removing<br>
the arbitrary phases – you would be doubling them.  If you multiply your eigenvectors by<br>
their complex conjugates, you would be cancelling out the arbitrary phases.</p>
<aside class="quote no-group">
<blockquote>
<p>So if i create a mask like <code>mask = eigenvalues &lt; 10**-7</code> and that to get eigenvectors with non-zero eigenvalues like <code>eigenvectors[mask]</code> would it be alright as it is pseudo-differentiable?</p>
</blockquote>
</aside>
<p>Yes, in the context you describe of having no non-zero degenerate eigenvalues, you should<br>
be fine.  (I suppose one could construct a perverse loss function that could mess things up,<br>
but I wouldn’t expect that to happen in a realistic case.)  When an eigenvalue crosses over<br>
your mask threshold, you could have non-differentiability (and discontinuity), but lots of<br>
models have that sort of behavior and can be trained just fine.</p>
<p>Best.</p>
<p>K. Frank</p>