Clamping to satisfy an inequality without messing up grads

sherlock.h: <p>Hello!</p>
<p>I have two tensors, <code>x</code> (B,) and <code>y</code> (B, N, N). Each matrix in <code>y</code> is theoretically guaranteed to have a real eigenvalue. I require <code>x</code> to always satisfy <code>1 + x.mul(minimum_real_eigenvalue(y))  &gt; 0</code>.</p>
<p>Basically, I’m going to use <code>x_cond = x.clamp(...)</code> or <code>x_cond = torch.min(...)</code> in downstream tasks.</p>
<p>Any ideas? Thank you!</p>

lucasegidio1: <p><code>clamp</code> and <code>min</code> will make your gradient vanish, indeed.</p>
<p>Maybe <a href="https://pytorch.org/docs/stable/generated/torch.nn.Softplus.html" rel="noopener nofollow ugc"><code>Softplus</code></a> (with a <code>beta&gt;5</code>) would be a better alternative?</p>

sherlock.h: <p>What I am currently doing is something like this:</p>
<pre data-code-wrap="python"><code class="lang-python">eigenvalues = torch.linalg.eigvals(y)
r_eigenvalues = torch.where(~torch.isreal(eigenvalues), 1.0, eigenvalues.real)
min_eigenvalues = r_eigenvalues.min(-1).values
x_cond = torch.where(x.mul(min_eigenvalues).add(1) &gt; 0,
                     x,
                     min_eigenvalues.mul(-1).pow(-1) - 1e-5)
</code></pre>
<p>Meaning that I am not exactly using <code>min</code> or <code>clamp</code>. My problem is that when I introduce this into the code instead of leaving <code>x</code> (which is a learnable <code>Parameter</code>) unbounded, I get an exploding gradient problem, even though when I inspect the values of <code>x</code> I find them reasonable (on the order of <code>1e-1</code> or <code>1e-2</code>).</p>
<p>How would you introduce <code>Softplus</code> here?</p>

lucasegidio1: <p>Hum, so let me ask a couple of questions to get more intuition on your problem.</p>
<p>By doing <code>r_eigenvalues = torch.where(~torch.isreal(eigenvalues), 1.0, eigenvalues.real)</code> are you assuming that the real eigenvalue is strictly less than <code>1.0</code>? Because if it is not, the <code>min</code> call in the subsequent line will not pick an eigenvalue but the <code>1.0</code> you added with where.</p>
<p>And do you really care only about the real eigenvalues? Or would it also work for minimum real part of all eigenvalues or the minimum singular value?</p>
<p>Well, if your answers are “yes” for both, I think you could do something like</p>
<pre data-code-wrap="python"><code class="lang-python">import torch


B = 2
N = 3

leaf = torch.ones(1, requires_grad=True)  # just to create a graph

x = torch.tensor([-2.0, 1.0]) * leaf
y = (
    torch.tensor(
        [
            [
                [-0.2671, -0.0748, -0.1718],
                [-0.1510, 0.2960, -1.6154],
                [-0.0786, -1.6944, 0.9237],
            ],
            [
                [-0.7459, -1.0099, -0.2937],
                [-0.4803, -0.8911, -0.3678],
                [0.3982, -0.7645, 0.7407],
            ],
        ]
    )
    * leaf
)


eigenvalues = torch.linalg.eigvals(y)
r_eigenvalues = torch.where(~torch.isreal(eigenvalues), torch.inf, eigenvalues.real)
min_eigenvalues, _ = r_eigenvalues.min(-1)

softplus = torch.nn.Softplus(beta=5.0)

x_cond = (softplus(1 + x.mul(min_eigenvalues)) - 1) / min_eigenvalues

print(1 + x * min_eigenvalues)  # [ 3.2149, -0.5520] &lt;- unconstrained x violates condition
print(1 + x_cond * min_eigenvalues)  # [3.2149, 0.0123] &lt;- constrained x_cond doesn't

x_cond[1].backward()
print(leaf.grad)  # got a non-zero grad
</code></pre>
<p>Then you use <code>x_cond</code> instead of <code>x</code> in whatever downstream operation you perform. Note that the constraint is not tightly active due to smoothness of Softplus. You can make it tighter by increasing beta but you may run into vanishing gradients. An alternative would be to use a <a href="https://pytorch.org/docs/stable/generated/torch.nn.LeakyReLU.html" rel="noopener nofollow ugc">LeakyReLU</a> instead of Softplus, but the constraint may be slightly violated.</p>

sherlock.h: <p>Thanks lucas! Using <code>softplus</code> as a <code>max</code> proxy is a neat trick.</p>
<p>My choice of 1 in the first <code>torch.where</code> rested on the fact that any real eigenvalue &gt; 1 will pass the test, since <code>x &gt; 0</code> always, which, in hindsight, I should’ve specified in the OP! I also wanted the subsequent <code>1 / min_eigenvalues</code> to not yield any weird <code>nan</code> or <code>inf</code> (had I chosen 0 instead of 1) that could mess up the grads, even though they would be in the discarded section of <code>torch.where</code>. Paranoia, basically. <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"></p>