Incorrect gradient computation

Gantavya_Bhatt1: <p>I have the following loss function, for which I want to compute the gradients.</p>
<pre><code class="lang-auto">z = torch.tensor([1.], requires_grad=True, dtype=torch.float64)
thresh = 1e-10
alpha = 1e-5
beta=1.
tau = 1.
loss = F.softplus(tau- (z)/(margin+torch.sign(margin)*thresh), beta=beta)*torch.abs(margin+torch.sign(margin)*thresh)
</code></pre>
<p>Mathematically, the gradient with respect to z is the same as:</p>
<pre><code class="lang-auto">manual_grad = F.sigmoid(beta*(tau- (z)/(margin+torch.sign(margin)*thresh)))*(-torch.sign(margin))
</code></pre>
<p>Moreover, the gradient also exists for margin = 0. (exact 0). However, when I compute the gradient for the loss above, then it comes out to be NaN at 0.</p>
<p>Can anyone help with it?</p>

KFrank: <p>Hi Gantavya!</p>
<aside class="quote no-group" data-username="Gantavya_Bhatt1" data-post="1" data-topic="169651" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gantavya_bhatt1/48/39424_2.png" class="avatar"> Gantavya_Bhatt1:</div>
<blockquote>
<pre><code class="lang-auto">loss = F.softplus(tau- (z)/(margin+torch.sign(margin)*thresh), beta=beta)*torch.abs(margin+torch.sign(margin)*thresh)
</code></pre>
<p>Mathematically, the gradient with respect to z is the same as:</p>
<pre><code class="lang-auto">manual_grad = F.sigmoid(beta*(tau- (z)/(margin+torch.sign(margin)*thresh)))*(-torch.sign(margin))
</code></pre>
<p>Moreover, the gradient also exists for margin = 0. (exact 0).</p>
</blockquote>
</aside>
<p>At a technical level, neither <code>loss</code> nor its gradient is defined when<br>
<code>margin = 0</code>.</p>
<p>Note that <code>torch.sign (torch.tensor ([0.0]))</code> is zero, so that the<br>
argument to <code>softplus()</code> diverges (and <code>softplus()</code> itself either diverges<br>
or becomes zero, depending on the sign of <code>z</code>)</p>
<p>You might hope that when <code>margin</code> is very small, but not quite zero, you<br>
can define a consistent gradient by computing the gradient for non-zero<br>
<code>margin</code> and then taking its limit as <code>margin</code> goes to zero.  But, as shown<br>
by your expression for <code>manual_grad</code>, this doesn’t work.</p>
<p>It is true that <code>manual_grad</code> evaluates to zero for <code>margin = 0.0</code>, but, in<br>
isolation, this isn’t meaningful.  When <code>margin</code> is very small and negative,<br>
<code>manual_grad</code> becomes one (to machine precision), but when <code>margin</code> is<br>
very small and positive, <code>manual_grad</code> underflows to zero.  These numerical<br>
computations are telling you that the gradient is discontinuous as a function<br>
of <code>margin</code> when <code>margin</code> is equal to zero, so trying to define the gradient<br>
by taking the limit as <code>margin</code> goes to zero leaves <code>margin</code> undefined.</p>
<p>Autograd reasonably gives you <code>nan</code> for this undefined value.  (Simply<br>
asserting that the gradient ought to be zero or constructing some expression<br>
that returns zero doesn’t change that fact that the gradient isn’t well defined<br>
when <code>margin = 0.0</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>