Imperfect gradient for torch.sigmoid()

KFrank: <p>Hi All!</p>
<p>The gradient computation for <code>torch.sigmoid()</code> is not as good as it<br>
could or should be.</p>
<p>Specifically, it underflows to zero sooner than it should.</p>
<p>This can be seen by direct inspection of the gradient values.  Also,<br>
the gradient should be symmetric around zero, and this condition<br>
is violated.  Notably, the gradient underflows early for positive values<br>
of <code>x</code>.</p>
<p>Also, the standard formula, <code>sigmoid (x)  =  1 / (1 + exp (-x))</code>,<br>
performs better, although still imperfectly, in that its gradient doesn’t<br>
underflow as soon as that of <code>torch.sigmoid()</code>.  (With the standard<br>
formula, the gradient underflows early for negative values of <code>x</code>.)</p>
<p>This is illustrated by the following script and its output:</p>
<pre><code class="lang-python">import torch
print (torch.__version__)

_ = torch.manual_seed (2022)

def sigmoidB (x):
    return 1.0 / (1.0 + torch.exp (-x))

t = torch.arange (-80, 85, 10).float()
t.requires_grad = True

sigA = torch.sigmoid (t)
sigA.sum().backward()
grdA = t.grad

t.grad = None
sigB = sigmoidB (t)
sigB.sum().backward()
grdB = t.grad

print ('t:', t)
print ('sigA:', sigA)
print ('sigB:', sigB)
print ('grdA:', grdA)
print ('grdB:', grdB)
print ('relative asymmetry in gradient:')
print ('(grdA - grdA.flip (0)) / torch.max (grdA, grdA.flip (0)):', (grdA - grdA.flip (0)) / torch.max (grdA, grdA.flip (0)))
print ('(grdB - grdB.flip (0)) / torch.max (grdB, grdB.flip (0)):', (grdB - grdB.flip (0)) / torch.max (grdB, grdB.flip (0)))
</code></pre>
<pre><code class="lang-nohighlight">1.10.0
t: tensor([-80., -70., -60., -50., -40., -30., -20., -10.,   0.,  10.,  20.,  30.,
         40.,  50.,  60.,  70.,  80.], requires_grad=True)
sigA: tensor([1.8049e-35, 3.9754e-31, 8.7565e-27, 1.9287e-22, 4.2484e-18, 9.3576e-14,
        2.0612e-09, 4.5398e-05, 5.0000e-01, 9.9995e-01, 1.0000e+00, 1.0000e+00,
        1.0000e+00, 1.0000e+00, 1.0000e+00, 1.0000e+00, 1.0000e+00],
       grad_fn=&lt;SigmoidBackward0&gt;)
sigB: tensor([1.8049e-35, 3.9754e-31, 8.7565e-27, 1.9287e-22, 4.2484e-18, 9.3576e-14,
        2.0612e-09, 4.5398e-05, 5.0000e-01, 9.9995e-01, 1.0000e+00, 1.0000e+00,
        1.0000e+00, 1.0000e+00, 1.0000e+00, 1.0000e+00, 1.0000e+00],
       grad_fn=&lt;MulBackward0&gt;)
grdA: tensor([1.8049e-35, 3.9754e-31, 8.7565e-27, 1.9287e-22, 4.2484e-18, 9.3576e-14,
        2.0612e-09, 4.5396e-05, 2.5000e-01, 4.5417e-05, 0.0000e+00, 0.0000e+00,
        0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00])
grdB: tensor([0.0000e+00, 0.0000e+00, 0.0000e+00, 1.9616e-22, 4.2484e-18, 9.3576e-14,
        2.0612e-09, 4.5396e-05, 2.5000e-01, 4.5396e-05, 2.0612e-09, 9.3576e-14,
        4.2484e-18, 1.9287e-22, 8.7565e-27, 3.9754e-31, 1.8049e-35])
relative asymmetry in gradient:
(grdA - grdA.flip (0)) / torch.max (grdA, grdA.flip (0)): tensor([ 1.0000e+00,  1.0000e+00,  1.0000e+00,  1.0000e+00,  1.0000e+00,
         1.0000e+00,  1.0000e+00, -4.5947e-04,  0.0000e+00,  4.5947e-04,
        -1.0000e+00, -1.0000e+00, -1.0000e+00, -1.0000e+00, -1.0000e+00,
        -1.0000e+00, -1.0000e+00])
(grdB - grdB.flip (0)) / torch.max (grdB, grdB.flip (0)): tensor([-1.0000e+00, -1.0000e+00, -1.0000e+00,  1.6765e-02,  0.0000e+00,
         7.2414e-08,  0.0000e+00,  1.6028e-07,  0.0000e+00, -1.6028e-07,
         0.0000e+00, -7.2414e-08,  0.0000e+00, -1.6765e-02,  1.0000e+00,
         1.0000e+00,  1.0000e+00])
</code></pre>
<p>(The current nightly build, version 1.11.0.dev20220123, yields the<br>
same result, as does performing the computation on the gpu, and<br>
performing the computation in double precision yields an equivalent<br>
result.)</p>
<p>See Yaroslav’s <a href="https://discuss.pytorch.org/t/custom-sigmoid-the-network-is-not-trained/142250">“Custom Sigmoid”</a> thread for some of the motivation<br>
behind this post.</p>
<p>Best.</p>
<p>K. Frank</p>

ptrblck: <p>CC <a class="mention" href="/u/alband">@albanD</a> for visibility.</p>