Why is softplus backward pass not equal to sigmoid?

username: Dementiy
cooked: <p>Hello, everyone. I can’t understand why is softplus backward pass not equal to sigmoid?</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.tensor(np.array([
    [-0.4578739 , -0.57322363, -0.85933977],
    [ 0.9095323 ,  0.78346789,  0.47258139],
    [-0.54425339, -0.92374175, -0.56345292]
]), requires_grad=True)
&gt;&gt;&gt; out = torch.nn.Softplus()(x)
&gt;&gt;&gt; out.backward(torch.DoubleTensor([[1., 1., 1.], [1., 1., 1.], [1.,1.,1.]]))
&gt;&gt;&gt; x.grad
tensor([
    [0.3875, 0.3605, 0.2975],
    [0.7129, 0.6864, 0.6160],
    [0.3672, 0.2842, 0.3627]
], dtype=torch.float64)

# Expected output after backward
&gt;&gt;&gt; torch.nn.Sigmoid()(out)
tensor([
    [0.6202, 0.6099, 0.5874],
    [0.7769, 0.7613, 0.7225],
    [0.6124, 0.5828, 0.6108]
], dtype=torch.float64, grad_fn=&lt;SigmoidBackward&gt;)
</code></pre>
<p>What am I missing?</p>

username: KFrank
cooked: <p>Hi Dmitriy!</p>
<aside class="quote no-group" data-username="Dementiy" data-post="1" data-topic="58760" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/dbc845/48.png" class="avatar"> Dementiy:</div>
<blockquote>
<p>Hello, everyone. I can’t understand why is softplus backward pass not equal to sigmoid?</p>
</blockquote>
</aside>
<p>You are perfectly correct that the derivative of <code>softplus</code><br>
is <code>sigmoid</code>, and that calling <code>backward()</code> should give<br>
you this derivative as the gradient with respect to <code>x</code>.</p>
<p>Your problem is that, for comparison, you are not calculating<br>
<code>sigmoid (x)</code> (but, rather, <code>sigmoid (out)</code>).</p>
<p>Here is a (pytorch 0.3.0) version of your script that makes<br>
the correct comparison:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)
import numpy as np

x = torch.autograd.Variable (torch.Tensor(np.array([
    [-0.4578739 , -0.57322363, -0.85933977],
    [ 0.9095323 ,  0.78346789,  0.47258139],
    [-0.54425339, -0.92374175, -0.56345292]
])), requires_grad = True)
out = torch.nn.Softplus()(x)
out.backward(torch.Tensor([[1., 1., 1.], [1., 1., 1.], [1.,1.,1.]]))

x.grad
torch.nn.Sigmoid()(x)
torch.nn.Sigmoid()(out)
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
0.3.0b0+591e73e
&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt;
&gt;&gt;&gt; x = torch.autograd.Variable (torch.Tensor(np.array([
...     [-0.4578739 , -0.57322363, -0.85933977],
...     [ 0.9095323 ,  0.78346789,  0.47258139],
...     [-0.54425339, -0.92374175, -0.56345292]
... ])), requires_grad = True)
&gt;&gt;&gt; out = torch.nn.Softplus()(x)
&gt;&gt;&gt; out.backward(torch.Tensor([[1., 1., 1.], [1., 1., 1.], [1.,1.,1.]]))
&gt;&gt;&gt;
&gt;&gt;&gt; x.grad
Variable containing:
 0.3875  0.3605  0.2975
 0.7129  0.6864  0.6160
 0.3672  0.2842  0.3627
[torch.FloatTensor of size 3x3]

&gt;&gt;&gt; torch.nn.Sigmoid()(x)
Variable containing:
 0.3875  0.3605  0.2975
 0.7129  0.6864  0.6160
 0.3672  0.2842  0.3627
[torch.FloatTensor of size 3x3]

&gt;&gt;&gt; torch.nn.Sigmoid()(out)
Variable containing:
 0.6202  0.6099  0.5874
 0.7769  0.7613  0.7225
 0.6124  0.5828  0.6108
[torch.FloatTensor of size 3x3]
</code></pre>
<p>You can see that <code>x.grad</code> matches <code>sigmoid (x)</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Dementiy
cooked: <p>Yes, now I see my mistake. Thanks!</p>