MySigmoid(torch.autograd.Function)

Sumanth_Nandamuri: <p>I am just experimenting with defining new autograd Function for sigmoid <strong>MySigmoid</strong> similar to <strong>MyReLU</strong> in the tutorial, but the output for loss is printing <strong>nan</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f3d4ef1c7aeb479ff9ca92cfab4d587595e42b8d.png" data-download-href="https://discuss.pytorch.org/uploads/default/f3d4ef1c7aeb479ff9ca92cfab4d587595e42b8d" title="52 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f3d4ef1c7aeb479ff9ca92cfab4d587595e42b8d_2_690x186.png" alt="52 AM" data-base62-sha1="yN2ewV7vk4oFhIlIDq23294Skbz" width="690" height="186" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f3d4ef1c7aeb479ff9ca92cfab4d587595e42b8d_2_690x186.png, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f3d4ef1c7aeb479ff9ca92cfab4d587595e42b8d_2_1035x279.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f3d4ef1c7aeb479ff9ca92cfab4d587595e42b8d_2_1380x372.png 2x" data-dominant-color="EDEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">52 AM</span><span class="informations">1536×416 49.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What is wrong?</p>

SimonW: <p>Not sure if it is root of cause, but you shouldn’t use numpy functions.</p>

Sumanth_Nandamuri: <p>same problem even if I try torch.exp</p>

SimonW: <p>I just tried your code on master and the forward works fine. When are you seeing loss becoming nan? First forward? After a backward?</p>

luis_vasquez: <p>Sorry to revive this topic:<br>
I managed to implement a working version of <strong>MySigmoid</strong>, but Im also having some issues:</p>
<pre><code class="lang-auto">import torch

class MySigmoid(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        ctx.save_for_backward(input)
        sigmoid_eval = 1.0/(1.0 + torch.exp(-input))
        return sigmoid_eval

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        grad_temp = grad_output.clone()
        grad_input = torch.exp(-input)/((1.0 + torch.exp(-input)).pow(2))
        return grad_input

dtype = torch.float
device = torch.device("cpu")

N, D_in, D_out = 1000, 1, 1
alpha = 1e-2



x = torch.cat(
                (   torch.randn(N, D_in, device=device, dtype=dtype) ,
                    torch.ones([N,1], dtype=dtype, device=device)   ),
                1)
y = x[:,0:1]*3.5 + 2.5*torch.randn(N,1)
w = torch.randn(D_in+1, D_out, device=device, dtype=dtype, requires_grad=True)

print(w.shape)
print(w.dtype)

for t in range(5000):
    sigmoid = MySigmoid.apply
    
    y_pred = sigmoid(x.mm(w))

    loss = (y_pred - y).pow(2).sum()
    print(t, loss.item())

    loss.backward()
    
    with torch.no_grad():
        w -=  alpha * w.grad
        w.grad.zero_()
</code></pre>
<p>I  made sure the dimensions were correct, but still having this issues:</p>
<ul>
<li>If I try to multiply <strong>w</strong> by a scalar I get this:</li>
</ul>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test.py", line 49, in &lt;module&gt;
    w -=  alpha * w.grad
TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'
</code></pre>
<ul>
<li>The convergence is taking too long (reason why Im trying to modify starting <strong>w</strong> values)</li>
</ul>
<p>I would appreciate if someone could give a lock to this idea, which I think is very useful for those of us who have just started using torch. Thanks in advance</p>

Jesse: <p>change to <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"><br>
grad_input =grad_temp* torch.exp(-input)/((1.0 + torch.exp(-input)).pow(2))</p>