Defining Custom leaky_relu functions

111179: <p>I try to defining custom leaky_relu function base on autograd, but the code shows “function MyReLUBackward returned an incorrect number of gradients (expected 2, got 1)”, can you give me some advice?<br>
Thank you so much for your help.<br>
the code as shown:</p>
<p>import torch<br>
from torch.autograd import Variable<br>
import math<br>
class MyReLU(torch.autograd.Function):<br>
<span class="mention">@staticmethod</span><br>
def forward(ctx, input,negative_slope):<br>
output = input.clamp(min=0)+input.clamp(max=0)*negative_slope<br>
ctx.save_for_backward(input)<br>
return output<br>
<span class="mention">@staticmethod</span><br>
def backward(ctx, grad_output):<br>
input, = ctx.saved_tensors<br>
negative_slope, = ctx.saved_tensors<br>
grad_input = grad_output.clone()<br>
return grad_input<br>
dtype = torch.float</p>
<p>N, D_in, H, D_out = 64, 1000, 100, 10</p>
<p>x = torch.randn(N, D_in, device=device, dtype=dtype)<br>
y = torch.randn(N, D_out, device=device, dtype=dtype)</p>
<p>w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)<br>
w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)</p>
<p>learning_rate = 1e-6<br>
for t in range(500):<br>
relu = MyReLU.apply<br>
y_pred = relu(x.mm(w1),0.01).mm(w2)<br>
loss = (y_pred - y).pow(2).sum()<br>
if t % 100 == 99:<br>
print(t, loss.item())<br>
loss.backward()<br>
with torch.no_grad():<br>
w1 -= learning_rate * w1.grad<br>
w2 -= learning_rate * w2.grad<br>
w1.grad.zero_()<br>
w2.grad.zero_()</p>

DAlolicorn: <p>Hi, the problem is that the backward function expect you to return all gradients for each input argument of forward function.<br>
Since you don’t need gradient for the argument negative_slope, just return None:</p>
<pre><code class="lang-auto">@staticmethod
def forward(ctx, input,negative_slope):
    output = input.clamp(min=0)+input.clamp(max=0)*negative_slope
    ctx.save_for_backward(input)
    ctx.slope = negative_slope
    return output
@staticmethod
def backward(ctx, grad_output):
    input, = ctx.saved_tensors
    slope = ctx.slope
    grad_input = grad_output.clone()
    grad_input = grad_input * (input &gt; 0).float() + grad_input * (input &lt; 0).float() * slope
    return grad_input, None
</code></pre>
<p>By the way corrected the backward and forward function without testing, hope it works.</p>

111179: <p>yep! Thank you !!! <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>

Chinmay_Rane: <p>Hi thank you the custom leaky. I was wondering. if i trained the negative slope value here, where please correct me if i am wrong. I need to pass the gradient required for the slope in backward propagation as i did below after calculating the gradient for slope</p>
<p><span class="mention">@staticmethod</span><br>
def forward(ctx, input,negative_slope):<br>
output = input.clamp(min=0)+input.clamp(max=0)*negative_slope<br>
ctx.save_for_backward(input)<br>
ctx.slope = negative_slope<br>
return output<br>
<span class="mention">@staticmethod</span><br>
def backward(ctx, grad_output):<br>
input, = ctx.saved_tensors<br>
slope = ctx.slope<br>
grad_input = grad_output.clone()<br>
grad_input = grad_input * (input &gt; 0).float() + grad_input * (input &lt; 0).float() * slope</p>
<pre><code>return grad_input, negative_slope_gradient
</code></pre>
<p>how can i get the updated negative slope after training and also, how can i check if it is training simultaneously. I, also understand we need to do negative_slope.requires_grad = True. Any help is appreciated. Thank you</p>

111179: <p>Hi, the <code>negative_slope</code> is a constant as input not learnable parameters. If you want to implement  a ReLU function with learnable parameters, you can refer to PReLU, for example: <a href="https://medium.com/@shoray.goel/prelu-activation-e294bb21fefa" rel="noopener nofollow ugc">https://medium.com/@shoray.goel/prelu-activation-e294bb21fefa</a></p>
<p>Hope it can help you!</p>

Chinmay_Rane: <p>Hi thanks for your response. the medium post was helpful. I designed a trainable parameter. .</p>
<p>I have question on the above leaky relu. have you tried running it on cuda? On CPU the forward and backward both work but when you shift device to cuda the code doesn’t access the backward function, it only access the forward pass. I have also created a new question in the hopes to get the answer here <a href="https://discuss.pytorch.org/t/custom-backward-function-using-function-from-torch-autograd-fails-on-cuda-but-works-on-cpu/114157" class="inline-onebox">Custom Backward function using Function from torch.autograd fails on cuda but works on cpu</a></p>
<p>Please let me know if it works for you. It definitely doesn’t work for me. I even tried the make_dirty → ctx.mark_dirty(output). I get the following error -&gt;RuntimeError: a leaf Variable that requires grad has been used in an in-place operation. then i also tried the ‘output.view_as(output)’ but i got ‘RuntimeError: Some elements marked as dirty during the forward method were not returned as output. The inputs that are modified inplace must all be outputs of the Function.’</p>

111179: <p>Sorry, I haven’t try it on GPU. But, I implement another custom activation function on GPU, for instance:<br>
class Surrogate_BP_Function(torch.autograd.Function):</p>
<pre><code class="lang-auto">    @staticmethod
    def forward(ctx, input):
        ctx.save_for_backward(input)
        out = torch.zeros_like(input).cuda()
        out[input &gt; 0] = 1.0
        return out

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad = grad_input * 0.3 * F.threshold(1.0 - torch.abs(input), 0, 0)
        return grad
</code></pre>
<p>It works on GPU.<br>
I hope it can help you.</p>

Chinmay_Rane: <aside class="quote no-group" data-username="111179" data-post="7" data-topic="72576">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/111179/48/22195_2.png" class="avatar"> 111179:</div>
<blockquote>
<pre><code class="lang-auto">@staticmethod
    def forward(ctx, input):
        ctx.save_for_backward(input)
        out = torch.zeros_like(input).cuda()
        out[input &gt; 0] = 1.0
        return out

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad = grad_input * 0.3 * F.threshold(1.0 - torch.abs(input), 0, 0)
        return grad
</code></pre>
</blockquote>
</aside>
<p>hey thanks for helping but i checked and it doesn’t seem to use your custom backward pass. it uses pytorches default i believe.</p>
<p>way to check- try using breakpoints maybe on spyder or jupyter. insert break points inside forward and backward.</p>
<p>2nd compare your loss values for both cuda and cpu.</p>