Relu with leaky derivative

username: nthn_clmnt
cooked: <p>My understanding is that for classification tasks there is the intuition that:<br>
(1) relu activation functions encourage sparsity, which is good (for generalization?) but that<br>
(2) a leaky relu solves the gradient saturation problem, which relu has, at the cost of sparsity.</p>
<p>Is it possible, in PyTorch, to write an activation function which on the forward pass behaves like relu but which has a small positive derivative for x &lt; 0?  In other words its forward pass is relu but its backward pass is like that of a leaky relu with some specified leak rate.  Of course this breaks the FTC, but that (probably?) doesn’t matter and you would have sparsity + non-saturating gradients if it worked.</p>
<p>I’m relatively new to deep learning, so sorry if this has already been tried. I also don’t have the experience to know if tuning activation functions leads to significant performance gains.</p>

username: tom
cooked: <p>For ad-hoc experimentation with backward not quite equal to forward, the pattern</p>
<pre><code class="lang-auto">y = x_backward + (x_forward - x_backward).detach()
</code></pre>
<p>works quite well. It get’s you <code>x_forward</code> in the forward, but the derivative will act as if you had <code>x_backward</code>. Stay clear of <code>NaN</code> (and <code>infinity - infinity</code> which is NaN). It’s a tad more expensive than a custom autograd.Function, probably.</p>
<p>Best regards</p>
<p>Thomas</p>

username: raaj043
cooked: <p>Hi Tom,<br>
Thats great idea! What I was looking for.<br>
Could you please let me know how to use it? I am little confused with x_backward? should this used after the loss.backward()?<br>
please help!</p>

username: ptrblck
cooked: <p><code>x_backward</code> would correspond to the activation in your <code>forward</code> method, which should be used during the backward pass. In this example, you could define <code>x_backward</code> as the <code>LeakyReLU</code> output and <code>x_forward</code> as the <code>ReLU</code> output.</p>

username: nthn_clmnt
cooked: <p>This should work for what you want, all wrapped up as the activation function ‘non_sat_relu’. The backward leak is set to 0.1, but it could of course be whatever you want.</p>
<pre><code class="lang-auto">import torch

class NSReLU(torch.autograd.Function):
    @staticmethod
    def forward(self,x):
        self.neg = x &lt; 0
        return x.clamp(min=0.0)
    @staticmethod
    def backward(self,grad_output):
        grad_input = grad_output.clone()
        grad_input[self.neg] *= 0.1
        return grad_input

def non_sat_relu(x):
    return NSReLU.apply(x)</code></pre>

username: ecdrid
cooked: <p><a class="mention" href="/u/nthn_clmnt">@nthn_clmnt</a>  Just curious, when using <code>@staticmethod</code>, why are we passing <code>self</code>?<br>
Plus we can also implement this via inheriting <code>nn.Module</code> as well, right?<br>
Thanks!</p>

username: raaj043
cooked: <p>Thank you! Its helpful</p>

username: raaj043
cooked: <p>Thank you for the detailed explanation and code. Much obliged</p>

username: nthn_clmnt
cooked: <p>I am following this:<br>
<a href="https://pytorch.org/docs/master/notes/extending.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/master/notes/extending.html</a><br>
and this:<br>
<a href="https://pytorch.org/docs/master/autograd.html#torch.autograd.Function.forward" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/master/autograd.html#torch.autograd.Function.forward</a></p>
<p>I guess the secret first argument <code>ctx</code> of forward and backward is not a class instance but something called context. I think you forced me to learn something about Python, so thank you <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> So though my code runs fine, it is suggestive of the wrong thing to call that argument self.</p>
<p>I haven’t seen any documentation that you can re-implement backward at the level of an <code>nn.Module</code> rather than a <code>torch.autograd.Function</code>.</p>

username: tom
cooked: <p>Just because this is a useful pattern, but inefficient for larger Tensors, here is a variant that is less concise in expression but avoids the unneeded overhead:</p>
<pre><code class="lang-python">import torch
x_forward = torch.randn(5)
x_backward = torch.randn(5, requires_grad=True)

# the following is an efficient alternative for
# x2 = x_backward + (x_forward - x_backward).detach()

class FWBWMismatch(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x_forward, x_backward):
        ctx.shape = x_backward.shape
        return x_forward
    @staticmethod
    def backward(ctx, grad_in):
        return None, grad_in.sum_to_size(ctx.shape)

x2 = FWBWMismatch.apply(x_forward, x_backward)
y = (x2**2).sum()
y.backward()
# check against expectation
print((x_backward.grad - 2*x_forward).max())
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

username: 111179
cooked: <p>Hi,  I would like to use torch.autograd to implement leaky relu activate function, but I had trouble in implementing LeakyReLU(x)=max(0,x)+negative_slope∗min(0,x)…can you give me some advice?<br>
Thank you very much!</p>

username: G.M
cooked: <pre><code class="lang-auto">tc.clamp(x, min=0)+tc.clamp(x, max=0)*SLOPE
</code></pre>
<p>or</p>
<pre><code class="lang-auto">tc.where(x&lt;0, x*SLOPE, X)
</code></pre>

username: 111179
cooked: <p>Than you for your reply.<br>
But how to define in backward function?<br>
My code is as follows, which are show “‘MyReLUBackward’ object has no attribute ‘clamp’”.<br>
import torch<br>
from torch.autograd import Variable<br>
class MyReLU(torch.autograd.Function):<br>
<span class="mention">@staticmethod</span><br>
def forward(self, input, negative_slope):<br>
self.save_for_backward(input)<br>
output = self.clamp(input,min=0)+self.clamp(input, max=0)*negative_slope<br>
return output<br>
<span class="mention">@staticmethod</span><br>
def backward(self, grad_output):<br>
input, = self.saved_tensors<br>
grad_input = grad_output.clone()<br>
return grad_input</p>
<p>input = Variable(torch.linspace(-3, 3, steps=5))<br>
print (input)<br>
leakyrelu = MyReLU().apply<br>
output= leakyrelu(input, 0.02)<br>
print (output)</p>

username: G.M
cooked: <p>You can follow the tutorial <a href="https://pytorch.org/docs/1.0.1/autograd.html#function" rel="nofollow noopener">here</a>. The derivatives for LeakyReLU when <code>x&gt;0</code> is <code>1</code> and <code>-NEGATIVE_SLOPE</code> when <code>x&lt;=0</code>. Like what <a class="mention" href="/u/nthn_clmnt">@nthn_clmnt</a> said, the argument <code>self</code> shouldn’t be named “self” becuase it is very confusing, it is actually a “context” object that holds information. When <code>apply</code> is called, both <code>forward</code>'s and <code>backward</code>'s first argument will be the context object, followed by other arguments, so <code>input</code> is the actual input you passed.</p>

username: 111179
cooked: <p>Thank you very much!</p>