Pass variable to custom tanh function

hohohihi: <p>Hi, i am making custom activation function here.<br>
I want to pass a variable(which is pre-defined outside of function) to the backward function.</p>
<p>Here is the code.<br>
But it says</p>
<blockquote>
<p>AttributeError: ‘custom_tanh_variableBackward’ object has no attribute ‘variable’</p>
</blockquote>
<p>So, i don’t know what to do.<br>
Help me.</p>
<pre><code class="lang-auto">self.activation = custom_tanh_variable(variable).apply


class custom_tanh_variable(torch.autograd.Function):

    def __init__(self,ratio_non_sparsity):

        super().__init__()
        self.variable= variable

    @staticmethod
    def forward(ctx,input):
        minus_input = input*(-1)
        tmp1 = torch.exp(input) - torch.exp(minus_input)
        tmp2 = torch.exp(input) + torch.exp(minus_input)
        ans  = torch.div(tmp1,tmp2)
        ctx.save_for_backward(ans)
        return ans

    @staticmethod
    def backward(ctx,grad_output):

        input, = ctx.saved_tensors
        tmp = ctx.variable
        print(tmp)

        return grad_output*(1-torch.square(input))
</code></pre>

ptrblck: <p>You are using <code>staticmethod</code>s so would have to pass the variable to the <code>forward</code> and/or <code>backward</code> method. If you need to register a parameters/buffer etc. create a custom <code>nn.Module</code>, register the data there, and call the custom <code>autograd.Function</code> in the module’s <code>forward</code>.</p>

hohohihi: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi, having good weekend?</p>
<ol>
<li>
<p>do you want me to do somehow like this(<a href="https://pytorch.org/docs/stable/notes/extending.html#adding-a-module" class="inline-onebox" rel="noopener nofollow ugc">Extending PyTorch — PyTorch 1.11.0 documentation</a>) ?</p>
</li>
<li>
<p>I cannot understand the ‘register’? you mean like this (<code>self.variable= variable</code>)?</p>
</li>
</ol>
<p>Thank you sir.</p>

ptrblck: <ol>
<li>
<p>Yes, the tutorial explains how to write a custom <code>nn.Module</code> which you could use as a template to implement yours.</p>
</li>
<li>
<p>Yes, assuming <code>variable</code> is an <code>nn.Parameter</code>. If it’s a buffer (i.e. a non-trainable tensor which should be used in the <code>forward</code> and stored in the <code>state_dict</code>) use <code>self.register_buffer</code>.</p>
</li>
</ol>

hohohihi: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> .</p>
<p>One last question. In here(<a href="https://pytorch.org/docs/stable/notes/extending.html#adding-a-module" class="inline-onebox" rel="noopener nofollow ugc">Extending PyTorch — PyTorch 1.11.0 documentation</a>), there is no backward function in nn.Module. i want to use custom backward function, as i originally did on autograd.function in my first post on this thread.</p>
<p>So, can i write <code>def backward</code> in nn.Module and call backward in autograd.function ?</p>
<p>Or i don’t have to do write def backward, because the nn.Module automatically build ‘backward’ from my own-written autograd.function(which is custom backward as my original post on this thread)?</p>

ptrblck: <aside class="quote no-group" data-username="hohohihi" data-post="5" data-topic="149313">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/bbce88/48.png" class="avatar"> hohohihi:</div>
<blockquote>
<p>So, can i write <code>def backward</code> in nn.Module and call backward in autograd.function ?</p>
</blockquote>
</aside>
<p>No need to write a custom <code>backward</code> function in <code>nn.Module</code> since you’ve already defined the custom <code>backward</code> in the <code>autograd.Function</code>. PyTorch is smart enough to call this <code>backward</code> method if you use your <code>autograd.Function</code> via <code>.apply</code> in <code>nn.Module.forward</code>.</p>

hohohihi: <p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I had tried various way. But i couldn’t find the way.<br>
Here is the code that i tried.</p>
<p>I think I still don’t know how to pass variable to backward function. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>class Module_custom_tanh(nn.Module):<br>
def <strong>init</strong>(self,val):<br>
super(Module_custom_tanh,self).<strong>init</strong>()<br>
self.val= val</p>
<pre><code>def forward(self,input):
    return custom_tanh.apply(input,self.val)
</code></pre>
</blockquote>
<pre><code class="lang-auto"> class custom_tanh(torch.autograd.Function):
     def __init__(self):
         super().__init__()

    @staticmethod
    def forward(ctx,input,val):
        minus_input = input*(-1)
        tmp1 = torch.exp(input) - torch.exp(minus_input)
        tmp2 = torch.exp(input) + torch.exp(minus_input)
        ans  = torch.div(tmp1,tmp2)
        ctx.save_for_backward(ans)
        ctx.register_buffer('val',val)
        return ans

    @staticmethod
    def backward(ctx,grad_output):
        input, = ctx.saved_tensors
        val= ctx.val
        print(val)
        return grad_output*(1-torch.square(input))
</code></pre>
<p>but this gets me <code>AttributeError: 'custom_tanh_non_sparsityBackward' object has no attribute 'register_buffer'</code></p>

ptrblck: <p><code>autograd.Function</code>s are stateless and cannot register buffers.<br>
If you want to register <code>self.val</code> as a buffer or parameter, do so in the <code>nn.Module</code> (i.e. <code>Module_custom_tanh</code>).<br>
In your <code>custom_tanh</code> pass all needed tensors to the <code>forward</code> and use <code>ctx.save_for_backward</code> if you need specific tensors in the <code>backward</code> function.</p>

hohohihi: <p>Plus, after resolving this issue <a href="https://discuss.pytorch.org/t/how-to-mark-argument-as-nondifferentiable-in-a-custom-autograd-function/86693" class="inline-onebox">How to mark argument as nondifferentiable in a custom autograd Function</a>, I am good <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>thank you sir <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>