How to define a parametrized tanh activation function with 2 trainable parameters?

renken: <p>Hi, i want to define anactivation function with 2 trainable parameters, k and c, which define the function.<br>
I want my neural net to calibrate those parameters aswell during the training procedure. Do you have an idea on how i can manage to do that in few lines? I am really new on pytorch.</p>
<p>Here is my code for the moment, with fixed values of k and c as you can see…</p>
<pre><code class="lang-auto">def transpose_conv(in_channels,out_channels,padding):
    return nn.Sequential(
           nn.ConvTranspose3d(in_channels, out_channels, 4, stride=2, padding=padding, bias=False), 
           nn.BatchNorm3d(out_channels,affine=True),
           nn.ReLU(inplace=True)
           )  
class Generator(nn.Module):
    def __init__(self):
        super().__init__()
        self.transpose_conv_1 = transpose_conv(hnnc,512,2)
        self.transpose_conv_2 = transpose_conv(512,256,2)
        self.transpose_conv_3 = transpose_conv(256,128,2)
        self.transpose_conv_4 = transpose_conv(128,64,2)
        self.transpose_conv_5 = transpose_conv(64,nc,3)
    def forward(self, x):
        c, k = 0.5, 0.25
        t_c_1  = self.transpose_conv_1(x)
        t_c_2  = self.transpose_conv_2(t_c_1)
        t_c_3  = self.transpose_conv_3(t_c_2)
        t_c_4  = self.transpose_conv_4(t_c_3)
        t_c_5  = self.transpose_conv_5(t_c_4)
        result = 0.5*(torch.tanh((1/k)*(t_c_5-c))+1)
        return result
</code></pre>
<p>Thank you in advance</p>

ptrblck: <p>You can create a custom <code>nn.Module</code> and define both parameters as trainable. Something like this could work:</p>
<pre><code class="lang-python">class MyActivation(nn.Module):
    def __init__(self):
        super().__init__()
        self.c = nn.Parameter(torch.tensor(0.5))
        self.k = nn.Parameter(torch.tensor(0.25))
        
    def forward(self, x):
        return 0.5*(torch.tanh((1/self.k)*(x-self.c))+1)
</code></pre>
<p>Of course you can also randomly initialize both parameters or pass the init value as arguments to <code>MyActivation</code> etc.</p>

renken: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="161720">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">class MyActivation(nn.Module):
    def __init__(self):
        super().__init__()
        self.c = nn.Parameter(torch.tensor(0.5))
        self.k = nn.Parameter(torch.tensor(0.25))
        
    def forward(self, x):
        return 0.5*(torch.tanh((1/self.k)*(x-self.c))+1)
</code></pre>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>Thank you for your comment and your code ! I tried this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-auto"># defining a trainable tanh function
class Trainable_Tanh(nn.Module):
    def __init__(self):
        super().__init__()
        self.c = nn.Parameter(torch.tensor(0.5))
        self.k = nn.Parameter(torch.tensor(0.25))   
    def forward(self, x):
        return 0.5*(torch.tanh((1/self.k)*(x-self.c))+1)

# defining the Generator
def transpose_conv_G(in_channels,out_channels,padding):
    return nn.Sequential(
           nn.ConvTranspose3d(in_channels, out_channels, 4, stride=2, padding=padding, bias=False), 
           nn.BatchNorm3d(out_channels,affine=True),
           nn.ReLU(inplace=True)
           )  
class Generator(nn.Module):
    def __init__(self):
        super().__init__()
        self.transpose_conv_1 = transpose_conv_G(hyper_nb_noise_channels,512,2)
        self.transpose_conv_2 = transpose_conv_G(512,256,2)
        self.transpose_conv_3 = transpose_conv_G(256,128,2)
        self.transpose_conv_4 = transpose_conv_G(128,64,2)
        self.transpose_conv_5 = transpose_conv_G(64,nc,3)
    def forward(self, x):
        c,k=0.5,0.25
        t_c_1  = self.transpose_conv_1(x)
        t_c_2  = self.transpose_conv_2(t_c_1)
        t_c_3  = self.transpose_conv_3(t_c_2)
        t_c_4  = self.transpose_conv_4(t_c_3)
        t_c_5  = self.transpose_conv_5(t_c_4)
        # calling the trainable tanh
        myTanh = Trainable_Tanh()
        result = myTanh(t_c_5)
        return result
</code></pre>
<p>I got 3 last questions please:</p>
<ul>
<li>is it a right way to call the newly created activation ?</li>
<li>am I supposed to perform somewhere “myTanh.zero_grad()”, “myTanh.eval()” ? or is it already taken into account when i do it with the entire net: netG = Generator().to(device) ; netG.zero_grad()</li>
<li>when i do a torch summary of my final network, i don’t observe 2 more trainable parameters. I have exactly the same number of trainabe parameters as before. It is strange because normally i introduced 2 additionnal parameters. Please could you tell me where i am wrong ?</li>
</ul>

ptrblck: <aside class="quote no-group" data-username="renken" data-post="3" data-topic="161720">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/renken/48/44622_2.png" class="avatar"> renken:</div>
<blockquote>
<p>is it a right way to call the newly created activation ?</p>
</blockquote>
</aside>
<p>No, since you are re-initializing the trainable activation module in each forward pass (so it won’t actually be trained at all).<br>
Treat this module as any other layer, initialize it in your <code>Generator.__init__</code> method, and use it in the <code>forward</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">class Generator(nn.Module):
    def __init__(self):
        super().__init__()
        self.myTanh = Trainable_Tanh()
        ...

    def forward(self, x):
        ...
        result = myTanh(t_c_5)
        ...
</code></pre>
<aside class="quote no-group" data-username="renken" data-post="3" data-topic="161720">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/renken/48/44622_2.png" class="avatar"> renken:</div>
<blockquote>
<p>am I supposed to perform somewhere “myTanh.zero_grad()”, “myTanh.eval()” ? or is it already taken into account when i do it with the entire net: netG = Generator().to(device) ; netG.zero_grad()</p>
</blockquote>
</aside>
<p>If you register <code>Trainable_Tanh</code> properly in the <code>Generator.__init__</code> method, its gradients will be zeroed out from the parent module and also its training flag will be changed from the parent. As mentioned before: just treat it as any other layer such as <code>nn.Linear</code>.</p>
<aside class="quote no-group" data-username="renken" data-post="3" data-topic="161720">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/renken/48/44622_2.png" class="avatar"> renken:</div>
<blockquote>
<p>when i do a torch summary of my final network, i don’t observe 2 more trainable parameters. I have exactly the same number of trainabe parameters as before.</p>
</blockquote>
</aside>
<p>I guess because you are never registering it as mentioned in the previous points.</p>

renken: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I added the trainable tanh as any other layer, but i still don’t have the parameters in my torch summary.</p>
<p>Did i miss something ?</p>
<pre><code class="lang-auto"># defining a trainable tanh function
class Trainable_Tanh(nn.Module):
    def __init__(self):
        super().__init__()
        self.c = nn.Parameter(torch.tensor(0.5))
        self.k = nn.Parameter(torch.tensor(0.25))
    def forward(self, x):
        return 0.5*(torch.tanh((1/self.k)*(x-self.c))+1)

# defining the Generator
def transpose_conv_G(in_channels,out_channels,padding):
    return nn.Sequential(
           nn.ConvTranspose3d(in_channels, out_channels, 4, stride=2, padding=padding, bias=False), 
           nn.BatchNorm3d(out_channels,affine=True),
           nn.ReLU(inplace=True)
           )  
class Generator(nn.Module):
    def __init__(self):
        super().__init__()
        self.transpose_conv_1 = transpose_conv_G(hyper_nb_noise_channels,512,2)
        self.transpose_conv_2 = transpose_conv_G(512,256,2)
        self.transpose_conv_3 = transpose_conv_G(256,128,2)
        self.transpose_conv_4 = transpose_conv_G(128,64,2)
        self.transpose_conv_5 = transpose_conv_G(64,nc,3)
        self.myTanh           = Trainable_Tanh()
    def forward(self, x):
        t_c_1  = self.transpose_conv_1(x)
        t_c_2  = self.transpose_conv_2(t_c_1)
        t_c_3  = self.transpose_conv_3(t_c_2)
        t_c_4  = self.transpose_conv_4(t_c_3)
        t_c_5  = self.transpose_conv_5(t_c_4)
        result = self.myTanh(t_c_5)
        return result
</code></pre>
<p>Here is the torch summary, and i see that the tanh layer don’t have the 2 additional trainable parameters… do you have an idea on how to fix my code please ?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8cea3509f1019d318c96a9c63661e621e6eecf0.png" data-download-href="https://discuss.pytorch.org/uploads/default/c8cea3509f1019d318c96a9c63661e621e6eecf0" title="summary"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8cea3509f1019d318c96a9c63661e621e6eecf0.png" alt="summary" data-base62-sha1="sEqdg3iEfBz3ApRTNkBaXkoCvAI" width="546" height="500" data-dominant-color="F0EFF0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">summary</span><span class="informations">555×508 46.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Your implementation looks correct and I also see the parameters:</p>
<pre><code class="lang-python">
hyper_nb_noise_channels = 1
nc = 1
model = Generator()
print(dict(model.named_parameters()))
...
'myTanh.c': Parameter containing:
tensor(0.5000, requires_grad=True), 'myTanh.k': Parameter containing:
tensor(0.2500, requires_grad=True)}

from torchinfo import summary
summary(model)
=================================================================
Layer (type:depth-idx)                   Param #
=================================================================
Generator                                --
├─Sequential: 1-1                        --
│    └─ConvTranspose3d: 2-1              32,768
│    └─BatchNorm3d: 2-2                  1,024
│    └─ReLU: 2-3                         --
├─Sequential: 1-2                        --
│    └─ConvTranspose3d: 2-4              8,388,608
│    └─BatchNorm3d: 2-5                  512
│    └─ReLU: 2-6                         --
├─Sequential: 1-3                        --
│    └─ConvTranspose3d: 2-7              2,097,152
│    └─BatchNorm3d: 2-8                  256
│    └─ReLU: 2-9                         --
├─Sequential: 1-4                        --
│    └─ConvTranspose3d: 2-10             524,288
│    └─BatchNorm3d: 2-11                 128
│    └─ReLU: 2-12                        --
├─Sequential: 1-5                        --
│    └─ConvTranspose3d: 2-13             4,096
│    └─BatchNorm3d: 2-14                 2
│    └─ReLU: 2-15                        --
├─Trainable_Tanh: 1-6                    2
=================================================================
Total params: 11,048,836
Trainable params: 11,048,836
Non-trainable params: 0
=================================================================
</code></pre>