Customize an activation function

bille_du: <p>If i want to customize an activation function, and can be easily called in torch.nn.functional. What should I do? Thanks</p>

smth: <p>what do you mean by “customize an activation function”?<br>
and why do you need it to be called in <code>torch.nn.functional</code>?</p>

shicai: <p>I guess, “customize an activation function” means “how to implement some custom activation functions of his own”.</p>

jekbradbury: <p>If you can write your activation function using Torch math operations, you don’t need to do anything else to “implement” it.</p>

fmassa: <p>Let’s implement a truncated gaussian for example.</p>
<pre><code class="lang-python">def trucated_gaussian(x, mean=0, std=1, min=0.1, max=0.9):
    gauss = torch.exp((-(x - mean) ** 2)/(2* std ** 2))
    return torch.clamp(gauss, min=min, max=max) # truncate
</code></pre>

bille_du: <p>I’m sorry I did not express what I meant, and I mean is how I could use my own function in a net  instead of using the activation function provided in the pytorch framework. So,which document can reference? thanks.</p>

Tudor_Berariu: <p>One way to do it is to derive from <code>nn.Module</code> and implement the <code>forward</code> method. Don’t bother about backpropagation if you use autograd compatible operations. For example, <a class="mention" href="/u/fmassa">@fmassa</a>’s truncated gaussian from above:</p>
<pre><code>import torch
import torch.nn as nn
from torch.autograd import Variable

class MyActivationFunction(nn.Module):

    def __init__(self, mean=0, std=1, min=0.1, max=0.9):
        super(MyActivationFunction, self).__init__()
        self.mean = mean
        self.std = std
        self.min = min
        self.max = max

    def forward(self, x):
        gauss = torch.exp((-(x - self.mean) ** 2)/(2* self.std ** 2))
        return torch.clamp(gauss, min=self.min, max=self.max)

my_net = nn.Sequential(
    nn.Linear(7, 5),
    MyActivationFunction()
)

y = my_net(Variable(torch.rand(10, 7)))
y.backward(torch.rand(10, 5))</code></pre>

fmassa: <p>Note that it’s not necessary to derive from <code>nn.Module</code> to write your functions, and everything can be handled with normal python functions, and backpropagation will just work fine (if you pass in a <code>Variable</code>).</p>

osm3000: <p><a class="mention" href="/u/fmassa">@fmassa</a> Can you please give an example on that?<br>
I tried to implement a simple exponential activation for my rnn</p>
<pre><code>class my_rnn(nn.Module):
    def __init__(self, input_size=2, hidden_size=20, num_layers=3, output_size=1,
                 batch_size=10):
        super(my_rnn, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.output_size = output_size
        self.batch_size = batch_size
        
        self.rnn = nn.RNN(input_size=self.input_size, hidden_size=self.hidden_size, 
                          num_layers=self.num_layers, batch_first=True, nonlinearity=self.exp_activation)
        # The last layer is applied on the last output only of the RNN (not like
        # TimeDistributedDense in Keras)
        self.linear_layer = nn.Linear(self.hidden_size, self.output_size)
        
        self.hidden = Variable(torch.zeros((self.num_layers, self.batch_size, self.hidden_size))).cuda()
        
    def forward(self, input_sequence):
        out_rnn, self.hidden = self.rnn(input_sequence, self.hidden)
        in_linear = out_rnn[:, -1, :]
        final_output = self.linear_layer(in_linear)
        return final_output
    
    def init_hidden(self):
        self.hidden = Variable(torch.zeros((self.num_layers, self.batch_size, self.hidden_size))).cuda()

    def exp_activation(self, data):
        return torch.exp(data)
</code></pre>
<p>but it gives this error <code>ValueError: Unknown nonlinearity '&lt;bound method my_rnn.exp_activation of my_rnn ()&gt;'</code></p>

Tudor_Berariu: <p><code>nn.RNN</code> supports only <code>tanh</code> or <code>relu</code> for <code>nonlinearity</code> <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py#L191" rel="nofollow noopener">[code]</a>. One easy way to solve your problem is to write your own loop over the sequence just like in this <a href="https://github.com/pytorch/examples/blob/master/time_sequence_prediction/train.py#L11" rel="nofollow noopener">example</a>.</p>

fmassa: <p>maybe you are trying to call a non <code>@staticmethod</code>?<br>
Anyway, if your <code>exp_activation</code> is outside of the class definition, you can use it normally.</p>
<pre><code class="lang-python">def exp_activation_square(x):
    return torch.exp(x) ** 2

x = Variable(torch.rand(3), requires_grad=True)
y = Variable(torch.rand(3), requires_grad=True)

z = exp_activation_square(x * y).sum()
z.backward()
</code></pre>

fmassa: <p>Ah, <a class="mention" href="/u/tudor_berariu">@Tudor_Berariu</a> is right, I didn’t see all your code so I didn’t get where the problem was.<br>
Yes, if you want to try another non-linearity which is not currently supported in <code>nn.RNN</code>, you need to write your own for loop.<br>
Although it wouldn’t be hard to add support for it in <code>nn</code>, but I’m not sure it’s a very common use-case</p>

osm3000: <p>Ah okay, I see your point now. Thank you <a class="mention" href="/u/fmassa">@fmassa</a> and <a class="mention" href="/u/tudor_berariu">@Tudor_Berariu</a></p>

bille_du: <p>thank you ！everyone,I will try it,</p>

Vinay6666: <p>visit <a href="https://quickkt.com/tutorials/artificial-intelligence/deep-learning/activation-function/" rel="nofollow noopener">https://quickkt.com/tutorials/artificial-intelligence/deep-learning/activation-function/</a><br>
for video tutorial on activation functions</p>

herleeyandi: <p>Hello I have a question for implementing activation function. How can we implement our own activation function that need parameter?, Now I want to make like thresholding function where the threshold is determined in training this is similar with PReLU but in here I have a custom additional operation.</p>

ptrblck: <p>Similar to <code>PReLU</code> you could implement your own activation function writing a custom <code>nn.Module</code> (just like writing your model).</p>

Abi_Mehranian: <p>you can write a customized act function like below (e.g. weighted Tanh)</p>
<pre><code>class weightedTanh(nn.Module):
    def __init__(self, weights = 1):
        super().__init__()
        self.weights = weights
        
    def forward(self, input):
        ex = torch.exp(2*self.weights*input)
        return (ex-1)/(ex+1)</code></pre>

herleeyandi: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for your reply, could you give a simple example?, I have tried several things like <a href="https://discuss.pytorch.org/t/how-to-make-the-parameter-of-torch-nn-threshold-learnable/4729/12">this discussion</a> , but its seem failed.</p>

saba: <p>Hi Ptrblck,</p>
<p>I write the p-Sigmigd to have 3 learnable parameters as (Alpha, Beta, and Sigma). Would you please tell me that this implementation is correct?</p>
<pre><code class="lang-auto">class SigmoiidLearn(nn.Module):
    def __init__(self):
        super(SigmoiidLearn, self).__init__()
        
        self.Alpha = nn.Parameter(torch.ones(1), requires_grad=True)
        self.Beta = nn.Parameter(torch.zeros(1),requires_grad=True)   
        self.Sigma = nn.Parameter(torch.ones(1),requires_grad=True)

    def forward(self,input):
        Sigma=self.Sigma
        Beta=self.Beta
        Alpha=self.Alpha
        Out=torch.zeros(input.shape)
        for ii in range(input.shape[0]):
            for ii1 in range(input.shape[1]):
                for ii2 in range(input.shape[2]):
                    for ii3 in range(input.shape[3]):
                        vv=input[ii,ii1,ii2,ii3]

                        Out[ii,ii1,ii2,ii3]=Sigma*(1/(1+torch.exp((-1*(vv*Alpha))+Beta)))

        return Out
</code></pre>
<p>and the function that I call it as :</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self,ngpu,nz,ngf):
        super(Net, self).__init__()
        self.ngpu=ngpu
        self.nz=nz
        self.ngf=ngf
           
        self.l1= nn.Sequential( nn.ConvTranspose2d(self.nz, self.ngf * 8, 3, 1, 0, bias=False),
            nn.BatchNorm2d(self.ngf * 8),
            nn.ReLU(True))
        self.l2=nn.Sequential(SigmoiidLearn())

    def forward(self, input1):
        x = self.l1(input1)
        x =  self.l2(x)

        return x
</code></pre>