Custom activation functions with trainable parameters

Jamesswiz: <p>I am trying to implement this paper <a href="https://arxiv.org/pdf/1801.09403.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1801.09403.pdf</a><br>
The idea is to define a custom activation function using basic functions like relu, tanh etc with affine or convex constraints.</p>
<p>Here is a class structure I created but can’t make it work</p>
<pre><code> class combine_act(nn.Module):
      def __init__(self, in_features,act,comb='affine'):
           super(combine_act,self).__init__()
  
           self.in_features = in_features
           self.act = act                       #list of activations to use
           self.comb = comb                #type of combination [affine or convex]
           self.N = len(self.act)      
  
           self.alpha = nn.Parameter(torch.randn(len(act)))
           self.alpha.requiresGrad = True
  
  
           Lin = lambda x: x
  
           activations = {'linear': Lin, 'relu': nn.ReLU(), 'tanh': nn.Tanh()}
  
  
           def my_softmax(self, x):
                 means = torch.mean(x, 1, keepdim=True)[0]
                 x_exp = torch.exp(x-means)
                 x_exp_sum = torch.sum(x_exp, 1, keepdim=True)
                 return x_exp/x_exp_sum
  

           
  
           def get_alpha(self):
               self.eps = 1e-7
               if comb=='affine':
               return self.alpha / self.alpha.sum(1, keepdim=True).clamp(min=self.eps)
           else:
               return self.my_softmax(self.alpha)     

    
           def forward(self, x):
               out=x-x 
               weight = self.get_alpha()
               for i in range(self.N):
                    activation = activations[self.act[i]]
                    out += weight[i]*activation(x)
               return out
</code></pre>
<p>Call to this function would look like this:<br>
x = nn.Linear(100,100)(x)<br>
x = combine_act([‘linear’, ‘relu’, ‘tanh’],comb=‘affine’)(x)</p>
<hr>
<p>I am getting this error:   ‘combine_act’ object has no attribute ‘dim’</p>
<p>Thanks in advance?</p>

ptrblck: <p>It seems you’ve posted an older code, as it’s currently not executable, e.g. <code>self.eps</code> is defined outside of a method, you are using <code>type=affine</code> which is also undefined. Could you post the current code again?</p>

Jamesswiz: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi, Please see the modified code.</p>

Jamesswiz: <pre><code>class combine_act(nn.Module):
     def __init__(self,in_features,comb='convex'):
          super(combine_act, self).__init__()
    
    self.in_features = in_features
    self.comb = comb                     #type of combination [affine or convex]
    
    Lin = lambda x: x

    self.activations = {Lin, nn.ReLU(), nn.Tanh()}           #list of activations to use 

    self.alpha = nn.Parameter(torch.randn(len(self.activations)))
    self.alpha.requiresGrad = True
    
    


    def my_softmax(self, x):
      means = torch.mean(x)
      x_exp = torch.exp(x-means)
      x_exp_sum = torch.sum(x_exp)
      return x_exp/x_exp_sum

    def unitnorm(self,x):
      eps = 1e-7
      return x / x.sum().clamp(min=eps)

   def get_alpha(self,a):
      if self.comb=='affine':
         return self.unitnorm(a).cuda()
      else:
         return self.my_softmax(a).cuda()

   def forward(self, x):
      out=x-x 
      weight = self.get_alpha(self.alpha)
      for i, activation in enumerate(self.activations):
         out += weight[i]*activation(x)
         return out
</code></pre>
<hr>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> I have a working prototype now. I have to do some extensive tests. (see code above)</p>
<p>But there still remains an issue with the  update of weights.<br>
In forward function I am computing weights from trainable parameter alpha. However, What I want is to update alpha directly i.e., do gradient update on alpha via autograd and then rescale values according to convex/affine constraints so that I have the correct model parameter.</p>
<p>Any suggestions?</p>
<p>Thanks</p>

ptrblck: <p><code>self.act</code> is still undefined? Should I replace it with <code>self.activations</code>?<br>
Would the code to execute this module be <code>x = combine_act(100, actcomb='affine')(x)</code>?</p>

Jamesswiz: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> sorry for the bug. Yes your are right, and I have updated the code now too.</p>
<p>There is no need of self.act now. I will fix that with a named dictionary later. Right now I am hard coding the activations I want to use in self.activations.<br>
Here is how you would call it</p>
<pre><code>x = nn.linear(100,100)(x)
x = combine_act(100,comb='affine')(x)</code></pre>

ptrblck: <p>Thanks for the code.<br>
It seems <code>alpha</code> gets valid gradients (they look pretty high, but I’m not sure, if that’s desired in your use case).</p>
<pre><code class="lang-python">class combine_act(nn.Module):
    def __init__(self,in_features,comb='convex'):
        super(combine_act, self).__init__()
    
        self.in_features = in_features
        self.comb = comb                     #type of combination [affine or convex]
    
        Lin = lambda x: x

        self.activations = {Lin, nn.ReLU(), nn.Tanh()}           #list of activations to use 

        self.alpha = nn.Parameter(torch.randn(len(self.activations)))

    def my_softmax(self, x):
        means = torch.mean(x)
        x_exp = torch.exp(x-means)
        x_exp_sum = torch.sum(x_exp)
        return x_exp/x_exp_sum

    def unitnorm(self,x):
        eps = 1e-7
        return x / x.sum().clamp(min=eps)

    def get_alpha(self,a):
        if self.comb=='affine':
           return self.unitnorm(a).cuda()
        else:
           return self.my_softmax(a).cuda()

    def forward(self, x):
        out=x-x 
        weight = self.get_alpha(self.alpha)
        for i, activation in enumerate(self.activations):
           out += weight[i]*activation(x)
        return out


x = torch.randn(1, 100).cuda()    
model = combine_act(100, comb='affine').cuda()

x = model(x)
x.mean().backward()
print(model.alpha.grad)
</code></pre>
<p>You could simply pass <code>alpha</code> to an optimizer, which would perform the update step.<br>
Could you clarify the use case a bit, if that wouldn’t work?</p>

Jamesswiz: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> alpha is parameter of a custom activation function, and is different each time I use it in multiple layers like batchnorm. How do I pass it to optimizer in my training routine? There should be away to do it automatically like linear or batchnorm  layer where we do loss.backward followed by optimizer.step() and all parameters gets updated?</p>
<p>My next query is:<br>
I want to get rid of weight variable and use alpha directly. Since alpha is nn.Parameter I can’t assign it with output of get_alpha() function. masked_scatter_ gives me CUDA errors.</p>
<p>Thanks</p>