How to write my custom activation function?

jarvico: <p>Hi all,</p>
<p>I need to write a custom activation function which should support backward derivative operation.</p>
<p>The behavior of the activation function should vary based on the recieved parameters a and b.</p>
<p>Below, I share my sample code using NumPy to explain my requirement better.</p>
<pre><code class="lang-auto">x = torch.linspace(-35,30,4000)
x_numpy=x.data.numpy()

def activation_function(inp,a,b):
  '''boundaries -31,28'''

  #finding the equation of two points (-31,0),(a,2)
  points = [(-31,0),(a,2)]
  x_coords, y_coords = zip(*points)
  A = vstack([x_coords,ones(len(x_coords))]).T
  m_1, c_1 = lstsq(A, y_coords)[0]

  #finding the equation of two points [(b,2),(28,0)]
  points = [(b,2),(28,0)]
  x_coords, y_coords = zip(*points)
  A = vstack([x_coords,ones(len(x_coords))]).T
  m_2, c_2 = lstsq(A, y_coords)[0]

  temp = np.array([])
  for i in inp:
    if i&lt;-31:
      temp = np.concatenate( (temp, [0] ) )
    elif i&gt;=-31 and i&lt;a:
      #temp = np.concatenate( (temp, [i+31] ) ) #for a=-29
      temp = np.concatenate( (temp, [m_1*i+c_1] ) )
    elif i&gt;=a and i&lt;=b:
      temp = np.concatenate( (temp, [2] ) )
    elif i&gt;b and i&lt;28:
      #temp = np.concatenate( (temp, [-0.03571*i+1] ) ) #for b=-28
      temp = np.concatenate( (temp, [m_2*i+c_2] ) )
    else:
      temp = np.concatenate( (temp, [0] ) )
  return temp

x_numpy_new_1 = activation_function(x_numpy,-29,-28)

plt.figure(figsize=[24,4])
plt.plot(x_numpy,x_numpy_new_1,c='r',label="my_custom_activation_function")
plt.ylim([-0.5,2.5])
plt.xticks(np.arange(min(x_numpy), max(x_numpy)+1, 1.0))
plt.legend(loc='best')
</code></pre>
<p>the result should look like this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/4/24af8b038777d028015f7112362cfee7d0e0d6a9.png" data-download-href="https://discuss.pytorch.org/uploads/default/24af8b038777d028015f7112362cfee7d0e0d6a9" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24af8b038777d028015f7112362cfee7d0e0d6a9_2_689x125.png" alt="1" data-base62-sha1="5exhRyRiRFHEjyROVJaHsPiUBzr" width="689" height="125" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24af8b038777d028015f7112362cfee7d0e0d6a9_2_689x125.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24af8b038777d028015f7112362cfee7d0e0d6a9_2_1033x187.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/4/24af8b038777d028015f7112362cfee7d0e0d6a9.png 2x" data-dominant-color="FCFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1231×224 25.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As you can see, the function takes a value of 0 below -31 and above 28. -31 and 28 are fixed numbers no matter what. In this example parameter ‘a’ was -29 and parameter ‘b’ was -28. And the function takes a value of 2 between ‘a’ and ‘b’. And between -31,‘a’ and ‘b’,28, I need to calculate the equations first to determine the output values.</p>
<p>Here is the output for other iteration where the function outputs a value of 2 between parameters -23, -22</p>
<pre><code class="lang-auto">x_numpy_new_2 = activation_function(x_numpy,-23,-22)

plt.figure(figsize=[24,4])
plt.plot(x_numpy,x_numpy_new_2,c='r',label="my_custom_activation_function")
plt.ylim([-0.5,2.5])
plt.xticks(np.arange(min(x_numpy), max(x_numpy)+1, 1.0))
plt.legend(loc='best')
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54ab806db97a87baedea62d58b5ec816dd5828c5.png" data-download-href="https://discuss.pytorch.org/uploads/default/54ab806db97a87baedea62d58b5ec816dd5828c5" title="2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54ab806db97a87baedea62d58b5ec816dd5828c5_2_690x122.png" alt="2" data-base62-sha1="c51xZ7Vt8Qw1SmbrZp4mkumlB53" width="690" height="122" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54ab806db97a87baedea62d58b5ec816dd5828c5_2_690x122.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54ab806db97a87baedea62d58b5ec816dd5828c5_2_1035x183.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/4/54ab806db97a87baedea62d58b5ec816dd5828c5.png 2x" data-dominant-color="FCFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">1239×220 25.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Final example where the function outputs 2 between parameters -17, -16</p>
<pre><code class="lang-auto">x_numpy_new_3 = activation_function(x_numpy,-17,-16)

plt.figure(figsize=[24,4])
plt.plot(x_numpy,x_numpy_new_3,c='r',label="my_custom_activation_function")
plt.ylim([-0.5,2.5])
plt.xticks(np.arange(min(x_numpy), max(x_numpy)+1, 1.0))
plt.legend(loc='best')
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e0d99d28091f5ac3403a9a775f7b0818788bcbd9.png" data-download-href="https://discuss.pytorch.org/uploads/default/e0d99d28091f5ac3403a9a775f7b0818788bcbd9" title="3"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/0/e0d99d28091f5ac3403a9a775f7b0818788bcbd9_2_689x127.png" alt="3" data-base62-sha1="w57bIuCALskhFqCNchVBMsSNQlr" width="689" height="127" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/0/e0d99d28091f5ac3403a9a775f7b0818788bcbd9_2_689x127.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/0/e0d99d28091f5ac3403a9a775f7b0818788bcbd9_2_1033x190.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/0/e0d99d28091f5ac3403a9a775f7b0818788bcbd9.png 2x" data-dominant-color="FCFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">3</span><span class="informations">1231×227 25.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any support will be highly appreciated.</p>
<p>Best wishes…</p>

ZdsAlpha: <p>You will need to split the ranges. An example of Abs function is:</p>
<pre><code class="lang-auto">def abs(x): (x &gt;= 0).float() * x - (x &lt; 0).float() * x
</code></pre>
<p>Here is example:</p>
<pre><code class="lang-auto">x = torch.rand((10,)) * 2 - 1
print(x)
# tensor([ 0.8855, -0.6897,  0.6398, -0.4933,  0.0078,  0.2351, -0.1769,  0.9939, -0.7596, -0.3463])
y = abs(x)
print(y)
# tensor([0.8855, 0.6897, 0.6398, 0.4933, 0.0078, 0.2351, 0.1769, 0.9939, 0.7596, 0.3463])
</code></pre>
<p>This function by nature is differentiable. You can select different ranges using conditions and multiply it with the function you want it to be in that range and then sum all ranges. e.g for linear function in different ranges range is:<br>
<code>((x &gt;= a1) &amp; (x &lt; a2)).float() * (m1*x + b1) + ((x &gt;= a2) &amp; (x &lt; a3)).float() * (m2*x+b2) + ...</code><br>
This is not an optimal way to write a piecewise function. Also if you forget to include a range the slope there will be zero and model may not converge. Also the more pieces you added the more computationally expensive it becomes. An alternative solution is to write different forward and backward passes. In forward pass you only compute the function as it is. In backward pass you return the slope of function at those points which should be constant for linear pieces. (It should become as fact as relu)</p>

jarvico: <p>Thanks for your comment. I am having difficulties to understand your approach.</p>
<p>I actually need a working code example of the implementation of my question:(</p>
<p>For example, using below code I can implement an activation function like a filter. But,it actually doesn’t solve my question.</p>
<pre><code class="lang-auto">ht = nn.Hardtanh()

class MYFUNC7(nn.Module):

    def __init__(self):

        super().__init__() 

    def forward(self, input):

        return my_act_func_filter(input) 

def my_act_func_filter(input):

    return (ht(input) + ht(-input+3))

y = torch.linspace(-5,5,400)
func7 = MYFUNC7()
output = func7(y).data.numpy()

plt.figure(figsize=[24,4])
plt.plot(y,output,c='r',label="Filter")
plt.ylim([-2.5,2.5])
plt.xticks(np.arange(min(y), max(y)+1, 1.0))
plt.legend(loc='best')

</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50eb363325749e6a13a17a5fde15e3a619282114.png" data-download-href="https://discuss.pytorch.org/uploads/default/50eb363325749e6a13a17a5fde15e3a619282114" title="4"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50eb363325749e6a13a17a5fde15e3a619282114_2_689x122.png" alt="4" data-base62-sha1="bxQ8so64XRzsRHL5rMMTCbneM2U" width="689" height="122" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50eb363325749e6a13a17a5fde15e3a619282114_2_689x122.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50eb363325749e6a13a17a5fde15e3a619282114_2_1033x183.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/0/50eb363325749e6a13a17a5fde15e3a619282114.png 2x" data-dominant-color="FDFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">4</span><span class="informations">1237×220 10.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But, I need a shape like below as I shared in my previous post:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54ab806db97a87baedea62d58b5ec816dd5828c5.png" data-download-href="https://discuss.pytorch.org/uploads/default/54ab806db97a87baedea62d58b5ec816dd5828c5" title="2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54ab806db97a87baedea62d58b5ec816dd5828c5_2_690x122.png" alt="2" data-base62-sha1="c51xZ7Vt8Qw1SmbrZp4mkumlB53" width="690" height="122" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54ab806db97a87baedea62d58b5ec816dd5828c5_2_690x122.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54ab806db97a87baedea62d58b5ec816dd5828c5_2_1035x183.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/4/54ab806db97a87baedea62d58b5ec816dd5828c5.png 2x" data-dominant-color="FCFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">1239×220 25.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And it will be configurable by the two parameters ‘a’ and ‘b’.</p>

ZdsAlpha: <p>You first need to design function in something like graphing calculator and solve y = f(x) for individual piece of the function and convert it to code.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.desmos.com/calculator/imlxfm7arp">
  <header class="source">
      <img src="https://www.desmos.com/assets/img/apps/graphing/favicon.ico" class="site-icon" width="" height="">

      <a href="https://www.desmos.com/calculator/imlxfm7arp" target="_blank" rel="noopener nofollow ugc">Desmos</a>
  </header>

  <article class="onebox-body">
    <img width="500" height="500" src="https://www.desmos.com/calc_thumbs/production/imlxfm7arp.png" class="thumbnail onebox-avatar">

<h3><a href="https://www.desmos.com/calculator/imlxfm7arp" target="_blank" rel="noopener nofollow ugc">Desmos | Graphing Calculator</a></h3>

  <p>Explore math with our beautiful, free online graphing calculator. Graph functions, plot points, visualize algebraic equations, add sliders, animate graphs, and more.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/9058c84eb30bd0edb0d01c70ad9eeb9c2e392f6d.png" data-download-href="https://discuss.pytorch.org/uploads/default/9058c84eb30bd0edb0d01c70ad9eeb9c2e392f6d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/9058c84eb30bd0edb0d01c70ad9eeb9c2e392f6d_2_690x196.png" alt="image" data-base62-sha1="kAWYdyrEsrx3kCd3DyRgXblM7xz" width="690" height="196" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/9058c84eb30bd0edb0d01c70ad9eeb9c2e392f6d_2_690x196.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/9058c84eb30bd0edb0d01c70ad9eeb9c2e392f6d_2_1035x294.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/0/9058c84eb30bd0edb0d01c70ad9eeb9c2e392f6d_2_1380x392.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2445×698 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is sample code (ignore where function is zero):</p>
<pre><code class="lang-auto">import torch
import numpy as np
import matplotlib.pyplot as plt

def actfn(x):
    return ((x &gt;= -31) &amp; (x &lt; -23)).float() * (x + 31) / 4 + \
           ((x &gt;= -23) &amp; (x &lt; -22)).float() * 2  + \
           ((x &gt;= -22) &amp; (x &lt;= 28)).float() * ((-x - 22) / 25 + 2)


y = torch.linspace(-32, 32, 400)
output = actfn(y)
plt.plot(y,output,c='r',label="Filter")
plt.ylim([-2.5,2.5])
plt.xticks(np.arange(min(y), max(y)+1, 1.0))
plt.legend(loc='best')
plt.show()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af6064da8c1e826a121699bf2ece046d9b31b3f2.png" data-download-href="https://discuss.pytorch.org/uploads/default/af6064da8c1e826a121699bf2ece046d9b31b3f2" title="Figure_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af6064da8c1e826a121699bf2ece046d9b31b3f2_2_690x200.png" alt="Figure_1" data-base62-sha1="p1s4ZbTi2LmBOrapgytV0frQN6a" width="690" height="200" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af6064da8c1e826a121699bf2ece046d9b31b3f2_2_690x200.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af6064da8c1e826a121699bf2ece046d9b31b3f2_2_1035x300.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/f/af6064da8c1e826a121699bf2ece046d9b31b3f2.png 2x" data-dominant-color="FCFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Figure_1</span><span class="informations">1219×354 18 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>