Understanding PyTorch Activation Functions: The Maths and Algorithms (Part 1) | by Juan Nathaniel | Towards Data Science

You can find part 2 here.
In the world of ML, the activation functions help a network to learn complex patterns in the input data (or embeddings). Comparing to our brains, the activation functions are akin to the terminal side of the neurons determining what packet of information is to be propagated to the subsequent neuron given a set of complex inputs.
In this post, we are going to mathematically formalize and implement some of the more popular activation functions in PyTorch. It will be divided based on the kind of outputs you are looking at, namely:
(Slightly) positive: ReLU and Leaky ReLU
Between 0 and 1: Sigmoid, Softmax
Between -1 and 1: Tanh
If you are looking to modify the inputs and taking only their positive values, then either the Rectified Linear Unit (ReLU) or Leaky ReLU is appropriate
Given an input x, ReLU will take the maximal value between 0 and x. Graphically, ReLU has the following transformative behavior.
Implementing the ReLU function in python can be done as follows:
import numpy as nparr_before = np.array([-1, 1, 2])def relu(x):    x = np.maximum(0,x)    return xarr_after = relu(arr_before)arr_after#array([0, 1, 2])
And in PyTorch, you can easily call the ReLU activation function.
import torch.nnrelu = nn.ReLU()input = torch.randn(2)output = relu(input)
There is a slight difference betweek ReLU and Leaky ReLU. Given an input x, Leaky ReLU will take the maximal value between 0 and x if the value is positive, otherwise it will multiply x with the provided negative slope. Graphically, ReLU has the following transformative behavior.
Implementing the Leaky ReLU activation function in python can be done as follows:
import numpy as nparr_before = np.array([-1, 1, 2])# If x>0 returns x, else returns x*negative_slopedef leaky_relu(x, neg_slope=0.01):    x = np.where(x>0, x, x*neg_slope)    return xarr_after = leaky_relu(arr_before)arr_after#array([-0.01, 1., 2.])
And in PyTorch, you can easily call the Leaky ReLU activation function.
import torch.nnneg_slope=0.01leaky_relu = nn.LeakyReLU(neg_slope) #Pass in negative slope valueinput = torch.randn(2)output = leaky_relu(input)
If you are looking to output values between 0 and 1 (useful for probabilistic classification-type of problem), then either the Sigmoid or Softmax activation function would be appropriate depending on some caveats discussed below.
Graphically, Sigmoid has the following transformative behavior which restricts outputs to [0,1].
Implementing the Sigmoid function in python can be done as follows:
import numpy as nparr_before = np.array([-1, 1, 2])def sigmoid(x):    x = 1 / (1 + np.exp(-x))    return xarr_after = sigmoid(arr_before)arr_after#array([0.26894142, 0.73105858, 0.88079708])
And in PyTorch, you can easily call the Sigmoid activation function.
import torch.nnsigmoid = nn.Sigmoid()input = torch.randn(2)output = sigmoid(input)
Softmax is similar to sigmoid activation function in that the output of each element lies in the range between 0 and 1 (ie. [0,1]). The difference lies in softmax normalizing the exponent terms such that the sum of the component equals to 1. Thus, softmax is often used for multiclass classification problem where the total probability across known classes generally sums up to 1.
Implementing the Softmax function in python can be done as follows:
import numpy as nparr_before = np.array([-1, 1, 2])def softmax(x):    numerator = np.exp(x)    denominator = np.sum(np.exp(x))    x = numerator / denominator    return xarr_after = softmax(arr_before)arr_after#array([0.03511903, 0.25949646, 0.70538451]) #Sums up to 1
And in PyTorch, you can easily call the Softmax activation function.
import torch.nnsoftmax = nn.Softmax(dim=1)input = torch.randn(2, 3)output = softmax(input)
If you are looking to transform values to be between -1 and 1, then Tanh could be an appropriate activation function.
Graphically, Tanh has the following activation behavior which restricts outputs to be between [-1,1].
Implementing the Tanh function in python can be done as follows:
import numpy as nparr_before = np.array([-1, 1, 2])def tanh(x):    x = (np.exp(x) - np.exp(-x)) / (np.exp(x) + np.exp(-x))    return xarr_after = tanh(arr_before)arr_after#array([-0.76159416,  0.76159416,  0.96402758])
And in PyTorch, you can easily call the Tanh activation function.
import torch.nntanh = nn.Tanh()input = torch.randn(2)output = tanh(input)
This post introduces the mathematical definition, graphical representation, and hands-on implementation of some of the more common activation functions in PyTorch. The next sequence of posts will discuss some of the more advanced activation functions that might be useful for different use cases. Stay tuned!
Do subscribe to my Email newsletter: https://tinyurl.com/2npw2fnz where I regularly summarize AI research papers in plain English and beautiful visualization.