What is activation function ?. One of most important parts of neural... | by Christophe Pere | Towards Data Science

The activation function defines the output of a neuron / node given an input or set of input (output of multiple neurons). It’s the mimic of the stimulation of a biological neuron.
The output of the activation function to the next layer (in shallow neural network: input layer and output layer, and in deep network to the next hidden layer) is called forward propagation (information propagation). It’s considered as a non linearity transformation of a neural network.
A notebook with all the code are available here: GitHub
Binary
Linear
Sigmoid
Tanh
ReLU
Leaky ReLU (LReLU)
Parametric ReLU (PReLU)
Exponential Linear Unit (eLU)
ReLU-6
Softplus
Softsign
Softmax
Swish
The binary activation function is the simpliest. It’s based on binary classifier, the output is 0 if values are negatives else 1. See this activation function as a threshold in binary classification.
The code for a binary activation function is:
def binary_active_function(x):    return 0 if x < 0 else 1
What is the output of this function ?
for i in [-5, -3, -1, 0, 2, 5]:    print(binary_active_function(i))output:    0    0    0    1    1    1
Or visualy:
Pro:
Binary classification
Cons:
Doesn’t work in multilabel classification
The derivative for the gradient calculation is always 0 so impossible to update weights
The next step after the binary function is to use a linear function instead of a step. The output is proportional to the input.
The corresponding code is:
def linear_active_function(a, x):    return a*x
We can compute it for different values of “a”:
$ x = numpy.linspace(-10, 10, 5000)$ y_1 = [linear_active_function(1, i) for i in x] # a = 1$ y_2 = [linear_active_function(2, i) for i in x] # a = 2$ y_1> [-10.0, -9.9, -9.8, -9.7, ..., 9.7, 9.8, 9.9, 10.0]
If we plot the results for a = 1, 2, 4 and 10:
Pros:
Binary and multiclass classification
Highly interpretable
Cons:
The derivative correspond to “a” so the update of weights and biaises during the backprogation will be constant.
Not efficient if the gradient is always the same.
Sigmoid is the most used activation function with ReLU and tanh. It’s a non-linear activation function also called logistic function. The output of this activation function vary between 0 and 1. All the output of neurons will be positive.
The corresponding code is as follow:
def sigmoid_active_function(x):    return 1./(1+numpy.exp(-x))
A quick computation:
$ x = numpy.linspace(-10, 10, 5000)$ y = [sigmoid_active_function(i) for i in x] $ y> [4.5397868702434395e-05, 4.5854103946941324e-05, ... , 0.9999532196250409, 0.9999536850759906, 0.9999541458960531]
If we plot the results:
Cons:
The problem of this function is that the output of each neuron can saturate. Values greater than 1 are shape as 1 and values smaller than 0 are shape as 0.
The best sensitivity for the sigmoid function is around the central point (0, 0.5).
A big problem appear during the saturation, the algorithm can not learn during this position (it’s the source of the vanishing gradient problem, corresponding to the absence of direction in the gradient).
The tangent hyperbolic function (tanh) is similar to the sigmoïd function in the way that their form are similar. Tanh is symmetric in 0 and the values are in the range -1 and 1. As the sigmoid they are very sensitive in the central point (0, 0) but they saturate for very large number (positive and negative). This symmetry make them better than the sigmoid function.
The corresponding code to apply a tanh function is:
def tanh_active_function(x):    return 2*sigmoid_active_function(2*x)-1
Compute the y values:
$ x = numpy.linspace(-10, 10, 5000)$ y = [tanh_active_function(i) for i in x] $ y> [-0.9999999958776927, -0.9999999957944167, ... , 0.9999999956227836, 0.9999999957094583, 0.9999999957944166]
And the corresponding result:
Pros:
Range between -1 and 1
The gradient is stronger than sigmoid ( derivatives are steeper)
Cons:
Like sigmoid, tanh also has a vanishing gradient problem
Saturation
The REctified Linear Unit was develop to avoid the saturation with big positive numbers. The non-linearity permit to conserve and learn the patterns inside the data and the linear part (>0 — also called piecewise linear function) make them easily interpretable.
The function below shows how to implement the ReLU function:
def relu_active_function(x):    return numpy.array([0, x]).max()
y computation:
$ x = numpy.linspace(-10, 10, 5000)$ y = [relu_active_function(i) for i in x] $ y> [0.0, 0.0, ... , 9.97, 9.98, 9.99]
The results:
Pros:
Easy to implement and very fast
True 0 value
Optimization are easy when activation function are linear
Most used in the neural networks ecosystem
Cons:
The function can not be differentiable when x = 0. The gradient descent can’t be computed for this point but, in practice that has not an influence. The linear part correspond to a slope with value 1 and the negative part is equal to zero.
“dying ReLU problem”: corresponds to the inactive part of the neurons if the output are 0. There no gradient when neurons are not active so if a large part of neurons are not activated it can result of poor performance of the model
Not appropriate for RNN class algorithm (RNN, LSTM, GRU)
This activation function is a modification of the ReLU activation function to avoid the “dying problem”. The function return a linear slope where a=0.01 which permit to keep neurons activated with a gradient flow.
See the code below:
def leaky_relu_active_function(x):    return 0.01*x if x < 0 else x
Compute the y axis to plot the results:
$ x = numpy.linspace(-10, 10, 5000)$ y = [leaky_relu_active_function(i) for i in x] $ y> [-0.1, -0.0999, ... , 9.97, 9.98, 9.99]
Plot the results:
Pros:
Correct the “dying ReLU problem”
Same comportement of the ReLU activation function for the part y=x
After the Leaky ReLU there is another activation function created to avoid the “dying ReLU problem”, the parametric or parametrised ReLU. The coefficient a is not lock at 0.01 (Leaky ReLU) but it free to estimate. It’s a generalization of the ReLU, the algorithm learn the rectifier parameter.
The code:
def parametric_relu_active_function(a, x):    return a*x if x < 0 else x
Compute the results for different a values:
$ x   = numpy.linspace(-10, 10, 5000)$ y_1 = [parametric_relu_active_function(0.25, i) for i in x] $ y_2 = [parametric_relu_active_function(0.5, i) for i in x]$ y_3 = [parametric_relu_active_function(0.75, i) for i in x]$ y_4 = [parametric_relu_active_function(1, i) for i in x]$ y_1> [-2.5, -2.4975, ... , 9.97, 9.98, 9.99]
Plot the results for a = 0.25, 0.5, 0.75, 1:
If a = 0 the parametric ReLU is equivalent to the ReLU activation function. If a=0.01 the parametric ReLU correspond to the Leaky ReLU.
Pros:
Generalize the ReLU activation function
Avoid the “dying ReLU problem”
The parameter “a” is learned by the neural network
eLU is another variation of the ReLU function. The negative part of the function is handled by the exponential function with a slow smooth.
The corresponding function:
def elu_active_function(a, x):    return a*(numpy.exp(x)-1) if x < 0 else x
y computation:
$ x   = numpy.linspace(-10, 10, 5000)$ y_1 = [elu_active_function(0.1, i) for i in x] $ y_2 = [elu_active_function(1, i) for i in x]$ y_3 = [elu_active_function(2, i) for i in x]$ y_4 = [elu_active_function(5, i) for i in x]$ y_1> [-0.09999546000702375, -0.09999541437933579, ... , 9.97, 9.98, 9.99]
Plot the results for a = 0.1, 1, 2, 4:
Pros:
ELU becomes smooth slowly until its output equal to -α whereas RELU sharply smoothes.
ELU is a strong alternative to ReLU.
Unlike to ReLU, ELU can produce negative outputs.
Cons:
For x > 0, it can blow up the activation with the output range of [0, inf].
Another variation of the ReLU function is the ReLU-6, 6 is an arbitrary parameter fixed by hand. The advantage is to shape the output for large positive number to the 6 value.
The corresponding code:
def relu_6_active_function(x):    return numpy.array([0, x]).max() if x<6 else 6
The y computation:
$ y = [relu_6_active_function(i) for i in x]
Plot the results:
The softplus activation function is an alternative of sigmoid and tanh functions. This functions have limits (upper, lower) but softplus is in the range (0, +inf).
The corresponding code:
def softplus_active_function(x):    return math.log(1+numpy.exp(x))
y computation:
$ y = [softplus_active_function(i) for i in x]
Plot the results:
This activation function is a variation of tanh but is not very used in practice. tanh and softsign functions are closely related, tanh converges exponentially whereas softsign converges polynomially.
The corresponding code:
def softsign_active_function(x):    return x / (1 + abs(x) )$ y = [softsign_active_function(i) for i in x]
Plot the results:
The softmax activation function is different from the other because it compute the probability distribution. The sum of the output is equal to 1.
The corresponding code:
def softmax_active_function(x):    return numpy.exp(x)/numpy.sum(numpy.exp(x))
Compute the output is different because it’s a probability distribution taking into account the sum of exponential, the function needed all the x points to compute the output y.
$ x = [0.8, 1.2, 2.4, 4.6]$ y = softmax_active_function(x)$ y> [0.01917691, 0.02860859, 0.09498386, 0.85723064]$ numpy.sum(y)> 1.0
Swish is the newer activation function, published by Google in 2017 it improves the performances of ReLU on deeper models. This function is a variation of sigmoid function because it can be expressed by: x*sigmoid(x).
Swish has the properties of one-sided boundedness at zero, smoothness, and non-monotonicity, which may play a role in the observed efficacy of Swish and similar activation functions.SWISH: A SELF-GATED ACTIVATION FUNCTION, Prajit Ramachandran∗ , Barret Zoph, Quoc V. Le, 2017
The corresponding codes:
def swish_active_function(x):    return x/(1+numpy.exp(-x))
Or:
def swish_active_function(x):    return x*sigmoid_active_function(x)
Compute the values:
$ x = numpy.linspace(-10, 10, 5000)$ y = [swish_active_function(i) for i in x]$ y> [-0.0004539786870243439, -0.0004967044303692657, ..., 9.699405586525717, 9.799456604457717, 9.89950329556963]
Plot the results:
Pros:
Differentiable on each point compare to ReLU
Here are presented some activation functions (most popular) with their code and representation. Hope this visualisation could permit to everyone to understand the output of neurons and determined which function is better for the problem they work on.
The last plot corresponds to the activation functions stack in one graphic.
https://medium.com/r/?url=https%3A%2F%2Fml-cheatsheet.readthedocs.io%2Fen%2Flatest%2Factivation_functions.html
https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/
Ian Goodfellow, Joshua Bengio and Aaron courville, Deep Learning (2016), MIT
Kevin Jarrett et al., What is the best multi-stage architecture for object recognition? (2009), IEEE
Vinod Nair and Geoffrey Hinton, Rectified Linear Units Improve Restricted Boltzmann Machines (2010), ACM
Xavier Glorot, Antoine Bordes, Yoshua Bengio, Deep Sparse Rectifier Neural Networks (2011), Proceedings of Machine Learning Research
Andrew L. Maas, Awni Y. Hannun and Andrew Y. Ng, Rectifier Nonlinearities Improve Neural Network Acoustic Models (2013), Stanford
Xavier Glorot, Yoshua Bengio, Understanding the difficulty of training deep feedforward neural networks (2010), Proceedings of Machine Learning Research
Kaiming He et al., Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015), arxiv.org
Activation function, Wikipedia
Vanishing gradient problem, Wikipedia
Rectifier (neural networks), Wikipedia
Piecewise Linear Function, Wikipedia
https://medium.com/@danqing/a-practical-guide-to-relu-b83ca804f1f7
Wenling Shang, Kihyuk Sohn, Diogo Almeida, Honglak Lee, Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units (2016), arxiv.org
Softmax function, Wikipedia
https://medium.com/@himanshuxd/activation-functions-sigmoid-relu-leaky-relu-and-softmax-basics-for-neural-networks-and-deep-8d9c70eed91e
Prajit Ramachandran, Barret Zoph, Quoc V. Le, Searching for activation functions (2017), arxiv.org