7 popular activation functions you should know in Deep Learning and how to use them with Keras and TensorFlow 2 | by B. Chen | Towards Data Science

In artificial neural networks (ANNs), the activation function is a mathematical “gate” in between the input feeding the current neuron and its output going to the next layer [1].
The activation functions are at the very core of Deep Learning. They determine the output of a model, its accuracy, and computational efficiency. In some cases, activation functions have a major effect on the model’s ability to converge and the convergence speed.
In this article, you’ll learn the following most popular activation functions in Deep Learning and how to use them with Keras and TensorFlow 2.
Sigmoid (Logistic)Hyperbolic Tangent (Tanh)Rectified Linear Unit (ReLU)Leaky ReLUParametric Leaky ReLU (PReLU)Exponential Linear Units (ELU)Scaled Exponential Linear Unit (SELU)
Sigmoid (Logistic)
Hyperbolic Tangent (Tanh)
Rectified Linear Unit (ReLU)
Leaky ReLU
Parametric Leaky ReLU (PReLU)
Exponential Linear Units (ELU)
Scaled Exponential Linear Unit (SELU)
Please check out Notebook for the source code.
The Sigmoid function (also known as the Logistic function) is one of the most widely used activation function. The function is defined as:
The plot of the function and its derivative.
As we can see in the plot above,
The function is a common S-shaped curve.
The output of the function is centered at 0.5 with a range from 0 to 1.
The function is differentiable. That means we can find the slope of the sigmoid curve at any two points.
The function is monotonic but the function’s derivative is not.
The Sigmoid function was introduced to Artificial Neural Networks (ANN) in the 1990s to replace the Step function [2]. It was a key change to ANN architecture because the Step function doesn’t have any gradient to work with Gradient Descent, while the Sigmoid function has a well-defined nonzero derivative everywhere, allowing Gradient Descent to make some progress at every step during training.
The main problems with the Sigmoid function are:
Vanishing gradient: looking at the function plot, you can see that when inputs become small or large, the function saturates at 0 or 1, with a derivative extremely close to 0. Thus it has almost no gradient to propagate back through the network, so there is almost nothing left for lower layers [2].Computationally expensive: the function has an exponential operation.The output is not zero centered:
Vanishing gradient: looking at the function plot, you can see that when inputs become small or large, the function saturates at 0 or 1, with a derivative extremely close to 0. Thus it has almost no gradient to propagate back through the network, so there is almost nothing left for lower layers [2].
Computationally expensive: the function has an exponential operation.
The output is not zero centered:
To use the Sigmoid activation function with Keras and TensorFlow 2, we can simply pass 'sigmoid' to the argument activation :
from tensorflow.keras.layers import DenseDense(10, activation='sigmoid')
To apply the function for some constant inputs:
import tensorflow as tffrom tensorflow.keras.activations import sigmoidz = tf.constant([-20, -1, 0, 1.2], dtype=tf.float32)output = sigmoid(z)output.numpy()
Another very popular and widely used activation function is the Hyperbolic Tangent, also known as Tanh. It is defined as:
The plot of the function and its derivative:
We can see that the function is very similar to the Sigmoid function.
The function is a common S-shaped curve as well.
The difference is that the output of Tanh is zero centered with a range from -1 to 1 (instead of 0 to 1 in the case of the Sigmoid function)
The same as the Sigmoid, this function is differentiable
The same as the Sigmoid, the function is monotonic, but the function’s derivative is not.
Tanh has characteristics similar to Sigmoid that can work with Gradient Descent. One important point to mention is that Tanh tends to make each layer’s output more or less centered around 0 and this often helps speed up convergence [2].
Since Tanh has characteristics similar to Sigmoid, it also faces the following two problems:
Vanishing gradient: looking at the function plot, you can see that when inputs become small or large, the function saturates at -1 or 1, with a derivative extremely close to 0. Thus it has almost no gradient to propagate back through the network, so there is almost nothing left for lower layers.Computationally expensive: the function has an exponential operation.
Vanishing gradient: looking at the function plot, you can see that when inputs become small or large, the function saturates at -1 or 1, with a derivative extremely close to 0. Thus it has almost no gradient to propagate back through the network, so there is almost nothing left for lower layers.
Computationally expensive: the function has an exponential operation.
To use the Tanh, we can simply pass 'tanh' to the argument activation:
from tensorflow.keras.layers import DenseDense(10, activation='tanh')
To apply the function for some constant inputs:
import tensorflow as tffrom tensorflow.keras.activations import tanhz = tf.constant([-20, -1, 0, 1.2], dtype=tf.float32)output = tanh(z)output.numpy()
The Rectified Linear Unit (ReLU) is the most commonly used activation function in deep learning. The function returns 0 if the input is negative, but for any positive input, it returns that value back. The function is defined as:
The plot of the function and its derivative:
As we can see that:
Graphically, the ReLU function is composed of two linear pieces to account for non-linearities. A function is non-linear if the slope isn’t constant. So, the ReLU function is non-linear around 0, but the slope is always either 0 (for negative inputs) or 1 (for positive inputs).
The ReLU function is continuous, but it is not differentiable because its derivative is 0 for any negative input.
The output of ReLU does not have a maximum value (It is not saturated) and this helps Gradient Descent
The function is very fast to compute (Compare to Sigmoid and Tanh)
It’s surprising that such a simple function works very well in deep neural networks.
ReLU works great in most applications, but it is not perfect. It suffers from a problem known as the dying ReLU.
Dying ReLU
During training, some neurons effectively die, meaning they stop outputting anything other than 0. In some cases, you may find that half of your network’s neurons are dead, especially if you used a large learning rate. A neuron dies when its weights get tweaked in such a way that the weighted sum of its inputs are negative for all instances in the training set. When this happens, it just keeps outputting 0s, and gradient descent does not affect it anymore since the gradient of the ReLU function is 0 when its input is negative.
Hands-on Machine Learning [2], page 329
To use ReLU with Keras and TensorFlow 2, just set activation='relu'
from tensorflow.keras.layers import DenseDense(10, activation='relu')
To apply the function for some constant inputs:
import tensorflow as tffrom tensorflow.keras.activations import reluz = tf.constant([-20, -1, 0, 1.2], dtype=tf.float32)output = relu(z)output.numpy()
Leaky ReLU is an improvement over the ReLU activation function. It has all properties of ReLU, plus it will never have dying ReLU problem. Leaky ReLU is defined as:
f(x) = max(αx, x)
The hyperparameter α defines how much the function leaks. It is the slope of the function for x < 0 and is typically set to 0.01. The small slope ensures that Leaky ReLU never dies.
To use the Leaky ReLU activation function, you must create a LeakyReLU instance like below:
from tensorflow.keras.layers import LeakyReLU, Denseleaky_relu = LeakyReLU(alpha=0.01)Dense(10, activation=leaky_relu)
Parametric leaky ReLU (PReLU) is a variation of Leaky ReLU, where α is authorized to be learned during training (instead of being a hyperparameter, it becomes a parameter that can be modified by backpropagation like any other parameters). This was reported to strongly outperform ReLU on large image datasets, but on smaller datasets it runs the risk of overfitting the training set [2].
To use Parametric leaky ReLU, you must create a PReLU instance like below:
from tensorflow.keras.layers import PReLU, Densepara_relu = PReLU()Dense(10, activation=para_relu)
Exponential Linear Unit (ELU) is a variation of ReLU with a better output for z < 0. The function is defined as:
The hyperparameter α controls the value to which an ELU saturates for negative net inputs.
The plot of the function and its derivative:
We can see in the plot above,
ELU modified the slope of the negative part of the function.
Unlike the Leaky ReLU and PReLU functions, instead of a straight line, ELU uses a log curve for the negative values.
According to the authors, ELU outperformed all the ReLU variants in their experiments [3].
According to [2, 3], the main drawback of the ELU activation is that it is slower to compute than the ReLU and its variants (due to the use of the exponential function), but during training this is compensated by the faster convergence rate. However, at test time, an ELU network will be slower than a ReLU network.
Implementing ELU in TensorFlow 2 is trivial, just specify the activation function when building each layer:
Dense(10, activation='elu')
To apply the function for some constant inputs:
import tensorflow as tffrom tensorflow.keras.activations import eluz = tf.constant([-20, -1, 0, 1.2], dtype=tf.float32)output = elu(z, alpha=1)output.numpy()
Exponential Linear Unit (SELU) activation function is another variation of ReLU proposed by Günter Klambauer et al. [4] in 2017. The authors showed that if you build a neural network composed exclusively of a stack of dense layers, and if all hidden layers use the SELU activation function, then the network will self-normalize (the output of each layer will tend to preserve mean 0 and standard deviation 1 during training, which resolves the vanishing/exploding gradients problem). This activation function often outperforms other activation functions very significantly.
SELU is defined as:
f(x) = scale * x                 , z > 0     = scale * α * (exp(x) - 1)   , z <= 0
where α and scale are pre-defined constants (α=1.67326324 and scale=1.05070098).
The plot of SELU and its derivative:
The main problem with SELU is that there are a few conditions for SELU to work:
SELU works only for a neural network composed exclusively of a stack of dense layers. It might not work for convolutional neural networks.
Every hidden layer’s weights must also be initialized using LeCun normal initialization.
Input features must be standardized with mean 0 and standard deviation.
To use SELU with Keras and TensorFlow 2, just set activation='selu' and kernel_initializer='lecun_normal':
from tensorflow.keras.layers import DenseDense(10, activation='relu', kernel_initializer='lecun_normal')
We have gone through 7 different activation functions in deep learning. When building a model, the selection of activation functions is critical. So which activation function should you use? Here is a general suggestion from the book Hands-on ML
Although your mileage will vary, in general SELU > ELU > leaky ReLU (and its variants) > ReLU > tanh > logistic. If the network’s architecture prevents it from self-normalizing, then ELU may perform better than SELU (since SELU is not smooth at z = 0). If you care a lot about runtime latency, then you may prefer leaky ReLU. If you don’t want to tweak yet another hyperparameter, you may just use the default α values used by Keras (e.g., 0.3 for the leaky ReLU). If you have spare time and computing power, you can use cross-validation to evaluate other activation functions, in particular, RReLU if your network is over‐fitting, or PReLU if you have a huge training set.
Hands-on ML, page 332
In this article, we have gone through 7 different activation functions in Deep Learning and how to use them with Keras and TensorFlow.
I hope this article will help you to save time in building and tuning your own Deep Learning model. I recommend you to check out the Keras documentation for the activation functions and to know about other things you can do.
Thanks for reading. Please check out the notebook for the source code and stay tuned if you are interested in the practical aspect of machine learning.
Building custom callbacks with Keras and TensorFlow 2
A practical introduction to Keras Callbacks in TensorFlow 2
Learning Rate Schedules in practice
The Google’s 7 steps of Machine Learning in practice: a TensorFlow example for structured data
3 ways to create a Machine Learning Model with Keras and TensorFlow 2.0
Batch normalization in practice: an example with Keras and TensorFlow 2.0
Early stopping in Practice: an example with Keras and TensorFlow
More can be found from my Github
[1] 7 Types of Neural Network activation function
[2] Hands-on Machine Learning
[3] Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
[4] Self-Normalizing Neural Networks