Why Rectified Linear Unit (ReLU) in Deep Learning and the best practice to use it with TensorFlow | by B. Chen | Towards Data Science

The activation functions are at the very core of Deep Learning. They determine the output of a model, its accuracy, and computational efficiency. In some cases, activation functions have a major effect on the model’s ability to converge and the convergence speed.
In this article, you’ll learn why ReLU is used in Deep Learning and the best practice to use it with Keras and TensorFlow 2.
Problems with Sigmoid and Tanh activation functionsWhat is Rectified Linear Unit (ReLU)Training a deep neural network using ReLUBest practice to use ReLU with He initializationComparing to models with Sigmoid and TanhLimitations of ReLU
Problems with Sigmoid and Tanh activation functions
What is Rectified Linear Unit (ReLU)
Training a deep neural network using ReLU
Best practice to use ReLU with He initialization
Comparing to models with Sigmoid and Tanh
Limitations of ReLU
Please check out Notebook for the source code. More tutorials can be found from the Github repo.
Historically, the two most widely used nonlinear activations are the Sigmoid and Hyperbolic Tangent (Tanh) activations functions.
The Sigmoid activation function (also known as the Logistic function), is traditionally a very popular activation function for neural networks. The input to the function is transformed into a value between 0 and 1. For a long time, through the early 1990s, it was the default activation used on neural networks.
The Hyperbolic Tangent, also known as Tanh, is a similar shaped nonlinear activation function that outputs value range from -1.0 and 1.0 (instead of 0 to 1 in the case of Sigmoid function). In the later 1990s and through the 2000s, the Tanh function was preferred over the Sigmoid activation function as models that used it were easier to train and often had a better predictive performance.
A general problem with both the Sigmoid and Tanh functions is vanishing gradients. Looking at the function plot, you can see that when inputs become small or large, the Sigmoid function saturates at 0 or 1 and the Tanh function saturates at -1 and 1, with a derivative extremely close to 0. Thus it has almost no gradient to propagate back through the network, so there is almost nothing left for lower layers [2]. The problem prevents network models from learning effectively, especially in deep networks,
Another problem with both the Sigmoid and Tanh functions is that they have an exponential operation, which can be computationally expensive. In our case, a model trained with ReLU has got a 25% improvement in terms of the convergence speed (Check out this result from the section “Comparing to models with Sigmoid and Tanh”).
The Rectified Linear Unit (ReLU) is the most commonly used activation function in deep learning. The function returns 0 if the input is negative, but for any positive input, it returns that value back. The function is defined as:
The plot of the function and its derivative:
As we can see that:
Graphically, the ReLU function is composed of two linear pieces to account for non-linearities. A function is non-linear if the slope isn’t constant. So, the ReLU function is non-linear around 0, but the slope is always either 0 (for negative inputs) or 1 (for positive inputs).
The ReLU function is continuous, but it is not differentiable because its derivative is 0 for any negative input.
The output of ReLU does not have a maximum value (It is not saturated) and this helps Gradient Descent
The function is very fast to compute (Compare to Sigmoid and Tanh)
It’s surprising that such a simple function works very well in deep neural networks.
For the demonstration purpose, we will build an image classifier to tackle Fashion MNIST, which is a dataset that has 70,000 grayscale images of 28-by-28 pixels with 10 classes.
Keras provides some utility functions to fetch and load common datasets, including Fashion MNIST. Let’s load Fashion MNIST
fashion_mnist = keras.datasets.fashion_mnist(X_train_full, y_train_full), (X_test, y_test) = fashion_mnist.load_data()
The dataset is already split into a training set and a test set. Here is the shape and data type of the training set:
>>> X_train_full.shape(60000, 28, 28)>>> X_train_full.dtypedtype('uint8')
We are going to train the neural network using Gradient Descent, we must scale the input feature down to the 0–1 range.
Now let’s build a deep neural network! There are 3 ways to create a machine learning model with Keras and TensorFlow 2.0. Since we are building a simple fully connected neural network and for simplicity, let’s use the easiest way: Sequential Model with Sequential(). Let’s create a deep neural network for Fashion MNIST with 50 hidden layers:
from tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import Dense, Flattendef create_model(activation):     model = keras.models.Sequential()    model.add(Flatten(input_shape=[28, 28]))    model.add(Dense(300, activation=activation))    for layer in range(49):        model.add(Dense(100, activation=activation))    model.add(Dense(10, activation="softmax"))            model.compile(        optimizer='sgd',         loss='sparse_categorical_crossentropy',         metrics=['accuracy']    )    return model
Our model has the following specifications:
The first layer (also known as the input layer) has the input_shape to set the input size (28, 28) which matches the training data. The input layer is a Flatten layer whose role is simply to convert each input image into a 1D array.
And then it is followed by 50Dense layers, one with 300 units, and the other 49 with 100 units.
The output Dense layer has 10 units and the softmax activation function.
Let’s create a model with ReLU:
model_relu = create_model(activation = 'relu')
To fit the model to training data:
history_relu = model_relu.fit(    X_train,     y_train,     epochs=10,    validation_data=(X_valid, y_valid),)
If all runs smoothly, we should get an output like below.
Train on 55000 samples, validate on 5000 samplesEpoch 1/1055000/55000 [==============================] - 41s 751us/sample - loss: 2.3027 - accuracy: 0.0991 - val_loss: 2.3028 - val_accuracy: 0.0914Epoch 2/1055000/55000 [==============================] - 42s 771us/sample - loss: 2.3027 - accuracy: 0.0979 - val_loss: 2.3029 - val_accuracy: 0.0914............Epoch 10/1055000/55000 [==============================] - 39s 715us/sample - loss: 2.3027 - accuracy: 0.0993 - val_loss: 2.3028 - val_accuracy: 0.0914
In this case, we can see that performance is quite poor on both the train and validation sets achieving around 10% accuracy. This suggests that the model as configured could not learn the problem nor generalize a solution.
The line plots of model accuracy on the train and validation sets during training tell a similar story.
Isn’t ReLU the default activation function in deep learning? What else is causing the problem?
One of the hyperparameters on training a deep neural network is the weight initializers. In general, there are 3 common initializers:
Random initialization
Glorot (Also known as Xavier) initialization
He initialization
If you google a little bit, you’ve likely also found out that one should use Glorot (Xavier) initialization if the activation function is a Tanh, and He initialization is recommended if the activation function is a ReLU. By default, Keras uses Glorot initialization and that was a problem with the previous model.
In addition, the learning rate is too high in the previous model because we have seen both the train and validation sets accuracy achieving around 10% during training. Let’s use a smaller learning rate.
To build layers with He initialization, we can simply set the argument kernel_initializer to 'he_normal'. To use a custom learning rate, we can set the argument optimizer to keras.optimizers.SGD(lr=1e-3).
def create_model(activation, k_init):     model = keras.models.Sequential()    model.add(Flatten(input_shape=[28, 28]))    model.add(Dense(300, activation=activation, kernel_initializer=k_init))    for layer in range(49):        model.add(Dense(100, activation=activation, kernel_initializer=k_init))    model.add(Dense(10, activation="softmax"))        model.compile(        optimizer=keras.optimizers.SGD(lr=1e-3),         loss='sparse_categorical_crossentropy',         metrics=['accuracy']    )        return modelmodel_with_he = create_model(    activation = 'relu',     k_init='he_normal')
Now, let’s fit the model to training data again:
history_relu_he = model_with_he.fit(    X_train,     y_train,     epochs=10,    validation_data=(X_valid, y_valid),)
This time you should get a much better output:
Train on 55000 samples, validate on 5000 samplesEpoch 1/1055000/55000 [==============================] - 38s 687us/sample - loss: 1.3974 - accuracy: 0.4666 - val_loss: 1.3329 - val_accuracy: 0.5898Epoch 2/1055000/55000 [==============================] - 38s 693us/sample - loss: 0.7728 - accuracy: 0.7079 - val_loss: 0.7561 - val_accuracy: 0.7154............Epoch 10/1055000/55000 [==============================] - 38s 687us/sample - loss: 0.4063 - accuracy: 0.8545 - val_loss: 0.4198 - val_accuracy: 0.8552
By plotting the model accuracy, we can see the model with He initialization shows a huge improvement to what we have seen before.
Now, let’s build 2 models, one with Sigmoid and the other with Tanh, fit them with the training data.
model_sigmoid = create_model(activation = 'sigmoid')model_tanh = create_model(activation = 'tanh')
After that, let’s compare the accuracy of all the models we have discussed:
In this case, the model trained with ReLU and He Initialization has the best performance, followed by the model trained with Tanh.
We can see that models with ReLU and Sigmoid are quite poor on both the train and validation sets achieving around 10% accuracy. This suggests that the two models as configured could not learn the problem nor generalize a solution.
In addition, we found the average execution time for the model trained with ReLU is about 33 seconds per epoch, comparing to the average 40 seconds per epoch for the model trained with Sigmoid or Tanh. It is a 25% improvement in terms of convergence speed. (Please check out log details from the Notebook)
ReLU works great in most applications, but it is not perfect. It suffers from a problem known as the dying ReLU.
During training, some neurons effectively die, meaning they stop outputting anything other than 0. In some cases, you may find that half of your network’s neurons are dead, especially if you used a large learning rate. A neuron dies when its weights get tweaked in such a way that the weighted sum of its inputs are negative for all instances in the training set. When this happens, it just keeps outputting 0s, and gradient descent does not affect it anymore since the gradient of the ReLU function is 0 when its input is negative.
Hands-on Machine Learning [1], page 329
To solve this problem, you may want to use a variant of the ReLU function, such as Leaky ReLU, Parametric Leaky ReLU (PReLU), Exponential Linear Unit (ELU), or Scaled Exponential Linear Unit (SELU). Please check out their introduction from the following article:
towardsdatascience.com
In this article, we have gone through the reason behind using the ReLU activation function in Deep Learning and how to use it with Keras and TensorFlow.
I hope this article will help you to save time in building and tuning your own Deep Learning model. I recommend you to check out the Keras documentation for the activation functions and to know about other things you can do.
Thanks for reading. Please check out the notebook for the source code and stay tuned if you are interested in the practical aspect of machine learning.
7 popular activation functions you should know in Deep Learning
Building custom callbacks with Keras and TensorFlow 2
A practical introduction to Keras Callbacks in TensorFlow 2
Learning Rate Schedules in practice
The Google’s 7 steps of Machine Learning in practice: a TensorFlow example for structured data
3 ways to create a Machine Learning Model with Keras and TensorFlow 2.0
Batch normalization in practice: an example with Keras and TensorFlow 2.0
Early stopping in Practice: an example with Keras and TensorFlow
More can be found from my Github
[1] Hands-on machine learning with Scikit-Learn and TensorFlow: concepts, tools, and techniques to build intelligent systems