Understanding ReLU: The Most Popular Activation Function in 5 Minutes! | by Bharath K | Towards Data Science

Activation functions in neural networks and deep learning play a significant role in the ignition of the hidden nodes to produce a more desirable output. The main purpose of the activation function is to introduce the property of non-linearity into the model.
In artificial neural networks, the activation function of a node defines the output of that node given an input or set of inputs. A standard integrated circuit can be seen as a digital network of activation functions that can be “ON” or “OFF,” depending on the input.
Sigmoid and tanh were monotonous, differentiable, and previously more popular activation functions. However, these functions suffer saturation over time, and this leads to problems occurring with vanishing gradients. An alternative and the most popular activation function to overcome this issue is the Rectified Linear Unit (ReLU).
The above diagram with the blue line is the representation of the Rectified Linear Unit (ReLU) whereas, the green line is a variant of ReLU called Softplus. The other variants of ReLU include Leaky ReLU, ELU, SiLU, etc., which are used for better performance in some tasks.
In this article, we will only consider the Rectified Linear Unit (ReLU) because it is still the most used activation function by default for performing a majority of the deep learning tasks. Its variants are usually used for specific purposes where they might have a slight edge over the ReLU.
This activation function was first introduced to a dynamical network by Hahnloser et al. in 2000 with strong biological motivations and mathematical justifications. It was demonstrated for the first time in 2011 to enable better training of deeper networks, compared to the widely used activation functions prior to 2011, e.g., the logistic sigmoid (which is inspired by probability theory and logistic regression) and its more practical counterpart, the hyperbolic tangent.
The rectifier is, as of 2017, the most popular activation function for deep neural networks. A unit employing the rectifier is also called a rectified linear unit (ReLU).
The main reason ReLU, despite being one of the best activation functions was not frequently used before recently. The reason for this was because it was not differentiable at the point 0. Researchers tended to use differentiable functions like sigmoid and tanh. However, it is now found that ReLU is the best activation function for deep learning.
The ReLU activation function is differentiable at all points except at 0. For values greater than 0, we just consider the max of the function. This can be written as:
f(x) = max{0, z}
In simple terms, this can also be written as follows:
if input > 0:    return inputelse:    return 0
All the negative values default to 0, and the maximum for the positive number is taken into consideration.
For the computation of the backpropagation of neural networks, the differentiation for the ReLU is relatively easy. The only assumption we will make is the derivative at the point 0, which will also be considered as 0. This is usually not such a big concern, and it works well for the most part. The derivative of the function is the value of the slope. The slope for negative values is 0.0, and the slope for positive values is 1.0.
The main advantages of the ReLU activation function are:
1. Convolutional layers and deep learning: They are the most popular activation functions for training convolutional layers and deep learning models.
2. Computational Simplicity: The rectifier function is trivial to implement, requiring only a max() function.
3. Representational Sparsity: An important benefit of the rectifier function is that it is capable of outputting a true zero value.
4. Linear Behavior: A neural network is easier to optimize when its behavior is linear or close to linear.
However, The main issue with the Rectified Linear Unit is that all the negative values become zero immediately, which decreases the ability of the model to fit or train from the data properly.
That means any negative input given to the ReLU activation function turns the value into zero immediately in the graph, which in turn affects the resulting graph by not mapping the negative values appropriately. This can however be easily fixed by using the different variants of the ReLU activation function like the Leaky ReLU and other functions discussed previously.
This was a short introduction to understanding the Rectified Linear Unit and its importance in deep learning technology as of today. It is way more popular than all the other activation functions, and for good reason as explained in this article.
I would highly recommend checking out the references to learn more about this concept. Feel free to check out some of my other articles from the links provided below. Thank you all for sticking on till the end, and have a wonderful day!