The Rectified Linear Unit or ReLU is represented below with a diagram. It computes the function f(x)=max(0,x). In other words, the activation is simply thresholded at zero. The Pros and Cons of ReLU are as follows: Pros: It does not saturate (during its +ve region) Computationally it is very efficient Generally models with ReLU neurons converge much faster than neurons with other activation functions. Cons: An issue with dealing with them is where they die, i.e. dead Relus. Because if activation of any relu neurons become zero then its gradients will be clipped to zero in back-propagation. This can be avoided if we are very careful with weights initialization and tuning learning rate. For example, a large gradient flowing through a ReLU neuron can cause the weights to update in such a way that the neuron will never activate on any data point again. If this happens, then the gradient flowing through the unit will forever be zero from that point on. That is, the ReLU units can irreversibly die during training since they can get knocked off the data manifold. For example, we may find that as much as 40% of our network can be “dead” (i.e. neurons that never activate across the entire training dataset) if the learning rate is set too high. With a proper setting of the learning rate this is less frequently an issue.