A unit employing the rectifier is called Rectifier Linear Unit or ReLU. Range is 0 to infinity. In case of Leaky ReLU, f(y) is ay and not zero. The leak helps to increase the range of the ReLU function. Usually the value of a is 0.01 or equivalent to that. When a is not 0.01, then it is called Randomized ReLU. Hence the range of Leaky ReLU is -infinity to infinity. Leaky ReLUs allow a small, positive gradient when the unit is not active.