Solving the Vanishing Gradient Problem with Self-Normalizing Neural Networks using Keras
By: Jonathan Quijas
Tags: Neural Networks, AI, Keras, Machine Learning, Convergence

Background

In their 2010 landmark paper, Xavier Glorot and Yoshua Bengio provided invaluable insights concerning the difficulty of training deep neural networks.

It turns out the then-popular choice of activation function and weight initialization technique were directly contributing to what is known as the Vanishing/Exploding Gradient Problem. This article provides a deep dive on the subject of Vanishing/Exploding Gradients and Gradient Clipping.

In succinct terms, this is when the gradients start shrinking or increasing so much that they make training impossible.

Saturating Activation Functions

Before the wide adoption of the now ubiquitous ReLU function and its variants, sigmoid functions (S-shaped) were the most popular choice of the activation function. One such example of a sigmoid activation is the logistic function:

One major disadvantage of sigmoid functions is that they saturate. In the case of the logistic function, the outputs saturate to either 0 or 1 for negative and positive inputs respectively. This leads to smaller and smaller gradients (very close to 0) as the magnitude of the inputs increases.

Visualization of the logistic and ELU activation functions, along with their corresponding derivatives. Due to saturation, the derivatives of the logistic function will tend to shrink. In contrast, the derivative of the ELU function will be constant for positive inputs.

Since the ReLU and its variants do not saturate, they alleviate this vanishing gradient phenomenon. Improved variants of the ReLU such as the ELU function (shown above) have smooth derivatives all across:

For any positive input, the derivative will always be 1

For small negative numbers, the derivative will not be close to zero

Smooth derivative all across

NOTE: It is therefore beneficial to have inputs with an expected value of 0, as well as a small variance. This would help maintain strong gradients across the network.

Poor Choice of Weight Initialization

Another important insight found in the paper was the effect of weight initialization using a normal distribution with 0 mean and standard deviation of 1, a widely-popular choice prior to the authors’ findings.

The authors showed that the particular combination of a sigmoid activation and weight initialization (a normal distribution with 0 mean and standard deviation of 1) makes the outputs have a larger variance than the inputs. This effect is compounded across the network, making the input of deeper layers have a much larger magnitude relative to the input of more shallow (earlier) layers. This phenomenon was later named Internal Covariate Shift in a 2015 landmark paper by Sergey Ioffe and Christian Szegedy.

As we saw above, this translates into smaller and smaller gradients when using sigmoid activations.

The problem is further accentuated with the logistic function, since it has an expected value of 0.5, rather than 0. The hyperbolic tangent sigmoid function has an expected value of 0, and thus behaves better in practice (but also saturates).

The authors argued that in order for the gradients to be stable during training, the inputs and outputs of all layers must preserve more or less the same variance across the entire network. This would prevent the signal from dying or exploding when propagating in a forward pass, as well as gradients vanishing or exploding during backpropagation.

The distribution generated with the LeCun Normal initialization leads to much more probability mass centered at 0 and has a smaller variance. This is in contrast to the distribution generated with the normal initialization (0 mean and standard deviation of 1), where there is a much wider spread of values (larger variance).

To achieve this, they proposed a weight initialization technique, named Glorot (or Xavier) initialization after the paper’s first author. It turns out that with a slight modification of the Glorot technique, we get LeCun initialization, named after Yann LeCun.