What is Gradient Clipping?. A simple yet effective way to tackle... | by Wanshun Wong | Towards Data Science

Recurrent Neural Networks (RNN) work very well with sequential data by utilizing hidden states that stores information about past inputs: the values of hidden states at time t depend on their values at time t -1 and also the inputs at time t. This architecture, while powerful, causes two problems in training: exploding gradients and vanishing gradients. In this article, we will look into gradient clipping which deals with the exploding gradients problem.
Exploding gradients refer to the problem that the gradients get too large in training, making the model unstable. Similarly, vanishing gradients refer to gradients getting too small in training. This prevents the network weights from changing their values. Both problems cause the model unable to learn from the training data. The following informal discussion is not that rigorous but is sufficient to give us an intuition about the source of exploding and vanishing gradients.
When we train a RNN by Backpropagation Through Time, it means we first unroll the RNN in time by creating a copy of the network for each time step, viewing it as a multi-layer feedforward neural network, where the number of layers is equal to the number of time steps. Then we do backpropagation on the unrolled network, taking into account the weight sharing:
where W is the recurrent weight matrix. It can be shown that the gradient of the loss function consists of a product of n copies of WT, where n is the number of layers going back in time. This product of matrices is the source of exploding and vanishing gradients.
For a scaler a ≠ 1, an shrink or grow exponentially. For example, consider n = 30. Then, for instance, 1.1n ≈ 17.45 and 0.9n ≈ 0.042. We pick 30 because it is quite common for a sentence in a Natural Language Processing task to have 30 words, and it is also quite typical for time series analysis to process 30 days of data. The situation for products of matrices (WT)n is very similar. The easiest way to see this is to assume WT is diagonalizable. Then WT = QDQ−1 for some diagonal matrix D = diag(λ1, ..., λx), and (WT)n = QDnQ−1 with Dn = diag(λ1n, ..., λxn).
We refer to [2] for a rigorous treatment of the exploding and vanishing gradients problems.
Gradient clipping is a technique that tackles exploding gradients. The idea of gradient clipping is very simple: If the gradient gets too large, we rescale it to keep it small. More precisely, if ‖g‖ ≥ c, then
g ↤ c · g/‖g‖
where c is a hyperparameter, g is the gradient, and ‖g‖ is the norm of g. Since g/‖g‖ is a unit vector, after rescaling the new g will have norm c. Note that if ‖g‖ < c, then we don’t need to do anything.
Gradient clipping ensures the gradient vector g has norm at most c. This helps gradient descent to have a reasonable behaviour even if the loss landscape of the model is irregular. The following figure shows an example with an extremely steep cliff in the loss landscape. Without clipping, the parameters take a huge descent step and leave the “good” region. With clipping, the descent step size is restricted and the parameters stay in the “good” region.
Tensorflow: tf.clip_by_global_norm.
PyTorch: torch.nn.utils.clip_grad_norm_.
Q: How do we choose the hyperparameter c?
A: We can train our neural networks for some epochs and look at the statistics of the gradient norms. The average value of gradient norms is a good initial trial.
Q: Can we use gradient clipping in training neural architectures other than RNN?
A: Yes. We can use gradient clipping for any neural architectures whenever we have exploding gradients.
Chapter 10.11 of [1] has a good overview of how gradient clipping works.[3] introduces a new smoothness condition to provide a theoretical explanation for the effectiveness of gradient clipping.
Chapter 10.11 of [1] has a good overview of how gradient clipping works.
[3] introduces a new smoothness condition to provide a theoretical explanation for the effectiveness of gradient clipping.
I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning (2016), MIT Press.R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training Recurrent Neural Networks (2013), ICML 2013.J. Zhang, T. He, S. Sra, and A. Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity (2020), ICLR 2020.
I. Goodfellow, Y. Bengio, and A. Courville. Deep Learning (2016), MIT Press.
R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training Recurrent Neural Networks (2013), ICML 2013.
J. Zhang, T. He, S. Sra, and A. Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity (2020), ICLR 2020.