A Quick Guide to Gradient Descent and its Variants
By: Riccardo Di Sipio
Tags: Machine Learning, Adam, Gradient Descent, Adabound, Sgd

Gradient Descent and its Variants| Towards AI

A Quick Guide to Gradient Descent and its Variants

And if you do not know where to start from, optimize the learning rate

In machine learning, one deals with some mathematical model z = F(x,θ) which is a function of some input variables x and a set of parameters θ. The model can be for example an artificial and possibly deep neural network. The name of the game is to find the set of parameters that minimize the error in predicting some targets y (or labels in classification problems) starting from some defined input x, i.e. to reduce the quantity F(x,θ)-y as much as possible. Borrowing the jargon and some ideas from optimization theory, if the model F(x,θ) is a differentiable function, i.e. it has a derivative for each point of its domain, in principle all one has to do is to follow its gradient ∇F(x,θ) towards the minimum, hence the name Gradient Descent (GD). It is the equivalent of reaching the ground level of some energy function in physics or going downhill until we end up in a valley. The catch is, complicated models usually have many minima that are “false” minima: if we keep looking, eventually we might find the true or global minimum.

Gradient Descent is an algorithm to find the minimum of a function in N dimensions. However, some functions are complicated and show multiple minima.

Basic Gradient Descent Algorithms

In the most basic form, GD is applied to all the training examples and hence it is called Batch Gradient Descent. The key ingredient is the learning rate parameter λ, which represents the size of the step we take to update the parameters θ when we go from one point to another in the parameter space:

Batch gradient descent is guaranteed to converge to the global minimum for convex functions (think of a parabola) and to a local minimum for non-convex functions. But then, how do we jump into the global minimum? Also, if the size of the dataset is very large, this procedure can be very slow, or simply does not fit into memory.

The Stochastic Gradient Descent (SGD) comes to the rescue: the update of the parameters is performed for each training example. SGD is noisy, but what looks at first as a problem is actually the key to its success: it permits to “jump out” from a minimum and keep exploring the parameter space. Practical experience suggests that if we reduce the learning rate slowly, at some point SGD converges to a minimum more efficiently than the naive batch GD. But now we have another problem: how do we figure out the optimal value of λ, or how should we change this parameter as we descend towards the minimum?

Optimized Gradient Descent Algorithms

SGD is known not to perform well in cases where the gradient changes quickly in some regions, and especially if that happens only along some dimensions (hence the name ravines). A smart idea was borrowed from metallurgy, where a sequence of heating and cooling is used to model certain properties. In optimization theory this is known as momentum: it accelerates SGD in the relevant direction by damping oscillations. The momentum parameter γ (usually set to 0.9) is analogous to the mass of a Newtonian particle that moves through a viscous medium in a conservative force field.

All of this seems very smart, but what happens if we reach the bottom of a valley and keep accelerating? We’ll end up somewhere else, and maybe that’s not what we want. So we should better change the value of the momentum to take into account how far we are from a minimum. That’s obtained with the Nesterov accelerated gradient (NAG), which “looks ahead” before the direction of the descent is updated:

Adaptive Algorithms

The ultimate level of smartness is to be able to fine-tune the learning rate λ depending on the parameters θ. For example, we may want to have a smaller λ where the gradient does not change much and a larger λ where the gradient changes more frequently. The Adagrad algorithm updates assign a learning rate for each parameter. The learning rates are updated adaptively, i.e. scaled with respect to the accumulated squared gradient at each iteration in each dimension. Adadelta improves the algorithm by accumulating past gradients only up to a fixed history depth. RMSprop divides the learning rate by an exponentially decaying average of squared gradients:

Finally, Adam (Adaptive Moment Estimation) keeps track of averaged squared gradients (as in RMSprop) but also of gradients, i.e. of the first (mean) and second (variance) moment of the gradient distribution. Recently, a further improved version of Adam has been proposed called AdaBound, claimed to be able to train a model “as fast as Adam and as good as SGD.”. It employs dynamic upper and lower bounds on learning rates.

What to use?

The answer, it goes without saying, depends on the problem. Not surprising, practical experience hints at an advantage when using adaptive algorithms such as RMSprop and Adam. Interestingly, there were some claims that SGD would be preferable when training a Generative-Adversarial Network. If you do not know where to start from, use Adam and remember: if you do not have much time and you can optimize only one hyper-parameter, focus on the learning rate.