Neural Network Optimization Algorithms | by Vadim Smolyakov | Towards Data Science

What are some of the popular optimization algorithms used for training neural networks? How do they compare?
This article attempts to answer these questions using a Convolutional Neural Network (CNN) as an example trained on MNIST dataset with TensorFlow.
SGD updates model parameters (theta) in the negative direction of the gradient (g) by taking a subset or a mini-batch of data of size (m):
The neural network is represented by f(x(i); theta) where x(i) are the training data and y(i) are the training labels, the gradient of the loss L is computed with respect to model parameters theta. The learning rate (eps_k) determines the size of the step that the algorithm takes along the gradient (in the negative direction in the case of minimization and in the positive direction in the case of maximization).
The learning rate is a function of iteration k and is a single most important hyper-parameter. A learning rate that is too high (e.g. > 0.1) can lead to parameter updates that miss the optimum value, a learning rate that is too low (e.g. < 1e-5) will result in unnecessarily long training time. A good strategy is to start with a learning rate of 1e-3 and use a learning rate schedule that reduces the learning rate as a function of iterations (e.g. a step scheduler that halves the learning rate every 4 epochs):
def step_decay(epoch):    lr_init = 0.001    drop = 0.5    epochs_drop = 4.0    lr_new = lr_init * \             math.pow(drop, math.floor((1+epoch)/epochs_drop))    return lr_new
In general, we want the learning rate (eps_k) to satisfy the Robbins-Monroe conditions:
The first condition ensures that the algorithm will be able to find a locally optimal solution regardless of the starting point and the second one controls oscillations.
Momentum accumulates exponentially decaying moving average of past gradients and continues to move in their direction:
Thus step size depends on how large and how aligned the sequence of gradients are, common values of momentum parameter alpha are 0.5 and 0.9.
Nesterov Momentum is inspired by Nesterov’s accelerated gradient method:
The difference between Nesterov and standard momentum is where the gradient is evaluated, with Nesterov’s momentum the gradient is evaluated after the current velocity is applied, thus Nesterov’s momentum adds a correction factor to the gradient.
AdaGrad is an adaptive method for setting the learning rate [3]. Consider two scenarios in the Figure below.
In the case of a slowly varying objective (left), the gradient would typically (at most points) have a small magnitude. As a result, we would need a large learning rate to quickly reach the optimum. In the case of a rapidly varying objective (right), the gradient would typically be very large. Using a large learning rate would result in very large steps, oscillating around but not reaching the optimum.
These two situations occur because the learning rate is set independent of the gradient. AdaGrad solves this by accumulating squared norms of gradients seen so far and dividing the learning rate by the square root of this sum:
As a result parameters that receive high gradients will have their effective learning rate reduced and parameters that receive small gradients will have their effective learning rate increased. The net effect is greater progress in the more gently sloped directions of parameter space and more cautious updates in the presence of large gradients.
RMSProp modifies AdaGrad by changing the gradient accumulation into an exponentially weighted moving average, i.e. it discards history from the distant past [4]:
Notice that AdaGrad implies a decreasing learning rate even if the gradients remain constant due to accumulation of gradients from the beginning of training. By introducing exponentially weighted moving average we are weighing recent past more heavily in comparison to distant past. As a result, RMSProp has been shown to be an effective and practical optimization algorithm for deep neural networks.
Adam derives from “adaptive moments”, it can be seen as a variant on the combination of RMSProp and momentum, the update looks like RMSProp except that a smooth version of the gradient is used instead of the raw stochastic gradient, the full Adam update also includes a bias correction mechanism [5]:
The recommended values are beta_1 = 0.9, beta_2 = 0.999, and eps = 1e-8.
A simple CNN architecture was trained on MNIST dataset using TensorFlow with 1e-3 learning rate and cross-entropy loss using four different optimizers: SGD, Nesterov Momentum, RMSProp and Adam. The Figure below shows the value of the training loss vs iterations:
We can see from the plot that Adam and Nesterov Momentum optimizers produce the lowest training loss!
All code is available in the following ipython notebook.
We compared different optimizers used in training neural networks and gained intuition for how they work. We found that SGD with Nesterov Momentum and Adam produce the best results when training a simple CNN on MNIST data in TensorFlow.
[1] Ian Goodfellow et. al., “Deep Learning”, MIT Press, 2016
[2] Andrej Karpathy, http://cs231n.github.io/neural-networks-3/
[3] Duchi, J. ,Hazan, E. and Singer, Y. “Adaptive subgradient methods for online learning and stochastic optimization”, JMLR, 2011.
[4] Tieleman, T. and Hinton, G. “Lecture 6.5 — RMSProp, COURSERA: Neural Networks for Machine Learning”, Technical Report, 2012.
[5] Diederik Kingma and Jimmy Ba, “Adam: A Method for Stochastic Optimization”, ICLR, 2015