Exploring Optimizers in Machine Learning
By: Nikita Sharma
Tags: Data Science, Gradient Descent, Heartbeat, Machine Learning, Optimization

Momentum Based Optimizers

These optimizers help models converge by focusing on movement in the correct direction (downhill) and reduces the variance in every other insignificant direction. Thus, adding another parameter ‘Y’ to our optimization algorithm to move steeply in the relevant direction helps us move faster towards convergence. The momentum parameter ‘Y’ typically has a value around 0.9. But, at the same time, one needs to hit and try multiple values and choose manually for the most optimal results.

Nesterov Accelerated Gradient (NAG)

If the value of the momentum is too high, our model can even cross the minima and again start going up. Thus, just like a ball, if it has too much energy, it would not stop on the flat surface after rolling down, but will continue to move up.

But, if our algorithm knows when to slow down, it will reach the minima. Thus, in NAG algorithms, the gradient for time step t also takes into account the estimated gradient for time step t+1, which gives a better idea whether to slow down or not. It works better than the conventional momentum algorithm.

In the above equations, (ϴ -ϒvt-1) is the next approximated values of the parameters and ∇J(ϴ -ϒvt-1) is the gradient with respect to the future values.

Advantages of NAG

It has an understanding of motion or future gradients. Thus, it can decrease momentum when the gradient value is lesser or the slope is low and can increase momentum when the slope is steep.

Disadvantages of NAG

NAG is not adaptive with respect to the parameter importance. Thus, all parameters are updated in a similar manner.

AdaGrad — Adaptive Gradient Algorithm

How cool would it be if our algorithm adapts different learning rates for different parameters? This is the intuition behind AdaGrad — some weights will have different learning rates based on how frequently or by how much their values are updated so as to perform optimally. Larger learning rate updates are made for less frequent parameters and smaller ones for the frequent parameters. The learning rate however, always decreases.

Here, as you can see, AdaGrad uses the sum of the square of past gradients to calculate the learning rate for each parameter (∇L(ϴ) is the gradient or partial derivative of cost function with respect to ϴ).

Advantages of AdaGrad

Works great for datasets which have missing samples or are sparse in nature.

Disadvantages of AdaGrad

The learning might be very slow, since, according to the formula above, division by bigger numbers (sum of past gradients become bigger and bigger with time) means that the learning rate is decreasing over time — therefore the pace of learning would also decrease.

RMS-Prop — Root Mean Square Propagation

RMS-Prop is almost similar to AdaGrad, with just a minute difference: It uses exponentially decaying average instead of the sum of gradients. Thus, RMS-Prop basically combines momentum with AdaGrad. Also, instead of using all the gradients for momentum, it takes into account only the most recent gradients for momentum calculation. This gives the model an understanding of whether it should increase or decrease its learning rate, given the current scenario.

Advantages of RMS-Prop

AdaGrad decreases the learning rate with each time step, but RMS-Prop can adapt to an increase or a decrease in the learning rate with each epoch.

Disadvantages of RMS-Prop

The learning can be pretty slow, same reason as in AdaGrad.

Adam — Adaptive Moment Estimation

Adam optimizer also uses adaptive learning rate technique to calculate current gradient from the first and second moments of the past gradients. Adam optimizer can be viewed as a combination of momentum and RMS-prop and is the most widely used optimizer for a wide range of problems.

Rather than influencing the learning rate with respect to first moments as in RMSProp, Adam also uses the average of the second moments of the gradients. Also, Adam calculates an exponential moving average of the gradient and the squared gradient. Thus, Adam can be considered as a combination of AdaGrad and RMS-Prop.

In the above formula, α is the learning rate, β1(usually ~0.9) is the exponential decay rate with respect to the first moments, β2 is the exponential decay rate with respect to the second moments (usually ~0.999). ∈ is just a small value to avoid division by zero.

Advantages of Adam

Adam optimizer is well suited for large datasets and is computationally efficient.

Disadvantages of Adam

There are few disadvantages as the Adam optimizer tends to converge faster, but other algorithms like the Stochastic gradient descent focus on the datapoints and generalize in a better manner. Thus, the performance depends on the type of data being provided and the speed/generalization trade-off.

Race to the Global Minima

Here is an animation of how different optimizers help the model reach the global minima for a particular data space:

Conclusion

In this post we discussed about various optimizers like gradient descent and its variations, Nesterov accelerated gradient, AdaGrad, RMS-Prop, and Adam along with their primary use cases. These are the most widely-used optimizer functions and are essential for developing efficient neural networks. Choosing an optimizer is influential to building efficient and fast neural networks.

I hope this article has helped you learn and understand more about these fundamental ML concepts. All feedback is welcome. Please help me improve!

Until next time!