Demystifying Optimizations for machine learning | by Ravindra Parmar | Towards Data Science

Optimization is the most essential ingredient in the recipe of machine learning algorithms. It starts with defining some kind of loss function/cost function and ends with minimizing the it using one or the other optimization routine. The choice of optimization algorithm can make a difference between getting a good accuracy in hours or days. The applications of optimization are limitless and is widely researched topic in industry as well as academia. In this article we’ll walk through several optimization algorithms used in the realm of deep learning. (You can go through this article to understand the basics of loss functions)
Stochastic gradient descent (SGD) is the simplest optimization algorithm used to find parameters which minimizes the given cost function. Apparently, for gradient descent to converge to optimal minimum, cost function should be convex. For the demonstration purpose, imagine following graphical representation for the cost function.
We start with defining some random initial values for parameters. The goal for optimization algorithm is to find parameter values which correspond to minimum value of cost function. Specifically, gradient descent starts with calculating gradients (derivatives) for each of the parameter w.r.t cost function. Those gradients gives us numerical adjustment we need to make to each parameter so as to minimize the cost function. This process continues until we hit the local/global minimum (cost function is minimum w.r.t it’s surrounding values). Mathematically,
for i in range(iterations_count):              param_gradients = evaluate_gradients(loss_function,                                          data,                                          params)    params -= learning_rate * param_gradients
Learning rate defines how much parameters should change in each iteration. In other words it controls how fast or slow we should converge to minimum. On one hand, small learning rate can take iterations to converge a large learning rate can overshoot minimum as you can see in the figure above.
Although easy enough to apply in practice, it has quite a few disadvantages when it comes to deep neural networks as these networks have large number of parameters to fit in. To illustrate issues with gradient descent let’s assume we have a cost function with two parameters only. Assume the cost function is very sensitive to changes in one of the parameter for example in vertical direction and less to other parameter i.e horizontal direction (This means cost function has high condition number).
If we run stochastic gradient descent on this function, we get a kind of zigzag behavior. In essence, SGD is making slow progress towards less sensitive direction and more towards high sensitive one and hence does not align in the direction of minimum. In practice, deep neural network could have millions of parameters and hence millions of directions to accommodate for gradient adjustments and hence compounding the problem.
Another issue with SGD is problem of local minimum or saddle points. Saddle points are points where gradient is zero in all directions. Consequently, our SGD will be stuck there only. On the other hand, local minimums are point which are minimum w.r.t surrounding however not minimum over all. As gradient will be zero at local minimum our gradient descent would report it as minimum value when global minimum is somewhere else.
To rectify the issues with vanilla gradient descent several advanced optimization algorithms were developed in recent years. We will look through them one by one.
In order to understand the dynamics behind advanced optimizations we first have to grasp the concept of exponentially weighted average. Let’s assume we are given data for temperatures per day of any particular city for all 365 days of a year. Plotting it, we get a graph at top left corner.
Now, if we wish to calculate the local average temperature across the year we would proceed as follows.
NOTE :-     alpha = 0.9 is randomly chosen weight.             t(i)  is temperature at ith day.     v(i)  is average temperature for ith day averaged over 1/(1 - alpha) days.v(0) = 0v(1) = 0.9 * v(0) + 0.1 * t(1)v(2) = 0.9 * v(1) + 0.1 * t(2)...v(i) = alpha * v(i-1) + (1 - alpha) * t(i)
At each day, we are calculating weighted average of previous day temperatures and current day temperature. Plot for above computation is shown at top right corner. This plot is averaging temperature over last 10 days (alpha = 0.9). Left bottom (green line) is showing the plot averaging data over last 50 days (alpha = 0.98).
An important point to notice here is as we are averaging over more number of days the plot will become less sensitive to changes in temperature. In contrast, if we average over less number of days the plot will be more sensitive to changes in temperature and hence wriggly behavior.
This increase in latency is due to the fact that we are giving more weight-age to previous day temperatures than current day temperature.
So far so good, but the question is what all this buys us. Quite similarly, by averaging gradients over past few values, we tend to reduce the oscillations in more sensitive direction and hence make it converge faster.
In practice, momentum based optimization algorithms are almost always faster then vanilla gradient descent. Mathematically,
moment = 0for i in range(iterations_count):              param_gradients = evaluate_gradients(loss_function,                                          data,                                          params)    moment = gamma * moment + param_gradients    params += learning_rate * moment           (where moment is building moving average of gradients.                 gamma gives kind of friction = 0.9 or 0.99).
The idea is, for each parameter, we store the sum of squares of all its historical gradients. This sum is later used to scale the learning rate.
Notice that, in contrast to previous optimizations, here we have different learning rate for each of the parameter.
squared_gradients = 0for i in range(iterations_count):              param_gradients = evaluate_gradients(loss_function,                                          data,                                          params)    squared_gradients += param_gradients * param_gradients    params -= learning_rate * param_gradients/                         (np.sqrt(squared_gradients) + 1e-8)                           {1e-8 is to avoid divide by zero}
Now the question is how this scaling is helping us when we have very high condition number for our loss function?
For parameters with high gradient values, the squared term will be large and hence dividing with large term would make gradient accelerate slowly in that direction. Similarly, parameters with low gradients will produce smaller squared terms and hence gradient will accelerate faster in that direction.
However notice that, as gradient is squared at every step, the moving estimate will grow monotonically over the course of time and hence the step size our algorithm will take to converge to minimum would get smaller and smaller.
And in a sense this is beneficial for convex problems as we are expected to slow down towards minimum in this case. However, the same gift becomes a curse in case of non-convex optimization problems as chance of getting stuck in saddle points increases.
This is a slight variation of AdaGrad and works better in practice as it addresses the issues left open by it. Similar to AdaGrad, here as well we will keep the estimate of squared gradient but instead of letting that squared estimate accumulate over training we rather let that estimate decay gradually. To accomplish this, we multiply the current estimate of squared gradients with the decay rate.
squared_gradients = 0for i in range(iterations_count):              param_gradients = evaluate_gradients(loss_function,                                          data,                                          params)    squared_gradients = decay_rate * squared_gradients + (1 -                    decay_rate) * param_gradients * param_gradients    params -= learning_rate * param_gradients/                              (np.sqrt(squared_gradients) + 1e-8)
This incorporates all the nice features of RMSProp and Gradient descent with momentum.
Specifically, this algorithm calculates an exponential moving average of gradients and the squared gradients whereas parameters beta_1 and beta_2 controls the decay rates of these moving averages.
first_moment = 0second_moment = 0for step in range(iterations_count):    param_gradients = evaluate_gradients(loss_function,                                          data,                                          params)    first_moment = beta_1 * first_moment + (1 - beta_1) *                                                       param_gradients    second_moment = beta_2 * second_moment + (1 - beta_2) *                                   param_gradients * param_gradients          params -= learning_rate * first_moment/(np.sqrt(second_moment) +                                                               1e-8)
Notice that we’ve initialized second_moment to zero. So, in the beginning, second_moment would be calculated as somewhere very close to zero. Consequently, we are updating parameters by dividing with a very small number and hence making large updates to parameter. That means initially, the algorithm would make larger steps. To rectify that we create an unbiased estimate of those first and second moment by incorporating current step. And then we make update to parameters based on these unbiased estimates rather than first and second moments. Mathematically,
first_moment = 0second_moment = 0for step in range(iterations_count):    param_gradients = evaluate_gradients(loss_function,                                          data,                                          params)    first_moment = beta_1 * first_moment + (1 - beta_1) *                                                       param_gradients    second_moment = beta_2 * second_moment + (1 - beta_2) *                                   param_gradients * param_gradients          first_bias_correction = first_moment/(1 - beta_1 ** step)    second_bias_correction = second_moment/(1 - beta_2 ** step)    params -= learning_rate * first_bias_correction/                           (np.sqrt(second_bias_correction) + 1e-8)
Figure below demonstrates the performance of each of the optimization algorithm as iterations pass by. Clearly adding momentum provides boost to accuracy. In practice, however, Adam is known to perform very well with large data sets and complex features.
www.coursera.org
You can find this for more mathematical background. Please let me know through your comments any modifications/improvements this article could accommodate.