Optimization: Gradient Descent
By: Yudha Aji Saputra
Tags: Gradient Descent, Machine Learning, Optimization, Mathematics, Data Science

Optimization

Why is this topic widely discussed in science? Whether we realize it or not, humans naturally apply the principle of optimization in everyday life. As a simple example, we will find the route with the shortest distance when we want to go from one point to another. There is one story, on campus, the entrance gate is located in one corner of a rectangular garden. Along the side of the garden is the main road. However, over time, it turned out that many students chose the path through the garden, rather than through the main road that had been provided, in order to get to their destination faster. Finally, so that the damage to the garden does not get worse, the campus chooses to facilitate the natural character of its students by making this path a permanent route.

Student paths illustration

Nature also applies optimization and many of these optimization processes are adopted into the optimization algorithm we know today. The Genetic Algorithm is inspired by natural selection which tries to get better offspring by combining better parents. The simulated Annealing algorithm is inspired by the thermodynamic process of metal annealing. Meanwhile, Particle Swarm Optimization (PSO) adopts a bee swarm process in finding food sources.

Then what is optimization? In simple terms, optimization is a term that refers to the effort to maximize or minimize an objective function that depends on a parameter. In the case of finding the shortest route, the objective function will be how far the distance traveled and the parameter is the selected track. In general, optimization problems can be written mathematically:

The following is a general definition of optimization.

Optimization problem: Maximizing or minimizing some function relative to some set, often representing a range of choices available in a certain situation. The function allows a comparison of the different choices for determining which might be “best.” Common applications: Minimal cost, maximal profit, minimal error, optimal design, optimal management, variation principles. Source.

Photo by Daniel Frank on Unsplash

In the context of machine learning, optimization is used in various stages. Most popularly, optimization is used to minimize the cost function or loss function. In addition, optimization can also be used to select numerical methods that minimize the computation process but still provide good accuracy. In the case of minimization, the optimization algorithm aims to:

Finding the global minima of objective functions, sufficient conditions for this are convex objective functions. In other words, every local minima in the objective function is a global minima. Finds the local minima for the non-convex objective function.

Local minima are defined as the parameter that gives the smallest objective function value in a subset of parameters. On the other hand, the global minima are the parameter that gives the smallest objective function value compared to all possible parameters. Mathematically it can be written:

Gradient Descent

One of the most commonly used optimization algorithms in the machine or deep learning is gradient descent. Gradient descent is basically an iterative algorithm used to find local minima. Imagine that you are a climber lost on top of a mountain in a storm. In those circumstances, your goal is a village at the foot of a mountain or at least a lower place to shelter from the storm and avoid the risk of hypothermia. Unfortunately, the storm covered the entire surface of the mountain with a thick fog that limited your view. What will you do to reach the village? One way you can try is to observe the slope of the ground around you, then move to the lower place with the steepest slope. After a few steps, you stop again and return to the same observation, then move again to the lower place with the steepest slope. By repeating these steps you can expect to find a village at the foot of the mountain. But is it sure to work? Isn’t it possible that this process only took you down to the valley and not down to the village? Yes, you may just be stuck in a valley or in a niche, where your observations find no lower ground.

The gradient descent algorithm works like the climber’s lost analogy, by calculating the slope (gradient) of the objective function you can determine the next step to find the local minima point. In the analogy above, the surface of the mountain is an objective function and each coordinate point on the mountain is a parameter. The valley or niche where you might get stuck is called a saddle point.

Procedure

Gradient descent can be written in iterative form:

x is a parameter of the objective function. Gradient descent can not only be applied to single-variable functions, but also to functions with finite-dimensional multi variables.

Alpha is the learning rate, the coefficient that determines how far steps are taken in each iteration. The learning rate needs to be chosen wisely. A learning rate that is too large has the potential to cause an overshoot, which is a situation where steps are taken too far so that the iteration misses the local minima point. Conversely, a learning rate that is too small will slow down the convergence process to the local minima.

Notice that, since we need to calculate the first derivative of the objective function, F must be differentiable around the observed point.

The stopping condition used in iterative algorithms is usually the difference between the steps is very small or the maximum number of iterations is reached.

Example

To explain the gradient descent procedure, let’s take a simple case example. Let’s say we want to apply linear regression to the following dummy data for height and weight:

Suppose w = weight and h = height, assuming that the relationship between weight and height is linear, we can construct a simple linear model:

where a is the intercept and b is the slope. To measure the accuracy of the linear regression model, we use a simple criterion, least-square error:

So that we can get the following loss function:

To make it easier to understand, let’s first assume that the value of b is constant at 0.64. As a result, the loss function f becomes a function of one variable, depending only on a.

Apply the gradient descent procedure:

Suppose that the initial point taken is a0=0 and the learning rate is 1

2. Update value

By applying the chain rule we get:

So that we get an iterative formula:

So,

3. Repeat

The image below shows the graph formed after running six iterations, note that the steps taken between a0 and a1 are quite far, and when the gradient gets smaller (the closer to the local minima) the steps are smaller. This means that when the algorithm finds a large value gradient, it will tolerate taking a large step, and will take more rigorous steps as the gradient approaches zero.

After six iterations, for slope b = 0.64, the intercept value a = 0.95 is obtained, which gives a gradient of 0.004. By continuing the iteration, the gradient will decrease closer to zero, while the intercept will converge to the local minima point.

After understanding the procedure and a little help from multivariate calculus, we can apply the same procedure to minimize f for two parameters at once.

you can try it! the correct answer is intercept = 0.95 and slope = 0.64.

Gradient Descent Variation

The gradient descent algorithm we discussed above is called ‘batch gradient descent’. We also discussed earlier that the batch gradient descent might get stuck at the local minima and fail to reach the global minima. Apart from being prone to being trapped at the local minima point, batch gradient descent also demands a heavy computation process for each iteration. In the previous example we only used the two-parameter objective function and three sample data, with such objective function specifications we have to calculate the gradient 6 times in each iteration. Imagine if the objective function which must be minimized has tens of parameters and millions of sample data!.

Then, the next question is how can we avoid local minima and always try to find global minima? Or is there a way to avoid repeated calculations due to the large number of samples?

There are several versions of gradient descent, each of which has its own advantages and disadvantages. In the following, we will briefly discuss three of them.

1. Batch gradient descent

In batch gradient descent, we use the entire data set to compute the gradient of the objective function to determine the next step (update value).

Advantage:

Theoretically simple and easy to understand.

Has a very stable path.

Disadvantage:

Perform repeated calculations for the same learning in a large data set.

Since it uses up all the data sets, convergence is slow.

Demands intensive computation and more memory.

A very stable movement is prone to be trapped in local minima.

2. Stochastic gradient descent

This algorithm uses one random data point from the data set to calculate the gradient for each iteration. Because for each iteration we update the value as well as update the objective function, the movement of this algorithm step will appear more random. The random motion is expected to minimize the possibility of getting stuck in the local minima, but the main benefit of stochastic gradient descent over batch gradient descent is of course much less repetitive computation.

Advantage:

Learning is much faster than batch gradient descent.

Repetitive computation is much reduced, as it only uses one sample per iteration.

Step changes significantly with each iteration, reducing the chance of getting stuck in the local minima.

Disadvantage:

Because the step changes significantly with each iteration, the resulting path will be noisy and the cost function will be more volatile.

3. Mini batch gradient descent

Mini batch gradient descent is an extension of stochastic gradient descent, this algorithm uses a ‘mini-batch’ training sample, instead of just using one training sample. The batch size selected depends on the size of the data set. Increasing the number of samples included in the calculation may be a bit burdensome for the computation process, but it will give less noise to the path and help make the movement more stable and converge faster.

Advantage:

Reduces noise and therefore leads to stable convergence.

Very helpful in finding the global minimum value.

Disadvantage: