Minimizing the cost function: Gradient descent
By: Xuankhanh Nguyen
Tags: Algorithms, Deep Learning, Editors Pick, Data Science, Machine Learning

This is a very long note. Grab a cup of coffee or tea and let’s get started.

A quick recap from my last note:

A review from last note

So we have our hypothesis function and we have a way of measuring how well it fits the data. We now need to estimate the parameters theta zero and theta one in the hypothesis function.

So here’s the problem setup. Assume that we have a function J, as theta zero, theta one. We want to minimize over theta zero and theta one of this function J(theta zero, theta one). And it turns out gradient descent is an algorithm for solving this general problem. We’re going to start with some initial guesses for theta zero and theta one. It doesn’t matter what they are, but a common choice would be we set theta zero to zero and set theta one to one. What we’re going to do in gradient descent is we’ll keep changing theta zero and theta one a little bit to try to reduce J(theta zero, theta one), until we wind at a minimum, or maybe at a local minimum.

Gradient descent is an efficient optimization algorithm that attempts to find a local or global minimum of the cost function.

Global minimum vs local minimum

A local minimum is a point where our function is lower than all neighboring points. It is not possible to decrease the value of the cost function by making infinitesimal steps. A global minimum is a point that obtains the absolute lowest value of our function, but global minima are difficult to compute in practice.

Cost Function vs Gradient descent

We might argue that if the cost function and gradient descent are both used to minimize something then what is the difference and can we use one instead of the other?

Well, a cost function is something we want to minimize. For example, our cost function might be the sum of squared errors over the training set. Gradient descent is a method for finding the minimum of a function of multiple variables.

So we can use gradient descent as a tool to minimize our cost function.

Suppose we have a function with n variables, then the gradient is the length-n vector that defines the direction in which the cost is increasing most rapidly. So in gradient descent, we follow the negative of the gradient to the point where the cost is a minimum. In machine learning, the cost function is a function to which we are applying the gradient descent algorithm.

I assume that the readers are already familiar with calculus but will provide a brief overview of how calculus concepts relate to optimization here. So don’t worry friends, just stay with me… it’s kind of intuitive!

Machine learning uses derivatives in optimization problems. Derivatives are used to decide whether to increase or decrease the weights to increase or decrease an objective function. If we can compute the derivative of a function, we know in which direction to proceed to minimize it.

Some notation

A quick review for derivative

Suppose we have a function y = f(x) . The derivative f’(x) gives the slope of f(x) at point x. It speciﬁes how to scale a small change in the input to obtain the corresponding change in the output. Let’s say, f(x) = 1/2 x²

We can reduce f(x) by moving in small steps with the opposite sign of the derivative. When f’(x) = 0,the derivative provides no information about which direction to move. Points where f’(x) = 0 are known as critical points.

The concept of convergence is a well defined mathematical term. It means that “eventually” a sequence of elements gets closer and closer to a single value. So what does it mean for an algorithm to converge? Technically what converges is not the algorithm, but a value the algorithm is manipulating or iterating. To illustrate this, let's say we are writing an algorithm that prints all the digits of pi.

Our algorithm starts printing numbers like:

x0 = 3.1

x1 = 3.14

x2 = 3.141

x3 = 3.1415

x4 = 3.14159

...

As we can see, the algorithm prints increasing numbers close to pi. We say our algorithm converges to pi. And we call such functions convex functions (like a bowl shape). Now, let us consider the formula of gradient descent:

Gradient descent formula

We implement this formula by taking the derivative (the tangential line to a function) of our cost function. The slope of the tangent line is the value of the derivative at that point and it will give us a direction to move towards. We make steps down the cost function in the direction with the steepest descent. The size of each step is determined by the parameter α (alpha), which is called the learning rate.

The learning rate determines the size of the steps that are taken by the gradient descent algorithm.

To reach a local minimum efficiently, we have to set our learning rate- parameter α appropriately, neither too high nor too low. Depending on where the initial point starts on the graph, it could end up at different points. Typically, the value of the learning rate is chosen manually, starting with 0.1, 0.01, or 0.001 as the common values.

In this case, gradient descent is taking too long to calculate; we need to increase the learning rate.

If our learning curve is just going up and down without reaching a lower point, we should try decreasing the learning rate.

Note:

If the learning rate is too big, the loss will bounce around and may not reach the local minimum.

If the learning rate is too small then gradient descent will eventually reach the local minimum but require a long time to do so.

The cost function should decrease over time if gradient descent is working properly.

How can we visualize this idea?

Let’s say we are at Mount Lyell (the highest point in Yosemite National Park), we hike down the hill following the path of the river. The job of gradient descent here is exactly what we aim to achieve— to reach the bottom-most point of the mountain. Mount Lyell is the data plotted in space, the surface representing the objective function, and the size of the step we move is the learning rate. The lowest point on the mountain is the value where the cost of the function reaches its minimum (the parameter α where our model presents more accuracy).

Assume also that Mount Lyell is shaped in such a way that the river will not stop at any place and will straightaway arrive at the foothill (like a bowl shape). In machine learning, we would have achieved our global minimum. However, straightforward optimization is not the case in real-life. The river may face a lot of pits on the way down. It might be trapped in the pits and fail to move downwards, which is a local minimum in machine learning.

When we are in a valley, there is no way we can descend the hill further. We can say we have converged. In machine learning, when gradient descent can’t reduce the cost function anymore and the cost remains near the same level, we can say it has converged to an optimum. The number of iterations for convergence may vary a lot.

The takeaway here is the initial values and learning rate. Depending on where we start at the first point, we could wind up at different local optima. Also, depending on the size of the step we take (learning rate) we might arrive at the foothill differently. These values are important in determining whether we will reach the foothill (global minima) or get trapped in the pits (local minima).