Machine Learning #6 — Enter Gradient Descent
By: Vedant Mathur
Tags: Artificial Intelligence, Machine Learning, Algorithms, Gradient Descent, Computer Science

In the last post, we delved deeper into the chasms of the cost function, exploring its intricacies in pursuit of a deeper understanding of the what significance it holds in the grand scheme of Linear Regression. We identified how to manipulate algebraic expressions involving this cost function to derive an exact and precise method that is a one-for-all solution to single-parameter linear regression problems.

With only one parameter, determining the value tied to the lowest cost is a rather simple task. However, when you introduce further parameters, this process drifts towards a more tricky terrain.

Figure 1.1 — A 3-dimensional graph of a cost function

The rather horrendous-looking graph above us represents a three-dimensional plot, featuring both parameters theta 0 and theta 1. It combs through each potential value for the two parameters, and evaluates the corresponding cost function to it.

So how do we read this cryptic and ambiguous graph? If we keep in mind our overarching goal of achieving a low cost, then it only follows that we should be funneling through the graph for a value that is closest, or equal to, 0.

Say we start off with a random parameter set of {0.7, 0.4}. Although I can’t read the graph by eye, let’s assume it corresponds to a cost function output of 2.1.

Now, how can we, in a non-random and exact manner, maneuver down from the point (0.7, 0.4, 2.1) to a point (theta 0, theta 1, 0). To that question, computer scientists quickly responded with an innovative solution, that will drive down the graph to the lowest possible cost — Gradient Descent.

What is Gradient Descent?

In essence, gradient descent refers to a ‘first-order iterative optimization algorithm for finding the minimum of a function’.

So yet again, we’ve seem to have stumbled upon another cryptic jumble of technical jargon. But, if we place this definition under the microscope, it actually is quite intuitive.

The phrase ‘first-order iterative’ simply refers to how this function improves with each ‘step’ it takes. It’s goal, at the end of the day, is to ‘optimize’ our cost function, as it scours through the graph in pursuit of the lowest cost value potential. And the rest is pretty self-explanatory. It’s overarching goal is to unconcealed and uncover the minimum of the cost function.

But how does it work? How can we articulate this almost dream-like concept into clear, concrete formulas and algorithms. The answer lies within the realms of recursive algorithms.

Now, to better illustrate the intricacies and underlying intuition behind gradient descent, let’s approach this problem from the lens of one parameter again, by eliminating theta 0.

A nice way to visualize how gradient descent actually works is to picture the lowest point (the vertex) blindfolded. It would check to see whether the cost of the hypothesis function is either too far to the left, or too far to the right. From that, it will pull that point as close to the lowest as possible, and keep on going until it finally arrives there.

Figured 1.2 — An illustration I created in Photoshop that visualizes the process of Gradient Descent on Parabolas

The diagram I drew up right above offers a simple window into how Gradient Descent works. Instead of just resorting to pure brute force, the process of Gradient Descent offers us an elegant and far more efficient way to approach the lowest cost value.

As can be seen, the method falls back on analyzing how high the cost is, and employing these figures to maneuver the point back and forth, gradually converging to the vertex with each iteration of the gradient descent algorithm.

And the best part is the math behind this isn’t actually as sophisticated as one would think.

So now, we’re faced with the problem. How can we mathematically represent this idea of moving towards the vertex? The answer, lies in the principles of differential calculus.

The first step in the road to solving this gradient descent problem is to derive a way to check if we’re right. So, how can we measure or determine if we have arrived at the vertex? To that, the answer is quite simple. If we peek at the properties of a vertex, it dawns upon us that it is also a critical point in the cost function, and therefore the tangent to the graph at that point (the derivative), would be 0. With this, we have unveiled the seed to which gradient descent sprouts from and upon — the instantaneous slope.

Let’s say we’re at a point in the graph to the right of the vertex as so:

Figured 1.3 — A simple parabola that illustrates the nature of a tangent to a point

As the graph illustrates, with a point too far to the right of the vertex, the derivative of the function at that point would be a positive number, as the slope of the tangent (the light blue line) is clearly greater than 0.

Similarly, it then becomes clear that if the hypothesis landed on a point too far to the left of the vertex, its derivative would yield a number less than 0 (negative).

Now, not only do we have a means to check if we have reached the vertex, but we now have ways to quantitatively measure exactly how far off we are, and where on the graph it is just by looking at the numbers.

From this, the fragmented pieces of the gradient descent algorithm begin to stitch themselves together. It is quite transparent that the derivative of the cost function will play a key role here, and from that we have the foundation of our cost function formula, which we can now build upon.

Let’s imagine our vertex is the point (4, 2), and we’re right now at the point (6, 4).

Figured 1.4 — A more mathematical representation of how tangents tie in with gradient descent

The equation for this cost function that I have graphed can be expressed as so:

Figured 1.4 — An arbitrary function that is modeled after a cost function

You’ll quickly notice this isn’t the typical form of expressing the cost function. Without the jumble of Sigmas and derivatives, it’s almost unrecognizable. But this is just an example scenario that I engineered to illustrate the mathematical intuition behind the cost function, as we are going to work backwards from this equation, seeing how we can push the point (6, 4), to (4, 2) in a precise, exact and universal way.

With some simple differentiation, we can derive the first-order derivative of this function.

Figured 1.5 — The derivative of the previous function (fig 1.4) with respect to theta

For those of you who aren’t familiar with differentiation, do not worry. In essence, it’s the study of how graphs behave on a microscopic rate of change, which points back towards our work on tangents at a point. The derivative of a function at a particular point gives us the slope of its tangent.

Now if we plug in our theta value of 6 into this derivative function, it yields a value of 2 (6–4 = 2). All that means is that at the point (6, 4) on the graph (figure 1.5), the slope of the tangent to that point is 2. Obviously, 2 is not equal to 0, so even a computer can tell that they aren’t there yet. But, you’ll notice that the more right up the parabola you go, the steeper and steeper it gets. In math, the formal way to express ‘steepness’, is a greater slope. Similarly, the further left we go from the point (6, 4), the flatter and lower our slope will get until it approaches the vertex. So, if we want our graph to approach 0, we want to go left. And to do that, we need to move the x-coordinate of our point by a negative value.

But let’s take a look at the other side of the fence. If our point is too far to the left of the graph, we can tell that its tangent will bear a slope less than 0 (in the negatives). In this case, our derivative value is too low, as we need to add to the point to approach an instantaneous slope of 0. Therefore, we end up adding to the x-coordinate.

Do you notice the pattern? If our derivative is positive, we subtract. If it’s negative, we add. And that’s the key that unlocks the real meat behind the basic gradient descent algorithm. We have to negate the sign of the derivative. And how do we do that? We turn to our friendly and familiar subtraction symbol.

So, now we know we have to subtract our derivative. But from what? Do we take it away from some arbitrary constant, or pick a specified point on the graph for consistency and further reference. The answer, in reality, lies somewhere in the middle.

To approach our vertex, we have to take our derivative and subtract it from our x-coordinate. On the cost function, the x-axis represents the parameter theta. Therefore, we can now derive that all we need to do is subtract the derivative of the cost function at that point from the parameter it represents.

Phew, so now we have a pretty clear picture of what a gradient descent algorithm would look like. Given what we’ve discussed already, it should resemble something like this:

Figure 1.6 — The first piece of our gradient descent algorithm

We’re almost there! We are just missing one important piece of the puzzle — the learning rate.

When tackling most, if not all, machine learning problems, we tend to introduce a constant that determines and manages (controls) the rate, or speed, of the reaction. This learning rate acts as a coefficient to the derivative value that will cap the speed at which the gradient descent is performed.

And so we now have it! A fully fledged gradient descent expression for a single-parameter cost function.

Figure 1.7 — The generic formula for a single-parameter gradient descent

After having a go at plugging in the actual formula instead of J(theta), we can derive a far more precise and simplified formula for gradient descent steps.

But now, instead of continuing this one parameter situation, let’s fall back to the two-parameter gradient descent.

Even though the three dimensional graph may not seem like it would have a vertex that has a derivative slope of 0, it actually does. And through this, pulling the gradient descent concept that we have derived for single-parameter hypotheses actually becomes a matter of just adding one more variable — theta zero.

And so all we need to do is re-introduce this wonderful parameter, and we’re back in business for a fully functional linear regression problem.

In case you were wondering why adding the second parameter is so important, just remember what we covered a few posts ago in the definition of the hypothesis function. If we only have one parameter, our hypothesis can be in the form:

Figure 1.8 — The hypothesis function for a one-parameter system

And so, after scouring through the foreign and alien symbols, what this actually highlights to us lies with how we can only actually have a linear equation in the form y = mx, not y = mx + c.

By doing this, we’re actually missing out on an infinite number of additional lines, all of which can only bring us closer to the actual plot of data we’re working to crack (as the c constant can be at a value of 0 too).

So, by reintroducing the second parameter, we can have our hypothesis in the form of all potential line equations.

Now, with both parameters, theta 0 and theta 1, we realize that our gradient descent algorithm actually shifts a little bit. This stems from how we now need to now only apply this gradient descent on one parameter, but both. Therefore, we’ll actually need two separate equations, both very similar in nature, yet still distinct in form.

We start off with our gradient descent formulas being represented as so:

Hooray! But having this derivative hanging over our heads there is quite irritating, as we would have to go through the grueling process of differentiating the function every time we wanting to run a step of gradient descent.

And so, the need arises for us to eliminate or simplify out this differential call in the formula. To do that, we finally have to enter the terrifying realms of calculus, employing both the first-principles of differentiation, coupled with the chain rule.

Let’s start off with deriving the full formula for theta 0. Now to achieve this, I will be employing the technique of partial derivatives, which is a way to tackle differentiation with multivariate equations (on a multivariable scale).

Now, after just taking a glance at this working out, I see how it looks complicated. However, all you need to understand is that we went from having abstract and tricky partial derivative to a clear, concise, and exact formula. From this, all we need to do is plug in the values, and sit back and watch gradient descent run its magic.

But we’re not quite done yet. We have derived the partial derivative with respect to theta 0, but what about theta 1? Well for that, we just follow a very similar process, so brace yourselves for another round of partial differentiation fun!

As you may notice, it is easy to draw parallels between our derivation for theta 0, and that for theta 1. The key distinction that leads to the extra x coefficient in the Sigma for theta one lies with how it has x as a trailing coefficient in the hypothesis, so when it is differentiated, unlike theta 0, it won’t yield a value of 1; rather, it will yield a value of x. That’s why we have that extra x coefficient tagging along at the end of the formula for theta 1.

In reality, even though the symbols and the functions may seem intimidating at first, if we look past its rather scratchy and rough exterior around the edges, it is actually quite intuitive. It takes the slope of the tangent to our current point on the cost function, multiplies it by the constant learning rate, and subtracts it from our current x-value.

Now, at first, I was wondering what significance subtracting the derivative had. If the slope was 2, it wouldn’t make that much of a difference is we subtracted 2 instead of say, 4 (or any other positive number). Really here, the only reason we needed any number was for the sign — whether it is positive or negative. But then I realized that there is no other way to determine the sign itself other than to expand out and simplify the differentiation of the partial multivariate polynomial. After that, it is anyways multiplied by the learning rate, so the actual absolute value of the number is almost irrelevant — it’s the sign that counts. Of course, that isn’t saying that we can just subtract 1000 instead of 10; that would make it impossible for the gradient descent to eventually converge on the point. But it did open my eyes into the deeper intuition that fuels this fascinating algorithm.

Even though these formulas may initially come off as truly arcane clusters of alien symbols, what it represents is truly remarkable. The way that this algorithm both elegantly and swiftly bests brute force is fascinating and enlightening. With just two clean formulas, we can take any graph of any cost function, and be sure that we end up at the very lowest point potential.

And there you have it. Two complete (but very unreadable) equations that we can employ to carry out our basic gradient descent algorithm. But the next step, as this is of course machine learning problem, is to bring in our favorite science in the world — computer science. How can we model this algorithm in programming. Fortunately, programming it is far easier than the math behind it, so you can take a breath — the hard part is over.

With just a couple loops and some basic algebraic manipulation, sketching up an interpretation of gradient descent in programming isn’t as tricky as it may seem.

So in the next post, we’ll explore the programmatic implications of this algorithm, and some do’s and don’ts while implementing gradient descent. If you’ve managed to make it to the end of this whole post in one sitting, good on you — this was quite heavy.

See you next time!