The Math of Gradient Descent With Univariate Linear Regression
By: Adam Dhalla
Tags: Linear Regression, Machine Learning, Mathematics, Gradient Descent, Data Analysis

In 1809, legendary mathematician Carl Friedrich Gauss, now in his early 30’s, published Theoria Motus Corporum Coelestium in Sectionibus Conicis Solem Ambientium — a landmark theoretical paper on the mathematical laws and formulas describing the orbit of celestial bodies around the sun.

The Prince of Mathematicians himself, Carl Friedrich Gauss, was in most definitions of the word, a genius

Within it, he claimed to have created the idea of least squares regression — a way to plot a line of best fit for a scatter plot of data points, no matter how dispersed the points are.

Gauss promptly received a frustrated letter from prolific mathematician Adrien-Marie Legendre saying that he, indeed, had discovered the exact same formula four years prior in 1805, and had published a paper, New Methods for Determination of the Orbits of Comets, which included the original notation for least squares regression.

An excerpt of Legendre’s 1805 paper on orbits of comets, which included the first published proposal of least squares regression, published four years before Gauss published his own.

Gauss argued that he had thought of the idea independently as far back as 1795, but did not publish it since he found it ‘trivial’. Although there’s now significant evidence supporting Gauss’ claim, the feud between Gauss and Legendre became one of the most widely publicized plagiarization-related scandals to every see the scientific world.

Despite that, we generally attribute the ‘invention’ of linear regression to both Gauss and Legendre, as well as later mathematicians who expanded the idea, such as Francis Galton.

Use case

Say we have a dataset with two columns, with an independent and dependent variable that display correlation.

Imagine we gathered data from 10 parks, and collected the amount of species of trees in the park, and collected data on how many bird species can be seen within the park. Plotting this data, we see that there’s a correlation (generally, the more tree species in a park, the higher amount of bird species)

Is there a way we can create a prediction algorithm that predicts the amount of birds found in a park if the amount of trees is given?

Specifically, can we automate the process of finding the equation for this line of best fit?

We would do this by plotting a line of best fit on to the scatter plot, to give us an equation that in “y=mx + b” form that gives us a prediction of y (species of birds) when given x (species of trees). All our predictions would lie on this line.

Hypothesis

The equation of our line of best fit is given by this equation. We call this our hypothesis, represented by the function h(x).

If this seems foreign, imagine it as the y = mx + b equation of a line, but θ0 as b, the y intercept, and θ1 as the slope, m.

Thus, in univariate (dealing with one variable, x) linear regression, we must find a way to tweak these two thetas to get a line that best represents the line of best fit. Here, θ0 is the bias, in that it shifts the entire line by default (the y intercept). θ1 represents the weight, or how much does the input x impact the output y, or in our scenario, what is the number that relates the amount of trees to the amount of birds?

The line of best fit changes as our θ1 and θ0 change. We are trying to find the ideal values of theta that best match the training data, here, the one in the bottom right.

Before we judge how to best tweak the theta values, we must first how we concretely evaluate the success of our prediction algorithm? This is where the mean squared error (MSE) cost function can help us.

Mean Squared Error (MSE) Cost Function

The mean squared error (cost function) finds the average squared difference between our predictions, and the actual y values in our training set. Our goal is to minimize the total cost.

For each training example, MSE adds up the lengths of all the dashed lines, which are the differences between each actual y values and the predicted values (h(x) - y) . Each of those differences are then squared to get rid of negatives (h(x) — y)², and then added to equal a single scalar “cost”. That cost is then averaged (1/2m).

This final scalar is how we evaluate how accurate our prediction algorithm is.

We are trying to choose the proper θ1 and θ0 to minimize the final scalar given by the cost function, which means the differences (dashed lines) are as short as possible.

Now, to automatically find the values for θ1 and θ0 that minimize the cost function, we can use a technique widely use in machine learning, gradient descent.

An introduction and intuitive understanding of gradient descent

Here is an example of what a graph of θ1 and θ0, with the result of the cost function as the z axis. via Andrew Ng/Stanford

Since we are trying to minimize the value of cost, we are trying to find the deepest points in this graph, which correlate to the θ1 and θ0 that give the lowest cost, J(θ1, θ0).

Gradient descent is an iterative algorithm that randomly initializes somewhere on this plane. It then adjusts the values of θ1 and θ0 until it gets to a point on the plane where the cost function is low.

Each star is where gradient descent re-runs.

Intuitively, what gradient descent does is upon each iteration (each star on the graph) it ‘looks around a full 360 degrees’ and chooses the direction of steepest (most efficient) descent. It then takes a baby step in that direction, and reiterates again, until it converges — when it gets to a point where the cost can’t go any lower.

Below are the mathematical formulas that describe how these steps are taken (note that the formulas on the right are directly equivalent, but have the derivative term, the fraction thing, calculated)

The gradient descent update rules for theta0 (bias) and theta1 (weight). For each iteration of gradient descent, the values of theta are updated according to these formulae. The formula on the right is equivalent, just with the partial derivatives of θ0 and θ1 calculated. For those comfortable with calculus, I’ll prove those derivations near the end of the article.

We can understand the logic of these update rules better with a one dimensional example — let’s modify our hypothesis to only have one theta — getting rid of the bias.

Getting rid of the bias term would mean the line of best fit plotted by the hypothesis must run through the origin.

Now, the cost is only a function of θ1 — cost is now represented by function J(θ1). The model of the cost function is now a 2d graph, and can be represented like below. We are continuing to look for the minima.

The derivative term d/dθ1 of J(θ1) is what we should focus on. The derivative is the slope at a single point — for those who are rusty on derivatives, think of this, simply, as the slope between two points very close to each other.

Ignore α (the ‘learning rate’ )for now, and just think of it as some scalar.

When the derivative/slope is positive (it is right of the minimum), it multiplies with the negative α, and subtracts value from θ1, bringing it closer to the minimum from the right.

When the derivative/slope is negative(it is left of the minimum), it multiplies with the negative α, and adds value to θ1, bringing it closer to the minimum from the left.

As an added bonus, the magnitude (absolute value) of the derivative decreases as it approaches the minimum (where the slope is 0), which means that the gradient update automatically slows down as the slope starts to flatten. Mathematically, this is because the product between α and the derivative shrinks, and thus the update to θ1 gets smaller.

Learning rate α

The learning rate dictates the size of the step for each iteration of gradient descent. The learning rate is a hyperparameter, which means that it must be manually chosen by the human operating the ML algorithm.

Intuitively, since the learning rate scales the derivative, it determines by how much each parameter is updated after each iteration of gradient descent. Finding the correct learning rate size depends on the project and data given.

Mathematical explanation for gradient descent

Partial derivatives

If you understand calculus, you’ll know what a derivative is — the rate at which some function changes with respect to it’s input.

Where df/dx is how f changes with respect to (w.r.t) x, df/dy is how f changes w.r.t y, and df/dz is how f changes w.r.t z.

Partial derivatives are similar, but only come into play in multivariable functions, (functions with multiple variables as inputs). Partial derivatives are calculated by simply taking the derivative of the variable in question, but by keeping the other variables as constants.

Each multivariable function has as many partial derivatives as it has input variables into the function.

Gradient

The gradient of a multivariable function is simply a vector of all of its partial derivatives, signified by ∇. For example, for our previous multivariable function, the gradient would be:

One of the characteristics of the gradient vector, when placed in the input space of the function, is that it always points in the direction of steepest ascent. (no matter the amount of dimensions). The proof for this is rather intuitive but requires an understanding of directional derivatives, as well as basic linear algebra (dot products).

This is a gradient field. It plots the gradients (arrows) at each point — the reason they change in shape and direction is that the value of the partial derivatives change (and thus, the gradient) from point to point. The important thing to notice is that they all point towards the angle of steepest ascent relative to their point. Martin Thoma

Sensibly, then, taking the negative gradient of the function would mean getting the directions of steepest descent. So, we could expect the best way to get each theta down is to adjust all thetas by the negative gradient (adjusting each theta with the negative partial derivative component that corresponds to the theta).

Gradient-based intuition for update rules

The mathematical intuition for these two update rules is actually quite simple once the initial ideas of multivariable (and single variable) calculus is understood.

Just focusing on the derivative terms of the two updates:

Notice when we do each update consecutively, by calculating the partial derivative of θ0 w.r.t the cost function J(θ0, θ1), and then right after, calculating the partial derivative of θ1 w.r.t the cost function, it’s kind of like computing the gradient, in that we’re calculating all the partial derivatives of the cost function J(θ0, θ1) at the same time.

Although we aren’t technically dealing with the gradient, we are informally doing the same thing.

Furthermore, when we zoom out, we can see that we are dealing with the negative partial derivatives (the -α is the same as a minus before the partial derivatives, because of the associative property of multiplication).

If we were to think of this as a vector operation, it could be adjusting a theta vector with [θ0, θ1] by the negative gradient scaled by some learning rate α.

The equivalent vector operation, where we can express the partial derivatives as a vector of partial derivatives; thus, a negative gradient.

Intuitively, you’re trying to minimize multiple little partial derivatives at the same time, which then add up to something significant (the gradient)

That’s the intuition for the update rule and how it relates to derivatives. Next, I’ll describe the straight derivation of the update rules we covered earlier.

Derivations of the partial derivatives w.r.t θ0, θ1

Why is this true?

The derivation is pretty simple year 1 calculus stuff, except for the brief use of multivariable calculus in determining the second product after using the chain rule.

This technically only derives the partial derivative segment of the overall update rule, but the rest (such as learning rate α, stay as they are).