Fit a Linear Regression Model with Gradient Descent from Scratch
By: Chris I.
Tags: Machine Learning, Gradient Descent, Tutorial, Linear Regression, Data Science

Fit a Linear Regression Model with Gradient Descent from Scratch

Implement gradient descent to find optimal weights for a simple linear regression.

We all know sklearn can fit models for us. But do we know what it’s actually doing when we call .fit() . Keep reading to find out.

Today we’ll write a set of functions which implement gradient descent to fit a linear regression model. Then we’ll compare our model’s weights to the weights from a fitted sklearn model.

Required background

Fitting = finding a model’s bias and coefficient(s) that minimize error.

Error = Mean Squared Error (MSE). The mean of the squared differences between actual and predicted values, across a dataset.

Simple Linear Regression = A model based on the equation of a line, “y=mx+b”. It takes a single feature as input, applies and bias and coefficient, and predicts y.

Also, coefficient and bias together will sometimes be referred to as just, “weights”.

Prepare the dataset

Download the California Housing Dataset from kaggle, and load it into a dataframe.

import pandas as pd

df = pd.read_csv('california-housing-dataset.csv')

df.head()

For this dataset, we typically try to predict median_house_value using all the other features.

But in our case, we’re concerned with fitting a simple linear regression (which only takes a single input feature) so we’ll choose median_income as that feature and ignore the rest. This won’t create the best model possible, but it will make implementing gradient descent simpler.

Count examples in the dataset.

len(df)

#=> 20640

That’s a lot of data. Let’s reduce the size by 75%.

df = df.sample(frac=0.25)

len(df)

#=> 5160

That’s more manageable.

Collect our input features and labels.

X = df['median_income'].tolist()

y = df['median_house_value'].tolist()

Fitting a Linear Regression with Gradient Descent

According to wikipedia,

Gradient descent is a first-order iterative optimization algorithm for finding the local minimum of a function. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or approximate gradient) of the function at the current point.

My translation:

Gradient descent uses the gradient of the error function to predict in which direction the coefficient, m , and bias, b , should be updated to reduce error on a given dataset.

The gradient of each weight, m and b , are calculated separately. A gradient is calculated by averaging a weight’s partial derivative across all examples.

The direction and steepness of the gradient determines in which direction weights are updated, and by how much. The latter is also influenced by a hyper-parameter, the learning rate.

This algorithm repeatedly iterates over the training set and updates weights until the cost function is at it’s minimum.

Implementation

You can find the partial derivatives of the MSE function (as below), all over the internet so we won’t derive it here.

We’ll implement the above with the following, while iterating across the dataset.

m_gradient += -(2/N) * x * (y - y_hat)

b_gradient += -(2/N) * (y - y_hat)

Let’s write a function which takes current weights, features, labels and learning rate, and outputs updated weights.

def bias_coef_update(m, b, X, Y, learning_rate):

m_gradient = 0

b_gradient = 0



N = len(Y)



# iterate over examples

for idx in range(len(Y)):

x = X[idx]

y = Y[idx]



# predict y with current bias and coefficient

y_hat = (m * x) + b m_gradient += -(2/N) * x * (y - y_hat)

b_gradient += -(2/N) * (y - y_hat)



# use gradient with learning_rate to nudge bias and coefficient

new_coef = m - (m_gradient * learning_rate)

new_bias = b - (b_gradient * learning_rate)



return new_coef, new_bias

The latter part of this function multiplies the gradient by the learning rate and uses the result to update current weights. The higher the learning rate the faster the model fits, at the cost of finding the exact local minimum (note: it will never actually reach the true minimum).

Write another function which iteratively applies the above function for a set number of epochs. This is a less sophisticated approach (for simplicity) than returning fitted weights at some predetermined gradient steepness.

def run(epoch_count=1000):

# store output to plot later

epochs = []

costs = []



m = 0

b = 0

learning_rate = 0.01 for i in range(epoch_count):

m, b = bias_coef_update(m, b, X, y, learning_rate)

print(m,b)



C = cost(b, m, x_y_pairs)



epochs.append(i)

costs.append(C)



return epochs, costs, m, b epochs, costs, m, b = run()

Let’s output the final cost and weights of the fitted model.

print(m)

print(b)

print(costs[-1]) # I've rounded these myself so they're nicer to look at

#=> 46,804

#=> 19,963

#=> 7,261,908,362

And chart how it improved over epochs.

import matplotlib.pyplot as plt plt.xlabel('Epoch')

plt.ylabel('Error')

plt.suptitle('Cost by epoch') plt.plot(epochs,costs, linewidth=1)

Cool. We can see that it made most of it’s progress within the first 100 epochs.

Fitting a Linear Regression with Sklearn

Now we’ll check the same values in a model fitted with sklearn.

Reshape features.

import numpy as np

X_array = np.array(X).reshape(5160,1)

Fit a model.

from sklearn.linear_model import LinearRegression

model = LinearRegression()

model.fit(X_array,y)

Check weights and error.

from sklearn.metrics import mean_squared_error m = model.coef_[0]

b = model.intercept_ mse = mean_squared_error(y_test, y_pred, sample_weight=None, multioutput='uniform_average') print(m)

print(b)

print(mse) # rounded

#=> 42,324

#=> 41,356

#=> 7,134,555,443

How did we do?

Sklearn’s tuning outperformed ours by a small margin, 7,134,555,443 VS 7,261,908,362 but we got pretty close.

Our bias is also quite different from the bias that sklearn found, 41,356 VS 19,963 .

Without digging into how values changed after each epoch (which you can easily do), I’d guess we could improve our model by both decreasing the learning rate and increasing the number of epochs.