Fit a Linear Regression Model with Gradient Descent from Scratch | by GreekDataGuy | Towards Data Science

We all know sklearn can fit models for us. But do we know what it’s actually doing when we call .fit(). Keep reading to find out.
Today we’ll write a set of functions which implement gradient descent to fit a linear regression model. Then we’ll compare our model’s weights to the weights from a fitted sklearn model.
Fitting = finding a model’s bias and coefficient(s) that minimize error.
Error = Mean Squared Error (MSE). The mean of the squared differences between actual and predicted values, across a dataset.
Simple Linear Regression = A model based on the equation of a line, “y=mx+b”. It takes a single feature as input, applies and bias and coefficient, and predicts y.
Also, coefficient and bias together will sometimes be referred to as just, “weights”.
Download the California Housing Dataset from kaggle, and load it into a dataframe.
import pandas as pddf = pd.read_csv('california-housing-dataset.csv')df.head()
For this dataset, we typically try to predict median_house_value using all the other features.
But in our case, we’re concerned with fitting a simple linear regression (which only takes a single input feature) so we’ll choose median_income as that feature and ignore the rest. This won’t create the best model possible, but it will make implementing gradient descent simpler.
Count examples in the dataset.
len(df)#=> 20640
That’s a lot of data. Let’s reduce the size by 75%.
df = df.sample(frac=0.25)len(df)#=> 5160
That’s more manageable.
Collect our input features and labels.
X = df['median_income'].tolist()y = df['median_house_value'].tolist()
According to wikipedia,
Gradient descent is a first-order iterative optimization algorithm for finding the local minimum of a function. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or approximate gradient) of the function at the current point.
My translation:
Gradient descent uses the gradient of the error function to predict in which direction the coefficient, m, and bias, b, should be updated to reduce error on a given dataset.
The gradient of each weight, m and b, are calculated separately. A gradient is calculated by averaging a weight’s partial derivative across all examples.
The direction and steepness of the gradient determines in which direction weights are updated, and by how much. The latter is also influenced by a hyper-parameter, the learning rate.
This algorithm repeatedly iterates over the training set and updates weights until the cost function is at it’s minimum.
You can find the partial derivatives of the MSE function (as below), all over the internet so we won’t derive it here.
We’ll implement the above with the following, while iterating across the dataset.
m_gradient += -(2/N) * x * (y - y_hat)b_gradient += -(2/N) * (y - y_hat)
Let’s write a function which takes current weights, features, labels and learning rate, and outputs updated weights.
def bias_coef_update(m, b, X, Y, learning_rate):    m_gradient = 0    b_gradient = 0        N = len(Y)        # iterate over examples    for idx in range(len(Y)):        x = X[idx]        y = Y[idx]                              # predict y with current bias and coefficient        y_hat = (m * x) + b        m_gradient += -(2/N) * x * (y - y_hat)        b_gradient += -(2/N) * (y - y_hat)            # use gradient with learning_rate to nudge bias and coefficient    new_coef = m - (m_gradient * learning_rate)    new_bias = b - (b_gradient * learning_rate)        return new_coef, new_bias
The latter part of this function multiplies the gradient by the learning rate and uses the result to update current weights. The higher the learning rate the faster the model fits, at the cost of finding the exact local minimum (note: it will never actually reach the true minimum).
Write another function which iteratively applies the above function for a set number of epochs. This is a less sophisticated approach (for simplicity) than returning fitted weights at some predetermined gradient steepness.
def run(epoch_count=1000):    # store output to plot later    epochs = []    costs = []            m = 0     b = 0     learning_rate = 0.01    for i in range(epoch_count):        m, b = bias_coef_update(m, b, X, y, learning_rate)        print(m,b)                C = cost(b, m, x_y_pairs)                epochs.append(i)        costs.append(C)        return epochs, costs, m, bepochs, costs, m, b = run()
Let’s output the final cost and weights of the fitted model.
print(m)print(b)print(costs[-1])# I've rounded these myself so they're nicer to look at#=> 46,804#=> 19,963#=> 7,261,908,362
And chart how it improved over epochs.
import matplotlib.pyplot as pltplt.xlabel('Epoch')plt.ylabel('Error')plt.suptitle('Cost by epoch')plt.plot(epochs,costs, linewidth=1)
Cool. We can see that it made most of it’s progress within the first 100 epochs.
Now we’ll check the same values in a model fitted with sklearn.
Reshape features.
import numpy as npX_array = np.array(X).reshape(5160,1)
Fit a model.
from sklearn.linear_model import LinearRegressionmodel = LinearRegression()model.fit(X_array,y)
Check weights and error.
from sklearn.metrics import mean_squared_errorm = model.coef_[0]b = model.intercept_mse = mean_squared_error(y_test, y_pred, sample_weight=None, multioutput='uniform_average')print(m)print(b)print(mse)# rounded#=> 42,324#=> 41,356#=> 7,134,555,443
Sklearn’s tuning outperformed ours by a small margin, 7,134,555,443 VS 7,261,908,362 but we got pretty close.
Our bias is also quite different from the bias that sklearn found, 41,356 VS 19,963.
Without digging into how values changed after each epoch (which you can easily do), I’d guess we could improve our model by both decreasing the learning rate and increasing the number of epochs.