Machine Learning: Python Linear Regression Estimator Using Gradient Descent
By: Benjamin Obi Tayo Ph.D.
Tags: Machine Learning, Gradient Descent, Python, Linear Regression, Optimization

Simple Linear Regression Estimator in Python | Towards AI

In this article, we describe how a simple python estimator can be built to perform linear regression using the gradient descent method. Let’s assume we have a one-dimensional dataset containing a single feature (X) and an outcome (y), and let’s assume there are N observations in the dataset:

A linear model to fit the data is given as:

where w0 and w1 are the weights that the algorithm learns during training.

Gradient Descent Algorithm

If we assume that the error in the model is independent and normally distributed, then the likelihood function is given as:

To maximize the likelihood function, we minimize the sum of squared errors (SSE) with respect to w0 and w1:

The objective function or our SSE function is often minimized using the gradient descent (GD) algorithm. In the GD method, the weights are updated according to the following procedure:

i.e., in the direction opposite to the gradient. Here, eta is a small positive constant referred to as the learning rate. This equation can be written in component form as:

If you would like to find out more about the GD algorithm and why it works, see the following article: https://medium.com/@benjaminobi/machine-learning-how-the-gradient-descent-algorithm-works-61682d8570b6

Implementation Using Python Estimator

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

class GradientDescent(object):

"""Gradient descent optimizer.

Parameters

------------

eta : float

Learning rate (between 0.0 and 1.0)

n_iter : int

Passes over the training dataset.



Attributes

-----------

w_ : 1d-array

Weights after fitting.

errors_ : list

Error in every epoch.

""" def __init__(self, eta=0.01, n_iter=10):

self.eta = eta

self.n_iter = n_iter



def fit(self, X, y):

"""Fit the data.



Parameters

----------

X : {array-like}, shape = [n_points]

Independent variable or predictor.

y : array-like, shape = [n_points]

Outcome of prediction.

Returns

-------

self : object

"""

self.w_ = np.zeros(2)

self.errors_ = []



for i in range(self.n_iter):

errors = 0

for j in range(X.shape[0]):

self.w_[1:] += self.eta*X[j]*(y[j] - self.w_[0] - self.w_[1]*X[j])

self.w_[0] += self.eta*(y[j] - self.w_[0] - self.w_[1]*X[j])

errors += 0.5*(y[j] - self.w_[0] - self.w_[1]*X[j])**2

self.errors_.append(errors)

return self def predict(self, X):

"""Return predicted y values"""

return self.w_[0] + self.w_[1]*X

Application of Python Estimator

a) Create dataset

np.random.seed(1)

X=np.linspace(0,1,10)

y = 2*X + 1

y = y + np.random.normal(0,0.05,X.shape[0])

b) Fit and Predict

gda = GradientDescent(eta=0.1, n_iter=100)

gda.fit(X,y)

y_hat=gda.predict(X)

c) Plot Output

plt.figure()

plt.scatter(X,y, marker='x',c='r',alpha=0.5,label='data')

plt.plot(X,y_hat, marker='s',c='b',alpha=0.5,label='fit')

plt.xlabel('x')

plt.ylabel('y')

plt.legend()

d) Calculate R-square value

R_sq = 1-((y_hat - y)**2).sum()/((y-np.mean(y))**2).sum()

R_sq

0.991281901588877

In summary, we have shown how a simple linear regression estimator using the GD algorithm can be built and implemented in Python. If you would like to see how the GD algorithm is used in a real machine learning classification algorithm, see the following Github repository.

Thanks for reading.