Building Your First Machine Learning Model: Linear Regression Estimator
By: Benjamin Obi Tayo Ph.D.
Tags: Machine Learning, Optimization, Python, Gradient Descent, Linear Regression

There are so many good packages that can be used for building predictive models. Some of the most common packages for predictive analytics include

Sci-kit learn package

Caret package

Tensorflow

It’s important that before using these packages, you master the fundamentals of predictive modeling, that way you are not using these packages simply as blackbox tools.

One way to understand the functioning of machine learning models is for you to actually learn how to build your own models. The simplest machine learning model is the linear regression model. Everyone new to data science should master the fundamentals of the linear regression estimator, since a majority of machine learning models (such as SVM, KNN, Logistic Regression, etc.) are very similar to the linear regression estimator.

In this article, we describe how a simple python estimator can be built to perform linear regression using the gradient descent method. Let’s assume we have a one-dimensional dataset containing a single feature (X) and an outcome (y), and let’s assume there are N observations in the dataset:

A linear model to fit the data is given as:

where w0 and w1 are the weights that the algorithm learns during training.

Gradient Descent Algorithm

If we assume that the error in the model is independent and normally distributed, then the likelihood function is given as:

To maximize the likelihood function, we minimize the sum of squared errors (SSE) with respect to w0 and w1:

The objective function or our SSE function is often minimized using the gradient descent (GD) algorithm. In the GD method, the weights are updated according to the following procedure:

i.e., in the direction opposite to the gradient. Here, eta is a small positive constant referred to as the learning rate. This equation can be written in component form as:

If you would like to find out more about the GD algorithm and why it works, see the following article: Machine Learning: How the Gradient Descent Algorithm Works.

Implementation Using Python Estimator

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt class GradientDescent(object):

"""Gradient descent optimizer.

Parameters

------------

eta : float

Learning rate (between 0.0 and 1.0)

n_iter : int

Passes over the training dataset.



Attributes

-----------

w_ : 1d-array

Weights after fitting.

errors_ : list

Error in every epoch.

""" def __init__(self, eta=0.01, n_iter=10):

self.eta = eta

self.n_iter = n_iter



def fit(self, X, y):

"""Fit the data.



Parameters

----------

X : {array-like}, shape = [n_points]

Independent variable or predictor.

y : array-like, shape = [n_points]

Outcome of prediction.

Returns

-------

self : object

"""

self.w_ = np.zeros(2)

self.errors_ = []



for i in range(self.n_iter):

errors = 0

for j in range(X.shape[0]):

self.w_[1:] += self.eta*X[j]*(y[j] - self.w_[0] - self.w_[1]*X[j])

self.w_[0] += self.eta*(y[j] - self.w_[0] - self.w_[1]*X[j])

errors += 0.5*(y[j] - self.w_[0] - self.w_[1]*X[j])**2

self.errors_.append(errors)

return self def predict(self, X):

"""Return predicted y values"""

return self.w_[0] + self.w_[1]*X

Application of Python Estimator

a) Create dataset

np.random.seed(1)

X=np.linspace(0,1,10)

y = 2*X + 1

y = y + np.random.normal(0,0.05,X.shape[0])

b) Fit and Predict

gda = GradientDescent(eta=0.1, n_iter=100)

gda.fit(X,y)

y_hat=gda.predict(X)

c) Plot Output

plt.figure()

plt.scatter(X,y, marker='x',c='r',alpha=0.5,label='data')

plt.plot(X,y_hat, marker='s',c='b',alpha=0.5,label='fit')

plt.xlabel('x')

plt.ylabel('y')

plt.legend()

d) Calculate R-square value

R_sq = 1-((y_hat - y)**2).sum()/((y-np.mean(y))**2).sum()

R_sq

0.991281901588877

In summary, we have shown how a simple linear regression estimator using the GD algorithm can be built and implemented in Python. If you would like to see how the GD algorithm is used in a real machine learning classification algorithm, see the following Github repository.