Linear Regression Model from Scratch
By: Jyotishman Ghatak
Tags: Machine Learning, Linear Regression, First Post, Python, Beginners Guide

To understand it better we are going to take an example and build a linear regression model using a given dataset.

Consider a real estate company “JG Real Estates” who wants to predict the price of the houses they are going to build. They want to figure out the right price that one should pay for it so that they can maximize their profit. To find it out they have opted linear regression algorithm.

Let’s consider this as the data set they are going to use.

In this data set, there are two features (area, number of rooms). These features predict the price that the house should have. Using the above dataset we are going to train and fit our model.

Well, till now you have seen too many theories on linear regression. Now it's time to gear up for some practical implementation.

The very first thing we need to do is to import the python libraries and load data in the pandas data frame. You can download the dataset from here.

Importing libraries and Dataset

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt column=["area","room","price"] df=pd.read_csv('ex1data2.txt',names=column,delimiter=',')

The next thing we gonna do is to create a feature matrix(area, rooms) and a vector of the target variable(price).

X=df.iloc[:,:2]

Y=df.iloc[:,2:3]

Feature Scaling

Now to make our linear regression algorithm more efficient we will perform feature scaling. But, what difference does it make???

Feature scaling is nothing but normalizing the range of values of the features. In simple words, feature scaling ensures that all the values of features are in a fixed range. Feature scaling isn’t required for every machine learning algorithm but it is essential for gradient descent-based algorithms and linear regression is one of them. Features on a similar scale help gradient descent to converge more quickly to minima.

There are different ways to scale features. But we are going to use Standard Scaler.

muX=X.mean()

sigmaX=X.std() X=(X-muX)/sigmaX

Y=(Y-Y.mean())/Y.std()

X=X.to_numpy()

Y=Y.to_numpy()

Now moving forward, we need to add the “bias term” to the feature matrix. Without bias term, the linear regression line will have to go through the origin every time and when all the features have value zero it will predict the value as zero, which might not be the answer in the training data. Therefore we add a column vector Xo to the feature matrix to easily fit the data. For convenience we use Xo=1.

A=np.ones((X.shape[0],1))

X=np.concatenate((A,X),axis=1)

X

Feature matrix with a bias term

So we have prepared the feature matrix and output vector.

Moving to the next part. Let’s recall the cost function that we have seen above. Before we implement it we need to set our hyperparameters and initialize the value of theta(coefficients of the equation) as 0.

theta=np.zeros((1,3)) #creates a column vector

iters=1000

alpha=0.03 #rate of descent

Cost Function and Gradient Descent

And now to implement cost function:

def costfunction(X,Y,theta):

m=len(X)

s=np.power((np.matmul(X,theta.T))-Y,2)

J=np.sum(s)/(2*m)

return J

Here J is the cost returned by the function. Let’s see the cost initially.

cost=costfunction(X,Y,theta)

cost

0.48936170212765967 is the cost returned by the function.

Now we have to minimize the cost function using gradient descent. As we have discussed above that gradient descent is an optimization algorithm that changes the value of the coefficient(theta) of our linear regression line such that the cost function is minimized.

def gradientdescent(X,Y,theta,alpha,iters):

m=len(X)

costg=np.zeros(iters)

for i in range(0,iters):

theta=theta-(alpha/m)*np.sum(X*(X@theta.T-Y),axis=0)

costg[i]=costfunction(X,Y,theta)

return theta,costg thetag,costg=gradientdescent(X,Y,theta,alpha,iters)

After the gradient descent function is invoked we get the minimized cost and our theta values(coefficients) are optimized such that it best fits the data.

Let’s see what values we got in theta

thetag

array([[-1.10727030e-16, 8.84764932e-01, -5.31777642e-02]])

Now, lets look at the cost we get after gradient descent

finalcost=costfunction(X,Y,thetag)

finalcost

Cost after gradient descent becomes 0.13068648053952175.

For better understanding, we will visualize how the cost function is getting minimized over multiple iterations.

Visualization of the Cost function

k=np.arange(iters);

plt.plot(k,costg)

plt.xlabel("Number of Iteration")

plt.ylabel("Cost J(theta))")

plt.show()

As we can see that the cost function gets gradually decreased and after some iterations it becomes constant. Hence, the cost function is minimized.

If you have come this far and understood the concepts well then voila! you have created a linear regression model from scratch successfully.

Now it’s time for you to get a reward from “JG Real Estates” as they are going to use your model for predictions 😉.