Linear Regression Using Boston Housing Dataset
By: Akhil Anand
Tags: Programming, Data Manipulation, Machine Learning, Linear Regression, AI

Overview

Boston Dataset is the information collected by U.S census Service concerning housing in Boston city.The data was originally published in 1978 containing nearly 500 samples.We can easily access this data with the help of sklearn library .Our main aim would be to predict housing price based on features present in dataset.let’s get started……

Step 1. Importing Libraries and Acquiring Dataset

Importing libraries

import numpy as np

import pandas as pd

import matplotlib.pyplot as plt

import seaborn as sns

from sklearn.datasets import load_boston

from sklearn.preprocessing import StandardScaler

from sklearn.model_selection import train_test_split

Acquiring Dataset

boston=load_boston()

type(boston) [0ut]>> sklearn.utils.Bunch

The type of boston data is utils.Bunch . sklearn stores data in the form of dictionary like object and Bunch consists of four keys through which we can understand data more precisely i.e:-{data,target,feature_names,DESCR}.

i. data: It gives information about features through which we can do our prediction it excludes target variable.

--------------------------------------------------------------------

boston.data[0] #printing first row of dataset [out]>>array([6.320e-03, 1.800e+01, 2.310e+00, 0.000e+00, 5.380e-01, 6.575e+00,6.520e+01, 4.090e+00, 1.000e+00, 2.960e+02, 1.530e+01, 3.969e+02,4.980e+00]) --------------------------------------------------------------------

boston.data.shape [out]>> (506, 13)

ii. target: It will print data present in target variable.

-------------------------------------------------------------------

#reshaping the target into 2d array so that it would be easily #converted into dataframe further

target=np.array(boston.target).reshape(-1,1)

target.shape

[out]>> (506,1)

--------------------------------------------------------------------

iii. feature_names :- It will print all the features(excluding target variables) present in dataset.

boston.feature_names [out]>> array(['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD','TAX', 'PTRATIO', 'B', 'LSTAT'], dtype='<U7')

iv. DESCR: It gives complete description about dataset including basic definition of features, what does each feature means which feature would act like target variable , is there any missing value present in dataset,Source of dataset ,author of dataset etc.

Figure 1

Next step would be to convert dataset into dataframe so that we can preprocess and visualize data and will find suitable feature for prediction.

#Converting dataset into dataframe

boston_df=pd.DataFrame(boston.data,columns=boston.feature_names)

boston_df["Price"]=boston.target #changing target variable name as #price

boston_df.head() #printing first five rows of dataframe

Figure 2

Step 2. Data Pre-Processing

i. Printing shape of dataframe

boston_df.shape #checking number of rows and columns present in data [out]>> (506, 14)

ii. Printing all the features present in dataframe & data type of each column

-------------------------------------------------------------------

boston_df.columns [out]>>Index(['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE',

'DIS', 'RAD', 'TAX','PTRATIO', 'B', 'LSTAT','Price'],dtype='object')

-------------------------------------------------------------------

boston_df.dtypes #Printing data type of each columns

Figure 3

iii. Is there any null value present in dataset?

boston_df.isnull().values.any() [out]>> False #There is no any null value present in dataset

iv. Statistical Description

boston_df.describe()

Figure 4

Step 3. Data Visualization

i. We will try to understand distribution of target variable boston_df.Price ,Whether target variable has been normally distributed or not.

plt.figure(figsize=(12,6))

sns.set('notebook')

sns.distplot(boston_df.Price,bins=20)

plt.show()

Figure 5

Upper figure shows the boston_df.Price has been normally distributed with some outliers.We can also conclude that maximum number of houses sold within price range of $20000-$24000.

ii. Now we will plot each features with respect to target variable to see whether features has linear relationship with target variable or not.

for i in x.columns: #plotting each features with respect to target

plt.scatter(x[i],y)

plt.xlabel(i)

plt.ylabel("Price")

plt.show()

figure 6(a)

figure 6(b)

If we will analyse above figures we can conclude that NOX , RM , DIS,LSTAT , AGE are showing near about linear character.Hence we can say that these features are too important for prediction of housing price.

iii. Checking multicolinearity using heat map:-

a. As we know while solving linear regression problem each features should be independent with one another if there will be some correlation between two independent variables(Features) then it leads to overfitting the model so with the help of heat map we will eliminate all those features which shows strong correlation with one another.

b. We also try to extract all those features which will have extensive correlation with target variable. Because , if feature will be strongly correlated with target variables then we would expect better prediction.

plt.figure(figsize=(12,6))

corr_val=boston_df.corr()

sns.heatmap(data=corr_val,annot=True)

Heatmap

Extracting all those features which is highly correlated (threshold value=0.5) with target variable.

#Etracting all features which is highly correlated with target

def HighlyCorrelated(data,threshold):

feature=[]

values=[]

for ele,index in enumerate(data.index):

if abs(data[index])> threshold:

feature.append(index)

values.append(data[index])

df=pd.DataFrame(data=values,index=feature,columns= ["Correlation"])

return df threshold=0.5

corr_df=HighlyCorrelated(corr_val.Price,threshold)

corr_df

output

Here RM , PTRATIO , LSTAT are highly correlated with target variable Price.Hence these variables will give better prediction.

Step 4. Detailed overview of model using O.L.S

O.L.S does not do prediction but gives summary about the model.By calling summary() we get to know various parameters that drives the linear regression model for prediction i.e:-β1,β0,p-value,r squared etc.

#As ols is statistical modeling hence first of all we need to import #it from statsmodels

import statsmodels.formula.api as sms

reg=sms.ols(formula="Price~CRIM+ZN+INDUS+CHAS+NOX+RM+AGE+DIS+RAD+TAX+PTRATIO+LSTAT",data=boston_df).fit()

print(reg.conf_int())

print(reg.summary())

O.L.S

a. If we will see table we have coef , which consists Intercept(β0) and β1(would be considered for variable we will take).let’s understand this by an example;

I would like to plot prediction of y^ based on LSTAT

Here we have β0(Taken from table of confidence interval)=31.6173 and β1=-0.5520(coef of LSTAT). Now we will make model from these inputs. i.e:- y_hat=β0 + β1*LSTAT = 31.6173+(-0.5520)*LSTAT

#Plotting linear regression

LSTAT=boston_df["LSTAT"]

plt.scatter(LSTAT,boston_df["Price"])

y_hat=31.6173+(-0.5520)*RM

plt.plot(LSTAT,y_hat,lw=5,c="green")

plt.xlabel("LSTAT",fontsize=20)

plt.ylabel("Price",fontsize=20)

Figure 7

The next thing in the table is p-value . If p-value will be 0.000 means result is statistically significant and accepts null hypothesis.

b. Here we have R Squared and adjusted R Squared by the help of O.L.S we will know which will be better fit for accuracy of model.

c. t represent student t distribution of features and z represent z-statistics of features.

print(reg.rsquared,"

","*"*100,"

",reg.rsquared_adj)

[out]>>

0.7343070437613076 ********************************************************************

0.7278398724126984

#here rsuared is better fit to check accuracy of model insted of adjusted r sqaured

Step 5. Modeling and Prediction

i. RM, LSTAT , PTRATIO are Extensively correlated with Price hence we will make our model based on these features only.

x =pd.DataFrame(boston_df.loc[:,["RM","LSTAT","PTRATIO"]])

y=boston_df.loc[:,"Price"]

y=np.array(y).reshape(-1,1)

print(x.shape)

print(y.shape) [out]>> (506, 3)

(506, 1)

ii. Splitting the dataset into training and testing set

Here i have taken 80% of dataset as training set and size of testing set test_size=0.2 would be 20%. random_state=int it will seed the value means if we will take same dataset on different system with same train and test set size we will get same accuracy percentage.

x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.2,random_state=1)

print("shape of training set of X:",x_train.shape)

print("shape of training set of y:",y_train.shape)

print("shape of testing set of x:",x_test.shape)

print("shape of testing set of y:",y_test.shape) [out]>> shape of training set of X: (404, 3)

shape of training set of y: (404, 1)

shape of testing set of x: (102, 3)

shape of testing set of y: (102, 1)

iii. Training and Testing

We will use LinearRegression to train our training and testing set.

from sklearn.linear_model import LinearRegression

lrm=LinearRegression()

lrm.fit(x_train,y_train)



Training set:

y_train_pred=lrm.predict(X_train)

df1=pd.DataFrame({"Actual_train":y_train,"Predicted_train":y_train_pred})

df2=df.head()

df2

Figure 8

df2.plot(kind="bar")

plt.show()

figure 9

Test set:

y_test_pred=lrm.predict(x_test)

df3=pd.DataFrame({"Actual_test":y_test,"Predicted_test":y_test_pred})

df4=df3.head()

df4

Figure 10

Now we will plot the actual value and predicted value .After plotting we will see the variations between them.

df4.plot(kind="bar")

plt.show()

Figure 11

From above figure we can see there is not much variation between predicted value and actual value hence we can say our predicted model insures that it will work well.

iv. Model Evaluation

Model Evaluation estimates generalized accuracy for unseen data.It also ensures whether our model is overfit or not. overfitted model works well for test dataset but it does not give good predicton for real world dataset so if there would not be large difference between the accuracy of train and test set then we can say that our model is ready for deployment.

Training set:

from sklearn.metrics import r2_score

from sklearn import metrics

print("MSE:",metrics.mean_squared_error(y_train,y_train_pred))

print("MAE:",metrics.mean_absolute_error(y_train,y_train_pred))

print("RMSE:",np.sqrt(metrics.mean_squared_error(y_train,y_train_pred)))

print("R_squared:",r2_score(y_train,y_train_pred))

Figure 12

Test Set:

print("MSE:",metrics.mean_squared_error(y_test,y_test_pred))

print("MAE:",metrics.mean_absolute_error(y_test,y_test_pred))

print("RMSE:",np.sqrt(metrics.mean_squared_error(y_test,y_test_pred)))

print("R_squared:",r2_score(y_test,y_test_pred))

Figure 13

R Squared is nearer to 1 and there is not much difference between R squared value of Train set and Test set hence we can say model is not overfitted.

Conclusion

Drop a comment if you have any query and suggestions. Keep visiting for more machine learning content.