Multiple Linear Regression in Python
By: Xuankhanh Nguyen
Tags: Programming, Machine Learning, Python, Artificial Intelligence, Data Science

A case study:

We have learned all we need to implement multiple linear regression. Now it’s time to see how it works on a dataset. I have learned so much by performing a multiple linear regression in Python. I hope you will learn a thing or two after reading my note.

I downloaded the World Happiness Report from Kaggle. You can search on Kaggle for competitions, datasets, and other solutions. In this example, we want to predict the happiness score based on multiple variables.

Features Analyzed

GDP : GDP per capita is a measure of a country’s economic output that accounts for its number of people.

: GDP per capita is a measure of a country’s economic output that accounts for its number of people. Support : Social support means having friends and other people, including family, turning to in times of need or crisis to give you a broader focus and positive self-image. Social support enhances the quality of life and provides a buffer against adverse life events.

: Social support means having friends and other people, including family, turning to in times of need or crisis to give you a broader focus and positive self-image. Social support enhances the quality of life and provides a buffer against adverse life events. Health : Healthy Life Expectancy is the average number of years that a newborn can expect to live in “full health” — in other words, not hampered by disabling illnesses or injuries.

: Healthy Life Expectancy is the average number of years that a newborn can expect to live in “full health” — in other words, not hampered by disabling illnesses or injuries. Freedom: Freedom of choice describes an individual’s opportunity and autonomy to perform an action selected from at least two available options, unconstrained by external parties.

Freedom of choice describes an individual’s opportunity and autonomy to perform an action selected from at least two available options, unconstrained by external parties. Generosity : The quality of being kind and generous.

: The quality of being kind and generous. Corruption: The Corruption Perceptions Index (CPI) is an index published annually by Transparency International since 1995, which ranks countries “by their perceived levels of public sector corruption, as determined by expert assessments and opinion surveys.

Import Modules, Read the Dataset and Define an Evaluation Table

To do some analysis, we need to set up our environment. First, we import some modules and read the data. The below output is the head of the data, but if you want to see more details, you might try removing # signs in front of the df_15.describe() and df_15.info()

# FOR NUMERICAL ANALYTICS

import numpy as np

# TO STORE AND PROCESS DATA IN DATAFRAME

import pandas as pd # BASIC VISUALIZATION PACKAGE

import matplotlib.pyplot as plt

# ADVANCED PLOTING

import seaborn as seabornInstance

# TRAIN TEST SPLIT

from sklearn.model_selection import train_test_split

# STATS

from sklearn import datasets, metrics

from sklearn.metrics import mean_squared_error

from sklearn.linear_model import LinearRegression

from sklearn.model_selection import cross_val_score #2015 data

df_15 = pd.read_csv('2015.csv')

#df_15.describe()

#df_15.info()

usecols = ['Rank','Country','Score','GDP','Support',

'Health','Freedom','Generosity','Corruption']

df_15.drop(['Region','Standard Error', 'Dystopia Residual'],axis=1,inplace=True)

df_15.columns = ['Country','Rank','Score','Support',

'GDP','Health',

'Freedom','Generosity','Corruption']

df_15['Year'] = 2015 #add year column

df_15.head()

output.

I only present the code for 2015 data as an example; you could do similar for other years.

Parts starting with Happiness , Whisker and the Dystopia.Residual are targets, just different targets. Dystopia Residual compares each countries scores to the theoretical unhappiest country in the world. Since the data from the years have a bit of a different naming convention, so we will abstract these to a familiar name.

target = ['Top','Top-Mid', 'Low-Mid', 'Low' ]

target_n = [4, 3, 2, 1] df_15["target"] = pd.qcut(df_15['Rank'], len(target), labels=target)

df_15["target_n"] = pd.qcut(df_15['Rank'], len(target), labels=target_n)

Combine all data file to final df

# APPENDING ALL TOGUETHER

finaldf = df_15.append([df_16,df_17,df_18,df_19])

# finaldf.dropna(inplace = True) #CHECKING FOR MISSING DATA

finaldf.isnull().any() # FILLING MISSING VALUES OF CORRUPTION PERCEPTION WITH ITS MEAN

finaldf.Corruption.fillna((finaldf.Corruption.mean()), inplace = True)

finaldf.head(10)

We can see the statistical detail of our dataset by using describe() function:

Further, we define an empty dataframe . This dataframe includes Root Mean Squared Error (RMSE) , R-squared , Adjusted R-squared , and mean of the R-squared values obtained by the k-Fold Cross-Validation , which are the essential metrics to compare different models. Having an R-squared value closer to one and smaller RMSE means a better fit. In the following sections, we will fill this dataframe with the results.

evaluation = pd.DataFrame({'Model':[],

'Details':[],

'Root Mean Squared Error (RMSE)': [],

'R-squared (training)': [],

'Adjusted R-squared (training)': [],

'R-squared (test)':[],

'Adjusted R-squared(test)':[],

'5-Fold Cross Validation':[]

})

Define a Function to Calculate the Adjusted R²

R-squared increases when the number of features increases. Because of this, sometimes, a more robust evaluator is preferred to compare the performance between different models. This evaluator is called adjusted R-squared.

The adjusted R-squared compensates for the addition of variables and only increases if the new predictor enhances the model above what would be obtained by probability. Conversely, it will decrease when a predictor improves the model less than what is predicted by chance. The definition of the adjusted R² is:

def adjustedR2(r2,n,k):

return r2-(k-1)/(n-k)*(1-r2)

Create a Simple Linear Regression Model

We want to predict happiness score, so our dependent variable here is score . I decided to use GPD as our independent variable, but if you're going to examine the relationship between the happiness score and another feature, you may prefer that feature.

train_data, test_data = train_test_split(finaldf, train_size = 0.8, random_state = 3)

lr = LinearRegression()

X_train = np.array(train_data['GDP'],

dtype = pd.Series).reshape(-1,1)

y_train = np.array(train_data['Score'], dtype = pd.Series)

lr.fit(X_train, y_train) X_test = np.array(test_data['GDP'],

dtype = pd.Series).reshape(-1,1)

y_test = np.array(test_data['Score'], dtype = pd.Series) pred = lr.predict(X_test)

#ROOT MEAN SQUARED ERROR

rmsesm = float(format(np.sqrt(metrics.mean_squared_error(y_test,pred)),'.3f'))

#R-SQUARED (TRAINING)

rtrsm = float(format(lr.score(X_train, y_train),'.3f'))

#R-SQUARED (TEST)

rtesm = float(format(lr.score(X_test, y_test),'.3f'))

cv = float(format(cross_val_score(lr,finaldf[['GDP']],finaldf['Score'],cv=5).mean(),'.3f')) print ("Average Score for Test Data: {:.3f}".format(y_test.mean()))

print('Intercept: {}'.format(lr.intercept_))

print('Coefficient: {}'.format(lr.coef_))

print('Happiness score = ',np.round(lr.intercept_,4),

'+',np.round(lr.coef_,4),'* GDP') r = evaluation.shape[0]

evaluation.loc[r] = ['Simple Linear Regression','-',rmsesm,rtrsm,'-',rtesm,'-',cv]

evaluation

By using these values and the below definition, we can estimate the happiness score manually. The equation we use for our estimations is called hypothesis function and defined as:

We also printed the intercept and coefficient for the simple linear regression.

Result

Since we have just two dimensions at the simple regression, it is easy to draw it. The below chart determines the result of the simple regression. It does not look like a perfect fit, but when we work with real-world datasets, having an ideal fit is not easy.

seabornInstance.set_style(style='whitegrid') plt.figure(figsize=(12,6))

plt.scatter(X_test,y_test,color='blue',label="Data", s = 12)

plt.plot(X_test,lr.predict(X_test),color="red",label="Predicted Regression Line")

plt.xlabel("GDP per Captita", fontsize=15)

plt.ylabel("Happiness Score", fontsize=15)

plt.xticks(fontsize=13)

plt.yticks(fontsize=13)

plt.legend() plt.gca().spines['right'].set_visible(False)

plt.gca().spines['top'].set_visible(False)

Create a Multiple Linear Regression Model

We used a simple linear regression and found a poor fit. To improve this model, we want to add more features. Now, it is time to create some complex models.

We determined features at first by looking at the previous sections and used them in our first multiple linear regression. As in the simple regression, we printed the coefficients which the model uses for the predictions. However, this time we must use the below definition for multiple linear regression:

We create a model with all features.

# MULTIPLE LINEAR REGRESSION 1

train_data_dm,test_data_dm = train_test_split(finaldf,train_size = 0.8,random_state=3) independent_var = ['GDP','Health','Freedom','Support','Generosity','Corruption']

complex_model_1 = LinearRegression()

complex_model_1.fit(train_data_dm[independent_var],train_data_dm['Score']) print('Intercept: {}'.format(complex_model_1.intercept_))

print('Coefficients: {}'.format(complex_model_1.coef_))

print('Happiness score = ',np.round(complex_model_1.intercept_,4),

'+',np.round(complex_model_1.coef_[0],4),'∗ Support',

'+',np.round(complex_model_1.coef_[1],4),'* GDP',

'+',np.round(complex_model_1.coef_[2],4),'* Health',

'+',np.round(complex_model_1.coef_[3],4),'* Freedom',

'+',np.round(complex_model_1.coef_[4],4),'* Generosity',

'+',np.round(complex_model_1.coef_[5],4),'* Corrption') pred = complex_model_1.predict(test_data_dm[independent_var])

rmsecm = float(format(np.sqrt(metrics.mean_squared_error(

test_data_dm['Score'],pred)),'.3f'))

rtrcm = float(format(complex_model_1.score(

train_data_dm[independent_var],

train_data_dm['Score']),'.3f'))

artrcm = float(format(adjustedR2(complex_model_1.score(

train_data_dm[independent_var],

train_data_dm['Score']),

train_data_dm.shape[0],

len(independent_var)),'.3f'))

rtecm = float(format(complex_model_1.score(

test_data_dm[independent_var],

test_data_dm['Score']),'.3f'))

artecm = float(format(adjustedR2(complex_model_1.score(

test_data_dm[independent_var],test_data['Score']),

test_data_dm.shape[0],

len(independent_var)),'.3f'))

cv = float(format(cross_val_score(complex_model_1,

finaldf[independent_var],

finaldf['Score'],cv=5).mean(),'.3f')) r = evaluation.shape[0]

evaluation.loc[r] = ['Multiple Linear Regression-1','selected features',rmsecm,rtrcm,artrcm,rtecm,artecm,cv]

evaluation.sort_values(by = '5-Fold Cross Validation', ascending=False)

The population regression line for n independent variables x(n) is defined to be Happiness score = 2.0977 + 1.1126 ∗ Support + 0.9613 * GDP + 1.3852 * Health + 0.7854 * Freedom + 0.2824 * Generosity + 1.2498 * Corrption . This line describes how the happiness score changes with the independent variables ( Support , GDP , Health , Freedom , Generosity , and Corruption )

Check Out the Correlation Among Independent Variables

mask = np.zeros_like(finaldf[usecols].corr(), dtype=np.bool)

mask[np.triu_indices_from(mask)] = True f, ax = plt.subplots(figsize=(16, 12))

plt.title('Pearson Correlation Matrix',fontsize=25) seabornInstance.heatmap(finaldf[usecols].corr(),

linewidths=0.25,vmax=0.7,square=True,cmap="Blues",

linecolor='w',annot=True,annot_kws={"size":8},mask=mask,cbar_kws={"shrink": .9});

It looks like GDP , Health , and Support are strongly correlated with the Happiness score . Freedom and correlates quite well with the, however, Freedom connects quite well with all data. Corruption still has a mediocre correlation with the Happiness score.

Multiple Linear Regression using Gradient Descent

We can implement various linear regression using gradient descent.