Multiple Linear Regression
By: Heena Girdher
Tags: Regression, Multiple Regression, Data Science, R, Machine Learning

Photo by Glenn Carstens-Peters on Unsplash

In this blog article, we will be covering the following topics:-

What is Multiple Linear Regression? Assumptions of Multiple Linear Regression. Practical Implementation of Multiple Linear Regression.

1. What is Multiple Linear Regression?

Multiple Linear Regression or Multiple Regression is used to model the linear relationship between a dependent variable and a set of independent variables. It is used when the output of a variable called dependent variable depends on a number of other variables called independent variables. In multiple linear regression, the value of y is discrete or numerical.

Example 1

The price of a Mobile Phone depends on the brand name, storage, camera, and age. The price is a dependent variable and brand name, storage, camera, and age are independent variables.

Example 2

The premium amount for medical insurance policy depends upon age, nature of work, previous medical history, and tenure of the insurance policy. Premium amount is a dependent variable and age, nature of work, medical history and tenure of an insurance policy are independent variables.

The equation for the multiple linear regression model is –

Y=B0+B1X1+B2X2+…+BnXn

Where Y is the dependent variable,

X1, X2…Xn are independent variables,

And B0, B1, B2, …, Bn are regression coefficients.

2. Assumptions

Multiple Linear Regression is based on the following assumptions:-

1. The dependent and independent variables are highly correlated and there exists a linear relationship between the dependent variable and independent variables.

2. The independent variables are not correlated with each other.

3. The residuals should be normally distributed which means that the errors do not contain important information.

4. The residuals should not contain any pattern.

The coefficient of determination (R²) indicates how good the regression model is. The value of R² ranges from 0 to 1. The closer the value of R² to 1, the better will be the regression model.

3. Practical Implementation of Multiple Linear Regression in R

To perform the experiment, I have downloaded a dataset ‘insurance.csv’ (https://www.kaggle.com/noordeen/insurance-premium-prediction) from Kaggle. The dataset contains the following features- age, bmi, children, sex, smoker, region, and expenses. The purpose of this experiment is to plot a multiple linear regression model based on several features such as age, bmi, children, sex, smoker, and region against their existing medical expenses to be used for predicting future medical expenses of individuals.

Install and load the required packages

First, open R Studio and then, install and load the required packages into R Studio.

Load the data

Load the ‘insurance.csv’ file into R Studio.

After executing the above code, we get the following output-

Convert the categorical columns to numerical form

To apply multiple linear regression, there is a need to convert categorical variables into numerical form. Here, we have changed the value of categorical variables to numerical form by using the revalue function.

After executing the above code, we get the following output-

I. Exploratory Data Analysis

The first step in Multiple Linear Regression is Exploratory Data Analysis. In this step, we check the Scatter Plot and Correlation Matrix to find the relationship between the variables.

Scatterplot

We want to look at pairwise scatterplots of the variables so that we can determine if any pairs of predictors appear highly correlated.

After executing the above code, we get the following output-

Print Correlation Matrix

The correlation matrix of the variables is plotted to check the correlation between variables.

After executing the above code, we get the following output-

II. Model Building

The second step in Multiple Linear Regression is Model Building.

After executing the above code, we get the following output-

The Summary shows that age, bmi, children, and smoker are significant variables whereas sex and region are insignificant variables. The value of R² is 0.7498.

III. Check Multicollinearity Problem

The third step in Multiple Linear Regression is checking Multicollinearity problems. If a Multicollinearity problem exists, then the duplicate variables are removed from the data. We will use the variance inflation factor (VIF) to check the Multicollinearity problem.

After executing the above code, we get the following output-

Since the VIF value of all the variables is less than 20, the Multicollinearity problem does not exist.

IV. Model Validation

After executing the above code, we get the following output-

The first plot is the residual plot. It indicates that there is a pattern in the Residuals Vs Fitted plot which means the errors contain important information.

The next plot is QQ-plot. Since most of the points in the QQ-plot do not fall on the line which indicates that the residuals do not follow a normal distribution.

The third plot created is the scale-location plot. This plot is similar to the residual plot but uses the square root of the standardized residuals instead of the residuals themselves. This makes trends in residuals more evident and, from this plot, we can see that there are patterns in our residuals.

The fourth plot is the leverage plot. This plot graphs the standardized residuals against their leverage. It also includes Cook’s distance boundaries. Any point outside of those boundaries would be an outlier in the x-direction. From the plot, we can see that there are outliers in the data.

Residual Vs Regressors Plot

After executing the above code, we get the following output-

Here, the independent variables are plotted against their errors. From the plot, we can see that age and bmi are showing a pattern or the linear relationship does not exist for age and bmi. Therefore, transformations are applied to make the relationship linear. Various transformations can be applied to data like quadratic, polynomial, logarithmic and exponential, etc.

V. Model deletion diagnostics

The outliers or influencers can be identified from the data using Cook’s distance, Hat value, and Bonferroni correction.

After executing the above code, we get the following output-

Now, we will remove outliers and apply transformations to the model to get a better value of R².

####Iteration 1

#Quadratic model

After executing the above code, we get the following output-

# Poly Model

After executing the above code, we get the following output-

# Reciprocal Model

After executing the above code, we get the following output-

# Exponential model

After executing the above code, we get the following output-

We have applied various transformations on the model and we can see that the Exponential model gives a better value of R² which is 0.7616.

#check the model for influential observations

After executing the above code, we get the following output-

After executing the above code, we get the following output-

#check the model for influential observations

After executing the above code, we get the following output-

After executing the above code, we get the following output-

By removing the influential observations, the accuracy of the exponential model has been increased from 0.7616 to 0.7821. In this way, by removing the outliers from the data, the accuracy of the model can be improved.