Multi-Linear Regression Using Python
By: Rafi Atha
Tags: Data Visualization, Python, Data Science, Linear Regression, Statistics

Model Validation

After building the model it is important for us to validate it’s performance. We can evaluate a model by looking at it’s coefficient of determination (R²), F-test, t-test, and also residuals. Before we continue we will rebuild our model using the statsmodel library with the OLS() function. Then we will print the model summary using the summary() function on the model. The model summary contains lots of important value we can use to evaluate our model.

Coefficient of Determination (R²)

Coefficient of determination is the portion of the total variation in the dependent variable that is explained by variation in the independent variable. R² scores are calculated as below:

In statsmodel we can obtain the R² value of our model by accesing the .rsquared attribute of the our model.

R2 score: 0.5214779360292285

R² range between 0 and 1, where R²=0 means there are no linear relationship between the variables and R²=1 shows a perfect linear relationship. In our case, we got R² score about 0.5214 which means 52.14% of our dependent variable can be explained using our independent variables.

F-Test (ANOVA)

F-test or ANOVA (Analysis of variance) in multi-linear regression can be used to determine whether our complex model perform better than a simpler model (e.g. model with only one independent variable). With F-test we can evaluate the significance of our model by calculating the probability of observing an F-statistic that is at least as high as the value that our model obtained. Similar to R² score, we can easily get the F-statistic and probability of said F-statistic by accessing the .fvalues and .f_pvalues attribute of our model as below.

F-statistic: 6.538606789020464

Probability of observing value at least as high as F-statistic: 0.01200637223318641

Because our f_pvalue is lower than 0.05 we can conclude that our model performs better than other simpler model.

T-test

The t-statistic is the coefficient divided by its standard error. The standard error is an estimate of the standard deviation of the coefficient, the amount it varies across cases. It can be thought of as a measure of the precision with which the regression coefficient is measured. Same as the F-test, the p-value show the probability of seeing a result as extreme as the one our model have. We can also get the p-value for all of our variables by calling the .pvalues attribute on the model.

const 0.019932

price 0.039788

advertising 0.014494

dtype: float64

Both of our independent variables, price and advertising, have p-value less than 0.05 which shows that there is sufficient evidence that price and advertising affects our pie sales.

Assumption Testing

Next, we will validate our model by doing residual analysis, below are the list of test or assumption we will do to check on our model validity:

Linearity

Normality

Multicollinearity

Autocorrelation

Homoscedasticity

Residual is the difference between the observed value and predicted value from our dataset. With statsmodel we can easily get the residual value of our model by simply accesing the .resid attribute of the model and then we can keep it in a new column called 'residual' in our df_pie dataframe.

Linearity

This assumes that there is a linear relationship between the independent variables and the dependent variable. In our case since we have multiple independent variables, we can do this by using a scatter plot to see our predicted values versus the actual values.

The scatter plots show residual point evenly spread around the diagonal line, so we can assume that there is linear relationship between our independent and dependent variables.

Normality

This assumes that the error terms of the model are normally distributed. We will examine the normality of the residuals by plotting it into histogram and looking at the p-value from the Anderson-Darling test for normality. We will use the normal_ad() function from statsmodel to calculate our p-value and then compare it to threshold of 0.05, if the p-value we get is higher than the threshold then we can assume that our residual is normally distributed.

p-value from the test Anderson-Darling test below 0.05 generally means non-normal: 0.6655438857701688

Residuals are normally distributed

From the code above we got our p-value of 0.6644 which can be considered normal because it’s above the 0.05 threshold. The histogram plot also show a normal distribution (despite it might be looking a little skewed because we only have 15 observation in our dataset). From both of those result we can assume that our residual are normally distributed.

Multicollinearity

This assumes that the predictors used in the regression are not correlated with each other. To identify if there are any correlation between our predictors we can calculate the Pearson correlation coefficient between each column in our data using the corr() function from Pandas dataframe. Then we can display it as a heatmap using heatmap() function from Seaborn.

Pearson correlation coefficient matrix of each variables:

pie_sales price advertising

pie_sales 1.000000 -0.443273 0.556320

price -0.443273 1.000000 0.030438

advertising 0.556320 0.030438 1.000000

The image shows that there are some positive relationship between advertising and pie_sales and a negative relationship between price and pie_sales. Both of this result support our resulting model from before. Most importanly, notice how the price and advertising have almost 0 correlation coefficient. This means both of our independent variable are not affecting each other and that there is no multicollinearity in our data.

Autocorrelation

Autocorrelation is correlation of the errors (residuals) over time. Used when data are collected over time to detect if autocorrelation is present. Autocorrelation exists if residuals in one time period are related to residuals in another period. We can detect autocorrelation by performing Durbin-Watson test to determine if either positive or negative correlation is present. In this step we will use the durbin_watson () function from statsmodel to calculate our Durbin-Watson score and then assess the value with the following condition:

If the Durbin-Watson score is less than 1.5 then there is a positive autocorrelation and the assumption is not satisfied

If the Durbin-Watson score is between 1.5 and 2.5 then there is no autocorrelation and the assumption is satisfied

If the Durbin-Watson score is more than 2.5 then there is a negative autocorrelation and the assumption is not satisfied

Durbin-Watson: 1.6831203020921253

Little to no autocorrelation



Assumption satisfied

Our model got a Durbin-Watson score of about 1.6831 which is between 1.5 and 2.5, so we can assume that there is no autocorrelation in our residual.

Homoscedasticity

This assumes homoscedasticity, which is the same variance within our error terms. Heteroscedasticity, the violation of homoscedasticity, occurs when we don’t have an even variance across the error terms. To detect homoscedasticity, we can plot our residual and see if the variance appears to be uniform.

Despite having only 15 data points our residual seems to have constant and uniform variance, so we can assume that it satisfied the homoscedasticity assumption.