Linear Regression Model with Python | by Idil Ismiguzel | Towards Data Science

Regression models are widely used machine learning tools allowing us to make predictions from data by learning the relationship between features and continuous-valued outcomes. Checking model assumptions and understanding whether they are satisfied or not is as important as checking the accuracy and goodness of the model.
In this article you will learn:
How to build a linear regression modelHow to assess the model by prediction accuracy and R-squaredHow to check model diagnostics by testing linearity, normality, homoscedasticity, autocorrelation, and multicollinearityHow to detect outliers with Cook’s distanceHow to find the influence of each observation by studentized residuals and H leverage.
How to build a linear regression model
How to assess the model by prediction accuracy and R-squared
How to check model diagnostics by testing linearity, normality, homoscedasticity, autocorrelation, and multicollinearity
How to detect outliers with Cook’s distance
How to find the influence of each observation by studentized residuals and H leverage.
While following the article, I encourage you to check out the Jupyter Notebook on my GitHub for the full analysis and code.
github.com
Regression analysis is used to model the relationship between a single dependent variable Y (aka response, target, or outcome) and one or more independent variables X (aka predictor or feature).
When we have one predictor it is “simple” linear regression and when we have more than one predictors it is “multiple” linear regression.
In general, data does not fall exactly on a linear line, so the regression equation should include an error term(ε).
The fitted (predicted) values typically denoted by Y-hat, and β-hat coefficients are chosen by the model as they minimize the squared distance (aka residuals) between the actual Y value and the best fit (Y-hat) minimizing the following expression:
The method of minimizing the sum of the squared residuals is called Ordinary Least Squares (OLS) regression.
We will be building the multiple linear regression model on the Boston housing dataset from the late 1970s. Data consists of a total of 506 cases with 14 attributes. Let’s have a look at it!
names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV']data = pd.read_csv("./housing.csv", delim_whitespace=True, names=names)data.head()
We will be predicting house prices (“MEDV”) by using all the other variables.
predictors = ["CRIM", "ZN", "INDUS", "CHAS", "NOX", "RM", "AGE", "DIS", "RAD", "TAX", "PTRATIO", "B", "LSTAT"]outcome = "MEDV"-> Create X and y datasetsX = data[predictors]y = data["MEDV"]
We will use the Statsmodels library for linear regression. (Scikit-learn can also be used as an alternative but here I preferred statsmodels to reach a more detailed analysis of the regression model). Having said that, we will still be using Scikit-learn for train-test split. Details below!
-> Because we are using statsmodels, we need to add the constant term to the X value.import statsmodels.api as smX = sm.add_constant(X)
We will split our dataset into train and test sets (80% for training, and 20% for testing). The regression model will learn from training data where the output is known, and later we will generalize the model on the test set. We will predict the test set’s y values output and comparing these predictions with the actual values. This is done to detect overfitting or underfitting problems.
-> Train-test splitfrom sklearn.model_selection import train_test_splittrain_X, test_X, train_y, test_y = train_test_split(X, y, train_size = 0.8, random_state = 42)-> Linear regression model model = sm.OLS(train_y, train_X)model = model.fit()print(model.summary2())
No. Observations is 404 (80% of the overall dataset)
R-squared (goodness of the model from 0 to 1) is 0.75
p-value >0.05 for some predictors: “ZN”, “INDUS”, “AGE”, “TAX” meaning that we would consider removing these variables from the predictors list! It is because p-values determine whether the relationships that we observe in this sample also exist in the larger population.
We will use model.predict on the test_X and compare the predictions with the actual test_y values.
predictions = model.predict(test_X)df_results = pd.DataFrame({‘Actual’: test_y, ‘Predicted’: predictions})
One of the most important metrics is root mean squared error; the square root of the average squared error in the predicted y values (y-hat). It measures the overall accuracy of the model.
Another important metric is R-squared ranging from 0 to 1; measures the proportion of variation in the data that is accounted in the regression model. It tells how well the model fits the data and R-squared=1 indicates that the regression predictions perfectly fit the data.
Let’s find RMSE and R-squared for the predictions.
from sklearn.metrics import r2_score, mean_squared_errorRMSE = np.sqrt(mean_squared_error(test_y, predictions))r2 = r2_score(test_y, predictions)print(RMSE, r2)
RMSE=4.92
R-squared = 0.66
As we see our model performance dropped from 0.75 (on training data) to 0.66 (on test data), and we are expecting to be 4.92 far off on our next predictions using this model.
Before we built a linear regression model, we make the following assumptions:
Linearity: The relationship between X and the mean of Y is linear.Normality: The residuals follow a normal distribution and the expected mean of the residuals is zero.Homoscedasticity: The variance of residual is constant for all values of X.No autocorrelation: Residuals are independent of each other.No multicollinearity: There is no high correlation between predictor variables.
Linearity: The relationship between X and the mean of Y is linear.
Normality: The residuals follow a normal distribution and the expected mean of the residuals is zero.
Homoscedasticity: The variance of residual is constant for all values of X.
No autocorrelation: Residuals are independent of each other.
No multicollinearity: There is no high correlation between predictor variables.
Before being confused about the model performance, it is crucial to check whether the assumptions are strongly satisfied or not. It could be difficult to fulfill all the assumptions perfectly with the real-world data, however, it is crucial to understand how far our model being interpreted correctly.
We assumed that there is a linear relationship between the predictors and the outcome, but this may not be correct. When we do modeling, we are trying to fit our data to a function that explains our data, but linear regression might not be the fitting estimator in this case.
We will test linearity with a scatter plot to see predicted values versus the actual values. Ideally, the data points should lie around the diagonal line on the plot.
-> Plot the actual vs predicted resultssns.lmplot(x='Actual', y='Predicted', data=df_results, fit_reg=False)-> Plot the diagonal lined_line= np.arange(df_results.min().min(), df_results.max().max())plt.plot(d_line, d_line, color='red', linestyle='--')plt.show()
In the Boston dataset, we can see that there is not a perfect relationship. On the lower and higher-end values, our predictions are biased.
We assumed that the error terms (residuals) of the model are normally distributed. This assumption might be violated due to the linearity assumption not been hold or due to outliers in the dataset.
We will test normality by checking with the quantile-quantile plot, Q-Q plot.
from statsmodels.graphics.gofplots import qqplotfig=qqplot(model.resid_pearson,line=’45',fit=’True’)plt.xlabel(“Theoretical quantiles”)plt.ylabel(“Sample quantiles”)plt.show()
The ideal match on the left shows data distribution follows the red line from the bottom left to the top right. In Boston dataset, we can see that our model biasing towards under-estimating.
You can go further and plot a histogram of residuals or perform normality tests such as Anderson-Darling or Shapiro-Wilk Test.
We assumed that the variance of error terms (residuals) is constant for all values of X. However, heteroscedasticity, the violation of homoscedasticity happens when we do not have the same variance across the error terms.
This violation can happen when the model gives different weights to the subsets of the data.
We will test homoscedasticity by plotting the residuals and checking if the variance is uniform.
fig, ax = plt.subplots(figsize=(5, 5))sns.regplot(model.fittedvalues,model.resid, scatter_kws={'alpha': 0.25}, line_kws={'color': 'C2', 'lw': 2}, ax=ax)ax.set_xlabel('predicted')ax.set_ylabel('residuals')plt.tight_layout()plt.show()
The ideal plot on the left shows a uniform distribution of the residuals however, we can’t say the same for Boston Dataset. We need to be careful about heteroscedasticity, which indicates that the prediction errors are different for a different range of predicted values meaning that our model could be incomplete.
Autocorrelation occurs when the residuals are not independent of each other, in other words, when the value of y_t is not independent of y_t-1. This is generally the case for time series data such as stock prices. To learn more on this you can check my article about time series forecasting.
towardsdatascience.com
We will perform the Durbin-Watson test to determine if the no autocorrelation assumption holds.
from statsmodels.stats.stattools import durbin_watsondurbinWatson = durbin_watson(model.resid)print(durbinWatson)
>>>Durbin-Watson =2.11
The Durbin-Watson statistic ranges between 0 and 4. A value of 2.0 means that there is no autocorrelation. Values between 0 and 2 indicate positive and values between 2 and 4 indicate negative autocorrelation. In our case, Durbin-Watson statistic is very close to 2.0 therefore we can say that no autocorrelation assumption is not violated.
We assumed that the predictor variables in the regression are not correlated with each other. Having this assumption violated does not make our model unusable but it is still important to understand and get rid of some unnecessary predictors if they are highly correlated with others.
We will plot a heatmap to see the correlation between predictors visually and we will also calculate the variance inflation factor (VIF), which measures the multicollinearity among the independent variables.
mask = np.zeros_like(train_X.drop("const", axis=1).corr())mask[np.triu_indices_from(mask)] = Truesns.heatmap(train_X.drop("const", axis=1).corr().round(2), annot=True, mask=mask, cmap="cividis")
As we can see between some variables we have a high (negative/positive) correlation in Boston Dataset.
And we calculate VIF:
from statsmodels.stats.outliers_influence import variance_inflation_factor as viffor i in range(len(X.columns)):    v=vif(np.matrix(X),i)    print("Variance inflation factor for {}: {}".format(X.columns[i],round(v,2)))
Variance inflation factor for CRIM: 1.79Variance inflation factor for ZN: 2.3Variance inflation factor for INDUS: 3.99Variance inflation factor for CHAS: 1.07Variance inflation factor for NOX: 4.39Variance inflation factor for RM: 1.93Variance inflation factor for AGE: 3.1Variance inflation factor for DIS: 3.96Variance inflation factor for RAD: 7.48Variance inflation factor for TAX: 9.01Variance inflation factor for PTRATIO: 1.8Variance inflation factor for B: 1.35Variance inflation factor for LSTAT: 2.94
A VIF of 1 indicates two variables are not correlated, a VIF between 1 and 5 indicates a moderate correlation, and a VIF above 5 indicates a high correlation.
As result shows we have 2 high correlated and 5 moderately correlated variables. It is better if we get rid of some of them.
Outlier detection using Cook’s distance plot
Cook’s distance determines the effect of deletion of a given observation from the dataset. We can leverage Cook’s distance while examining if an observation is a potential outlier or an influential variable. Here is how to plot Cook’s distance.
from statsmodels.stats.outliers_influence import OLSInfluence as influenceinf=influence(model)(i, d) = inf.cooks_distanceplt.title("Cook's distance plot")plt.stem(np.arange(len(i)), i, markerfmt=",")plt.show()
From Cook’s plot, we can understand which are the observations we need to pay more attention to and decide whether to drop them or not. (As a rule, the observation has a high influence if the Cook’s distance is greater than 4/N-k-1 ( N=number of observations, k= number of predictors, yellow horizontal line in the plot)
Influence plots show the studentized residuals versus the leverage of each observation as measured by the hat matrix.
from statsmodels.stats.outliers_influence import OLSInfluence influence = OLSInfluence(model)fig, ax = plt.subplots()ax.axhline(-2.5, linestyle='-', color='C1')ax.axhline(2.5, linestyle='-', color='C1')ax.scatter(influence.hat_matrix_diag, influence.resid_studentized_internal,  s=1000 * np.sqrt(influence.cooks_distance[0]), alpha=0.5)ax.set_xlabel(‘H Leverage’)ax.set_ylabel(‘Studentized Residuals’)ax.set_title(“BOSTON DATASET \nInfluence Plot”)plt.tight_layout()plt.show()
In contrast to the ideal plot, in Boston Dataset we can see that some data points have low leverage but large residuals. These highly influential data points need a delicate evaluation before including in the model.
In this analysis, we found out that some of the assumptions we made before building a linear regression model have been violated and caused significant problems in the model performance and outcome predictions. Even though meeting with the assumptions with real-world data sounds a bit superficial, we still need to check and know the flaws of our model and if a possible fix or improve them, before making important decisions based on the outputs.
If you have outcomes like here we have with the Boston dataset, I would suggest you to try modeling with Random Forest Regressor or XGBoost to understand if you can improve model performance. You can enrich with principal component analysis-PCA for feature extraction to drop the least important predictors while still keeping the most valuable parts of all the variables. Moreover, you can also perform K-fold Cross Validation(CV) by dividing entire dataset into k folds and being sure that each fold is used as a testing set.
I hope you enjoyed reading the article and find it useful for your analyses!
If you liked this article, you can read my other articles here and follow me on Medium. Let me know if you have any questions or suggestions.✨
Enjoy this article? Become a member for more!