Linear Regression Explained. A High Level Overview of Linear... | by Jason Wong | Towards Data Science

Regression analysis is a statistical methodology that allows us to determine the strength and relationship of two variables. Regression is not limited to two variables, we could have 2 or more variables showing a relationship. The results from the regression help in predicting an unknown value depending on the relationship with the predicting variables. For example, someone’s height and weight usually have a relationship. Generally, taller people tend to weigh more. We could use regression analysis to help predict the weight of an individual, given their height.
When there is a single input variable, the regression is referred to as Simple Linear Regression. We use the single variable (independent) to model a linear relationship with the target variable (dependent). We do this by fitting a model to describe the relationship. If there is more than predicting variable, the regression is referred to as Multiple Linear Regression.
You may have heard of Ordinary Least Squares Regression. When we are attempting to find the “best fit line”, the regression model can sometimes be referred to as Ordinary Least Squares Regression. This just means that we’re using the smallest sum of squared errors. The error is the difference between the predicted y value subtracted from the actual y value. The difference is squared so there is an absolute difference, and summed.
error = y_actual - y_predicted
Take a look at the graph to the left. When using the data points (green dots) to draw a regression line, we’re actually working with estimations. Our goal here is to find the line that best describes the data. When working with estimations, we can make use of the “hat” notation (^). The formula for drawing the “best fit line” when working with estimations will be the same as the straight line formula, but with hat notation.
Straight Line Formula: y = mx + c        - Where {m} is the slope and {c} is the intercept
Straight Line Formula with estimations:
ŷ =m̂ x+ĉ
Or
ŷ =β̂ 0+β̂ 1x
We’re just estimating a proper intercept and slope. When we have drawn the “best fit line” we are ready to make some predictions. However, since our prediction is based on the parameter values we estimate, when we predict new y values given x, we will have error or vertical offset (blue lines). This error is denoted as |ŷ −y| where ŷ is on our regression line and y is the actual observed value (green dot).
When performing simple linear regression, the four main components are:
Dependent Variable — Target variable / will be estimated and predicted
Independent Variable — Predictor variable / used to estimate and predict
Slope — Angle of the line / denoted as m or β1
Intercept — Where function crosses the y-axis / denoted as c or β0
The last two, slope and intercept, are the coefficients/parameters of a linear regression model, so when we calculate the regression model, we’re just calculating these two. In the end, we’re trying to find the best-fit line describing the data, out of an infinite number of lines. To find the slope of a line, we can choose a random part of the line, and divide the change in x by the change in y.
Δy — Change in y
Δx — Change in x
We need to calculate some statistical measures before calculating the “best fit line”:
The formula above is multiplying the slope by the mean of x and subtracting that value from the mean of y.
Take another look at the plot above, we can see that:
Δy = 4.66
Δx = 8.64
4.66(change in y) / 8.64(change in x) = 0.54 (slope)
c (Intercept) = 6.38, where the line intersects with the y-axis
m (Slope) = 0.54
Δy = ((np.mean(X) * np.mean(Y)) - np.mean(X*Y))Δx = ((np.mean(X)**2) - np.mean(X*Y))m = Δy / Δx
c = np.mean(Y) - m * np.mean(X)
If we have a new value (x), we can calculate the prediction (y) with the data we already have.
new_x_value = 9y_predicted = (m * new_x_value) + cy_predicted#Output:11.233796296296294
We know that the goal of linear regression is to find the “best fit line” that describes the data. However, we saw above that the line won’t fully represent the relationship between variables. There will also be some error (y_actual - y_predicted). The R-Squared measure can be used to determine a how well a model fits the data. This measure is also known as the Coefficient of Determination.
R-Squared takes a simple model which uses the mean of the actual_y values to predict new_y values. The model will always predict this mean as the new_y regardless of the x value. This simple model is compared to a fit regression model to determine how well its fit.
The formula above can be read as:
Regression, being a parametric technique, relies on parameters learned from the data. This also means that the data must fulfill certain assumptions. These assumptions are necessary for obtaining reliable results. If the assumptions aren’t fulfilled, our predictions may be biased.
The plots used below were created using the Advertising dataset from Kaggle.
There is a linear relationship between the dependent variable (y) and the independent variable (x). We can check for linearity by creating a scatter plot.
The plot above is of TV advertisement spending and sales. This plot shows that there is a linear relationship between the two variables. We can interpret the plot to the right by saying “As TV advertisement spending increases, so do the sales. If you think you’re violating this assumption, try log-scaling your data.
This assumption states that the residuals (difference between actual_y and predicted_y) of a model are normally distributed. This assumption can be checked by created histograms or Q-Q-Plots.
Q-Q-Plots (quantile-quantile-plots) are scatterplots of two sets of quantiles plotted against each other.
To check the normality assumption using qq-plots, we’re looking for a pretty straight line. It is worth noting that this is only a visual check. Another method of checking the normality assumption is the Jarque-Bera (JB) test.
The plot above is a Q-Q-Plot of TV advertisement spending and sales. The “straight line” indicates that the residuals of the model are normally distributed.
Multicollinearity describes the correlation between the predictor variables. This assumption states that the predictor variables are independent. We can check this assumption by creating pair plots and/or heat maps. Another method would be to calculate the Variance Inflation Factor (VIF).
The variance inflation factor is a measure for the increase of the variance of the parameter estimates if an additional variable, given by exog_idx is added to the linear regression. It is a measure for multicollinearity of the design matrix, exog.
— Statsmodels User Guide
#Pairplotsns.pairplot(sales, vars = ['TV', 'radio', 'newspaper']);
Looking at the pairplot above, we can see that out of the three features in the “sales” dataset, there isn’t a high correlation between them.
#Heatmap (correlation matrix)sns.heatmap(sales.drop('sales', axis=1).corr(), annot=True, cmap='plasma');
Looking at the heatmap above, we can see the actual correlation coefficients if we set the annot= parameter to True. Radio and newspaper seem to show a correlation coefficient of 0.35 which tells us the features are not highly correlated.
#Variance Inflation Factor (VIF)from statsmodels.stats.outliers_influence import variance_inflation_factorr = sales[["newspaper", "radio"]].valuesvif_df = pd.DataFrame()vif_df["VIF"] = [variance_inflation_factor(r, i) for i in range(2)]vif_df["feature"] = ["newspaper", "radio"]vif_df
In general, a VIF value of 5 is too high. Looking at the dataframe to the left we see little multicollinearity between the newspaper and radio features.
What would it mean if we did have features showing multicollinearity? For example, let’s say that the newspaper and radio features show multicollinearity. This would make it difficult for us to separate the effects of just newspaper on sales.
Homoscedasticity refers to the variability of the dependent variable being equal across the independent variable values. We can check this assumption by creating a scatterplot of the model predictions and residuals, we’re looking for the residuals to equal across the regression line. We could also use a significance test such as the Breusch-Pagan test.
sns.scatterplot(fsm.predict(), fsm.resid);
Looking at the plot above, we can see the values do not look to be forming a pattern on the right side but are forming a pattern on the left side. This plot shows that the error increases with the predicted values, so it is heteroscedastic.
#Breusch-Pagan testlm, lm_p_value, fvalue, f_p_value = het_breuschpagan(fsm_resids, sales[["TV"]])print("F-statistic p-value:", f_p_value)
The Breusch-Pagan test returning a p-value this low tells us we can reject the null hypothesis (homoscedasticity), and therefore, we’re violating the Homoscedasticity assumption.
Autocorrelation refers to the model residuals not being independent. If there were to be a correlation in the error terms, our model’s accuracy would decrease. This assumption can be checked using the Durbin-Watson test or creating an error plot. For the Durbin-Watson (DB) test, we’re looking for a value between 1.5–2.5. A few things to know regarding the DB test:
2: No Autocorrelation
0–1.9: Positive Autocorrelation
2.1–4: Negative Autocorrelation
#Durbin-Watson test in OLS summaryfsm.summary()
Looking at the OLS Summary, we can see the Durbin-Watson score is 1.935, this score tells us there is no correlation between the model residuals.
Now that we’ve checked the assumptions, let’s fit a linear regression model and evaluate the summary table.
Importing necessary libraries and data
import pandas as pdimport numpy as npfrom matplotlib import pyplot as plt%matplotlib inlineimport seaborn as snsfrom scipy import statsfrom statsmodels.formula.api import olsfrom statsmodels.stats.diagnostic import het_breuschpaganfrom statsmodels.stats.outliers_influence import variance_inflation_factorfrom sklearn.linear_model import LinearRegression
sales = pd.read_csv('data/Advertising.csv', index_col=0)sales = sales[['TV', 'sales']]sales.head()
The sales feature is our target (dependent) and TV is our predictor (independent).
Creating formula for OLS model
formula = 'sales~TV'
Fitting the model
model = ols(formula=formula, data=sales).fit()
Viewing the model summary
model.summary()
The left part of the top table provides information on the data and model
The right part of the top table provides information on how well the model is fit
The middle table is a coefficient report
The bottom table provides information on residuals, autocorrelation, and multicollinearity
R-Squared: Percent of variance explained by the model.
Adj. R-Squared: R-Squared where additional independent variables are penalized
F-statistic: Significance of fit
Prob (F-statistic): Probability of seeing F-statistic from a sample
Log-likelihood: Log of the likelihood function
AIC: Akaike Information Criterion, penalizes model when more independent variables are added.
BIC: Bayesian Information Criterion, similar to AIC but with higher penalties
coef: Estimated coefficient value
std err: Standard error of the coefficient estimate
t: Measure of statistical significance for coefficient
P>|t|: Probability value that the coefficient is equal to 0
[0.025 0.975]: Lower and upper halves of 95% confidence interval
Omnibus: Omnibus D’Angostino’s test, statistical test for skewness and kurtosis
Prob(Omnibus): Omnibus statistic as a probability
Skew: Measure of data mean symmetry
Kurtosis: Measure of shape of the distribution
Durbin-Watson: Test for autocorrelation
Jarque-Bera (JB): Test for skewness & kurtosis
Prob (JB): Jarque-Bera statistic as a probability
Cond. No.: Test for multicollinearity
There you have it, a breakdown of linear regression analysis. Regression analysis is one of the first modeling techniques to learn as a data scientist. It can helpful when forecasting continuous values, e.g., sales, temperature. There are quite a few formulas to learn but they’re necessary to understand what’s happening “under the hood” when we run linear regression models. As you saw above there are many ways to check the assumptions of linear regression, hopefully you now have a better understanding of them. Thanks so much for taking the time to check out this post!
Bevans, R. (2020, October 26). Simple Linear Regression: An Easy Introduction & Examples. Retrieved from https://www.scribbr.com/statistics/simple-linear-regression/
Bevans, R. (2020, October 26). Multiple Linear Regression: A Quick and Simple Guide. Retrieved from https://www.scribbr.com/statistics/multiple-linear-regression/
International Encyclopedia of the Social Sciences. . Encyclopedia.com. 16 Oct. 2020 . (2020, November 27). Retrieved from https://www.encyclopedia.com/social-sciences/applied-and-social-sciences-magazines/ordinary-least-squares-regression
Jcf2d, W. B. (n.d.). University of Virginia Library Research Data Services Sciences. Retrieved from https://data.library.virginia.edu/understanding-q-q-plots/
Stephanie. (2020, September 16). Q Q Plots: Simple Definition & Example. Retrieved from https://www.statisticshowto.com/q-q-plots/
Assumptions of Linear Regression. (2020, June 22). Retrieved from https://www.statisticssolutions.com/assumptions-of-linear-regression/
McCarty, K. (2018, June 18). Interpreting Results from Linear Regression — Is the data appropriate? Retrieved from https://www.accelebrate.com/blog/interpreting-results-from-linear-regression-is-the-data-appropriate
Ordinary Least Squares regression (OLS). (n.d.). Retrieved from https://www.xlstat.com/en/solutions/features/ordinary-least-squares-regression-ols
Breusch pagan test — ols_test_breusch_pagan. (n.d.). Retrieved from https://olsrr.rsquaredacademy.com/reference/ols_test_breusch_pagan.html
Introduction. (n.d.). Retrieved from https://www.statsmodels.org/stable/index.html
Dey, S. (2020, April 01). Advertising.csv. Retrieved from https://www.kaggle.com/bumba5341/advertisingcsv