Linear Regression Models and Influential Points | by Francis Adrian Viernes | Towards Data Science

Linear regression models play a huge role in the analytics and decision-making process of many companies, owing in part to their ease of use and interpretability.
There are instances, however, that the presence of certain data points affects the predictive power of such models. These data points are known as influential points.
We note in the previous paragraph that influential data points affect the predictive power of linear regression models. And influential data points do so, by greatly influencing the regression coefficient/s.
It is easy to mistake these points with “outliers”, however, they have different definitions. Not all outliers are considered influential points. In fact, in some cases, the presence of outliers, although unusual, may not change the regression line.
For example, if you have the data points: (1,1), (5,5), (6,6), (3,3), and (500,500), you can consider the last point as an outlier, but the regression life remains unchanged.
Instead, we should breakdown these extreme values into extreme y-values (high residuals/outliers) and extreme x-values (high leverage). In some cases, the observation may have both high residuals and high leverage.
An outlier is an observation with extreme y-values. Because the extreme values occur in the dependent or target variable, these observations have high residuals.
Leverage is a measure of how far the value of a predictor variable (e.g. independent or usually the x-variable) from the mean of that variable.
An observation is said to have high leverage if the value of the predictor variable is unusual and far from the rest.
An observation can have a high residual and high leverage and may or may not be an influential point.
To have a better grasp of the concept, let us try to practice on the California Housing Kaggle dataset. From my experience, real estate datasets are prone to significant outlier data and would make for a good exercise.
#Preliminariesimport pandas as pdimport numpy as npimport itertoolsfrom itertools import chain, combinationsimport statsmodels.api as smfrom statsmodels.formula.api import olsfrom statsmodels.graphics.regressionplots import *import matplotlib.pyplot as pltimport seaborn as snssns.set_theme()from sklearn.model_selection import train_test_splitimport mathimport time
Load the dataset:
california = pd.read_csv('data/housing.csv')california.head()
Preprocessing for ordinary least squares regression:
#Dummify categorical variablescalifornia = pd.get_dummies(california)#Remove observations that are NaN as these cannot be processed by our package#Remember to reset the index - as you have removed some observations and the influence diagnostics dataframe would not use the original dataset's indexcalifornia = california[np.isfinite(california).all(1)].reset_index(drop="True")cols = california.columns.to_list()#Targettarget= california.median_house_value.values#Featuresfeatures = [col for col in cols if col not in ['longitude', 'latitude','median_house_value']] #latitude and longitude are not good predictor variables (they are not meaningful as float variables)
Splitting the datasets into train and test sets:
X_train, X_test, Y_train, Y_test = train_test_split(X, target, test_size = 0.20, random_state = 5)
Run our initial OLS:
model_1 = sm.OLS(Y_train, sm.add_constant(X_train)).fit()print_model_1 = model_1.summary()print(print_model_1)
Influential points are generally identified either through visual means or through statistical diagnostics.
Conveniently, the get_influence method of the statsmodel package generates a table with influence diagnostics that we can use to determine these influential points.
influence = model_1.get_influence()inf_sum = influence.summary_frame()print(inf_sum.head())student_resid = influence.resid_studentized_external(cooks, p) = influence.cooks_distance(dffits, p) = influence.dffitsleverage = influence.hat_matrix_diagprint ('\n')print ('Leverage vs. Studentized Residuals')sns.regplot(leverage, model_1.resid_pearson,  fit_reg=False)plt.title('Leverage vs. Studentized Residuals')plt.xlabel('Leverage')plt.ylabel('Studentized Residuals')
As we have identified outliers as having high residuals, we can view this using the summary table we have generated using the get_influence().
Quick note: A studentized residual is a quotient resulting from the division of the observation’s residual over its estimated standard deviation. In a way, it is kind of a student’s t-statistic, with the estimate of error varying between points. This is a very popular technique in outlier detection.
Generally, sources agree that observation with an absolute value of 3 on the studentized residual is considered unusual and is, therefore, an outlier. However, you may want to be more aggressive and set it to something lower than 3.
studentized_resids =  concatenated_df.student_resid#Print the median house value of the outliersprint(concatenated_df.median_house_value[abs(studentized_resids) > 2])
#Highest Negative Residualsstudentized_resid_sort = concatenated_df.sort_values(by = 'student_resid')print ('*'*20 + ' Most Negative Residuals ' + '*'*20)print (studentized_resid_sort.head())print ('\n')#Highest Positive Residualsprint ('*'*20 + ' Largest Positive Residuals ' + '*'*20)print (studentized_resid_sort.tail())print ('\n')
The NaN in the positive residuals is really huge numbers divided by 0 because of the process known as deletion residuals, or externally Studentized residuals.
A nice explanation can be found here.
Now, that we have identified observations that have high residuals or outliers, then we can apply a criterion to determine observations with high leverage.
The formula for the leverage cutoff is:
#Number of observationsn = california.shape[0]#Predictorsk = california[features].shape[1]#Leveragecutoff_leverage = ((2*k)+2)/nleverage = concatenated_df.hat_diagleverage_sort = concatenated_df.sort_values(by = 'hat_diag', ascending = False)print ('*'*20 + ' Highest Leverage Data Points ' + '*'*20)print (leverage_sort.head())
Now, let’s try to make sense of what we have so far: outliers and high-leverage data points.
Some sources would agree that influential data points are both outliers and have high leverage. And we can implement this for this exercise, but in reality, even just qualifying as an outlier or high leverage may be enough for an observation to be an influential point.
#Merge the observations that are outliers and have high leverageoutliers = pd.DataFrame((concatenated_df.student_resid[abs(studentized_resids) > 3]))high_leverage = pd.DataFrame((concatenated_df.hat_diag[abs(leverage) > cutoff_leverage]))#Influential Datasetinfluential_points =pd.merge(outliers,high_leverage, left_index=True, right_index=True)display(influential_points)
Influence plot can help us visualize these points:
fig, ax = plt.subplots(figsize=(12,8))fig = sm.graphics.influence_plot(model_1, ax= ax, criterion="cooks", alpha = 0.5)
We can likewise use the yellowbrick package to visualize influential points.
We see that any observations that extend above the red line are influential data points.
The yellowbrick graph we generated used Cook’s distance as a measure of overall influence. The other one used as a measure of overall influence is DFFITS (Difference in fits).
Cook’s distances are nonnegative values and the higher they are, the more influential the observation is. The common cutoff used by most is three times the mean of the dataset’s Cook’s D for an observation to be classified as influential.
cutoff_cooks =(concatenated_df.loc[:,"cooks_d"].mean())*3outliers_2 = pd.DataFrame((concatenated_df.cooks_d[abs(concatenated_df.cooks_d) > cutoff_cooks]))display(outliers_2)
For DFFITS, the conventional cutoff value uses the same variables in the determination of cutoff of leverage we had earlier but using this formula:
#Cutoff for DFFITScutoff_dffits = 2* math.sqrt(k/n)print(concatenated_df.dffits[abs(concatenated_df.dffits) > cutoff_dffits])
Unlike Cook’s distances, dffits can either be positive or negative. With a “0” value, the point lies exactly on the regression line.
While dffits and Cook’s distance measures the general influence of an observation, dfbetas measure the influence of an observation brought about by a specific variable. As you can see, this is quite useful for multiple linear regression models.
#Variables for our plots latercolors = ['#e6194b', '#3cb44b', '#ffe119', '#4363d8', '#f58231', '#911eb4',           '#46f0f0', '#f032e6', '#bcf60c', '#fabebe', '#008080']markers = ['o','v', '^', 'p', '_', '+',          'd', 'h', '*','|', 's']dfbetas = [x for x in concatenated_df.columns.to_list() if ('dfb' in x)]dfbeta_cutoff =  2/(n)**(0.5)fig,ax = plt.subplots(figsize=(12,8))for i in range(len(dfbetas)):    ax.scatter(concatenated_df.median_house_value,                concatenated_df[dfbetas[i]],                color = colors[i],                marker = markers[i],              label = dfbetas[i],                alpha=0.5,              s=100)    ax.legend(loc='best')    plt.plot((0, 550000), (dfbeta_cutoff, dfbeta_cutoff), '-.r*')plt.plot((0, 550000), (-dfbeta_cutoff, -dfbeta_cutoff), '-.r*')
Now that we have information on the possible influential data points, let us remove them and try to improve the predictive capacity and fit of our models.
#Influential Datapoints that we manually identified as both Outliers and have High Leverageindex_1 = outliers.index.to_list()#Outliers Identified Using Cooks Distancesindex_2 = outliers_2.index.to_list()#Outliers Indentidied Using DFFITSindex_3 = outliers_3.index.to_list()indices = [index_1,index_2, index_3]for i in range(len(indices)):    X = california[features].drop(indices[i]).values    y = california['median_house_value'].drop(indices[i]).values    X_train, X_test, Y_train, Y_test = train_test_split(X, y, test_size = 0.20, random_state = 5)    model= sm.OLS(Y_train, sm.add_constant(X_train)).fit()    print(model.summary())    print(f'The R-squared for the model after removing influential points from {i} is {round(model.rsquared,3)}')
Let us compare our three models and check if our adjustments did improve the predictive capacity of the model.
For R-squared, the improvement is negligible (0.002) for all three revisions. One possible explanation for this is that we have a huge dataset. The influence of a few points in the model’s accuracy becomes less. You can verify this in the Boston Housing dataset from sklearn.datasets. This means that removing influential points is so much more important for smaller datasets than it is for a huge one.
It is in the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC), however, where we see the benefits of removing our influential points. Both the AIC and BIC are used as selection criteria in choosing among linear regression models. The lower the number for these two, the better the fit of the model.
For the dataset above, removing influential points using dffits resulted in the best fit among the models that we have generated. It is worth noting, however, that in some cases, the other two methods may generate the better model.
Another thing to consider is this: while we may want to improve the predictive capacity of the model, excluding influential data points may not necessarily be what we want. For example, if the purpose of the model is the identification of those extreme, influential instances, say for example loan defaults, removing these points will make our model not learn what features lead to these influential instances.
Full code found on my GitHub page.