Huber and Ridge Regressions in Python: Dealing with Outliers | by Michael Grogan | Towards Data Science

Traditional linear regression can prove to have some shortcomings when it comes to handling outliers in a set of data.
Specifically, if a data point lies very far away from other points in the set — this can significantly influence the least squares regression line, i.e. the line that approximates the overall direction of the set of data points will be skewed by the presence of outliers.
In an attempt to guard against this shortcoming, it is possible to use modified regression models that are robust to outliers. In this particular instance, we will take a look at the Huber and Ridge regression models.
The dataset that is used in this instance is the Pima Indians Diabetes dataset as originally from the National Institute of Diabetes and Digestive and Kidney Diseases and made available under the CC0 1.0 Universal (CC0 1.0) Public Domain Dedication license.
For this particular scenario, a regression model will be built to predict body mass index (BMI) levels across patients.
When looking at a boxplot of BMI, we can see that there exist significant outliers as evidenced by the data points in the top half of the graph.
# Creating plotplt.boxplot(bmi)plt.ylabel("BMI")plt.title("Body Mass Index")# show plotplt.show()
A correlation matrix across the data was then created:
corr = a.corr()corr
Here is a visual using seaborn:
sns.heatmap(a.corr());
Of the independent variables, Skin Thickness and Glucose were chosen as the two variables that are hypothesised to have a significant effect on BMI.
A regression was run across the training data which produced the following results:
According to the above results:
A 1 unit increase in Skin Thickness leads to a 0.1597 increase in BMI — holding all other variables constant.
A 1 unit increase in Glucose levels leads to a 0.0418 increase in BMI — holding all other variables constant.
We see that these two variables are highly significant at all levels (given p-values of 0). While the R-Squared of 17.5% is low — this does not necessarily mean that the model is bad. Given that there are a wide range of variables that can influence BMI fluctuations — this simply indicates that there are many variables that this model does not account for. However, the two independent variables being used show a highly significant relationship.
Predictions were generated across the validation set and a root mean squared error (RMSE) value was calculated. RMSE is used in this instance as the score is more sensitive to outliers. The higher the score, the more error in the forecast predictions.
>>> olspredictions = results.predict(X_val)>>> print(olspredictions)[36.32534071 31.09716546 28.67493585 34.29867119 35.03070074... 36.50971909 35.97416079 36.57591618 35.10923948 34.3672371]
The root mean squared error is calculated:
>>> mean_squared_error(y_val, olspredictions)>>> math.sqrt(mean_squared_error(y_val, olspredictions))5.830559762277098>>> np.mean(y_val)31.809333333333342
The value of the root mean squared error is 5.83 relative to the mean of 31.81 across the validation set.
Now, we will generate Huber and Ridge regression models. In the same way, predictions will be made against the validation set using these models and the root mean squared error will be calculated.
Both the Huber and Ridge regressions are designed to produce a regression line that is less sensitive to outliers than standard linear regression.
However, the way in which these models work in doing so differ slightly.
Specifically, Huber regression relies on what is known as the M-estimate, or measures of location that are less sensitive to outliers than the mean (as described in the Oxford Dictionary of Statistics (Upton and Cook, 2014)).
A Ridge regression uses what is known as L2 regularization — which makes the weights of the outlier values smaller so as to have less of an effect on the regression line. Additionally, L2 regularization attempts to estimate the mean of the data to avoid overfitting, whereas L1 regularization (as used in a Lasso regression) attempts to estimate the median of the data.
In this example, predictions will be made using the Huber and Ridge regressions, in order to compute the RMSE of the predictions against the validation set. The best-performing model will then be used to make predictions across the test set.
Here is a sample Huber regression:
hb1 = linear_model.HuberRegressor(epsilon=1.1, max_iter=100, alpha=0.0001, warm_start=False, fit_intercept=True, tol=1e-05)
In particular, the value of epsilon measures the number of samples that should be classified as outliers. The smaller this value, the more robust the model is to outliers.
From this standpoint, five separate Huber regressions with varying epsilon values were computed.
hb1 = linear_model.HuberRegressor(epsilon=1.1, max_iter=100, alpha=0.0001, warm_start=False, fit_intercept=True, tol=1e-05)hb2 = linear_model.HuberRegressor(epsilon=1.8, max_iter=1000, alpha=0.0001, warm_start=False, fit_intercept=True, tol=1e-05)hb3 = linear_model.HuberRegressor(epsilon=2.5, max_iter=1000, alpha=0.0001, warm_start=False, fit_intercept=True, tol=1e-05)hb4 = linear_model.HuberRegressor(epsilon=3.2, max_iter=1000, alpha=0.0001, warm_start=False, fit_intercept=True, tol=1e-05)hb5 = linear_model.HuberRegressor(epsilon=3.9, max_iter=1000, alpha=0.0001, warm_start=False, fit_intercept=True, tol=1e-05)
Predictions were made across the five models against the validation set. Using the first model as an example:
>>> hubermodel1 = hb1.fit(X_train,y_train)>>> huberpredictions1 = hb1.predict(X_val)>>> print(huberpredictions1)[35.67051275 29.43501067 27.18925225 33.91769821 34.47019359... 31.52694684 31.0940833  35.37464065 30.99181107 36.11032014]
The calculated mean squared errors were as follows:
hb1 = 5.803
hb2 = 5.800
hb3 = 5.816
hb4 = 5.825
hb5 = 5.828
We see that the RMSE values are all slightly less than the 5.83 computed when using the OLS regression. hb2, or the model with an epsilon value of 1.8, showed the best performance — albeit marginal.
In the same manner as previously, predictions are made using the Ridge regression and the RMSE is subsequently calculated:
>>> rg = Ridge(fit_intercept=True, alpha=0.0, random_state=0, normalize=True)>>> ridgemodel = rg.fit(X_train,y_train)>>> ridgepredictions = rg.predict(X_val)[36.32534071 31.09716546 28.67493585 34.29867119 35.03070074...31.79765405 36.18771132 31.86883756 36.98120033 35.68182273]
The mean squared error is virtually identical to that calculated using the OLS regression:
>>> mean_squared_error(y_val, ridgepredictions)>>> math.sqrt(mean_squared_error(y_val, ridgepredictions))5.8305
Given that the Huber regression with epsilon = 1.8 showed the best performance against the validation set (albeit marginal), let’s see how it performs against the test set.
In this case, a portion of the Pima Indians Diabetes dataset was made physically separate from the rest of the data — in order to examine how the model would perform across unseen data.
Here are the predictions:
>>> btest = t_bmi>>> btest=btest.values>>> bpred = hb2.predict(atest)>>> bpredarray([33.14957023, 30.36001456, 32.93500157, 28.91518701, ...34.72666166,36.29947658, 39.13505914, 33.77051646])
The RMSE and mean value of the test set are calculated:
>>> math.sqrt(mean_squared_error(btest, bpred))5.744712507435319>>> np.mean(btest)32.82418300653595
With the RMSE value accounting for just over 17% of the mean value — the model does reasonably well at predicting BMI values across the test set.
As we have seen, the RMSE values across all models were more or less similar — with the Huber regressions delivering slightly lower RMSE values. However, depending on the size of the outliers, there are instances where Huber and Ridge regressions can significantly outperform OLS.
In this example, you have seen:
How to visually detect outliers in a data sample
The differences between Huber and Ridge regressions
How to modify outlier sensitivity within a Huber regression
Use of RMSE to determine model accuracy
Many thanks for your time, and any questions or feedback are greatly appreciated. You can find more of my content at michael-grogan.com.
Disclaimer: This article is written on an “as is” basis and without warranty. It was written with the intention of providing an overview of data science concepts, and should not be interpreted as professional advice. The findings and interpretations in this article are those of the author and are not endorsed by or affiliated with any third-party mentioned in this article. The author has no relationship with any third parties mentioned in this article.