Robust Regression: All You Need to Know & an Example in Python
By: Serafeim Loukas
Tags: Machine Learning, Data Science, Robustness, Outliers, Linear Regression

3. Working example

3.1.The artificial dataset

First, to illustrate the difference between the regular linear regression and the RANSAC robust model, we will create some data that have outliers. The example is based on the examples from the sklearn documentation page [1].

Our artificial dataset will consist of: one dependent variable (y) and one independent variable (X) with 1000 observations from which 50 are outliers.

import numpy as np

from matplotlib import pyplot as plt

from sklearn import linear_model, datasets n_samples = 1000

n_outliers = 50 X, y, coef = datasets.make_regression(n_samples=n_samples, n_features=1, n_informative=1, noise=10, coef=True, random_state=0) # Add outlier data

np.random.seed(0)

X[:n_outliers] = 3 + 0.5 * np.random.normal(size=(n_outliers, 1))

y[:n_outliers] = -3 + 10 * np.random.normal(size=n_outliers) print("The independent variable X has {} observations/samples".format(X.shape[0]))

print("The dependent variable y has shape {}".format(y.shape))

Code output.

Let’s also plot the data to visualize the artificial data and see the outliers.

plt.scatter(X,y)

plt.show()

Code output.

It is clear that we have 1000 observations from which 50 are outliers (right cloud of points).

3.2. The model fitting

Now let’s fit a regular regression model and a robust model on the data and check the estimated regression coefficients.

# Fit line using all data

lr = linear_model.LinearRegression()

lr.fit(X, y) # Robustly fit linear model with RANSAC algorithm

ransac = linear_model.RANSACRegressor()

ransac.fit(X, y)

inlier_mask = ransac.inlier_mask_

outlier_mask = np.logical_not(inlier_mask) # Predict data of estimated models

line_X = np.arange(X.min(), X.max())[:, np.newaxis]

line_y = lr.predict(line_X)

line_y_ransac = ransac.predict(line_X) # Compare estimated coefficients

print(“Estimated coefficients (true, linear regression, RANSAC):”)

print(coef, lr.coef_, ransac.estimator_.coef_)

Code output.

We observe something very interesting here. The true coefficient is 82.19 while the estimated by the regural regression is 54.17and the one estimated by the robust regreassion is 81.63. We can verify that the robust model is performing well.

Note: “coef” was returned by the function “datasets.make_regression” when we created the data (see first code block).

3.3. Visualizing the fitted regression lines

lw = 2

plt.scatter(X[inlier_mask], y[inlier_mask], color='yellowgreen', marker='.', label='Inliers')

plt.scatter(X[outlier_mask], y[outlier_mask], color='gold', marker='.', label='Outliers')

plt.plot(line_X, line_y, color='navy', linewidth=lw, label='Regural Linear regression')

plt.plot(line_X, line_y_ransac, color='royalblue', linewidth=lw, label='RANSAC regression')

plt.legend(loc='lower right')

plt.xlabel("Input")

plt.ylabel("Response")

plt.show()

We observe again that the robust model is performing well ignoring the outliers.

Conclusions

The presence of outliers in a dataset can introduce bias and lead to underperforming linear regression models.

To deal with outliers we can use robust regression algorithms.

In this article we saw that the robust model is performing well when outliers are presence whereas regular linear regression models are highly affected and biased by these outliers.