Are outliers ruining your machine learning predictions? Search for an optimal solution | by Kaushik Choudhury | Towards Data Science

In the world of data, we all love Gaussian distribution (also known as a normal distribution). In real-life, seldom we have normal distribution data. It is skewed, missing data points or has outliers.
As I mentioned in my earlier article, the strength of Scikit-learn inadvertently works to its disadvantage. Machine learning developers esp. with relatively lesser experience implements an inappropriate algorithm for prediction without grasping particular algorithms salient feature and limitations. We have seen earlier the reason we should not use the decision tree regression algorithm in making a prediction involving extrapolating the data.
The success of any machine learning modelling always starts with understanding the existing dataset on which model will be trained. It is imperative to understand the data well before starting any modelling. I will even go to an extent to say that the prediction accuracy of the model is directly proportional to the extent we know the data.
Objective
In this article, we will see the effect of outliers on various regression algorithms available in Scikit-learn, and learn about the most appropriate regression algorithm to apply in such a situation. We will start with a few techniques to understand the data and then train a few of the Sklearn algorithms with the data. Finally, we will compare the training results of the algorithms and learn the potential best algorithms to apply in the case of outliers.
Training Dataset
The training data consists of 200,000 records of 3 features (independent variable) and 1 target value (dependent variable). The true coefficient of the features 1, feature 2 and feature 3 is 77.74, 23.34, and 7.63 respectively.
Step 1- First, we will import the packages required for data analysis and regressions.
We will be comparing HuberRegressor, LinearRegression, Ridge, SGDRegressor, ElasticNet, PassiveAggressiveRegressor and Linear Support Vector Regression (SVR), hence we will import the respective packages.
Most of the time, few data points are missing in the training data. In that case, if any particular features have a high proportion of null values then it may be better not consider that feature. Else, if a few data points are missing for a feature then either can drop those particular records from training data, or we can replace those missing values with mean, median or constant values. We will import SimpleImputer to fill the missing values.
We will import the Variance Inflation Factor to find the severity of multicollinearity among the features. We will need Matplotlib and seaborn to draw various plots for analysis.
from sklearn.linear_model import HuberRegressor,LinearRegression ,Ridge,SGDRegressor,ElasticNet,PassiveAggressiveRegressorfrom sklearn.svm import LinearSVRimport pandas as pdfrom sklearn.impute import SimpleImputerfrom statsmodels.stats.outliers_influence import variance_inflation_factorimport numpy as npimport matplotlib.pyplot as pltimport seaborn as sns
Step 2- In the code below, training data containing 200.000 records are read from excel file into the PandasDataframe called “RawData”. Independent variables are saved into a new DataFrame.
RawData=pd.read_excel("Outlier Regression.xlsx")Data=RawData.drop(["Target"], axis=1)
Step 3-Now we will start by getting a sense of the training data and understanding it. In my opinion, a heatmap is a good option to understand the relationship between different features.
sns.heatmap(Data.corr(), cmap="YlGnBu", annot=True)plt.show()
It shows that none of the independent variables (features) is closely related to each other. In case you would like to learn more on the approach and selection criteria of independent variables for regression algorithms, then please read my earlier article on it.
How to identify the right independent variables for Machine Learning Supervised Algorithms?
Step 4- After getting a sense of the correlation among the features in the training data next we will look into the minimum, maximum, median etc. of each feature value range. This will help us to ascertain whether there are any outliers in the training data and the extent of it. Below code instructs to draw boxplots for all the features.
sns.boxplot(data=Data, orient="h",palette="Set2")plt.show()
In case you don’t know to read the box plot then please refer the Wikipedia to learn more on it. Feature values are spread across a wide range with a big difference from the median value. This confirms the presence of outlier values in the training dataset.
Step 5- We will check if there are any null values in the training data and take any action required before going anywhere near modelling.
print (Data.info())
Here we can see that there are total 200,000 records in the training data and all three features have few values missing. For example, feature 1 has 60 values (200000 –199940) missing.
Step 6- We use SimpleImputer to fill the missing values with the mean values of the other records for a feature. In the below code, we use the strategy= “mean” for the same. Scikit-learn provides different strategies viz. mean, median, most frequent and constant value to replace the missing value. I will suggest you please self explore the effect of each strategy on the training model as a learning exercise.
In the code below, we have created an instance of SimpleImputer with strategy “Mean” and then fit the training data into it to calculate the mean of each feature. Transform method is used to fill the missing values with the mean value.
imputer = SimpleImputer(strategy="mean")imputer.fit(Data)TransformData = imputer.transform(Data)X=pd.DataFrame(TransformData, columns=Data.columns)
Step 7- It is good practice to check the features once more after replacing the missing values to ensure we do not have any null (blank) values remaining in our training dataset.
print (X.info())
We can see that now all the features have non-null i.e non-blank values for 200,000 records.
Step 8- Before we start training the algorithms, let us check the Variance inflation factor (VIF) among the independent variables. VIF quantifies the severity of multicollinearity in an ordinary least squares regression analysis. It provides an index that measures how much the variance (the square of the estimate’s standard deviation) of an estimated regression coefficient is increased because of collinearity. I will encourage you all to read the Wikipedia page on Variance inflation factor to gain a good understanding of it.
vif = pd.DataFrame()vif["features"] = X.columnsvif["vif_Factor"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]print(vif)
In the above code, we calculate the VIF of each independent variables and print it. In general, we should aim for the VIF of less than 10 for the independent variables. We have seen earlier in the heatmap that none of the variables is highly correlated, and the same is reflecting in the VIF index among the features.
Step 9- We will extract the target i.e. dependent variable values from the RawData dataframe and save it in a data series.
y=RawData["Target"].copy()
Step 10- We will be evaluating the performance of various regressors viz. HuberRegressor, LinearRegression, Ridge and others on outlier dataset. In the below code, we created instances of the various regressors.
Huber = HuberRegressor()Linear = LinearRegression()SGD= SGDRegressor()Ridge=Ridge()SVR=LinearSVR()Elastic=ElasticNet(random_state=0)PassiveAggressiveRegressor= PassiveAggressiveRegressor()
Step 11- We declared a list with instances of the regressions to pass it in sequence in a for a loop later.
estimators = [Linear,SGD,SVR,Huber, Ridge, Elastic,PassiveAggressiveRegressor]
Step 12- Finally, we will train the models in sequence with the training data set and print the coefficients of the features calculated by the model.
for i in estimators:    reg= i.fit(X,y)    print(str(i)+" Coefficients:", np.round(i.coef_,2))    print("**************************")
We can observe a wide range of coefficients calculated by different models based on their optimisation and regularisation factors. Feature 1 coefficient calculated coefficient varies from 29.31 to 76.88.
Due to a few outliers in the training dataset a few models, like linear and ridge regression predicted coefficients nowhere near the true coefficients. Huber regressor is quite robust to the outliers ensuring loss function is not heavily influenced by the outliers while not completely ignoring their effects like TheilSenRegressor and RANSAC Regressor. Linear SVR also more options in the selection of penalties and loss functions and performed better than other models.
Learning Action for you- We trained different models with a training data set containing outliers and then compared the predicted coefficients with actual coefficients. I will encourage you all to follow the same approach and compare the prediction metrics viz. R2 score, mean squared error (MSE), RMSE of different models trained with outlier dataset.
Hint — You may be surprised to see the R2 (coefficient of determination) regression score function for the models in comparison to the coefficient prediction accuracy we have seen in this article. In case, you stumble upon on any point then, feel free to reach out to me.
Key Takeaway
As mentioned in my earlier article and keep stressing that main focus for us machine learning practitioners are to consider the data, prediction objective, algorithms strengths and limitations before starting the modelling. Every additional minute we spend in understanding the training data directly translates into prediction accuracy with the right algorithms. We don’t want to use a hammer to unscrew and screwdriver to nail in the wall.
If you want to learn more on a structured approach to identifying the right independent variables for Machine Learning Supervised Algorithms then please refer my article on this topic.
"""Full Code"""from sklearn.linear_model import HuberRegressor, LinearRegression ,Ridge ,SGDRegressor,  ElasticNet, PassiveAggressiveRegressorfrom sklearn.svm import LinearSVRimport pandas as pdimport matplotlib.pyplot as pltimport seaborn as snsfrom sklearn.impute import SimpleImputerfrom sklearn.preprocessing import StandardScalerfrom statsmodels.stats.outliers_influence import variance_inflation_factorimport numpy as npRawData=pd.read_excel("Outlier Regression.xlsx")Data=RawData.drop(["Target"], axis=1)sns.heatmap(Data.corr(), cmap="YlGnBu", annot=True)plt.show()sns.boxplot(data=Data, orient="h",palette="Set2")plt.show()print (Data.info())print(Data.describe())imputer = SimpleImputer(strategy="mean")imputer.fit(Data)TransformData = imputer.transform(Data)X=pd.DataFrame(TransformData, columns=Data.columns)print (X.info())vif = pd.DataFrame()vif["features"] = X.columnsvif["vif_Factor"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]print(vif)y=RawData["Target"].copy()Huber = HuberRegressor()Linear = LinearRegression()SGD= SGDRegressor()Ridge=Ridge()SVR=LinearSVR()Elastic=ElasticNet(random_state=0)PassiveAggressiveRegressor= PassiveAggressiveRegressor()estimators = [Linear,SGD,SVR,Huber, Ridge, Elastic,PassiveAggressiveRegressor]for i in estimators:    reg= i.fit(X,y)    print(str(i)+" Coefficients:", np.round(i.coef_,2))    print("**************************")