How to Handle/Detect Outliers for machine learning?
By: Surendra Kumar
Tags: Data Science, Outliers, Data Analysis, Machine Learning, Exploratory Data Analysis

Why It is important to identify outliers?

Often outliers are discarded because of their effect on the total distribution and statistical analysis of the dataset. This is certainly a good approach if the outliers are due to an error of some kind (measurement error, data corruption, etc.), however often the source of the outliers is unclear. There are many situations where occasional ‘extreme’ events cause an outlier that is outside the usual distribution of the dataset but is a valid measurement and not due to an error. In these situations, the choice of how to deal with the outliers is not necessarily clear and the choice has a significant impact on the results of any statistical analysis done on the dataset. The decision about how to deal with outliers depends on the goals and context of the research and should be detailed in any explanation about the methodology.

What is an Outlier?

An outlier is any data point that is distinctly different from the rest of your data points. In general, outliers belong to one of two categories: a mistake in the data or a true outlier. A mistake in the data could be as simple as typing 10000 rather than 100.00, the weight of a person as 0 or 200, and the second type, a true outlier, would be something like salary of CEO in salary prediction dataset.

I will divide the topic into two parts.

1. How to detect outliers.

2. How to handle outliers.

How to detect outliers: — We can find outliers using data visualization/graphical methods and statistical methods. In graphical methods, we could use a scatter plot and a box plot. In statistical methods, we can use percentile capping, IQR (interquartile range), and standard deviation.

Graphical methods

Scatter plot: — We draw the scatter plot and any data points far away from the population can be considered as outliers. Data points in red can be considered as outliers.

Box plot: — When reviewing a Boxplot, an outlier is defined as a data point that is located outside the fences (“whiskers”) of the boxplot (e.g. outside 1.5 times the interquartile range above the upper quartile and below the lower quartile). In the below, figure data points above the inner fences are possible outliers.

Box Plot for Univariate Outlier detection

Statistical methods

Percentile capping: — Any data points less than the value at the first percentile or greater than the value at the 99th percentile could be possible outliers.

IQR (interquartile range): — If a value is higher than the 1.5*IQR above the upper quartile (Q3), the value will be considered as an outlier. Similarly, if a value is lower than the 1.5*IQR below the lower quartile (Q1), the value will be considered as an outlier.

IQR = Q3 -Q1

The lower limit of acceptable range = Q1–1.5* (Q3-Q1)

Upper limit of acceptable range = Q3 + 1.5* (Q3-Q1)

Box Plot with Statistical method

Standard Deviation Method: — If a value is higher or lower by three Standard Deviation from the mean is considered as an outlier. It is based on the characteristics of a normal distribution for which 99.87% of the data appear within this range.

2. How to handle outliers: — Most parametric statistics, like means, standard deviations, and correlations, and every statistic based on these, are highly sensitive to outliers. And since the assumptions of common statistical procedures, like linear regression and ANOVA, are also based on these statistics, outliers can really mess up your analysis. Just how much an outlier affects your analysis depends on a few factors. One factor is dataset size. In a large dataset, each individual point carries less weight, so an outlier is less worrisome than the same data point would be in a smaller dataset. Another consideration is “how much” of an outlier a point might be — just how far out of line with the rest of your dataset a single point is. A point that is ten times as large as your upper boundary will do more damage than a point that is twice as large. Now we have two options either to drop them or do some data transformation or assign a new value to the outliers. First, run the analysis using both outliers and without outliers and make a decision, which action to perform.

When to drop outliers: -

· You know that it’s wrong, For example, if you have a really good sense of what range the data should fall in, like people’s ages, you can safely drop values that are outside of that range.

· You have a lot of data and outliers are few, so your sample won’t be hurt by dropping a questionable outlier.

· In the case of the salary of the CEO, or another true outlier, sometimes it’s best to completely remove that record from your dataset to keep that person or event from skewing your analysis.

· If the outlier creates a relationship where there isn’t one otherwise, we can safely drop those outliers.

When not to drop outliers: — There are a lot of outliers. Outliers are rare by definition. If, for example, 30% of your data is outliers, then it actually means that there’s something interesting going on with your data that you need to look further into.

· Cap your outliers’ data, another way to handle true outliers is to cap them (Winsorization). For example, if you’re using income, you might find that people above a certain income level behave in the same way as those with a lower income. We can use percentile capping. Values that are less than the value at the 1st percentile are replaced by the value at the 1st percentile, and values that are greater than the value at the 99th percentile are replaced by the value at the 99th percentile. The winsorization at the 5th and 95th percentile is also common.

· Assign a new value, if an outlier seems to be due to a mistake in your data, you try imputing a value. Common imputation methods include using the mean or median of a variable based on the distribution of data or utilizing a regression model to predict the missing value.

· Try a transformation, Square root, and log transformations (It’s often preferred when the response variable follows exponential distribution or is right-skewed) both pull in high numbers. This can make assumptions work better if the outlier is a dependent variable and can reduce the impact of a single point if the outlier is an independent variable.

· Another option is to try a different model. Tree-based models like Random Forests and Gradient Boosting techniques, which are less impacted by outliers.