Handling Outliers in Machine Learning
By: Abhay Parashar
Tags: Machine Learning, Python, Artificial Intelligence, Education, Data Science

Methods To Find Outliers

Now We have understood what an outlier is and the different types of outliers now let’s see different methods to find outliers.

There are two basic methods:

Percentile Box Plot

Percentile

In this method, we choose a minimum percentile and maximum percentile. Usually, the minimum percentile is 5%, and the maximum percentile is 95%. Then We Fetch out all the data points outside the percentile range, which means those values that are greater than 95% value or smaller than 5% value, and consider them as outliers.

Example: In a dataset, if 5% is 45 and 95% is 1000, then all the values that are below 45 or greater than 1000 are considered as outliers.

Practical Example:

## Let's First Create a Dummy DataFrame With Outliers

lst = [random.randint(0,100) for i in range(0,100)]

## Adding a manual outlier

global_outlier = [300]

df = pd.DataFrame(lst+global_outlier,columns=['number']) ## Minimum Percentile Value

min_val = df.quantile(0.05) ## Maximum Percentile Value

max_val = df.quantile(0.95) ## Finding All the Outliers

df[(df['number']<min_val[0])| (df['number']>max_val[0])] ##############OUTPUT################

number

23 2

64 1

66 99

84 2

89 99

100 300



######YOUR OUTPUT MAY BE DIFFERENT BECAUSE WE ARE USING RANDOM MODULE TO GENERATE SAMPLE DATAFRAME####

Box Plot

A box plot is a graphical display for describing the distribution of data. Box plots use the median and the lower and upper quartiles.

Pandas data frame has a built-in boxplot function. Let’s use the above to create a data frame and try to find the outliers.

df.boxplot(column=['number'])

300 value as an outlier

Let’s use both the techniques and try to find outliers in a real dataset like Titanic. Visit my Github repo and download the cleaned version of the dataset with no nan values from here.

Percentile

########## DETECTING OUTLIERS USING PERCENTILE ###############

df = pd.read_csv('data/titanic_with_no_nan.csv')

max_val = df.Age.quantile(0.95)

min_val = df.Age.quantile(0.05)

df2 = df[(df['Age']<min_val) | (df['Age']>max_val)]

print("Number of Outliers Detected in Age:",df2.shape[0]) #########OUTPUT#########

Number of Outliers Detected in Age: 86

Box Plot

########## DETECTING OUTLIERS USING BOX PLOT ###############

df = pd.read_csv('data/titanic_with_no_nan.csv') ### LET'S USE SEABORN BOX PLOTS

import seaborn as sns

sns.boxplot(df['Age'])

Outliers in the ‘Age’ Column in Titanic Dataset

Different Ways to Handle Outliers

There are two ways to handle outliers.

Remove All the outliers. Replace Outliers Values with a suitable value

Removing all the outliers

In this method, we first find the min and max value quantiles, and then we simply remove all the values by not picking them in further processing.

import matplotlib.pyplot as plt

import seaborn as sns

import warnings

import pandas as pd

warnings.filterwarnings("ignore")

fig, axes = plt.subplots(1,2)

plt.tight_layout(0.2) ## DataFrame

df = pd.read_csv('data/titanic_with_no_nan.csv')

print("Before Shape:",df.shape) ## Max and Min Quantile

max_val = df.Age.quantile(0.95)

min_val = df.Age.quantile(0.05) ## Removing all the outliers

df2 = df[(df['Age']>min_val) & (df['Age']<max_val)] ## Visulization

print("After Shape:",df2.shape)

sns.boxplot(df['Age'],orient='v',ax=axes[0])

axes[0].title.set_text("Before")

sns.boxplot(df2['Age'],orient='v',ax=axes[1])

axes[1].title.set_text("After")

plt.show()

Replacing Outliers Values with a suitable value

Using Quantile Method

In this method, we first find the min and max quantile. After that, we find all the values outside the quantile range and replace them with min or max quantile value accordingly.

import numpy as np

import matplotlib.pyplot as plt

import warnings

warnings.filterwarnings("ignore")

fig, axes = plt.subplots(1,2)

plt.tight_layout(0.2)

df = pd.read_csv('data/titanic_with_no_nan.csv')

print("Previous Shape With Outlier: ",df.shape)

sns.boxplot(df['Age'],orient='v',ax=axes[0])

axes[0].title.set_text("Before") ########### HANDLING OUTLIER ######

max_val = df.Age.quantile(0.95)

min_val = df.Age.quantile(0.05) df2 = df ####### REPLACING ALL THE Large values with MAX QUANTILE VALUE ####

df2['Age'] = np.where(df2['Age']>max_val,max_val,df2['Age']) print("Shape After Removing Outliers:", df2.shape) sns.boxplot(df2['Age'],orient='v',ax=axes[1])

axes[1].title.set_text("After")

plt.show()

Using IQR

IQR or interquartile range is a measurement of variability based on dividing the dataset into different quantiles.

Quantiles are divided into Q1, Q2, and Q3, where Q1is the middle value of the first half of the dataset. Q2 is the median value, and Q3 is the middle value of the second half of the dataset.

IQR is equal to Q3 minus Q1.

Q1 = df.column.quantile(0.25)

Q3 = df.column.quantile(0.75)

IQR = Q3-Q1

After calculating IQR, we calculate the lower limit and upper limit and then simply discard all the values that are less or above the limit and replace them with lower and upper limit accordingly.

NOTE: It will Also Work For Data That is Left skewed or Right Skewed

df = pd.read_csv('data/titanic_with_no_nan.csv')

import numpy as np

import matplotlib.pyplot as plt

import warnings

warnings.filterwarnings("ignore")

fig, axes = plt.subplots(1,2)

plt.tight_layout(0.2)

print("Previous Shape With Outlier: ",df.shape)

sns.boxplot(df['Age'],orient='v',ax=axes[0])

axes[0].title.set_text("Before") ########### HANDLING OUTLIER ######

Q1 = df.Age.quantile(0.25)

Q3 = df.Age.quantile(0.75)

print(Q1,Q3) IQR = Q3-Q1

print(IQR) lower_limit = Q1 - 1.5*IQR

upper_limit = Q3 + 1.5*IQR

print(lower_limit,upper_limit) df2 = df

df2['Age'] = np.where(df2['Age']>upper_limit,upper_limit,df2['Age'])

df2['Age'] = np.where(df2['Age']<lower_limit,lower_limit,df2['Age']) print("Shape After Removing Outliers:", df2.shape) sns.boxplot(df2['Age'],orient='v',ax=axes[1])

axes[1].title.set_text("After")

plt.show()

Bonus Tip

It is not always easy as it looks to find the outliers and then handle them. In such a situation, we can use a different machine learning model that is not sensitive to outliers.

1. Naivye Bayes Classifier--- Not Sensitive To Outliers

2. SVM-------- Not Sensitive To Outliers

3. Decision Tree Regressor or Classifier---- Not Sensitive

4. Ensemble(RF,XGboost,GB)------------Not Sensitive

5. KNN--------------------------- Not Sensitive 6. Linear Regression------------- Sensitive

7. Logistic Regression----------- Sensitive

8. Kmeans------------------------ Sensitive

9. Hierarichal------------------- Sensitive

10. PCA-------------------------- Sensitive

11. Neural Networks-------------- Sensitive

Github & Notebook Link