Univariate Outlier Detection in Python
By: Philip Wilkinson
Tags: Data Science, Anomaly, Anomaly Detection, Outlier Detection, Exploratory Data Analysis

Univariate Outlier Detection in Python

Photo by Will Myers on Unsplash

Outlier detection can often be an important part of any exploratory data analysis. This is because in the real world, data is often messy and many different things can affect the underlying data. It is thus important to be able to identify different methods to be able to identify these from the underlying data.

The first thing to ask, then, is “what is an outlier?” An outlier can be classed as a data point, or several data points, that don’t fit the pattern, data structure, or within the normal bounds to what we would expect for the data that we have. This is a very subjective definition as outliers depend heavily on the context in which you are examining as to whether a data point is an outlier or not.

This outlier could be the result of many different issues:

Human error

Instrument error

Experimental error

Intentional creation

Data processing error

Sampling error

Natural outlier

The purpose for being able to identify this outlier of course can also be different. This could be because an outlier would indicate something has changed in the action that produces the data which is useful in the case of:

Fraud detection

Intrusion detection

Fault diagnostics

Time series monitoring

Health monitoring

Where an outlier would indicate maybe something has gone wrong in the process or the nature of the process generating the data has changed. This would thus entail identifying outliers on the basis of underlying accepted normal data.

In other cases it is useful to be able to remove outliers from existing data to ensure a model works. For example:

Recommendation engines

Time series forecasting

Scientific experimentation

Model building

Where outliers in existing data would affect model implementation such as in linear regression or classification tasks. Thus it would be important to identify outliers from existing data where we are not exactly sure what normal behavior is yet.

These two tasks can thus be separated, with the first referring to novelty detection in newly generated data, and the latter referring to anomaly detection in underlying data. It is primarily to this second domain that we turn our attention to although the methods outlined below can also be applied to the first scenario as well.

The dataset that I use for this is that of Pokémon through the first seven generations containing a total of 801 Pokémon. Specifically, I will focus on analysing the distribution of the HP attribute from this dataset to be able to detect any potential anomalies. Of course, any anomalies in the dataset detected by these methods may not actually be anomalies, but we can make a choice in light of the results.

Distribution plot

The first method for this is by examining the distribution of the underlying dataset through a distribution plot. This can be implemented using the following code:

#create the displot

sns.displot(pokemon["hp"],

kde = True) #label the axis

plt.xlabel("HP", fontsize = 15)

plt.ylabel("Count", fontsize = 15)

plt.title("Pokemon HP distribution", fontsize = 15) plt.show()

Image by author

From this we can see that the distribution appears positively skewed with a few points that fall to the extreme right of this distribution. On the basis of the visual inspection we could say that Pokémon that have HP greater than 200 could potentially be outliers in this dataset. This can be extracted from the dataset using:

pokemon_above_150_HP = pokemon[pokemon["HP"] > 150]

pokemon_above_150_HP

Image by author

Which we can see results in 8 potential outliers, however this is just based on purely visual inspection.

Z-score

We can see that the distribution of the HP attribute is roughly normal, albeit with some degree of skew. We could thus supplement the visual inspecting by using the Z-score. This is based on the following formula:

Where 𝜇 is the mean of the data and 𝜎 is the standard deviation. The purpose of this is to see how many standard deviations away is the data point from the mean of the dataset. The further away the data point is, as indicated by a higher Z-score, the more likely we could argue that the data point is an outlier. This can be implemented in Python as:

#calculate the Z score

pokemon["HP_z"] = (pokemon["hp"] - pokemon["hp"].mean())/pokemon["hp"].std()

On this basis, we can use the Z-score to examine data that lies greater than 2, 2.5, 3 or 3.5 standard deviations from the mean. The choice of which Z-score to act as the cutoff will depend on the underlying data, but here choosing a cut-off of 3 extracts the following Pokémon:

Image by author

The main difference here is that this includes those with HP of 150 or greater, rather than just greater than 150. This therefore suggests that the initial visual inspection and choice of cut off was relatively accurate in terms of this dataset.

Boxplot

Another good method for this is that of the Boxplot. This again is another visualisation method that can be used to identify any outliers from the dataset. This is done on the basis that any value that is less than the lower quartile less 1.5 times the inter quartile range or that is greater than the upper quartile plus 1.5 times the interquartile range can be seen as an outlier. On the plot, this shows as separate points being identified beyond the min and max of the boxplot itself. This can be implemented using the Seaborn library as:

#create the boxplot

ax = sns.boxplot(x = pokemon["HP"]) #add labels to the plot

ax.set_xlabel("HP", fontsize = 15)

ax.set_ylabel("Variable", fontsize = 15)

ax.set_title("pokemon HP boxplot", fontsize =20, pad = 20) plt.show()

Image by author

In contrast to the distribution plot, this box plot suggests more clearly what data points could be classed as outliers and we can see that this suggests not only are there outliers at the higher end of the distribution but also one at the lower end of the distribution.

Tukey fences

These points can be extracted using the values of the interquartile range as follows:

#extract the upper and lower quantiles

pokemon_HP_lq = pokemon["HP"].quantile(0.25)

pokemon_HP_uq = pokemon["HP"].quantile(0.75)

#extract the inter quartile range

pokemon_HP_iqr = pokemon_HP_uq - pokemon_HP_lq #get the upper and lower bounds

lower_bound = pokemon_HP_lq - 1.5*pokemon_HP_iqr

upper_bound = pokemon_HP_uq + 1.5*pokemon_HP_iqr #extract values outside these bounds

Pokemon_IQR_outliers = pokemon[(pokemon.HP <= lower_bound) | (pokemon.HP >= upper_bound)]

Pokemon_IQR_outliers

Image by author

Which suggests many more outliers than the previous inspection. Indeed, this suggests one Pokémon whose HP falls below the lower bound and twenty-three that fall above the upper bound.

An alternative to this, given the high number of potential outliers identified, is to use alternate values to multiply the inter-quartile range by. To get extreme outliers from this instead of multiplying by only 1.5, we could multiply by 3. In doing so we get:

Image by author

Which suggests only six outliers at the upper end of the distribution. Which one you select will depend on the type of outliers you are looking to identify.

Clustering

The final method to potentially use is that of clustering. The aim of clustering is to be able to identify groups from the dataset, where potential outliers may fit in extreme groups or in a group of their own depending on how different their characteristics are from the underlying dataset. For univariate analysis this may be limited in terms of its ability to identify clearly distinct groups, but visualising the results could potentially identify outliers much clearer than just examining the underling distribution. Using K-Means clustering therefore this can be implemented as:

#convert the HP values to a float to be able to use numpy

HP_raw = pokemon["hp"].values.astype("float")

#use the kmeans function from scipy

centroids, avg_distance = kmeans(HP_raw, 4)

#extract the groups from the data

groups, cdit = vq(HP_raw, centroids) #plot the results

#assign groups back to the dataframe

pokemon["HP_groups"] = groups #plot the scatter plot

fig = px.scatter(pokemon, x = "hp", y = pokemon.index,

color = "HP_groups",

hover_name = "name")

#add a title

fig.update_layout(title = "K-Means outlier detection",

title_x = 0.5)

#show the plot

fig.show()

Image by author

From which we could more potentially identify outliers from both the lower and upper ends of the distribution.

The fact that these methods suggest different potential outliers all from the same dataset shows the fact that outlier detection is not an exact science and judgement calls have to be made. An important part of this will also be deciding what needs to be done with these outliers and what they mean in terms of removing them from the dataset, imputing values to cover these outliers or acknowledging their existing but continuing with the analysis as is.

This of course will depend on the purpose of identifying the outliers, the underlying dataset and the analysis that will be performed in the future. While practical applications of trying to find outliers from a single variable from a whole dataset may be limited, these methods can be used to supplement multivariate outlier detection algorithms such as Isolation Forest, Local Outlier Factor or Elliptic Envelope where outlier scores can be analysed to decide contamination values.

Code available: https://github.com/PhilipDW183/Outlier_detection

Dataset available: https://www.kaggle.com/rounakbanik/pokemon?select=pokemon.csv