Multivariate outlier detection in Python | by Philip Wilkinson | Towards Data Science

In my previous medium article I introduced five different methods for Univariate outlier detection: Distribution plot, Z-score, Boxplot, Tukey fences and clustering. This highlighted the fact that several different methods can be used to detect outliers in your data, but that each of these can lead to different conclusions. As such, in selecting which method to use you should pay attention to the context of the data and what domain knowledge would also suggest would be classed as an outlier.
Often however, data is collected from multiple sources, sensors and time periods creating multiple variables that could interact with your target variable. This means that analysis or machine learning methods are often applied in the cases where you have more than one variable to analyse. This means that it is often more crucial to be able to detect outliers as a result of the interaction between these variables rather than just detecting outliers from a single variable. This article therefore seeks to identify several different methods for this purpose.
As before, the Pokémon dataset is used to demonstrate these methods, with data from 801 Pokémon from 7 seasons. This will focus on the Attack and Defense attributes from this dataset to be able to detect any potential anomalies as seen in the picture below. Of course, any anomalies in the dataset by these methods may not actually be anomalies, but we can make a choice in light of these results.
As we can see from this plot there is generally a positive linear relationship between defense and attack in this dataset but there appears to be a few outliers.
Box plot and Tukey fences
One of the first methods that can be used as a baseline for being able to detect outliers from mutli-variate datasets is that of boxplots and Tukey fences. While these are able to detect outliers from a single variable distribution, rather than the interaction between them, we can use this as a baseline to compare to other methods later one. We can first visualise this as:
#create the plotax = sns.boxplot(data = pokemon[["attack", "defense"]], orient = "h", palette = "Set2")#add labelsax.set_xlabel("Value", fontsize = 20, labelpad = 20)ax.set_ylabel("Attributes", fontsize = 20, labelpad = 20)ax.set_title("Boxplot of pokemon Attack \nand Defense attributes", fontsize = 20,            pad = 20)#edit ticksax.tick_params(which = "both", labelsize = 15)
This suggests that there could be outliers at the upper end of both distributions. To extract these we can use Tukey fences based on values that are above the upper bound of the upper quartile plus 1.5 times the inter-quartile range and below the lower bound of the lower quartile less 1.5 times the inter-quartile range:
#create a function to calculate IQR boundsdef IQR_bounds(dataframe, column_name, multiple):    """Extract the upper and lower bound for outlier detection using IQR        Input:        dataframe: Dataframe you want to extract the upper and lower bound from        column_name: column name you want to extract upper and lower bound for        multiple: The multiple to use to extract this            Output:        lower_bound = lower bound for column        upper_bound = upper bound for column"""        #extract the quantiles for the column    lower_quantile = dataframe[column_name].quantile(0.25)    upper_quantile = dataframe[column_name].quantile(0.75)    #cauclat IQR    IQR = upper_quantile - lower_quantile        #extract lower and upper bound    lower_bound = lower_quantile - multiple * IQR    upper_bound = upper_quantile + multiple * IQR        #retrun these values    return lower_bound, upper_bound#set the columns we wantcolumns = ["attack", "defense"]#create a dictionary to store the boundscolumn_bounds = {}#iteratre over each column to extract boundsfor column in columns:    #extract normal and extreme bounds    lower_bound, upper_bound =  IQR_bounds(pokemon, column, 1.5)    #send them to the dictionary    column_bounds[column] = [lower_bound, upper_bound]#create the normal dataframepokemon_IQR_AD = pokemon[(pokemon["attack"] < column_bounds["attack"][0]) |                          (pokemon["attack"] > column_bounds["attack"][1]) |                         (pokemon["defense"] < column_bounds["defense"][0]) |                          (pokemon["defense"] > column_bounds["defense"][1])                        ]
Where we get 15 potential outliers detection from the entire dataset. As we can see, this results in outliers that are above 180 attack and/or greater than 150 defense. This therefore suggests linear cut offs for identifying these outliers, but does so on the basis of single variates, rather than there interaction.
Isolation forest
The first method that is capable of dealing with the interaction between these variables is that of Isolation forest. This is often a good starting point, especially for higher dimension datasets. It is a tree ensemble method that is built on the basis of decision trees, just like random forests, whereby trees are partitioned by first randomly selecting a feature and then selecting a random split value between the maximum and minimum value of the selected feature. In principle, outliers are less frequent than regular observations and are different in terms of their value. Thus, by using random partitions, they should be identified as closer to the root of the tree, with fewer splits necessary.
For this, the algorithm requires us to specify the contamination parameter which tells the algorithm how much of the data is expected to be anomalous. In our case, following the Tukey fences analysis, this can be set to 0.02 to signify we think that 2% of the data may be anomalous, equating to 16 points. This can be implemented as:
from sklearn.ensemble import IsolationForest#create the method instanceisf = IsolationForest(n_estimators = 100, random_state = 42, contamination = 0.02)#use fit_predict on the data as we are using all the datapreds = isf.fit_predict(pokemon[["attack", "defense"]])#extract outliers from the datapokemon["iso_forest_outliers"] = predspokemon["iso_forest_outliers"] = pokemon["iso_forest_outliers"].astype(str)#extract the scores from the data in terms of strength of outlierpokemon["iso_forest_scores"] = isf.decision_function(pokemon[["attack", "defense"]])#print how many outliers the data suggestsprint(pokemon["iso_forest_outliers"].value_counts())# Out:1    785-1   16
Which we have 16 outliers as expected. We can then plot this as:
#this plot will be repeated so it is better to create a functiondef scatter_plot(dataframe, x, y, color, title, hover_name):    """Create a plotly express scatter plot with x and y values with a colour        Input:        dataframe: Dataframe containing columns for x, y, colour and hover_name data        x: The column to go on the x axis        y: Column name to go on the y axis        color: Column name to specify colour        title: Title for plot        hover_name: column name for hover            Returns:        Scatter plot figure    """    #create the base scatter plot    fig = px.scatter(dataframe, x = x, y=y,                    color = color,                     hover_name = hover_name)    #set the layout conditions    fig.update_layout(title = title,                     title_x = 0.5)    #show the figure    fig.show()#create scatter plotscatter_plot(pokemon, "attack", "defense", "iso_forest_outliers",             "Isolation Forest Outlier Detection",            "name")
Where we can see that unlike Tukey Fences, there is no clear cut off at the top or bottom of each variable and that points in the lower left hand corner have also been identified as outliers. We can see the range of scores assigned to these variables on the basis of their position in the decision trees as well.
#create the same plot focusing on the scores from the datasetscatter_plot(pokemon, "attack", "defense", "iso_forest_scores",             "Isolation Forest Outlier Detection Scores",            "name")
Where we can see what points may have been classed as outliers, with points in the centre mass receiving higher scores suggesting these are core points, while those on the periphery receiving lower scores suggesting they could be outliers.
We can also use the univariate anomaly detection methods to be able to inform the choice of the contamination percentage by identifying outliers from the scores results as:
Which these methods can be used to suggest where to set the cut-off for the contamination parameter and can be used in the same way for all following algorithms where a score can be extracted from.
Local outlier factor
Another algorithm that can be used is the Local Outlier Factor algorithm. This is a calculation that examines the neighbors of a point to be able to find its density and then compares this to the density of its neighbors. If the density of a point is found to be much smaller than that of its neighbors, suggesting isolation, than this could be identified as an outlier.
The benefit of this algorithm is that it takes into account both the local and global properties of the dataset into account as it focuses on how isolated the sample is in respect to its neighbors. For this, we need to specify the number of neighbors (default 20) to compare the density to, and the distance metric to be used (the default is “minkowski” which generalises both the Euclidean and Manhattan distance). This can thus be applied as:
#import the algorithmfrom sklearn.neighbors import LocalOutlierFactor#initialise the algorithmlof = LocalOutlierFactor(n_neighbors = 20)#fit it to the training data, since we don't use it for novelty than this is finey_pred = lof.fit_predict(pokemon[["attack", "defense"]])#extract the predictions as stringspokemon["lof_outliers"] = y_pred.astype(str)#print the number of outliers relative to non-outliersprint(pokemon["lof_outliers"].value_counts())#extract the outlier scorespokemon["lof_scores"] = lof.negative_outlier_factor_#Out:1   767-1  34
Where using the default value of 20 neighbors gives us 39 potential outliers. We can again examine the distribution of the outliers and the scores:
As again, we can see that similar to the previous algorithm outliers are detected on the edge of the main mass of points. This will be because of the emphasis on density where points lying outside of the main region are likely to be identified as outliers as they don’t have points surrounding them completely. As before, we can also use univariate anomaly detection methods to analyse the scores to adjust the hyperparameters in the original algorithm.
DBScan
Similarly, DBScan is another algorithm that can also detect outliers on the basis of distance between points. This is a clustering algorithm and behaves differently to LOF by selecting points not already assigned to a cluster, determines if it is a core point by seeing if there at least a given number of samples within a given distance. If so, then it is designated as a core point along with all points within direct reach of that point as identified by the distance metric. This is then repeated for each point within the cluster until the edge of the cluster is identified where there are no more points within the specified distance that can be identified as core points (having the minimum number of points within the specified distance).
If a point does not fall within any of the potential clusters then it is deemed as an outlier on the basis that it does not fit within the existing density or cluster of points. This can be implemented as:
#import the algorithmfrom sklearn.cluster import DBSCAN#initiate the algorithm#set the distance to 20, and min_samples as 5outlier_detection = DBSCAN(eps = 20, metric = "euclidean", min_samples = 10, n_jobs = -1)#fit_predict the algorithm to the existing dataclusters = outlier_detection.fit_predict(pokemon[["attack", "defense"]])#extract the labels from the algorithmpokemon["dbscan_outliers"] = clusters#label all others as inliers pokemon["dbscan_outliers"] = pokemon["dbscan_outliers"].apply(lambda x: str(1) if x>-1 else str(-1))#print the vaue countsprint(pokemon["dbscan_outliers"].value_counts())# Out:1    787-1   14
With the results as:
The key difference here is that only the upper edges of both attributes were selected as outliers from this algorithm, similar to that of the Tukey Fences. This is because they will be beyond the reach of core points where points in the lower left will be within the minimum distance of core points.
The benefit of this is that it can be used when the distribution of values in the feature space cannot be assumed, it is easily implemented in Sklearn and it is intuitive to understand. But, selecting the optimal parameters is often difficult and it struggles to be used in a predictive capacity.
Elliptic envelope
Elliptic envelope is another anomaly detection algorithm but one that assumes a Gaussian distribution as part of the data. This works by creating an imaginary elliptical area around a given dataset where values falling inside that ellipses are taken to be normal data, and anything falling outside that are assumed to be outliers.
Implementation of this model, as for Isolation Forest, requires the specification of the contamination parameter to suggest how many outliers are to be expected for the model. As before, we can set this to 0.02 and this can be implemented as:
#import the necessary library and functionalityfrom sklearn.covariance import EllipticEnvelope#create the model, set the contamination as 0.02EE_model = EllipticEnvelope(contamination = 0.02)#implement the model on the dataoutliers = EE_model.fit_predict(pokemon[["attack", "defense"]])#extract the labelspokemon["EE_outliers"] = outliers#change the labelspokemon["EE_outliers"] = pokemon["EE_outliers"].apply(lambda x: str(-1) if x == -1 else str(1))#extract the scorepokemon["EE_scores"] = EE_model.score_samples(pokemon[["attack", "defense"]])#print the value counts for inlier and outliersprint(pokemon["EE_outliers"].value_counts())#out:1   785-1  16
We can then visualise the results and scores as follows:
We can see here that the ellipses has been identified surrounding the centre of the distribution of points (i.e. the centre mass). The score itself is the negative Mahalanobis distance which is the inverse of the measure of distance between the point and the distribution (the ellipses). Thus, points further away from the distribution receive lower scores.
A primary difference between the points detected by the algorithm as compared to before is that points towards the centre left have not been identified as outliers here, suggesting these these have been counted within the ellipses.
Difficulty with this method include the requirement of normality in distribution of the variables and that as before we do not know the exact value of the contamination parameter. However, assuming normality, as before univariate analysis can be undertaken on the scores to identify outliers.
Ensemble
Finally, one way to strengthen the confidence in these methods is to not only use a single one but to combine the predictions of all methods. This can be done as:
#extract the sum of the outlier countpokemon['outliers_sum'] = (pokemon['iso_forest_outliers'].astype(int)+                           pokemon['lof_outliers'].astype(int)+                           pokemon['dbscan_outliers'].astype(int)+                          pokemon['EE_outliers'].astype(int))#print the value counts for each scaleprint(pokemon["outliers_sum"].value_counts())# out: 4    758 2     24 0      9-4      8-2      2
Where from this only 8 were identified as outliers in all algorithms. We can thus visualise this as:
Where we can see where predictions from different algorithms overlap. At this point, it is subjective judgement as to which algorithm to trust, whether that is a single one or multiple, and will depend on domain expertise in that particular context.
From this example, the eight Pokémon identified are given as:
Where the ensemble results are taken to be the key metric.
Beyond this, other algorithms can also be used including: One Class SVM, PCA and autoencoders. The choice of this of course will depend on the domain, the tools available and the dimensionality of the dataset. However, these results show that not all algorithms will produce the same result and that some subjective judgement will be required.
Code available: https://github.com/PhilipDW183/Outlier_detection
Dataset available: https://www.kaggle.com/rounakbanik/pokemon?select=pokemon.csv