Fast, Accurate Anomaly Detection Based on Copulas (COPOD)
By: Alexandra Amidon
Tags: Machine Learning, Outlier Detection, Anomaly Detection, Unsupervised Learning, Artificial Intelligence

Copula-Based Outlier Detection — COPOD — is a new algorithm for anomaly detection. It has been implemented in the PyOD python package.

It has several key features that make it stand out from competing algorithms:

Deterministic

No hyperparameters! (This is important because is difficult to tune hyperparameters for outlier detection tasks because the true labels are rare, unknown, or difficult to obtain)

Top-performance on benchmark anomaly detection datasets

Interpretable and easy to visualize anomalies

Fast and computationally efficient

Scales to high-dimensional datasets

The biggest challenge of COPOD is the complex mathematics behind the algorithm. This article contains two sections:

Summary of the key concepts of copulas and description of the COPOD algorithm. Tutorial of COPOD in python, showing how you can easily detect anomalies in data with this method.

Copula Models for Anomaly Detection

Copulas defined

What is a copula?

Simply stated, a copula describes the dependence structure between random variables.

Specifically, “a copula is a multivariate cumulative distribution function for which the marginal probability distribution of each variable is uniform on the interval [0, 1]” (Wikipedia).

In other words, you can describe the probability distributions in a high-dimensional dataset by separating out the marginal probability distribution for each variable from the copula, which is the multivariate probability distribution describing the dependence between variables.

“Copulas are functions that enable us to separate marginal distributions from the dependency structure of a given multivariate distribution.”

The COPOD algorithm

The COPOD algorithm does several mathematical estimations and transformations to compute an outlier score for each row.

Algorithm summary. Described in detail below.

Step 1: For each dimension (aka column) in a dataset —

Fit left tail empirical cumulative distribution function Fit right tail empirical cumulative distribution function Compute skewness coefficient

Step 2: Using the functions fitted in step 1, compute

the left tail empirical copula the right tail empirical copula the skewness-corrected empirical copula values for each row

Basically, this results in 3 new values per scalar value (row+column) in the data frame.

The copula values can be interpreted as left tail, right tail, and skewness-corrected probabilities, respectively.

Step 3: Compute anomaly scores for each row of your dataset based on the values computed in step 2. This step relies on the fact that smaller tail probabilities (aka couplas) result in larger negative-log values.

For each row,

sum the negative log of left tail empirical copula

sum the negative log of right tail empirical copula

sum the negative log of skewness corrected empirical copula

The anomaly score for a row is the maximum value of these sums.

Outlier scores

Interpretation: A higher anomaly score indicates that there is a low probability of the data instance because it lies on the tail of the data distribution.

Outlier scores lie between (0, ∞) and are directly comparable. They do not directly indicate outlier probability. Instead, they measure the likelihood of a row, relative to other points in the dataset.

To get outlier predictions from COPOD, there are two options:

Set a threshold on the outlier scores; any row with a score that exceeds the threshold is an anomaly Select to top-k or top-kth-percentile outlier scores.

(In PyOD , thresholds are automatically set based on the given contamination , or expected portion of anomalies in the training data. More on this later.)

Python Tutorial

COPOD is implemented in the PyOD python and is very easy to use.

First, load or generate data for outlier detection.

Here, I use the generate_data function from PyOD to generate a synthetic dataset with 200 training samples and 100 test samples. The normal samples are generated by a multivariate Gaussian distribution; the outlier samples are generated using a uniform distribution.

Both train and test datasets have 5 features and 10% of rows are labeled as anomalies. I add a bit of random noise to the data to make it slightly harder to perfectly separate normal and outlier points.

Below, I plot the train and test datasets, projected onto 2 dimensions using PCA.

Synthetic train and test datasets, reduced to 2 dimensions by PCA.

Training and Detecting

Next, I fit COPOD on the training data and evaluate on the test data.

It took less than 1 second to run the fit and decision_function methods.

In PyOD, a (fitted) outlier detector has two key functions: decision_function and predict .

decision_function returns an anomaly score for each row.

returns an anomaly score for each row. predict returns an array of 0’s and 1’s, indicating whether each row is predicted to be normal (0) or an outlier (1). The predict function simply applies a threshold to the anomaly score returned by the decision_function . The threshold is automatically calibrated based on the specified contamination rate parameter set when initializing the detector (e.g. clf=COPOD(contamination=0.1 ). The contamination indicates the expected percentage of outliers in the training data.

COPOD does not require training data!

In the real world, you might not have labeled training data — that’s okay! You can still use COPOD to detect outliers.

As you see in the example above, I initialized the COPOD algorithm and simply passed data through the decision_function without fitting. The outlier scores of outlier points were generally much greater than those for inliers, or normal data points.

In application, it is usually useful to select thresholds for COPOD anomaly scores in order to identify which data to investigate as anomalies.

Cool Feature: Anomaly Explanation

COPOD can explain which feature(s) contributed most to the outlier score. This is especially helpful when there are many features in your dataset or you want to tell a human reviewer why the algorithm selected a particular row as an outlier.

Dimensional Outlier Graphs for rows #182 and #193.

The two Dimensional Outlier Graphs above plot the feature-level outlier scores for two true outliers (rows) detected as outliers by COPOD with the blue line. The x-axis indicates the feature and the y-axis indicates the outlier score for that feature alone. The 90th and 99th percentile outlier scores are also plotted.

In the left plot (row #182), the outlier score for dimension 3 exceeds the 99th percentile, meaning it is the main reason the row is an outlier.

In the right plot (row #193), the outlier scores for dimensions 1 and 2 both exceed the 99th percentile and are both reasons the row is an outlier.

Thank you for reading!

COPOD is a fast, powerful, and easy-to-use anomaly detection algorithm. I hope it finds a place in your outlier detection toolbox.

To learn more about PyOD and anomaly detection, check out these other articles.

Not a Medium Member? Join Today!

References

[2009.09463] COPOD: Copula-Based Outlier Detection (arxiv.org)

pyod/copod_example.py at master · yzhao062/pyod · GitHub