Towards a Unified Framework of Clustering-based Anomaly Detection

## Abstract

Abstract Unsupervised Anomaly Detection (UAD) plays a crucial role in identifying abnormal patterns within data without labeled examples, holding significant practical implications across various domains.
Although the individual contributions of representation learning and clustering to anomaly detection are well-established, their interdependencies remain under-explored due to the absence of a unified theoretical framework. Consequently, their collective potential to enhance anomaly detection performance remains largely untapped.
To bridge this gap, in this paper, we propose a novel probabilistic mixture model for anomaly detection to establish a theoretical connection among representation learning, clustering, and anomaly detection.
By maximizing a novel anomaly-aware data likelihood, representation learning and clustering can effectively reduce the adverse impact of anomalous data and collaboratively benefit anomaly detection. Meanwhile, a theoretically substantiated anomaly score is naturally derived from this framework. Lastly, drawing inspiration from gravitational analysis in physics, we have devised an improved anomaly score that more effectively harnesses the combined power of representation learning and clustering.
Extensive experiments, involving 17 baseline methods across 30 diverse datasets, validate the effectiveness and generalization capability of the proposed method, surpassing state-of-the-art methods.

## 1 Introduction

Unsupervised Anomaly Detection (UAD)
refers to the task dedicated to identifying abnormal patterns or instances within data in the absence of labeled examples Chandola et al. ().
It has long received extensive attention in the past decades for its wide-ranging applications in numerous practical scenarios, including
financial auditing Bakumenko and Elragal (), healthcare monitoring Salem et al. () and e-commerce sector Kou et al. ().
Due to the lack of explicit label guidance, the key to UAD is to uncover the dominant patterns that widely exist in the dataset so that samples do not conform to these patterns can be recognized as anomalies.
To achieve this, early works Chalapathy and Chawla () have heavily relied on powerful unsupervised representation learning methods to extract the normal patterns from high-dimensional and complex data such as images, text, and graphs.
More recent works Song et al. (); Aytekin et al. () have utilized clustering, a widely observed natural pattern in real-world data, to provide critical global information for anomaly detection and achieved tremendous success.

Figure: Figure 1: Interdependent relationships among representation learning, clustering, and anomaly detection.
Refer to caption: extracted/5632434/images/fig1.png

While the individual contributions of representation learning and clustering to anomaly detection are well-established, their interrelationships remain largely unexplored. Intuitively, discriminative representation learning can leverage accurate clustering results to differentiate samples from distinct clusters in the embedding space (i.e., ➀). Similarly, it can utilize accurate anomaly detection to avoid preserving abnormal patterns (i.e., ➁).
For accurate clustering, it can gain advantages from representation learning by operating in the discriminative embedding space (i.e., ➂). Meanwhile, it can potentially benefit from accurate anomaly detection by excluding anomalies when formulating clusters (i.e., ➃).
Anomaly detection can greatly benefit from both discriminative representation learning and accurate clustering (i.e., ➄ & ➅). However, these benefits hinge on the successful identification of anomalies and the reduction of their detrimental impact on the aforementioned tasks.
As depicted in Figure 1, the integration of these three elements exhibits a significant reciprocal nature. In summary, representation learning, clustering, and anomaly detection are interdependent and intricately intertwined. Therefore, it is crucial for anomaly detection to fully leverage and mutually enhance the relationships among these three components.

Despite the intuitive significance of the interactions among representation learning, clustering, and anomaly detection, existing methods have only made limited attempts to exploit them and fall short of expectations. On one hand, some methods Zong et al. () have acknowledged the interplay among these three factors, but their focus remains primarily on the interactions between two factors at a time, making only targeted improvements. For instance, some strategies include explicitly removing outlier samples during the clustering process Chawla and Gionis () or designing robust representation learning methods Cho et al. () to mitigate the influence of anomalies.
On the other hand, recent methods Song et al. () have begun to explore the simultaneous optimization of these three factors within a single framework. However, these attempts are still in the stage of merely superimposing the objectives of the three factors without a unified theoretical framework. This lack of a guiding framework prevents the adequate modeling of the interdependencies among these factors, thereby limiting their collective contribution to a unified anomaly detection objective.
Consequently, we aim to address the following question: Is it possible to employ a unified theoretical framework to jointly model these three interdependent objectives, thereby leveraging their respective strengths to enhance anomaly detection?

In this paper, we try to answer this question and propose a novel model named UniCAD for anomaly detection.
The proposed UniCAD integrates representation learning, clustering, and anomaly detection into a unified framework, achieved through the theoretical guidance of maximizing the anomaly-aware data likelihood.
Specifically, we explicitly model the relationships between samples and multiple clusters in the representation space using the probabilistic mixture models for the likelihood estimation.
Moreover, we creatively introduce a learnable indicator function into the objective of maximum likelihood to explicitly attenuate the influence of anomalies on representation learning and clustering.
Under this framework, we can theoretically derive an anomaly score that indicates the abnormality of samples, rather than heuristically designing it based on clustering results as existing works do.
Furthermore, building upon this theoretically supported anomaly score and inspired by the theory of universal gravitation, we propose a more comprehensive anomaly metric that considers the complex relationships between samples and multiple clusters. This allows us to better utilize the learned representations and clustering results from this framework for anomaly detection.

To sum up, we underline our contributions as follows:

- •
We propose a unified theoretical framework to jointly optimize representation learning, clustering, and anomaly detection, allowing their mutual enhancement and aid in anomaly detection.
- •
Based on the proposed framework, we derive a theoretically grounded anomaly score and further introduce a more comprehensive score with the vector summation, which fully releases the power of the framework for effective anomaly detection.
- •
Extensive experiments have been conducted on 30 datasets to validate the superior unsupervised anomaly detection performance of our approach, which surpassed the state-of-the-art through comparative evaluations with 17 baseline methods.

## 2 Related Work

Typical unsupervised anomaly detection (UAD) methods calculate a continuous score for each sample to measure its anomaly degree.
Various UAD methods have been proposed based on different assumptions, making them suitable for detecting various types of anomaly patterns,
including
subspace-based models Kriegel et al. (),
statistical models Goldstein and Dengel (),
linear models Wold et al. (); Manevitz and Yousef (),
density-based models Breunig et al. (); Peterson (),
ensemble-based models Pevnỳ (); Liu et al. (),
probability-based models Reynolds et al. (); Zong et al. (); Li et al. (, ),
neural network-based models Ruff et al. (); Xu et al. (),
and cluster-based models He et al. (); Chawla and Gionis ().
Considering the field of anomaly detection has progressed by integrating clustering information to enhance detection accuracy Li et al. (); Zhou et al. (), we primarily focus on and analyze anomaly patterns related to clustering, incorporating a global clustering perspective to assess the degree of anomaly.
Notable methods in this context include CBLOF He et al. (), which evaluates anomalies based on the size of the nearest cluster and the distance to the nearest large cluster.
Similarly, DCFOD Song et al. () introduces innovation by applying the self-training architecture of the deep clustering Xie et al. () to outlier detection.
Meanwhile, DAGMM Zong et al. () combines deep autoencoders with Gaussian mixture models, utilizing sample energy as a metric to quantify the anomaly degree.
In contrast, our approach introduces a unified theoretical framework that integrates representation learning, clustering, and anomaly detection, overcoming the limitations of heuristic designs and the overlooked anomaly influence in existing methods.

## 3 Methodology

In this section, we first define the problem we studied and the notations used in this paper.
Then we elaborate on the proposed method UniCAD.
More specifically, we first introduce a novel learning objective that optimizes representation learning, clustering, and anomaly detection within a unified theoretical framework by maximizing the data likelihood.
A novel anomaly score with theoretical support is also naturally derived from this framework.
Then, inspired by the concept of universal gravitation, we further propose an enhanced anomaly scoring approach that leverages the intricate relationship between samples and clustering to detect anomalies effectively.
Finally, we present an efficient iterative optimization strategy to optimize this model and provide a complexity analysis for the proposed model.

###### Definition 1 (Unsupervised Anomaly Detection) .

Given a dataset $\mathbf{X}\in\mathbb{R}^{N\times D}$ comprising $N$ instances with $D$-dimensional features, unsupervised anomaly detection aims to learn an anomaly score ${o}_{i}$ for each instance $\mathbf{x}_{i}$ in an unsupervised manner so that the abnormal ones have higher scores than the normal ones.

###### Definition 1 (Unsupervised Anomaly Detection) .

### 3.1 Maximizing Anomaly-aware Likelihood

Previous research has demonstrated the importance of discriminative representation and accurate clustering in anomaly detection Song et al. ().
However, the presence of anomalous samples can significantly disrupt the effectiveness of both representation learning and clustering Duan et al. ().
While some existing studies have attempted to integrate these three separate learning objectives, the lack of a unified theoretical framework has hindered their mutual enhancement, leading to suboptimal results.

To tackle this issue, in this paper, we propose a unified and coherent approach that considers representation learning, clustering, and anomaly detection by maximizing the likelihood of the observed data.
Specifically, we denote the parameters of representation learning as $\Theta$, the clustering parameter as $\Phi$, and the dynamic indicator function for anomaly detection as $\delta(\cdot)$.
These parameters are optimized simultaneously by maximizing the likelihood of the observed data $\mathbf{X}$:

$$ $\max\log p(\mathbf{X}|\Theta,\Phi)=\max\sum_{i=1}^{N}\delta(\mathbf{x}_{i}) \log p(\mathbf{x}_{i}|\Theta,\Phi)=\max\sum_{i=1}^{N}\delta(\mathbf{x}_{i}) \log\sum_{k=1}^{K}p(\mathbf{x}_{i},{c}_{i}=k|\Theta,\Phi),$ (1) $$

where ${c}_{i}$ represents the latent cluster variable associated with $\mathbf{x}_{i}$, and ${c}_{i}=k$ denotes the probabilistic event that $\mathbf{x}_{i}$ belongs to the $k$-th cluster.
The $\delta(\mathbf{x}_{i})$ is an indicator function that determines whether a sample $\mathbf{x}_{i}$ is an anomaly of value 0 or a normal sample of value 1.

#### 3.1.1 Joint Representation Learning and Clustering with p ⁢ ( 𝐱 i | Θ , Φ ) 𝑝 conditional subscript 𝐱 𝑖 Θ Φ p(\mathbf{x}_{i}|\Theta,\Phi) italic_p ( bold_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | roman_Θ , roman_Φ )

Based on the aforementioned advantages of MMs, we estimate the likelihood $p(\mathbf{x}_{i}|\Theta,\Phi)$ with mixture models defined as:

$$ $\begin{split}p(\mathbf{x}_{i}|\Theta,\Phi)=\sum_{k=1}^{K}p(\mathbf{x}_{i},{c}_ {i}=k|\Theta,\Phi)&=\sum_{k=1}^{K}p({c}_{i}=k)\cdot p(\mathbf{x}_{i}|{c}_{i}=k ,\Theta,\bm{\mu}_{k},\Sigma_{k})\\ &=\sum_{k=1}^{K}\omega_{k}\cdot p(\mathbf{x}_{i}|{c}_{i}=k,\Theta,\bm{\mu}_{k} ,\Sigma_{k}),\end{split}$ (2) $$

where $\Phi=\{\omega_{k},\bm{\mu}_{k},\Sigma_{k}\}$.
The mixture model is parameterized by the prototypes $\bm{\mu}_{k}$, covariance matrices $\Sigma_{k}$, and mixture weights $\omega_{k}$ from all clusters.
$\sum_{k=1}^{K}\omega_{k}=1$, and $k=1,2,\cdots,K$.

In practice, the samples are usually attributed to high-dimensional features and it is challenging to detect anomalies from the raw feature space Ruff et al. ().
Therefore, modern anomaly detection methods Ruff et al. (); Zong et al. () often map raw data samples $\mathbf{X}=\{\mathbf{x}_{i}\}\in\mathbb{R}^{N\times D}$ into a low-dimensional representation space $\mathbf{Z}=\{\mathbf{z}_{i}\}\in\mathbb{R}^{N\times d}$ with a representation learning function $\mathbf{z}_{i}=f_{\Theta}(\mathbf{x}_{i})$ and detect anomalies within this latent representation space.

Following this widely adopted practice, we model the distribution of samples in the latent representation space with a multivariate Student’s-$t$ distribution giving its cluster ${c}_{i}=k$.
The Student’s-$t$ distribution is robust against outliers due to its heavy tails.
Bayesian robustness theory leverages such distributions to dismiss outlier data, favoring reliable sources, making the Student’s-$t$ process preferable over Gaussian processes for data with atypical information Andrade ().
Thus the probability distribution of generating $\mathbf{x}_{i}$ with latent representation $\mathbf{z}_{i}$ given its cluster ${c}_{i}=k$ can be expressed as:

$$ $p(\mathbf{x}_{i}|{c}_{i}=k,\Theta,\bm{\mu}_{k},\Sigma_{k})=\frac{\Gamma(\frac{ \nu+1}{2})|\Sigma_{k}|^{-1/2}}{\Gamma(\frac{\nu}{2})\sqrt{\nu\pi}}\left(1+ \frac{1}{\nu}D_{M}(\mathbf{z}_{i},\bm{\mu}_{k})^{2}\right)^{-\frac{\nu+1}{2}},$ (3) $$

where $\mathbf{z}_{i}=f_{\Theta}(\mathbf{x}_{i})$ denotes the representation obtained from the data mapped through the neural network parameterized by $\Theta$.
$\Gamma$ denotes the gamma function while $\nu$ is the degree of freedom.
$\Sigma_{k}$ is the scale parameter.
$D_{M}(\mathbf{z}_{i},\bm{\mu}_{k})=\sqrt{(\mathbf{z}_{i}-\bm{\mu}_{k})^{T}
\Sigma_{k}^{-1}(\mathbf{z}_{i}-\bm{\mu}_{k})}$ represents the Mahalanobis distance McLachlan ().
In the unsupervised setting, as cross-validating $\nu$ on a validation set or learning it is unnecessary, $\nu$ is set as 1 for all experiments Xie et al. (); Van Der Maaten ().
The overall marginal likelihood of the observed data $\mathbf{x}_{i}$ can be simplified as:

$$ $p(\mathbf{x}_{i}|\Theta,\Phi)=\sum_{k=1}^{K}\omega_{k}\cdot\frac{\pi^{-1}\cdot |\Sigma_{k}|^{-1/2}}{1+D_{M}(\mathbf{z}_{i},\bm{\mu}_{k})^{2}}.$ (4) $$

#### 3.1.2 Anomaly Indicator δ ⁢ ( 𝐱 i ) 𝛿 subscript 𝐱 𝑖 \delta(\mathbf{x}_{i}) italic_δ ( bold_x start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) and Score o i subscript 𝑜 𝑖 {o}_{i} italic_o start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT

As we have discussed, the indicator function $\delta(\mathbf{x}_{i})$ not only benefits both representation and clustering but also directly serves as the output of anomaly detection.
Ideally, with the percentage of outliers denoted as $l$, an optimal solution for $\delta(\mathbf{x}_{i})$ that maximizes the objective function $J(\Theta,\Phi)$ entails setting all $\delta(\mathbf{x}_{i})=0$ for $\mathbf{x}_{i}$ among the $l$ percent of outliers with lowest generation possibility $p(\mathbf{x}_{i}|\Theta,\Phi)$, and otherwise $\delta(\mathbf{x}_{i})=1$ is set for the remaining normal samples.
Therefore, the indicator function is determined as:

$$ $\delta(\mathbf{x}_{i})=\begin{cases}0,&\text{if }p(\mathbf{x}_{i}|\Theta,\Phi) \text{ is among the }l\ \text{lowest},\\ 1,&\text{otherwise}.\end{cases}$ (5) $$

As this method involves sorting the samples based on the generation probability as being anomalous, the values of $p(\mathbf{x}_{i}|\Theta,\Phi)$ can serve as a form of anomaly score, a classic approach within the mixture model framework Reynolds et al. (); Zong et al. ().
This suggests that the likelihood of a sample being anomalous is inversely related to its generative probability since a lower generative probability indicates a higher chance of the sample being an outlier.
Thus the anomaly score of sample $\mathbf{x}_{i}$ can be defined as:

$$ ${o}_{i}=\frac{1}{p(\mathbf{x}_{i}|\Theta,\Phi)}=\frac{1}{\sum_{k=1}^{K}\omega_ {k}\cdot\frac{\pi^{-1}\cdot|\Sigma_{k}|^{-1/2}}{1+D_{M}(\mathbf{z}_{i},\bm{\mu }_{k})^{2}}}.$ (6) $$

### 3.2 Gravity-inspired Anomaly Scoring

In practical applications, it is proved that anomaly scores derived from generation probabilities often yield suboptimal performance Han et al. ().
This observation prompts a reconsideration of how to fully leverage the complex relationships among samples or even across multiple clusters for anomaly detection.
In this section, we first provide a brief introduction to the concept of Newton’s Law of Universal Gravitation Newton () and then demonstrate how the anomaly score is intriguingly similar to this cross-field principle.
Finally, we discuss the advantages of introducing the vector sum operation into the anomaly score inspired by the analogy.

#### 3.2.1 Analog Anomaly Scoring and Force Analysis

To begin with, Newton’s Law of Universal Gravitation Newton () stands as a fundamental framework for describing the interactions among entities in the physical world.
According to this law, every object in the universe experiences an attractive force from another object.
In classical mechanics, force analysis involves calculating the vector sum of all forces acting on an object, known as the resultant force, which is crucial in determining an object’s acceleration or change in motion:

$$ $\vec{\mathbf{F}}_{i,\text{total}}=\sum_{k=1}^{K}\vec{\mathbf{F}}_{ik},\ \ \text{with}\ \vec{\mathbf{F}}_{ik}=\frac{G\cdot m_{i}m_{k}}{r_{ik}^{2}}\cdot \vec{\mathbf{r}}_{ik},$ (7) $$

where $\vec{\mathbf{F}}_{ik}$ represents the $k$-th force acting on the object $i$.
This force is proportional to the product of their masses, ($m_{i}$ and $m_{k}$), and inversely proportional to the square of the distance $r_{ik}$ between them.
$G$ represents the gravitational constant, and $\vec{\mathbf{r}}_{ij}$ is the unit direction vector.

Similarly, if denoting:
$\widetilde{\mathbf{F}}_{ik}=p(\mathbf{x}_{i},{c}_{i}=k|\Theta,\Phi)=\omega_{k}
\cdot\frac{\pi^{-1}\cdot|\Sigma_{k}|^{-1/2}}{1+D_{M}(\mathbf{z}_{i},\bm{\mu}_{
k})^{2}},$
the score of Equation (6) bears analogies to the summation of the magnitudes of forces as:

$$ ${o}_{i}=\frac{1}{\sum_{k=1}^{K}\widetilde{\mathbf{F}}_{ik}},\ \ \text{with}\ \widetilde{\mathbf{F}}_{ik}=\frac{\widetilde{G}\cdot\widetilde{m}_{i} \widetilde{m}_{k}}{\widetilde{r}_{ik}^{2}},$ (8) $$

where
$\widetilde{G}=\pi^{-1}$,
$\widetilde{m}_{k}=\omega_{k}|\Sigma_{k}|^{-1/2}$,
$\widetilde{m}_{i}=1$, and
$\widetilde{r}_{ik}=\sqrt{1+D_{M}(\mathbf{z}_{i},\bm{\mu}_{k})^{2}}$.
Here, $\widetilde{r}_{ik}$ is taken as the measure of distance within the representation space, modified slightly by an additional term for smoothness.
The constant $\widetilde{G}$ serves a role akin to the gravitational constant in this analogy, whereas $\widetilde{m}_{k}$ resembles the concept of mass for the cluster.
The notation $\widetilde{m}_{i}$ suggests a standardization where the mass of each data point is considered uniform and not differentiated.

#### 3.2.2 Anomaly Scoring with Vector Sum

Comparing Equation (7) with Equation (8), what still differs is that, unlike a simple sum of the scalar value,
the resultant force $\vec{\mathbf{F}}_{i,\text{total}}$
employs the vector sum and incorporates both the magnitude
and direction $\widehat{\mathbf{r}}_{ik}$ of each force.
This distinction is crucial because forces in different directions can neutralize each other with a large angle between them or enhance each other’s effects with a small angle.
Inspired by this difference, we consider modeling the relationship between samples and clusters as a vector, and aggregating them through vector summation.
The vector-formed anomaly score ${o}_{i}^{V}$ is defined as:

$$ ${o}_{i}^{V}=\frac{1}{\|\sum_{k=1}^{K}\widetilde{\mathbf{F}}_{ik}\cdot\vec{ \mathbf{r}}_{ik}\|},$ (9) $$

where
$\vec{\mathbf{r}}_{ik}$ represents the unit direction vector in the representation space from the sample $\mathbf{z}_{i}$ to the cluster prototype $\bm{\mu}_{k}$
, and $\|\cdot\|$ represents the $L_{2}$ norm.

### 3.3 Iterative Optimization

Given the challenge posed by the interdependence of the parameters of the network $\Theta$ and those of the mixture model $\{\omega_{k},\bm{\mu}_{k},\Sigma_{k}\}$ in joint optimization, we propose an iterative optimization procedure.
The pseudocode for training the model is presented in Algorithm 1 in the appendix.

#### 3.3.1 Update Φ Φ \Phi roman_Φ

To update the parameters of the mixture model $\Phi=\{\omega_{k},\bm{\mu}_{k},\Sigma_{k}\}$, we use the Expectation-Maximization (EM) algorithm to maximize equation (1) Peel and McLachlan ().
The detailed derivation is included in Appendix B.

E-step.
During the E-step of iteration $(t+1)$, our goal is to compute the posterior probabilities
of each data point belonging to the $k$-th cluster within the mixture model.
Given the observed sample $\mathbf{x}_{i}$ and the current estimates of the parameters $\Theta^{(t)}$ and $\Phi^{(t)}$, the expected value of the likelihood function of latent variable ${c}_{k}$, or the posterior possibilities, can be expressed as:

$$ $\begin{split}\bm{\tau}_{ik}^{(t+1)}=p({c}_{i}=k|\mathbf{x}_{i},\Theta,\Phi^{(t )})=\frac{p(\mathbf{x}_{i},{c}_{i}=k|\Theta,\Phi^{(t)})}{\sum_{j=1}^{K}p( \mathbf{x}_{i},{c}_{i}=j|\Theta,\Phi^{(t)})}=\frac{{\widetilde{\mathbf{F}}}_{ ik}^{(t)}}{\sum_{j=1}^{K}\widetilde{\mathbf{F}}_{ij}^{(t)}}.\end{split}$ (10) $$

The scale factorPeel and McLachlan () serving as an intermediate result for subsequent updates in the M-step is
:

$$ $\mathbf{u}_{ik}^{(t+1)}=\frac{2}{1+D_{M}(\mathbf{z}^{(t)}_{i},\bm{\mu}^{(t)}_{ k})}.$ (11) $$

M-step.
In the M-step of iteration $(t+1)$, given the gradients $\frac{\partial J(\Theta,\Phi)}{\partial\omega_{k}}=0$, $\frac{\partial J(\Theta,\Phi)}{\partial\bm{\mu}_{k}}=0$, and $\frac{\partial J(\Theta,\Phi)}{\partial\Sigma_{k}}=0$, we derive the analytical solutions for the mixture model parameters $\omega_{k}$, $\bm{\mu}_{k}$, and $\Sigma_{k}$.
Assume the anomalous ratio is $l\in[0,1]$, the number of the normal samples is $n=\text{int}(l*N)$.
The updating process for $\{\omega_{k}^{(t+1)},\bm{\mu}_{k}^{(t+1)},\bm{\Sigma}_{k}^{(t+1)}\}$ is as follows:

- •
The mixture weights $\omega_{k}$ are updated by averaging the posterior probabilities over all data points with the number of samples
, reflecting the relative presence of each component in the mixture:
$\omega_{k}^{(t+1)}=\sum_{i=1}^{n}\bm{\tau}_{ik}^{(t+1)}/n.$
(12)
- •
The prototypes $\bm{\mu}_{k}$ are updated to be the weighted average of the data points, where weights are the posterior probabilities:
$\bm{\mu}_{k}^{(t+1)}=\sum_{i=1}^{n}\left(\bm{\tau}_{ik}^{(t+1)}\mathbf{u}_{ik}
^{(t+1)}\mathbf{z}_{i}\right)/\sum_{i=1}^{n}\left(\bm{\tau}_{ik}^{(t+1)}
\mathbf{u}_{ik}^{(t+1)}\right).$
(13)
- •
The covariance matrices $\Sigma_{k}$ are updated by considering the dispersion of the data around the newly computed prototypes:
$\bm{\Sigma}_{k}^{(t+1)}=\frac{\sum_{i=1}^{n}\bm{\tau}_{ik}^{(t+1)}\mathbf{u}_{
ik}^{(t+1)}(\mathbf{z}_{i}-\bm{\mu}_{k}^{(t+1)})(\mathbf{z}_{i}-\bm{\mu}_{k}^{
(t+1)})^{\intercal}}{\sum_{j=1}^{K}\bm{\tau}_{ij}^{(t+1)}}.$
(14)

#### 3.3.2 Update Θ Θ \Theta roman_Θ

We focus on anomaly-aware representation learning and use stochastic gradient descent to optimize the network parameters $\Theta$, by minimizing the following joint loss:

$$ $\mathcal{L}=-J(\Theta,\Phi)+g(\Theta),$ (15) $$

where $J(\Theta,\Phi)=\log p(\mathbf{X}|\Theta,\Phi)$.
An additional constraint term $g(\Theta)$ is introduced to prevent shortcut solution Geirhos et al. ().
In practice, an autoencoder architecture is implemented, utilizing a reconstruction loss $g(\Theta)=\|x-\hat{x}\|^{2}$ as the constraint.

These updates are iteratively performed until convergence, resulting in optimized model parameters that best fit the given data according to the mixture model framework.

## 4 Experiments

### 4.1 Datasets & Baselines

We evaluated UniCAD on an extensive collection of datasets, comprising 30 tabular datasets that span 16 diverse fields. We specifically focused on naturally occurring anomaly patterns, rather than synthetically generated or injected anomalies, as this aligns more closely with real-world scenarios.
The detailed descriptions are provided in Table 4 of Appendix D.1.
Following the setup in ADBench Han et al. (), we adopt an inductive setting to predict newly emerging data, a highly beneficial approach for practical applications.

To assess the effectiveness of UniCAD, we compared it with 17 advanced unsupervised anomaly detection methods, including:
(1) traditional methods: SOD Kriegel et al. () and HBOS Goldstein and Dengel ();
(2) linear methods: PCA Wold et al. () and OCSVM Manevitz and Yousef ();
(3) density-based methods: LOF Breunig et al. () and KNN Peterson ();
(4) ensemble-based methods: LODA Pevnỳ () and IForest Liu et al. ();
(5) probability-based methods: DAGMM Zong et al. (), ECOD Li et al. (), and COPOD Li et al. ();
(6) cluster-based methods: DBSCAN Ester et al. (), CBLOF He et al. (), DCOD Song et al. () and KMeans-- Chawla and Gionis (); and
(7) neural network-based methods: DeepSVDD Ruff et al. () and DIF Xu et al. ().
These baselines encompass the majority of the latest methods, providing a comprehensive overview of the state-of-the-art.
For a detailed description, please refer to Appendix D.2.

### 4.2 Experiment Settings

In the unsupervised setting, we employ the default hyperparameters from the original papers for all comparison methods.
Similarly, the UniCAD also utilizes a fixed set of parameters to ensure a fair comparison.
For all datasets, we employ a two-layer MLP with a hidden dimension of $d=128$ and ReLU activation function as both encoder and decoder.
We utilize the Adam optimizer Kingma and Ba () with a learning rate of $1e^{-4}$ for 100 epochs.
For the EM process, we set the maximum iteration number to 100 and a tolerance of $1e^{-3}$ for stopping training when the objectives converge.
The number of components in the mixture model is set as $k=10$, and the proportion of the outlier is set as $l=1\%$.
We evaluate the methods using Area Under the Receiver Operating Characteristic (AUC-ROC) and Area Under the Precision-Recall Curve (AUC-PR) metrics Han et al. (), reporting the average ranking (Avg. Rank) across all datasets.
All experiments are run 3 times with different seeds, and the mean results are reported.

Table: Table 1: AUCROC of 10 unsupervised algorithms on 30 tabular benchmark datasets. In each dataset, the algorithm with the highest AUCROC is marked in red, the second highest in blue, and the third highest in green.

### 4.3 Performance and Analysis

Performance Comparison.
Table 1 presents a comparison of UniCAD with 10 unsupervised baseline methods across 30 tabular datasets using the AUC-ROC metric. The experimental results, which encompass 17 baselines, are included in Tables  5 and  6 of Appendix D.3, with additional experiments on other data domains presented in Appendix E.
Our proposed UniCAD achieves the top average ranking, exhibiting the best or near-best performance on a larger number of datasets and confirming advanced capabilities.
It is noteworthy that there is no one-size-fits-all unsupervised anomaly detection method suitable for every type of dataset, as demonstrated by the observation that other methods have also achieved some of the best results on certain datasets.
However, our model showcased a remarkable ability to generalize across most datasets featuring natural anomalies, as evidenced by statistical average ranking.
As for clustering-based methods such as KMeans–, DCOD, and CBLOF, they mostly rank in the top tier among all baseline methods, supporting the advantage of combining deep clustering with anomaly detection.
However, our method significantly outperformed these methods by mitigating their limitations and further providing a unified framework for joint representation learning, clustering, and anomaly detection.

Effectiveness of Vector Sum in Anomaly Scoring.
As demonstrated in Table 1, we compare the anomaly score $\mathbf{o}_{i}$ derived directly from the generation possibility with its vector summation form $\mathbf{o}_{i}^{V}$.
According to our statistical findings, we observe that vector scores $\mathbf{o}_{i}^{V}$ consistently outperform scalar scores $\mathbf{o}_{i}$.
This indicates that the introduction of the vector summation, analogous to the concept of resultant force, makes a substantial difference in anomaly detection scenarios involving multiple clusters.
The performance gains of the vector sum scores strongly demonstrate the effectiveness of the UniCAD in capturing the subtle differences in the distinctions among multiple clusters and underscore the utility of this factor in the context of anomaly detection based on clustering.

Figure: (a) Optimization Analysis
Refer to caption: extracted/5632434/images/iterations.png

Analysis of EM Iterative Optimization.
To comprehend the iterative training within our model, we have illustrated the performance variations accompanying the increase in iteration counts in Figure 2(a).
Specifically, we monitored the iteration number $t$ for the satimage-2 dataset, ranging from 0 to 10, while maintaining other default parameters constant.
Both AUC-ROC and AUC-PR performance curves displayed consistent trends, with minor fluctuations only during the initial phase.
The performance remained relatively stable throughout the last steps, illustrating the effectiveness and convergence of iterative EM optimization.

Runtime Comparison.
We present a analysis of the runtime performance of various methods, including our proposed approach, as detailed in Table 2. Our experiments, conducted on the backdoor dataset, reveal that while non-deep learning methods exhibit lower runtime, they often simplify the problem space excessively, failing to capture the complex non-linear relationships present in the data. In contrast, our method, when compared to existing deep learning techniques, demonstrates a significant reduction in computational time. This indicates that our approach not only manages to efficiently model complex patterns but also achieves an optimal balance between computational efficiency and modeling capability.

### 4.4 Ablation Studies

Table: Table 2: Runtime Comparison. The runtime is reported in seconds (s).

**Table 3: Ablation study on AUC-ROC scores, calculated across 30 datasets.**
| Metric | w/ Gauss. | w/o $J(\Theta,\Phi)$ | w/o $\delta(\mathbf{x}_{i})$ | Full Model |
| --- | --- | --- | --- | --- |
| Avg. Rank (w/ baselines & variants) | 6.2 | 6.6 | 5.0 | 4.2 |

In this section, we examine the contributions of different components in UniCAD.
Tables 3 reports the results. We make three major observations.
Firstly, the anomaly detection performance experiences a significant drop when replacing the Student’s t distribution with a Gaussian distribution for the Mixture Model, highlighting the robustness of the Student’s t distribution in unsupervised anomaly detection.
Secondly, omitting the likelihood maximization loss (w/o $J(\Theta,\Phi)$) also results in a considerable decrease in overall performance.
This observation underscores the importance of deriving both the optimization objectives and anomaly scores from the likelihood generation probability through a theoretical framework, which allows for unified joint optimization of anomaly detection and clustering in the representation space.
Furthermore, the indicator function $\delta(\mathbf{x}_{i})$ also contributes to a performance increase.
These results further confirm the effectiveness of our UniCAD in mitigating the negative influence of anomalies in the clustering process, as the existence of outliers may significantly degrade the performance of clustering.
In summary, all these ablation studies clearly demonstrate the effectiveness of our theoretical framework in simultaneously considering representation learning, clustering, and anomaly detection.

### 4.5 Sensitivity of Hyperparameters

In this section, we conducted a sensitivity analysis on key hyperparameters of the model applied to the donors dataset, focusing on the number of clusters $k$ and the proportion of the outlier set $l$.
The results of this analysis are illustrated in Figure 2.
Notably, the optimal range for $l$ tends to be lower than the actual proportion of anomalies in the dataset.
Furthermore, a pattern was observed with the number of clusters $k$, where the model performance initially improved with an increase in $k$, followed by a subsequent decline.
This suggests the existence of an optimal range for the number of clusters, which should be carefully selected based on the specific application context.

## 5 Conclusion

This paper presents UniCAD, a novel model for Unsupervised Anomaly Detection (UAD) that seamlessly integrates representation learning, clustering, and anomaly detection within a unified theoretical framework.
Specifically, UniCAD introduces an anomaly-aware data likelihood based on the mixture model with the Student-t distribution to guide the joint optimization process, effectively mitigating the impact of anomalies on representation learning and clustering. This framework enables a theoretically grounded anomaly score inspired by universal gravitation, which considers complex relationships between samples and multiple clusters.
Extensive experiments on 30 datasets across various domains demonstrate the effectiveness and generalization capability of UniCAD, surpassing 15 baseline methods and establishing it as a state-of-the-art solution in unsupervised anomaly detection.
Despite its potential, the proposed method’s applicability to broader fields like time series and multimodal anomaly detection requires further exploration and validation, highlighting a significant area for future work.

## Appendix A Iterative Training Algorithm

The pseudocode for training the model is presented in Algorithm 1.
Initially, all parameters undergo random initialization.
In subsequent iterations, following the initial round, the outlier set $L$ undergoes updates based on the anomaly score ${o}_{i}$.
This is succeeded by the adjustment of the network parameters $\Theta$ based on $\mathbf{x}_{i}$, further optimizing the performance of $\Theta$ through the utilization of the estimated parameters ${\bm{\mu}_{k},\omega_{k},\Sigma_{k}}$.
The essence of the algorithm is embedded in its alternating optimization strategy, iteratively refining the accuracy of representation learning and mixed model parameter estimation, thereby augmenting the overall training effectiveness of the model.

Figure: Algorithm 1 Model training for UniCAD

## Appendix B Derivation of EM Algorithm

This appendix provides the detailed derivation of the Expectation-Maximization (EM) algorithm for optimizing the parameters of a mixture model based on Student’s t-distribution. The focus is on deriving analytical solutions for the maximization of the parameters $\Phi=\{\bm{\mu}_{k},\Sigma_{k},\omega_{k}\}$
of the mixture components. The EM algorithm alternates between two steps:

In the E-step, we calculate the posterior probabilities $\bm{\tau}_{ik}$, representing the probability of data point $i$ belonging to cluster $k$, given the current parameters. The posterior probabilities for a Student’s t-distribution mixture model are formulated as:

$$ $\bm{\tau}_{ik}=\frac{\omega_{k}\cdot p(\mathbf{z}_{i}|\bm{\mu}_{k},\Sigma_{k}) }{\sum_{j=1}^{K}\omega_{j}\cdot p(\mathbf{z}_{i}|\bm{\mu}_{j},\Sigma_{j})},$ (16) $$

where $\bm{\tau}(\mathbf{z}_{i}|\bm{\mu}_{k},\Sigma_{k})$ denotes the Student’s t-distribution for data point $i$ with respect to cluster $k$, and $K$ is the number of mixture components.

The Student’s t-distribution is depicted as a hierarchical conditional probability, resembling a Gaussian distribution with an accuracy scale factor $\mathbf{u}$, where its latent variable follows a gamma distribution. Adopting a degree of freedom $\nu=1$, the value of $\mathbf{u}_{ik}$ is given by:

$$ $\mathbf{u}_{ik}=\frac{\nu+1}{\nu+D_{M}(z_{i},\bm{\mu}_{k})}=\frac{2}{1+D_{M}(z _{i},\bm{\mu}_{k})}$ (17) $$

In the M-step, we update the parameters $\Phi=\{\omega_{k}$, $\bm{\mu}_{k}$, and $\Sigma_{k}\}$ using the derivatives obtained in the previous steps.
In our model, the likelihood function for a Student’s-t Distribution Mixture Model (SMM) is represented as:

$$ $L(\omega,\bm{\mu},\Sigma)=\sum_{i=1}^{N}\sum_{k=1}^{K}\omega_{k}\cdot\frac{\pi ^{-1}\cdot|\Sigma_{k}|^{-\frac{1}{2}}}{1+(\mathbf{z}_{i}-\bm{\mu}_{k})^{T} \Sigma_{k}^{-1}(\mathbf{z}_{i}-\bm{\mu}_{k})},$ (18) $$

where $\omega_{k}$ are the mixture weights, $\Sigma_{k}$ the covariance matrices, $\bm{\mu}_{k}$ the means, and $\mathbf{z}_{i}$ the data points.

The derivative with respect to $\omega_{k}$ must consider the constraint that the sum of the mixture weights equals 1, i.e., $\sum_{k}\omega_{k}=1$. Hence, we introduce a Lagrange multiplier $\lambda$ to address this constraint and construct the Lagrangian $L^{\prime}$:

$$ $L^{\prime}(\omega,\bm{\mu},\Sigma,\lambda)=L(\omega,\bm{\mu},\Sigma)+\lambda \left(1-\sum_{k=1}^{K}\omega_{k}\right),$ (19) $$

The derivative with respect to $\omega_{k}$ is:

$$ $\frac{\partial L^{\prime}}{\partial\omega_{k}}=\frac{\partial L}{\partial \omega_{k}}-\lambda,$ (20) $$

Substituting the definition of $L(\omega,\bm{\mu},\Sigma)$, we obtain:

$$ $\frac{\partial L}{\partial\omega_{k}}=\sum_{i}\frac{p(\mathbf{z}_{i}|\bm{\mu}_ {k},\Sigma_{k})}{\sum_{j=1}^{K}\omega_{j}\cdot p(\mathbf{z}_{i}|\bm{\mu}_{j}, \Sigma_{j})}=\sum_{i}\frac{\bm{\tau}_{ik}}{\omega_{k}},$ (21) $$

Figure: Figure 3: Score comparison with other methods.
Refer to caption: extracted/5632434/images/explore_score_full_v2.png

To solve for $\omega_{k}$, we first multiply both sides of the equation by $\omega_{k}$ and apply the constraint condition:

$$ $\sum_{k}\omega_{k}\left(\sum_{i}\frac{\bm{\tau}_{ik}}{\omega_{k}}-\lambda \right)=0,$ (22) $$

Upon further organization, we find that the Lagrange multiplier $\lambda$ actually equals the total number of data points $N$ (since $\sum_{i}\bm{\tau}_{ik}=N_{k}$, where $N_{k}$ is the expected total number of data points belonging to the $k$th component, and the sum of all $N_{k}$ equals the total number of data points $N$).

Finally, we can solve for $\omega_{k}$:

$$ $\omega_{k}=\frac{\sum_{i}\bm{\tau}_{ik}}{N},$ (23) $$

This result indicates that the weight $\omega_{k}$ of each mixture component equals the proportion of the posterior probabilities of the data points it contains relative to all data points.

To update $\bm{\mu}_{k}$ and $\Sigma_{k}$, we consider the conditional expectation of the data log-likelihood function:

$$ $\displaystyle Q(\bm{\mu}_{k},\Sigma_{k})=$ $\displaystyle\sum_{i=1}^{N}\bm{\tau}_{ik}\left(-\log(\pi)-\frac{1}{2}\log{| \sigma_{k}|}+\frac{1}{2}\log u_{ik}\right.$ (24) $\displaystyle\left.-\frac{1}{2}\mathbf{u}_{ik}(\mathbf{z}_{i}-\bm{\mu}_{k})^{T }\Sigma_{k}^{-1}(\mathbf{z}_{i}-\bm{\mu}_{k})\right)$ $$

Maximizing $Q(\bm{\mu}_{k},\Sigma_{k})$ with respect to $\bm{\mu}_{k}$ leads to:

$$ $\frac{\partial Q}{\partial\bm{\mu}_{k}}=\frac{1}{2}\sum_{i=1}^{N}\bm{\tau}_{ik }\mathbf{u}_{ik}(2\Sigma_{k}^{-1}\bm{\mu}_{k}-2\Sigma_{k}^{-1}\mathbf{z}_{ik})$ (25) $$

Setting $\frac{\partial Q}{\partial\bm{\mu}_{k}}=0$ results in the updated mean $\bm{\mu}_{k}^{(t+1)}$:

$$ $\bm{\mu}_{k}^{(t+1)}=\sum_{i=1}^{n}\left(\bm{\tau}_{ik}^{(t+1)}\mathbf{u}_{ik} ^{(t+1)}\mathbf{z}_{i}\right)/\sum_{i=1}^{n}\left(\bm{\tau}_{ik}^{(t+1)} \mathbf{u}_{ik}^{(t+1)}\right).$ (26) $$

Considering the derivative of $Q(\bm{\mu}_{k},\Sigma_{k})$ with respect to $\Sigma_{k}^{-1}$:

$$ $\frac{\partial Q}{\partial\Sigma_{k}^{-1}}=\frac{1}{2}\sum_{i=1}^{N}\bm{\tau}_ {ik}\left(\Sigma_{k}-\mathbf{u}_{ik}(\mathbf{z}_{i}-\bm{\mu}_{k})\times( \mathbf{z}_{i}-\bm{\mu}_{k})^{T}\right).$ (27) $$

Setting $\frac{\partial Q}{\partial\bm{\mu}_{k}}=0$ yields the updated covariance matrix $\bm{\Sigma}_{k}^{(t+1)}$:

$$ $\bm{\Sigma}_{k}^{(t+1)}=\frac{\sum_{i=1}^{n}\bm{\tau}_{ik}^{(t+1)}\mathbf{u}_{ ik}^{(t+1)}(\mathbf{z}_{i}-\bm{\mu}_{k}^{(t+1)})(\mathbf{z}_{i}-\bm{\mu}_{k}^{ (t+1)})^{T}}{\sum_{j=1}^{K}\bm{\tau}_{ij}^{(t+1)}}.$ (28) $$

## Appendix C Anomaly Score with Vector Sum

Figure: (a) Scalar Sum
Refer to caption: extracted/5632434/images/scalar-v2.png

### C.1 Advantages

Here we discuss the advantages of employing vector sum in anomaly score with a toy example.

The application of the vector sum principle extends beyond physical mechanics and finds relevance in various domains.
In relational embedding Bordes et al. [], for example, relationships can be represented as vectors.
Aggregating these vectors allows for capturing complexities like transitivity, symmetry, and antisymmetry.

Similarly, in our context, the vector sum can help capture more complex relationships along clusters.
Consider Figure 4 as an example, where a sample $v$ is attracted by two groups of cluster prototypes $\left(\{\bm{\mu}_{1},\bm{\mu}_{2}\},\ \{\bm{\mu}_{3},\bm{\mu}_{4}\}\right)$ with the same mass and sample-prototype distances ($\widetilde{m}_{1}=\widetilde{m}_{2}=\widetilde{m}_{3}=\widetilde{m}_{4}$, $\widetilde{r}_{v1}=\widetilde{r}_{v2}=\widetilde{r}_{v3}=\widetilde{r}_{v4}$).
Without considering the direction of the forces, the two groups of prototypes would attract the sample with equal forces.
However, we argue that the two groups of prototypes should exert different influences.
A sample close to two clusters with a large difference ($\{\bm{\mu}_{1},\bm{\mu}_{2}\}$) is more likely to be an anomaly compared to a sample that is close to two clusters with a smaller difference ($\{\bm{\mu}_{3},\bm{\mu}_{4}\}$).
For example, in a social network, a user who equally likes two extremely different communities, like money-saving tips and luxury items, is more anomalous than a user who equally likes two similar communities, like private jets and luxury items.
Applying the vector sum, the total force of $\{\bm{\mu}_{1},\bm{\mu}_{2}\}$ is much smaller than that of $\{\bm{\mu}_{3},\bm{\mu}_{4}\}$.
As the anomaly score is inversely related to the total force, it is more anomalous when equally attracted by $\{\bm{\mu}_{1},\bm{\mu}_{2}\}$ with large difference.
This indicates that the vector sum successfully captures subtle differences in the distinctions among multiple clusters, thereby assisting in the identification of more accurate anomalies.

### C.2 Toy Example

In the appendix, as illustrated in Figure 3, we investigated a toy example. We discussed a specific pattern of anomalies termed group anomalies, where a small number of anomalous samples cluster together.
It is crucial to note that we do not claim this anomaly pattern is common in real-world data; our goal is merely to point out a specific anomaly pattern that is challenging for traditional cluster-based anomaly detection methods to detect. Specifically, we utilize three Gaussian distributions with high variance (each generating 300 data samples) and one with lower variance (generating 30 data samples). Because the samples from the smaller Gaussian follow a different generative mechanism and represent a minority in the dataset, we consider them anomalies.

We set the cluster number for KMeans-- and GMM at four, indicating that the Gaussian distribution comprising anomalous samples was also recognized as a cluster. KMeans-- employs a cluster-based approach, using the distance to the nearest cluster center as the anomaly score, while GMM uses a probability-based approach, considering the samples’ likelihood in the mixture model as the anomaly score. However, both approaches are ineffective in this scenario. Rather than identifying the small cluster as anomalous, they tend to misidentify samples on the peripheries of larger clusters as anomalies.

By contrast, our scoring method views the entire small cluster as more likely anomalous, followed by outlier samples on the margins of the larger clusters. This visualization provides a perspective that distinguishes our method from previous efforts.

## Appendix D Experimental Supplementary

### D.1 Benchmark Datasets Details

Due to space constraints in the main text, we utilized 30 public datasets from ADBench Han et al. [], covering all different types of data. The details of the 30 datasets are presented in Table 4.

Table: Table 4: Statistics of tabular benchmark datasets.

### D.2 Baselines Details

A comprehensive overview of the unsupervised anomaly detection methods is presented below.

#### D.2.1 Traditional Models

- •
Subspace Outlier Detection (SOD) Kriegel et al. []: Identifies outliers in varying subspaces of a high-dimensional feature space, targeting anomalies that emerge in lower-dimensional projections.
- •
Histogram-based Outlier Detection (HBOS) Goldstein and Dengel []: Assumes feature independence and calculates outlyingness via histograms, offering scalability and efficiency.

#### D.2.2 Linear Models

- •
Principal Component Analysis (PCA) Wold et al. []: Utilizes singular value decomposition for dimensionality reduction, with anomalies indicated by reconstruction errors.
- •
One-class SVM (OCSVM) Manevitz and Yousef []: Defines a decision boundary to separate normal samples from outliers, maximizing the margin from the data origin.

#### D.2.3 Density-based Models

- •
Local Outlier Factor (LOF) Breunig et al. [] : Measures local density deviation, marking samples as outliers if they lie in less dense regions compared to their neighbors.
- •
K-Nearest Neighbors (KNN) Peterson []: Anomaly scores are assigned based on the distance to the k-th nearest neighbor, embodying a simple yet effective approach.

#### D.2.4 Ensemble-based Models

- •
Lightweight On-line Detector of Anomalies (LODA) Pevnỳ [] : An ensemble method suitable for real-time processing and adaptable to concept drift through random projections and histograms.
- •
Isolation Forest (IForest) Liu et al. []: Isolates anomalies by randomly selecting features and split values, leveraging the ease of isolating anomalies to identify them efficiently.

#### D.2.5 Probability-based Models

- •
Deep Autoencoding Gaussian Mixture Model (DAGMM) Zong et al. []: Combines a deep autoencoder with a GMM for anomaly scoring, utilizing both low-dimensional representation and reconstruction error.
- •
Empirical-Cumulative-distribution-based Outlier Detection (ECOD) Li et al. []: Uses ECDFs to estimate feature densities independently, targeting outliers in distribution tails.
- •
Copula Based Outlier Detector (COPOD) Li et al. []: A hyperparameter-free method leveraging empirical copula models for interpretable and efficient outlier detection.

#### D.2.6 Cluster-based Models

- •
DBSCAN Ester et al. []: A density-based clustering algorithm that identifies clusters based on the density of data points, effectively separating high-density clusters from low-density noise, and is widely used for anomaly detection in spatial data.
- •
Clustering Based Local Outlier Factor (CBLOF) He et al. []: Calculates anomaly scores based on cluster distances, using global data distribution.
- •
KMeans-- Song et al. []: Extends k-means to include outlier detection in the clustering process, offering an integrated approach to anomaly detection.
- •
Deep Clustering-based Fair Outlier Detection (DCFOD) Chawla and Gionis []: Enhances outlier detection with a focus on fairness, combining deep clustering and adversarial training for representation learning.

#### D.2.7 Neural Network-based Models

- •
Deep Support Vector Data Description (DeepSVDD) Ruff et al. []: Minimizes the volume of a hypersphere enclosing network data representations, isolating anomalies outside this sphere.
- •
Deep Isolation Forest for Anomaly Detection (DIF) Xu et al. []: Utilizes deep learning to enhance traditional isolation forest techniques, offering improved anomaly detection in complex datasets with minimal parameter tuning.

Each method’s unique mechanism and application context provide a rich landscape of techniques for unsupervised anomaly detection, illustrating the field’s diverse methodologies and the breadth of approaches to tackling anomaly detection challenges.

### D.3 Supplementary Experimental Results

In the appendix, we detail the statistical analysis conducted to compare the performance of various anomaly detectors. We obtained this diagram by conducting a Friedman test (p-value: 4.657e-19), indicating significant differences among different detectors. We utilized average ranks and the Nemenyi test to generate the critical difference diagram, as shown in Figure 5. It is noteworthy that the vector version exhibits significantly superior performance compared to the scalar version across more methods.
The detailed outcomes for the AUCROC and AUCPR metrics, spanning 30 datasets and against 17 baseline approaches, are showcased in Table 5 and Table 6.

Table: Table 5: AUCROC of 17 unsupervised algorithms on 30 tabular benchmark datasets. In each dataset, the algorithm with the highest AUCROC is marked in red, the second highest in blue, and the third highest in green.

Table: Table 6: AUCPR of 17 unsupervised algorithms on 30 tabular benchmark datasets. In each dataset, the algorithm with the highest AUCPR is marked in red, the second highest in blue, and the third highest in green.

Figure: (a) AUC-ROC
Refer to caption: extracted/5632434/images/auc_cd_30.png

### D.4 Complexity Analysis

The complexity of each iteration in UniCAD involves three parts: constructing the outlier set, updating the network parameters $\Theta$, and optimizing the mixture model using the EM algorithm.
Constructing the outlier set requires a sorting operation, for which we use Numpy’s built-in quantile calculation with a time complexity of $\mathcal{O}(N\log N)$.
Considering the number of network parameters along with the computation of the loss function, the computational complexity for optimizing $\Theta$ is approximately $\mathcal{O}(TNDd+TNKd)$.
The EM algorithm for the Student’s t mixture model includes two main steps: the E-step, where the complexity for computing the probability (or responsibility) of each data point belonging to each component is approximately $\mathcal{O}(NKd)$, and the M-step, where the full computational complexity of updating the parameters (mean, covariance matrix) of each component is $\mathcal{O}(NKd^{2})$.
In practice, we use diagonal covariance matrices, which reduces the update complexity to roughly $\mathcal{O}(NKd)$.
If the EM algorithm requires $T$ round to converge, its time complexity is approximately $\mathcal{O}(TNKd)$.
Therefore, the time complexity for $t$-iterations is $\mathcal{O}(tN(\log N+Td(D+K)))$.

## Appendix E Additional Experiments on Graph

### E.1 Baselines

Our proposed method was compared with 16 graph domain baseline methods grouped into three categories as follows:

- •
Contrastive Learning-based Methods:
This group includes CoLA Liu et al. [], SLGAD Zheng et al. [], CONAD Xu et al. [], and ANEMONE Jin et al. [].
These methods primarily assume that the contrastive loss between anomalous nodes and their neighborhoods is more significant.
- •
Autoencoder-based Methods:
This category consists of MLPAE Sakurada and Yairi [], GCNAE Kipf and Welling [], DOMINANT Ding et al. [], GUIDE Yuan et al. [], ComGA Luo et al. [], AnomalyDAE Fan et al. [], ALARM Peng et al. [], DONE/AdONE Bandyopadhyay et al. [] and
AAGNN Zhou et al. [].
These methods focus on the reconstruction errors of anomalous nodes during the process of reconstructing the graph structure or features.
- •
Clustering-based Methods: This category of methods encompasses SCAN Xu et al. [], CBLOF He et al. [], and DCFOD Song et al. []. These methods generally identify anomalies by detecting if a sample deviates from the clustering.

**Table 7: Statistics of graph benchmark datasets.**
| Dataset | # Nodes | # Edges | # Features | # Anomaly | Category |
| --- | --- | --- | --- | --- | --- |
| Disney | 124 | 670 | 28 | 6 | co-purchase network |
| Weibo | 8,405 | 407,963 | 400 | 868 | social media network |
| Reddit | 10,984 | 168,016 | 64 | 366 | user-subreddit network |
| T-Finance | 39,357 | 42,445,086 | 10 | 1,803 | trading network |

### E.2 Datasets

We assess the performance of our model using four graph benchmark datasets containing organic anomalies.
Table 7 presents the statistical summary for each dataset. These datasets contain naturally occurring real-world anomalies and are valuable for assessing the performance of anomaly detection algorithms in real-world scenarios. The sources and compositions of these datasets are as follows:

- •
WeiboJiang [] is a labeled graph comprising user posts extracted from the social media platform Tencent Weibo.
The user-user graph establishes connections between users who exhibit similar topic labels.
A user is considered anomalous if they have engaged in a minimum of five suspicious events, whereas normal nodes represent users who have not.
- •
RedditKumar et al. [] consists of a user-subreddit graph extracted from the popular social media platform Reddit. This publicly accessible dataset encompasses user posts within various subreddits over a month.
Each user is assigned a binary label indicating whether they have been banned on the platform.
Our assumption is that banned users exhibit anomalous behavior compared to regular Reddit users.
- •
DisneyMüller et al. [] is a co-purchase network of movies that includes attributes such as price, rating, and the number of reviews.
The ground truth labels, indicating whether a movie is considered anomalous or not, were assigned by high school students through majority voting.
- •
T-FinanceTang et al. [] aims to identify anomalous accounts within a trading network.
The nodes in this network represent unique anonymous accounts, each characterized by ten features related to registration duration, recorded activity, and interaction frequency.
Graph edges denote transaction records between accounts.
If a node is associated with activities such as fraud, money laundering, or online gambling, human experts will designate it as an anomaly.

### E.3 Experiment Settings

Table: Table 8: AUC-ROC and AUC-PR of 16 unsupervised algorithms on 4 graph benchmark datasets.

In this experiment, we compared graph-based methods on relational data. For methods originally designed around feature vectors, including CBLOF, DCFOD, and our approach, we uniformly employed the same graph representation learning technique as described in BGRL Thakoor et al. []. Specifically, we used a two-layer Graph Convolutional Network (GCN) for encoding, which produced output embeddings with a dimensionality of 128. The training epochs were set to 3000, including a warm-up period of 300 epochs. The hidden size of the predictor was set to 512, and the momentum was fixed at 0.99.

### E.4 Performance Analysis

The performance of UniCAD compared to 16 baseline methods on the four datasets are summarized in Table 8.
From the results, we have the following observations:
Our model consistently outperforms the baseline methods on most datasets, underlining its effectiveness in anomaly detection even within graph data contexts.
This highlights the superiority of UniCAD in detecting anomalies in real-world graph data.

When comparing UniCAD with the four contrastive learning-based methods, it exhibits a distinct advantage, outperforming them by a substantial margin across all metrics.
Unlike contrastive learning methods that rely on the local neighborhood for anomaly detection, UniCAD leverages the global clustering distribution.
This key difference contributes to its consistently superior performance.
Although CONAD incorporates human prior knowledge about anomalies, enabling it to outperform other similar methods on the Weibo and Disney datasets, it still falls short compared to our proposed UniCAD.

Compared to the autoencoder-based methods, UniCAD offers the advantage of lower memory requirements along with better performance.
Graph autoencoders typically reconstruct the entire adjacency matrix during full graph training, resulting in memory usage of at least $\mathcal{O}(N^{2})$.
In contrast, UniCAD, as a clustering-based method, only requires $\mathcal{O}(N\times K)$.
Among the autoencoder-based methods, GCNAE, DONE, and AdONE can be extended to the T-Finance dataset as they only reconstruct the sampled subgraphs rather than the entire adjacency matrix.
However, UniCAD still showcases superior performance while being more memory-efficient.

UniCAD also demonstrates superior performance compared to various other clustering-based methods, including traditional structural clustering (SCAN) methods that treat the embedding from BGRL as tabular data (CBLOF, DCFOD).

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. In the main results, Isolation Forest appears to perform surprisingly well and is, on average, the best-performing baseline method. Could you provide a discussion on this observation?

2. I noticed that several GAD methods are included in the appendix, despite my understanding that the focus of this paper is on AD for i.i.d. data. Why were these methods chosen over other i.i.d. data baselines, such as those for images? It would be helpful if the authors could explain their inclusion. Additionally, if the authors find it important to include GAD baselines, it is recommended to compare them with more recent methods, e.g., those published in 2023 and 2024.

3. Will the proposed framework be effective for image AD?
soundness: 2
strengths: The proposed framework introduces an integrated approach to unsupervised AD by jointly optimizing representation learning, clustering, and AD within a single model. The analogy of gravitational force for anomaly scoring is interesting. On average, the proposed method outperforms the selected baselines in terms of average performance.
summary: Summary: This paper proposes an unsupervised anomaly detection framework called UniCAD, which unifies representation learning, clustering, and anomaly detection. Traditional methods often treat these components separately, which limits their potential to enhance each other. UniCAD introduces a probabilistic mixture model that maximizes an "anomaly-aware" data likelihood to jointly optimize representation learning and clustering, reducing the impact of anomalies. Additionally, UniCAD derives an anomaly score with theoretical support, enhanced by a gravitational force-inspired metric that leverages relationships between samples and clusters. The proposed method outperforms baseline methods in terms of average rank
weaknesses: 1. The model jointly learns a mixture model similar to DAGMM and incorporates a clustering objective as a constraint. While the techniques used in this combination are relatively common in AD, this is not necessarily an issue if the combination demonstrates a unique synergy. The significance of the paper could be improved if the authors provided more insight into why this specific combination is more advantageous than others. For instance, how would performance differ if hypersphere-based constraints like Deep SVDD were combined with mixture models, or if other SSL methods were paired with the clustering objective? I look forward to other reviewers' opinions on the significance of this technical contribution.

2. Some recent baselines are missing. There are several comprehensive surveys on AD available through Google Scholar, and benchmark papers such as ADBench from NeurIPS 2022 could serve as a useful reference point.

3. The default neural network architecture is fairly basic. It might be worth exploring stronger backbones, as these could have a significant impact on the evaluation results.

4. A diagram of the proposed framework would greatly enhance clarity and help readers better understand the model's structure and components.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: N/A
soundness: 3
strengths: 1. The proposed heuristic appears simple yet effective
  2. The paper is well written
  3. The experimental section is strong
  4. Ablation experiments are very useful
summary: The paper presents a new heuristic to compute the anomaly scores under a mixture of models assumption which is motivated by the laws of gravitation. Instead of using scalar addition of the probabilities to compute anomaly score, the proposed technique uses vector addition.
weaknesses: 1. Comparison with gravitation is a stretch and should be left out. While in some sense using vector addition might appear intuitive, mathematically, the underlying data generative model under a Mixture of Models assumption is probabilistic which does not correspond to any 'force' based model; probability at any point does not have a direction. Therefore, the reasoning behind the use of vectors can only be taken as a heuristic with little formal rigorous justification. Despite this weakness, the paper shows a potential for effectiveness and is quite novel.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: Elaborate on the novelty in light of existing works.
soundness: 3
strengths: The paper employs understandable mathematical notations, unlike the majority of deep learning papers.
summary: This paper proposes an unsupervised anomaly detection method based on clustering. The clustering approach relies on a latent representation and a t-distribution, which has been applied in density estimation tasks for decades.

In addition to clustering, the authors propose likelihood-based and gravity-based anomaly scoring methods with neighbor averaging.
weaknesses: The t-distribution-based clustering approach is established. The gravity-based, likelihood-based, and neighbor averaging techniques are also well-known in the field, raising concerns about the paper's novelty.

Additionally, the paper appears to mischaracterize previous research. The use of mixture models for anomaly detection has been documented for at least 25 years (e.g., Yamanishi et al., KDD 2000). Latent subspace-based formulations are also known, such as in Pesevski et al. (2018).

@inproceedings{yamanishi2000line,
  title={On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms},
  author={Yamanishi, Kenji and Takeuchi, Jun-Ichi and Williams, Graham and Milne, Peter},
  booktitle={Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining},
  pages={320--324},
  year={2000}
}


@article{pesevski2018subspace,
  title={Subspace clustering with the multivariate-t distribution},
  author={Pesevski, Angelina and Franczak, Brian C and McNicholas, Paul D},
  journal={Pattern Recognition Letters},
  volume={112},
  pages={297--302},
  year={2018},
  publisher={Elsevier}
}

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: With vector sum, an instance in between two protoypes/centroids could
have an anomaly score of zero, because the two directions are
opposite. Is that desirable?

Sec 3.3.2.  Why can an autoencoder reduce a shortcut solution?
soundness: 3
strengths: Unifying representation learning, clustering, and anomaly deteciont 
into a single likelihood expression is insteresting.

Inspired by physics, incorporating direction in addition to distance
in calculating anomaly score is interesting.

Empirical results indicate that the method has a higher average rank
than 10 existing methods.  

The paper is generally well written.
summary: The authors propose an unified approach to anomaly detection by
combining clustering, representation learning and anomaly detection
via maximising the likelihood of the observed data.  The likelihood of
data is deomposed into the anomaly indicator multipled by the
likelihood of clustered data.  The likelihood of clustered data is a
mixture model with cluster prototypes/centrods and covariance, which
is learned via EM.  Clustering is based on learned representations of
the data via a feature extractor.  The anomaly indicator is zero if an
instance is some predeteremined lowest percentile in the likelihood of
clustered data.  That is, the anomaly indicator is zero if the
instance is unlikely.  The anomaly score is one over the likelihood of
clustered data.

Pointing out the similarity between Newton's gravitational law and the
their likelihood of clustered data, they propose incorporating
direction in addition to magnitude in calculating the anomaly score.
The direction is from the instance to a prototype/centroid.  They call
this method vector sum.

The overall loss function has the negative log likelihood of data and
the reconstruction loss of an autoencoder for the input instances.

They compare their method UniCAD with 10 existing algorithms over 29
tabular datasets.  Empricial results indicate UniCAD with vector sum
has a higher average rank.  Ablation studies indicate the contribution
of different components.
weaknesses: The clustering approach is similar to typical learning of Gaussian
Mixture Models (GMMs) with EM, except with a t-distribution instead of
a Gausian distribution.

Only two of the 10 existing alogrithms involve representation
learning.  Without representation learning, an algorithm is in an
inherent disadvantage when compared to UniCAD.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: - What was the process of selecting values for parameters l and K for the purpose of experiment in Table 1?
- In Appendix Figure 4, if the central cluster was just a regular cluster, would the point near the centre of the central cluster receive a very high outlier score?
soundness: 2
strengths: Several aspects caught my attention when I was reading this paper. Each aspect has strengths and weaknesses. Thus each aspect will be discussed twice, once here and once in the "Weaknesses" section.

Framework: Simultaneous incorporation of representation learning, clustering, and outlier filtering is a powerful framework. It has the most potential compared to considering these factors in isolation or ignoring some of these factors. It is also helpful that the method is applicable in inductive setting.

Physics-inspired design: The use of the vector sum improves performance. It also makes an interesting and thought-provoking read.

Evaluation: The authors evaluated their method using a large number of datasets (30) and baselines (17).

Presentation: Overall, the paper is well-written and easy to follow. Providing a comprehensive study in Appendix has been very useful.
summary: The paper addresses a long standing problem of anomaly detection. To this end, the authors propose a unified framework. This framework simultaneously incorporates 3 important ideas: recovering latent view of data (representation learning), leveraging spatial patterns in data (clustering), and preventing the outliers from contaminating the data model (anomaly detection). Inspired by classical mechanics, the authors further improve the framework by using vector summation in outlier scoring. Extensive experiments with 30 datasets and over a dozen baselines show that the proposed framework can generally outperform previous methods.
weaknesses: Framework (theory): The authors did not provide a solid theoretical foundation behind the propose framework. For example, is Algorithm 1 guaranteed to converge? Is it possible that the algorithm will indefinitely oscillate, e.g., considering points 1, 2, 3 outliers in one iteration, then considering 4, 5, 6 in the next iteration, then considering 1, 2, 3 outliers again, then considering 4, 5, 6 outliers, etc. Moreover, Equation 1, while making intuitive sense, does not appear to correspond to any coherent statistical model. E.g., in a statistical model, one would expect that any point has a probability of being an outlier, even if it's not in the top l by score. Also, does the equation imply that for outliers p(x | params)=1? Given that a point is an outlier, what is p(x | params, x is outlier)?

Framework (novelty): There has been a long history of robust model fitting, including the idea of ignoring potential outliers (e.g., in RANSAC). The proposed framework is an instantiation of this idea. The paper has some novelty, but the novelty is not ground breaking.

Physics-inspired design: Vector sum is an aspect that I found quite interesting, but also most worrying. Thank you for including a deep dive example in Appendix Figure 4. However, what if the central cluster was just a regularly sized cluster? In this case, would a point in the centre of the central cluster receive a very high outlier score (because all "forces" neutralize)? But intuitively, this would be the least abnormal point in the data. I know that in the experiments using the vector sum appeared to increase performance, but I would have liked to see a deep dive from some of these datasets. I don't understand why using vector sum is not harming performance. Also, in Figure 2, you cannot compare the score relative to (mu_1, mu_2) with the score relative to (mu_3, mu_4), because these will be probabilities conditioned on different parameters. That is, you cannot compare outlier scores from different model estimates.

Evaluation (overfitting): Thank you for providing hyperparameter analysis. However, it does show that selecting the right values is important. This raises the concern of whether the parameters were overfit to the datasets. Even though the same setting were used for all datasets, I'm concerned whether the result was overfitted (overfit to the mega-dataset consisting of 30 datasets).

Evaluation (stat. significance): I'm concerned whether critical difference diagrams in Appendix Figure 5 imply that the winning method is not statistically significantly different from the 2nd, 3rd, etc. This is based on horizontal lines crossing several methods.

Presentation: There has been some minor issues with writing
- There is a typesetting issue with citations. Citations should be in brackets. E.g., according to Smith (Smith et al., 2010) or according to Smith [3]
- Some parts of the paper mention 17 baselines, while other parts mention 15
- Items 1, 2, 3, ... in the Introduction are mentioned before Figure 1 is cited
- Why are some arrows crossed in Figure 1?
- The paper mentions "Based on the aforementioned advantages of MMs, ... ", but there were no discussion of MM. Also, acronym MM is not defined
- In Section 3.3.1 (M-step), J is not defined
- In Section 4.4, what does "omitting the likelihood maximization loss" mean?
- Color coding in Table 1 is very confusing. To me "green" is something good, and I though "green" denoted the winner