Title: Towards a Unified Framework of Clustering-based Anomaly Detection
Venue: NeurIPS 2024
Accepted: no
Keywords: Anomaly Detection

Clustering
Abstract: Unsupervised Anomaly Detection (UAD) plays a crucial role in identifying abnormal patterns within data without labeled examples, holding significant practical implications across various domains. Although the individual contributions of representation learning and clustering to anomaly detection are well-established, their interdependencies remain under-explored due to the absence of a unified theoretical framework. Consequently, their collective potential to enhance anomaly detection performance remains largely untapped. To bridge this gap, in this paper, we propose a novel probabilistic mixture model for anomaly detection to establish a theoretical connection among representation learning, clustering, and anomaly detection. By maximizing a novel anomaly-aware data likelihood, representation learning and clustering can effectively reduce the adverse impact of anomalous data and collaboratively benefit anomaly detection. Meanwhile, a theoretically substantiated anomaly score is naturally derived from this framework. Lastly, drawing inspiration from gravitational analysis in physics, we have devised an improved anomaly score that more effectively harnesses the combined power of representation learning and clustering. Extensive experiments, involving 17 baseline methods across 30 diverse datasets, validate the effectiveness and generalization capability of the proposed method, surpassing state-of-the-art methods.

Abstract

Unsupervised Anomaly Detection (UAD) plays a crucial role in identifying abnormal patterns within data without labeled examples, holding significant practical implications across various domains. Although the individual contributions of representation learning and clustering to anomaly detection are well-established, their interdependencies remain under-explored due to the absence of a unified theoretical framework. Consequently, their collective potential to enhance anomaly detection performance remains largely untapped. To bridge this gap, in this paper, we propose a novel probabilistic mixture model for anomaly detection to establish a theoretical connection among representation learning, clustering, and anomaly detection. By maximizing a novel anomaly-aware data likelihood, representation learning and clustering can effectively reduce the adverse impact of anomalous data and collaboratively benefit anomaly detection. Meanwhile, a theoretically substantiated anomaly score is naturally derived from this framework. Lastly, drawing inspiration from gravitational analysis in physics, we have devised an improved anomaly score that more effectively harnesses the combined power of representation learning and clustering. Extensive experiments, involving 17 baseline methods across 30 diverse datasets, validate the effectiveness and generalization capability of the proposed method, surpassing state-of-the-art methods.

1 Introduction

Unsupervised Anomaly Detection (UAD) refers to the task dedicated to identifying abnormal patterns or instances within data in the absence of labeled examples [8]. It has long received extensive attention in the past decades for its wide-ranging applications in numerous practical scenarios, including financial auditing [3], healthcare monitoring [44] and e-commerce sector [23]. Due to the lack of explicit label guidance, the key to UAD is to uncover the dominant patterns that widely exist in the dataset so that samples do not conform to these patterns can be recognized as anomalies. To achieve this, early works [7] have heavily relied on powerful unsupervised representation learning methods to extract the normal patterns from high-dimensional and complex data such as images, text, and graphs. More recent works [45, 2] have utilized clustering , a widely observed natural pattern in real-world data, to provide critical global information for anomaly detection and achieved tremendous success.

While the individual contributions of representation learning and clustering to anomaly detection 31 are well-established, their interrelationships remain largely unexplored. Intuitively, discriminative 32 representation learning can leverage accurate clustering results to differentiate samples from distinct 33 clusters in the embedding space (i.e., ➀ ). Similarly, it can utilize accurate anomaly detection to 34 avoid preserving abnormal patterns (i.e., ➁ ). For accurate clustering , it can gain advantages from 35 representation learning by operating in the discriminative embedding space (i.e., ➂ ). Meanwhile, it 36

Representation

Clustering

Learning

Anomaly

MutuallyDependent

Detection

andEnhanced

！

can potentially benefit from accurate anomaly detection by excluding anomalies when formulating 37 clusters (i.e., ➃ ). Anomaly detection can greatly benefit from both discriminative representation 38 learning and accurate clustering (i.e., ➄ & ➅ ). However, these benefits hinge on the successful 39 identification of anomalies and the reduction of their detrimental impact on the aforementioned 40 tasks. As depicted in Figure 1, the integration of these three elements exhibits a significant reciprocal 41 nature. In summary, representation learning, clustering, and anomaly detection are interdependent and 42 intricately intertwined. Therefore, it is crucial for anomaly detection to fully leverage and mutually 43 enhance the relationships among these three components . 44

Despite the intuitive significance of the interactions among representation learning, clustering, and 45 anomaly detection, existing methods have only made limited attempts to exploit them and fall short 46 of expectations. On one hand, some methods [58] have acknowledged the interplay among these 47 three factors, but their focus remains primarily on the interactions between two factors at a time, 48 making only targeted improvements. For instance, some strategies include explicitly removing outlier 49 samples during the clustering process [9] or designing robust representation learning methods [10] to 50 mitigate the influence of anomalies. On the other hand, recent methods [45] have begun to explore 51 the simultaneous optimization of these three factors within a single framework. However, these 52 attempts are still in the stage of merely superimposing the objectives of the three factors without a 53 unified theoretical framework. This lack of a guiding framework prevents the adequate modeling of 54 the interdependencies among these factors, thereby limiting their collective contribution to a unified 55 anomaly detection objective. Consequently, we aim to address the following question: Is it possible 56 to employ a unified theoretical framework to jointly model these three interdependent objectives, 57 thereby leveraging their respective strengths to enhance anomaly detection? 58

59

60

61

62

63

64

65

66

67

68

69

70

71

In this paper, we try to answer this question and propose a novel model named UniCAD for anomaly detection. The proposed UniCAD integrates representation learning, clustering, and anomaly detection into a unified framework, achieved through the theoretical guidance of maximizing the anomaly-aware data likelihood. Specifically, we explicitly model the relationships between samples and multiple clusters in the representation space using the probabilistic mixture models for the likelihood estimation. Moreover, we creatively introduce a learnable indicator function into the objective of maximum likelihood to explicitly attenuate the influence of anomalies on representation learning and clustering. Under this framework, we can theoretically derive an anomaly score that indicates the abnormality of samples, rather than heuristically designing it based on clustering results as existing works do. Furthermore, building upon this theoretically supported anomaly score and inspired by the theory of universal gravitation, we propose a more comprehensive anomaly metric that considers the complex relationships between samples and multiple clusters. This allows us to better utilize the learned representations and clustering results from this framework for anomaly detection.

72

73

74

To sum up, we underline our contributions as follows:

We propose a unified theoretical framework to jointly optimize representation learning, clustering, and anomaly detection, allowing their mutual enhancement and aid in anomaly detection.

Based on the proposed framework, we derive a theoretically grounded anomaly score and further 75 introduce a more comprehensive score with the vector summation, which fully releases the power 76 of the framework for effective anomaly detection. 77

Extensive experiments have been conducted on 30 datasets to validate the superior unsupervised 78 79

anomaly detection performance of our approach, which surpassed the state-of-the-art through comparative evaluations with 17 baseline methods. 80

81

2 Related Work

Typical unsupervised anomaly detection (UAD) methods calculate a continuous score for each sample 82 to measure its anomaly degree. Various UAD methods have been proposed based on different 83 assumptions, making them suitable for detecting various types of anomaly patterns, including 84 subspace-based models [24], statistical models [16], linear models [49, 32], density-based models [6, 85 38], ensemble-based models [39, 29], probability-based models [40, 58, 28, 27], neural network86 based models [42, 51], and cluster-based models [18, 9]. Considering the field of anomaly detection 87 has progressed by integrating clustering information to enhance detection accuracy [26, 56], we 88 primarily focus on and analyze anomaly patterns related to clustering, incorporating a global clustering 89 perspective to assess the degree of anomaly. Notable methods in this context include CBLOF [18], 90 which evaluates anomalies based on the size of the nearest cluster and the distance to the nearest large 91 cluster. Similarly, DCFOD [45] introduces innovation by applying the self-training architecture of 92 the deep clustering [50] to outlier detection. Meanwhile, DAGMM [58] combines deep autoencoders 93 with Gaussian mixture models, utilizing sample energy as a metric to quantify the anomaly degree. 94 In contrast, our approach introduces a unified theoretical framework that integrates representation 95 learning, clustering, and anomaly detection, overcoming the limitations of heuristic designs and the 96 overlooked anomaly influence in existing methods. 97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

3 Methodology

In this section, we first define the problem we studied and the notations used in this paper. Then we elaborate on the proposed method UniCAD. More specifically, we first introduce a novel learning objective that optimizes representation learning, clustering, and anomaly detection within a unified theoretical framework by maximizing the data likelihood. A novel anomaly score with theoretical support is also naturally derived from this framework. Then, inspired by the concept of universal gravitation, we further propose an enhanced anomaly scoring approach that leverages the intricate relationship between samples and clustering to detect anomalies effectively. Finally, we present an efficient iterative optimization strategy to optimize this model and provide a complexity analysis for the proposed model.

Definition 1 (Unsupervised Anomaly Detection) . Given a dataset X ∈ R N × D comprising N instances with D -dimensional features, unsupervised anomaly detection aims to learn an anomaly score o i for each instance x i in an unsupervised manner so that the abnormal ones have higher scores than the normal ones.

3.1 Maximizing Anomaly-aware Likelihood

Previous research has demonstrated the importance of discriminative representation and accurate clustering in anomaly detection [45]. However, the presence of anomalous samples can significantly disrupt the effectiveness of both representation learning and clustering [12]. While some existing studies have attempted to integrate these three separate learning objectives, the lack of a unified theoretical framework has hindered their mutual enhancement, leading to suboptimal results.

To tackle this issue, in this paper, we propose a unified and coherent approach that considers representation learning, clustering, and anomaly detection by maximizing the likelihood of the observed data. Specifically, we denote the parameters of representation learning as Θ , the clustering parameter as Φ , and the dynamic indicator function for anomaly detection as δ ( · ) . These parameters are optimized simultaneously by maximizing the likelihood of the observed data X :

where c i represents the latent cluster variable associated with x i , and c i = k denotes the probabilistic 123 event that x i belongs to the k -th cluster. The δ ( x i ) is an indicator function that determines whether a 124 sample x i is an anomaly of value 0 or a normal sample of value 1. 125

3.1.1 Joint Representation Learning and Clustering with p ( x i | Θ , Φ) 126

Based on the aforementioned advantages of MMs, we estimate the likelihood p ( x i | Θ , Φ) with mixture 127 models defined as: 128

129

130

131

132

133

134

135

136

137

138

139

140

141

where Φ = { ω k , µ k , Σ k } . The mixture model is parameterized by the prototypes µ k , covariance matrices Σ k , and mixture weights ω k from all clusters. ∑ K k =1 ω k = 1 , and k = 1 , 2 , · · · , K .

In practice, the samples are usually attributed to high-dimensional features and it is challenging to detect anomalies from the raw feature space [41]. Therefore, modern anomaly detection methods [42, 58] often map raw data samples X = { x i } ∈ R N × D into a low-dimensional representation space Z = { z i } ∈ R N × d with a representation learning function z i = f Θ ( x i ) and detect anomalies within this latent representation space.

Following this widely adopted practice, we model the distribution of samples in the latent representation space with a multivariate Student'st distribution giving its cluster c i = k . The Student'st distribution is robust against outliers due to its heavy tails. Bayesian robustness theory leverages such distributions to dismiss outlier data, favoring reliable sources, making the Student'st process preferable over Gaussian processes for data with atypical information [1]. Thus the probability distribution of generating x i with latent representation z i given its cluster c i = k can be expressed as:

where z i = f Θ ( x i ) denotes the representation obtained from the data mapped through the neural 142 network parameterized by Θ . Γ denotes the gamma function while ν is the degree of freedom. 143 √ -1

Σ k is the scale parameter. D M ( z i , µ k ) = ( z i -µ k ) T Σ k ( z i -µ k ) represents the Mahalanobis 144 distance [33]. In the unsupervised setting, as cross-validating ν on a validation set or learning it is 145 unnecessary, ν is set as 1 for all experiments [50, 48]. The overall marginal likelihood of the observed 146 data x i can be simplified as: 147

3.1.2 Anomaly Indicator δ ( x i ) and Score o i

148

As we have discussed, the indicator function δ ( x i ) not only benefits both representation and clustering 149 but also directly serves as the output of anomaly detection. Ideally, with the percentage of outliers 150 denoted as l , an optimal solution for δ ( x i ) that maximizes the objective function J (Θ , Φ) entails 151 setting all δ ( x i ) = 0 for x i among the l percent of outliers with lowest generation possibility 152 p ( x i | Θ , Φ) , and otherwise δ ( x i ) = 1 is set for the remaining normal samples. Therefore, the 153 indicator function is determined as: 154

As this method involves sorting the samples based on the generation probability as being anomalous, 155 the values of p ( x i | Θ , Φ) can serve as a form of anomaly score, a classic approach within the mixture 156 model framework [40, 58]. This suggests that the likelihood of a sample being anomalous is inversely 157 related to its generative probability since a lower generative probability indicates a higher chance of 158 the sample being an outlier. Thus the anomaly score of sample x i can be defined as: 159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

179

180

181

182

183

184

185

186

187

188

189

190

191

3.2 Gravity-inspired Anomaly Scoring

In practical applications, it is proved that anomaly scores derived from generation probabilities often yield suboptimal performance [17]. This observation prompts a reconsideration of how to fully leverage the complex relationships among samples or even across multiple clusters for anomaly detection . In this section, we first provide a brief introduction to the concept of Newton's Law of Universal Gravitation [35] and then demonstrate how the anomaly score is intriguingly similar to this cross-field principle. Finally, we discuss the advantages of introducing the vector sum operation into the anomaly score inspired by the analogy.

3.2.1 Analog Anomaly Scoring and Force Analysis

To begin with, Newton's Law of Universal Gravitation [35] stands as a fundamental framework for describing the interactions among entities in the physical world. According to this law, every object in the universe experiences an attractive force from another object. In classical mechanics, force analysis involves calculating the vector sum of all forces acting on an object, known as the resultant force , which is crucial in determining an object's acceleration or change in motion:

where ⃗ F ik represents the k -th force acting on the object i . This force is proportional to the product of 174 their masses, ( m i and m k ), and inversely proportional to the square of the distance r ik between them. 175 G represents the gravitational constant, and ⃗ r ij is the unit direction vector. 176

Similarly, if denoting: ˜ F ik = p ( x i , c i = k | Θ , Φ) = ω k · π -1 ·| Σ k | -1 / 2 1+ D M ( z i , µ k ) 2 , the score of Equation (6) 177 bears analogies to the summation of the magnitudes of forces as: 178

where ˜ G = π -1 , ˜ m k = ω k | Σ k | -1 / 2 , ˜ m i = 1 , and ˜ r ik = √ 1 + D M ( z i , µ k ) 2 . Here, ˜ r ik is taken as the measure of distance within the representation space, modified slightly by an additional term for smoothness. The constant ˜ G serves a role akin to the gravitational constant in this analogy, whereas ˜ m k resembles the concept of mass for the cluster. The notation ˜ m i suggests a standardization where the mass of each data point is considered uniform and not differentiated.

3.2.2 Anomaly Scoring with Vector Sum

Comparing Equation (7) with Equation (8), what still differs is that, unlike a simple sum of the scalar value, the resultant force ⃗ F i, total employs the vector sum and incorporates both the magnitude and direction ̂ r ik of each force. This distinction is crucial because forces in different directions can neutralize each other with a large angle between them or enhance each other's effects with a small angle. Inspired by this difference, we consider modeling the relationship between samples and clusters as a vector, and aggregating them through vector summation. The vector-formed anomaly score o V i is defined as:

where ⃗ r ik represents the unit direction vector in the representation space from the sample z i to the 192 cluster prototype µ k , and ∥ · ∥ represents the L 2 norm. 193

194

195

196

197

198

199

200

201

202

203

204

205

3.3 Iterative Optimization

Given the challenge posed by the interdependence of the parameters of the network Θ and those of the mixture model { ω k , µ k , Σ k } in joint optimization, we propose an iterative optimization procedure. The pseudocode for training the model is presented in Algorithm 1 in the appendix.

3.3.1 Update Φ

To update the parameters of the mixture model Φ = { ω k , µ k , Σ k } , we use the ExpectationMaximization (EM) algorithm to maximize equation (1) [36]. The detailed derivation is included in Appendix B.

E-step. During the E-step of iteration ( t +1) , our goal is to compute the posterior probabilities of each data point belonging to the k -th cluster within the mixture model. Given the observed sample x i and the current estimates of the parameters Θ ( t ) and Φ ( t ) , the expected value of the likelihood function of latent variable c k , or the posterior possibilities, can be expressed as:

The scale factor[36] serving as an intermediate result for subsequent updates in the M-step is : 206

M-step. In the M-step of iteration ( t + 1) , given the gradients ∂J (Θ , Φ) ∂ω k = 0 , ∂J (Θ , Φ) ∂ µ k = 0 , and ∂J (Θ , Φ) ∂ Σ k = 0 , we derive the analytical solutions for the mixture model parameters ω k , µ k , and Σ k . Assume the anomalous ratio is l ∈ [0 , 1] , the number of the normal samples is n = int ( l ∗ N ) . The updating process for { ω ( t +1) k , µ ( t +1) k , Σ ( t +1) k } is as follows:

The mixture weights ω k are updated by averaging the posterior probabilities over all data points with the number of samples , reflecting the relative presence of each component in the mixture:

The prototypes µ k are updated to be the weighted average of the data points, where weights are the 213 posterior probabilities: 214

The covariance matrices Σ k are updated by considering the dispersion of the data around the newly 215 computed prototypes: 216

207

208

209

210

211

212

217

218

219

220

221

222

223

224

3.3.2 Update Θ

We focus on anomaly-aware representation learning and use stochastic gradient descent to optimize the network parameters Θ , by minimizing the following joint loss:

where J (Θ , Φ) = log p ( X | Θ , Φ) . An additional constraint term g (Θ) is introduced to prevent shortcut solution [15]. In practice, an autoencoder architecture is implemented, utilizing a reconstruction loss g (Θ) = ∥ x -ˆ x ∥ 2 as the constraint.

These updates are iteratively performed until convergence, resulting in optimized model parameters that best fit the given data according to the mixture model framework.

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

4 Experiments

4.1 Datasets & Baselines

We evaluated UniCAD on an extensive collection of datasets, comprising 30 tabular datasets that span 16 diverse fields. We specifically focused on naturally occurring anomaly patterns, rather than synthetically generated or injected anomalies, as this aligns more closely with real-world scenarios. The detailed descriptions are provided in Table 4 of Appendix D.1. Following the setup in ADBench [17], we adopt an inductive setting to predict newly emerging data, a highly beneficial approach for practical applications.

To assess the effectiveness of UniCAD, we compared it with 17 advanced unsupervised anomaly detection methods, including: (1) traditional methods : SOD [24] and HBOS [16]; (2) linear methods : PCA [49] and OCSVM [32]; (3) density-based methods : LOF [6] and KNN [38]; (4) ensemble-based methods : LODA [39] and IForest [29]; (5) probability-based methods : DAGMM[58], ECOD [28], and COPOD [27]; (6) cluster-based methods : DBSCAN[13], CBLOF [18], DCOD [45] and KMeans- [9]; and (7) neural network-based methods : DeepSVDD [42] and DIF [51]. These baselines encompass the majority of the latest methods, providing a comprehensive overview of the state-ofthe-art. For a detailed description, please refer to Appendix D.2.

4.2 Experiment Settings

In the unsupervised setting, we employ the default hyperparameters from the original papers for all comparison methods. Similarly, the UniCAD also utilizes a fixed set of parameters to ensure a fair comparison. For all datasets, we employ a two-layer MLP with a hidden dimension of d = 128 and ReLU activation function as both encoder and decoder. We utilize the Adam optimizer [21] with a learning rate of 1 e -4 for 100 epochs. For the EM process, we set the maximum iteration number to 100 and a tolerance of 1 e -3 for stopping training when the objectives converge. The number of components in the mixture model is set as k = 10 , and the proportion of the outlier is set as l = 1% . We evaluate the methods using Area Under the Receiver Operating Characteristic (AUC-ROC) and Area Under the Precision-Recall Curve (AUC-PR) metrics [17], reporting the average ranking (Avg. Rank) across all datasets. All experiments are run 3 times with different seeds, and the mean results are reported.

4.3 Performance and Analysis

Performance Comparison . Table 1 presents a comparison of UniCAD with 10 unsupervised baseline methods across 30 tabular datasets using the AUC-ROC metric. The experimental results, which encompass 17 baselines, are included in Tables 5 and 6 of Appendix D.3, with additional experiments on other data domains presented in Appendix E. Our proposed UniCAD achieves the top average ranking, exhibiting the best or near-best performance on a larger number of datasets and confirming advanced capabilities. It is noteworthy that there is no one-size-fits-all unsupervised anomaly detection method suitable for every type of dataset, as demonstrated by the observation that other methods have also achieved some of the best results on certain datasets. However, our model showcased a remarkable ability to generalize across most datasets featuring natural anomalies, as evidenced by statistical average ranking. As for clustering-based methods such as KMeans--, DCOD, and CBLOF, they mostly rank in the top tier among all baseline methods, supporting the advantage of combining deep clustering with anomaly detection. However, our method significantly outperformed these methods by mitigating their limitations and further providing a unified framework for joint representation learning, clustering, and anomaly detection.

Effectiveness of Vector Sum in Anomaly Scoring . As demonstrated in Table 1, we compare the anomaly score o i derived directly from the generation possibility with its vector summation form o V i . According to our statistical findings, we observe that vector scores o V i consistently outperform scalar scores o i . This indicates that the introduction of the vector summation, analogous to the concept of resultant force, makes a substantial difference in anomaly detection scenarios involving multiple clusters. The performance gains of the vector sum scores strongly demonstrate the effectiveness of the UniCAD in capturing the subtle differences in the distinctions among multiple clusters and underscore the utility of this factor in the context of anomaly detection based on clustering.

276

277

278

279

280

281

282

283

284

285

286

287

288

(a) Optimization Analysis

(b) AUC-ROC Surface

(c) AUC-PR Surface

99.8

0.93

96

0.88

0.23

0.28

AUC-ROC(%)

99.6

AUC-PR(%)

E8'0

0.18

0.78

0.13

94

99.4

17

17

92

13

13

0.001

0.001

99.2

0.002

k

0.002

9

k

0.007

0.007

2

4

6

8

10.018

0.050

5

0.050

Iterations

0.135

1

0.135

1

Analysis of EM Iterative Optimization . To comprehend the iterative training within our model, we have illustrated the performance variations accompanying the increase in iteration counts in Figure 2a. Specifically, we monitored the iteration number t for the satimage-2 dataset, ranging from 0 to 10, while maintaining other default parameters constant. Both AUC-ROC and AUC-PR performance curves displayed consistent trends, with minor fluctuations only during the initial phase. The performance remained relatively stable throughout the last steps, illustrating the effectiveness and convergence of iterative EM optimization.

Runtime Comparison. We present a analysis of the runtime performance of various methods, including our proposed approach, as detailed in Table 2. Our experiments, conducted on the backdoor dataset, reveal that while non-deep learning methods exhibit lower runtime, they often simplify the problem space excessively, failing to capture the complex non-linear relationships present in the data. In contrast, our method, when compared to existing deep learning techniques, demonstrates a significant reduction in computational time. This indicates that our approach not only manages

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

to efficiently model complex patterns but also achieves an optimal balance between computational efficiency and modeling capability.

4.4 Ablation Studies

In this section, we examine the contributions of different components in UniCAD. Tables 3 reports the results. We make three major observations. Firstly , the anomaly detection performance experiences a significant drop when replacing the Student's t distribution with a Gaussian distribution for the Mixture Model, highlighting the robustness of the Student's t distribution in unsupervised anomaly detection. Secondly , omitting the likelihood maximization loss (w/o J (Θ , Φ) ) also results in a considerable decrease in overall performance. This observation underscores the importance of deriving both the optimization objectives and anomaly scores from the likelihood generation probability through a theoretical framework, which allows for unified joint optimization of anomaly detection and clustering in the representation space. Furthermore , the indicator function δ ( x i ) also contributes to a performance increase. These results further confirm the effectiveness of our UniCAD in mitigating the negative influence of anomalies in the clustering process, as the existence of outliers may significantly degrade the performance of clustering. In summary, all these ablation studies clearly demonstrate the effectiveness of our theoretical framework in simultaneously considering representation learning, clustering, and anomaly detection.

4.5 Sensitivity of Hyperparameters

In this section, we conducted a sensitivity analysis on key hyperparameters of the model applied to the donors dataset, focusing on the number of clusters k and the proportion of the outlier set l . The results of this analysis are illustrated in Figure 2. Notably, the optimal range for l tends to be lower than the actual proportion of anomalies in the dataset. Furthermore, a pattern was observed with the number of clusters k , where the model performance initially improved with an increase in k , followed by a subsequent decline. This suggests the existence of an optimal range for the number of clusters, which should be carefully selected based on the specific application context.

5 Conclusion

This paper presents UniCAD, a novel model for Unsupervised Anomaly Detection (UAD) that seamlessly integrates representation learning, clustering, and anomaly detection within a unified theoretical framework. Specifically, UniCAD introduces an anomaly-aware data likelihood based on the mixture model with the Student-t distribution to guide the joint optimization process, effectively mitigating the impact of anomalies on representation learning and clustering. This framework enables a theoretically grounded anomaly score inspired by universal gravitation, which considers complex relationships between samples and multiple clusters. Extensive experiments on 30 datasets across various domains demonstrate the effectiveness and generalization capability of UniCAD, surpassing 15 baseline methods and establishing it as a state-of-the-art solution in unsupervised anomaly detection. Despite its potential, the proposed method's applicability to broader fields like time series and multimodal anomaly detection requires further exploration and validation, highlighting a significant area for future work.

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

References

J Ailton A Andrade. On the robustness to outliers of the student-t process. Scandinavian Journal of Statistics , 50(2):725-749, 2023.

Caglar Aytekin, Xingyang Ni, Francesco Cricri, and Emre Aksu. Clustering and unsupervised anomaly detection with l 2 normalized deep auto-encoder representations. In 2018 International Joint Conference on Neural Networks (IJCNN) , pages 1-6. IEEE, 2018.

Alexander Bakumenko and Ahmed Elragal. Detecting anomalies in financial data using machine learning algorithms. Systems , 10(5):130, 2022.

Sambaran Bandyopadhyay, Saley Vishal Vivek, and MN Murty. Outlier resistant unsupervised deep architectures for attributed network embedding. In Proceedings of the 13th international conference on web search and data mining , pages 25-33, 2020.

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems , 26, 2013.

Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data , pages 93-104, 2000.

Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407 , 2019.

Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR) , 41(3):1-58, 2009.

Sanjay Chawla and Aristides Gionis. k-means-: A unified approach to clustering and outlier detection. In Proceedings of the 2013 SIAM international conference on data mining , pages 189-197. SIAM, 2013.

Hyunsoo Cho, Jinseok Seol, and Sang-goo Lee. Masked contrastive learning for anomaly detection. arXiv preprint arXiv:2105.08793 , 2021.

Kaize Ding, Jundong Li, Rohit Bhanushali, and Huan Liu. Deep anomaly detection on attributed networks. In Proceedings of the 2019 SIAM International Conference on Data Mining , pages 594-602. SIAM, 2019.

Lian Duan, Lida Xu, Ying Liu, and Jun Lee. Cluster-based outlier detection. Annals of Operations Research , 168:151-168, 2009.

Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd , volume 96, pages 226-231, 1996.

Haoyi Fan, Fengbin Zhang, and Zuoyong Li. Anomalydae: Dual autoencoder for anomaly detection on attributed networks. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 5685-5689. IEEE, 2020.

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665-673, 2020.

Markus Goldstein and Andreas Dengel. Histogram-based outlier score (hbos): A fast unsupervised anomaly detection algorithm. KI-2012: poster and demo track , 1:59-63, 2012.

Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. Adbench: Anomaly detection benchmark. Advances in Neural Information Processing Systems , 35:32142-32159, 2022.

Zengyou He, Xiaofei Xu, and Shengchun Deng. Discovering cluster-based local outliers. 372 Pattern recognition letters , 24(9-10):1641-1650, 2003. 373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

Meng Jiang. Catching social media advertisers with strategy analysis. In Proceedings of the First International Workshop on Computational Methods for CyberSafety , pages 5-10, 2016.

Ming Jin, Yixin Liu, Yu Zheng, Lianhua Chi, Yuan-Fang Li, and Shirui Pan. Anemone: Graph anomaly detection with multi-scale contrastive learning. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 3122-3126, 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 , 2016.

Yufeng Kou, Chang-Tien Lu, Sirirat Sirwongwattana, and Yo-Ping Huang. Survey of fraud detection techniques. In IEEE International Conference on Networking, Sensing and Control, 2004 , volume 2, pages 749-754. IEEE, 2004.

Hans-Peter Kriegel, Peer Kröger, Erich Schubert, and Arthur Zimek. Outlier detection in axis-parallel subspaces of high dimensional data. In Advances in Knowledge Discovery and Data Mining: 13th Pacific-Asia Conference, PAKDD 2009 Bangkok, Thailand, April 27-30, 2009 Proceedings 13 , pages 831-838. Springer, 2009.

Srijan Kumar, Xikun Zhang, and Jure Leskovec. Predicting dynamic embedding trajectory in temporal interaction networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , pages 1269-1278, 2019.

Jinbo Li, Hesam Izakian, Witold Pedrycz, and Iqbal Jamal. Clustering-based anomaly detection in multivariate time series data. Applied Soft Computing , 100:106919, 2021.

Zheng Li, Yue Zhao, Nicola Botta, Cezar Ionescu, and Xiyang Hu. Copod: copula-based outlier detection. In 2020 IEEE international conference on data mining (ICDM) , pages 1118-1123. IEEE, 2020.

Zheng Li, Yue Zhao, Xiyang Hu, Nicola Botta, Cezar Ionescu, and George Chen. Ecod: Unsupervised outlier detection using empirical cumulative distribution functions. IEEE Transactions on Knowledge and Data Engineering , 2022.

Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining , pages 413-422. IEEE, 2008.

Yixin Liu, Zhao Li, Shirui Pan, Chen Gong, Chuan Zhou, and George Karypis. Anomaly detection on attributed networks via contrastive self-supervised learning. IEEE transactions on neural networks and learning systems , 33(6):2378-2392, 2021.

Xuexiong Luo, Jia Wu, Amin Beheshti, Jian Yang, Xiankun Zhang, Yuan Wang, and Shan Xue. Comga: Community-aware attributed graph anomaly detection. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining , pages 657-665, 2022.

Larry M Manevitz and Malik Yousef. One-class svms for document classification. Journal of machine Learning research , 2(Dec):139-154, 2001.

Goeffrey J McLachlan. Mahalanobis distance. Resonance , 4(6):20-26, 1999.

Emmanuel Müller, Patricia Iglesias Sánchez, Yvonne Mülle, and Klemens Böhm. Ranking outlier nodes in subspaces of attributed graphs. In 2013 IEEE 29th international conference on data engineering workshops (ICDEW) , pages 216-222. IEEE, 2013.

Isaac Newton. Philosophiae naturalis principia mathematica , volume 1. G. Brookman, 1833.

David Peel and Geoffrey J McLachlan. Robust mixture modelling using the t distribution. Statistics and computing , 10:339-348, 2000.

Zhen Peng, Minnan Luo, Jundong Li, Luguo Xue, and Qinghua Zheng. A deep multi-view framework for anomaly detection on attributed networks. IEEE Transactions on Knowledge and Data Engineering , 34(6):2539-2552, 2020.

Leif E Peterson. K-nearest neighbor. Scholarpedia , 4(2):1883, 2009. 421

Tomáš Pevn` y. Loda: Lightweight on-line detector of anomalies. Machine Learning , 102:275422 304, 2016. 423

424

425

Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics , 741(659-663), 2009.

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE , 109(5):756-795, 2021.

Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning , pages 4393-4402. PMLR, 2018.

Mayu Sakurada and Takehisa Yairi. Anomaly detection using autoencoders with nonlinear dimensionality reduction. In Proceedings of the MLSDA 2014 2nd workshop on machine learning for sensory data analysis , pages 4-11, 2014.

Osman Salem, Yaning Liu, Ahmed Mehaoua, and Raouf Boutaba. Online anomaly detection in wireless body area networks for reliable healthcare monitoring. IEEE journal of biomedical and health informatics , 18(5):1541-1551, 2014.

Hanyu Song, Peizhao Li, and Hongfu Liu. Deep clustering based fair outlier detection. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1481-1489, 2021.

Jianheng Tang, Jiajin Li, Ziqi Gao, and Jia Li. Rethinking graph neural networks for anomaly detection. In International Conference on Machine Learning , pages 21076-21089. PMLR, 2022.

Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L Dyer, Remi Munos, Petar Veliˇ ckovi´ c, and Michal Valko. Large-scale representation learning on graphs via bootstrapping. arXiv preprint arXiv:2102.06514 , 2021.

Laurens Van Der Maaten. Learning a parametric embedding by preserving local structure. In Artificial intelligence and statistics , pages 384-391. PMLR, 2009.

Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems , 2(1-3):37-52, 1987.

Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning , pages 478-487. PMLR, 2016.

Hongzuo Xu, Guansong Pang, Yijie Wang, and Yongjun Wang. Deep isolation forest for anomaly detection. IEEE Transactions on Knowledge and Data Engineering , 2023.

Xiaowei Xu, Nurcan Yuruk, Zhidan Feng, and Thomas AJ Schweiger. Scan: a structural clustering algorithm for networks. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 824-833, 2007.

Zhiming Xu, Xiao Huang, Yue Zhao, Yushun Dong, and Jundong Li. Contrastive attributed network anomaly detection with data augmentation. In Advances in Knowledge Discovery and Data Mining: 26th Pacific-Asia Conference, PAKDD 2022, Chengdu, China, May 16-19, 2022, Proceedings, Part II , pages 444-457. Springer, 2022.

Xu Yuan, Na Zhou, Shuo Yu, Huafei Huang, Zhikui Chen, and Feng Xia. Higher-order structure based anomaly detection on attributed networks. In 2021 IEEE International Conference on Big Data (Big Data) , pages 2691-2700. IEEE, 2021.

Yu Zheng, Ming Jin, Yixin Liu, Lianhua Chi, Khoa T Phan, and Yi-Ping Phoebe Chen. Generative and contrastive self-supervised learning for graph anomaly detection. IEEE Transactions on Knowledge and Data Engineering , 2021.

468

469

470

471

472

473

474

475

476

Shuang Zhou, Xiao Huang, Ninghao Liu, Qiaoyu Tan, and Fu-Lai Chung. Unseen anomaly detection on networks via multi-hypersphere learning. In Proceedings of the 2022 SIAM International Conference on Data Mining (SDM) , pages 262-270. SIAM, 2022.

Shuang Zhou, Qiaoyu Tan, Zhiming Xu, Xiao Huang, and Fu-lai Chung. Subtractive aggregation for attributed network anomaly detection. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 3672-3676, 2021.

Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and

Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International conference on learning representations , 2018.

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

Algorithm 1 Model training for UniCAD

A Iterative Training Algorithm

The pseudocode for training the model is presented in Algorithm 1. Initially, all parameters undergo random initialization. In subsequent iterations, following the initial round, the outlier set L undergoes updates based on the anomaly score o i . This is succeeded by the adjustment of the network parameters Θ based on x i , further optimizing the performance of Θ through the utilization of the estimated parameters µ k , ω k , Σ k . The essence of the algorithm is embedded in its alternating optimization strategy, iteratively refining the accuracy of representation learning and mixed model parameter estimation, thereby augmenting the overall training effectiveness of the model.

B Derivation of EM Algorithm

This appendix provides the detailed derivation of the Expectation-Maximization (EM) algorithm for optimizing the parameters of a mixture model based on Student's t-distribution. The focus is on deriving analytical solutions for the maximization of the parameters Φ = { µ k , Σ k , ω k } of the mixture components. The EM algorithm alternates between two steps:

In the E-step , we calculate the posterior probabilities τ ik , representing the probability of data point i belonging to cluster k , given the current parameters. The posterior probabilities for a Student's t-distribution mixture model are formulated as:

where τ ( z i | µ k , Σ k ) denotes the Student's t-distribution for data point i with respect to cluster k , and K is the number of mixture components.

The Student's t-distribution is depicted as a hierarchical conditional probability, resembling a Gaussian distribution with an accuracy scale factor u , where its latent variable follows a gamma distribution. Adopting a degree of freedom ν = 1 , the value of u ik is given by:

In the M-step , we update the parameters Φ = { ω k , µ k , and Σ k } using the derivatives obtained in 498 the previous steps. In our model, the likelihood function for a Student's-t Distribution Mixture Model 499

GroupAnomalies

NearestCluster

GenerativeProbability

ResultantForce

(SMM) is represented as: 500

where ω k are the mixture weights, Σ k the covariance matrices, µ k the means, and z i the data points. 501

The derivative with respect to ω k must consider the constraint that the sum of the mixture weights 502 equals 1, i.e., ∑ k ω k = 1 . Hence, we introduce a Lagrange multiplier λ to address this constraint 503 and construct the Lagrangian L ′ : 504

The derivative with respect to ω k is: 505

Substituting the definition of L ( ω, µ , Σ) , we obtain: 506

To solve for ω k , we first multiply both sides of the equation by ω k and apply the constraint condition: 507

Upon further organization, we find that the Lagrange multiplier λ actually equals the total number of 508 data points N (since ∑ i τ ik = N k , where N k is the expected total number of data points belonging 509 to the k th component, and the sum of all N k equals the total number of data points N ). 510

Finally, we can solve for ω k : 511

This result indicates that the weight ω k of each mixture component equals the proportion of the 512 posterior probabilities of the data points it contains relative to all data points. 513

To update µ k and Σ k , we consider the conditional expectation of the data log-likelihood function: 514

516

(b) Vector Sum

μ1

μ1

μ4

μ4

Fv3 + Fv4

Fu1 +Fo2F3+F

Foui+Po2

2

14

μ3

μ3

μ2

μ2

Maximizing Q ( µ k , Σ k ) with respect to µ k leads to: 515

Setting ∂Q ∂ µ k = 0 results in the updated mean µ ( t +1) k :

Considering the derivative of Q ( µ k , Σ k ) with respect to Σ -1 k : 517

Setting ∂Q ∂ µ k = 0 yields the updated covariance matrix Σ ( t +1) k :

C Anomaly Score with Vector Sum

C.1 Advantages

Here we discuss the advantages of employing vector sum in anomaly score with a toy example.

The application of the vector sum principle extends beyond physical mechanics and finds relevance in various domains. In relational embedding [5], for example, relationships can be represented as vectors. Aggregating these vectors allows for capturing complexities like transitivity, symmetry, and antisymmetry.

Similarly, in our context, the vector sum can help capture more complex relationships along clusters. Consider Figure 4 as an example, where a sample v is attracted by two groups of cluster prototypes ( { µ 1 , µ 2 } , { µ 3 , µ 4 } ) with the same mass and sample-prototype distances ( ˜ m 1 = ˜ m 2 = ˜ m 3 = ˜ m 4 , ˜ r v 1 = ˜ r v 2 = ˜ r v 3 = ˜ r v 4 ). Without considering the direction of the forces, the two groups of prototypes would attract the sample with equal forces. However, we argue that the two groups of prototypes should exert different influences. A sample close to two clusters with a large difference ( { µ 1 , µ 2 } ) is more likely to be an anomaly compared to a sample that is close to two clusters with a smaller difference ( { µ 3 , µ 4 } ). For example, in a social network, a user who equally likes two extremely different communities, like money-saving tips and luxury items, is more anomalous than a user who equally likes two similar communities, like private jets and luxury items. Applying the vector sum, the total force of { µ 1 , µ 2 } is much smaller than that of { µ 3 , µ 4 } . As the anomaly score is inversely related to the total force, it is more anomalous when equally attracted by { µ 1 , µ 2 } with large difference. This indicates that the vector sum successfully captures subtle differences in the distinctions among multiple clusters, thereby assisting in the identification of more accurate anomalies .

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

C.2 Toy Example

In the appendix, as illustrated in Figure 3, we investigated a toy example. We discussed a specific pattern of anomalies termed group anomalies , where a small number of anomalous samples cluster together. It is crucial to note that we do not claim this anomaly pattern is common in real-world data; our goal is merely to point out a specific anomaly pattern that is challenging for traditional clusterbased anomaly detection methods to detect. Specifically, we utilize three Gaussian distributions with high variance (each generating 300 data samples) and one with lower variance (generating 30 data samples). Because the samples from the smaller Gaussian follow a different generative mechanism and represent a minority in the dataset, we consider them anomalies.

We set the cluster number for KMeans-- and GMM at four, indicating that the Gaussian distribution comprising anomalous samples was also recognized as a cluster. KMeans-- employs a cluster-based approach, using the distance to the nearest cluster center as the anomaly score, while GMM uses a probability-based approach, considering the samples' likelihood in the mixture model as the anomaly score. However, both approaches are ineffective in this scenario. Rather than identifying the small cluster as anomalous, they tend to misidentify samples on the peripheries of larger clusters as anomalies.

By contrast, our scoring method views the entire small cluster as more likely anomalous, followed by outlier samples on the margins of the larger clusters. This visualization provides a perspective that distinguishes our method from previous efforts.

D Experimental Supplementary

D.1 Benchmark Datasets Details

Due to space constraints in the main text, we utilized 30 public datasets from ADBench [17], covering all different types of data. The details of the 30 datasets are presented in Table 4.

D.2 Baselines Details

A comprehensive overview of the unsupervised anomaly detection methods is presented below.

D.2.1 Traditional Models

Subspace Outlier Detection (SOD) [24]: Identifies outliers in varying subspaces of a highdimensional feature space, targeting anomalies that emerge in lower-dimensional projections.

Histogram-based Outlier Detection (HBOS) [16]: Assumes feature independence and calculates outlyingness via histograms, offering scalability and efficiency.

D.2.2 Linear Models

Principal Component Analysis (PCA) [49]: Utilizes singular value decomposition for dimensionality reduction, with anomalies indicated by reconstruction errors.

One-class SVM (OCSVM) [32]: Defines a decision boundary to separate normal samples from outliers, maximizing the margin from the data origin.

D.2.3 Density-based Models

Local Outlier Factor (LOF) [6] : Measures local density deviation, marking samples as outliers if they lie in less dense regions compared to their neighbors.

K-Nearest Neighbors (KNN) [38]: Anomaly scores are assigned based on the distance to the k-th nearest neighbor, embodying a simple yet effective approach.

D.2.4 Ensemble-based Models

Lightweight On-line Detector of Anomalies (LODA) [39] : An ensemble method suitable for real-time processing and adaptable to concept drift through random projections and histograms.

Isolation Forest (IForest) [29]: Isolates anomalies by randomly selecting features and split values, leveraging the ease of isolating anomalies to identify them efficiently.

586

D.2.5 Probability-based Models

Deep Autoencoding Gaussian Mixture Model (DAGMM) [58]: Combines a deep autoencoder 587 with a GMM for anomaly scoring, utilizing both low-dimensional representation and reconstruction 588 error. 589

590

591

Empirical-Cumulative-distribution-based Outlier Detection (ECOD) [28]: Uses ECDFs to estimate feature densities independently, targeting outliers in distribution tails.

592

593

594

595

596

597

598

599

600

601

Copula Based Outlier Detector (COPOD) [27]: A hyperparameter-free method leveraging empirical copula models for interpretable and efficient outlier detection.

D.2.6 Cluster-based Models

DBSCAN [13]: A density-based clustering algorithm that identifies clusters based on the density of data points, effectively separating high-density clusters from low-density noise, and is widely used for anomaly detection in spatial data.

Clustering Based Local Outlier Factor (CBLOF) [18]: Calculates anomaly scores based on cluster distances, using global data distribution.

KMeans-- [45]: Extends k-means to include outlier detection in the clustering process, offering an integrated approach to anomaly detection.

Deep Clustering-based Fair Outlier Detection (DCFOD) [9]: Enhances outlier detection with a 602 focus on fairness, combining deep clustering and adversarial training for representation learning. 603

Dataset

SOD

HBOS

PCA

OC

SVM

LOF

KNN

LODA

IForest

DA

GMM

ECOD

COPOD

DB

SCAN

CBLOF

DCOD

KMeans--

Deep

SVDD

DIF

UniCAD

(Scalar)

UniCAD

(Vector)

annthyroid

77.38

60.15

66.24

57.23

70.20

71.69

41.02

82.01

56.53

78.66

76.80

50.08

62.28

55.01

64.99

76.09

66.76

75.27

72.72

backdoor

68.77

71.56

80.16

85.04

85.79

80.58

66.38

72.15

55.98

86.08

80.97

76.55

81.91

79.57

89.11

78.83

92.87

87.28

89.24

breastw

93.97

98.94

95.13

80.30

40.61

97.01

98.49

98.32

N/A

99.17

99.68

85.20

96.86

99.02

97.05

63.36

77.45

98.15

98.56

campaign

69.16

78.55

72.78

65.70

59.04

72.27

51.67

71.71

56.03

76.10

77.69

50.60

64.34

63.16

63.51

54.42

67.53

73.52

73.64

celeba

48.44

76.18

79.38

70.70

38.95

59.63

60.17

70.41

44.74

76.48

75.68

50.36

73.99

91.41

56.76

45.17

65.29

81.38

82.00

census

62.12

64.89

68.74

54.90

47.46

66.88

37.14

59.52

59.65

67.63

69.07

58.50

60.17

72.84

63.33

54.16

59.66

67.90

67.84

glass

73.36

77.23

66.29

35.36

69.20

82.29

73.13

77.13

76.09

65.83

72.43

54.55

78.30

78.07

77.30

55.71

84.57

79.52

82.17

Hepatitis

67.83

79.85

75.95

67.75

38.06

52.76

64.87

69.75

54.80

75.22

82.05

68.12

73.05

48.38

64.64

57.45

74.24

75.53

80.62

http

78.04

99.53

99.72

99.59

27.46

3.37

12.48

99.96

N/A

98.10

99.29

49.97

99.60

99.53

99.55

60.38

99.49

99.53

99.52

Ionosphere

86.37

62.49

79.19

75.92

90.59

88.26

78.42

84.50

73.41

73.15

79.34

81.12

90.79

57.78

91.36

53.94

89.74

92.04

90.37

landsat

59.54

55.14

35.76

36.15

53.90

57.95

38.17

47.64

43.92

36.10

41.55

50.17

63.69

33.40

55.31

62.48

54.84

49.60

57.37

Lymphography

71.22

99.49

99.82

99.54

89.86

55.91

85.55

99.81

72.11

99.52

99.48

74.16

99.81

81.19

100.00

71.91

83.67

99.29

99.73

mnist

60.10

60.42

85.29

82.95

67.13

80.58

72.27

80.98

67.23

74.61

77.74

50.00

79.96

65.23

82.45

50.98

88.16

86.00

86.64

musk

74.09

100.00

100.00

80.58

41.18

69.89

95.11

99.99

76.85

95.40

94.20

50.00

100.00

42.19

72.16

66.02

98.22

99.92

100.00

pendigits

66.29

93.04

93.73

93.75

47.99

72.95

89.10

94.76

64.22

93.01

90.68

55.33

96.93

94.33

94.37

27.32

93.79

95.12

95.52

Pima

61.25

71.07

70.77

66.92

65.71

73.43

65.93

72.87

55.93

63.05

69.10

51.39

71.49

72.16

70.44

49.49

67.28

75.16

74.87

satellite

63.96

74.80

59.62

59.02

55.88

65.18

61.98

70.43

62.33

58.09

63.20

55.52

71.32

55.97

67.71

57.40

74.52

72.46

77.65

satimage-2

83.08

97.65

97.62

97.35

47.36

92.60

97.56

99.16

96.29

96.28

97.21

75.74

99.84

86.01

99.88

55.68

99.63

99.87

99.88

shuttle

69.51

98.63

98.62

97.40

57.11

69.64

60.95

99.56

97.92

99.13

99.35

50.40

93.07

97.20

69.97

51.81

97.00

99.15

98.75

skin

60.35

60.15

45.26

49.45

46.47

71.46

45.75

68.21

N/A

49.08

47.55

50.00

68.03

64.34

65.47

45.69

66.36

72.26

69.69

Stamps

73.26

90.73

91.47

83.86

51.26

68.61

87.18

91.21

88.89

87.87

93.40

52.08

69.89

93.41

79.78

59.48

87.95

91.37

94.18

thyroid

92.81

95.62

96.34

87.92

86.86

95.93

74.30

98.30

79.75

97.94

94.30

53.57

94.74

78.55

92.26

52.14

96.26

97.66

97.48

vertebral

40.32

28.56

37.06

37.99

49.29

33.79

30.57

36.66

53.20

40.66

25.64

49.74

41.01

38.13

38.14

37.81

47.20

33.11

47.37

vowels

92.65

72.21

65.29

61.59

93.12

97.26

70.36

73.94

60.58

62.24

53.15

57.50

92.12

51.56

93.45

49.87

81.02

88.38

92.09

Waveform

68.57

68.77

65.48

56.29

73.32

73.78

60.13

71.47

49.35

62.36

75.03

66.41

71.27

63.47

74.35

53.94

75.33

71.81

74.29

WBC

94.60

98.72

98.20

99.03

54.17

90.56

96.91

99.01

N/A

99.11

99.11

87.43

96.88

94.92

97.45

62.46

81.27

97.68

98.93

Wilt

53.25

32.49

20.39

31.28

50.65

48.42

26.42

41.94

37.29

36.30

33.40

49.96

34.50

44.71

34.91

45.90

39.46

48.95

52.56

wine

46.11

91.36

84.37

73.07

37.74

44.98

90.12

80.37

61.70

77.22

88.65

40.33

27.14

82.18

27.36

64.26

41.69

82.72

95.25

WPBC

51.28

51.24

46.01

45.35

41.41

46.59

49.31

46.63

47.80

46.65

49.34

52.22

45.32

49.67

45.01

44.01

44.69

48.02

49.90

Avg. Rank

11.00

8.26

8.98

11.59

13.59

10.00

13.24

7.09

13.24

9.19

8.29

14.21

8.07

10.90

8.71

15.48

8.38

5.41

3.59

(a) AUC-ROC

(b) AUC-PR

4

6

8

10

12

14

6

8

10

12

COPOD(7.3)

Galaxy (3.6)

IForest(6.8)

GMM(6.6)

(14)DeepSVDD

(12)LOF

(12)COF

(12)DAGMM

Galaxy (4.5)

IForest(7.5)

PCA(7.5)

(12)DeepSVDD

(12) COF

(12)LOF

(11)DAGMM

(12)LODA

HBOS (7.7)

(10)SOD

HBOS (7.6)

(10)OCSVM

COPOD(7.8)

(10)LODA

KMeans--(7.8)

(9.8)SOD

KMeans--(8.1)

(9.9)OCSVM

KNN (8.2)

(8.4)ECOD

GMM (8.2)

(8.6)ECOD

PCA(8.2)

KNN (8.3)

D.2.7 Neural Network-based Models 604

Deep Support Vector Data Description (DeepSVDD) [42]: Minimizes the volume of a hyper605 sphere enclosing network data representations, isolating anomalies outside this sphere. 606

Deep Isolation Forest for Anomaly Detection (DIF) [51]: Utilizes deep learning to enhance 607 traditional isolation forest techniques, offering improved anomaly detection in complex datasets 608 with minimal parameter tuning. 609

610

611

612

Each method's unique mechanism and application context provide a rich landscape of techniques for unsupervised anomaly detection, illustrating the field's diverse methodologies and the breadth of approaches to tackling anomaly detection challenges.

613

D.3 Supplementary Experimental Results

In the appendix, we detail the statistical analysis conducted to compare the performance of various 614 anomaly detectors. We obtained this diagram by conducting a Friedman test (p-value: 4.657e-19), 615 indicating significant differences among different detectors. We utilized average ranks and the 616 Nemenyi test to generate the critical difference diagram, as shown in Figure 5. It is noteworthy that 617 the vector version exhibits significantly superior performance compared to the scalar version across 618 more methods. The detailed outcomes for the AUCROC and AUCPR metrics, spanning 30 datasets 619 and against 17 baseline approaches, are showcased in Table 5 and Table 6. 620

D.4 Complexity Analysis 621

The complexity of each iteration in UniCAD involves three parts: constructing the outlier set, 622 updating the network parameters Θ , and optimizing the mixture model using the EM algorithm. 623

Dataset

SOD

HBOS

PCA

OC

SVM

LOF

KNN

LODA

IForest

DA

GMM

ECOD

COPOD

DB

SCAN

CBLOF

DCOD

KMeans--

Deep

SVDD

DIF

UniCAD

(Scalar)

UniCAD

(Vector)

annthyroid

18.84

16.99

16.12

10.37

15.71

16.74

7.06

30.47

9.64

25.35

16.58

7.60

13.74

10.01

15.41

21.75

18.93

26.37

25.03

backdoor

37.07

4.96

31.29

8.79

26.14

44.37

13.84

4.75

5.47

10.72

7.69

21.04

7.03

6.77

15.47

55.70

41.46

37.77

36.36

breastw

84.88

97.71

95.11

82.70

28.55

92.19

97.04

96.04

N/A

98.54

99.40

78.42

91.94

96.83

92.25

48.60

50.65

94.47

95.90

campaign

19.14

38.01

27.90

29.25

14.59

27.18

14.11

32.26

14.54

36.65

38.58

11.43

20.88

19.61

18.86

16.75

26.52

27.66

27.12

celeba

2.36

13.82

15.89

10.73

1.73

3.14

4.04

8.96

1.95

13.96

13.69

2.32

11.22

17.48

3.19

2.73

5.44

15.12

14.66

census

8.54

8.68

10.02

6.82

5.48

9.04

5.03

7.78

9.03

9.46

9.92

7.52

7.52

10.92

8.13

8.42

7.42

9.70

9.75

glass

18.73

11.82

10.05

8.02

20.11

20.26

13.37

10.99

24.58

15.35

9.78

6.88

11.57

9.66

14.66

8.46

18.86

13.29

15.33

Hepatitis

24.73

37.73

36.65

29.44

13.67

21.95

30.90

26.25

22.93

32.80

41.50

22.31

36.54

19.53

25.14

30.04

34.93

36.08

43.37

http

8.32

44.79

56.43

46.86

3.82

0.70

0.67

90.83

N/A

16.61

35.19

0.37

47.53

44.03

45.09

13.39

41.72

43.53

43.52

Ionosphere

85.88

41.78

73.92

74.54

88.07

90.41

73.04

80.41

64.97

64.69

69.89

63.04

89.77

47.63

91.36

43.24

87.45

89.55

87.61

landsat

26.38

22.03

16.18

16.21

24.69

24.65

18.86

19.81

24.48

16.24

17.48

20.80

31.05

15.57

22.40

36.92

24.35

20.84

23.27

Lymphography

22.00

91.83

97.02

93.59

23.08

38.69

44.54

97.31

19.52

90.87

88.68

7.66

97.31

12.34

100.00

34.58

32.84

91.69

96.66

mnist

19.15

12.51

39.93

33.20

20.90

35.53

25.86

27.71

23.75

17.45

21.35

9.21

30.60

23.59

37.12

20.18

44.55

41.19

41.94

musk

7.59

100.00

99.89

10.61

2.82

9.65

47.60

99.61

32.76

50.13

34.79

3.16

100.00

2.87

37.55

8.78

70.70

97.65

99.96

pendigits

4.46

29.27

23.65

23.52

3.78

6.50

18.71

26.05

4.67

30.65

21.22

2.94

32.87

22.21

32.67

1.53

23.75

24.86

21.68

Pima

48.24

56.61

54.03

50.00

47.18

55.14

44.09

55.82

41.55

50.45

55.19

36.65

52.99

50.24

53.50

35.02

46.34

54.66

54.23

satellite

47.23

67.25

59.64

57.61

37.68

50.01

61.94

65.92

58.33

52.22

56.58

37.56

61.43

43.31

54.68

41.77

68.92

71.68

75.13

satimage-2

26.11

78.04

85.69

82.71

4.30

39.14

80.52

93.45

22.07

64.49

76.55

12.08

97.09

8.12

97.13

2.58

72.90

97.33

97.31

shuttle

20.27

96.40

92.35

85.29

13.76

20.38

48.75

97.62

93.20

90.45

96.56

7.68

79.89

81.82

32.66

12.41

67.23

92.05

92.36

skin

24.61

23.70

17.40

19.03

18.25

28.72

18.44

26.08

N/A

18.37

17.99

20.89

28.34

26.29

25.58

19.06

25.36

28.87

28.72

Stamps

20.28

35.24

41.09

31.39

21.29

23.53

34.60

39.49

43.73

33.21

43.10

11.03

24.46

47.36

35.63

12.07

34.68

42.39

50.94

thyroid

23.56

50.98

44.34

21.23

20.81

34.98

14.68

63.11

16.06

51.06

19.64

9.44

29.88

10.56

31.69

2.70

50.36

60.99

60.06

vertebral

11.79

9.23

10.49

10.94

14.24

10.57

9.68

10.46

15.24

11.84

8.89

13.11

11.43

11.58

10.54

10.62

14.31

9.78

12.96

vowels

38.88

13.41

8.92

8.24

34.42

63.41

13.82

15.12

12.22

10.56

4.14

13.27

35.14

3.58

49.10

4.58

14.97

26.52

32.42

Waveform

9.66

5.86

5.79

4.37

11.33

13.04

4.71

6.24

3.11

4.76

6.90

5.33

17.93

4.26

19.74

4.41

11.28

6.49

7.83

WBC

54.00

73.56

82.29

89.87

5.57

66.55

78.67

90.49

N/A

86.19

86.19

30.25

67.31

33.43

71.88

8.99

13.32

68.69

83.14

Wilt

5.53

3.84

3.13

3.62

5.05

4.73

3.36

4.23

4.00

3.93

3.69

5.33

3.74

4.62

3.76

4.65

4.05

4.80

5.19

wine

7.95

43.08

30.87

21.56

7.77

8.43

48.82

25.96

17.51

23.54

45.71

8.11

5.98

24.44

6.27

18.78

8.38

21.40

49.59

WPBC

25.62

23.04

23.01

22.93

20.29

21.49

25.39

22.42

22.49

21.24

22.81

23.86

21.08

22.86

20.58

25.00

20.73

22.71

24.90

Avg. Rank

10.83

8.19

8.31

11.14

13.24

9.36

11.79

7.29

11.96

9.36

9.53

14.91

8.53

11.97

9.03

13.41

9.10

6.31

4.74

Constructing the outlier set requires a sorting operation, for which we use Numpy's built-in quantile 624 calculation with a time complexity of O ( N log N ) . Considering the number of network parameters 625 along with the computation of the loss function, the computational complexity for optimizing Θ is 626 approximately O ( TNDd + TNKd ) . The EM algorithm for the Student's t mixture model includes 627 two main steps: the E-step, where the complexity for computing the probability (or responsibility) 628 of each data point belonging to each component is approximately O ( NKd ) , and the M-step, where 629 the full computational complexity of updating the parameters (mean, covariance matrix) of each 630 component is O ( NKd 2 ) . In practice, we use diagonal covariance matrices, which reduces the 631 update complexity to roughly O ( NKd ) . If the EM algorithm requires T round to converge, its 632 time complexity is approximately O ( TNKd ) . Therefore, the time complexity for t -iterations is 633 O ( tN (log N + Td ( D + K ))) . 634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

E Additional Experiments on Graph

E.1 Baselines

Our proposed method was compared with 16 graph domain baseline methods grouped into three categories as follows:

Contrastive Learning-based Methods : This group includes CoLA [30], SLGAD [55], CONAD [53], and ANEMONE [20]. These methods primarily assume that the contrastive loss between anomalous nodes and their neighborhoods is more significant.

Autoencoder-based Methods : This category consists of MLPAE [43], GCNAE [22], DOMINANT [11], GUIDE [54], ComGA [31], AnomalyDAE [14], ALARM [37], DONE/AdONE [4] and AAGNN [57]. These methods focus on the reconstruction errors of anomalous nodes during the process of reconstructing the graph structure or features.

Clustering-based Methods : This category of methods encompasses SCAN [52], CBLOF [18], and DCFOD [45]. These methods generally identify anomalies by detecting if a sample deviates from the clustering.

E.2 Datasets

We assess the performance of our model using four graph benchmark datasets containing organic anomalies. Table 7 presents the statistical summary for each dataset. These datasets contain naturally occurring real-world anomalies and are valuable for assessing the performance of anomaly detection algorithms in real-world scenarios. The sources and compositions of these datasets are as follows:

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

Weibo [19] is a labeled graph comprising user posts extracted from the social media platform Tencent Weibo. The user-user graph establishes connections between users who exhibit similar topic labels. A user is considered anomalous if they have engaged in a minimum of five suspicious events, whereas normal nodes represent users who have not.

Reddit [25] consists of a user-subreddit graph extracted from the popular social media platform Reddit. This publicly accessible dataset encompasses user posts within various subreddits over a month. Each user is assigned a binary label indicating whether they have been banned on the platform. Our assumption is that banned users exhibit anomalous behavior compared to regular Reddit users.

Disney [34] is a co-purchase network of movies that includes attributes such as price, rating, and the number of reviews. The ground truth labels, indicating whether a movie is considered anomalous or not, were assigned by high school students through majority voting.

T-Finance [46] aims to identify anomalous accounts within a trading network. The nodes in this network represent unique anonymous accounts, each characterized by ten features related to registration duration, recorded activity, and interaction frequency. Graph edges denote transaction records between accounts. If a node is associated with activities such as fraud, money laundering, or online gambling, human experts will designate it as an anomaly.

E.3 Experiment Settings

In this experiment, we compared graph-based methods on relational data. For methods originally 672 designed around feature vectors, including CBLOF, DCFOD, and our approach, we uniformly 673 employed the same graph representation learning technique as described in BGRL [47]. Specifically, 674 we used a two-layer Graph Convolutional Network (GCN) for encoding, which produced output 675 embeddings with a dimensionality of 128. The training epochs were set to 3000, including a warm-up 676 period of 300 epochs. The hidden size of the predictor was set to 512, and the momentum was fixed 677 at 0.99. 678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

E.4 Performance Analysis

The performance of UniCAD compared to 16 baseline methods on the four datasets are summarized in Table 8. From the results, we have the following observations: Our model consistently outperforms the baseline methods on most datasets, underlining its effectiveness in anomaly detection even within graph data contexts. This highlights the superiority of UniCAD in detecting anomalies in real-world graph data.

When comparing UniCAD with the four contrastive learning-based methods, it exhibits a distinct advantage, outperforming them by a substantial margin across all metrics. Unlike contrastive learning methods that rely on the local neighborhood for anomaly detection, UniCAD leverages the global clustering distribution. This key difference contributes to its consistently superior performance. Although CONAD incorporates human prior knowledge about anomalies, enabling it to outperform other similar methods on the Weibo and Disney datasets, it still falls short compared to our proposed UniCAD.

Compared to the autoencoder-based methods, UniCAD offers the advantage of lower memory requirements along with better performance. Graph autoencoders typically reconstruct the entire adjacency matrix during full graph training, resulting in memory usage of at least O ( N 2 ) . In contrast, UniCAD, as a clustering-based method, only requires O ( N × K ) . Among the autoencoder-based methods, GCNAE, DONE, and AdONE can be extended to the T-Finance dataset as they only reconstruct the sampled subgraphs rather than the entire adjacency matrix. However, UniCAD still showcases superior performance while being more memory-efficient.

UniCADalso demonstrates superior performance compared to various other clustering-based methods, including traditional structural clustering (SCAN) methods that treat the embedding from BGRL as tabular data (CBLOF, DCFOD).

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims presented in the abstract and introduction are consistent with the paper's contributions and accurately outline the scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: The limitations of the method's application scope are discussed in Section 5 of the paper, along with considerations for future work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The full set of assumptions and complete proofs for each theoretical result are provided and can be found in the appendix, specifically in Section B, ensuring that the theoretical framework is transparent and verifiable.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All necessary information for reproducing the main experimental results, including dataset links, baseline comparisons, and the methodologies of our proposed approach, are comprehensively included within the submission files, facilitating transparency and reproducibility of the research findings.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The paper ensures open access to both the data and code necessary for reproducing the main experimental results, complemented by detailed instructions in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

889

890

891

892

893

894

895

896

897

898

899

900

901

902

903

904

905

906

907

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Detailed information regarding the training and test setups, including data splits, hyperparameters and their selection process, the type of optimizer used, and other relevant details, are thoroughly documented in Section 4.2 of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports the statistical significance of the experiments by detailing the results of the Friedman test and the Nemenyi test in Appendix D.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

908

909

910

911

912

913

914

915

916

917

918

919

920

921

922

923

924

925

926

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

955

956

957

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Detailed information on the compute resources, including the type of compute workers (CPU/GPU), memory, and execution time for each experiment, is provided in the supplementary materials, enabling accurate reproduction of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research adheres to the NeurIPS Code of Ethics, including considerations for anonymity, fairness, and transparency, with no deviations reported or necessary under current laws or regulations.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper thoroughly discusses both the potential positive impacts, such as enhancements in anomaly detection for critical applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

958

959

960

961

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper appropriately credits the creators of the utilized assets, including code, data, and models, and explicitly mentions the licenses and terms of use, ensuring compliance with the original terms set by the asset owners.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

1044

1045

1046

1047

1048

1049

1050

1051

1052

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses unsupervised anomaly detection by proposing a method named UniCAD. The authors aim to enhance anomaly detection performance by establishing a theoretical connection between representation learning, clustering, and anomaly detection. They introduce a unified framework that jointly optimizes these three components, using a probabilistic mixture model and a Student's-t distribution for robust representation learning and clustering. The framework also includes an anomaly-aware data likelihood objective, which reduces the impact of anomalous data on the learning process. Additionally, the authors propose a gravity-inspired anomaly scoring method that leverages relationships between samples and clusters.\n\nSTRENGTHS:\n1. Modeling the connection between representation learning, clustering, and anomaly detection is highly relevant. This paper effectively demonstrates how these three tasks are interrelated and can be jointly optimized to improve anomaly detection performance.\n\n2. The paper is well-written, presenting its hypothesis and method clearly. \n\n3. The results are impressive and demonstrate the effectiveness of UniCAD.\n\nWEAKNESSES:\n1. The ablation study on the hyperparameters $k$ and $l$ is insufficient. The authors only present results from a single dataset, satimage-2, where their method achieves an almost perfect score. It would be more informative to perform ablation studies across all 30 datasets or at least a subset where the model also shows lower performance. This broader analysis would demonstrate how these hyperparameters affect the average ranking of the method, similar to the results reported in the paper's table.\n\n2. The authors introduce a $g(\\Theta)$ term to prevent shortcut solutions, mentioning it in Equation 15. However, they do not discuss its importance or impact on performance after its introduction. Key questions remain unanswered, such as how the $g(\\Theta)$ term affects the model's performance, what happens if it is removed, and how the autoencoder is implemented. These details are crucial, as the regularization term may significantly influence the results.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors mention some of the limitations, but they do not address the potential negative impact of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes UniCAD, a theoretically unified framework for representation learning, clustering, and anomaly detection. This paper first introduces the mixture of Student-t distribution $p(x|\\Theta, \\Phi)$ with degree of freedom $\\nu=1$ based on a representation learner $f_\\Theta$ using NN. Then, this paper combines with an anomaly indicator $\\delta$ for maximum likelihood estimation. Parameters $(\\Theta, \\Phi)$ are optimized by EM algorithm and SGD. In addition, when detecting anomalies, an improved score is used with reference to gravity. The UniCAD achieved good performance on experiments with various datasets.\n\nSTRENGTHS:\n- This paper is well written and easy to follow.\n- Good experimental results.\n\nWEAKNESSES:\n- We have several questions about the proposed method and experiments. Please see Qustions.\n- The comparison with DeepSVDD and DIF is excellent, but I think the paper also needs to be compared with other Deep anomaly detection methods. For example, DROCC [1].\n\n[1] Goyal, Sachin, et al. \"DROCC: Deep robust one-class classification. \"International conference on machine learning. PMLR, 2020.\n\nQUESTIONS:\n- if the degrees of freedom is fixed to 1, the benefit of the t-distribution disappears. Why not learn the degrees of freedom as well?\n- I think $\\tilde{F}_{ik}$ is a scalar and the norm of $\\tiled{r}_{ik}$ is 1, so I don't see the difference between Eq. (9) and Eq. (8).\n- As shown in Figure 2, I think the proposed method strongly depends on the hyperparameters $l$ and $k$. Is there any criteria for setting these values?\n- The method in this paper seems incremental. (Especially until 3.1.) I think the main novelty is the gravity-based anomaly score described in 3.2. To what extent does this anomaly score improve performance compared to the regular anomaly score? Also, can you give a theoretical explanation?\n\n\nIf the above concerns are remedied and a comparison is made between SOTA's Deep Anomaly Detection methodology, I intend to raise the score.\n\nLIMITATIONS:\n- Hyper-parameter sensitivity seems to be one limitation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel probabilistic mixture model for unsupervised anomaly detection (UAD) that unifies representation learning, clustering, and anomaly detection into a single theoretical framework. The proposed UniCAD model addresses the lack of a unified approach in existing methods, which often consider these components separately or in pairs. The experimental results show that UniCAD consistently outperformed other methods in terms of AUC-ROC and AUC-PR. The model’s iterative optimization process using EM was also highlighted as effective and convergent.\n\nSTRENGTHS:\n- This paper introduces a novel integration of a probabilistic mixture model that unifies representation learning, clustering, and anomaly detection into a single theoretical framework.\n- The proposed approach is well-motivated (Fig. 1) and supported by a robust theoretical foundation that maximizes anomaly-aware data likelihood, ensuring the model effectively leverages the interplay between representation learning, clustering, and anomaly detection.\n- The paper is well-written, offering clear and comprehensive explanations of the proposed method, including detailed theoretical derivations and intuitive motivations for the design choices. The methodology section is particularly well-structured, logically outlining the steps and equations involved in the proposed model.\n- The comprehensive evaluation design underscores the robustness of the proposed method.\n\nWEAKNESSES:\n- The connection between force analysis and anomaly detection, particularly between Equations 7 and 8 in Section 3.2.1, could benefit from further justification. While the analogy is interesting, it may not be immediately intuitive for all readers.\n- The iterative optimization process may pose scalability issues for large datasets. An in-depth analysis and discussion of this would further strengthen the quality of this research.\n- Although the model maps data to a low-dimensional representation space, the effectiveness of this mapping for very high-dimensional datasets could be explored further.\n\nQUESTIONS:\nI have no specific questions.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose UniCAD to jointly model representation learning,\nclustering and anomaly detection.  The main objective is maximizing\nthe product of anomaly indicator (1 is normal, 0 is anomaly) and the\njoint probability of instance x_i and cluster c_k given parameters for\nrepresentation learning theta and clustering phi.  The joint\nprobability is decomposed into the prior of c_k and likelihood of\np(x_i|c_k), which is modeled by a Student's-t distribution on the\ndistance between representation z_i and mean mu_k with covariance\nSigma_k.  p(x_i) is the marginal over c_k.  Anomaly indicator delta is\nzero for p(x_i) in the lowest l percentage.  The anomaly score is\n1/p(x_i).\n\nCompared to Newton's law of Universal Gravitation, the anomaly score\nfunction has similar components, except for the unit vector r_ik\n(which indicates the directions of forces, beyond the\nmagnitudes). Hence, they incorporated r_ik into their anomaly score\nfunction.\n\nFor updating the clustering parameter phi (mixture weights, means,\ncovariance), they use EM.  In the E-step, they estimate the posterior\np(c_k|x_i).  In the M-step, they estimate phi.  For updating\nrepresentation parameters theta, they use gradient descent to minimize\nnegative log likelihood of instances, together with a reconstruction\nloss via an autoencoder to prevent shortcut solutions.\n\nFor empirical comparisons, they use 30 tabular data sets and 17\nexisting algorithms.  The proposed approach generally outperforms the\nothers in terms of average rank in AUCROC.  The vector version of\nanomaly score function is ranked higher than the scalar version.  On\ncomputation time, UniCAD is in the middle among 5 algorithms.  Ablation\nstudies indicate the contributions of the different components.\n\nSTRENGTHS:\nThe main contribution is combining representation learning,\nclustering, and anomaly detection in a unified single probabilistic\nformulation, which is interesting.\n\nThe empirical results indicate that UniCAD compares favorably against\n17 existing techniques on 30 tabular datasets.  Compared to four\nexisting algorithms, computation is not the most intensive.\n\nThe paper is generally well written.\n\nWEAKNESSES:\nThe clustering part is similar to a typical Gaussian mixture model for clustering via EM, except for t-distribution instead of Gaussian and the scaling factor.\n\nTwo neural-network-based approaches were compared.  As UniCAD utilizes\nrepresentation learning, comparing with more approaches that utilize\nrepresentation learning would be significant.  Approaches without\nrepresentation learning have an inherent disadvantage.\n\nSome parts could be clarified--see Questions.\n\nQUESTIONS:\nEq 11: What is the motivation for the scale factor u_ik, used in Eq 13\nand 14?\n\nHow is K, number of clusters, determined?\n\nSince the method has representation learning, how does the method\nprevent the trivial solution of having most/all instances in one\ncluster?\n\nSec 3.2.1: Consider (a simple case of) only two clusters with the same \ncovariance and \"mass\", but different centroids.  If an instance is in the middle\nbetween the two clusters, the two r_ik vectors will be in opposite directions,\nresulting in an anomaly score of zero.  In gravitational forces, the\ntwo opposite forces cancel out.  However, in anomaly detection, that\nmight not be desirable, particularly, when the two clusters are far\naway.  Any insights?\n\nWhat is the \"schedule\" for updating phi and theta?  Updating one to\nconvergence before updating the other to convergence?\n\nIf one \"round\" is updating phi to convergence and then theta to\nconvergence?  Are there multiple rounds?  If so, what is the overall\nstopping criterion?\n\nLIMITATIONS:\nLimitations of the proposed approach do not seem to be mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}