Towards a Unified Framework of Clustering-based Anomaly Detection

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 14:19:42

Abstract: Unsupervised Anomaly Detection (UAD) plays a crucial role in identifying abnormal patterns within data without labeled examples, holding significant practical implications across various domains. Although the individual contributions of representation learning and clustering to anomaly detection are well-established, their interdependencies remain under-explored due to the absence of a unified theoretical framework. Consequently, their collective potential to enhance anomaly detection performance remains largely untapped. To bridge this gap, in this paper, we propose a novel probabilistic mixture model for anomaly detection to establish a theoretical connection among representation learning, clustering, and anomaly detection. By maximizing a novel anomaly-aware data likelihood, representation learning and clustering can effectively reduce the adverse impact of anomalous data and collaboratively benefit anomaly detection. Meanwhile, a theoretically substantiated anomaly score is naturally derived from this framework. Lastly, drawing inspiration from gravitational analysis in physics, we have devised an improved anomaly score that more effectively harnesses the combined power of representation learning and clustering. Extensive experiments, involving 17 baseline methods across 30 diverse datasets, validate the effectiveness and generalization capability of the proposed method, surpassing state-of-the-art methods.

Main paper: 
 introduction: 
 Unsupervised Anomaly Detection (UAD) refers to the task dedicated to identifying abnormal patterns or instances within data in the absence of labeled examples [8] . It has long received extensive attention in the past decades for its wide-ranging applications in numerous practical scenarios, including financial auditing [3] , healthcare monitoring [44] and e-commerce sector [23] . Due to the lack of explicit label guidance, the key to UAD is to uncover the dominant patterns that widely exist in the dataset so that samples do not conform to these patterns can be recognized as anomalies. To achieve this, early works [7] have heavily relied on powerful unsupervised representation learning methods to extract the normal patterns from high-dimensional and complex data such as images, text, and graphs. More recent works [45, 2] have utilized clustering, a widely observed natural pattern in real-world data, to provide critical global information for anomaly detection and achieved tremendous success.While the individual contributions of representation learning and clustering to anomaly detection are well-established, their interrelationships remain largely unexplored. Intuitively, discriminative representation learning can leverage accurate clustering results to differentiate samples from distinct clusters in the embedding space (i.e., ➀). Similarly, it can utilize accurate anomaly detection to avoid preserving abnormal patterns (i.e., ➁). For accurate clustering, it can gain advantages from representation learning by operating in the discriminative embedding space (i.e., ➂). Meanwhile, it can potentially benefit from accurate anomaly detection by excluding anomalies when formulating clusters (i.e., ➃). Anomaly detection can greatly benefit from both discriminative representation learning and accurate clustering (i.e., ➄ & ➅). However, these benefits hinge on the successful identification of anomalies and the reduction of their detrimental impact on the aforementioned tasks. As depicted in Figure 1 , the integration of these three elements exhibits a significant reciprocal nature. In summary, representation learning, clustering, and anomaly detection are interdependent and intricately intertwined. Therefore, it is crucial for anomaly detection to fully leverage and mutually enhance the relationships among these three components.Despite the intuitive significance of the interactions among representation learning, clustering, and anomaly detection, existing methods have only made limited attempts to exploit them and fall short of expectations. On one hand, some methods [58] have acknowledged the interplay among these three factors, but their focus remains primarily on the interactions between two factors at a time, making only targeted improvements. For instance, some strategies include explicitly removing outlier samples during the clustering process [9] or designing robust representation learning methods [10] to mitigate the influence of anomalies. On the other hand, recent methods [45] have begun to explore the simultaneous optimization of these three factors within a single framework. However, these attempts are still in the stage of merely superimposing the objectives of the three factors without a unified theoretical framework. This lack of a guiding framework prevents the adequate modeling of the interdependencies among these factors, thereby limiting their collective contribution to a unified anomaly detection objective. Consequently, we aim to address the following question: Is it possible to employ a unified theoretical framework to jointly model these three interdependent objectives, thereby leveraging their respective strengths to enhance anomaly detection?In this paper, we try to answer this question and propose a novel model named UniCAD for anomaly detection. The proposed UniCAD integrates representation learning, clustering, and anomaly detection into a unified framework, achieved through the theoretical guidance of maximizing the anomaly-aware data likelihood. Specifically, we explicitly model the relationships between samples and multiple clusters in the representation space using the probabilistic mixture models for the likelihood estimation. Moreover, we creatively introduce a learnable indicator function into the objective of maximum likelihood to explicitly attenuate the influence of anomalies on representation learning and clustering. Under this framework, we can theoretically derive an anomaly score that indicates the abnormality of samples, rather than heuristically designing it based on clustering results as existing works do. Furthermore, building upon this theoretically supported anomaly score and inspired by the theory of universal gravitation, we propose a more comprehensive anomaly metric that considers the complex relationships between samples and multiple clusters. This allows us to better utilize the learned representations and clustering results from this framework for anomaly detection.To sum up, we underline our contributions as follows:• We propose a unified theoretical framework to jointly optimize representation learning, clustering, and anomaly detection, allowing their mutual enhancement and aid in anomaly detection.• Based on the proposed framework, we derive a theoretically grounded anomaly score and further introduce a more comprehensive score with the vector summation, which fully releases the power of the framework for effective anomaly detection.• Extensive experiments have been conducted on 30 datasets to validate the superior unsupervised anomaly detection performance of our approach, which surpassed the state-of-the-art through comparative evaluations with 17 baseline methods. 
related work: 
 Typical unsupervised anomaly detection (UAD) methods calculate a continuous score for each sample to measure its anomaly degree. Various UAD methods have been proposed based on different assumptions, making them suitable for detecting various types of anomaly patterns, including subspace-based models [24] , statistical models [16] , linear models [49, 32] , density-based models [6, 38] , ensemble-based models [39, 29] , probability-based models [40, 58, 28, 27] , neural networkbased models [42, 51] , and cluster-based models [18, 9] . Considering the field of anomaly detection has progressed by integrating clustering information to enhance detection accuracy [26, 56] , we primarily focus on and analyze anomaly patterns related to clustering, incorporating a global clustering perspective to assess the degree of anomaly. Notable methods in this context include CBLOF [18] , which evaluates anomalies based on the size of the nearest cluster and the distance to the nearest large cluster. Similarly, DCFOD [45] introduces innovation by applying the self-training architecture of the deep clustering [50] to outlier detection. Meanwhile, DAGMM [58] combines deep autoencoders with Gaussian mixture models, utilizing sample energy as a metric to quantify the anomaly degree.In contrast, our approach introduces a unified theoretical framework that integrates representation learning, clustering, and anomaly detection, overcoming the limitations of heuristic designs and the overlooked anomaly influence in existing methods. 
methodology: 
 In this section, we first define the problem we studied and the notations used in this paper. Then we elaborate on the proposed method UniCAD. More specifically, we first introduce a novel learning objective that optimizes representation learning, clustering, and anomaly detection within a unified theoretical framework by maximizing the data likelihood. A novel anomaly score with theoretical support is also naturally derived from this framework. Then, inspired by the concept of universal gravitation, we further propose an enhanced anomaly scoring approach that leverages the intricate relationship between samples and clustering to detect anomalies effectively. Finally, we present an efficient iterative optimization strategy to optimize this model and provide a complexity analysis for the proposed model. Definition 1 (Unsupervised Anomaly Detection). Given a dataset X ∈ R N ×D comprising N instances with D-dimensional features, unsupervised anomaly detection aims to learn an anomaly score o i for each instance x i in an unsupervised manner so that the abnormal ones have higher scores than the normal ones.Previous research has demonstrated the importance of discriminative representation and accurate clustering in anomaly detection [45] . However, the presence of anomalous samples can significantly disrupt the effectiveness of both representation learning and clustering [12] . While some existing studies have attempted to integrate these three separate learning objectives, the lack of a unified theoretical framework has hindered their mutual enhancement, leading to suboptimal results.To tackle this issue, in this paper, we propose a unified and coherent approach that considers representation learning, clustering, and anomaly detection by maximizing the likelihood of the observed data. Specifically, we denote the parameters of representation learning as Θ, the clustering parameter as Φ, and the dynamic indicator function for anomaly detection as δ(•). These parameters are optimized simultaneously by maximizing the likelihood of the observed data X:EQUATIONwhere c i represents the latent cluster variable associated with x i , and c i = k denotes the probabilistic event that x i belongs to the k-th cluster. The δ(x i ) is an indicator function that determines whether a sample x i is an anomaly of value 0 or a normal sample of value 1.Based on the aforementioned advantages of MMs, we estimate the likelihood p(x i |Θ, Φ) with mixture models defined as:EQUATIONwhere Φ = {ω k , µ k , Σ k }. The mixture model is parameterized by the prototypes µ k , covariance matrices Σ k , and mixture weights ω k from all clusters.K k=1 ω k = 1, and k = 1, 2, • • • , K.In practice, the samples are usually attributed to high-dimensional features and it is challenging to detect anomalies from the raw feature space [41] . Therefore, modern anomaly detection methods [42, 58] often map raw data samples X = {x i } ∈ R N ×D into a low-dimensional representation space Z = {z i } ∈ R N ×d with a representation learning function z i = f Θ (x i ) and detect anomalies within this latent representation space.Following this widely adopted practice, we model the distribution of samples in the latent representation space with a multivariate Student's-t distribution giving its cluster c i = k. The Student's-t distribution is robust against outliers due to its heavy tails. Bayesian robustness theory leverages such distributions to dismiss outlier data, favoring reliable sources, making the Student's-t process preferable over Gaussian processes for data with atypical information [1] . Thus the probability distribution of generating x i with latent representation z i given its cluster c i = k can be expressed as:EQUATIONwhere z i = f Θ (x i ) denotes the representation obtained from the data mapped through the neural network parameterized by Θ. Γ denotes the gamma function while ν is the degree of freedom.Σ k is the scale parameter. D M (z i , µ k ) = (z i -µ k ) T Σ -1 k (z i -µ k )represents the Mahalanobis distance [33] . In the unsupervised setting, as cross-validating ν on a validation set or learning it is unnecessary, ν is set as 1 for all experiments [50, 48] . The overall marginal likelihood of the observed data x i can be simplified as:p(x i |Θ, Φ) = K k=1 ω k • π -1 • |Σ k | -1/2 1 + D M (z i , µ k ) 2 .(4)As we have discussed, the indicator function δ(x i ) not only benefits both representation and clustering but also directly serves as the output of anomaly detection. Ideally, with the percentage of outliers denoted as l, an optimal solution for δ(x i ) that maximizes the objective function J(Θ, Φ) entails setting all δ(x i ) = 0 for x i among the l percent of outliers with lowest generation possibility p(x i |Θ, Φ), and otherwise δ(x i ) = 1 is set for the remaining normal samples. Therefore, the indicator function is determined as:EQUATIONAs this method involves sorting the samples based on the generation probability as being anomalous, the values of p(x i |Θ, Φ) can serve as a form of anomaly score, a classic approach within the mixture model framework [40, 58] . This suggests that the likelihood of a sample being anomalous is inversely related to its generative probability since a lower generative probability indicates a higher chance of the sample being an outlier. Thus the anomaly score of sample x i can be defined as:EQUATION)In practical applications, it is proved that anomaly scores derived from generation probabilities often yield suboptimal performance [17] . This observation prompts a reconsideration of how to fully leverage the complex relationships among samples or even across multiple clusters for anomaly detection. In this section, we first provide a brief introduction to the concept of Newton's Law of Universal Gravitation [35] and then demonstrate how the anomaly score is intriguingly similar to this cross-field principle. Finally, we discuss the advantages of introducing the vector sum operation into the anomaly score inspired by the analogy.To begin with, Newton's Law of Universal Gravitation [35] stands as a fundamental framework for describing the interactions among entities in the physical world. According to this law, every object in the universe experiences an attractive force from another object. In classical mechanics, force analysis involves calculating the vector sum of all forces acting on an object, known as the resultant force, which is crucial in determining an object's acceleration or change in motion:EQUATIONwhere ⃗ F ik represents the k-th force acting on the object i. This force is proportional to the product of their masses, (m i and m k ), and inversely proportional to the square of the distance r ik between them.G represents the gravitational constant, and ⃗ r ij is the unit direction vector.Similarly, if denoting:F ik = p(x i , c i = k|Θ, Φ) = ω k • π -1 •|Σ k | -1/2 1+D M (zi,µ k ) 2, the score of Equation (6) bears analogies to the summation of the magnitudes of forces as:EQUATIONwhereG = π -1 , m k = ω k |Σ k | -1/2 , m i = 1, and r ik = 1 + D M (z i , µ k ) 2 .Here, r ik is taken as the measure of distance within the representation space, modified slightly by an additional term for smoothness. The constant G serves a role akin to the gravitational constant in this analogy, whereas m k resembles the concept of mass for the cluster. The notation m i suggests a standardization where the mass of each data point is considered uniform and not differentiated.Comparing Equation (7) with Equation ( 8), what still differs is that, unlike a simple sum of the scalar value, the resultant force ⃗ F i,total employs the vector sum and incorporates both the magnitude and direction r ik of each force. This distinction is crucial because forces in different directions can neutralize each other with a large angle between them or enhance each other's effects with a small angle. Inspired by this difference, we consider modeling the relationship between samples and clusters as a vector, and aggregating them through vector summation. The vector-formed anomaly score o V i is defined as:EQUATIONwhere ⃗ r ik represents the unit direction vector in the representation space from the sample z i to the cluster prototype µ k , and ∥ • ∥ represents the L 2 norm.Given the challenge posed by the interdependence of the parameters of the network Θ and those of the mixture model {ω k , µ k , Σ k } in joint optimization, we propose an iterative optimization procedure.The pseudocode for training the model is presented in Algorithm 1 in the appendix.To update the parameters of the mixture model Φ = {ω k , µ k , Σ k }, we use the Expectation-Maximization (EM) algorithm to maximize equation ( 1) [36] . The detailed derivation is included in Appendix B.During the E-step of iteration (t + 1), our goal is to compute the posterior probabilities of each data point belonging to the k-th cluster within the mixture model. Given the observed samplex i and the current estimates of the parameters Θ (t) and Φ (t) , the expected value of the likelihood function of latent variable c k , or the posterior possibilities, can be expressed as:EQUATION)The scale factor [36] serving as an intermediate result for subsequent updates in the M-step is :EQUATION.M-step. In the M-step of iteration (t + 1), given the gradients ∂J(Θ,Φ)EQUATION∂J(Θ,Φ) ∂Σ k= 0, we derive the analytical solutions for the mixture model parameters ω k , µ k , and Σ k .Assume the anomalous ratio is l ∈ [0, 1], the number of the normal samples is n = int(l * N ). The updating process for {ωEQUATION, Σ} is as follows:• The mixture weights ω k are updated by averaging the posterior probabilities over all data points with the number of samples , reflecting the relative presence of each component in the mixture:EQUATION• The prototypes µ k are updated to be the weighted average of the data points, where weights are the posterior probabilities:EQUATION• The covariance matrices Σ k are updated by considering the dispersion of the data around the newly computed prototypes:EQUATION)We focus on anomaly-aware representation learning and use stochastic gradient descent to optimize the network parameters Θ, by minimizing the following joint loss:EQUATIONwhere J(Θ, Φ) = log p(X|Θ, Φ). An additional constraint term g(Θ) is introduced to prevent shortcut solution [15] . In practice, an autoencoder architecture is implemented, utilizing a reconstruction loss g(Θ) = ∥x -x∥ 2 as the constraint.These updates are iteratively performed until convergence, resulting in optimized model parameters that best fit the given data according to the mixture model framework.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: The full set of assumptions and complete proofs for each theoretical result are provided and can be found in the appendix, specifically in Section B, ensuring that the theoretical framework is transparent and verifiable.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
datasets & baselines: 
 We evaluated UniCAD on an extensive collection of datasets, comprising 30 tabular datasets that span 16 diverse fields. We specifically focused on naturally occurring anomaly patterns, rather than synthetically generated or injected anomalies, as this aligns more closely with real-world scenarios. The detailed descriptions are provided in Table 4 of Appendix D.1. Following the setup in ADBench [17] , we adopt an inductive setting to predict newly emerging data, a highly beneficial approach for practical applications.To assess the effectiveness of UniCAD, we compared it with 17 advanced unsupervised anomaly detection methods, including: (1) traditional methods: SOD [24] and HBOS [16] ; (2) linear methods:PCA [49] and OCSVM [32] ; (3) density-based methods: LOF [6] and KNN [38] ; (4) ensemble-based methods: LODA [39] and IForest [29] ; (5) probability-based methods: DAGMM [58] , ECOD [28] ,and COPOD [27] ; (6) cluster-based methods: DBSCAN [13] , CBLOF [18] , DCOD [45] and KMeans-- [9] ; and (7) neural network-based methods: DeepSVDD [42] and DIF [51] . These baselines encompass the majority of the latest methods, providing a comprehensive overview of the state-ofthe-art. For a detailed description, please refer to Appendix D.2.In the unsupervised setting, we employ the default hyperparameters from the original papers for all comparison methods. Similarly, the UniCAD also utilizes a fixed set of parameters to ensure a fair comparison. For all datasets, we employ a two-layer MLP with a hidden dimension of d = 128 and ReLU activation function as both encoder and decoder. We utilize the Adam optimizer [21] with a learning rate of 1e -4 for 100 epochs. For the EM process, we set the maximum iteration number to 100 and a tolerance of 1e -3 for stopping training when the objectives converge. The number of components in the mixture model is set as k = 10, and the proportion of the outlier is set as l = 1%.We evaluate the methods using Area Under the Receiver Operating Characteristic (AUC-ROC) and Area Under the Precision-Recall Curve (AUC-PR) metrics [17] , reporting the average ranking (Avg.Rank) across all datasets. All experiments are run 3 times with different seeds, and the mean results are reported.Performance Comparison. Table 1 presents a comparison of UniCAD with 10 unsupervised baseline methods across 30 tabular datasets using the AUC-ROC metric. The experimental results, which encompass 17 baselines, are included in Tables 5 and 6 of Appendix D.3, with additional experiments on other data domains presented in Appendix E. Our proposed UniCAD achieves the top average ranking, exhibiting the best or near-best performance on a larger number of datasets and confirming advanced capabilities. It is noteworthy that there is no one-size-fits-all unsupervised anomaly detection method suitable for every type of dataset, as demonstrated by the observation that other methods have also achieved some of the best results on certain datasets. However, our model showcased a remarkable ability to generalize across most datasets featuring natural anomalies, as evidenced by statistical average ranking. As for clustering-based methods such as KMeans--, DCOD, and CBLOF, they mostly rank in the top tier among all baseline methods, supporting the advantage of combining deep clustering with anomaly detection. However, our method significantly outperformed these methods by mitigating their limitations and further providing a unified framework for joint representation learning, clustering, and anomaly detection.Effectiveness of Vector Sum in Anomaly Scoring. As demonstrated in Table 1 , we compare the anomaly score o i derived directly from the generation possibility with its vector summation form o V i .According to our statistical findings, we observe that vector scores o V i consistently outperform scalar scores o i . This indicates that the introduction of the vector summation, analogous to the concept of resultant force, makes a substantial difference in anomaly detection scenarios involving multiple clusters. The performance gains of the vector sum scores strongly demonstrate the effectiveness of the UniCAD in capturing the subtle differences in the distinctions among multiple clusters and underscore the utility of this factor in the context of anomaly detection based on clustering. The performance remained relatively stable throughout the last steps, illustrating the effectiveness and convergence of iterative EM optimization.Runtime Comparison. We present a analysis of the runtime performance of various methods, including our proposed approach, as detailed in Table 2 . Our experiments, conducted on the backdoor dataset, reveal that while non-deep learning methods exhibit lower runtime, they often simplify the problem space excessively, failing to capture the complex non-linear relationships present in the data. In contrast, our method, when compared to existing deep learning techniques, demonstrates a significant reduction in computational time. This indicates that our approach not only manages to efficiently model complex patterns but also achieves an optimal balance between computational efficiency and modeling capability.In this section, we examine the contributions of different components in UniCAD. Tables 3 reports the results. We make three major observations. Firstly, the anomaly detection performance experiences a significant drop when replacing the Student's t distribution with a Gaussian distribution for the Mixture Model, highlighting the robustness of the Student's t distribution in unsupervised anomaly detection.Secondly, omitting the likelihood maximization loss (w/o J(Θ, Φ)) also results in a considerable decrease in overall performance. This observation underscores the importance of deriving both the optimization objectives and anomaly scores from the likelihood generation probability through a theoretical framework, which allows for unified joint optimization of anomaly detection and clustering in the representation space. Furthermore, the indicator function δ(x i ) also contributes to a performance increase. These results further confirm the effectiveness of our UniCAD in mitigating the negative influence of anomalies in the clustering process, as the existence of outliers may significantly degrade the performance of clustering. In summary, all these ablation studies clearly demonstrate the effectiveness of our theoretical framework in simultaneously considering representation learning, clustering, and anomaly detection.In this section, we conducted a sensitivity analysis on key hyperparameters of the model applied to the donors dataset, focusing on the number of clusters k and the proportion of the outlier set l.The results of this analysis are illustrated in Figure 2 . Notably, the optimal range for l tends to be lower than the actual proportion of anomalies in the dataset. Furthermore, a pattern was observed with the number of clusters k, where the model performance initially improved with an increase in k, followed by a subsequent decline. This suggests the existence of an optimal range for the number of clusters, which should be carefully selected based on the specific application context.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: All necessary information for reproducing the main experimental results, including dataset links, baseline comparisons, and the methodologies of our proposed approach, are comprehensively included within the submission files, facilitating transparency and reproducibility of the research findings.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
conclusion: 
 This if i = 1 then 4:X i ← X; Re-order the point inX such that o 1 ≥ • • • ≥ o n ; 7:L i ← {x 1 , . . . , x ⌊N * l⌋ };8:X i ← X \ L i ; 9:end if while |J(Θ, Φ) -J old (Θ, Φ)| > λ do 12:J old (Θ, Φ) = J(Θ, Φ);13:Calculate τ with Equation (10);14:Update {ω k , µ k , Σ k } with Equation ( 12), ( 13) and ( 14);  
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Detailed information on the compute resources, including the type of compute workers (CPU/GPU), memory, and execution time for each experiment, is provided in the supplementary materials, enabling accurate reproduction of the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper thoroughly discusses both the potential positive impacts, such as enhancements in anomaly detection for critical applications.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper appropriately credits the creators of the utilized assets, including code, data, and models, and explicitly mentions the licenses and terms of use, ensuring compliance with the original terms set by the asset owners.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]  
 
 Appendix: This appendix provides the detailed derivation of the Expectation-Maximization (EM) algorithm for optimizing the parameters of a mixture model based on Student's t-distribution. The focus is on deriving analytical solutions for the maximization of the parameters Φ = {µ k , Σ k , ω k } of the mixture components. The EM algorithm alternates between two steps:In the E-step, we calculate the posterior probabilities τ ik , representing the probability of data point i belonging to cluster k, given the current parameters. The posterior probabilities for a Student's t-distribution mixture model are formulated as:EQUATIONwhere τ (z i |µ k , Σ k ) denotes the Student's t-distribution for data point i with respect to cluster k, and K is the number of mixture components.The Student's t-distribution is depicted as a hierarchical conditional probability, resembling a Gaussian distribution with an accuracy scale factor u, where its latent variable follows a gamma distribution.Adopting a degree of freedom ν = 1, the value of u ik is given by:EQUATIONIn the M-step, we update the parameters Φ = {ω k , µ k , and Σ k } using the derivatives obtained in the previous steps. In our model, the likelihood function for a Student's-t Distribution Mixture Model (SMM) is represented as:EQUATION)where ω k are the mixture weights, Σ k the covariance matrices, µ k the means, and z i the data points.The derivative with respect to ω k must consider the constraint that the sum of the mixture weights equals 1, i.e., k ω k = 1. Hence, we introduce a Lagrange multiplier λ to address this constraint and construct the Lagrangian L ′ :EQUATIONThe derivative with respect to ω k is:EQUATIONSubstituting the definition of L(ω, µ, Σ), we obtain:EQUATIONTo solve for ω k , we first multiply both sides of the equation by ω k and apply the constraint condition:EQUATIONUpon further organization, we find that the Lagrange multiplier λ actually equals the total number of data points N (since i τ ik = N k , where N k is the expected total number of data points belonging to the kth component, and the sum of all N k equals the total number of data points N ).Finally, we can solve for ω k :EQUATIONThis result indicates that the weight ω k of each mixture component equals the proportion of the posterior probabilities of the data points it contains relative to all data points.To update µ k and Σ k , we consider the conditional expectation of the data log-likelihood function: Maximizing Q(µ k , Σ k ) with respect to µ k leads to:EQUATIONEQUATIONSetting ∂Q ∂µ k = 0 results in the updated mean µEQUATION)Considering the derivative ofEQUATIONSetting ∂Q ∂µ k = 0 yields the updated covariance matrix Σ:EQUATION)C Anomaly Score with Vector SumHere we discuss the advantages of employing vector sum in anomaly score with a toy example.The application of the vector sum principle extends beyond physical mechanics and finds relevance in various domains. In relational embedding [5] , for example, relationships can be represented as vectors. Aggregating these vectors allows for capturing complexities like transitivity, symmetry, and antisymmetry.Similarly, in our context, the vector sum can help capture more complex relationships along clusters. Consider Figure 4 as an example, where a sample v is attracted by two groups of cluster prototypes ({µ 1 , µ 2 }, {µ 3 , µ 4 }) with the same mass and sample-prototype distances (m 1 = m 2 = m 3 = m 4 , r v1 = r v2 = r v3 = r v4 ).Without considering the direction of the forces, the two groups of prototypes would attract the sample with equal forces. However, we argue that the two groups of prototypes should exert different influences. A sample close to two clusters with a large difference ({µ 1 , µ 2 }) is more likely to be an anomaly compared to a sample that is close to two clusters with a smaller difference ({µ 3 , µ 4 }). For example, in a social network, a user who equally likes two extremely different communities, like money-saving tips and luxury items, is more anomalous than a user who equally likes two similar communities, like private jets and luxury items. Applying the vector sum, the total force of {µ 1 , µ 2 } is much smaller than that of {µ 3 , µ 4 }. As the anomaly score is inversely related to the total force, it is more anomalous when equally attracted by {µ 1 , µ 2 } with large difference. This indicates that the vector sum successfully captures subtle differences in the distinctions among multiple clusters, thereby assisting in the identification of more accurate anomalies.In the appendix, as illustrated in Figure 3 , we investigated a toy example. We discussed a specific pattern of anomalies termed group anomalies, where a small number of anomalous samples cluster together. It is crucial to note that we do not claim this anomaly pattern is common in real-world data; our goal is merely to point out a specific anomaly pattern that is challenging for traditional clusterbased anomaly detection methods to detect. Specifically, we utilize three Gaussian distributions with high variance (each generating 300 data samples) and one with lower variance (generating 30 data samples). Because the samples from the smaller Gaussian follow a different generative mechanism and represent a minority in the dataset, we consider them anomalies.We set the cluster number for KMeans--and GMM at four, indicating that the Gaussian distribution comprising anomalous samples was also recognized as a cluster. KMeans--employs a cluster-based approach, using the distance to the nearest cluster center as the anomaly score, while GMM uses a probability-based approach, considering the samples' likelihood in the mixture model as the anomaly score. However, both approaches are ineffective in this scenario. Rather than identifying the small cluster as anomalous, they tend to misidentify samples on the peripheries of larger clusters as anomalies.By contrast, our scoring method views the entire small cluster as more likely anomalous, followed by outlier samples on the margins of the larger clusters. This visualization provides a perspective that distinguishes our method from previous efforts.Due to space constraints in the main text, we utilized 30 public datasets from ADBench [17] , covering all different types of data. The details of the 30 datasets are presented in Table 4 .A comprehensive overview of the unsupervised anomaly detection methods is presented below.• Subspace Outlier Detection (SOD) [24] : Identifies outliers in varying subspaces of a highdimensional feature space, targeting anomalies that emerge in lower-dimensional projections.• Histogram-based Outlier Detection (HBOS) [16] : Assumes feature independence and calculates outlyingness via histograms, offering scalability and efficiency.• Principal Component Analysis (PCA) [49] : Utilizes singular value decomposition for dimensionality reduction, with anomalies indicated by reconstruction errors.• One-class SVM (OCSVM) [32] : Defines a decision boundary to separate normal samples from outliers, maximizing the margin from the data origin.• Local Outlier Factor (LOF) [6] : Measures local density deviation, marking samples as outliers if they lie in less dense regions compared to their neighbors.• K-Nearest Neighbors (KNN) [38] : Anomaly scores are assigned based on the distance to the k-th nearest neighbor, embodying a simple yet effective approach.• Lightweight On-line Detector of Anomalies (LODA) [39] : An ensemble method suitable for real-time processing and adaptable to concept drift through random projections and histograms.• Isolation Forest (IForest) [29] : Isolates anomalies by randomly selecting features and split values, leveraging the ease of isolating anomalies to identify them efficiently. • Deep Support Vector Data Description (DeepSVDD) [42] : Minimizes the volume of a hypersphere enclosing network data representations, isolating anomalies outside this sphere.• Deep Isolation Forest for Anomaly Detection (DIF) [51] : Utilizes deep learning to enhance traditional isolation forest techniques, offering improved anomaly detection in complex datasets with minimal parameter tuning.Each method's unique mechanism and application context provide a rich landscape of techniques for unsupervised anomaly detection, illustrating the field's diverse methodologies and the breadth of approaches to tackling anomaly detection challenges.In the appendix, we detail the statistical analysis conducted to compare the performance of various anomaly detectors. We obtained this diagram by conducting a Friedman test (p-value: 4.657e-19), indicating significant differences among different detectors. We utilized average ranks and the Nemenyi test to generate the critical difference diagram, as shown in Figure 5 . It is noteworthy that the vector version exhibits significantly superior performance compared to the scalar version across more methods. The detailed outcomes for the AUCROC and AUCPR metrics, spanning 30 datasets and against 17 baseline approaches, are showcased in Table 5 and Table 6 .The complexity of each iteration in UniCAD involves three parts: constructing the outlier set, updating the network parameters Θ, and optimizing the mixture model using the EM algorithm. Our proposed method was compared with 16 graph domain baseline methods grouped into three categories as follows:• Contrastive Learning-based Methods: This group includes CoLA [30] , SLGAD [55] ,CONAD [53] , and ANEMONE [20] . These methods primarily assume that the contrastive loss between anomalous nodes and their neighborhoods is more significant.• Autoencoder-based Methods: This category consists of MLPAE [43] , GCNAE [22] , DOMI-NANT [11] , GUIDE [54] , ComGA [31] , AnomalyDAE [14] , ALARM [37] , DONE/AdONE [4] and AAGNN [57] . These methods focus on the reconstruction errors of anomalous nodes during the process of reconstructing the graph structure or features.• Clustering-based Methods: This category of methods encompasses SCAN [52] , CBLOF [18] ,and DCFOD [45] . These methods generally identify anomalies by detecting if a sample deviates from the clustering.We assess the performance of our model using four graph benchmark datasets containing organic anomalies. Table 7 presents the statistical summary for each dataset. These datasets contain naturally occurring real-world anomalies and are valuable for assessing the performance of anomaly detection algorithms in real-world scenarios. The sources and compositions of these datasets are as follows: • Weibo [19] is a labeled graph comprising user posts extracted from the social media platform Tencent Weibo. The user-user graph establishes connections between users who exhibit similar topic labels. A user is considered anomalous if they have engaged in a minimum of five suspicious events, whereas normal nodes represent users who have not.• Reddit [25] consists of a user-subreddit graph extracted from the popular social media platform Reddit. This publicly accessible dataset encompasses user posts within various subreddits over a month. Each user is assigned a binary label indicating whether they have been banned on the platform. Our assumption is that banned users exhibit anomalous behavior compared to regular Reddit users.• Disney [34] is a co-purchase network of movies that includes attributes such as price, rating, and the number of reviews. The ground truth labels, indicating whether a movie is considered anomalous or not, were assigned by high school students through majority voting.• T-Finance [46] aims to identify anomalous accounts within a trading network. The nodes in this network represent unique anonymous accounts, each characterized by ten features related to registration duration, recorded activity, and interaction frequency. Graph edges denote transaction records between accounts. If a node is associated with activities such as fraud, money laundering, or online gambling, human experts will designate it as an anomaly. In this experiment, we compared graph-based methods on relational data. For methods originally designed around feature vectors, including CBLOF, DCFOD, and our approach, we uniformly employed the same graph representation learning technique as described in BGRL [47] . Specifically, we used a two-layer Graph Convolutional Network (GCN) for encoding, which produced output embeddings with a dimensionality of 128. The training epochs were set to 3000, including a warm-up period of 300 epochs. The hidden size of the predictor was set to 512, and the momentum was fixed at 0.99.Justification: The limitations of the method's application scope are discussed in Section 5 of the paper, along with considerations for future work.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute. 


Decision: This paper presents an unsupervised anomaly detection framework that consists of two components: (i) a framework that jointly learns latent feature representation, data clustering structure and a generative mixture model; (ii) a gravity-inspired anomaly scoring function based on the mixture model output. Experimental results on 30 data sets show this framework consistently achieves the top three performance when compared with 10 baseline methods. 

This paper has a plausible initiative and all reviewers commend the extensive and good experimental results. All reviewers found the paper more or less acceptable. However, after screening through the paper and all reviewer-author discussions, I have a strong sentiment of holding back on an "acceptance" recommendation. My main concern is the limited novelty and depth. Below are detailed comments. 

-- A reviewer pointed out the limited technical novelty of component (i). Authors did not directly defend and instead emphasized that component (ii) is novel. I agree with the reviewer. Component (i) is basically a standard mixture model based on student-t distribution and latent feature plus a standard alternate optimization process between mixture model learning (using EM) and latent feature learning (using stochastic gradient descent). Based on these, I judge that (i) carries very limited technical novelty. 

-- If component (i) alone can boost detection performance, then it can at least be defended as being "useful". However, this is not clear from the paper since the ablation study results in Table 3 do not include performance without component (ii). Interestingly, authors had a perfect opportunity to address this point when a reviewer questioned the impact of (ii) e.g., they could have provided new detection results based on (i) and without (ii). However, authors did not directly respond and instead pointed to Table 1 which compares two variants of (ii) based on (i) -- as they provided new results for many other questions but chose to dodge this one, I have reason to suspect that (i) is not very useful and the superior performance of the proposed framework mainly comes from (ii). This further weakens the contribution of (i). 
 
-- I  noticed components (i) and (ii) run independently, since (i) is about learning a mixture model and (ii) is about scoring anomalies based on (any) mixture model output. Therefore, conceptually, the paper does not really "integrate" latent feature/mixture model learning with anomaly detection, but merely "combine" them -- if the scoring function in (ii) had not been hand-crafted as in the paper but jointly learned with other components in (i), that would be an "integration" and carry more merit. I think "integration" is an oversold concept in this paper.

-- If my above judgements are correct, then the whole merit of this paper is carried by the gravity-inspired anomaly scoring function in Section 3.2. While it does seem interesting and novel, I think it alone is not sufficient for publishing at a top-notch venue like NeurIPS. 

Overall, I judge the work has a plausible initiative but falls short on technical novelty and depth. I encourage authors to deepen the work (e.g., get a real "integration" of all components) and resubmit for another round of review.

Reviews: review: paper_summary: paper_summary: The authors propose UniCAD to jointly model representation learning,
clustering and anomaly detection.  The main objective is maximizing
the product of anomaly indicator (1 is normal, 0 is anomaly) and the
joint probability of instance x_i and cluster c_k given parameters for
representation learning theta and clustering phi.  The joint
probability is decomposed into the prior of c_k and likelihood of
p(x_i|c_k), which is modeled by a Student's-t distribution on the
distance between representation z_i and mean mu_k with covariance
Sigma_k.  p(x_i) is the marginal over c_k.  Anomaly indicator delta is
zero for p(x_i) in the lowest l percentage.  The anomaly score is
1/p(x_i).

Compared to Newton's law of Universal Gravitation, the anomaly score
function has similar components, except for the unit vector r_ik
(which indicates the directions of forces, beyond the
magnitudes). Hence, they incorporated r_ik into their anomaly score
function.

For updating the clustering parameter phi (mixture weights, means,
covariance), they use EM.  In the E-step, they estimate the posterior
p(c_k|x_i).  In the M-step, they estimate phi.  For updating
representation parameters theta, they use gradient descent to minimize
negative log likelihood of instances, together with a reconstruction
loss via an autoencoder to prevent shortcut solutions.

For empirical comparisons, they use 30 tabular data sets and 17
existing algorithms.  The proposed approach generally outperforms the
others in terms of average rank in AUCROC.  The vector version of
anomaly score function is ranked higher than the scalar version.  On
computation time, UniCAD is in the middle among 5 algorithms.  Ablation
studies indicate the contributions of the different components.
strength_weakness: strength_weakness: The clustering part is similar to a typical Gaussian mixture model for clustering via EM, except for t-distribution instead of Gaussian and the scaling factor.

Two neural-network-based approaches were compared.  As UniCAD utilizes
representation learning, comparing with more approaches that utilize
representation learning would be significant.  Approaches without
representation learning have an inherent disadvantage.

Some parts could be clarified--see Questions.
questions: questions: Eq 11: What is the motivation for the scale factor u_ik, used in Eq 13
and 14?

How is K, number of clusters, determined?

Since the method has representation learning, how does the method
prevent the trivial solution of having most/all instances in one
cluster?

Sec 3.2.1: Consider (a simple case of) only two clusters with the same 
covariance and "mass", but different centroids.  If an instance is in the middle
between the two clusters, the two r_ik vectors will be in opposite directions,
resulting in an anomaly score of zero.  In gravitational forces, the
two opposite forces cancel out.  However, in anomaly detection, that
might not be desirable, particularly, when the two clusters are far
away.  Any insights?

What is the "schedule" for updating phi and theta?  Updating one to
convergence before updating the other to convergence?

If one "round" is updating phi to convergence and then theta to
convergence?  Are there multiple rounds?  If so, what is the overall
stopping criterion?
limitations: limitations: Limitations of the proposed approach do not seem to be mentioned.
paper_summary: paper_summary: This paper introduces a novel probabilistic mixture model for unsupervised anomaly detection (UAD) that unifies representation learning, clustering, and anomaly detection into a single theoretical framework. The proposed UniCAD model addresses the lack of a unified approach in existing methods, which often consider these components separately or in pairs. The experimental results show that UniCAD consistently outperformed other methods in terms of AUC-ROC and AUC-PR. The model’s iterative optimization process using EM was also highlighted as effective and convergent.
strength_weakness: strength_weakness: - The connection between force analysis and anomaly detection, particularly between Equations 7 and 8 in Section 3.2.1, could benefit from further justification. While the analogy is interesting, it may not be immediately intuitive for all readers.
- The iterative optimization process may pose scalability issues for large datasets. An in-depth analysis and discussion of this would further strengthen the quality of this research.
- Although the model maps data to a low-dimensional representation space, the effectiveness of this mapping for very high-dimensional datasets could be explored further.
questions: questions: I have no specific questions.
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: This paper proposes UniCAD, a theoretically unified framework for representation learning, clustering, and anomaly detection. This paper first introduces the mixture of Student-t distribution $p(x|\Theta, \Phi)$ with degree of freedom $\nu=1$ based on a representation learner $f_\Theta$ using NN. Then, this paper combines with an anomaly indicator $\delta$ for maximum likelihood estimation. Parameters $(\Theta, \Phi)$ are optimized by EM algorithm and SGD. In addition, when detecting anomalies, an improved score is used with reference to gravity. The UniCAD achieved good performance on experiments with various datasets.
strength_weakness: strength_weakness: - We have several questions about the proposed method and experiments. Please see Qustions.
- The comparison with DeepSVDD and DIF is excellent, but I think the paper also needs to be compared with other Deep anomaly detection methods. For example, DROCC [1].

[1] Goyal, Sachin, et al. "DROCC: Deep robust one-class classification. "International conference on machine learning. PMLR, 2020.
questions: questions: - if the degrees of freedom is fixed to 1, the benefit of the t-distribution disappears. Why not learn the degrees of freedom as well?
- I think $\tilde{F}_{ik}$ is a scalar and the norm of $\tiled{r}_{ik}$ is 1, so I don't see the difference between Eq. (9) and Eq. (8).
- As shown in Figure 2, I think the proposed method strongly depends on the hyperparameters $l$ and $k$. Is there any criteria for setting these values?
- The method in this paper seems incremental. (Especially until 3.1.) I think the main novelty is the gravity-based anomaly score described in 3.2. To what extent does this anomaly score improve performance compared to the regular anomaly score? Also, can you give a theoretical explanation?


If the above concerns are remedied and a comparison is made between SOTA's Deep Anomaly Detection methodology, I intend to raise the score.
limitations: limitations: - Hyper-parameter sensitivity seems to be one limitation.
paper_summary: paper_summary: The paper addresses unsupervised anomaly detection by proposing a method named UniCAD. The authors aim to enhance anomaly detection performance by establishing a theoretical connection between representation learning, clustering, and anomaly detection. They introduce a unified framework that jointly optimizes these three components, using a probabilistic mixture model and a Student's-t distribution for robust representation learning and clustering. The framework also includes an anomaly-aware data likelihood objective, which reduces the impact of anomalous data on the learning process. Additionally, the authors propose a gravity-inspired anomaly scoring method that leverages relationships between samples and clusters.
strength_weakness: strength_weakness: 1. The ablation study on the hyperparameters $k$ and $l$ is insufficient. The authors only present results from a single dataset, satimage-2, where their method achieves an almost perfect score. It would be more informative to perform ablation studies across all 30 datasets or at least a subset where the model also shows lower performance. This broader analysis would demonstrate how these hyperparameters affect the average ranking of the method, similar to the results reported in the paper's table.

2. The authors introduce a $g(\Theta)$ term to prevent shortcut solutions, mentioning it in Equation 15. However, they do not discuss its importance or impact on performance after its introduction. Key questions remain unanswered, such as how the $g(\Theta)$ term affects the model's performance, what happens if it is removed, and how the autoencoder is implemented. These details are crucial, as the regularization term may significantly influence the results.
questions: questions: See weaknesses.
limitations: limitations: The authors mention some of the limitations, but they do not address the potential negative impact of the work.

Comments: title: Response from Reviewer BFMu
Thanks for the rebuttal.
comment: > I think that is for fixed representation (ie, no representation learning). Consider the simple case of 2 clusters with centroids mu_1 and mu_2. With representation learning, I think Eq 2 (and hence Eq 1) can be maximized if all instances are at/near mu_1. That is, one cluster with all instances having roughly the same representation.

Thank you for your response. We also noted this important point, **as indicated in section 3.3.2, L220-L222**. When updating the deep network parameters $\Theta$, solely optimizing J(Θ, Φ) can lead to shortcut solutions, causing the loss to quickly become infinite. Therefore, during representation learning, we introduced the Autoencoder reconstruction loss to prevent all samples from being mapped to the same representation.


> When an instance is in between two centroids, the resultant force is close to zero and the anomaly score is going toward infinity. Numerically, in order for an instance that is far away to get a similar anomaly score, the distance needs to go toward infinity. Hence, I think the resultant force seems to give an instance in between two centroids a much higher anomaly score than I would desire.

Regarding your point about the anomaly score when an instance is between two centroids, we are open to further discussing this scenario. When an instance is located between two centroids and the resultant force is close to zero, its anomaly score indeed becomes higher compared to other types of anomalies. **This situation is also applicable in real-world scenarios.** For example, in identifying **bots (anomalous users)** in social media platforms, their behavior might resemble multiple user groups, making it difficult to classify them into any single cluster, and thus **placing them at the intersections of multiple clusters**. Our model would mark them as potential anomalies. Conversely, users from **niche groups** with unique behaviors might be far from the cluster centers, and our model would **assign them a lower score than the bots**.

Considering that in an unsupervised setting, there is no one-size-fits-all anomaly score that can cater to all types of anomalies, it is essential to choose an appropriate metric based on the specific dataset and scenario.
> The main reason is that the whole dataset consists of several clusters, thus the assumption of mixture model can better fit the dataset distribution than single model.

I think that is for fixed representation (ie, no representation learning).  Consider the simple case of 2 clusters with centroids mu_1 and mu_2.   With representation learning, I think Eq 2 (and hence Eq 1) can be maximized if all instances are at/near mu_1.  That is, one cluster with all instances having roughly the same representation.

> an instance between two centroids and an instance far away from the centroids—would be assigned a high anomaly score.

When an instance is in between two centroids, the resultant force is close to zero and the anomaly score is going toward infinity.  Numerically, in order for an instance that is far away to get a similar anomaly score, the distance needs to go toward infinity.  Hence, I think the resultant force seems to give an instance in between two centroids a much higher anomaly score than I would desire.
I appreciate the effort the authors put into preparing the rebuttal. I have no further comments and will be keeping my rating as it is.
My concerns have been addressed to some extent, especially with the additional experiments involving DROCC. While the method itself appears to be incremental, its performance on tabular data is excellent. I will raise my score.
> To include more comparisons with methods with representation learning, I suggest reducing the ones without representation learning.

Thank you for your suggestions. We will incorporate them into the revised version.

> Where is that?

We apologize for not clearly stating the location, these results are placed in **Appendix E**.

> What is the main reasoning?

Thanks for your further question. The main reason is that the whole dataset consists of several clusters, thus the assumption of mixture model can better fit the dataset distribution than single model. This is also supported by the research such as GMM\[1\]. According to the Approximation Theorem, a mixture model **can approximate any continuous probability density function by increasing the number of distributions**. If all samples were assigned to a single cluster, there **would still be room for improvement in the maximum likelihood solution**.

As a result, maximizing the likelihood of mixture model can help improve the clustering performance while avoiding the trivial solution that assign all samples into single cluster.

> In my simple example, why is an instance between two centroids the most anomalous situation. Wouldn't an instance very far away from the two centroids be more anomalous?

We apologize for the misunderstanding caused by the use of "most" in the previous response. In our newly proposed anomaly score, **both situations you described—an instance between two centroids and an instance far away from the centroids—would be assigned a high anomaly score**.

In the example you provided, due to the opposing effects of the two centroids, the resultant force $\vec{{\mathbf{F}}}\_{i}$ is smaller, resulting in a higher anomaly score ${o}\_i^V$, meaning that samples with ambiguous category semantics are more likely to be identified as anomalies. Conversely, the probability-based anomaly score fails to detect this type of anomaly, making it less adaptable to a broader range of data.

**References:**

\[1\] Reynolds, Douglas A. "Gaussian mixture models." Encyclopedia of biometrics 741.659-663 (2009).
Thank you very much for reading our rebuttal!
We are very pleased that your concerns have been addressed. We will update the content mentioned in our rebuttal in the camera-ready version. 
Thank you again for your valuable suggestions!
I would like to thank the authors for addressing my concerns. I have read the author's responses and all other reviews, and I am keeping my rating the same.
Thanks for your response

> We have included an additional NN-based comparison method (DROCC) in our experiments. As shown in our general response, our model significantly outperforms it.

To include more comparisons with methods with representation learning, I suggest reducing the ones without representation learning.

> Furthermore, on graph data, we compared our method with 13 deep anomaly detection approaches based on representation learning.

Where is that?

> Optimizing the maximum likelihood objective leads to better results with a mixture model compared to a single cluster.

What is the main reasoning?

>  we consider to be the most anomalous situation. Since our anomaly score is inversely proportional to the resultant force.

In my simple example, why is an instance between two centrods the most anomalous situation.   Wouldn't an instance very far away from the two centroids be more anomalous?