Anomaly Detection for Tabular Data with Internal Contrastive Learning

1 INTRODUCTION . In the one-class classification problem , one learns a model with the goal of identifying whether a test sample belongs to the same distribution from which the training set is sampled ( Schölkopf et al. , 1999 ) . Methods for solving this problem , therefore , define a criterion that is satisfied for the samples of the training set , while being less likely to hold for samples from other ( unseen ) distributions . When considering perceptual data , one can rely on the structure of the input . For example , images can be rotated , and the discrimination between the various rotations is class-dependent and , therefore , indicative of the class ( Golan & El-Yaniv , 2018 ) . In this work , we consider tabular data , in which there is no prior information on the structure of the data . If one assumes that no such structure exists , i.e. , that the variables are independent , then the criterion is defined by combining the per-feature scores . This , however , is not competitive in cases in which the features in each sample vector are not independent ( Ahirwar et al. , 2012 ) . By making the assumption that the dependency structures are class-dependent , one can construct , for example , a low-dimensional subspace and expect out-of-distribution classes to lie outside it ( Jolliffe , 1986 ) . In this work , we make the assumption that the way in which a subset of the variables in the feature vector is related to the rest of the variables is class-dependent . Our method considers subsets of consecutive variables . In this manner , for a given input sample xi ∈ Rd , we obtain a set of pairs { ( aji , b j i ) } mj=1 , where a j i is a vector of k consecutive features from xi , b j i is the vector of all other feature values , and m = d− k + 1 . All vectors aji have the same length and vary according to the first coordinate of xi , from which the subsets are collected . Given a training set , we learn a neural mapping F : Rd−k → Ru for the vectors of type b and a mapping G : Rk → Ru for vectors of type a such that the mutual information in the latent space of dimensionality u between matching elements ( aji , b j i ) is maximized . The same networks F , G are learned for all samples i of the training set and for all starting indices j . The maximization of the mutual information is done through contrastive learning ( Oord et al. , 2018 ) . At test time , the anomaly score that is being used y ( x ) is exactly the loss that is being minimized during training . This puts our method on solid ground , in contrast to many other outlier detection methods , in which networks are trained for one goal , and anomaly identification assumes that some regularity is learned . Additionally , since the method employs a sliding window approach , it is straightforward to identify the specific features in the input vector that contribute to a high anomaly score . This provides our method with a natural and direct interpretability . Assumptions and design choices Our method is generic and assumes little on the structure of the data . We make the following set of assumptions and associated design choices . First , we assume that one can learn networks F , G that identify matching elements ( aji , b j i ) . Second , we assume that when training F , G on data from a single class , the learned networks are class-specific in the sense that the recognition rate for matching elements that belong to samples from other classes would be considerably lower . Both of these assumptions are validated in Appendix A . Note that the contrastive learning task that F and G are trained to solve can be easily solved with a very short deterministic program that is class-independent , which checks the overlap between bji and aji . However , the success of our method implies that by training our neural method , one learns class-specific models . Furthermore , simulations on synthetic data demonstrate that learning the class-independent solution requires a lengthy training process and does not converge to a perfect classifier . These results , which shed light on self-supervised training , are given in Sec . 5 . Our assumptions are further validated by an extensive set of experiments , in which our method , using a single architecture and the same setrule of default hyperparameters selection , obtains state-of-the-art results in one-class classification of tabular data . The increase in accuracy compared to existing methods is sizable and consistent across benchmarks . It is further shown that the method is insensitive to its hyperparameters , demonstrating that the validity of the assumption is a stable phenomenon . Even with k = 1 ( masking of a single feature ) , the method outperforms all baselines by a gap . Two softer assumptions are introduced by performing specific design choices ( first soft assumption ) or implementing the method in a specific way ( second ) . ( i ) For simplicity , F , G can learn to match aji with bji regardless of j . In an alternative implementation , one can learn a pair of networks F j , Gj for every j or employ positional encoding for j . Since our method is effective with a single pair of networks , this option was not explored . ( ii ) The original order in which the features are given is used , based on the intuition that in several datasets , the order is not random . Since our method , by default , considers consecutive vector elements , nearby features are stacked together . Naturally , the strength of this effect is dataset-dependent . We quantify it in Sec . 4 , and show , by gathering statistics over a large corpus of varied datasets , that the original order does not provide a clear advantage over a random order of features ( consecutive features after applying permutation results in random , non-overlapping , subsets of features ) . This is also evident from the results presented in Appendix A . 2 RELATED WORK . The main application of one-class classification methods is anomaly detection ; that is , identifying outliers after observing a set of mostly normal ( the opposite of abnormal ) samples ( Chandola et al. , 2009 ; Pang et al. , 2020 ; Ruff et al. , 2021 ) . A straightforward way to perform this task is to model a distribution based on the training samples and then estimate the likelihood of each test sample . For this purpose , one can employ non-parametric methods , such as kernel density estimation ( Parzen , 1962 ) , or the very recent COPOD method ( Li et al. , 2020 ) that is based on an empirical copula model . Parametric methods include Gaussian and Gaussian mixture models ( Zong et al. , 2018 ) , as well as adversarial learning ( Schlegl et al. , 2017 ) . An alternative to density estimation approaches relies on regularized classifiers . The classical methods were mostly kernel-based methods , in which the role of the regularization term is to ensure that the fitted model is tight around the observed samples ( Schölkopf et al. , 1999 ) . Many of the first deep learning anomaly detection methods employed such classical one-class methods on top of auto-encoder based representations ( Hawkins et al. , 2002 ; Sakurada & Yairi , 2014 ; Xia et al. , 2015 ; Xu et al. , 2015b ; Erfani et al. , 2016 ; An & Cho , 2015 ) . More recent methods apply a suitable one-class loss , in order to learn a neural network-based representation in an end-to-end manner ( Ruff et al. , 2018 ) . In order to further avoid the problem of representation collapse , the DROCC method ( Goyal et al. , 2020 ) applies virtual adversarial training ( Miyato et al. , 2018 ) to create virtual negative samples around the training samples . Self-supervised learning , in which an unsupervised learning problem is turned into a discriminative learning problem , was introduced to anomaly detection by Golan & El-Yaniv ( 2018 ) . Their method predicts the image transformation that is applied to an image . Assuming that this classification problem is class-dependent , the membership score is based on the success of the learned classifier on a given test image . GOAD ( Bergman & Hoshen , 2020 ) improved this method , by learning an embedding space , in which the classifier considers the distances to the center of the set of training samples after applying each transformation . The method is also made suitable for tabular data , in which case random linear projections replace the geometric transformations . Our work is based on a different self-supervised task called masking , in which part of the data is held out and is predicted by the rest of the data . This form of self supervision is commonly used for learning representation in NLP ( Mikolov et al. , 2013 ) and in computer vision ( Pathak et al. , 2016 ) . As far as we can ascertain , masking was not used for one-class classification before . The idea of contrastive learning has emerged in metric learning , where it was used to train a Siamese network ( Chopra et al. , 2005 ) . However , its main application is in unsupervised representation learning ( Hadsell et al. , 2006 ) . The learned embedding brings associated samples closer together , while pushing away other samples . The framework of noise contrastive estimation ( Gutmann & Hyvärinen , 2010 ) casts this type of learning as a form of mutual information maximization . Many of the most recent contrastive learning methods perform unsupervised learning by anchoring an image together with its transformed version , while distancing other images ( He et al. , 2020 ; Misra & van der Maaten , 2019 ; Chen et al. , 2020 ) . Recently , the contrastive learning method of Chen et al . ( Chen et al. , 2020 ) was adapted for the problem of one-class classification ( Tack et al. , 2020 ) . The obtained score combines the norm of the representation ( Taigman et al. , 2015 ) with the the maximal similarity to any sample of the training set to define an anomaly score . The performance is further enhanced by contrasting two sets of image transformations : those that maintain the same-identity property vs. those that lead to a different training identity . Another method applies contrastive learning that is tailored to one-class learning in order to learn an image representation , followed by a one class classifier ( Sohn et al. , 2021 ) . These image transformation-based techniques are not applicable to tabular data , since there is no group of transformations that the content of generic vectors is invariant to . The contemporaneous work by Qiu et al . ( 2021 ) employs contrastive loss for identifying anomalies in tabular data . Similar to our work , one employs a per-sample contrastive loss . However , there are crucial differences between our work and the NeuTraL AD method : ( 1 ) NeuTraL AD learns specific masks , while we apply the entire set of the masks specified by a window size k. ( 2 ) The role of NeuTraL AD masks is to mask-out parts that are irrelevant for specific classes . In our case , we perform a two sided matching that identifies the masked part form the original . ( 3 ) NeuTraL AD learns a single feature extractor ( “ encoder ” ) for both the original and transformed data . In our case , the two sides of the contrastive loss are of very different dimensions ( d − k and k ) and we employ two different encoders . Furthermore , we explore in our ablation ( and theoretically motivate ) the importance of using encoders that are of different architectures . ( 4 ) Their method is built such that there is diversity between the different views and similarity between each view and the original . This requires the views to be spread , in the latent space , around the original sample . In our case , the constraints are such that the crop matches the complement elements of the vector more than all other crops . There is no requirement that the other crops are dissimilar between themselves .

Summary: ## After rebuttal I am impressed by the comprehensive experiments and baselines and increase my score. Authors provide very complete baselines and also compare in other datasets, which greatly reduces the concern that the proposed rule overfits to the ODDS benchmark.   ## Summary This paper learns a contrastive representation that helps differentiate between normal and abnormal data in the tabular data. Contrary to other contrastive learning methods which learns to differentiate between different examples, this paper instead differentiates between in-window vector and out-window vector for each example under a sliding k-sized window (k is a hyperparameter). Since the features are unordered, they shuffle the feature orders to get multiple score and average them. They show that it outperforms other baselines in a large suite of tabular data, ODDS benchmark, with default hyperparameter rules (e.g. the k is set based on number of samples N and d). They compare mainly with recent methods DROCC, GOAD and COPOD.