Anomaly Detection for Tabular Data with Internal Contrastive Learning

1 INTRODUCTION . In the one-class classification problem , one learns a model with the goal of identifying whether a test sample belongs to the same distribution from which the training set is sampled ( Schölkopf et al. , 1999 ) . Methods for solving this problem , therefore , define a criterion that is satisfied for the samples of the training set , while being less likely to hold for samples from other ( unseen ) distributions . When considering perceptual data , one can rely on the structure of the input . For example , images can be rotated , and the discrimination between the various rotations is class-dependent and , therefore , indicative of the class ( Golan & El-Yaniv , 2018 ) . In this work , we consider tabular data , in which there is no prior information on the structure of the data . If one assumes that no such structure exists , i.e. , that the variables are independent , then the criterion is defined by combining the per-feature scores . This , however , is not competitive in cases in which the features in each sample vector are not independent ( Ahirwar et al. , 2012 ) . By making the assumption that the dependency structures are class-dependent , one can construct , for example , a low-dimensional subspace and expect out-of-distribution classes to lie outside it ( Jolliffe , 1986 ) . In this work , we make the assumption that the way in which a subset of the variables in the feature vector is related to the rest of the variables is class-dependent . Our method considers subsets of consecutive variables . In this manner , for a given input sample xi ∈ Rd , we obtain a set of pairs { ( aji , b j i ) } mj=1 , where a j i is a vector of k consecutive features from xi , b j i is the vector of all other feature values , and m = d− k + 1 . All vectors aji have the same length and vary according to the first coordinate of xi , from which the subsets are collected . Given a training set , we learn a neural mapping F : Rd−k → Ru for the vectors of type b and a mapping G : Rk → Ru for vectors of type a such that the mutual information in the latent space of dimensionality u between matching elements ( aji , b j i ) is maximized . The same networks F , G are learned for all samples i of the training set and for all starting indices j . The maximization of the mutual information is done through contrastive learning ( Oord et al. , 2018 ) . At test time , the anomaly score that is being used y ( x ) is exactly the loss that is being minimized during training . This puts our method on solid ground , in contrast to many other outlier detection methods , in which networks are trained for one goal , and anomaly identification assumes that some regularity is learned . Additionally , since the method employs a sliding window approach , it is straightforward to identify the specific features in the input vector that contribute to a high anomaly score . This provides our method with a natural and direct interpretability . Assumptions and design choices Our method is generic and assumes little on the structure of the data . We make the following set of assumptions and associated design choices . First , we assume that one can learn networks F , G that identify matching elements ( aji , b j i ) . Second , we assume that when training F , G on data from a single class , the learned networks are class-specific in the sense that the recognition rate for matching elements that belong to samples from other classes would be considerably lower . Both of these assumptions are validated in Appendix A . Note that the contrastive learning task that F and G are trained to solve can be easily solved with a very short deterministic program that is class-independent , which checks the overlap between bji and aji . However , the success of our method implies that by training our neural method , one learns class-specific models . Furthermore , simulations on synthetic data demonstrate that learning the class-independent solution requires a lengthy training process and does not converge to a perfect classifier . These results , which shed light on self-supervised training , are given in Sec . 5 . Our assumptions are further validated by an extensive set of experiments , in which our method , using a single architecture and the same setrule of default hyperparameters selection , obtains state-of-the-art results in one-class classification of tabular data . The increase in accuracy compared to existing methods is sizable and consistent across benchmarks . It is further shown that the method is insensitive to its hyperparameters , demonstrating that the validity of the assumption is a stable phenomenon . Even with k = 1 ( masking of a single feature ) , the method outperforms all baselines by a gap . Two softer assumptions are introduced by performing specific design choices ( first soft assumption ) or implementing the method in a specific way ( second ) . ( i ) For simplicity , F , G can learn to match aji with bji regardless of j . In an alternative implementation , one can learn a pair of networks F j , Gj for every j or employ positional encoding for j . Since our method is effective with a single pair of networks , this option was not explored . ( ii ) The original order in which the features are given is used , based on the intuition that in several datasets , the order is not random . Since our method , by default , considers consecutive vector elements , nearby features are stacked together . Naturally , the strength of this effect is dataset-dependent . We quantify it in Sec . 4 , and show , by gathering statistics over a large corpus of varied datasets , that the original order does not provide a clear advantage over a random order of features ( consecutive features after applying permutation results in random , non-overlapping , subsets of features ) . This is also evident from the results presented in Appendix A . 2 RELATED WORK . The main application of one-class classification methods is anomaly detection ; that is , identifying outliers after observing a set of mostly normal ( the opposite of abnormal ) samples ( Chandola et al. , 2009 ; Pang et al. , 2020 ; Ruff et al. , 2021 ) . A straightforward way to perform this task is to model a distribution based on the training samples and then estimate the likelihood of each test sample . For this purpose , one can employ non-parametric methods , such as kernel density estimation ( Parzen , 1962 ) , or the very recent COPOD method ( Li et al. , 2020 ) that is based on an empirical copula model . Parametric methods include Gaussian and Gaussian mixture models ( Zong et al. , 2018 ) , as well as adversarial learning ( Schlegl et al. , 2017 ) . An alternative to density estimation approaches relies on regularized classifiers . The classical methods were mostly kernel-based methods , in which the role of the regularization term is to ensure that the fitted model is tight around the observed samples ( Schölkopf et al. , 1999 ) . Many of the first deep learning anomaly detection methods employed such classical one-class methods on top of auto-encoder based representations ( Hawkins et al. , 2002 ; Sakurada & Yairi , 2014 ; Xia et al. , 2015 ; Xu et al. , 2015b ; Erfani et al. , 2016 ; An & Cho , 2015 ) . More recent methods apply a suitable one-class loss , in order to learn a neural network-based representation in an end-to-end manner ( Ruff et al. , 2018 ) . In order to further avoid the problem of representation collapse , the DROCC method ( Goyal et al. , 2020 ) applies virtual adversarial training ( Miyato et al. , 2018 ) to create virtual negative samples around the training samples . Self-supervised learning , in which an unsupervised learning problem is turned into a discriminative learning problem , was introduced to anomaly detection by Golan & El-Yaniv ( 2018 ) . Their method predicts the image transformation that is applied to an image . Assuming that this classification problem is class-dependent , the membership score is based on the success of the learned classifier on a given test image . GOAD ( Bergman & Hoshen , 2020 ) improved this method , by learning an embedding space , in which the classifier considers the distances to the center of the set of training samples after applying each transformation . The method is also made suitable for tabular data , in which case random linear projections replace the geometric transformations . Our work is based on a different self-supervised task called masking , in which part of the data is held out and is predicted by the rest of the data . This form of self supervision is commonly used for learning representation in NLP ( Mikolov et al. , 2013 ) and in computer vision ( Pathak et al. , 2016 ) . As far as we can ascertain , masking was not used for one-class classification before . The idea of contrastive learning has emerged in metric learning , where it was used to train a Siamese network ( Chopra et al. , 2005 ) . However , its main application is in unsupervised representation learning ( Hadsell et al. , 2006 ) . The learned embedding brings associated samples closer together , while pushing away other samples . The framework of noise contrastive estimation ( Gutmann & Hyvärinen , 2010 ) casts this type of learning as a form of mutual information maximization . Many of the most recent contrastive learning methods perform unsupervised learning by anchoring an image together with its transformed version , while distancing other images ( He et al. , 2020 ; Misra & van der Maaten , 2019 ; Chen et al. , 2020 ) . Recently , the contrastive learning method of Chen et al . ( Chen et al. , 2020 ) was adapted for the problem of one-class classification ( Tack et al. , 2020 ) . The obtained score combines the norm of the representation ( Taigman et al. , 2015 ) with the the maximal similarity to any sample of the training set to define an anomaly score . The performance is further enhanced by contrasting two sets of image transformations : those that maintain the same-identity property vs. those that lead to a different training identity . Another method applies contrastive learning that is tailored to one-class learning in order to learn an image representation , followed by a one class classifier ( Sohn et al. , 2021 ) . These image transformation-based techniques are not applicable to tabular data , since there is no group of transformations that the content of generic vectors is invariant to . The contemporaneous work by Qiu et al . ( 2021 ) employs contrastive loss for identifying anomalies in tabular data . Similar to our work , one employs a per-sample contrastive loss . However , there are crucial differences between our work and the NeuTraL AD method : ( 1 ) NeuTraL AD learns specific masks , while we apply the entire set of the masks specified by a window size k. ( 2 ) The role of NeuTraL AD masks is to mask-out parts that are irrelevant for specific classes . In our case , we perform a two sided matching that identifies the masked part form the original . ( 3 ) NeuTraL AD learns a single feature extractor ( “ encoder ” ) for both the original and transformed data . In our case , the two sides of the contrastive loss are of very different dimensions ( d − k and k ) and we employ two different encoders . Furthermore , we explore in our ablation ( and theoretically motivate ) the importance of using encoders that are of different architectures . ( 4 ) Their method is built such that there is diversity between the different views and similarity between each view and the original . This requires the views to be spread , in the latent space , around the original sample . In our case , the constraints are such that the crop matches the complement elements of the vector more than all other crops . There is no requirement that the other crops are dissimilar between themselves .

Summary: - The authors propose a novel anomaly detection framework that can be useful for tabular data. - The authors utilize the contrastive learning to learn the relationship between features in the tabular data. Then, using the loss as the anomaly scores for evaluation. - The authors provide extensive experimental results which are promising.