Year: 2021

Abstract:
We consider the task of finding out-of-class samples in tabular data, where little can be assumed on the structure of the data. In order to capture the structure of the samples of the single training class, we learn mappings that maximize the mutual information between each sample and the part that is masked out. The mappings are learned by employing a contrastive loss, which considers only one sample at a time. Once learned, we can score a test sample by measuring whether the learned mappings lead to a small contrastive loss using the masked parts of this sample. Our experiments show that our method leads by a sizable accuracy gap in comparison to the literature and that the same default set rule of hyperparameters selection provides state-of-the-art results across benchmarks.

heading: 1 INTRODUCTION
text: In the one-class classification problem, one learns a model with the goal of identifying whether a test sample belongs to the same distribution from which the training set is sampled (Schölkopf et al., 1999). Methods for solving this problem, therefore, define a criterion that is satisfied for the samples of the training set, while being less likely to hold for samples from other (unseen) distributions.
When considering perceptual data, one can rely on the structure of the input. For example, images can be rotated, and the discrimination between the various rotations is class-dependent and, therefore, indicative of the class (Golan & El-Yaniv, 2018). In this work, we consider tabular data, in which there is no prior information on the structure of the data.
If one assumes that no such structure exists, i.e., that the variables are independent, then the criterion is defined by combining the per-feature scores. This, however, is not competitive in cases in which the features in each sample vector are not independent (Ahirwar et al., 2012). By making the assumption that the dependency structures are class-dependent, one can construct, for example, a low-dimensional subspace and expect out-of-distribution classes to lie outside it (Jolliffe, 1986).
In this work, we make the assumption that the way in which a subset of the variables in the feature vector is related to the rest of the variables is class-dependent. Our method considers subsets of consecutive variables. In this manner, for a given input sample xi ∈ Rd, we obtain a set of pairs {(aji , b j i )}mj=1, where a j i is a vector of k consecutive features from xi, b j i is the vector of all other feature values, and m = d− k + 1. All vectors aji have the same length and vary according to the first coordinate of xi, from which the subsets are collected.
Given a training set, we learn a neural mapping F : Rd−k → Ru for the vectors of type b and a mapping G : Rk → Ru for vectors of type a such that the mutual information in the latent space of dimensionality u between matching elements (aji , b j i ) is maximized. The same networks F,G are learned for all samples i of the training set and for all starting indices j. The maximization of the mutual information is done through contrastive learning (Oord et al., 2018).
At test time, the anomaly score that is being used y(x) is exactly the loss that is being minimized during training. This puts our method on solid ground, in contrast to many other outlier detection methods, in which networks are trained for one goal, and anomaly identification assumes that some regularity is learned. Additionally, since the method employs a sliding window approach, it is straightforward to identify the specific features in the input vector that contribute to a high anomaly score. This provides our method with a natural and direct interpretability.
Assumptions and design choices Our method is generic and assumes little on the structure of the data. We make the following set of assumptions and associated design choices. First, we assume that one can learn networks F,G that identify matching elements (aji , b j i ). Second, we assume that when training F,G on data from a single class, the learned networks are class-specific in the sense that the recognition rate for matching elements that belong to samples from other classes would be considerably lower. Both of these assumptions are validated in Appendix A.
Note that the contrastive learning task that F and G are trained to solve can be easily solved with a very short deterministic program that is class-independent, which checks the overlap between bji and aji . However, the success of our method implies that by training our neural method, one learns class-specific models. Furthermore, simulations on synthetic data demonstrate that learning the class-independent solution requires a lengthy training process and does not converge to a perfect classifier. These results, which shed light on self-supervised training, are given in Sec. 5.
Our assumptions are further validated by an extensive set of experiments, in which our method, using a single architecture and the same setrule of default hyperparameters selection, obtains state-of-the-art results in one-class classification of tabular data. The increase in accuracy compared to existing methods is sizable and consistent across benchmarks. It is further shown that the method is insensitive to its hyperparameters, demonstrating that the validity of the assumption is a stable phenomenon. Even with k = 1 (masking of a single feature), the method outperforms all baselines by a gap.
Two softer assumptions are introduced by performing specific design choices (first soft assumption) or implementing the method in a specific way (second). (i) For simplicity, F,G can learn to match aji with bji regardless of j. In an alternative implementation, one can learn a pair of networks F j , Gj for every j or employ positional encoding for j. Since our method is effective with a single pair of networks, this option was not explored. (ii) The original order in which the features are given is used, based on the intuition that in several datasets, the order is not random. Since our method, by default, considers consecutive vector elements, nearby features are stacked together. Naturally, the strength of this effect is dataset-dependent. We quantify it in Sec. 4, and show, by gathering statistics over a large corpus of varied datasets, that the original order does not provide a clear advantage over a random order of features (consecutive features after applying permutation results in random, non-overlapping, subsets of features). This is also evident from the results presented in Appendix A.

heading: 2 RELATED WORK
text: The main application of one-class classification methods is anomaly detection; that is, identifying outliers after observing a set of mostly normal (the opposite of abnormal) samples (Chandola et al., 2009; Pang et al., 2020; Ruff et al., 2021). A straightforward way to perform this task is to model a distribution based on the training samples and then estimate the likelihood of each test sample. For this purpose, one can employ non-parametric methods, such as kernel density estimation (Parzen, 1962), or the very recent COPOD method (Li et al., 2020) that is based on an empirical copula model. Parametric methods include Gaussian and Gaussian mixture models (Zong et al., 2018), as well as adversarial learning (Schlegl et al., 2017).
An alternative to density estimation approaches relies on regularized classifiers. The classical methods were mostly kernel-based methods, in which the role of the regularization term is to ensure that the fitted model is tight around the observed samples (Schölkopf et al., 1999). Many of the first deep learning anomaly detection methods employed such classical one-class methods on top of auto-encoder based representations (Hawkins et al., 2002; Sakurada & Yairi, 2014; Xia et al., 2015; Xu et al., 2015b; Erfani et al., 2016; An & Cho, 2015). More recent methods apply a suitable one-class loss, in order to learn a neural network-based representation in an end-to-end manner (Ruff et al., 2018). In order to further avoid the problem of representation collapse, the DROCC method (Goyal et al., 2020) applies virtual adversarial training (Miyato et al., 2018) to create virtual negative samples around the training samples.
Self-supervised learning, in which an unsupervised learning problem is turned into a discriminative learning problem, was introduced to anomaly detection by Golan & El-Yaniv (2018). Their method predicts the image transformation that is applied to an image. Assuming that this classification problem is class-dependent, the membership score is based on the success of the learned classifier on a given test image. GOAD (Bergman & Hoshen, 2020) improved this method, by learning an
embedding space, in which the classifier considers the distances to the center of the set of training samples after applying each transformation. The method is also made suitable for tabular data, in which case random linear projections replace the geometric transformations. Our work is based on a different self-supervised task called masking, in which part of the data is held out and is predicted by the rest of the data. This form of self supervision is commonly used for learning representation in NLP (Mikolov et al., 2013) and in computer vision (Pathak et al., 2016). As far as we can ascertain, masking was not used for one-class classification before.
The idea of contrastive learning has emerged in metric learning, where it was used to train a Siamese network (Chopra et al., 2005). However, its main application is in unsupervised representation learning (Hadsell et al., 2006). The learned embedding brings associated samples closer together, while pushing away other samples. The framework of noise contrastive estimation (Gutmann & Hyvärinen, 2010) casts this type of learning as a form of mutual information maximization. Many of the most recent contrastive learning methods perform unsupervised learning by anchoring an image together with its transformed version, while distancing other images (He et al., 2020; Misra & van der Maaten, 2019; Chen et al., 2020).
Recently, the contrastive learning method of Chen et al. (Chen et al., 2020) was adapted for the problem of one-class classification (Tack et al., 2020). The obtained score combines the norm of the representation (Taigman et al., 2015) with the the maximal similarity to any sample of the training set to define an anomaly score. The performance is further enhanced by contrasting two sets of image transformations: those that maintain the same-identity property vs. those that lead to a different training identity. Another method applies contrastive learning that is tailored to one-class learning in order to learn an image representation, followed by a one class classifier (Sohn et al., 2021). These image transformation-based techniques are not applicable to tabular data, since there is no group of transformations that the content of generic vectors is invariant to.
The contemporaneous work by Qiu et al. (2021) employs contrastive loss for identifying anomalies in tabular data. Similar to our work, one employs a per-sample contrastive loss. However, there are crucial differences between our work and the NeuTraL AD method: (1) NeuTraL AD learns specific masks, while we apply the entire set of the masks specified by a window size k. (2) The role of NeuTraL AD masks is to mask-out parts that are irrelevant for specific classes. In our case, we perform a two sided matching that identifies the masked part form the original. (3) NeuTraL AD learns a single feature extractor (“encoder”) for both the original and transformed data. In our case, the two sides of the contrastive loss are of very different dimensions (d − k and k) and we employ two different encoders. Furthermore, we explore in our ablation (and theoretically motivate) the importance of using encoders that are of different architectures. (4) Their method is built such that there is diversity between the different views and similarity between each view and the original. This requires the views to be spread, in the latent space, around the original sample. In our case, the constraints are such that the crop matches the complement elements of the vector more than all other crops. There is no requirement that the other crops are dissimilar between themselves.

heading: 3 METHOD
text: We are given a training set of n in-class samples S = {xi}, each a vector of d dimensions. Our goal is to design a score y : Rd → R that maps samples from the sample domain to a low value if they are sampled from the underlying distribution from which S is sampled and to a high one otherwise.
The method has two hyperparameters that specify dimensions: k < d determines the size of the subset of features we consider, and u determines their embedding size. A third hyperparameter τ is the temperature constant of the loss.
The method first constructs a set of m = d + 1 − k pairs Φ(xi) = {(aji , b j i )} from each training sample xi. Each pair in this set is obtained by extracting k consecutive variables from xi. Let a j i , 1 ≤ j ≤ m be the vector [xji , x j+1 i , . . . , x j+k−1 i ], where superscripts denote elements of the vector xi. We define b j i = [x 1 i , x 2 i , . . . , x j−1 i , x j+k i , . . . , x d i ] to be the vector of the other d− k elements in xi. The method then learns two mappings F,G that maximize the mutual information between F (b j i ) and G(aji ), where (a j i , b j i ) ∈ Φ(xi), i = 1 . . . n. The same mappings are learned for all samples and regardless of the index j ∈ [1,m].
The mutual information is maximized through the use of the noise contrastive estimation framework (Oord et al., 2018), see Fig. 1 for an illustration of the contrastive relations in our setting. In this framework, there is a query q, a positive sample v+, and negative ones v−, all vectors in Ru. Contrastive learning maximizes the similarity of the query with the positive sample, while minimizing it with the negative samples. In our case, the vectors are given, for some i, j, j′ 6= j as: q = F (bji ),v + = G(aji ),v − = G(aj ′ i ).
Almost all current contrastive learning methods employ normalization of the vectors q and v+,v− such that their unit norm is 1. In our method, this normalization is performed after a first normalization step that is applied at each vector dimension in Ru separately, by considering all of the sub-vectors of the input vector xi. The motivation for the first normalization step is that scale, by itself, can lead to the identification of a certain feature. This scaling of the features is often class-independent, since it frequently depends more on the nature of the feature than on the class, while the classification task is required to be class-dependent.
The normalized network FN considers the u × m matrix B = [F (b1i ), F (b2i ), . . . , F (bmi )] and normalizes each of its rows to have an L2 norm of 1 to obtain a matrix BN . We define the normalized network FN such that FN (bji ) is the j−th column BN , further normalized to an L2-norm of one. Although we omit it from the operand list, FN (bji ) depends not only on b j i but on all b-type vectors in Φ(xi) through normalization. Similarly, GN (a j i ) is defined by considering the matrix A
N , which is a double-normalized version of the matrix that contains all vectors of the form G(aji ), where j varies by column.
The contrastive loss ` is defined as an m-way classification problem, in which the cross-entropy loss for a given temperature τ is used. The logit used is the pseudo-probability for the positive sample v+ being selected over m− 1 negatives given the query q (Wu et al., 2018; He et al., 2020). Note that the normalized versions of these vectors are being used.
`(F,G,Φ(xi), j) = − ln exp(FN (bji ) ·GN (a j i )/τ)∑m
j′=1 exp(F N (bji ) ·GN (a
j′ i )/τ) . (1)
Once the networks F andG are trained with the contrastive loss, we define the one-class classification score as y(x) = ∑ j `(F,G,Φ(x), j) , where Φ(x) is constructed for sample x similarly to the construction of Φ(xi) for training sample xi.
TRAINING, PARAMETERS, AND ARCHITECTURE
During training, the method directly minimizes the anomaly score of the training set S, given by L = ∑ x∈S y(x). Training employs the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 10−3 . It stops when the loss is smaller than 10−3 for datasets with d < 40. For larger input dimensions, this criterion would lead to long training sessions, and we use 10−2 instead.
We fix τ = 0.01 and u = 200. The value of u, which is often much larger than k, provides enough capacity throughout all experiments, without the need to tune it for each problem. We set k proportionally to the input dimension d. For d smaller than 40, we set k = 2, for d in the range [40, 160] we employ k = 10, and for d > 160, k takes the value d − 150. However, as our experiments in Sec. 4 show, the method is not very sensitive to the value of k.
We found that we can make use of the fact that the features are unordered and simply combine multiple scores, each obtained on a different permutation of the features. In our experiments (Sec. 4), repeating this way is only very seldom detrimental to accuracy, and it improves performance for small d and very small n. On the other hand, it does add to the overall runtime.
In order to make use of this bagging effect, when needed, we set the number of repeats to be r = 1+b100(log(n)+d)−1c. For each repeat after the first, we randomly permute the set of features. The score that the method returns is the mean of the scores obtained from each such repeat.
F and G are two fully connected networks with LeakyRELU activations (Xu et al., 2015a) using a slope coefficient of 0.2 in all layers, except for the first layer of F , which has a tanh activation. The reason is that we wish to distance the embedding of the a-part of x and its b part, making a simple matching between the parts, which overlap for bji and a j′ i when j ′ 6= j, more challenging. See Sec. 5 for a detailed discussion. F has two hidden layers, with u and 2u hidden units, each followed by Batch Normalization (Ioffe & Szegedy, 2015). G is similar, only that due to the smaller input sizes, the hidden layers have u/4 and u/2 units and Batch Normalization is applied only after the first layer.

heading: 4 EXPERIMENTS
text: Datasets Based on the terminology of the field of anomaly detection, we use the term “normal” to describe the class observed during training, and abnormal to describe samples from the other class or classes. The experiments were conducted on two groups of datasets: (i) a collection of four datasets commonly used for reporting anomaly detection for tabular data, (ii) a much more comprehensive set of tabular datasets for benchmarking outlier detection. The first set of datasets contains two small-scale medical datasets (Arrhythmia and Thyroid), as well as two cyber-intrusion detection datasets (KDD and KDDRev) which are considerably larger. Following Zong et al. (2018) and others, the categorical attributes are presented to the network as one-hot vectors.
The second set employs the “Multi-dimensional point datasets” from the Outlier Detection DataSets (ODDS)1. It contains 31 datasets, including two of the four datasets above. The datasets Heart and Yeast were omitted, as it is not clear from the description which class is the normal one. See Appendix B for the dimensionality of each dataset.
Evaluation protocol and scores Following Zong et al. (2018); Bergman & Hoshen (2020), the training set contains a random subset of 50% of the normal data. The test set contains the rest of the normal data, as well as all the anomalies. In the first set of experiments, the mean and standard deviation (SD) of the F1 score are reported. For the experiments we run, this is computed over 500 random splits for the smaller datasets (Arrhythmia and Thyroid) and 10 splits for the larger ones (KDD and KDD-Rev). For the baseline methods, the sample size varies and in some cases the results reported in the literature are given without SD. Following the existing protocol, the decision threshold for scoring the methods is chosen such that the number of test samples above this threshold (i.e.,
1http://odds.cs.stonybrook.edu/, accessed January 2021
classified as anomalies) is the number of anomalies in the test set. For the second set of experiments, in which all runs were made by us, we report (sometimes in the appendix), in addition to F1, the AUC. Since the AUC varies less dramatically than F1 scores, it is more suitable for comparison across datasets. It also has the advantage of not requiring setting a threshold.
Baseline methods For the first set of experiments, a comprehensive set of literature baselines is presented. One-Class SVM (OC-SVM) (Schölkopf et al., 1999)) and DAGMM Zong et al. (2018) results are reported as computed by Zong et al. (2018). GOAD (Bergman & Hoshen, 2020), Local Outlier Factor (LOF) (Breunig et al., 2000), and an ensemble method (FB-AE) that employs autoencoders as the base classifier, feature bagging as the source of randomization, and reconstruction error as the anomaly score, are reported by Bergman & Hoshen (2020). A recent baseline, DROCC Goyal et al. (2020), is rerun by us based on their code, since the published results sampled the test set using a different protocol2. Finally, we include the COPOD baseline (Li et al., 2020), based on our runs, as a modern non-deep-learning method..
For the second set of experiments, we focus on the most recent deep methods: COPOD, GOAD and DROCC. Since GOAD uses three different architectures in their code, we report results for all three. The first architecture is the one used by Bergman & Hoshen (2020) for the small datasets, the second is used for KDD, and the third is the one used for KDDrev. Similarly, DROCC (Goyal et al., 2020) employs three architectures in their code: one for Thyroid, one for Arrhythmia, and one of Abalone, and we run all three. Our method employs the same architecture in all experiments, except that k is adjusted according to d, see Sec. 3. As non-deep baselines, we run multiple methods including classical methods and modern ones: IForest (Liu et al., 2008), k-nearest neighbours (KNN), PIDForest (Gopalan et al., 2019), RRCF (Guha et al., 2016), and COPOD (Li et al., 2020). All of these are based on our own runs, using the PyOD anomaly detection python package Zhao et al. (2019) for KNN and IForest, RRCF python package and the official code published by the authors of PIDForest in Github. The defualt parameters were used, except for the suprisignly strong baseline of KNN, for which we report results for the best parameters found on the test set, see Appendix C.
Results The results for the first set of experiments are reported in Tab. 1. Our method outperforms the literature baselines by a significant margin on Arrhythmia and Thyroid, where the baselines obtain a moderate F1 score. On the larger datasets - KDD and KDDRev, where the performance of GOAD is very high - we outperform it, obtaining a near-perfect score. On all datasets, the pvalue of the Wilcoxon test between our method and the best competing method is lower than 0.001. From the baseline methods, the feature bagging auto encoder (FB-AE) and GOAD seem to be the strongest, while DROCC, with the correct protocol, is not as competitive. COPOD, despite being shown to be successful on many other benchmarks, does not perform particularly well on this first set.
The results for the second set of experiments are obtained by our runs and compared with GOAD and DROCC, each with three different architectures, as well as with COPOD. The latter has been
2In their implementation, (1) the number of normals in the test set equals the number of anomalies in the data, and the rest are used for training and (2) the threshold in test time to determine F1 is constant across datasets at 20% of the test set.
shown to be highly effective on the ODDS collection, when tested with a random 60%/40% train/test split protocol (Li et al., 2020). The mean and SD for the F1 score across 20 runs are reported in Tab. 2, see Appendix C for the AUC results. As can be seen, in the vast majority of the cases (19), our method obtains the highest performance. COPOD has four datasets for which it leads over all other methods, and GOAD (when taking the max over all versions) leads on 5 datasets. DROCC is less competitive. For all cases in which our method achieves the highest score, which is 79% of the cases, the pvalue of the Wilcoxon test is below 0.0001 when compared to the next best method.
The results for the second set of experiments are obtained by our runs and compared with both deep and classical methods (we separate these into two tables due to layout considerations). The deep baselines, GOAD and DROCC, are run each with three different architectures. The mean and SD for the F1 score across 20 runs are reported in Tab. 2, see Appendix C for the AUC results.
A similar experiment is conducted in comparison to classical methods: IForest (Liu et al., 2008), KNN, PIDForest (Gopalan et al., 2019), RRCF (Guha et al., 2016), and COPOD (Li et al., 2020). The latter has been shown to be highly effective on the ODDS collection, when tested with a random 60%/40% train/test split protocol (Li et al., 2020).
As can be seen, in Tab. 2,3, our method obtains the highest mean performance. The mean rank reported in the tables considers the results of both tables together, and our mean rank is considerably higher than all classical and deep baselines. KNN, perhaps the simplest method, obtains the second best performance. While we list the KNN results for K = 5, which is the best test-set parameter found, KNN outperforms the other baselines for other values of K as well, see Appendix C.
For all cases in which our method achieves the highest score, not including the cases in which it ties at 100% with other methods, the pvalue of the Wilcoxon test is below 0.0001 when compared to the next best method.
To further visualize these multiple-benchmark results, we employ a Dolan-More profile. In such profiles, there is a single plot per method, based on the statistics of the performance it obtains compared to the best method for each benchmark. Specifically, the ratio of benchmarks for which the method obtains up to a fraction θ of the maximal score obtained by all methods. This is plotted for
0 ≤ θ ≤ 1. and A leading method would is expected to obtain a ratio of 1.0 closer to θ = 1, which means that it obtains, on all datasets, performance that is within a relatively narrow multiplicative margin of the best method. A dominating method would present a graph with higher values along the y-axis for all θ values, which means that for every margin, it has more datasets in which it is within margin of the best results than any other method. Fig. 2 presents our results for the F1 score, showing that our method leads by a significant margin over the even 11 other alternatives.
Ablations and parameter sensitivity We selected four representative datasets (’Wine’, ’Glass’, ’Thyroid, ’Letter’), which vary in the number of dimensions, the number of samples, and the performance level, and ran an ablation analysis on them. The variants we compare include: (i) a variant of our method, in which the tanh activation of the first layer of F is replaced by a LeakyRELU, (ii) a variant in which only the first of the two normalizations of the query and vectors that F and G output takes place (iii) a variant in which only the second normalization takes place, i.e, normalization occurs in the conventional way, (iv) a variant in which the order of the two normalizations is reversed, and (v) a variant with no normalization, i.e., F,G are trained and used instead of FN and GN .
The results are reported in Tab. 4. As can be seen, the tanh activation in F improves results, to a varying degree, on the four datasets. Normalization tends to help across datasets. However, on Thyroid, applying no normalization at all provides better results. Omitting only the 2nd normalization, which is often used in contrastive learning, is detrimental in all cases. Omitting the 1st normalization, while keeping the 2nd, also hurts performance. Reversing the order of the two normalizations, which changes the scales of the features before this scale is normalized for, is detrimental.
On the same four datasets we also evaluate the sensitivity of the method to its hyperparameters: k, u, τ , and r. In each experiment, we fix three parameters and vary the fourth. The results, provided in the supplementary Fig. 7 of appendix C, indicate that the method is largely insensitive to its parameters. This robustness is further supported by using the same hyperparameters on a large number of datasets.
To further verify that the method works for any k, we tested for Arrhythmia, which has a dimensionality of 274, values of k between 1 and 150. The best F1 score was obtained at 63 (in the experiments it is 124), but all values were relatively similar and between 58.59% and 67.2% (the AUC values were in the range 80.7%–84.3%). For all k, the F1 and AUC are better than all baselines, see Appendix C for the full results. Further evidence of the robustness of the k parameter, which is the main hyperparameter, since we fix u = 200 and τ = 0.01, is provided by simply using k = 1 across all ODDS datasets. In 30% of the experiments, this leads to better performance than the k used according to Sec. 3. The mean F1 is lower by 2.6% (AUC lower by 1.7%), still outperforming all baselines, see Appendix C for the full results. Note that in this case k << u.
We further investigate the sensitivity of our method to the network architecture. When changing both F and G, a variant with only one hidden layer obtained, on average on ODDS, an F1 score that is lower by 2.8% (AUC by 1%; still much better than all baselines in both scores), and a variant with an extra hidden layer in both had a mean F1 that is lower by 10.2% (AUC difference of 4%) than the default architecture. See Appendix C for the full tables and Appendix F for a discussion.
As described in Sec. 3, the score is computed multiple times after permuting the features, and the number of repeats (r) depends on the dimensionality d and number of samples n. In Fig. 7(d), we present the effect of the number of repeats on performance for multiple relatively small datasets, at different performance levels. Shown are the mean F1 over 10 runs and also, as error bars, the SD. We observe that adding repeats typically helps, albeit modestly. It also tends to reduce the variance between runs. A drop in performance between the first and the second repeats may indicate that the order of features is informative. However, this does not happen often.
To study the potential benefit that our method obtains from the particular order of the features in which the data is given, we have compared our results (with no permutations) to those obtained when employing a single, fixed permutation. The average F1 score our method obtains across the entire ODDS collection is 64.1%, and after permutation 64.4% (mean over all datasets, with 10 repeats). This trend, however, is dominated by a few datasets, and in AUC terms, the results are 87.2% vs. 87.1%, where the original order leads. In both AUC and F1 the original order leads in the same 50% of the datasets. The lack of a clear advantage for the original order is also apparent in the auxiliary results of Appendix A.
The same experiment also verifies that our improved performance does not arise solely from the bagging effect of having r > 1. Comparing the case of r = 1 to the baselines reveals that our average F1 score of 64.1% (AUC 87.2%) is higher by a sizable margin then all of GOAD’s architectures, which have a mean F1 of 51.3–55.9% (depending on the architecture; AUC of 79.6–81.0%) and COPOD’s 44.7% mean F1 score (AUC 79.1%).
Interpretability Since we apply a sliding window approach and obtain the anomaly score by simple integration, we can point back to specific features in the input vector that are overrepresented in factors that indicate the anomaly. This provides a natural explainability capability, which, as far as we can ascertain, was not shown for the deep-learning based anomaly detection methods (interpretability can be provided for such methods indirectly, e.g., by tracking network activations (Bach et al., 2015) or linear approximations (Ribeiro et al., 2016)).
As an example, consider the “Thyroid” dataset. In the ODDS repository, the normal class contains samples labeled “normal” or “sub-normal functioning” and the anomalies are from the class “hyperfunctioning”. Examining the loss score per each sliding window (and not just the mean over all of them) reveals that in 83% of the anomalies detected the feature with the highest loss value (implying the lowest correlation with the rest of the vector compared with the training set) was “T3 concentration”, in 12% it was “TSH levels” and the rest were “T4U concentration”. This makes medical sense since hyperthyroidism is characterized by high T3 and T4U levels, and low TSH.
To examine the ability of the model to identify dependencies, we conducted a random data experiment as follows: we generated random data for vectors over R4 by sampling from a Gaussian distribution with means [1, 2, 3, 4] and a covariance matrix that implies a high covariance of 0.85 between the features of indices 2,3, zero covariance elsewhere, and unit variance for all of the features. We also sampled random data that was identical in all parameters except for having no correlation between any of the features, i.e., using a diagonal covariance matrix. The model (k = 2, r = 1) was trained on half the data with correlation between features and tested on the other half with the addition of all non-correlated data. The mean AUC over 10 runs was 88%, which is considerably higher than the baselines, see Appendix C (Tab. 10). More interestingly, features 2,3 were highlighted by the model as those with the highest anomaly score. This was inferred by simply aggregating the losses of the sliding windows each feature is included in, per sample. In 89% of the samples detected as anomalies, either the second or third features were ranked highest (with almost equal frequency). This is reassuring, since the only difference between the two sampling procedures was in the dependencies between the features, and not in their mean values or variance.

heading: 5 DISCUSSION
text: The self-supervised task we seek to solve through the learning of the mappings F and G is relatively simple and can be solved without learning in O(dm), since it amounts to identifying whether bji and aj ′
i overlap, in which case j ′ 6= j, or not, which implies that j′ = j. Since this decision process is not class-specific, should we be surprised that the learned representations seem to be highly distinctive of class membership? Consider by way of analogy the self-supervised learning of natural images by employing a constrastive loss between an anchor image, its transformation, and another image, e.g., (Chen et al., 2020). Identifying the transformation between two images, which is no more difficult than finding out whether they are related by a transformation, is done reliably with neural networks using point matching (Sarlin et al., 2020), or using “direct methods” (DeTone et al., 2016).
These methods are not class-specific. However, the representations learned by applying geometric transformations are extremely distinctive of the class.
To study this further with a random data experiment, in which the vectors x are composed of d independent variables sampled uniformly in [−1, 1], we learn networks F and G and observe the success in identifying index j, vs. the other indices. The results for d = 6, 15, 30 are reported in Fig. 3. As can be seen (our method, all experiments), the loss is being reduced while training. However, the network is not able to immediately obtain perfect performance, especially for larger d. As mentioned in Sec. 3, the architectures of F and G are slightly different, based on the motivation that this would make the class-independent solution less accessible during training. As shown in Fig. 3, when replacing the architecture ofG to be identical to that of F , the class-independent learning has a lower error on most epochs. Lastly, for data where the features are different from one another the problem becomes less challenging. A second random data experiment sets the mean value of each variable uniformly in [0,1] and the data is i.i.d Gaussian with a unit variance. In this case, the network reaches the same loss in one-tenth of the number of epochs, see Appendix C (Fig. 9).
It has been argued by Bergman & Hoshen (2020) that the most common real-world anomaly detection applications are for tabular data. Such data tends to have considerably higher structural variance between datasets than perceptual data. The type of features (continuous or categorical), the number of features, and dependencies between features vary greatly from one dataset to the next. This variance makes the development of a generic anomaly detection method challenging, and we find a significant difference in performance across methods. We note, however, that our method is more stable than recent ones, and handles multiple datasets using the same architecture, except for minimal tuning directly related to the dimensionality of the data. For example, on the four datasets commonly used in the literature, the method is applied exactly the same, except for Thyroid, where due to the low input dimensionality (d = 6), the value of k = 2 was used. In contrast, GOAD, for example, has dataset-dependent stopping criteria (early stopping or 25 epochs) and employs three architectures on these four datasets.
Limitations In some symmetric cases, for specific parameters and with no permutation, our method would fail in the task of anomaly detection. Specifically, any scenario in which for both normal and anomalous data the pairs constructed from each instance {(aji , b j i )}mj=1 are identical would be impossible for the network to differentiate between. For example normal data of the form (0,1,0,0,0) and anomalous data (0,0,0,1,0) with k = 2 and r = 1 violates the second assumption listed in Sec. 1. See Appendix A for simulations of this example.

heading: 6 CONCLUSIONS
text: We present a generic one-class classification method for tabular data. The method assumes that it is possible to identify missing features based on the rest and employs a contrastive loss for learning without any other auxiliary loss. In an extensive set of experiments, the method presents a significant advantage over existing anomaly detection methods. The method requires no tuning between different datasets and is stable with respect to its hyperparameters.
REPRODUCIBILITY STATEMENT
The full implementation of our method and scripts for reproducing the experiments are attached as a supplementary zip file. This archive includes a README file and a list of requirements that support seamless reproducibility.
The runtime statistics are reported in Appendix E. We note that our method requires modest computational resources, such as those available for free on Google Colab, further supporting widespread reproducibility.
A detailed depiction of the (few) hyperparameters used is given in Sec. 3. A sensitivity analysis presented in Sec. 4 demonstrates that the method is insensitive to specific parameter values.

heading: B ODDS DATASET STATISTICS
text: Tab. 5 presents the number of samples, the dimensionality, and the number of samples not from the main class for the various datasets that belong to the collection “Multi-dimensional point datasets” from the Outlier Detection DataSets (ODDS, http://odds.cs.stonybrook.edu/, accessed May 2021).

heading: C ADDITIONAL RESULTS
text: In Tab. 6 we present results for the second set of experiments, this time reporting AUC (the manuscript reported the F1 scores in Tab. 2.).
In Fig. 7, we present the results of a parameter sensitivity study performed on four datasets. The results indicate that the method is largely insensitive to the parameters: the sliding window size k, the dimensionality of the latent space u, the softmax temperature τ , and the number of random permutations r. In each experiment, the tested parameter was varied, while all other values were default.
In Fig. 8 we show the results for varying k in the range [1...150] for the Arrhythmia dataset. For each k value, the reported results are averages over 10 runs. The best F1 score was obtained at 63 (the k value selected at our default settings for this dataset is 124), but all values were relatively similar and between 58.59% and 67.2% (the AUC values were in the range 80.7%–84.3%). For all k, the F1 and AUC are better than all baselines.
In Tab. 7 we show the results for a sensitivity test, in which we set the hyperparameter k as 1 for all datasets, compared to the original results reported in the paper. In 30% of the experiments, this leads to better performance than the k used according to the default setting. The mean F1 is lower by 2.6% (AUC lower by 1.7%), still outperforming all baselines.
Fig. 9 shows the results for the random data experiment, where features are sampled independently from a Gaussian distribution with a different mean per feature, sampled from U[0,1]. These runs are compared to results where the features are sampled from U[-1,1] i.i.d. Also, for each distribution we show the results for two variations of our network - one with the default architecture, and one where we change the F network so it is similar to G (i.e. without TanH activation).
As can be seen, for data sampled from Gaussian distribution the network converges considerably faster, which suggests that solving the class-independent task is easier when the features have distinctive differences. Also, in both variations of the data distribution, the TanH architecture variant requires more epochs to converge to similar values, supporting our hypothesis that the design of our architecture makes it harder for the network to learn the class-independent solution.
In Tab. 8 we report the results for running different variations of our architectures: a variant with a single hidden layer for both F and G, one with an extra layer for both, and the default architecture.
In Tab. 9 we report results for a test designed for studying whether the order of the features effects the results of the model. To this end,we measured the performance after employing a single fixed permutation on the datasets, compared to not employing any permutation at all (the original order as provided by the dataset). Permuting the features had a minor effect, and across the collection of datasets, there is no clear advantage to the original order.
Tab. 10 reports the results of the experiment in which both the normal and the out-of-class distributions are sampled from 4D Gaussians with a mean vector of [1, 2, 3, 4]> and a per-variable variance of one. The in-class samples where sampled from a convariance matrix with a correlation of 0.85 between variables two and three, but uncorrelated otherwise. For the out-of-class samples, the covariance matrix was diagonal. This is a challenging problem, since variables two and three can appear to be similar even for an out-of-class sample. As can be seen, our method greatly outperforms the baseline methods.
The KNN baseline, while being the simplest baseline, was shown to be the strongest one. In the experiments of the paper we report results for the highest performing parameter, which is K = 5 neighbours. The mean F1 score for various values of K tested are: 67.1, 66.3, 67.7, 66.8, 66.4 for K of 1,2,5,10,20, respectively.

heading: D AN ALTERNATIVE METHOD (REPORTING A NEGATIVE RESULT)
text: We briefly summarize an alternative method that was tried, which failed to produce the results we hoped for. Earlier in the development process, we had an intuition that making a comparison with external samples might be beneficial for our model, and we experimented with drawing the negative samples from other instances, i.e. instead of using:
q = F (bji ),v + = G(aji ),v
− = G(aj ′
i ) (2)
for some i, j, j′ 6= j, we used:
q = F (bji ),v + = G(aji ),v − = G(aji′) (3)
for i, i′ 6= i, j. The results obtained by employing this method on the ODDS repository were significantly lower in terms of F1 and AUC performance on almost all datasets, and were not competitive with the baseline methods.

heading: E RUNTIME
text: In Tab. 11 we show the average runtime in seconds for each of the datasets. We include two versions: one with the default amount of repeats rdefault = 1 + b100(log(n) + d)−1c and another one with rfaster = min(2, rdefault).
We observe that the difference in performance between the two versions is typically minor, and the faster method might be preferred if runtime is a top priority. The code includes an option for running the model in the faster mode.
The google colab infrastructure was used to run the experiments. GPU: Tesla K80, 12GB GDDR5 VRAM; CPU: Single core Xeon Processors @2.3Ghz; RAM: 24 GB. For the GOAD baseline that required larger memory, we used a 32GB GPU and 512GB RAM.

heading: F DISCUSSION OF ALTERNATIVE ARCHITECTURES
text: The architecture we employ for F and G is a straightforward fully connected architecture with two hidden layers. One can hypothesize that with less capacity, the network may not be able to learn the required mappings as effectively, making it incapable of utilizing the first assumption. However, with added capacity, the network may shift from the class specific solution to a generic solution, validating
our 2nd assumption. See Galanti et al. (2018) for a remotely related phenomenon in the completely different unsupervised learning task of mapping between two visual domains: with too few layers the mapping fails to produce samples in the target domain, for too many layers, the mappings are not specific to the input image.
The hypothesis is partly validated empirically in the results reported in Appendix. C Tab. 8. A variant with only one hidden layer obtained only 1% less mean AUC (still better by a gap than all baselines) on the ODDS benchmark than our original results, and a variant with an extra hidden layer had an AUC 4% lower.
In order to study whether this drop in performance is due to assumption 2, we examine the average log likelihood. The mean loss for the “normals” category is 0.016 and 0.014 for the original architechture and one extra layer respectively, and the mean anomaly class losses are 1.54 and 0.96. We, therefore, observe that the ability to solve the problem (assumption 1) is almost the same and slightly improves with the added capacity. However, the gap between the normal class and the outlier samples has shrunk dramatically with the extra layer.
Method r = 0 r = 1 r = 0 r = 1
Method AUC (percent; Mean ± SD) COPOD (Li et al., 2020) 52.2 ± 3.2 Isolation Forest (Liu et al., 2008) 62.3 ± 4.6 OC-SVM (Schölkopf et al., 1999) 66.8 ± 1.1 GOAD KDD architecture (Bergman & Hoshen, 2020) 74.0 ± 3.2 GOAD Thyroid architecture (Bergman & Hoshen, 2020) 73.7 ± 1.3 Ours 88.1 ± 1.5

Review summaries:
This paper introduces an anomaly detection algorithm for tabular data based on contrastive learning in the semi-supervised setting (training set assumed to be only normal data). The contrastive learning task is based on masking: the features from a single training example are split into two groups (pairs), one being a subsequence of the features, and the other its complement set; the learning task is then to differentiate pairs of splits that overlap (positive) vs not (negative) (from the same split or not) for each sample. The contrastive loss defined by this task is used to train the model and as the anomaly score. The proposed method is extensively evaluated on a range of tabular datasets where it outperforms recent methods.

- The authors propose a novel anomaly detection framework that can be useful for tabular data. - The authors utilize the contrastive learning to learn the relationship between features in the tabular data. Then, using the loss as the anomaly scores for evaluation. - The authors provide extensive experimental results which are promising.

## After rebuttal I am impressed by the comprehensive experiments and baselines and increase my score. Authors provide very complete baselines and also compare in other datasets, which greatly reduces the concern that the proposed rule overfits to the ODDS benchmark.   ## Summary This paper learns a contrastive representation that helps differentiate between normal and abnormal data in the tabular data. Contrary to other contrastive learning methods which learns to differentiate between different examples, this paper instead differentiates between in-window vector and out-window vector for each example under a sliding k-sized window (k is a hyperparameter). Since the features are unordered, they shuffle the feature orders to get multiple score and average them. They show that it outperforms other baselines in a large suite of tabular data, ODDS benchmark, with default hyperparameter rules (e.g. the k is set based on number of samples N and d). They compare mainly with recent methods DROCC, GOAD and COPOD. 

This paper proposes a contrastive learning method for unsupervised anomaly detection on general multivariate (tabular) data. The idea of the method is to train two encoders, one that embeds a subset of k features and one that embeds the residual d − k features, such that the resulting embeddings are closely aligned via the contrastive loss, where all other subsets of k features are taken as negative examples respectively. This is then repeated over all subsets of features of size k. Thus, the intuition of the approach is to extract common dependencies (e.g., correlations, statistical redundancy, etc.) between the features by achieving a small contrastive loss over the training data, thereby obtaining a high contrastive loss for anomalies (which presumably lack these common dependencies). An empirical evaluation on the Arrhythmia, Thyroid, KDD, and KDDRev datasets (where results from the literature exist) as well as 30 additional datasets from the ODDS library show that the proposed method performs favorably over other recent competing methods (DROCC, GOAD, COPOD) on tabular data.