Anomaly Detection for Tabular Data with Internal Contrastive Learning
Authors: Tom Shenkar
Lior Wolf
Venue: ICLR 2022 Poster (2022)
Keywords: Anomaly detection
Tabular data

Abstract:  We consider the task of finding out-of-class samples in tabular data, where little can be assumed on the structure of the data. In order to capture the structure of the samples of the single training class, we learn mappings that maximize the mutual information between each sample and the part that is masked out. The mappings are learned by employing a contrastive loss, which considers only one sample at a time. Once learned, we can score a test sample by measuring whether the learned mappings lead to a small contrastive loss using the masked parts of this sample. Our experiments show that our method leads by a sizable accuracy gap in comparison to the literature and that the same default set of hyperparameters provides state-of-the-art results across benchmarks.

Published as a conference paper at ICLR 2022
ANOMALY DETECTION FOR TABULAR DATA WITH INTERNAL CONTRASTIVE LEARNING
Tom Shenkar & Lior Wolf
Blavatnik School of Computer Science, Tel Aviv University tomshenka r gma 1, com, wo fêcs tau.ac.................. il
ABSTRACT
We consider the task of finding out-of-class samples in tabular data, where little can be assumed on the structure of the data. In order to capture the structure of the samples of the single training class, learn mappings that maximize the mutual information between each sample and the part that is masked out. The mappings are learned by employing a contrastive loss, which considers only one sample at a time. Once learned, we can score a test sample by measuring whether the learned mappings lead to small contrastive loss using the masked parts of this sample. Our experiments show that our method leads by a sizable accuracy gap in comparison the literature and that the same default rule of hyperparameters selection provides state-of-ther thettrrtt results across benchmarks.
INTRODUCTION
In the one-class classification problem, one learns a model with the goal of identifying whether test sample belongs the same distribution from which the training set sampled (Schölkopf et al.) 1999). Methods for solving this problem, therefore, define a criterion that is satisfied for the samples of the training set, while being less likely hold for samples from other (unseen) distributions.
When considering perceptual data, one can rely on the structure of the input. For example, images can be rotated, and the discrimination between the various rotations is class-dependent and, therefore, indicative of the class (Golan & El- Yaniv 2018). In this work, we consider tabular data, in which there is no prior information on the structure of the data.
If one assumes that no such structure exists, i.e., that the variables are independent, then the criterion defined by combining the per-feature scores. This, however, is not competitive in cases in which the features in each sample vector not independent (Ahirwar et al. 2012). By making the assumption that the dependency structures are deessddeen,,nn, one can construct, for example, a low-dimensional subspace and expect
classes to lie outside it (Jolliffe) 1986).
In this work, we make the assumption that the way in which a subset of the variables in the feature vector is related to the rest of the variables is class-dependent. Our method considers subsets of consecutive variables. In this manner, for a given input sample 2; E Rdd we obtain a set of pairs {(a 6)) where af is vector of k consecutive features from K;, by is the vector of all other feature values, and m = d + All vectors af have the same length and vary according to the first coordinate of z;, from which the subsets are collected.
Given a training set, we learn neural mapping F dd R for the vectors of type and mapping G : I - R2 for vectors of type a such that the mutual information in the latent space of dimensionality between matching elements (a), b/) is maximized. The same networks F, G' are learned for all samples mutual information is done through contrastive learning (Oord et al. 2018).
the training set and for all starting indices j. The maximization of the
At test time, the anomaly score that is being used y(x) is exactly the loss that is being minimized during training. This puts method solid ground, in contrast to many other outlier detection methods, in which networks are trained for one goal, and anomaly identification assumes that some regularity is learned. Additionally, since the method employs sliding window approach, it is

---

Published as a conference paper at ICLR 2022
straightforward to identify the specific features in the imput vector that contribute high anomaly score. This provides our method with a natural and direct interpretability.
Assumptions and design choices Our method is generic and assumes little on the structure of the data. We make the following set of assumptions and associated design choices. First, we assume that one can learn networks F, G that identify matching elements (a/, b2). Second, we assume that when training F, G' on data from a single class, the learned networks are class-specific in the sense that the recognition rate for matching elements that belong to samples from other classes would be considerably lower. Both of these assumptions are validated in Appendix A
Note that the contrastive learning task that F' and G are trained to solve can be easily solved with a very short deterministic program that is class- class-indepenan independent, which checks the overlap between by and a.. However, the success of our method implies that by training our neural method, one learns class-specific models. Furthermore, simulations on synthetic data demonstrate that learning the plass-independent solution requires lengthy training process and does not converge to perfect classifier. These results, which shed light on self-supervised training, are given in Sec. [
Our assumptions further validated by an extensive set of experiments, in which our method, using a single architecture and the same rule of hyperparameters selection, obtains state-of-the-ard results in one-class classification of tabular data. The increase in accuracy compared to existing methods is sizable and consistent across benchmarks. It is further shown that the method is insensitive to its hyperparameters, demonstrating that the validity of the assumption is a stable phenomenon. Even with h = 1 (masking of single feature), the method outperforms all baselines by a gap.
Two softer assumptions are introduced by performing specific design choices (first soft assumption) or implementing the method in a specific way (second). (i) For simplicity, F, G can learn to match of with by regardless of j. In an alternative implementation, one can learn a pair of networks FJ, G for every or employ positional encoding for j. Since our method is effective with single pair of networks, this option was not explored. (ii) The original order in which the features are given is used, based on the intuition that in several datasets, the order is not random. Since our method, by default, considers consecutive vector elements, nearby features are stacked together. Naturally, the strength of this effect is deepedet.... We quantify it in Sec. 4 and show, by gathering statisties over a large corpus of varied datasets, that the original order does not provide a clear advantage over a random order of features (consecutive features after applying permutation results in random, non-overlapping. subsets of features). This is also evident from the results presented in Appendix
2 RELATED WORK
The main application of one-class classification methods is anomaly detection; that is, identifying outliers after observing a set of mostly normal (the opposite of abnormal) samples (Chandola et al. 2009 Pang et al. 2020 Ruff et al. 202T). A straightforward way to perform this task is to model a distribution based on the training samples and then estimate the likelihood of each test sample. For this purpose, one can employ non-parametric methods, such as kernel density estimation (Parzen 1962). or the very recent COPOD method (Li et al. 2020) that is based on empirical copula model. Parametric methods include Gaussian and Gaussian mixture models (Zong et al. 2018). as well as adversarial learning (Schlegl et al.) (017). An alternative to density estimation approaches relies on regularized classifiers. The classical methods were mostly kernel-based methods, in which the role of the regularization term is to ensure that the fitted model is tight around the observed samples (Schölkopf et al. 1999). Many of the first deep learning anomaly detection methods employed such classical one class methods on top of auto- encodrr based representations (Hawkins et al. 2002)
Sakurada & Yairi 2014| Xia et al. 2015 Xu et al. 2015b: Erfani et al.) 2016 An & Chol (0155). More recent methods apply a suitable one-class loss in order learn a neural network-basee representation
in an end-to-end manner (Ruff et al., 2018). In order to further avoid the problem of representation
collapse, the DROCC method (Goyal et al. 2020) applies virtual adversarial training (Miyato et al..
2018) to create virtual negative samples around the training samples.
Self-supervised learning, in which an unsupervised learning problem is turned into a discriminative learning problem, was introduced to anomaly detection by Golan & El- Yaniv (2018). Their method predicts the image transformation that is applicd to an image. Assuming that this classification problem is class-dependent, the membership score is based on the success of the learned classifier
2

---

Published as a conference paper at ICLR 2022
on given test image. GOAD (Bergman & Hoshen 2020) improved this method, by learning an embedding space, in which the classifier considers the distances to the center of the set of training samples after applying each transformation. The method is also made suitable for tabular data, in which case random linear projections replace the geometric transformations. Our work is based on a different self-supervised task called masking, in which part of the data is held out and is predicted by the rest of the data. This form of self supervision is commonly used for learning representation in NLP (Mikolov al., 2013) and in computer vision (Pathak al. 2016). As far we can ascertain, masking was not used for one-class classification before.
The idea of contrastive learning has emerged in metric learning, where was used train Siamese network (Chopra et al., 2005). However, its main application is in unsupervised representation learning (Hadsell et al. 2006). The learned embedding brings associated samples closer together, while pushing away other samples. The framework of noise contrastive estimation (Gutmann & Hyvärinen] 2010) casts this type of learning as a form of mutual information maximization. Many of the most recent contrastive learning methods perform unsupervised learning by anchoring an image together with its transformed version, while distancing other images (He et al. 2020 Misra & van der
Maaten 2019 Chen et al. 2
Recently, the contrastive learning method of Chen et al. (Chen et al. 2020) was adapted for the problem of one-class classification (Tack et al. 2020). The obtained score combines the norm of the representation (Taigman et al. 2015) with the the maximal similarity to any sample of the training set to define an anomaly score. The performance is further enhanced by contrasting two sets of image transformations: those that maintain the same-identity property vs. those that lead to different training identity. Another method applies contrastive learning that is tailored to one-class learning in order to learn an image representation, followed by a one class classifier (Sohn et al. 202T). These image transformation-aee based techniques are not applicable to tabular data, since there no group of transformations that the content of generic vectors is invariant to.
The contemporaneous NeuTraL AD work by Qiu et al. (202T) employs per-sample contrastive loss for identifying anomalies in tabular data, similar to our work. However, there are crucial differences: (1) NeuTraL AD learns specific masks, while we apply the entire set of the masks specified by a window size k. (2) The role of NeuTraL AD masks is to mask-out parts that are irrelevant for specific classes. In our case, we perform a two sided matching that identifies the masked part form the original. (3) NeuTraL AD learns single feature extractor ("encoder") for both the original and transformed data. In our case, the two sides of the contrastive loss are of very different dimensions (d - and k) and we employ two different encoders. Furthermore, motivate) the importance of using encoders that are of different architectures. (4) Their method is built such that there is diversity between the different views and similarity between each view and the original. This requires the views to be spread, in the latent space, around the original sample. In our case, the constraints are such that the crop matches the complement elements of the vector more than all other crops. There is no requirement that the other crops are dissimilar between themselves.
explore in our ablation (and theoretically
3 METHOD
We are given a training set of n in-class samples S = {x;), each a vector of d dimensions. Our goal is to design score y : Rd R that maps samples from the sample domain to a low value if they sampled from the underlying distribution from which S is sampled and to a high one otherwise.
The method has two hyperparameters that specify dimensions: h determines the size of the subset of features we consider, and u determines their embedding size. A third hyperparameter is the temperature constant of the loss.
The method first constructs set of d 1 - l pairs (<x)) {(a), b()) from each training sample z4. Each pair in this set is obtained by extracting &o consecutive variables from T;, Let az.
1 < j S m be the vector [ x +1
s+k-1, where superscripts denote elements of the vector -r.- We define b/- = 2f 2f...., x. i+k ..... x=f] to be the vector of the other d - k elements in .r The method then learns two mappings F, G that maximize the mutual information between F(b)) and G(a)). where (a. 6/) E ((x,), 1... n. The same mappings are learned for all samples and regardless of the index j E [1,m].
3

---

Published as a conference paper at ICLR 2022
0.9
0.8
Raloo of doxorets
labolnni
BROCC (Thymac)
GO4D (Kof)
0.4
GOAD (Kalle]]
GOAD (y
0.3
COROD
POOEST
ESN
0.5
3 bb - of af af af a aj a? Xi aj
Figure Dolan-More profile for ODDSs Fl scores. The x-axis is varying threshold (0). The plot considers the performance of each method in comparison to the best performing method. The y-axis is, for given method, the ratio of datasets in which the obtained score is above 0 times the maximal score over all methods. Experiments with missing baselines were omitted for fairness.
The underlying learning problem. Given a sample vector Fj, we consider the sub- vector of and its complementary b2. The net- works are trained to produce similar embeddings for this pair of vectors, while distancing the em- bedding of of for j' + 3 from that of 1,3
Figure 1:
The mutual information is maximized through the use of the noise contrastive estimation frame- work (Oord et al. 2018). see Fig. for an illustration of the contrastive relations our setting. In this framework, there is query q. positive sample v+, and negative ones v_, all vectors in R". Contrastive learning maximizes the similarity the query with the positive sample, while minimizing it with the negative samples. In our case, the vectors are given, for some j. # as:
= =F(b(), ++ G(a)), v = =Gaa ).
Almost all current contrastive learning methods employ normalization of the vectors q and v+ vz such that their unit norm is 1. In our method, this normalization is performed after a first normalization step that is applied at each vector dimension in Ra separately, by considering all of the sub-vectors of the input vector z;, The motivation for the first normalization step is that scale, by itself, can lead to the identification of a certain feature. This scaling the features is often dlass-independent, since frequently depends more the nature of the feature than the class, while the classification task is required to be class-dependent.
The normalized network F considers the normalizes each of its rows to have an 1.2 norm of 1 to obtain a matrix B., We define the normalized network EN such that E (b²) is the j-th th column BN, further normalized to an L2-norm of one. Although we omit from the operand list, FM (b{) depends not only on by but on all b-type vectors in ((x;) through normalization. Similarly, G. (a!) is detined by the matrix A which is double- normalized version of the matrix that contains vectors of the form G(aa), where varies by column.
matrix B [F(b]).
F(b,)) and
The contrastive loss É is defined as an m-way classification problem, in which the cross-entropy loss for given temperature
used. The logit used is the pseudo-probability for the positive sample +t being selected over m - 1 negatives given the query (Wu et al. 2018 He et al. 2020). Note that the normalized versions of these vectors are being used.
{(F, G. ((x)).)) - In exp(FN F. (V) G. (a))/T)
(1)
Σグ= exp(F EN (b) GN (a (4()//))
Once the networks F and G are trained with the contrastive loss, we define the one-class classification score as fzz) = E, {(F, G, ((r),)) where ((x) is constructed for sample z similarly to the construction of ((x,) for training sample x.,
TRAINING, PARAMETERS, AND ARCHITECTURE
During training, the method directly minimizes the anomaly score of the training set S, given by f. Exes ((r). Training employs the Adam optimizer (Kingma & Bal 2014) with learning rate
4

---

Published as a conference paper at ICLR 2022
of 1003 3 It stops when the loss is smaller than 10-3 for datasets with d < 40. For larger input dimensions, this criterion would lead to long training sessions, and we use 10 20-22pm insteed..
We fix 0.01 and enough capacity throughout all experiments, without the need to tune it for each problem. We set k proportionally to the input dimension d. For d smaller than 40,
200. The value of u, which is often much larger than k. provides
set k 2. for d in the range [40, 160] we employ k = 10, and for d > 160, & takes the value d - 150. However, as our experiments in Sec. 4 show, the method is not very sensitive to the value of k.
We found that we can make use of the fact that the features are unordered and simply combine multiple scores, each obtained on different permutation of the features. In our experiments (Sec. 4. repeating this way is only very seldom detrimental to accuracy, and it improves performance for small d and very small n. On the other hand, it does add to the overall runtime.
In order to make use of this bagging effect, when needed, we set the number of repeats to be r = 1+ [10g())g(n) + d)=1] For each repeat after the first, we randomly permute the set of features. The score that the method returns is the mean of the scores obtained from each such repeat.
F and G are two fully connected networks with LeakyRELU activations (Xu al. 2015a) using a slope coefficient of 0.2 in all layers, except for the first layer of F, which has tonh activation. The reason is that we wish to distance the embedding of the a-part of x and its part, making a simple
matching between the parts, which overlap for b/ and af when j' * j. more challenging. See Sec. for a detailed discussion. F has two hidden layers, with u and 2u hidden units, each followed by Batch Normalization (Ioffe & Szegedy 2015). G' is similar, only that due to the smaller input sizes, the hidden layers have u/4 and u/2 units and Batch Normalization is applied only after the first layer.
4 EXPERIMENTS
Datasets Based on the terminology of the field of anomaly detection, we use the term "normal" to describe the class observed during training, and abnormal describe samples from the other class or classes. The experiments were conducted on two groups of datasets: (i) collection of four datasets commonly used for reporting anomaly detection for tabular data, (ii) much more comprehensive set of tabular datasets for benchmarking outlier detection. The first set of datasets contains two small-scale medical datasets (Arrhythmia and Thyroid), as well as two cyber-intrusio detection datasets (KDD and KDDRev) which are considerably larger. Following Zong et al. (2118) and others, the categorical attributes are presented to the network as one-hot vectors. The second set employs the Mullii-dimesioonll point datasets" from the Outlier Detection DataSets (ODDS It contains 31 datasets, including two of the four datasets above. The datasets Heart and Yeast were omitted, as it is not clear from the description which class is the normal one. See Bpp.Btth the dimensionality data.
Evaluation protocol and scores Following Zong et al. (2018): Bergman & Hoshen (2020). the training set contains random subset of 50% of the normal data. The test set contains the rest of the normal data, as well as all the anomalies. In the first set of experiments, the mean and standard deviation (SD) of the Fl score are reported. For the experiments we run, this is computed over 500 random splits for the smaller datasets (Arrhythmia and Thyroid) and 10 splits for the larger ones (KDD and KDD-Rev). For the baseline methods, the sample size varies and in some cases the results reported in the literature are given without SD. Following the existing protocol, the decision threshold for scoring the methods is chosen such that the number of test samples above this threshold (i.e.. classified as anomalies) is the number of anomalies in the test set. For the second set of experiments, in which all runs were made by us, we report (sometimes in the appendix), in addition to F1, the AUC. Since the AUC varies less dramatically than Fl scores, it is more suitable for comparison across datasets. also has the advantage of not requiring setting a threshold.
Baseline methods For the first set of experiments. comprehensive set of literature baselines is presented. One-Class SVM (OC-SVM) (Schölkopf et al. 1999h) and DAGMM Zong et al. (0018) results are reported computed by Zong al. (2018). GOAD (Bergman & Hoshen 2220)) Local Outlier Factor (L.FF) (Breunig et al. 2000), and an ensemble method (FB-AE) that employs autoen- coders as the base classifier, feature bagging as the source of randomization, and reconstruction error as the anomaly score, are reported by Bergman & Hoshen 2200). A recent baseline, DROCC Goyal
http:/ /tot:://ddd.... cs. wtt.oids..sss.ttooooooooooo st.onybrookoooo
accessed January 2021
5

---

Published as a conference paper at ICLR 2022
Table 1: Mean Fl (percents) and standard deviation over multiple resampling the four datasets commonly used in the literature. 'DRCCC is reported based on our runs due to protocol discrepancies in the published code; some experiments are missing due to limitations of the published code.
Arrhythmia
Thyroid
KDD
KDDRev
Method
Pi Score SD P1 Score SD P1 Score
SD
F1
Score
SD
OC-SVM (Schölkopf et al.
1999)
45.8
38.9
79.5
83.2
LOF (Breunig et al. 2000)
50.0
0.0
52.7
0.0
83.8
5.2
81.6
3.6
DAGMM ZZng al. 2018
49.8
47.8
93.7
93.8
FB-AE (Chen et al. 2017)
51.5
1.6
75.0
0.8
92.7
0.3
95,9
0.4
DROCC Goyal ct al. 2020
38.8
6.2
72.7
3.1
N/A
N/A
N/A
N/A
GOAD Bergman & Hoshen
2020
52.0
2.3
74.5
1.1
98.4
0.2
98,9
0.3
COPOD LL et al.
2020
58.2
1.4
30.8
0.5
44.5
0.1
30.9
0.4
Ours
61.8
1.8
76.8
1.2
99.4
0.1
99.2
0.3
al. 2020), rerun by us based on their code, since the published results sampled the test set using a different protoco Finally, we include the COPOD baseline (Li et al. 2020), based on our runs, as a modern con-deep-learning method..
For the second set of experiments, we focus on the most recent deep methods: GOAD and DROCC. Since GOAD uses three different architectures in their code, we report results for all three. The first architecture is the one used by Bergman & Hoshen (2020) for the small datasets, the second used for KDD, and the third is the one used for KDDrev. Similarly, DROCC (Goyal et al. 2020) employs three architectures in their code: one for Thyroid, one for Arrhythmia, and one of Abalone, and we run all three. Our method employs the same architecture all experiments, except that & is adjusted according to d, see Sec. 3 As non-deep baselines,
multiple methods including classical methods and modern ones: IForest (Liu et al. 2008), k-nearest neighbours (KNN), PIDForest (Gopalan et al.. 2019), RRCF (Guha al. (2016), and COPOD (Li et al. 2020). All of these are based on our own runs, using the PyOD anomaly detection python package ZZhao et al. (2019) for KNN and IForest, RRCF python package and the official code published by the authors of PIDForest in Github. The defualt parameters were used, except for the suprisignly strong baseline of KNN, for which we report results for the best parameters found on the test set, see Appendix C
Results The results for the first set of experiments are reported in Tab. ! Our method outperforms the literature basclines by a significant margin on Arrhythmia and Thyroid, where the basclines obtain moderate F1 score. On the larger datasets KDD and KDDRev, where the performance of GOAD very high- we outperform it, obtaining near- rear-perfect score. On all datasets, the pvalue of the Wilcoxon test between baseline methods, the feature bagging auto encoder (FB-AE) and GOAD seem to be the strongest, while DROCC, with the correct protocol, is not competitive. COPOD, despite being shown to be successful on many other benchmarks, does not perform particularly well on this first set.
method and the best competing method is lower than 0.001. From the
The results for the second set of experiments are obtained by runs and compared with both deep and classical methods (we separate these into two tables due to layout considerations). The deep baselines, GOAD and DROCC,
each with three different architectures. The mean and SD for the F1 score across 20 runs are reported in Tab. see Appendix [or the AUC results.
A similar experiment is conducted in comparison to classical methods: IForest (Liu al. 2008). KNN, PIDForest (Gopalan et al.) 2019). RRCF (Guha et al. 2016), and COPOD (Li et al. 2020). The latter has been shown be highly effective the ODDS collection, when tested with random 60%/40% train/test split protocol (Li ei al. 2020).
As can be seen, in Tab. 2 3 our method obtains the highest mean performance. The mean rank reported in the tables considers the results of both tables together, and our mean rank is considerably higher than all classical and deep baselines. KNN, perhaps the simplest method, obtains the second
In their implementation, (1) the number of normals in the test set equals the number of anomalies in the data, and the rest are used for training and (2) the threshold in test time to determine F1 is constant across datasets 20% of the test

---

Published as a conference paper at ICLR 2022
Table 2: Fl for the ODDS benchmarks for deep models. To avoid selecting the architecture for the baseline methods, we report all versions, Missing experiments are due to the limitations of the published code.
Method
DROCC (Thyroid)
DROCC
DROCC (Abalone)
GOAD (Thyroid)
GOAD (kddrev)
GOAD (kdd)
Ours
(Arrhythmia)
Wine
20.0:119.0
32.0::3544
63.01:20.0
67.0199.
76.0::10.8
42.2:1.6.9
90.0:16.3
Lympho
0.0.10.0
38.3123.6
65.015.0
68.3113.0
67.7.17.8
46.0121.5
86.7.16.0
Glass
22.2++7.22
13.3+12.0
14.5+11.1
12.7+399
25.7+12.00
24.0+15.1
27.2-10.6
Vertebral
25.7:55.
27.0=15.9
9,346.1
16.319.6
26.94.522
25.514477
26.0:7.77
Wbe
0.0+0..0
18.6+16.0
9.046.2
66.2-2299
16.8:16.1
57.2-6.9
67.623.6
Ecoli
N/A
N/A
N/A
61.413177
69.31.23.7
66.11227.8
70.0
Ionosphere
29.9+6.8
76.3+6.4
76,9+2.8
83.4+22.6
88.1+2.3
88.7+2.7
93.2+
Arrhythmia
38.8+6.2
37,9+800
37.1+6.8
52.0+2.33
51.6+4..0
45.2+7.6
61.8+188
Breastw
15.347.7
63,8+29.3
93.0=3.7
96.0=0.6
73.549.4
94.8-1.00
96.1=0.7
Pima
40.6.1..33
55.218.0
66.014.1
66.013.11
57.31199
60.2.12.0
59.1122.2
Vowels
33.0.16..4
20,411500
66.21888
31.114.2
78.61229
72.611.55
1.6
Letter
39,0+4.8
31.3+6.5
55.6+3.6
20.7+1.7
53.8+2.2
48.6+3.0
62.8+2.44
Cardio
62.646.1
53,3+12.9
49.8+3.2
78.6-2.5
48.945.8
58,444.8
71.0=2.4
Scismic
17.742.5
17.922.7
19.1=0.9
24.1=1100
18.6+1.9
19.4226
1..7=119
Musk
1.313.3
99.710099
99411.
100.010.0
100.0
100.0
100.010.0
Speech
3.4+2.4
2.1+199
4,3+2.0
4.8+2.3
8.9+2,9
4.442.4
5.2+1.2
Thyroid
68.443.2
69.745.7
72.7+3.1
72.5+2.8
17.2+9.4
32,9+9.9
76.8+1.2
Abalone
44.3217.6
11.6210.5
17.9113
57.6#2.2
6.241.4
6.61100
68.7+2.3
Optidigits
18.415.4
26.5112.6
30,515.2
0.3.0.33
45.812.6
36.5199
10.3
Satimage-2
10,2422.
33.7119.
4.81166
90.710.77
10.4
21.712.
92.400.7
Satellite
61.3++6.3
68.1+0.7
52,2+1.5
64.2+0.4
67.9+2.0
70.1+0.8
73,2+116
Pendigits
7.92299
10.6+7.9
11.0226
40.1=5.0
25.143.6
19.414.5
82.3-4.5
Aunthyroid
63.8+4.77
55.6+5.2
64,243.3
50.3#6.3
61,47788
68.0=3.77
45.41188
Mnist
NIA
NIA
N/A
66.911.3
67.51122
66.2.11.5
8..9
Mammo.
34.1+2.2
315+662
32.6+2.1
33.7+6.1
16.5+1.3
16.0+1.5
29.4+1.4
Shuttle
N/A
N/A
N/A
73.5+5..1
N/A
98.4+0.22
9..
Mullcross
NIA
N/A
N/A
99.7=0.8
N/A
36.4+17.0
100.0+0..0
Forest
NIA
N/A
N/A
0.1:0.2
N/A
15.0.4.33
44.014.1
Kdd
NIA
N/A
N/A
79.613399
N/A
98.4-0022
99.110.1
Kdd-rev
N/A
N/A
N/A
98.0+F..1
98.9+00.3
98.8+10.1
99.2
mean
21.7.14.6
29.6.18.5
33.6.13.3
55.914.2
42.814.8
51.316.3
69.7.12.9
rank 12)
9.1
8.8
8.0
5.3
6.5
5.7
2.6
best performance. While list the KNN results for K 5, which is the best test-set parameter found, KNN outperforms the other baselines for other values of as well, see Appendix
For all cases in which method achieves the highest score (not including the ties at 100%), the pvalue of the Wilcoxon test is below 0.0001 when compared to the next best method.
To further visualize these multiple-henchman profiles, there is a single plot per method, based on the statistics of the performance it obtains compared to the best method for each benchmark. Specifically, the ratio benchmarks for which the method obtains up to fraction of the maximal score obtained by all methods. This is plotted for which means that
results, employ Dolan-More profile. In such
0 0 < A leading method is expected obtain ratio of 1.0 closer obtains, on all datasets, performance that is within relatively narrow multiplicative margin of the best method. dominating method would present a graph with higher values along the y-axis for all 0 values, which means that for every margin, it has more datasets in which it is within margin of the best results than any other method. Fig. presents our results for the Fl score, showing that our method leads by significant margin
the 11 other alternatives.
Interpretability Since we apply sliding window approach and obtain the anomaly score by simple integration,
point back to specific features in the input vector that are overrepresented in factors that indicate the anomaly. This provides a natural explainability capability, which, as far as we ascertain, was not shown for the deep-learning based anomaly detection methods (interpretability can be provided for such methods indirectly, e.g., by tracking network activations (Bach et al. 2015 or linear approximations (Ribeiro al. 2016)).
7

---

Published as a conference paper at ICLR 2022
Table 3: Fl results traditional models. The results reported to value of K that found to provide the highest mean Fl score.
Method
COPOD
IForest
KNN
PIDForest
RRCF
Ours
Wine
60.01:4.5
64.0112.88
94.01:499
50.016.4
69.0111.4
90.011..
Lympho
85.015.0
71.747.6
80.0+11.7
70.011..0
36.7218.0
86.716.0
Glass
11.1=0.0
11.1=0.0
11.1
8.946.0
15.6+13.3
27.2+:00.6
Vertebral
1.7.1.77
13.013..8
10.01:4.5
12.015.2
8.044.88
26.011.77
Wbc
71.410.0
70.013..7
63.8+2.3
65.713.7
54.816.1
67.613.6
Ecoli
25.6+111.2
58.9+:22.2
77.8+3.33
25.6+111.2
28.9
70.0
lonosphere
1.88+
80.8+2.11
88.6+11.6
67.1+3.9
72.0+
93.2+11.
Arthythmia
18.211.4
60.913.3
61.812.2
22.7+2255
50.613.3
61.8188
Breastw
96.4+0.6
97.2+0.55
96.0+0.7
70.6+7.6
63.0+1.8
96.1+0.7
Pima
62.3+-111
69.6+1.2
65.3+110
65.9+2.9
55.4+1.77
59.1+2.2
Vowels
4.8
25.814.7
64.413.7
23.243.2
18.014.6
90.811.
Letter
129++0.7
15.6+3..3
45.0+-2.6
14244233
17.4+2222
62.8+244
Cardio
6..0+114
73.5+4.11
67.6+0.9
43.0+-2.5
43.9+-277
71.0+2.4
Seismic
19.2
73.911.5
30.6+1.4
29.211.6
24.14:3.
20.7
Musk
49.6+1.2
52.0+5..
100.0+0.0
35.4+0.00
38.4+6.5
100.0+0.0
Speech
3.3+0.0
4.94119
5.1+1.0
2.+11.9
3.942.8
5.2+1.2
Thyroid
30.810.5
78.9
57.3+13
72.013.2
31.914.7
76.8+1.2
Abalone
50.3+6.4
53.4+-1.7
43.4+4.8
58.6+-1.6
36.946.4
68.7+22.3
Optidigits
3.40-.33
15.8+4.3
90.0+112
22.5+16.8
1.340.7
66.3++10.1
S
77.910.9
86.5
93.8+1.2
35.5#0.4
47.913.4
92.410.7
Satellite
56.710.2
69.6+0.5
76.3+0.4
46.943.77
55.411.3
73.2+1.
Pendigits
34.910.6
52.116.4
91.0+11
44.61:5.3
16.3+226
82.3+4.5
Annthyroid
31.5+0.5
57.3+1.3
37.8+0.6
65.412.77
32.1+0.8
1.
Mnist
38.510.4
51,24255
69.410.9
32.615.77
33.5+1.7
85.9
Mammo.
53,410.9
39.013.3
38.8+1.5
28.114.3
27.1119
29.4
Shuttle
96.0+0.0
96.4+0.8
97.3+0.2
70.7+11.
32.0+2.2
98.4+0.1
Mullcross
66.010.1
99.1
100.0:0.0
67.412.1
100.0:0..0
100.0
Forest
18.210.2
11.111.
92.110.3
8.142.8
9.911..
44.014.11
Kdd
44.5+:0.1
95.6+22.
98.9+0.44
92.1+2.2
74.7+0099
99.4+0.1
Kdd-rev
30.9
96.412244
85.2+0.33
50.9
9.8
99.2-L0.3
mean
44.7 7.1
58.2+4.0
67.712.2
43.41:..9 7.7
37.0
69.7+22.9
mean rank (out of 12)
4.7
3.8
8.4
2.6
As example, consider the "Thyroid" dataset. In the ODDS repository, the normal class contains samples labeled "normal" or "sub-normal functioning" and the anomalies are from the class "hyper- functioning". Examining the loss of them) reveals that in 83% of the anomalies detected the feature with the highest loss value (implying the lowest correlation with the rest of the vector compared with the training set) was "T3 concentration". in 12% it was "TSH levels" and the rest were "T4U concentration". This makes medical sense since hyperthyroidism is characterized by high T3 and T4U levels, and low TSH.
per each sliding window (and just the mean over all
To examine the ability of the model to identify dependencies, we conducted random data experiment follows: we generated random data for vectors over IR by sampling from Gaussian distribution with means [1, 2, 3, and a covariance matrix that implies a high covariance of 0.85 between the features of indices 2,3, zero covariance elsewhere, and unit variance for all of the features. We also sampled random data that was identical in all parameters except for having no correlation between any of the features. The model (k = 2,r 1) was trained on half the data with correlation between features and tested on the other half with the addition of all non-correlated data. The mean AUC over 10 runs was 88%, which is considerably higher than the baselines, see Appendix (Tab. 10 More interestingly. features 2,3 were highlighted by the model as those with the highest anomaly score. This was inferred by simply aggregating the losses the sliding windows each feature is included in, per sample. In 89% of the samples detected as anomalies, either the second third features were
8

---

Published as a conference paper at ICLR 2022
ranked highest (with almost equal frequency). This is reassuring, since the only difference between the two sampling procedures was in the dependencies between the features.
DISCUSSION
The self-supervised task we seek to solve through the learning of the mappings F and G is relatively simple and he solved without learning in (dd)), since it amounts identifying whether big and af overlap, in which case j' * j. or not, which implies that j' j. Since this decision process is not class-specific, should we be surprised that the learned representations seem to be highly distinctive of class membership? Consider by way of analogy the self-supervised learning of natural images by employing a constrastive loss between an anchor image, its transformation, and another image, e.g., (Chen et al. 2020). Identifying the transformation between two images, which is no more difficult than finding out whether they are related by a transformation, is done reliably with neural networks using point matching (Sarlin et al.) 2020), using "direct methods" (DeTTone et al. 2016). These methods are not class-specific. However, the representations learned by applying geometric transformations extremely distinctive of the class.
To study this further with a random data experiment, in which the vectors x are composed of d independent variables sampled uniformly in 1], we learn networks F and Gi and observe the success in identifying index j. vs. the other indices. The results for = 6, 15, 30 are reported in Fig. 6 As can be seen (our method, all experiments), the loss is being reduced while training. However, the network is not able to immediately obtain perfect performance, especially for larger d. As mentioned in Sec. 3 the architectures of F and G are slightly different, based on the motivation that this would make the bluss-independen solution less accessible during training. As shown in Fig. 6 when replacing the architecture of G to be identical that of F, the class-indepee independenn learning has a lower error on most epochs. Lastly, for data where the features are different from one another the problem becomes less challenging. A second random data experiment sets the mean value of each variable uniformly in [0,1] and the data is i.i.d Gaussian with a unit variance. In this case, the network reaches the same loss in one-tenth of the number of epochs, see Appendix (Fig, D)
Tabular data tends to have considerably higher structural variance between datasets than perceptual data. The type of features (continuous or categorical), the number of features, and dependencies between features vary greatly from one dataset the next. This variance makes the development of generic anomaly detection method challenging, and we find a significant difference in performance across methods. We note, however, that our method is more stable than recent ones, and handles multiple datasets using the same architecture, except for minimal tuning directly related to the dimensionality of the data. For example, on the four datasets commonly used in the literature, the method is applied exactly the same, except for Thyroid, where due to the low input dimensionality (d 6), the value of k = was used. In contrast, GOAD, for example, has dataset-dependent stopping criteria (early stopping or 25 epochs) and employs three architectures on these four datasets.
6 CONCLUSIONS
We present a generic one-class classification method for tabular data. The method assumes that is possible to identify missing features based on the rest and employs contrastive loss for learning without any other auxiliary loss. an extensive set of experiments, the method presents a significant advantage over existing anomaly detection methods. The method requires no tuning between different datasets and is stable with respect to its hyperparameters.
Reproducibility Statement The full implementation of our method and scripts for reproducing the experiments are attached as a supplementary zip file. This archive includes a README file and list of requirements that support seamless reproducibility. The runtime statistics are reported in Appendix G We note that our method requires modest computational resources, such as those available for free Google Colab, further supporting widespread reproducibility. A detailed depiction of the (few) hyperparameters used is given in Sec. A sensitivity analysis presented in Sec. demonsttates that the method is insensitive specific parameter values.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant ERC CoG 725974).
REFERENCES
Dilip Kumar Ahirwar, Sumit Kumar Saxena, and MS Sisodia. Anomaly detection by naive bayes & rbf network. International Journal of Advanced Research in Computer Science and Electronics Engineering, 2012.
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruction probability. Special Lecture on IE, 2(1):1-18, 2015.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0131400 2015.
Liron Bergman and Yedid Hoshen. Classification-bass anomaly detection for general data. arXiv preprint arXiv:2005.023555 2020.
Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander: Lof: identifying density- based local outliers. In ACM sigmod record, volume 29, pp. 93-104, ACM, 2000.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR), 413):15, 2009.
Jinghui Chen, Saket Sathe, Charu Aggarwal, and Deepak Turaga. Outlier detection with autoencoder ensembles. In ICDM, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2005.
Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Deep image homography estimation. arXiv preprint /606.06606.033 2016.
Sarah M Erfani, Sutharshan Rajasegarar, Shanika Karunasekera, and Christopher Leckie. High- dimensional and lcrge-scale anomaly detection using linear one-class svm with deep learning. Pattern Recognition, 58:121-134, 2016.
Tomer Galanti, Sagie Benaim, and Lior Wolf. Risk bounds for unsupervised cross-domain mapping with ipms. arXiv preprint (rXivv/807.08500 2018.
Izhak Golan and Ran Yanya.... Deep anomaly detection using geometric transformations. In Advances in Neural Information Processing Systems, pp. 9758 97699 2018.
Parikshit Gopalan, Vatsal Sharan, and Udi Wieder. Pidforest: anomaly detection via partial identifi-
cation. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 15809-15819, 2019.
Sachin Goyal, Aditi Raghunathan, Moksh Jain, Harsha Vardhan Simhadri, and Prateek Jain. DROCC: Deep robust one-class classification. In International Conference on Machine Learning, pp- 3711-3721. PMLR, 2020.
Sudipto Guha, Nina Mishra, Gourav Roy, and Okke Schrijvers. Robust random cut forest based anomaly detection on streams. In International conference on machine learning, pp. 2712-2211. PMLR, 2016.
10

---

Published as a conference paper at ICLR 2022
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2010.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 066), volume 2. pp. 1735-1742. IEEE, 2006.
Simon Hawkins, Hongxing He, Graham Williams, and Rohan Baxter. Outlier detection using replicator neural networks. In International Conference on Data Warehousing and Knowledge Discovery, pp. 180. Springer, 2002.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, pp. 448-456, 2015.
lan T Jolliffe. Principal components in regression analysis. In Principal component analysis, pp- 129-155. Springer, 1986.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: wwtp.:4422.9900 69800, 2014.
Zheng Li, Yue Zhao, Nicola Botta, Cezar Ionescu, and Xiyang Hu. Copod: Copula-based outlier detection. In IEEE International Conference Data Mining (ICDM). IEEE, 2020.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining, pp. 413-422. IEEE, 2008.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pp. 3111-3119, 2013.
Ishan Misra and Laurens der Maaten. Self-supervised learning of pretext-invariant rep.araarrrttrreeeeeetaattotoo arXiv preprint 19/2.0/ 2019.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and superviee learning. IEEE transactions on pattern analysis and machine intelligence, 11(8):1979-199 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXiv:1807.03744 2018
Guansong Pang, Anton van den Hengel, Chunhua Shen, and Longbing Cao. Deep reinforcement learning for unknown anomaly detection, 2020.
Emanuel Parzen. On estimation of a probability density function and mode. The annals of mathemat- ical statistics, 33(3): 1962.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544 2016.
Chen Qiu, Timo Pfrommer, Marius Kloft, Stephan Mandt, and Maja Rudolph. Neural transformation learning for deep anomaly detection beyond images. arXiv preprint arXiv:2103. 16440, 2021.
Marco Ribeiro, S Singh, and C Guestrin. Lime: Explaining the predictions of any machine learning classifier. Github Repository https://github. com/marcotcr/lime, 2016.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In ICMI, 2018.
11

---

Published as a conference paper at ICLR 2022
Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and -lauu-Robbrt Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 2021.
Mayu Sakurada and Takehisa Yairi. Anomaly detection using autoencoders with nonlinear dimen- sionality reduction. In Proceedings of the MLSDA 2014 2nd Workshop Machine Learning for Sensory Data Analysis, pp- 4. ACM, 2014.
Paul-Edouard Sarlin, Daniel Denone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4938-4947, 2020.
Thomas Schlegl, Philipp Sceböck, Sebastian M Waldstein. Ursula Schmidt-Erfurry and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International conference on information processing in medical imaging, pp. 146-157. Springer, 2017.
Bernhard Schölkopf, Robert C Williamson, Alex Smola, John Shawe-Taylon and John Platt. Support vector method for novelty detection. Advances in neural information processing systems, 12: 582-588, 1999.
Kihyuk Sohn, Chun-Liang Li, Jinsung Yoon, Minho Jin, and Tomas Pfister. Learning and evaluat- ing representations for deep one-class classification. In International Conference Learning Representations, 2021. URL
net/ forum (i-oummPtt
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi:
Novelty detection via contrastive learning on distributionally shifted instances. In 34th Conference on Neural Information Processing Systems (NeurIPS) 2020. Neural Information Processing Systems, 2020.
Yaniv Taigman, Ming Yang, Marc' Aurelio Ranzato, and Lior Wolf. Web-scale training for face identification. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2746-2754, 2015.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In IEEE Conference on Computer Vision and Pattern 2018.
Recognition (CVPR), 2018
Y. Xia, X. Cao, F. Wen, G. Hua, J.d outlier removal. 2015 IEEE International Conference on Computer Vision (ICCV), 2015.
J. Sun. Learning discriminative reconstructions for unsupervised
Bing Xu. Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv: 00853,, 2015a.
Dan Xu, Elisa Ricci, Yan Yan, Jingkuan Song, and Nicu Sebe. Learning deep representations of appearance and motion for anomalous event detection. arXiv preprint arXiv:1510.01553. 2015b.
Yue Zhao, Zain Nasrullah, and Zheng Li. Pyod: A python toolbox for scalable outlier detection.
Journal of Machine Learning Research, 20(96):1-7, 2019. URL URL
v20/ 1 9- 011 html
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International Conference on Learning Representations, 2018
A VALIDATING THE BASIC ASSUMPTIONS
We validate the ability to learn to match between (a,, b() against options of the form (af b/) for j by training networks F, G and measuring the log- likelikoodd of the correct answer (Eq, I for samples from the normal class. We then repeat this for oft-of-class samples to validate that the learned networks are class dependent.
12

---

Published as a conference paper at ICLR 2022
As can be seen in Fig. 3 and for the vast majority of the Outlier Detection DataSets (ODDS, StSSSttt::////o /ddss CS stonybrook. edu/ accessed May 2021), for the normal class the log likelihood Is much higher than chance (assumption 1). It is also clear that it is almost always much higher than that of the out-of-class samples (assumption 2), even though the log likelihood for the out-of-class samples is sometimes higher than chance.
point datasets" from
We note that where these assumptions do not hold, e.g.. for Pima and Vertebral datasets, where the second assumption does hold, the success rate of our method is relatively low (Tab. 2
The correlation between the success in detecting anomalies and the classification score is presented in Fig. 5 Presented are both the classification log likelihood (assumption as well the difference in log likelihood between in-class and out-of-class samples, Evidently, the negative log log-likelihood score of assumption is low in almost all cases, but is not highly correlated with the Fl score (r -0.1608, p 0.3959). The difference in classification success that is associated with assumption 2 is, however, highly correlated with the identification performance (r = 0.75688, p = 1.3 10 -9)
We use the same experiments to also check the soft assumption that the original order may be, in random permutation of the features. As can be seen, there is no clear advantage to the original order of the features. In the Mulcross dataset, for example, the permuted variation leads to a stronger distinction between the normal and the anomalous class. The lack of a clear advantage for the original order is further supported by the experimental results in Sec. which show that in 50% of the datasets the original order preferable and in the rest the permuted order is.
some cases, preferable
Finally, we explore the counterexample presented in the limitations part of Sec. using the specific parameters for which the normal and the out-of-class samples cannot be distinguished. As can be seen in Fig. T (bottom right), in this case the first assumption holds, but the second one does not. This leads to an inability to identify abnormal samples.
B ODDS DATASET STATISTICS
Tab. 5 presents the number of samples, the dimensionality, and the number of samples not from the main class for the various datasets that belong the collection "Multi dimtii-dimensoonal point datasets" from the Outlier Detection DataSets (ODDS, (ODDS./ttp:///ddee cs. www.sstooooooc.ooooooooooo edu/ accessed May 2021).
C ADDITIONAL RESULTS
In Tab, we present results for the second set of experiments, this time reporting AUC (the manuscript reported the Fl scores in Tab. 2.).
In Fig. 7 we present the results of parameter sensitivity study performed on four datasets. The results indicate that the method is largely insensitive to the parameters: the sliding window size k. the dimensionality of the latent space u, the softmax temperature т, and the number of random permutations each experiment, the tested parameter was varied, while all other values were default.
In Fig. we show the results for varying in the range [50. for the Arrhythmia dataset. For each I value, the reported results are averages over 10 runs. The best F1 score was obtained at 63 (the k value selected at our default settings for this dataset is 124), but all values were relatively similar and between 58.59% and 67. 2% (the AUC values were in the range (0.%%)84%%) For all k, the Fl and AUC are better than all baselines.
In Tab. we we show the results for sensitivity test, in which set the hyperparameter & as I for all datasets, compared to the original results reported in the paper. 30% of the experiments, this leads to better performance than the k used according to the default setting. The mean F1 is lower by 2.6% (AUC lower by 1.7%). still outperforming all baselines.
Fig. 9 shows the results for the random data experiment, where features sampled independently from a Gaussian distribution with a different mean per feature, sampled from U[O.1]. These runs are compared to results where the features are sampled from U[-1,1] Also, for each distribution
13

---

Published as a conference paper at ICLR 2022
00-
10-
Roxal
-33
-00
-25
Diger
orgee
Pereiee
Haxal
Testure
Oen
Inarghas
-31
Testuw riter
Weww:
labe
AB2705
01-
Noray
10
-30
Festu
lemphe
Varrograpty
05
Nermel
Voree
Herei
23
RETRURKS
-35
Festures
Testure
Fenkurs
The negative log likelihood (Eq. 1 for matching between the sub-vectors (a. 6.) with networks F, G for multiple ODDS datasets. The results are shown in each plot, from left right, for in-class samples using the original order of vector elements, out-of-class samples using the same order, in-class samples with a fixed-permutation order features, out-of-class samples with this permuted order. The samples in the box plots were collected across all test samples and segment indices j. without summing over j. The horizontal line indicates the negative log likelihood of random guess. (Part 1/2, continued on the next page)
Figure 3:
14

---

Published as a conference paper at ICLR 2022
00-
Aumeyy
+10
+13
-60
restures cole!
Fosturss onder
Speech
110
Fasture axde
Testues cilor
Festuresooee
Therst
Nexal
01yee
06-
Testume
Figure 4: (Part 2/2; see Fig for details). Bottom right: results on synthetic dataset that was created to demonstrate case, for specific k; r, in which learning is possible but not in a class dependent manner.
15

---

Published as a conference paper at ICLR 2022
Normal
Difference between classes
1
Log10 NLL
0
-1
-2
20
40
60
80
100
Figure 5: The anomaly detection performance (Fl score, x-axis) vs. the classification performance on the normal class(log likelihood, assumption 1) and the difference in performance between the normal and the anomaly class (assumption 2).
16

---

Published as a conference paper at ICLR 2022
show the results for two variations of our network one with the default architecture, and one where we change the network so it is similar to G (i.e. without TanH activation).
As can be seen, for data sampled from Gaussian distribution the network converges considerably faster, which suggests that solving the caass-indeeendent task is easier when the features have distinctive differences. Also, in both variations of the data distribution, the TanH architecture variant requires more epochs to converge to similar values, supporting our hypothesis that the design of our architecture makes harder for the network to learn the class-independent solution.
In Tab. 8 we report the results for running different variations of our architectures: a variant with a single hidden layer for both F and G, one with extra layer for both, and the default architecture.
In Tab. 9 we report results for a test designed for studying whether the order of the features effects the results of the model. To this end, we measured the performance after employing single fixed permutation on the datasets, compared to not employing any permutation at all (the original order as provided by the dataset). Permuting the features had minor effect, and across the collection of datasets, there is no clear advantage to the original order.
Tab. reports the results of the experiment in which both the normal and the out-of-class distributions are sampled from 4D Gaussians with a mean vector of [1, 2, 3, 4] and per-variable variance of one. The in-class samples where sampled from convariance matrix with a correlation of 0.85 between variables two and three, but uncorrelated otherwise. For the out-of-class samples, the covariance matrix was diagonal. This is a challenging problem, since variables two and three can appear to be similar even for an out-of-class sample. As can be seen, our method greatly outperforms the baseline methods.
The KNN baseline, while being the simplest baseline, experiments of the paper we report results for the highest performing parameter, which is K = neighbours. The mean F1 score for various values of K tested are: 67.1, 66.3, 67.7, 66.8, 66.4 for K of 1 1,0,5,10,20,, respectively.
shown to be the strongest one. In the
D ABLATIONS AND PARAMETER SENSITIVITY
We selected four representative datasets ('Wine', 'Glass', 'Thyroid, 'Letter'), which vary in the number of dimensions, the number of samples, and the performance level, and ran an ablation analysis on them. The variants we compare include: (i) a variant of our method, in which the tanh activation of the first layer of F is replaced by LeakyRELU, (ii) variant in which only the first of the two normalizations of the query and vectors that F and G output takes place (iii) a variant in which only the second normalization takes place, i.e, normalization occurs in the conventional way, (iv) a variant in which the order of the two normalizations is reversed, and (v) a variant with no normalization, i.e., F, G are trained and used instead of W and G...
The results are reported in Tab. 4 As can be seen, the lanh activation in F improves results, to varying degree, on the four datasets. Normalization tends to help across datasets. However, on Thyroid. applying no normalization at all provides better results. Omitting only the 2nd normalization, which is often used in contrastive learning, detrimental in all cases. Omitting the 1st normalization, while keeping the 2nd, also hurts performance. Reversing the order of the two normalizations, which changes the scales of the features before this scale is normalized for, is detrimental.
On the same four datasets we also evaluate the sensitivity of the method to its hyperparameters: k, u. T, and r. In each experiment, we fix three parameters and vary the fourth. The results, provided in the supplementary Fig. 7 of appendix C indicate that the method is largely insensitive to its parameters. This robustness is further supported by using the same hyperparameters on a large number of datasets.
To further verify that the method works for any k. we tested for Arrhythmia, which has a dimen- sionality of 274, values & between and 150. The best Fl score was obtained at 63 (in the experiments it is 124), but all values were relatively similar and between 58.59% and 67.2% (the AUC values were in the range 80.7%-84. 84.3%). For all k the Fl and AUC are better than all baselines, Appendix for the full results. Further evidence of the robustness of the k parameter, which is the main hyperparameter, since we fix u = 200 and 7 = 0.01, is provided by simply using & = 1 across all ODDS datasets. In 30% of the experiments, this leads to better performance than the k
17

---

Published as a conference paper at ICLR 2022
Table 4: Ablation results (F1 in percents and standard deviation multiple resampling).
Wine
Glass
Thyroid
Letter
Variant
Fi Score
SD
Fi Score
SD
Fi Score
SD
Fi Score
SD
Ours
90.0
6.3
28.9
10.6
76.8
1.2
62.8
2.4
(1) No tanh
70.0
7.7
15.6
5.4
76.1
3.8
54.4
3.0
(ii) Only 1st
85.0
9.2
21.1
0.07
68.7
0.02
52.2
4.7
(iii) Only 2nd norm
87.0
7.8
25.6
10.0
76.5
1.8
57.2
6.6
(iv) 2nd norm then 1st
84.0
4.8
22.2
11.0
68.4
2.2
57.1
3.0
(v) No normalization
87.0
6.4
21.1
10.5
78.1
1.8
51.0
7.2
Go the sento a ardidddarr F
l the sers andiacaars
G the suno ahdieddrr
0.5
Looe
0.1
3.25
Epoch
Epoch
(a)
(b)
(c)
Figure 6: Convergence of the loss completely random data, in order evaluate the ability perform class-independent learning. (a) d = 6. (b) d = 15. (c) = 30.
used according to Sec. 3 The mean F1 lower by 2.6% (AUC lower by 1.7%), still outperforming all baselines, see Appendix for the full results. Note that in this case h << u.
We further investigate the sensitivity of our method the network architecture. When changing both F and G, variant with only one hidden layer obtained, average ODDS, an Fl score that is lower by 2.8% (AUC by 1%; still much better than all baselines in both scores), and a variant with an extra hidden layer in both had mean Fl that is lower 10.2% (AUC difference of 4%) than the default architccture. See Appendix C for the full tables and Appendix H for a discussion.
As described in Sec. the score is computed multiple times after permuting the features, and the number of repeats (r) depends on the dimensionality d and number of samples n. In Fig. dd) we present the effect of the number of repeats performance for multiple relatively small datasets. at different performance levels. Shown are the mean Fl over 10 runs and also, as error bars, the SD. We observe that adding repeats typically helps, albeit modestly. also tends reduce the variance between runs. A drop in performance between the first and the second repeats may indicate that the order of features informative. However, this does happen often.
To study the potential benefit that our method obtains from the particular order of the features in which the data given, we have compared results (with permutations) those obtained when employing a single, fixed permutation. The average F1 score our method obtains across the entire ODDS collection is 64.1%, and after permutation 64.4% (mean over all datasets, with 10 repeats). This trend, however, is dominated by a few datasets, and in AUC terms, the results are 87.2% vs. 87.1% 9, where the original order leads. both AUC and the original order leads in the same 50% of the datasets. The lack of clear advantage for the original order is also apparent in the auxiliary results of Appendix A
The same experiment also verifies that our improved performance does not arise solely from the bagging effect of having r > 1. Comparing the case of r = 1 to the baselines reveals that our average Fl score of 64.1% (AUC 87.2% (%%) is higher by sizable margin then all of GOAD's architectures, which have a mean of $1.3-55,9%% (depending on the architecture; AUC 79.6 79.6-81.09 81.0%) and COPOD's 44.7% mean F1 score (AUC 79.1%)
18

---

Published as a conference paper at ICLR 2022
E LIMITATIONS
In some symmetric cases, for specific parameters and with no permutation, our method would fail in the task of anomaly detection. Specifically, any scenario in which for both normal and anomalous data the pairs constructed from each instance {(o by 1- are identical would be impossible for the network to differentiate between. For example normal data of the form (0, (0,1,0,0.0) and anomalous data See Appendix [A for
10,0,0,1,0) with and r violates the second assumption listed Sec. T simulations of this example.
F AN ALTERNATIVE METHOD (REPORTING NEGATIVE RESULT)
We briefly summarize alternative method that tried, which failed to produce the results hoped for. Earlier in the development process, we had an intuition that making a comparison with external samples might be beneficial for our model, and experimented with drawing the negative samples from other instances, i.e. instead of using:
q = F(b),, w+ = -G(a)), G(a!")
(2)
for some i,j.J' A j. we used:
q q= F u+ = G(d)), = G(a))
(3)
for i, * i,j. The results obtained by employing this method on the ODDS repository were significantly lower in terms of F1 and AUC performance almost all datasets, and were not competitive with the baseline methods.
G RUNTIME
In Tab. I1 we show the average runtime in seconds for each of the datasets. We include two versions: one with the default amount repeats fauf 1 + [100(log(n) + dd)-j and another one with faster = minn2, T deutt)))).
We observe that the difference in performance between the two versions typically minor, and the faster method might be preferred if runtime is a top priority. The code includes an option for running the model in the faster mode.
The google colab infrastructure was used to run the experiments. GPU: Tesla K80, 12GB GDDR5 VRAM: CPU: Single core Xeon Processors @2.3Ghz; RAM: 24 GB. For the GOAD baseline that required larger memory, used 32GB GPU and 512GB RAM.
H DISCUSSION OF ALTERNATIVE ARCHITECTURES
The architecture we employ for F and straightforward fully connected architecture with two hidden layers. One can hypothesize that with less capacity, the network may not be able to learn the required mappings as effectively, making it incapable of utilizing the first assumption. However, with added capacity, the network may shift from the class specific solution to generic solution, validating our 2nd assumption. See Galanti et al. (2018) for a remotely related phenomenon in the completely different unsupervised learning task of mapping between two visual domains: with too few layers the mapping fails to produce samples in the target domain, for too many layers, the mappings are not specific to the input image.
The hypothesis is partly validated empirically in the results reported in Appendix. C Tab. g A variant with only one hidden layer obtained only 1% less mean AUC (still better by gap than all baselines) on the ODDS benchmark than our original results, and a variant with an extra hidden layer had an AUC 4% lower.
In order to study whether this drop in performance is due to assumption 2, we examine the average log likelihood. The mean loss for the "normals" category is 0.016 and 0.014 0014 for the original architechture
19

---

Published as a conference paper at ICLR 2022
and one extra layer respectively, and the mean anomaly class losses are 1.54 and 0.96. We, therefore, observe that the ability to solve the problem (assumption 1) is almost the same and slightly improves with the added capacity. However, the gap between the normal class and the outlier samples has shrunk dramatically with the extra layer.
Table The numher of samples, the dimensionality, and the number of samples not from the main class for the datasets used in the second set of experiments.
Dataset
d
Outliers
Wine
129
13
10 (7.7%)
Lympho
148
(4.1%)
Glass
214
9 (4.2%)
Vertebral
240
30 (12.5%)
WBC
278
30
21 (5.6%)
Ecoli
336
9 (2.6%)
Jonosphere
351
33
126 (36%)
Arrhythmia
452
274
66 (15%)
BreastW
683
239 (35%)
Pima
768
8
268 (35%)
Vowels
1456
12
50 (3.4%%
Letter Recognition
1600
32
100 (6.25%)
Cardio
1831
176 (9.6%)
Seismic
2584
11
170 (6.5%)
Musk
3062
166
97 (3.2%)
Speech
3686
400
61 (1.65%)
Thyroid
3772
93 (2.5%)
Abalone
4177
29 (0.69%)
Optdigits
5216
64
150 (3%)
Satimage-2
5803
36
(1.2%)
Satellite
6435
36
2036 (32%)
Pendigits
6870
16
156 (2.27%)
Annthyroid
7200
534 (7.42%)
Mnist
7603
100
700 (9.2%)
Mammography
11183
260 (2.32%)
Shuttle
49097
3511 (7%)
KDDCUP99-Res
121597
120
24319 (20%)
Mulcross
262144
26214 (10%)
ForestCover
286048
10
2747 (0.9%)
KDDCUP99
494021
120
97277 (19.6%)
20

---

Published as a conference paper at ICLR 2022
Table 6: AUC results for the ODDS benchmarks. In order to avoid selecting the architecture for the baseline methods, we used available versions these methods. The missing DROCC experiments are due to the limitations of the published code.
Method
DROCC (Thyroid)
DROCC (Arrhtthmia)
DROCC (Abalone)
GOAD (Thyroid)
GOAD (kddrev)
GOAD (kdd)
COPOD
Ours
Wine
53.5-22.3
60.1132.3
90.918.2
95.2.11.
97.311.7
86.319,5
87,5
99.510066
Lympho
6,4=5.2
58.6=30.4
83.7+12.4
94,825.6
79.7=11.1
59.911499
99.420.4
99.5=0.3
Glass
63.5+99.1
55.5=21.8
75.4+8.9
62.24:1..0
85.5+7.0
82.1-6.3
63.743..
88.1=5.0
Vertebral
55.015..1
58.0115.4
41.2.10.1
47.01112.8
52.213.9
49.411422
32.611.2
51.113.2
WBC
6.8=1.8
41.3125.0
33.4
95.420.77
66.1=11.5
86.6:1299
96.31005
95.421.1
Bcoli
N/A
N/A
N/A
8..711..4
87.2.13.3
84.7.16.8
81.0.11.2
86.511.2
lonosphere
19.6.15.8
83.515..6
80.0.1..8
92,411.3
96.3111.
96.111.1
80.312.1
98.110.4
Arrhythmia
53.2:7..0
52.718.6
51.248.11
80.021.9
73.345.11
64.328.8
80.5
81.720.6
Breastw
7.7-8.6
64.4133.0
96.61..33
98.71.088
80.819.5
97.7-1088
99,4-10.2
99.1
Pima
36.2.14.6
5491110
69.114.9
68.7.3399
59.312.2
63.2.1.3
65.2.10.7
59.412.8
Vowels
79.4+995
72.0+119
95.3+2.1
81.0+2.4
98.5+0033
97.6+00.5
49.6+1.0
99.7+0.1
Letter
77.6+:3.3
73.3+-5.4
90.0++1.2
60.9+0077
89.9++0.5
87.6+0.99
50.1
92.8
Cardio
84.3.1.00
73881188
73.5.3..2
94,811.7
81.3.1.55
84.611..0
92.210.33
92.7
Scismic
58.24-2.8
60.3++4.5
56.7+1.3
69.5+1.5
67.2+1.2
67.9+112
70.8+0.4
62.9+1.0 1.0
Musk
2.3-5.1
100.0+0.0
100.0+0.0
100.0+0.0
100.0+0.00
100.0+0.0
94.5++0.2
100.0+0.00
Speech
51.24:5.6
50.5+:4.0
52.6434
47.1=1.3
65,343.2
54.1=4.4
49.1
58.9+2.7
Thyroid
95.6++0.9
96.1+:2.5
98.1+
94.5
77.1+8.8
89.2+3.00
94.1
08.5=0.1
Abulone
82.4-13.8
52.9125.8
70.619.7
89,210.9
46.013.7
54.317.8
86,3103
94.310.6
Optidigits
84.2:t4.6
89.0+:466
89.5+2.1
66.923.3
95.740055
93.1=119
68.0=0.4
97.5=115
Satimage
19.1+114
87.5+8.88
11.5+112
99.1+0.1
86.5+7.11
93.2+1177
97.4+0.1
99.8=0.1
Satellite
64.6.18.9
73.11:1.
50.242.2
69.110.8
76,3110
78.2.10.
63.510.2
80.6
Pendigits
58.947..6
50.8-15.4
76.645.
87.5+3.9
89.242.9
85.123.4
90.420.2
99.5=0.1
Annthyroid
92.9112.3
86.5.13.6
93411.
89.6.14.9
93.2.10.9
77.410.4
80.
MNIST
N/A
N/A
N/A
90,910.4
89.4.00.7
87.71100
77.21002
98.210.00
Mammography
81.0+:1.
85.0#2.1
82001
66.326.4
57.2+199
54.5#2.3
90.5
81.1=2.0
Mulcross
N/A
N/A
NIA
100.0.0.00
N/A
51.315588
93.2.10.0
100.0.10.0
Shuttle
N/A
NAA
N/A
88,415,5
N/A
99.910.00
99.41000
100.0.10.0
Forest
N/A
15.9+6.6
N/A
76.0+5.3
88.++0.0
96,2+0.6
KDD
N/A
92.81100
99,81000
78.610..1
99,910.0
KDD-Rev
N/A
N/A
N/A
N/A
N/A
99.710.2
75.110.2
99.810.1
21

---

Published as a conference paper at ICLR 2022
Table 7: Fl and AUC results for l 1 and the default settings of mentioned in the paper
F1
AUC
Method
k
k
de
k faulu
Wine
88.0
90.0
99.3
99.5
Lympho
86.7
86.7
99.5
99.5
Glass
22.2
27.2
87.3
88.1
Vertebral
28.0
26.0
52.4
51.1
WBC
71.4
67.6
95.6
95.4
Ecoli
62.2
70.0
86.0
86.5
Ionosphere
93.2
93.2
97.8
98.1
Arrhythmia
59.7
61.8
81.0
81.7
Breastw
94.8
96.1
98.4
99.1
Pima
60.0
59.1
60.5
59.4
Vowels
86.4
90.8
99.5
99.7
Letter
56.8
62.8
89.5
92.8
Cardio
72.0
71.0
92.1
92.7
Seismic
23.4
20.7
64.9
62.9
Musk
100.0
100.0
100.0
100.0
Speech
3.6
5.2
54.2
58.9
Thyroid
78.1
76.8
98.1
98.5
Abalone
69.7
68.7
93.1
94.3
Optidigits
37.9
66.3
92.3
97.5
Satimage
91.8
92.4
99.7
99.8
Satellite
72.6
73.2
81.1
80.6
Pendigits
71.0
82.3
98.7
99.5
Annthyroid
45.5
45.4
79.6
80.5
MNIST
82.0
85.9
96.4
98.2
Mammography
28.9
29.4
81.5
81.1
Shuttle
98.3
98.4
100.0
100.0
KDD-Rec Rev
99.0
99.2
99.8
99.8
Mulcross
100.0
100.0
100.0
100.0
Forest
28.3
44.0
86.6
96.2
KDD
98.6
99.4
99.9
99.9
mean
67.0
69.6
88.8
89.7
22

---

Published as a conference paper at ICLR 2022
Table F1 and AUC results for variations the architecture: Using only one hidden layer, adding extra hidden layer and the default architecture
F1
AUC
Method
Single- layer
Extra- layer
Default
Single- layer
Extra- layer
Default
Wine
74.0
88.0
90.0
96.8
98.3
99.5
Lympho
86.7
76.7
86.7
99.5
98.6
99.5
Glass
17.8
13.3
27.2
84.2
82.0
88.1
Vertebral
16.7
30.0
26.0
48.3
59.1
51.1
WBC
69.5
67.6
67.6
94.8
94.4
95.4
Ecoli
57.8
62.2
70.0
86.6
87.4
86.5
Ionosphere
94.4
91.6
93.2
98.3
97.3
98.1
Arrhythmia
62.1
62.7
61.8
80.9
81.5
81.7
Breastw
95,4
95.6
96.1
98.6
98.6
99.1
Pima
59.5
57.1
59.1
60.1
$7.2
59.4
Vowels
90.0
73.6
90.8
99.6
97.8
99.7
Letter
51.2
28.2
62.8
87.0
70.1
92.8
Cardio
66.6
78.4
71.0
90.3
96.5
92.7
Seismic
23.2
19.8
20.7
61.0
63.3
62.9
Musk
100.0
67.0
100.0
100.0
96.8
100.0
Speech
4.9
2.3
5.2
53.3
45.2
58.9
Thyroid
73.3
58.1
76.8
97.5
96.8
98.5
Abalone
68.3
56.6
68.7
94.2
92.6
94.3
Optidigits
47.1
2.7
66.3
94.0
72.0
97.5
Satimage
91.0
91.0
92.4
99.7
99.7
99.8
Satellite
73.3
69.2
73.2
79.5
75.8
80.6
Pendigits
85.1
54.6
82.3
99.6
94.2
99.5
Annthyroid
43.7
43.6
45.4
79.7
78.0
80.5
MNIST
82.3
52.4
85.9
96.7
86.1
98.2
Mammography
31.7
27.7
29.4
81.1
68.4
81.1
Shuttle
98.0
98.1
98.4
99.9
100.0
100.0
KeV-Rev
95.0
96.5
99.2
98.4
98.9
99.8
Muleross
100.0
94.1
100.0
100.0
99.3
100.0
Forest
49.2
13.6
44.0
97.0
83.5
96.2
KDD
97.4
99.1
99.4
99.8
100.0
99.9
mean
66.8
59.0
69.6
88.6
85,6
89.7
23

---

Published as a conference paper at ICLR 2022
Table 9: Fl and AUC results for: no permutations at all (r 0) and for one fixed permutation (r = 1)
F1
AUC
Method
7 ==
F = 1
r ==
7
Wine
74.0
73.2
95.1
94.7
Lympho
85.0
82.8
99.4
99.1
Glass
12.2
22.9
74.5
81.5
Vertebral
14.3
14.2
52.6
42.3
WBC
63.3
65.3
94.1
93.5
Ecoli
66.7
57.8
87.6
86.4
Ionosphere
90.6
92.8
96.5
97.8
Arrhythmia
62.4
61.3
80.4
81.0
Breastw
95.1
94.5
98.5
98.1
Pima
59.3
58.5
59.2
58.8
Vowels
80.4
79.9
98.4
98.4
Letter
49.6
55.3
88.7
89.2
Cardio
66.8
67.0
89.7
89.8
Seismic
20.8
19.8
62.6
60.1
Musk
100.0
100.0
100.0
100.0
Speech
3.6
2.0
52.3
55.5
Thyroid
75.5
69.8
98.3
96.4
Abalone
61.4
64.1
88.5
92.2
Optidigits
68.5
55.8
97.6
95.9
Satimage
90.6
91.7
99.7
99.8
Satellite
73.8
73.3
83.3
80.6
Pendigits
69.8
69.8
97.6
97.7
Aunthyroid
41.9
44.6
72.9
79.8
MNIST
83.7
84.8
97.3
97.9
Mammography
21.9
24.8
75.9
73.7
Shuttle
97.7
98.0
99.9
99.9
KDD-Rev
99.2
99.2
99.9
99.8
Muleross
73.5
75.8
90.5
86.7
Forest
23.5
35.1
86.2
89.5
KDD
98.9
98.3
99.9
99.9
mean
64.1
64.4
87.2
87.2
Table 10: AUC results for synthetic data experiment with 4D data that has the same class for both classes, only that the normal class variables two and three are correlated. Shown are mean results and Standard Deviations over 10 repeats.
Method
AUC (percent; Mean + SD)
COPOD Li et al. 2020)
52.2 + 3.2
Isolation
Forest Liu et al.
2008
62.3 4.6
OC-SVM (cchökkopt al. 1999)
66.8 + 1.1
GOAD KDD architecture [Bergman Hoshen 2020
74.0 + 3.2
GOAD Thyroid architecture (Bergman & Hoshen 2020
73.7 1.3
Ours
88.1 + 1.5
24

---

Published as a conference paper at ICLR 2022
0.8
0.8
0.6
0.6
msen
Glass
Thyraid
0.4
Glass
Thyroid
Letter
Lettor
0.2
2
3
50
100
150
200
250
300
K
(a)
(b)
0.0
0.8
08
0.7
0.6
Vertebral
Wire
korossh proosshrr
Glass
Breaste
Thyroid
0.4
0.3
0.2
0.2
0.4
0.6
0.8
3
10
Aumber perm Luatioons
(c)
(d)
Figure 7: Sensitivity with respect to (a) k, (b) (c) т, and (d) the number of random permutations All other values taken the default value.
1.0
1.0
www.
0.0
(a)
(b)
Figure Varying the Arrhythmia dataset. (a) results (b) AUC results.
25

---

Published as a conference paper at ICLR 2022
15
Unioorm our archieettture
10
100
Unfform similrr to
Garooiat- that archiistuu
13P
Gousston G to
107
10
10
1015
10
10
10
(a)
(b)
(c)
Figure 9: Convergence of the loss on random data, sampled from uniform and Gaussian distributions.
(a) 6. (b) 15. (c) d 30.
Table 11: Runtime in seconds and F1-Score for fdefault = 1 + min(2, Idefoutt)
[100(log(n) + +d)
and faster
Dataset
fadf runtime
ffaster runtime
fdefautt F1
ffattrr F1
wine
21.1
8.0
90.0%
73.0%
lympho
35.5
15.0
86.7%
83.3%
glass
24.9
5.2
27.2%
25.6%
vertebral
30.3
5.8
26.0%
17.0%
wbc
33.2
22.7
67.6%
66.7%
ecoli
35.5
8.4
70.0%
66.7%
ionosphere
28.0
18.8
93.2%
92.5%
arrhythmia
33.5
same
61.8%
same
breastw
38.9
12.2
96.1%
94.6%
pima
27.1
9.8
59.1%
58.2%
vowels
83.7
28.0
90.8%
86.6%
abalone
44.4
8.4
68.7%
66.2%
letter
107.6
72.5
62.8%
59.5%
cardio
105.9
52.8
71.0%
67.8%
seismic
159,9
79.2
20.7%
21.0%
musk
381.3
100.0%
speech
662.4
same
5.2%
same
thyroid
76.7
17.1
76.8%
73.1% %
optdigits
332.5
same
66.3%
same
satimage
430.4
286.8
92.4%
92.1% 1%
satellite
336.7
222.3
73.2%
74.7%
pendigits
224.1
89.9
82.3%
76.3%
annthyroid
80.2
21.7
45,4%
43.5%
mnist
223.1
same
85.9%
same
mammography
150.8
42.5
29.4%
24.2%
shuttle
212.4
89.1
98.4%
same
forest cover
240.3
107.9
44.0%
37.5%
mullcross
279.5
43.6
100.0%
99.0%
kdd
1674.1
same
99,4%
same
kddrev
1771.3
same
99.2%
same
26


Reviews:
[{"review_id": "qIEC3wdnP0p", "reviewer": "Reviewer_iWxT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am very surprised this method performs so well, since this k-sliding window does not seem to make sense for tabular data since heterogeneous types of features exist (e.g. discrete, continuous or skewed distribution) and the order of features should not matter and thus the sliding-window approach should not work unlike images. But my concern is much relieved after seeing the complete set of experiments and the stability analysis, as well as some limitations the authors admit in the discussion.\n\nMy concerns are as follows (ordered by high to low concerns):\n1. Since it's in the tabular data, I hope the authors can compare with long-lasted traditional baselines such as RRCF and KNN in ODDS benchmark (authors can use the default hyperparameter in the sklearn). \n2. The claimed that default set of hyperparameters achieve good performance is a bit exaggerated. A better way to phrase is a default \"rule\" of hyperparameter setting. Because this method still tunes hyperparameters but based on some rules-of-thumbs like the model stops when loss < 10^{-3} when d < 40 else stops when loss < 0.01, or k has to set to d - 150 when k > 150 etc. Overall I still like the stability of this method.\n3. I am wondering if authors try adding the location index into the inputs of the model? This might help improve the model to learn better representation based on index j without learning independent model for each j.\n4. The Dolan-More profile should be described more clearly. The text is quite confusing right now.\n5. In the result of Arrhythmia, the claim is the best F1 score is 63 but all values are between 58.59% to 67.3%. Isn't the best score 67.3?\n\n", "summary_of_the_paper": "## After rebuttal\nI am impressed by the comprehensive experiments and baselines and increase my score. Authors provide very complete baselines and also compare in other datasets, which greatly reduces the concern that the proposed rule overfits to the ODDS benchmark.\n\n\n## Summary\nThis paper learns a contrastive representation that helps differentiate between normal and abnormal data in the tabular data. Contrary to other contrastive learning methods which learns to differentiate between different examples, this paper instead differentiates between in-window vector and out-window vector for each example under a sliding k-sized window (k is a hyperparameter). Since the features are unordered, they shuffle the feature orders to get multiple score and average them. They show that it outperforms other baselines in a large suite of tabular data, ODDS benchmark, with default hyperparameter rules (e.g. the k is set based on number of samples N and d). They compare mainly with recent methods DROCC, GOAD and COPOD. ", "summary_of_the_review": "Pros:\n+ Good rule of hyperparameters which helps reproduce this method.\n+ Complete set of experiments and ablation study for the architecture choices.\n+ Good stability study of the sliding window k\n+ Not sure how authors find out the effectiveness of tanh and two normalization schemes, but they seem to perform well.\n\nCons:\n- No comparison to long-lasted baselines such as RRCF and KNN in the ODDS.\n- I think the claim should be changed to \"a default rule of hyperparameter selections performs well\".\n\nI would put the marginal acceptance for now but I will be happy to raise my score if the baselines are provided in the ODDS.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "msjP7TYpPkN", "reviewer": "Reviewer_uGiC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Interesting use of contrastive learning for anomaly detection on tabular data\n- Strong empirical performance outperforming recent baselines; extensive comparisons on ~30 datasets \n- Ablation studies and sensitivity analyses for hyperparameters are performed\n- Paper is well written and easy to follow; limitations and assumptions are discussed\n- Code is provided for reproducibility \n\nWeaknesses:\n- My major concern is that this work is very similar to the recent ICML 2021 paper \"Neural Transformation Learning for Deep Anomaly Detection Beyond Images\", which aims to learn the appropriate masking strategy for tabular data; this is in some sense a more general formulation compared to the fixed masking strategy used for contrastive learning here. Comparison and discussion of this work should be included though it can be to some extent be considered contemporaneous, and does not perform as extensive comparisons on tabular data.\n- It is a bit unsatisfying that the self-supervised task is solvable exactly without the need for the learned neural projections yet the success of the method relies on such neural projections; to be fair the paper does include some discussion of this point. Perhaps a more thorough exploration of the need for capacity constraints on the neural network will provide more insight into the method; one could experiment with different network architectures (e.g depth) and/or investigate how anomaly detection performance varies with overlap classification performance.  \n\nOther comments/suggestions:\n- It may be easier to compare the methods in Table 2 if the ranks and average ranks of the methods are provided.\n- Typos:\n  * In Table 3 caption: resutls -> results\n  * In and around Eq (1) - $\\Phi$ instead of $\\phi$\n", "summary_of_the_paper": "This paper introduces an anomaly detection algorithm for tabular data based on contrastive learning in the semi-supervised setting (training set assumed to be only normal data). The contrastive learning task is based on masking: the features from a single training example are split into two groups (pairs), one being a subsequence of the features, and the other its complement set; the learning task is then to differentiate pairs of splits that overlap (positive) vs not (negative) (from the same split or not) for each sample. The contrastive loss defined by this task is used to train the model and as the anomaly score. The proposed method is extensively evaluated on a range of tabular datasets where it outperforms recent methods.", "summary_of_the_review": "This is a paper presenting an interesting application of contrastive learning for anomaly detection on tabular data, with extensive experiments showing the strong empirical performance of the method. Ablation studies and sensitivity analyses are provided, as is the code for reproducibility. The main issue with this paper is the similarity to the somewhat contemporaneous ICML 2021 paper listed above, which should be discussed and compared to. On balance the relative novelty and strength of the experiments lean me towards acceptance. \n\n**Post Response Update:**\nThe authors have included a discussion of the related ICML 2021 paper and highlighted the differences, as well as added additional comparisons to baselines that support the superiority of the proposed method. I think the paper is a clear accept now.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IAvORr2yE6A", "reviewer": "Reviewer_e9Pz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Assumption\n- In the introduction, the authors assume that the feature relationship is class-dependent. \n- It would be good if the authors can provide some examples about this assumption and whether this assumption is practical.\n\n2. Consecutive features & Same number of features\n- The authors define the \"consecutive features\". \n- However, in tabular data, the order of the features are independent. \n- In that point of view, is there any reason that the authors are utilizing the \"consecutive\" features to extract the subset of the features?\n- Also, can we provide some more flexibility for selecting the subsets of the features? Like a different number of features? \n\n3. Intuitions in Figure 1\n- What is the intuition of selecting negative pairs?\n- I understand that the positive pair can be a_i^j and b_i^j.\n- However, it is hard to understand why the negative pair is a_i^m (where m is not j).\n- For instance, in Figure 1, a_i^6 is a subset of b_i^3. I understand that we use different embedding functions. However, still the inputs are the subset.\n- Maybe we can set the negative pair as a_m^j where (m is not j). \n\n4. Hyper-parameter optimization\n- It seems like in the problem setting, we do not have a validation set. (Because we only have in-sample data).\n- In that case, how did the authors optimize the hyper-parameters of the models? Like the encoder dimensions, learning rates, k, etc?\n- There are some rules of thumbs in page 4. But not sure how they are determined. Are they determined by checking the testing performances? If yes, I think it is a somewhat unfair way of doing the experiments.\n- Especially, it seems very heuristic of the thresholds determined by the dimensions (d). d < 40, d> 160, etc.\n\n5. Baselines\n- The authors have relevant baselines in the paper. However, some are missing.\n- Maybe it would be good if the authors add the following baselines: (i) Isolated Forests, (ii) PIDForest.\n\n6. Sensitive analyzes\n- The authors provide various sensitive analyzes in the appendix and those are very helpful.\n- However, in Figure 6 and 7, with very small k (like k = 1), the performances are similar to large k which is somewhat weird for me.\n- For instance, if k = 1, it means that we do contrastive learning between one feature and d-1 features which makes highly imbalanced networks F and G. \n- I am not sure whether this kind of learning would achieve meaningful representations for anomaly detection.", "summary_of_the_paper": "- The authors propose a novel anomaly detection framework that can be useful for tabular data.\n- The authors utilize the contrastive learning to learn the relationship between features in the tabular data. Then, using the loss as the anomaly scores for evaluation.\n- The authors provide extensive experimental results which are promising.", "summary_of_the_review": "Strength\n- The experimental results are quite promising and extensive.\n- The proposed method is simple and easy to reproduce.\n\nWeakness\n- The intuitions of the proposed method is limited. It is actually my biggest concern on this paper. If this can be improved during the rebuttal period, I will increase my scores. Please see my detailed comments above.\n- Some sensitive analyses are somewhat weird (e.g., k = 1).\n- Some additional baselines are needed to complete the experiments.\n\n-----------------------------------\n\nThank you for the detailed responses.\nI carefully read the responses.\nSome concerns (e.g., assumptions, consecutive features, and additional baselines) are resolved. Thank you.\nHowever, there are still some remaining questions.\n\n1. Intuition for negative pairing.\n- It is still not clear why the authors select those pairs as the negative pairs?\n- In figure 1, b_i^3 and a_i^6 are highly overlapped but still negative pair. \n- It would be good if the authors can explain about this clearer with proper intuitions.\n\n2. k=1\n- If k = 1, we do contrastive learning with the feature (with dimension = 1) and the feature (with dimension = d-1). \n- If the entire feature dimensions are large, the differences would be larger.\n- I am not sure whether this is the right way to do contrastive learning between these large different features.\n\nAlthough other reviewers are leaning to acceptance, I will stay with my original scores (5) until those above concerns are resolved.\n\nThank you.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "H7mhv4nh6kG", "reviewer": "Reviewer_yptt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Pros*\n+ Anomaly detection on general multivariate data (where few prior knowledge is available) is an important problem with many applications (fraud detection, etc.) that is relevant to the community.\n+ The proposed method performs favorably over recent state-of-the-art methods (DROCC, GOAD, COPOD) on many datasets. Overall, the experimental evaluation is extensive (4 + 30 datasets), scientifically rigorous (including Wilcoxon signed rank tests for best vs. second best performing method), and includes sensitivity analyses for critical method hyperparameters.\n+ The paper is overall structured well and easy to follow.\n+ The work is placed well into the existing literature.\n\n*Cons*\n- The methodological novelty is rather low (contrastive learning for anomaly detection has been recently explored [1, 2], though limited to image data).\n- There is not much of an explanation or intuition as to *why* the proposed method seems to work well. \n\n\n*Questions*\n(1) Could you elaborate more on why simply taking consecutive features seems to work well, although tabular data is generally permutation invariant? I appreciate the empirical evidence provided in Appendix A, but I struggle to understand the underlying reason for why this simple heuristic seems to work well on many datasets.\n\n\n*References*\n[1] K. Sohn, C.-L. Li, J. Yoon, M. Jin, and T. Pfister. Learning and evaluating representations for deep one-class classification. ICLR, 2021. \n[2] J. Tack, S. Mo, J. Jeong, and J. Shin. CSI: Novelty detection via contrastive learning on distributionally shifted instances. NeurIPS, 2020.", "summary_of_the_paper": "This paper proposes a contrastive learning method for unsupervised anomaly detection on general multivariate (tabular) data. The idea of the method is to train two encoders, one that embeds a subset of k features and one that embeds the residual d − k features, such that the resulting embeddings are closely aligned via the contrastive loss, where all other subsets of k features are taken as negative examples respectively. This is then repeated over all subsets of features of size k. Thus, the intuition of the approach is to extract common dependencies (e.g., correlations, statistical redundancy, etc.) between the features by achieving a small contrastive loss over the training data, thereby obtaining a high contrastive loss for anomalies (which presumably lack these common dependencies). An empirical evaluation on the Arrhythmia, Thyroid, KDD, and KDDRev datasets (where results from the literature exist) as well as 30 additional datasets from the ODDS library show that the proposed method performs favorably over other recent competing methods (DROCC, GOAD, COPOD) on tabular data.", "summary_of_the_review": "I think this work presents a simple method that shows significant improvements over previous methods in an extensive and scientifically rigorous evaluation, which makes this work a solid contribution to an important problem (anomaly detection on multivariate, tabular data). For this reason, I am leaning towards accepting this work, though I have some remaining questions and concerns (see above).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes contrastive learning for tabular data to improve anomaly detection. 
Strengths:
- Interesting and important problem.
- Usage of contrastive learning for anomaly detection in general multi-variate datasets is novel (as prior work mostly focuses on images)
- Extensive experiments with comparisons to multiple baselines on multiple datasets
- Well-written paper

The reviewers raised some concerns about novelty (in particular, the relationship to the closely related paper "Neural Transformation Learning for Deep Anomaly Detection Beyond Images"), hyperparameter tuning and additional baselines. The authors did a great job of addressing the concerns and multiple reviewers raised their scores. During the discussion phase, the consensus decision leaned towards accept. I recommend acceptance and encourage the authors to address any remaining concerns in the final version.

Additional AC comments:
- Please make sure that the camera ready version does not exceed page limits. https://iclr.cc/Conferences/2022/CallForPapers 
- " As far as we can ascertain, masking was not used for one-class classification before.": There's some related work on pre-training BERT for OOD detection (cf. https://arxiv.org/abs/2004.06100 or https://arxiv.org/abs/2106.03004) which might be worth discussing.
Decision: Accept (Poster)