Title: UAINETS: FROM UNSUPERVISED TO ACTIVE DEEP ANOMALY DETECTION

Abstract: This work presents a method for active anomaly detection which can be built upon existing deep learning solutions for unsupervised anomaly detection. We show that a prior needs to be assumed on what the anomalies are, in order to have performance guarantees in unsupervised anomaly detection. We argue that active anomaly detection has, in practice, the same cost of unsupervised anomaly detection but with the possibility of much better results. To solve this problem, we present a new layer that can be attached to any deep learning model designed for unsupervised anomaly detection to transform it into an active method, presenting results on both synthetic and real anomaly detection datasets.
INTRODUCTION

:

from which our 'normal' data instances are generated: X normal ∼ p normal (x) = p (x|y = 0), where x is an instance's available information1 and y is a label saying if the point is anomalous or not. There is also a different probability density function from which anomalous data instances are sampled: X anom ∼ p anom (x) = p (x|y = 1).

A full dataset is composed of both normal and anomalous instance, being sampled from a probability distribution that follows:

(X, Y ) full ∼ p full (x, y) = p (y) p (x|y) X full ∼ p full (x) = p (y = 0) p normal (x) + p (y = 1) p anom (x) = (1 -λ)p normal (x) + λp anom (x)

where λ is an usually small constant representing the probability of a random data point being anomalous (λ = p(y = 1)), this constant can be either known a priori or not. Chandola et al. (2009) divides anomaly detection learning systems in three different types:

• Supervised: A training and a test set are available with curated labels for non-anomalous and anomalous instances. This case is similar to an unbalanced supervised classification setting: D train/test = (X, Y ) train/test ∼ p full (x, y) • Semi-Supervised: A training set is available containing only non-anomalous instances and the challenge is to identify anomalous instances in a test set. This is also called novelty detection: D train = X train ∼ p normal (x) D test = X test ∼ p full (x) • Unsupervised: A dataset containing both non-anomalous and anomalous instance is available and the challenge is to identify anomalous instances in it. There is no concept of a test set since anomalous instances must be sorted in the dataset itself: D = X ∼ p full (x)

2.1. UNSUPERVISED ANOMALY DETECTION:

In this work, we will focus on unsupervised anomaly detection. Here, in possession of the full set of points X ∼ p full (x), we want to find a subset X anom ⊂ X which is composed of the anomalous instances. The full distribution p full is a mixture of distributions and if these distributions overlap very closely, it may be impossible to learn the individual distributions beyond a certain accuracy threshold (Dasgupta et al., 2005) . It is a well-known result that general mixture models are unidentifiable (Aragam et al., 2018; Bordes et al., 2006) . In the sequence, we further show that we gain no information on p anom from p full for any small λ without a prior on the anomalies' probability distribution. This differs from the usual unidentifiability of mixture models result in that we make no assumptions on the prior for p normal , showing all valid distributions of p anom are equally probable.

Theorem 1. No free anomaly theorem. Consider two independent arbitrary probability distributions p normal and p anom . For a small number of anomalies λ ≈ 0, p full = p gives us no further knowledge on the distribution of p anom :

p(p anom |p full = p) ≈ Uniform(P 2 ), λ ≈ 0 where P 2 = {p r , ∀p r ∈ P | λ ∈ [0; 1], λ • p r ≤ p} lim λ→0 P 2 = {p, ∀p ∈ P | supp(p) ⊆ supp(p)}

From Theorem 1 we can conclude that unsupervised anomaly detection requires a prior on the anomalies distribution. A more tangible example of this can be seen in Figure 2 , where we present a synthetic data distribution composed of three classes of data clustered in four visibly separable clusters. Anomaly detection is an undecidable problem in this setting without further information, since it is impossible to know if the low density cluster is composed of anomalies or the anomalies are the unclustered low density points (or a combination of both).

If we used a high capacity model to model the data distribution in Figure 2 , the low density points (Right) would be detected as anomalous. If we used a low capacity model, the cluster (Center) would probably present a higher anomaly score. Our choice of algorithm implicitly imposes a prior on the detected anomalies. Theorem 1 highlights this and makes the need to consider priors explicit. In a more practicle example, assume we are working with clinical data. In this setting, some low density clusters may indicate diseases (anomalies), while other low density clusters may be caused by uncontrolled factors in the data, such as high performance athletes. At the same time, rare diseases might seem like scattered (low density) points. We want to be able to distinguish between anomalies and 'uninteresting' low probability points. 

3. MODEL:

The usual strategy when working with unsupervised anomaly detection problems is training a parameterized model p θ (x) to capture the full data distribution p full (x) (e.g. a PCA, or AutoEncoder), and, since λ is, by definition, a small constant, assuming p full (x) ≈ p normal (x) and assuming points with low probability are anomalous (Zhou & Paffenroth, 2017 ). An anomaly score s(x) is then defined as s(x) = 1 p(x) . There are three main problems with this strategy: (1) if anomalous items are more common than expected, p full might be a poor approximation of p normal ; (2) if anomalous items are tightly clustered in some way, high capacity models may learn to identify that cluster as a high probability region; (3) if anomalous items are as rare as expected, since we only have access to p full , Theorem 1 states we have no information about p anom without further assumptions on its probability distribution.

Most unsupervised anomaly detection systems also already rely on further verification of the results by human experts, due to their uncertain performance. Being mostly used as a ranking system to get high probability instances in the top of a 'list' to be further audited by these experts.

From Theorem 1, we conclude it is impossible to have an universal and reliable unsupervised anomaly detection system, while we know that most such systems already rely on the data being later audited by human experts. These arguments together argue in favor of an active learning strategy for anomaly detection, including the auditor experts in the system's training loop. Thus, anticipating feedback and benefiting from it to find further anomalous instances, which results in a more robust system. Having an extremely unbalanced dataset in this problem (λ ≈ 0) is also another justification for an active learning setting, which has the potential of requiring exponentially less labeled data than supervised settings (Settles, 2012) .

3.1. ACTIVE ANOMALY DETECTION:

With these motivations, we argue in favor of active anomaly detection methods, which despite its many advantages remains an under-explored approach to this problem. Nonetheless, recent work has shown promising results (Veeramachaneni et al., 2016; Das et al., 2016; 2017) . In unsupervised anomaly detection, we start with a dataset D = {x|x ∼ p full (x)} and want to rank elements in this dataset so that we have the highest possible recall/precision for a certain budget b, which is the number of elements selected to be audited by an expert, with no prior information on anomaly labels.

In active anomaly detection, we also start with a completely unlabeled anomaly detection dataset D = {x|x ∼ p full (x)}, but instead of ranking anomalies and sending them all to be audited at once by our expert, we select them in small parts, waiting for the experts feedback before continuing. We iteratively select the most probable k b elements to be audited 2 , wait for the expert to select their label, and continue training our system using this information, as shown in Algorithm 1. This requires the same budget b as an unsupervised anomaly detection system, while having the potential of achieving a much better performance. labels ← labels ∪ expert.audit(top_k)

Algorithm 1 Active Anomaly Detection:

7: i ← i + k

With this in mind, we develop the Universal Anomaly Inference (UAI) layer. This layer can be incorporated on top of any deep learning based white box anomaly detection system which provides an anomaly score for ranking anomalies. It takes as input both a latent representation layer (l(x)), created by the model, and its output anomaly score (s(x)), and passes it through a classifier to find an item's anomaly probability.

EQUATION

This is motivated by recent works stating learned representations have a simpler statistical structure (Bengio et al., 2013) , which makes the task of modeling this manifold and detecting unnatural points much simpler (Lamb et al., 2018) . In this work, we model the UAI layer using a simple logistic regression as our classifier, but any architecture could be used here:

EQUATION

where W act ∈ R 1,d+1 is a linear transformation, b act ∈ R is a bias term and σ(•) is the sigmoid function. We learn the values of W and b using back-propagation with a cross entropy loss function, while allowing the gradients to flow through l, but not through s, since s might be non-differentiable.

For the rest of this document, we will refer to the networks with a UAI layer as UaiNets. An example of this architecture is shown in Figure 1 .

cross_entropy(x y , x y )), which use standard multi layer perceptrons. Both architectures are described in details in Appendix A.1. To test our algorithm we start by analyzing its performance on synthetic data created with different properties (Section 4.1). We then present results using UaiNets on real anomaly detection datasets (Section 4.2) and in a semi-supervised setting (Section 4.3).

4.1. SYNTHETIC DATA:

When designing experiments, we had the objective of showing that our model can work with different definitions of anomaly, while completely unsupervised models will need, by definition, to trade-off accuracy in one setting for accuracy in the other. While this may seem straight forward, these results can show how robust our approach is to the choice of underlying architecture, analyzing how well they do when their underlying architecture has a bad prior for that specific "type" of anomaly. With this in mind, we used the MNIST dataset and defined four sets of experiments:3 

1. MNIST 0 : For the first set of experiments, we reduced the presence of the 0 digit class to only 10% of its original number of samples, making it only 1/91 ≈ 1.1% of the dataset samples. The 0s still present in the dataset had its class randomly changed to x y ∼ Uniform([1; 9]) and were defined as anomalies. 2. MNIST 0-2 : The second set of experiments follows the same dataset construction, but we reduce the number of instances of numbers 0, 1 and 2, changing the labels of the remaining items in these categories to x y ∼ Uniform([3; 9]), and again defining them as anomalous.

In this dataset anomalies composed 3/73 ≈ 4.1% of the dataset. 3. MNIST hard : The third set of experiments aims to test a different type of anomaly. In order to create this dataset, we first trained a weak one hidden layer MLP classifier on MNIST and selected all misclassified instances as anomalous, keeping them in the dataset with their original properties (x x and x y ). In this dataset anomalies composed ≈ 3.3% of the dataset. 4. MNIST pca : In this set of experiments, for each image class (x y ), we used a PCA to reduce the dimensionality of MNIST images (x x ) to 2 and selected the 5% instances with the largest reconstruction error as anomalies. We kept all 60,000 instances in the dataset with their original properties (x x and x y ) and in this dataset anomalies composed 5% of the dataset.

Results for these experiments are shown in Figure 3 and the main conclusion taken from them is that, even though our algorithm might not get better results than its underlying model for every budget-dataset pair, it is robust to different types of anomalies, which is not the case for the underlying completely unsupervised models. While Class gives really good results in MNIST 0 and MNIST 0-2 datasets, it does not achieve the same performance in MNIST hard and MNIST pca , which might indicate it is better at finding clustered anomalies than low density ones. At the same time, DAE has good results for MNIST pca and MNIST hard , but bad ones for MNIST 0 and MNIST 0-2 , which indicates it is better at finding low density anomalies than clustered ones. Nevertheless, both UaiNets are robust in all four datasets, being able to learn even on datasets which are hard for their underlying models, although they might have a cold start to produce results.

4.2. REAL DATA:

Here we analyze our model's performance on public benchmarks composed of real anomaly detection datasets. We employ 11 datasets in our analysis: KDDCUP; Thyroid; Arrhythmia; KDDCUP-Rev; Yeast; Abalone; Cardiotocography (CTG); Credit Card; Covtype; Mammography (MMG); Shuttle (Lichman, 2013; Dheeru & Taniskidou, 2017; Pozzolo et al., 2015; Woods et al., 1993) . We compare our algorithm against: DAE (Vincent et al., 2008) ; DAGMM (Zong et al., 2018) ; LODA-AAD (Das et al., 2016) ; and Tree-AAD (Das et al., 2017) . Table 1 presents results for these real datasets. In these experiments, DAGMM (clean) was trained on a semi-supervised anomaly detection setting, using clean datasets during training, DAGMM (dirty) and DAE were trained in an unsupervised setting, while LODA-AAD, Tree-AAD and DAE uai were trained in an active anomaly detection setting. We can clearly see from these results that DAE produces fairly bad results for all datasets analyzed here, nevertheless, even using a simple architecture as its underlying model, DAE uai produces similar or better results to the best baselines on the 11 datasets, even when the baselines were trained in completely clean training sets. DAE uai also usually presents better results than LODA-AAD and Tree-AAD, which are similarly trained in an active setting.

One possible criticism to our method is that the importance of the proposed approach becomes more relevant the fewer the proportion of anomalous instances, which seems self-defeating. But we see that the largest difference from the active methods to the other algorithms was in Covtype, which has less than 1% anomalies but 286,048 instances. When working with large datasets (>1M instances), even if only 0.1% of the dataset is contaminated there is still the chance to benefit from this feedback to improve performance. The active algorithms are also more robust than the others, DAGMM used different hyperparameters for each experiment, while DAE uai and AAD use the same for all (except for k which was reduced from 10 to 3 for the datasets with less than 100 anomalies).

4.3. A MORE PRACTICAL ANOMALY DETECTION SETTING:

Another practical scenario where our model could be applied is a mixture of semi-supervised and unsupervised anomaly detection. In this case, we have a dataset which contains anomalies that we want to find and audit. At the same time, new data instances, which may include new types of anomalies not seen before, can be added to the dataset at any time and we would like to detect anomalies in this dataset as well.

D train = X train ∼ p full (x) = (1 -λ)p normal (x) + λp anom (x) D test = X test ∼ p full (x) = (1 -λ 1 -λ 2 )p normal (x) + λ 1 p anom (x) + λ 2 p anom_new (x)

With this in mind, we ran an experiment training DAE uai and LODA-AAD on KDDCUP-Rev in the same way as in Section 4.2, while evaluating it on its test set for different budgets. This test set The evaluation was done by selecting the most anomalous instances found by each model on the test set and calculating the recall for both seen and unseen anomalies in that group. Results for this experiment can be seen in Figure 4 . In this figure, the right y axis shows the number of anomalies detected in the training set for a certain budget and corresponds to the light blue lines. The left y axis present the recall for the test dataset. We see that DAGMM is not so effective on this test set, while DAE is able to detect well novelty (new classes). We also see that DAE uai is significantly better at detecting known types of anomalies, while it maintains a recall close to the best on new unseen classes, giving better results than LODA-AAD for both seen and unseen classes of anomalies. 

5. RELATED WORKS:

Anomaly Detection This field has been amply studied and good overviews can be found in (Hodge & Austin, 2004; Chandola et al., 2009) . Although many algorithms have been recently proposed, classical methods for outlier detection, like LOF Breunig et al. (2000) and OC-SVM (Schölkopf et al., 2001) , are still used and produce good results. Recent work on anomaly detection has focused on statistical properties of "normal" data to identify these anomalies, such as Maurus & Plant (2017) , which uses Benford's Law to identify anomalies in social networks, and (Siffer et al., 2017) , which uses Extreme Value Theory to detect anomalies. Other works focus on specific types of data, (Zheng et al., 2017) focuses on spatially contextualized data, while (Perozzi et al., 2014; Perozzi & Akoglu, 2016; Li et al., 2017; Liu et al., 2017) focus on graph data. Recently, energy based models (Zhai et al., 2016) and GANs (Schlegl et al., 2017) have been successfully used to detect anomalies, but autoencoders are still more popular in this field. Zhou & Paffenroth (2017) propose a method to train robust autoencoders, drawing inspiration from robust statistics (Huber, 2011) and more specifically robust PCAs, (Yang et al., 2017) focuses on clustering, and trains autoencoders that generate latent representations which are friendly for k-means. The work most similar to ours is DAGMM (Zong et al., 2018) , where they train a deep autoencoder and use its latent representations, together with its reconstruction error, as input to a second network, which they use to predict the membership of each data instance to a mixture of gaussian models, training the whole model end-to-end in an semi-supervised manner for novelty detection.

Active Anomaly Detection Despite its many advantages, active anomaly detection remains an under-explored approach to this problem, nevertheless, over the years some really interesting work has been developed in this topic. In (Pelleg & Moore, 2005) , the authors solve the rare-category detection problem by proposing an active learning strategy to datasets with extremely skewed distributions of class sizes. Abe et al. (2006) reduces outlier detection to classification using artificially generated examples that play the role of potential outliers and then applies a selective sampling mechanism based on active learning to the reduced classification problem. In (Görnitz et al., 2013) , the authors proposed a Semi-Supervised Anomaly Detection (SSAD) method based in Support Vector Data Description (SVDD) (Tax & Duin, 2004) , which he expanded to a semi-supervised setting, where he accounts for the presence of labels for some anomalous instances, and with an active learning approach to select these instances to label. Veeramachaneni et al. (2016) propose an active approach that combines unsupervised and supervised learning to select items to be labeled by experts, with each approach selecting k 2 instances at a time. The most similar prior works to ours in this setting are (Das et al., 2016) , which proposed an algorithm that can be employed on top of any ensemble methods based on random projections, and (Das et al., 2017) , which expands Isolation Forests to work in an active setting. Our work differs from these prior works mainly in that we prove the necessity of priors for unsupervised anomaly detection, further motivating the Active Anomaly Detection framework, and in our proposed model. UAI layers can be assembled on top of any Deep Learning based anomaly detection architecture, which is the state of the art for unsupervised anomaly detection, to make it work in an active anomaly detection setting. Besides, after each iteration with experts both LODA-AAD and Tree-AAD have a time complexity O(t), where t is the number of already labeled instances, while each iteration of UaiNets runs in constant time O(1) with respect to t.

6. DISCUSSIONS AND FUTURE WORK:

We proposed here a new architecture, Universal Anomaly Inference (UAI), which can be applied on top of any deep learning based anomaly detection architecture. We show that, even on top of very simple architectures, like a DAE, UaiNets can produce similar/better results to state-of-the-art unsupervised/semi-supervised anomaly detection methods. We also give both theoretical and practical arguments motivating active anomaly detection, arguing that, in most practical settings, there would be no detriment to using this instead of a fully unsupervised approach.

We further want to make clear that we are not stating our method is better than our semi-supervised baselines (DAGMM, DCN, DSEBM-e). Our contributions are orthogonal to theirs. We propose a new approach to this hard problem which can be built on top of them, this being our main contribution in this work. To the best of our knowledge, this is the first work which applies deep learning to active anomaly detection. We use the strongest points of these deep learning algorithms (their learned representations and anomaly scores) to build an active algorithm, presenting an end-to-end architecture which learns representations by leveraging both the full dataset and the already labeled instances.

Important future directions for this work are using the UAI layers confidence in its output to dynamically choose between either directly using its scores, or using the underlying unsupervised model's anomaly score to choose which instances to audit next. Another future direction would be testing new architectures for UAI layers, in this work we restricted all our analysis to simple logistic regression. A third important future work would be analyzing the robustness of UaiNets to mistakes being made by the labeling experts. Finally, making this model more interpretable, so that auditors could focus on a few "important" features when labeling anomalous instances, could increase labeling speed and make their work easier.

A EXPERIMENTS DESCRIPTIONS:

In this section we give detailed descriptions of the experiments. Section A.1 presents the used model architectures for both DAE and Class models, as well as DAE uai and Class uai . Section A.2 presents details on the synthetic MNIST datasets and on the hyper-parameters used for the experiments. Finally, Section A.3 contains detailed descriptions on the used datasets, baselines and experimental settings for the experiments on real anomaly detection datasets.

A.1 MODEL ARCHITECTURES:

To show our algorithm can be assembled on top of any deep learning model, we tested it using two simple but very different anomaly detection models. The first model we test it on top of is a normal Denoising AutoEncoder (DAE). A DAE is a neural network mainly composed by an encoder, which transforms the input into a latent space, and a decoder, which reconstructs the input using this latent representation, typically having a loss function that minimizes the reconstruction error L 2 norm:

l = f enc (x + ) ∼ N (0, ϕ) x = f dec (l) L = ||x -x|| 2 2 (4)

where both f enc and f dec are usually feed forward networks with the same number of layers, l ∈ R d is a d-dimensional latent representation and is a zero mean noise, sampled from a Gaussian distribution with a ϕ standard deviation. When used in anomaly detection, the reconstruction error is usually used as an approximation of the inverse of an item's probability, and as its anomaly score:

s dae (x) = 1 p(x) = ||x -x|| 2 2 (5)

We then create a DAE uai network by assembling the proposed UAI layer on top of the DAE:

EQUATION

where uai(•) is the classifier chosen for the UAI layer. This architecture can be seen in Figure 1 .

Another typical approach to unsupervised anomaly detection is, when given a dataset with labeled data X = (x x , x y ), training a classifier (Class) to predict x y from x x5 and using the cross-entropy of an item as an approximation to the inverse of its probability distribution:

EQUATION

where f class (•) is typically a feed forward neural network with p layers, from which we can use its last hidden layer (h p-1 ) as the data's latent representation to be used in the Class uai .

EQUATION

This architecture can be seen in Figure 5 . For all experiments in this work, unless otherwise stated, the DAE's encoder and decoder had independent weights and we used both the DAE and Class models with 3 hidden layers and hidden sizes [256, 64, 8] . This means the latent representations provided to the UAI layers are l ∈ R 8 . We implemented all experiments using TensorFlow (Abadi et al., 2016) , and used a learning rate of 0.01, batch size of 256 and the RMSprop optimizer with the default hyper-parameters. For the active learning models, we pre-train the DAE/Class model for 5000 optimization steps, select k = 10 items to be labeled at a time, and further train for 100 iterations after each labeling call. To deal with the cold start problem, for the first 10 calls of select_top, we use the base anomaly score (s) of the DAE/Class model to make this selection, using the UAI one for all later labeling decisions. Detailed statistics on the synthetic MNIST datasets can be seen in Table 2 . MNIST 0 and MNIST 0-2 were mainly generated with the purpose of simulating the situation in Figure 2 (Center), where anomalies were present in sparse clusters. At the same time, MNIST hard and MNIST pca were designed to present similar characteristics to the situation in Figure 2 (Right), where anomalous instances are in sparse regions of the data space.

A.3 REAL DATA:

For these experiments, most datasets were used as suggested in (Dheeru & Taniskidou, 2017 ), but we processed the KDDCUP, Thyroid, Arrhythmia and KDDCUP-Rev datasets in the same manner as (Zong et al., 2018) to be able to better compare with their results:

• KDDCUP (Lichman, 2013): The KDDCUP99 10 percent dataset from the UCI repository. Since it contains only 20% of instances labeled as "normal" and the rest as "attacks", "normal" instances are used as anomalies, since they are in a minority group. This dataset contains 34 continuous features and 7 categorical ones. We transform these 7 categorical features into their one hot representations, and obtain a dataset with 120 features.

• Thyroid (Lichman, 2013) : A dataset containing data from patients which can be divided in three classes: normal (not hypothyroid), hyperfunction and subnormal functioning. In this dataset, we treat the hyperfunction class as an anomaly, with the other two being treated as normal. It can be obtained from the ODDS repository.6 

• Arrhythmia (Lichman, 2013) : This dataset was designed to create classification algorithms to distinguish between the presence and absence of cardiac arrhythmia. In it, we use the smallest classes (3, 4, 5, 7, 8, 9, 14, and 15) as anomalies and the others are treated as normal. This dataset can also be obtained from the ODDS repository.

• KDDCUP-Rev (Lichman, 2013) : Since "normal" instances are a minority in the KDDCUP dataset, we keep all "normal" instances and randomly draw "attack" instances so that they compose 20% of the dataset.

We compare our algorithm against:

• DAE ( Vincent et al., 2008) : Denoising Autoencoders are autoencoder architectures which are trained to reconstruct instances from noisy inputs.

• DAGMM (Zong et al., 2018) : Deep Autoencoding Gaussian Mixture Model is a stateof-the-art model for semi-supervised anomaly detection which simultaneously learns a latent representation, using deep and uses both this latent representation and the autoencoder's reconstruction error to learn a Gaussian Mixture Model for the data distribution.

• LODA-AAD (Das et al., 2016) : Lightweight on-line detector of anomalies (LODA) Active Anomaly Discovery (AAD) is a work which uses the active anomaly detection framework on top of LODA (Pevnỳ, 2016) , which is a method based on ensembles of weak anomaly detection models.

• Tree-AAD (Das et al., 2017) : This work learns weights for each node in an Isolation Forest anomaly detection model, by incorporating knowledge gained through active anomaly detection.

Since there is no validation/test set in unsupervised anomaly detection, we cannot tune hyperparameters on a validation set. Because of this, to make the DAE baselines more competitive, we got the results for several different hyper-parameter configurations and present only the best among them. This is not a realistic approach, but we only do it to our baselines, while for our proposed algorithm we keep hyper-parameters fixed for all experiments. We even keep our hidden sizes fixed to [256, 64, 8] on thyroid, which only contains 6 features per instance, since our objective here is not getting the best possible results, but showing the robustness of our approach. The only hyper-parameter change we make in UAI networks is that, since there are fewer anomalies in some datasets, we set our active learning approach to choose k = 3 instances at a time, instead of 10, for datasets with less than 100 anomalies.

Results for DAGMM are from our implementation of this model and follow the same procedures, architectures and hyper-parameters as described in (Zong et al., 2018) , being trained in a semisupervised setting. The results for LODA-AAD and Tree-AAD were run using the code made available by the authors and with the same steps as DAE uai .7 For all experiments, results for LODA-AAD, Tree-AAD, DAE and DAE uai used the number of anomalies in the dataset as the budget b.

B DETAILED RESULTS:

In this section, we present more detailed results for both the synthetic (Section B.1) and real (Section B.2) anomaly detection datasets, which couldn't fit on the main paper due to lack of space. We also present results for synthetic anomaly detection experiments on Fashion-MNIST (Section B.3).

B.1 DETAILED RESULTS ON MNIST:

We present here detailed results for small budgets (b ≤ 5000) on the MNIST experiments, with graphs zoomed in for these budget values. Analyzing Figure 6 we see that for some of these datasets UaiNets present a cold start, producing worse results for small budgets. Nonetheless, after this cold start, they produce better results in all MNIST experiments. An interesting future work would be to measure the confidence in the UaiNet's prediction to dynamically choose between using its anomaly score or the underlying network's one, which could solve/reduce this cold start problem. , 2018) , while DAGMM * are results from our implementation of DAGMM. Unfortunately, we were not able to reproduce their results in the Thyroid dataset, getting a high variance in the results. LODA-AAD does not scale well to large datasets, so to run it on KDDCUP and KDDCUP-Rev we needed to limit its memory about the anomalies it had already learned, forgetting the oldest ones. This reduced its runtime complexity from O(b 2 ) to O(b) in our tests, where b is the budget limit for the anomaly detection task. We did the same (limit memory) for Tree-AAD on KDDCUP.

On this table we can see that DAE uai produces better results than LODA-AAD on all analyzed datasets and than Tree-AAD on three out of four. Our proposed method also, besides presenting results comparable to state-of-the-art DAGMM trained on a clean dataset, is much more stable, having a lower standard deviation than the baselines in almost all datasets.

B.3 EXPERIMENTS ON FASHION-MNIST:

In this Section, we present results for experiments on synthetic anomaly detection datasets based on Fashion-MNIST (Xiao et al., 2017) . To create these datasets we follow the same procedures as done for MNIST in Section 4.1, generating four datasets: Fashion-MNIST 0 ; Fashion-MNIST 0-2 ; Fashion-MNIST hard ; Fashion-MNIST pca . Detailed statistics of these datasets can be seen in Table 4 . We run experiments on these datasets following the exact same procedures as in Section 4.1. Figure 7 shows the results for Fashion-MNIST 0 and Fashion-MNIST 0-2 , while Figure 8 show the results for Fashion-MNIST hard and Fashion-MNIST pca . These figures show similar trends to the ones for MNIST, although algorithms find anomalies in these datasets harder to identify. In one run of Fashion-MNIST 0 , DAE uai needed several examples to start learning and for Fashion-MNIST hard , Class uai takes a long time to start producing better results than Class. Nevertheless, UaiNets are still much more robust than the underlying networks to different types of anomalies, producing good results in all four datasets, even when its underlying network gives weak results on that dataset.

C FURTHER ANALYSIS:

In this section we further study UaiNets, analyzing the evolution of hidden representations and anomaly scores through training (Section C.1), and the dependence of results on the number of audited anomalies (Section C.2).

C.1 LEARNED REPRESENTATIONS AND ANOMALY SCORES:

In this section, we show visualizations of the learned representations (l dae/class ) and anomaly scores (s dae/class ) of UaiNets' underlying networks, presenting their evolution as more labels are fed into the network through the active learning process. With this purpose, we retrain UaiNets on both MNIST 0-2 and MNIST hard , with a hidden size of [256, 64, 1], so that its latent representation is one dimensional (l(x) ∈ R 1 ), and plot these representations vs the anomaly scores (s) of the base network (either DAE or Class) for different budgets (b).

Figure 9 shows the evolution of DAE uai 's underlying l dae (x) and s dae (x). In it, we can see that initially (Figures 9 (a, d )) anomalies and normal data instances are not separable in this space. Nevertheless, with only a few labeled instances (b = 250) the space becomes much easier to separate, while for b = 2000 the space is almost perfectly linearly separable.8 

Figure 10 shows the same evolution for Class uai 's underlying l class (x) and s class (x). In it, we can also see the same patterns, as initially anomalies and normal data instances are not separable, but with a few labeled instances anomalies become much more identifiable.

The main conclusion taken from these visualizations is how the gradient flow through l is important, since it helps the network better separate data in these spaces, allowing good performance even when the underlying networks are not good at identifying a specific type of anomaly. 

C.2 ANOMALY CHOICES EVOLUTION THROUGH TRAINING:

This experiments aim at showing how the networks choice quality evolves with the access to more labels. Here, we present the choices DAE uai network would make having access to a fixed number of expert labels. With this in mind, we train the networks in the same way as in Section 4.2, but stop after reaching a specific budget (b), showing the choices made up to that point, and after that with no further training.

Figure 11 shows the evolution of DAE uai anomaly choices as it is fed more expert knowledge. We can see that with only a few labels it already fairs a lot better than its underlying network. In KDDCUP with only 3,000 labeled instances, which is less than 1% of the dataset, it can correctly find 80,000 anomalies with a high precision, while the DAE with no expert knowledge does a lot worse. On Thyroid and KDDCUP-Rev, with ≈ 10% of the dataset labeled (b = 531 and b = 4000, respectively) it finds all or almost all anomalies in the dataset correctly. The Arrhythmia dataset is a lot smaller and with few anomalies, so DAE uai improves on DAE in a smaller scale here, but it still does fairly better than the underlying network.9 

D PROOFS:

D.1 LEMMA 1. MIXTURE PROBABILITY LEMMA Lemma 1. Mixture probability lemma. Consider two independent arbitrary probability distributions p 1 and p 2 . Given only a third distribution p + = p composed of the weighted average of the two:

p + = (1 -λ) • p 1 + λ • p 2 , 0 ≤ λ ≤ 1

and considering P i as the residual probability distribution hyperplanes:

P 1 = p r = p-λ•p 1-λ , ∀p ∈ P | λ ∈ [0; 1], λ • p ≤ p = {p r , ∀p r ∈ P | λ ∈ [0; 1], (1 -λ) • p r ≤ p} P 2 = p r = p-(1-λ)•p λ , ∀p ∈ P | λ ∈ [0; 1], (1 -λ) • p ≤ p = {p r , ∀p r ∈ P | λ ∈ [0; 1], λ • p r ≤ p}

Without further assumptions on p 2 (without a prior on its probability distribution), we only know that p(p 1 |p + = p) = p(p 1 |p 1 ∈ P 1 ) and p(p

2 |p + = p α ) = p(p 1 = p α -λ α • p 2 |p 2 ∈ P 2 ).

Proof. Given p + = p we know that:

p 1 + λ α • p 2 = p α with λ α = λ

1-λ and p α = p 1-λ . Assuming the distribution of p 2 is independent of p 1 , and with no further assumptions on it, p 2 is random and uniform on the set of all possible probability distributions, where P is the hyperspace containing all probability distributions, with an hyper-volume m. Now we can try to find p(p 1 |p + = p):

p(p 1 |p + = p) = p (p + = p|p 1 ) • p(p 1 ) p(p + = p) (1) = p (p + = p|p 1 ) • p(p 1 ) p(p + = p) , p 1 ∈ P 1 (2) = p p 2 = p α -p 1 λ α |p 1 • p(p 1 ) x p p 1 = x, p 2 = p α -x λ α dx (3) = p p 2 = p α -p 1 λ α • p(p 1 ) x p p 1 = x, p 2 = p α -x λ α dx (4) = 1 m • p(p 1 ) x p p 1 = x|p 2 = p α -x λ α • p p 2 = p α -x λ α dx (5) = 1 m • p(p 1 ) x∈P1 p(p 1 = x) • 1 m dx + x / ∈P1 0dx = p(p 1 ) x∈P1 p(p 1 = x)dx p(p 1 |p + = p α ) = p(p 1 |p 1 ∈ P 1 )

The equality in (1) comes from the definition of the space P 1 , which is the space of all possible values of p 1 that could result in p + = p, so if p 1 / ∈ P 1 , then p (p + = p|p 1 ) = 0. Equality (2) is a simple variable substitution where p (p + = p) = p p 1 = x, p 2 = p α -x λα .

(3) comes from the assumption that p 2 and p 1 are independent. Equality (4) results from p 2 ∼ U nif orm(P ) and P having a volume m. Finally, Equality ( 5) is a result from the fact that p α -x λ α ∈ P ⇔ x ∈ P 1 .

With a similar strategy we can find p(p 2 |p + = p):

p(p 2 |p + = p) = p(p + = p|p 2 ) • p(p 2 ) p(p + = p) = p(p 1 = p α -λ α • p 2 |p 2 ) • p(p 2 ) x p p 1 = x, p 2 = p α -x λ α dx = p(p 1 = p α -λ α • p 2 |p 2 ) • p(p 2 ) x p p 1 = x, p 2 = p α -x λ α dx , p 2 ∈ P 2 (1) = p(p 1 = p α -λ α • p 2 |p 2 ) • p(p 2 ) x p p 1 = x, p 2 = p α -x λ α dx , p 1 ∈ P 1 = p(p 1 = p α -λ α • p 2 ) • p(p 2 ) x p p 1 = x, p 2 = p α -x λ α dx = p(p 1 = p α -λ α • p 2 ) • 1 m x p p 1 = x|p 2 = p α -x λ α • p p 2 = p α -x λ α dx = p(p 1 = p α -λ α • p 2 ) • 1 m x∈P1 p(p 1 = x) • 1 m dx + x / ∈P1 0dx = p(p 1 = p α -λ α • p 2 ) x∈P1 p(p 1 = x)dx = p(p 1 = p α -λ α • p 2 |p 1 ∈ P 1 ) (2) = p(p 1 = p α -λ α • p 2 |p 2 ∈ P 2 ) p(p 2 |p + = p) = p(p 1 = p α -λ α • p 2 |p 2 ∈ P 2 )

where Equality (1) and (2) result from the fact that p 1 ∈ P 1 ⇔ p 2 ∈ P 2 , given a specific value of p + = p. This completes this proof.

D.2 LEMMA 2. EXTREME MIXTURES LEMMA:

Lemma 2. Extreme mixtures lemma. Consider two independent arbitrary probability distributions p 1 and p 2 . Given only a third probability distribution p + = p composed of the weighted mixture of the two, and for a small λ ≈ 0, we can find a small residual hyperplane P 1 , which tends to {p}.

P 1 ≈ {p r = p -λ • p, ∀p ∈ P | λ • p ≤ p} λ ≈ 0 (9)

We can also find a very large residual hyperplane P 2 for p 2 , which tends to:

EQUATION

where supp(•) is the support of a probability distribution.

Proof. In this proof, we start with the arbitrary residual hyperplanes P r and find restrictions in the limits of λ → 0 and λ → 1. For a β ≈ 0: This finishes this proof.

lim β→0 P r = lim β→0 {p r = p-β•p 1-β , ∀p ∈ P | β • p ≤ p} = lim β→0 {p r = p -β • p, ∀p ∈ P | β • p ≤ p} = {p} P r ≈ {p r = p -β • p, ∀p ∈ P | β • p ≤ p} β ≈ 0 P 1 ≈ {p r = p -λ • p, ∀p ∈ P | λ • p ≤ p} λ ≈ 0 P 2 ≈ {p r = p -(1 -λ) • p, ∀p ∈ P | (1 -λ) • p ≤ p} λ ≈ 1 ∴ β ≈ 0

E FURTHER PROOFS:

In this section, we prove upper and lower bounds on the maximum distance a probability distribution p 1 can be from p + , based on the value of λ. This can be directly applied to p normal for small values of λ and to p anom for large ones.

Theorem 2. Upper Bound on Mixture Probability Distance For two independent arbitrary probability distributions p 1 and p 2 , given only a third probability distribution p + composed of the weighted mixture of the two:

p + = (1 -λ) • p 1 + λ • p 2

We have an upper bound on the distance measures δ(p + , p 1 ) and ||p + -p 1 || given by:

δ(p + , p 1 ) ≤ 1 2 log 1 1 -λ ||p + -p 1 || ≤ 2 log 1 1 -λ

which is a tight bound for λ ≈ 0. In this equation δ(•) is the total variation distance between two probability distributions and || • || is the L 1 norm.

Proof. Pinsker's inequality states that if p and q are two probability distributions on a common measurable space (A, F): (1-λ)•p1(x)+λ•p2(x) dx ≤ x p 1 (x) log p1(x)

(1-λ)•p1(x) dx = x p 1 (x) log 1 1-λ dx = log 1 1-λ x (p 1 (x)dx) = log 1 1-λ which concludes the proof that:

δ(p + , p 1 ) ≤ 1 2 log 1 1 -λ ||p + -p 1 || ≤ 2 log 1 1 -λ

Theorem 3. Lower Bound on Maximum Mixture Probability Distance For two independent arbitrary probability distributions p 1 and p 2 , given only a third probability distribution p + composed of the weighted mixture of the two:

p + = (1 -λ) • p 1 + λ • p 2

We have a lower bound on the maximum possible distance measures δ(p + , p 1 ) and ||p + -p 1 || for a chosen maximizing p 1 given by:

max p1 δ(p + , p 1 ) ≥ λ • |A -1| |A| max p1 ||p + -p 1 || ≥ 2λ |A -1| |A|

which is a tight bound for λ ≈ 1, considering the maximum L 1 distance between two probability distributions is 2.

Proof. We can prove a lower bound on the maximized distance of a probability distribution p 1 from p + by expanding the distance equations: where in (a) we lower bound based on the probability distribution that would have the smallest possible superior distance to a later maximized probability distribution p 1 . This probability distribution p 1 can always maximize its superior distance to p 2 by: p 1 (a) = 1 , if a = argmin x (p 2 (x)) 0 , else

In (b) we choose the uniform distribution as the one that would reduce this superior distance and in (c) we set p 1 (a) = 1 for a random a, since p 2 is uniform. With a similar strategy we find:

max p1 ||p + -p 1 || ≥ 2λ |A -1| |A|

This concludes this proof.

x, in our notation, is the information known about a data instance. This can be further composed of what would actually be x and y in a supervised setting, such as an image and its corresponding class label. We will reference this as xx and xy here.

EXPERIMENTSIn this section, we test our new UAI layer on top of two distinct architectures: a Denoising AutoEncoder (DAE, with s dae (x) = ||x -x|| 22 ) and a Classifier (Class, with s class (x) =2 Although this might seem like a simplistic approach, selecting the top k anomalies is a good strategy in practical settings, since we want to have the most anomalies for any given budget. Besides, because anomaly detection is already a highly imbalanced problem, we might not get anomalous instances even when picking only the top anomalous results, so actively searching for them is usually a good choice. This approach follows recent work in active anomaly detection(Veeramachaneni et al., 2016;Das et al., 2016; 2017)

Implementation details, such as the used architecture and hyper-parameters can be found in Appendix A, as well as further details about the synthetic MNIST datasets. Using MNIST for the generation of synthetic anomaly detection datasets follows recent works(Zhou & Paffenroth, 2017;Zhai et al., 2016). Due to lack of space we only report full results here, but the same plots zoomed in for small budgets (b ≤ 5000) can be found in Appendix B.1. We also report the same experiments with similar results on Appendix B.3 for the MNIST-Fashion dataset.

Further descriptions of these datasets and baselines can be found in Appendix A.3, as well as descriptions of the used architectures and hyper-parameters. More detailed results, standard deviations and comparison to other baselines are presented in Appendix B.2.

Note that, even though in this problem we have class labels (xy), we have no anomaly labels of objects (y), so this is still an unsupervised anomaly detection problem.

http://odds.cs.stonybrook.edu

https://github.com/shubhomoydas/ad_examples

Gifs showing this choice evolution will be made available with the final publication.

Gifs showing this choice evolution will be made available with the final publication.

Figures and Tables:

Figure 1: DAE uai architecture.

Figure 2: Example of undecidable anomalous data distribution: (Left) Raw data distribution; (Center) Possible Clustered Anomalies; (Right) Possible Low Density Anomalies.

1: procedure ACTIVEANOMALYDETECTION(D, expert, b, k)

Figure 3: (Color online) Results for different MNIST experiments. Lines represent median of five runs with different seeds and confidence intervals represent max and min results for each budget b.

Figure 4: Semi supervised experiment. This test set contains 67,908 instances, out of which 3,817 are previously seen anomalies and 3,498 unseen, totaling 7,315 anomalies. These results show how the Recall@7315 on the test set improve, for both seen and unseen classes of anomalies, as the budget increases during the active training.(Left) Anomalies in training set. (Right) New (unseen) anomalies.

Figure 5: Class uai architecture.

Figure 7: (Color online) Results for Fashion-MNIST 0 and Fashion-MNIST 0-2 with different zooms on x-axis. Lines represent median of five runs with different seeds and confidence intervals represent max and min results for each budget b.

Figure 8: (Color online) Results for Fashion-MNIST hard and Fashion-MNIST pca with different zooms on x-axis. Lines represent median of five runs with different seeds and confidence intervals represent max and min results for each budget b.

Figure 9: (Color online) Underlying latent representations (l dae ) vs anomaly score (s dae ) for DAE uai network as training progresses on MNIST 0-2 and MNIST hard .

we start with the other definition of P r :lim β→1 P r = lim β→1 {p r , ∀p r ∈ P | (1 -β) • p r ≤ p} = lim β→1 {p r , ∀p r ∈ P | supp(p r ) ⊆ supp(p), (1 -β) • p r ≤ p} = {p r , ∀p r ∈ P | supp(p r ) ⊆ supp(p)} P r ≈ {p r , ∀p r ∈ P | supp(p r ) ⊆ supp(p)} β ≈ 1 P 1 ≈ {p r , ∀p r ∈ P | supp(p r ) ⊆ supp(p)} λ ≈ 1 P 2 ≈ {p r , ∀p r ∈ P | supp(p r ) ⊆ supp(p)} λ ≈ 0 ∴ β ≈ 1This finishes this proof.D.3 THEOREM 1. NO FREE ANOMALY THEOREMTheorem 1. Consider two independent arbitrary probability distributions p normal and p anom . For a small number of anomalies λ ≈ 0, the knowledge that p full = p gives us no further knowledge on the distribution of p anom :p(p anom |p full = p) ≈ Uniform(P 2 ), λ ≈ 0Proof. Consider in Lemmas 1 and 2 that p 2 = p anom ∼ Uniform(P ). We then have that, for a small value of λ ≈ 0:p(p 2 |p + = p α ) = p(p 1 = p α -λ α • p 2 |p 2 ∈ P 2 ) ≈ p(p 1 = p α |p 2 ∈ P 2 ) = Uniform(P 2 )

δ(p, q) = sup{|p(A) -q(A)| : A ∈ F} ≤ 1 2 • D KL (p||q) ||p -q|| ≤ 2 • D KL (p||q)where D KL (p||q) is the Kullback-Leibler divergence in nats. So we have that:δ(p + , p 1 ) ≤ 1 2 • D KL (p 1 ||p + )and this Kullback-Leibler divergence is itself upper-bounded by:D KL (p 1 ||p + ) = λ)•p1(x)+λ•p2(x) dxwhere this maximum Kullback-Leibler divergence is achieved when p 1 and p 2 are disjoint probability distributions:D KL (p 1 ||p + ) ≤ max p2 x p 1 (x) log p1(x)

max p1 δ(p + , p 1 ) = max p1 sup{|p + (A) -p 1 (A)| : A ∈ F} = max p1 sup{|(1 -λ) • p 1 + λ • p 2 -p 1 (A)| : A ∈ F} = max p1 sup{|λ • p 2 (A) -λ • p 1 (A)| : A ∈ F} = λ • max p1 sup{|p 2 (A) -p 1 (A)| : A ∈ F} (a) ≥ λ • max p1 min p2 sup{|p 2 (A) -p 1 (A)| : A ∈ F} (b) = λ • max p1 sup{|U nif orm(A) -p 1 (A)| : A ∈ F} (c) = λ • sup{|U nif orm(A) -δ(A)| : A ∈ F} = λ • |A-1| |A| max p1 δ(p + , p 1 ) ≥ λ • |A-1| |A|

Results on Real Datasets showing average F1 scores of five independent runs.

Table content: <table><tr><td colspan="2">Train Set</td><td colspan="2">KDDCUP</td><td>Arrhythmia</td><td colspan="2">Thyroid</td><td>KDDCUP-Rev</td><td>Yeast</td></tr><tr><td colspan="2"># Instances</td><td colspan="2">494,021</td><td>3,772</td><td>452</td><td/><td>121,597</td><td>1,191</td></tr><tr><td colspan="2"># Features</td><td>120</td><td/><td>6</td><td>274</td><td/><td>120</td><td>8</td></tr><tr><td colspan="4"># Anomalies (%) 97,278 (20%)</td><td>93 (2.5%)</td><td colspan="2">66 (15%)</td><td>24,319 (20%)</td><td>55 (4.6%)</td></tr><tr><td colspan="2">DAGMM (clean)</td><td>0.94</td><td/><td>0.50</td><td>0.44</td><td/><td>0.94</td><td>0.11</td></tr><tr><td colspan="2">DAGMM (dirty)</td><td>0.43</td><td/><td>0.46</td><td>0.46</td><td/><td>0.31</td><td>0.02</td></tr><tr><td colspan="2">LODA-AAD</td><td>0.88</td><td/><td>0.45</td><td>0.51</td><td/><td>0.83</td><td>0.31</td></tr><tr><td colspan="2">Tree-AAD</td><td>0.89</td><td/><td>0.29</td><td>0.86</td><td/><td>0.50</td><td>0.32</td></tr><tr><td/><td>DAE</td><td>0.39</td><td/><td>0.35</td><td>0.09</td><td/><td>0.16</td><td>0.23</td></tr><tr><td colspan="2">DAEuai</td><td>0.94</td><td/><td>0.47</td><td>0.57</td><td/><td>0.91</td><td>0.33</td></tr><tr><td>Train Set</td><td colspan="2">Abalone</td><td>CTG</td><td colspan="2">Credit Card</td><td colspan="2">Covtype</td><td>MMG</td><td>Shuttle</td></tr><tr><td># Instances</td><td colspan="2">1,920</td><td>1,700</td><td>284,807</td><td/><td colspan="2">286,048</td><td>11,183</td><td>12,345</td></tr><tr><td># Features</td><td>9</td><td/><td>22</td><td>30</td><td/><td>54</td><td>6</td><td>9</td></tr><tr><td colspan="8"># Anomalies (%) 29 (1.5%) 45 (2.6%) 492 (0.17%) 2,747 (0.9%) 260 (2.3%) 867 (7.0%)</td></tr><tr><td>DAGMM (clean)</td><td>0.16</td><td/><td>0.27</td><td>0.34</td><td/><td colspan="2">0.18</td><td>0.07</td><td>0.48</td></tr><tr><td>DAGMM (dirty)</td><td>0.05</td><td/><td>0.18</td><td>0.31</td><td/><td colspan="2">0.01</td><td>0.00</td><td>0.48</td></tr><tr><td>LODA-AAD</td><td>0.54</td><td/><td>0.52</td><td>0.57</td><td/><td colspan="2">0.97</td><td>0.42</td><td>0.97</td></tr><tr><td>Tree-AAD</td><td>0.53</td><td/><td>0.69</td><td>0.76</td><td/><td colspan="2">0.94</td><td>0.59</td><td>0.92</td></tr><tr><td>DAE</td><td>0.08</td><td/><td>0.13</td><td>0.36</td><td/><td colspan="2">0.15</td><td>0.27</td><td>0.17</td></tr><tr><td>DAEuai</td><td>0.55</td><td/><td>0.66</td><td>0.64</td><td/><td colspan="2">0.86</td><td>0.60</td><td>0.93</td></tr></table>



Table content: <table><tr><td/><td/><td colspan="3">: MNIST Anomaly Datasets Statistics</td><td/></tr><tr><td/><td colspan="5"># Dimensions # Classes # Instances # Anomalies Anomaly Ratio</td></tr><tr><td>MNIST 0</td><td>784</td><td>9</td><td>54,679</td><td>602</td><td>1.1%</td></tr><tr><td>MNIST 0-2</td><td>784</td><td>7</td><td>43,199</td><td>1,822</td><td>4.2%</td></tr><tr><td>MNIST hard</td><td>784</td><td>10</td><td>60,000</td><td>2,108</td><td>3.5%</td></tr><tr><td>MNIST pca</td><td>784</td><td>10</td><td>60,000</td><td>2,996</td><td>5%</td></tr><tr><td colspan="2">A.2 SYNTHETIC DATA</td><td/><td/><td/><td/></tr></table>

Detailed results on real datasets showing mean and standard deviations of five runs.

Table content: <table><tr><td>Dataset</td><td>Method</td><td>Anomalies in Train Set</td><td>Precision</td><td>Recall</td><td>F1</td></tr><tr><td/><td>OC-SVM</td><td>0%</td><td>0.7457</td><td>0.8523</td><td>0.7954</td></tr><tr><td/><td>OC-SVM</td><td>5%</td><td>0.1155</td><td>0.3369</td><td>0.1720</td></tr><tr><td/><td>PAE</td><td>0%</td><td>0.7276</td><td>0.7397</td><td>0.7336</td></tr><tr><td/><td>DSEBM-r</td><td>0%</td><td>0.1972</td><td>0.2001</td><td>0.1987</td></tr><tr><td/><td>DSEBM-e</td><td>0%</td><td>0.7369</td><td>0.7477</td><td>0.7423</td></tr><tr><td/><td>DSEBM-e</td><td>5%</td><td>0.5345</td><td>0.5375</td><td>0.5360</td></tr><tr><td/><td>DCN</td><td>0%</td><td>0.7696</td><td>0.7829</td><td>0.7762</td></tr><tr><td>KDDCUP</td><td>DCN DAGMM</td><td>5% 0%</td><td>0.6763 0.9297</td><td>0.6893 0.9442</td><td>0.6827 0.9369</td></tr><tr><td/><td>DAGMM</td><td>5%</td><td>0.8504</td><td>0.8643</td><td>0.8573</td></tr><tr><td/><td>DAGMM  *</td><td>0%</td><td colspan="3">0.9290 (0.0344) 0.9435 (0.0349) 0.9362 (0.0346)</td></tr><tr><td/><td>DAGMM  *</td><td>5%</td><td colspan="3">0.8827 (0.0682) 0.8965 (0.0693) 0.8896 (0.0688)</td></tr><tr><td/><td>DAGMM  *</td><td>20%</td><td colspan="3">0.4238 (0.0187) 0.4304 (0.0190) 0.4271 (0.0188)</td></tr><tr><td/><td>LODA-AAD</td><td>20%</td><td colspan="3">0.8756 (0.1255) 0.8756 (0.1255) 0.8756 (0.1255)</td></tr><tr><td/><td>Tree-AAD</td><td>20%</td><td colspan="3">0.8940 (0.0261) 0.8940 (0.0261) 0.8940 (0.0261)</td></tr><tr><td/><td>DAE</td><td>20%</td><td colspan="3">0.3905 (0.2581) 0.3905 (0.2581) 0.3905 (0.2581)</td></tr><tr><td/><td>DAEuai</td><td>20%</td><td colspan="3">0.9401 (0.0191) 0.9401 (0.0191) 0.9401 (0.0191)</td></tr><tr><td/><td>OC-SVM</td><td>0%</td><td>0.3639</td><td>0.4239</td><td>0.3887</td></tr><tr><td/><td>PAE</td><td>0%</td><td>0.1894</td><td>0.2062</td><td>0.1971</td></tr><tr><td/><td>DSEBM-r</td><td>0%</td><td>0.0404</td><td>0.0403</td><td>0.0403</td></tr><tr><td/><td>DSEBM-e</td><td>0%</td><td>0.1319</td><td>0.1319</td><td>0.1319</td></tr><tr><td/><td>DCN</td><td>0%</td><td>0.3319</td><td>0.3196</td><td>0.3251</td></tr><tr><td>Thyroid</td><td>DAGMM DAGMM  *</td><td>0% 0%</td><td colspan="3">0.4766 0.4375 (0.1926) 0.4468 (0.1967) 0.4421 (0.1947) 0.4834 0.4782</td></tr><tr><td/><td>DAGMM  *</td><td>0.5%</td><td colspan="3">0.2875 (0.1505) 0.2936 (0.1537) 0.2905 (0.1521)</td></tr><tr><td/><td>DAGMM  *</td><td>2.5%</td><td colspan="3">0.4542 (0.2995) 0.4638 (0.3059) 0.4590 (0.3027)</td></tr><tr><td/><td>LODA-AAD</td><td>2.5%</td><td colspan="3">0.5097 (0.0712) 0.5097 (0.0712) 0.5097 (0.0712)</td></tr><tr><td/><td>Tree-AAD</td><td>2.5%</td><td colspan="3">0.8586 (0.0087) 0.8586 (0.0087) 0.8586 (0.0087)</td></tr><tr><td/><td>DAE</td><td>2.5%</td><td colspan="3">0.0860 (0.0725) 0.0860 (0.0725) 0.0860 (0.0725)</td></tr><tr><td/><td>DAEuai</td><td>2.5%</td><td colspan="3">0.5742 (0.0582) 0.5742 (0.0582) 0.5742 (0.0582)</td></tr><tr><td/><td>OC-SVM</td><td>0%</td><td>0.5397</td><td>0.4082</td><td/></tr><tr><td/><td>PAE</td><td>0%</td><td>0.4393</td><td>0.4437</td><td>0.4403</td></tr><tr><td/><td>DSEBM-r</td><td>0%</td><td>0.1515</td><td>0.1513</td><td>0.1510</td></tr><tr><td/><td>DSEBM-e</td><td>0%</td><td>0.4667</td><td>0.4565</td><td>0.4601</td></tr><tr><td/><td>DCN</td><td>0%</td><td>0.3758</td><td>0.3907</td><td>0.3815</td></tr><tr><td>Arrhythmia</td><td>GADMM GADMM  *</td><td>0% 0%</td><td colspan="3">0.4909 0.4902 (0.0514) 0.5051 (0.0530) 0.4975 (0.0522) 0.5078 0.4983</td></tr><tr><td/><td>GADMM  *</td><td>3%</td><td colspan="3">0.4530 (0.0573) 0.4666 (0.0591) 0.4597 (0.0582)</td></tr><tr><td/><td>GADMM  *</td><td>15%</td><td colspan="3">0.4500 (0.0597) 0.4636 (0.0615) 0.4567 (0.0606)</td></tr><tr><td/><td>LODA-AAD</td><td>15%</td><td colspan="3">0.4485 (0.0136) 0.4485 (0.0136) 0.4485 (0.0136)</td></tr><tr><td/><td>Tree-AAD</td><td>15%</td><td colspan="3">0.2882 (0.0257) 0.2882 (0.0257) 0.2882 (0.0257)</td></tr><tr><td/><td>DAE</td><td>15%</td><td colspan="3">0.3485 (0.0392) 0.3485 (0.0392) 0.3485 (0.0392)</td></tr><tr><td/><td>DAEuai</td><td>15%</td><td colspan="3">0.4727 (0.0225) 0.4727 (0.0225) 0.4727 (0.0225)</td></tr><tr><td/><td>OC-SVM</td><td>0%</td><td>0.7148</td><td>0.9940</td><td>0.8316</td></tr><tr><td/><td>PAE</td><td>0%</td><td>0.7835</td><td>0.7817</td><td>0.7826</td></tr><tr><td/><td>DSEBM-r</td><td>0%</td><td>0.2036</td><td>0.2036</td><td>0.2036</td></tr><tr><td/><td>DSEBM-e</td><td>0%</td><td>0.2212</td><td>0.2213</td><td>0.2213</td></tr><tr><td/><td>DCN</td><td>0%</td><td>0.2875</td><td>0.2895</td><td>0.2885</td></tr><tr><td>KDDCUP-Rev</td><td>GADMM GADMM  *</td><td>0% 0%</td><td colspan="3">0.9370 0.9391 (0.1553) 0.9391 (0.1553) 0.9391 (0.1553) 0.9390 0.9380</td></tr><tr><td/><td>GADMM  *</td><td>5%</td><td colspan="3">0.3184 (0.1358) 0.3559 (0.2096) 0.3341 (0.1658)</td></tr><tr><td/><td>GADMM  *</td><td>20%</td><td colspan="3">0.3051 (0.1059) 0.3053 (0.1060) 0.3052 (0.1059)</td></tr><tr><td/><td>LODA-AAD</td><td>20%</td><td colspan="3">0.8339 (0.1081) 0.8339 (0.1081) 0.8339 (0.1081)</td></tr><tr><td/><td>Tree-AAD</td><td>20%</td><td colspan="3">0.5032 (0.2984) 0.5032 (0.2984) 0.5032 (0.2984)</td></tr><tr><td/><td>DAE</td><td>20%</td><td colspan="3">0.1626 (0.0609) 0.1626 (0.0609) 0.1626 (0.0609)</td></tr><tr><td/><td>DAEuai</td><td>20%</td><td colspan="3">0.9117 (0.0160) 0.9125 (0.0170) 0.9121 (0.0165)</td></tr><tr><td>et al.</td><td/><td/><td/><td/><td/></tr></table>



Table content: <table><tr><td/><td colspan="4">: Fashion-MNIST Anomaly Datasets Statistics</td><td/></tr><tr><td/><td colspan="5"># Dimensions # Classes # Instances # Anomalies Anomaly Ratio</td></tr><tr><td>Fashion-MNIST 0</td><td>784</td><td>9</td><td>54,610</td><td>610</td><td>1.1%</td></tr><tr><td>Fashion-MNIST 0-2</td><td>784</td><td>7</td><td>43,765</td><td>1,765</td><td>4.0%</td></tr><tr><td>Fashion-MNIST hard</td><td>784</td><td>10</td><td>60,000</td><td>9,656</td><td>16.1%</td></tr><tr><td>Fashion-MNIST pca</td><td>784</td><td>10</td><td>60,000</td><td>3,000</td><td>5.0%</td></tr></table>

Peer review (iclr2019):
 The paper provided a convincing and intuitive motivation regarding the need for active learning in unsupervised anomaly detection. However the proposed approach of requesting expert feedback for the top ranked anomalies is straightforward and unsurprising, given past work on active learning. The experiments on synthetic data are also unsurprising. Moreover these are based on a questionable premise: the instances that are "hard" to classify are treated as anomalies. This is not very realistic. Regarding the real data experiments: In Table 1 the results for DAE_uai are based on which budget b? How does the result vary with b?