Deep Semi-Supervised Anomaly Detection
Authors: Lukas Ruff
Robert A. Vandermeulen
Nico Görnitz
Alexander Binder
Emmanuel Müller
Klaus-Robert Müller
Marius Kloft
Venue:  (2020)
Keywords: anomaly detection
deep learning
semi-supervised learning
unsupervised learning
outlier detection
one-class classification
deep anomaly detection
deep one-class classification

Abstract: Deep approaches to anomaly detection have recently shown promising results over shallow methods on large and complex datasets. Typically anomaly detection is treated as an unsupervised learning problem. In practice however, one may have---in addition to a large set of unlabeled samples---access to a small pool of labeled samples, e.g. a subset verified by some domain expert as being normal or anomalous. Semi-supervised approaches to anomaly detection aim to utilize such labeled samples, but most proposed methods are limited to merely including labeled normal samples. Only a few methods take advantage of labeled anomalies, with existing deep approaches being domain-specific. In this work we present Deep SAD, an end-to-end deep methodology for general semi-supervised anomaly detection. We further introduce an information-theoretic framework for deep anomaly detection based on the idea that the entropy of the latent distribution for normal data should be lower than the entropy of the anomalous distribution, which can serve as a theoretical interpretation for our method. In extensive experiments on MNIST, Fashion-MNIST, and CIFAR-10, along with other anomaly detection benchmark datasets, we demonstrate that our method is on par or outperforms shallow, hybrid, and deep competitors, yielding appreciable performance improvements even when provided with only little labeled data.

Published as a conference paper at ICLR 2020
DEEP SEMI-SUPERVISED ANOMALY DETECTION
Lukas Ruff
Robert A. Vandermeulen'
Nico Görnitz 12
Alexander Binder Emmanuel Müller
Klaus-Robert Müller Marius Kloft
'Technical University of Berlin, Germany
de, Berlin, Germany
Singapore University of Technology Design, Singapore
"Bonn-Aachen International Center for Information Technology, Germany
KKorea University, Seoul, Republic of Korea
Max Planck Institute for Informatics, Saarbrücken, Germany
Technical University of Kaiserslautern, Germany
(lukas. rufff, vandermeulen, nico. waco.goernitee @tu- in in.idni..d de
edu. ag mieller@bit. wniuunnn..daa
coaus-rbber..oo www.meeeeeetttbeeiinnn.e.. kloftlcs. uni 1a1kkl.60 de
ABSTRACT
Deep approaches to anomaly detection have recently shown promising results over shallow methods on large and complex datasets. Typically anomaly detection is treated unsupervised learning problem. In practice however, may have- in addition to a large set of unlabeled samples-access to a small pool of labeled samples, e.g. subset verified by some domain expert being normal or anoma- lous. Semiisupervised approaches to anomaly detection aim to utilize such labeled samples, but most proposed methods are limited merely including labeled nor- mal samples. Only few methods take advantage of labeled anomalies, with ex- isting deep approaches being domain-speciffi end-to-end deep methodology for general semi-supervised anomaly detection. We further introduce an rnformatioonthhoreee framework for deep anomaly detec- tion based on the idea that the entropy of the latent distribution for normal data should be lower than the entropy of the anomalous distribution, which can serve as theoretical interpretation for method. In extensive experiments MNIST, Mashion-MNIST. and CIFAR-I0. 10, along with other anomaly detection benchmark datasets, we demonstrate that method is par outperforms shallow, hy- brid, and deep competitors, yielding appreciable performance improvements even when provided with only little labeled data.
this work we present Deep SAD,
INTRODUCTION
Anomaly detection (AD) CChnndola et al. 2009 Pimentel et al. 2014) is the task of identifying unusual samples in data. Typically AD methods attempt to learn "compact" description of the data in an unsupervised manner assuming that most of the samples are normal (i.e., not anomalous). For example, in one-class classification (Moya et al., 1993, [Schölkopf et al., 200T) the objective is to find set of small measure which contains most of the data and samples not contained in that set are deemed anomalous. Shallow unsupervised AD methods such as the One-Class SVM (Schölkopf et al., |000]] Tax & Duin, 2004). Kernel Density Estimation (Parzen 1962 Kim & Scott, 2012 Vandermeulen & Scott 2013). ). or Isolation Forest (Liu et al.] 2008) often require manual nigh-dimensional data and are limited in their scalability to
feature engineering to be effective
large datasets. These limitations have sparked great interest in developing novel deep approaches to
unsupervised AD (Erfani et al. 2016 Zhai al. 2016 Chen et al.) 2017 Ruff et al.) 2018 Deecke
et al.
2018 Ruff et al. 2019 Golan & El- Yaniy 2018 Pang et al. 2019 Hendrycks et al. 2019a b).
Majority work was done while RV was at TU Kaiserslautem, Germany.
1 Part of the work was done while MK was sabbatical visitor of the DASH Center the University of Southern California, United States.

---

Published as a conference paper at ICLR 2020
anomaly score
low
high
unlabeled
(a) Training data
Unsupervised AD (OC-SVM)
Supervised classificr (SVM)
(d) Semi Semi-supervised classifier
(e) Semi supervvee LPUE
(I) Semi-supervinn AD (ours)
Figure 1: The need for semi-supervised anomaly detection: The training data (shown in (a)) consists of (mostly normal) unlabeled data (gray) as well few labeled normal samples (blue) and labeled anomalies (orange). Figures (f)-(f) show the decision boundaries of the various learning paradigms testing time along with novel anomalies that occur (bottom left in each plot). Our semi-supervised AD approach takes advantage of all training data: unlabeled samples, labeled normal samples, as well as labeled anomalies. This strikes a balance between one-class learning and classification.
Unlike the standard unsupervised AD setting, many real-worke applications one may also have access to some verified (i.e., labeled) normal or anomalous samples in addition to the unlabeled data. Such samples could be hand labeled by domain expert for instance. This leads a -upervieee AD problem: given (mostly normal but possibly containing some anomalous contamination) un- labeled samples 21,..... Fy and m labeled samples (æ ......., . (mm= Fm)) where ý +1 and -1 denote normal and anomalous samples respectively, the task is to learn model that com- pactly characterizes the "normal class."
The term semi-supervised anomaly detection has been used to describe two different AD settings. Most existing
AD methods, both shallow (Muñoz-Mari et al. 2010
Blanchard
et al., 2010 Chandola et al.] 2009) and deep (Song al. [017]] Akcay et al. 2018 Chalapathy
& Chawla 2019). only incorporate the use of labeled normal samples but not labeled anoma-
lies, i.e. they
precisely instances of Learning from Positive (i.e., normal) and Unlabeled
Examples (LPUE) (Denis)
1998 Zhang & Zuo 2008). A few works (Wang et al., 2005 Liu &
Zheng 2006 Görnitz al. 2013 have investigated the general semi-supervised AD setting where one also utilizes labeled anomalies, however existing deep approaches are domain or data-type spe-
cific (Ergen al. 2017 Kiran al. 2018 Min et al. 2018).
Research on deep semi- semi-supervised learning has almost exclusively focused on classification as the downstream task (Kingma et al., 2014, Rasmus al., 2015 Odena 2016 Dai et al.) 2017 Oliver
et al. 2018). Such semi-supervised classifiers typically assume that similar points are likely to be of same class, this known as the cluster assumption (Zhu 2005 Chapelle et al. 2009). This as- sumption, however, only holds for the "normal class" in AD, but is crucially invalid for the "anomaly class" since anomalics are not necessarily similar to one another. Instead, cemi-supervised AD ap- proaches must find compact description of the normal class while also correctly discriminating the labeled anomalies (Görnitz et al. 2013). Figure illustrates the differences between various learning paradigms applied to AD toy example.
We introduce Deep SAD (Deep Semi-supervised Anomaly Detection) in this work, an end-to-end deep method for general semi-supervised AD. Our main contributions the following:
2

---

Published as a conference paper at ICLR 2020
We introduce Deep SAD, generalization of the unsupervised Deep SVDD method (Ruff ct al. 2018) to the supi-super AD setting.
We present an information-thewe
framework for deep AD, which can serve an inter-
pretation of our Deep SAD method and similar approaches.
We conduct extensive experiments in which we establish experimental scenarios for the general semi-supervised AD problem where we also introduce novel baselines.
2 AN INFORMATTION-THORRRT VIEW ON DEEP ANOMALY DETECTION
The study of the theoretical foundations of deep learning an active and ongoing research effort
(Montavon et al.) 2011 Tishby & Zas Zaslavsky 2015] Cohen et al. 2016: Eldan & Shamir 2016
Neyshabur et al. [017 Raghu al. [20 7 Zhang et al. 201 | Achille & Soatto | Arora al,
2018 Belkin et al. 2018 8. Wiatowski & Bölcskei 2018 Lapuschkin et al., 2019). One important line of research that has emerged rooted information theory (Shannon) 1948). In the supervised classification setting where one has input variable X, latent variable % (e.g.. the final layer of a deep network), and output variable Y (i.c., the label), the well-known Information Bottleneck principle (Tishby al. 1999: Tishby & Zaslavsky 2015 Shwartz-Ziv & Tishby] 2017; [Alem et al.) 2017; Saxe et 20188 2018) provides an explanation for representation learning as the trade-off between finding minimal compression Z of the input X while retaining the informativeness of Z for predicting the label Y. Put formally, supervised deep learning seeks to minimize the mutual information I(X; 2) between the input X and the latent representation Z while maximizing the mutual information I(Z;Y) Y) between Z and the classification task Y, i.e.
min p(a(a)
I(X; aI(YY))
where p(z(zr) is modeled by a deep network and the hyperparameter > 0 controls the trade-off between compression (i.e., complexity) and classification accuracy.
For unsupervised deep learning. due the absence of labels Y and thus the lack of a clear task, other information-theone theoretic learning principles have been formulated. Of these, the Infomax principle Bell & Sejnowski 1995| Hjelm et al.) 2019) is one of the most prevalent and widely used principles. In contrast to (1. the objective of Infomax is to maximize the mutual information I(X; Z) between the data X and its latent representation Z:
(innker 1988;
max p(z(z)
I(X; + 2(x;))++R((2) (Z))
This is typically done under some additional constraint or regularization R(Z) on the representation Z with hyperparameter / > 0) to obtain statistical properties desired for some specific downstream task. Examples where the Infomax principle has been applied include tasks such as independent component analysis (Bell & Sejnowski, 1995), clustering (Slonim al. 2005 Ji et al. 2018), gen-
erative modeling (Chen et al. 2016; Hoffman & Johnson 2016 Zhao et al.
2017 Alemi et al.)
2018). and unsupervised representation learning in general (Hjelm et al. 2019)
We observe that the Infomax principle has also been applied in previous deep representations for
AD. Most notably autoencoders (Rumelhart et al. 1986 Hiinonn & Salakhutdinov 2006). which
the predominant approach to deep AD (Hawkins et al. 2002] (aauradaa & Yain 2014 Andrews
al., 2016 Erfani et al. 2016; Zhai et al. [016]] Chen et al. 2017/CChalapath
Chawla 2019).
can be understood as implicitly maximizing the mutual information Z(X; 2) via the reconstruction objective (Vincent et al. 2008) under some regularization of the latent code Z. Choices for regu- larization include sparsity (Makhzani & Frey] 2014), the distance to some latent prior distribution, e.g. measured via the KL divergence (Kingma & Welling, 2013) Rezende et al. 20144, an adversar- ial loss (Makhzani et al. 2015). or simply bottleneck in dimensionality. Such restrictions for AD share the idea that the latent representation of the normal data should be in some sense "compact."
As illustrated in Figure a supervised (or semi-supervised) classification approach to AD only learns to recognize anomalies similar to those seen during training, due to the class cluster assump- tion (Chapelle et al. 2009). However, anything not normal is by definition anomaly and thus anomalies do not have to be similar. This makes supervised (or semi-supervised) classification learning principles such (I) ill-defined for AD. We instead build upon principle (2) to motivate deep method for general seml-supervised AD, where we include the label information Y through a novel representation learning regularization objective R(Z)= R(Z; Y) that is based entropy.
3

---

Published as a conference paper at ICLR 2020
3 DEEP SEMI-SUPERVISED ANOMALY DETECTION
In the following, we introduce Deep SAD, a deep method for general semi-supervised AD. To for- mulate objective, first briefly explain the unsupervised Deep SVDD method (Ruff et al. 2018) which we then generalize to the semi-supervised AD setting.
3.1 UNSUPERVISED DEEP SVDD AND ENTROPY MINIMIZATION
For input space X' C TR D and output space 2 Rdd let o1: W) X - 2 be neural network with L hidden layers and corresponding set of weights W = {W'.... W4). The objective of Deep SVDD is train the neural network 6 learn a transformation that minimizes the volume of a Data-enclosing hypersphere in output space 2 centered on predetermined point c. Given n (unlabeled) training samples 21....,II X, the
Deep SVDD objective is
A 1 Σ |o(0;; W) - cl1
Σ 2 Em
min W
A>0.
(3)
Penalizing the mean squared distance of the mapped samples to the hypersphere center forces the network to extract those common factors of variation which are most stable within the dataset. As consequence normal data points tend to get mapped near the hypersphere center, whereas anomalies are mapped further away (Ruff et al. 2018). The second term a standard weight decay regularizer.
Deep SVDD is optimized via SGD using backpropagation. For initialization, Ruff et al. (2018) first pre-train an autoencoder and then initialize the weights W of the network & with the converged weights of the encoder. After initialization, the hypersphere center e is set as the mean of the network outputs obtained from an initial forward pass of the data. Once the network is trained, the anomaly score for test point given by the distance from p(æ; W) to the center of the hypersphere:
s(x) = [ W) cll-
We now argue that Deep SVDD may not only be interpreted in geometric terms minimum volume estimation (Scott& & Nowak 2006), but also in probabilistic terms as entropy minimization over the latent distribution. For latent random variable with covariance Σ, pdf p(z). and support Z Rd we have the following bound on entropy
1 H(Z) = log p(2) = =- log dz - log (2rT)) det 2),
(5)
2
which holds with equality iff Z is jointly Gaussian (Cover & Thomas 2012). Assuming the latent distribution Z follows an isotropic Gaussian, Z N (1, o²1) with σ
get
1 H(Z)= log((2ne)) det oeto²1)
(0gg((2 yd 1) _
d
log(2no²)) & logo?
(6)
2
ie. for fixed dimensionality d, the entropy of is proportional to its bog-variance.
Now observe that the Deep SVDD objective 33 (disregarding weight decay regularization) is equiv- alent to minimizing the empirical variance and thus minimizes an upper bound on the entropy of latent Gaussian. Since the Deep SVDD network is pre-trained
autoencoding objective that implicitly maximizes the mutual information I[X; Z) (Vincent et al., 2008). we may interpret Deep SVDD following the Infomax principle (2) with the additional "compactness" objective that the latent distribution should have minimal entropy.
3.2
DEEP SAD
We now introduce our method for deep semi-supervised anomaly detection: Deep SAD. Assume that, in addition to the unlabeled samples I...... E X' with X' S RDD we also have access to m labeled samples (æ1, (i)................. ...... (Øm, Fm) E X x with y [-1,+1} where ğ denotes known normal samples and ÿ -1 known anomalies. We define our Deep SAD objective as follows:
min
1 n
lo(a;; W) c|12 m+m
Σ (l1(0;; -))(
c
A
W 12..
(7)
2
4

---

Published as a conference paper at ICLR 2020
We employ the same loss term as Deep SVDD for the unlabeled data in our Deep SAD objective and thus recover Deep SVDD (3) as the special case when there is no labeled training data available (m 0). In doing this we also incorporate the assumption that most of the unlabeled data is normal.
For the labeled data, we introduce a new loss term that is weighted via the hyperparameter 17 > 0 which controls the balance between the labeled and the unlabeled term. Setting 7/ > puts more emphasis on the labeled data whereas < emphasizes the unlabeled data. For the labeled normal samples (ÿ +1), we also impose a quadratic loss on the distances of the mapped points to the center c, thus intending to overall learn a latent distribution which concentrates the normal data. Again, one might consider 17 > to emphasize labeled normal over unlabeled samples. For the labeled anomalies (j = 1) in contrast, we penalize the inverse of the distances such that anomalies must be mapped further away from the center | Note that this is in line with the common assumption that anomalies are not concentrated (Schölkopf & Smola 2002, Steinwarrt et al.] (005). In our experiments we found that simply setting 7 = yields a consistent and substantial performance improvement. A sensitivity analysis on is in Section 4.3
We define the Deep SAD anomaly score again by the distance of the mapped point to the center e as given in Eq. 4 and optimize our Deep SAD objective ( via SGD using backpropagation. We provide a summary of the Deep SAD optimization procedure and further details in Appendix [
In addition to the inverse squared norm loss we experimented with several other losses including the negative squared norm loss, negative robust losses, and the hinge loss. The negative squared norm loss, which is unbounded from below, resulted in an ill-posed optimization problem and caused optimization to diverge. Negative robust losses, such as the Hampel loss, introduce one or more scale parameters which are difficult to select or optimize in conjunction with the changing representation learned by the network. Like Ruff et al. (2018), we observed that the hinge loss was difficult to optimize and resulted in poorer performance. The inverse squared norm loss instead is bounded from below and smooth, which are crucial properties for losses used in deep learning (Goodfellow al. 2016), and ultimately performed the best while remaining conceptually simple.
Following our insights on the connection between Deep SVDD and entropy minimization from Section 3.1 we may interpret our Deep SAD objective as modeling the latent distribution of normal data, Z = =[{Y=+1}, to have low entropy, and the latent distribution of anomalies, 2 = ----1),, 1). to have high entropy. Minimizing the distances to the center (i.e., minimizing the empirical variance) for the mapped points of labeled normal samples (j- = +1) induces latent distribution with low entropy for the normal data. In contrast, penalizing low variance via the inverse squared norm loss for the mapped points of labeled anomalies (ÿ- = 1) induces latent distribution with high entropy for the anomalous data. That is, the network must attempt to map known anoma- lies to some heavy-tailed distribution. We argue that such a model better captures the nature of anomalies, which can be thought of as being generated from an infinite mixture of distributions that are different from the normal data distribution, indubitably distribution that has high entropy. Our objective notably does not impose any cluster assumption on the mmomaly-generatinn
distribution x - as is typically made in supervised or sumi-super classification approaches (Zhu) 2005 [Chpellle et al., 2009). We can express this interpretation in terms of principle 2 with an entropy regularization objective on the latent distribution:
I(X: + B (H(ZT)- - -H(Z))). p(=
max
(8)
To maximize the mutual information I(X; 2), Deep SAD also relies on autoencoder pre-training Vincent et al., 2008; Ruff et al. 2018).
4 EXPERIMENTS
We evaluate Deep SAD on MNIST, Fashion-MNIST. and CIFAR-10 well as on classic AD bench- mark datasets. We compare to shallow, hybrid, as well as deep unsupervised, semi-supervised and supervised competitors. We refer other recent works (Ruff et al. 2018 Golan & El- Yaniv 2018 Hendrycks et al. 2019a) for further comparisons between unsupervised deep AD methods
"To ensure numerical stability, we add a machine epsilon (eos 10 to the denominator of the inverse. -SAD--
"Our code is availahle at: httpp/////giitub..ooo////aaa/a//
5

---

Published as a conference paper at ICLR 2020
4.1 COMPETING METHODS
We consider the OC-SVM (Schölkopf et al. 2001 and SVDD TTxx & Duin 2004 with Gaussian kernel (which in this case are equivalent), Isolation Forest (Liu et al. 2008). and KDE (Parzen 1962) for shallow unsupervised baselines. For deep unsupervised competitors, we consider well- established (convolutional) autoencoders and the state-of-the-art unsupervised Deep SVDD method
RRuff et al. 2018). To avoid confusion, we note again that some literature (Song et al. 2017: Cha- lapathy & Chawla 2019) refer to the methods above as being "semi-supervised if they are trained on only labeled normal samples. For general semi-supervised AD approaches that also take advan- tage of labeled anomalies, we consider the state-of-the shallow SSAD method (Görnitz et al.)
2013) with Gaussian kernel. As mentioned earlier, there are no deep competitors for general semi- supervised AD that are applicable to general data types. To get comprehensive comparison we therefore introduce a novel hybrid SSAD bascline that applies SSAD to the latent codes of autoen- coder models. Such hybrid methods have demonstrated solid performance improvements over their raw feature counterparts on high-dimensional data (Erfani et al.) 2016) Nicolau et al. 2016). We also include such hybrid variants for all unsupervised shallow competitors. To also compare
deep semi-supervised learning method that targets classification as the downstream task, we add the well-known Semi-Supervised Deep Generative Model (SS-DGM) (Kingma al. 2014) where we use the latent class probability estimate (normal vs. anomalous) as the anomaly score. To complete the full learning spectrum, we also include a fully supervised deep classifier trained on the binary cross-entropy loss.
In our experiments we deliberately grant the shallow and hybrid methods an unfair advantage by selecting their hyperparameters to maximize AUC on a subset (10%) of the test set to minimize hyperparameter selection issues. To control for architectural effects between the deep methods, we always use the same (LeNet-type) deep networks. Full details on network architectures and hyperparameter selection can be found in Appendices [D and E Due to space constraints, in the main text we only report results for methods which showed competitive performance and defer results for the underperforming methods in Appendix F
4.2 EXPERIMENTAL SCENARIOS ON MNIST, FASHION-MNIST, AND CIFAR-10
Semi-supervised anomaly detection setup MNIST, Fashion-MNIST, and CIFAR- 10 all have ten classes from which we derive ten AD setups on each dataset following previous works (Ruff al.] 2018 Chalapathy et al. 2018 Golan & Yaniv 2018). In every setup, we set one of the ten classes to be the normal class and let the remaining nine classes represent anomalies. We use the original training data of the respective normal class as the unlabeled part of our training set. Thus we start with a clean AD setting that fulfills the assumption that most (in this case all) unlabeled samples are normal. The training data of the respective nine anomaly classes then forms the data pool from which we draw anomalies for training to create different scenarios. We compute the commonly used AUC measure the original respective test sets using ground truth labels to make quantitative comparison, i.e. ū = +1 for the normal class and ;j -1 for the respective nine anomaly classes We rescale pixels to [0, 1] via min-max feature scaling the only data pre-poccessing step.
Experimental scenarios We examine three scenarios in which we vary the following three ex- perimental parameters: (i) the ratio of labeled training data 7r (ii) the ratio of pollution 7 in the unlabeled training data with (unknown) anomalies, and (iii) the number of anomaly classes ky in- cluded in the labeled training data.
(i) Adding labeled anomalies In this scenario, we investigate the effect that including labeled anomalies during training has on detection performance to see the benefit of a general semi- supervised AD approach over other paradigms. To do this we increase the ratio of labeled training data " = m/(n+m) by adding more and more known anomalies ....... Hmm with ū, = -1 to the training set. The labeled anomalies are sampled from one of the nine anomaly classes (k: = 1). For testing, we then consider all nine remaining classes as anomalies, i.e. there are eight novel classes at testing time. We do this to simulate the unpredictable nature of anomalies. For the unlabeled part of the training set, we keep the training data of the respective normal class, which we leave unpolluted in this experimental setup, i.e. 7p 0. We iterate this training set generation process per AD setup always over all the nine respective anomaly classes and report the average results over the ten AD setups nine anomaly classes, i.e. over 90 experiments per labeled ratio 7/.

---

Published as a conference paper at ICLR 2020
(ii) Polluted training data Here investigate the robustness of the different methods in- creasing pollution ratio "p of the training set with unlabeled anomalies. To do so we pollute the unlabeled part of the training set with anomalies drawn from all nine respective anomaly classes in each AD setup. We fix the ratio of labeled training samples at " = 0.05 where we again draw sam- ples only from k 1 anomaly class in this scenario. We repeat this training set generation process per AD setup over all the nine respective anomaly classes and report the average results over the resulting 90 experiments per pollution ratio 7pp We hypothesize that learning from labeled anoma- lies in semi-supervised AD approach alleviates the negative impact pollution has on detection performance since similar unknown anomalies in the unlabeled data might be detected.
(iii) Number of known anomaly classes In the last scenario, we compare the detection perfor- mance at various numbers of known anomaly classes. In scenarios (i) and (ii), we always sample labeled anomalies only from one out of the nine anomaly classes (Az 1). In this scenario, we now increase the number of anomaly classes ky included in the labeled part of the training set. Since have a limited number of anomaly classes (nine) in each AD setup, expect the supervised classifier to catch up at some point. We fix the overall ratio of labeled training examples again at N 0.05 and consider pollution ratio of % 0.1 for the unlabeled training data in this scenario. We repeat this training set generation process for ten seeds each of the ten AD setups and report the average results over the resulting 100 experiments per number kj. For each seed, the kz classes are drawn uniformly at random from the nine respective anomaly classes.
MNIST
AUC
1
0.8
Method
Fashion- MNIST
OC-SVM Raw
OC-SVM Hybrid
AUC
Deep SVDD
SSAD Raw
0.6
SSAD Hybrid
CIFAR- 10
Supervised
ACC
Deep SAD
0.0
0.05
0.2
Ratio of labeled anomalies " in the training set.
Figure 2: Results of scenario (i), where we increase the ratio of labeled anomalies ? the training set. We report avg. AUC with st. dev. over 90 experiments at various ratios n A "x" indicates statistically significant (a = 0.05) difference between the 1 and 2nd best method.
Results The results of scenarios (iiiiii) are shown in Figures In addition to the avg. AUC
with st. dev., we report the outcome of Wilcoxon signed-rank tests (Wilcoxon) 1945) applied to the first and second best performing method to indicate statistically significant (a- 0.05) differences in performance. Figure 2 demonstrates the benefit of our semi-supervised approach to AD espe- cially on the most complex CIFAR-I0 dataset, where Deep SAD performs best. Figure 2 moreover confirms that a supervised classification approach is vulnerable to novel anomalies at testing time when only little labeled training data is available. In comparison, Deep SAD generalizes to novel anomalies while also taking advantage of the labeled examples. Note that our novel hybrid SSAD baseline also performs well. Figure [egure3sshow shows that the detection performance of all methods decreases with increasing data pollution. Deep SAD proves to be most robust again especially Finally, Figure shows that the more diverse the labeled anomalies in the training set, the better the detection performance becomes. We can again see that the supervised method is very sensitive to the number of anomaly classes but catches up at some point as suspected. This does not oceur with CIFAR-10, however, where n 0.05 labeled training samples seems to be insufficient for classification. Overall, we see that Deep SAD is particularly beneficial on the more complex data.
CIFAR- 10.

---

In addition, we experiments under the same experimental settings while varying the dimen- sion d {24 ... 2n} of the output space Deep SAD Published as a conference paper at ICLR 2020
MNIST
AUC
0.9
0.7
Method
Fashion MNIST
OC-SVM Row
1.0
OC-SVM Hybrid
AUC
Deep SVDD
SSAD Raw
0.4
SSAD Hybrid
CIFAR 10
0.9
Supervised
AUC
Deep SAD
0.5
0.0
0.01
0,05
0.2
Pollution ratio " of the unlabeled training data
Figure 3: Results of scenario (ii), where we pollute the unlabeled part of the training set with (un- known) anomalies. We report avg. AUC with st. dev. over 90 experiments at various ratios 7pp A "x" indicates statistically significant (a 0.05) difference between the 1st and 2dd best method.
MNIST
1.0
I
AUC
0.7
Method
Fashion-MNIH
OC-SVM Row
1.0
1
OC-SVM Hybrid
AUC
0.8
Deep SVDD
0.6
SSAD Raw
SSAD Hybrid
CIFAR-10
Supervised
ADC
Deep SAD
0.7
Number of known anomaly classes kj
Figure 4: Results of scenario (iii), where we increase the number of anomaly classes ky included in the labeled training data. We report avg. AUC with st. dev. over 100 experiments for various k;. A "*" indicates a statistically significant (a = 0.05) difference between the 1st and 2nd best method.
4.3 SENSITIVITY ANALYSIS
We run Deep SAD experiments the ten AD setups described above on each dataset for 77 e (10-2 ..... 102) analyze the sensitivity of Deep SAD with respect to the hyperparame- ter 1 0. In this analysis, we the experi- mental parameters to their default, ? = 0.05, 7p 0.1, and A 1, and again iterate all nine anomaly classes in every AD setup. The results shown in Figure(Jsuggeest that Deep SAD is fairly robust against changes of the hy- perparameter 7)
1.0
0.9
0.8
0.7
MNIST
Pishioe PInlso-MNNST MNIST
X
CIFAR:
0.01
0.1
1.0
10.0
100.0
7
Figure 5: Deep SAD sensitivity analysis w.r.t. 7, We report avg. AUC with st. dev. over 90 experi- ments for various values of hyperparameter
Z C ddd to infer the sensitivity with respect the representation dimensional- ity, where we keep 7 = 1. The results are given in Figure in Appendix A There we also com-
8

---

Published as a conference paper at ICLR 2020
pare to our hybrid SSAD baseline, which was the strongest competitor. Interestingly observe that detection performance increases with dimension d, converging to an upper bound in performance. This suggests that would want to set large enough to have sufficiently high mutual information [[;; 2) before compressing to a compact characterization.
4.4 CLASSIC ANOMALY DETECTION BENCHMARK DATASETS
In a final experiment, we also examine the detection performance of the various methods on some well-cstablished AD benchmark datasets (Rayanal 2016). We run these experiments to evaluate the deep versus the shallow approaches on non-image datasets that are rarely considered in deep AD literature. Here we observe that the shallow kernel methods seem to have slight edge the relatively small, low-dimensional benchmarks. Nonetheless, Deep SAD proves competitive and the small differences observed might be explained by the advantage grant the shallow methods in their hyperparameter selection. We give the full details and results in Appendix B
Our results and other recent works (Ruff al., 2018 Golan & El-Yamiv, 2018; Hendrycks et al. 2019a) overall demonstrate that deep methods are especially superior on complex data with hierar- chical structure. Unlike other deep approaches (Ergen et al., 2017 Kiran et al. 2018, Min et al.) 2018 Deecke et al., (2018) Golan & EI- Y (2018), however, our Deep SAD method is not domain or data-type specific. Due its good performance using both deep and shallow networks we expect Deep SAD extend well to other data types.
CONCLUSION AND FUTURE WORK
In this work introduced Deep SAD, deep method for general semi-supervised anomaly detec- tion. Our method is generalization of the unsupervised Deep SVDD method (Ruff et al. (0188) to the semi-supervised setting. The results of our experimental evaluation suggest that general semi- supervised anomaly detection should always be preferred whenever some labeled information on both normal samples anomalies available.
Moreover, we formulated an rnformmtiion-hhereeee framework for deep anomaly detection based on the Infomax principle. Using this framework, we interpreted our method as minimizing the entropy of the latent distribution for normal data and maximizing the entropy of the latent distribution for anomalous data. We introduced this framework with the aim of forming basis for new methods as well as rigorous theoretical analyses in the future, e.g. studying deep anomaly detection under the rate-distortion curve (Alemi et al. 2018).
ACKNOWLEDGMENTS
LR acknowledges support by the German Ministry of Education and Research (BMBF) in the project ALICE III (011SI 8049B). MK and RV acknowledge support by the German Research Foundation (DFG) award KL 2698/2-1 and by the German Ministry of Education and Research (BMBF) awards 03110023A, 011118051, and 03180700 AB is grateful for support by the National Research Foundation of Singapore, STEE-SUTD Cyber Security Laboratory, and the Ministry of Educa- tion. Singapore, under its program 10E2011-T2-2-11 man Ministry of Education and Research (BMBF) through the Berlin Center for Machine Learning (011S18037D, KRM acknowledges partial financial support by the German Ministry of Education and Research (BMBF) under grants 011S 400134,, 01IS 11025,, DIIS180377 O1GQ1115 and 01GQ0850; Deutsche Forschungsgesellschaftt (DFG) under grant Math+, EXC 2046/1, project-ID 390685689, and by the Technology Promotion (ITTP) grant funded by the Korea government (No. 2017-0-00451, No. 2017-0-01779).
NG acknowledges support by the Ger-
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep rep- resentations. Journal of Machine Learning Research, 19(1):1947-1980 2018.
Samet Akcay, Amir anomaly detection via adversarial training. In ACCV, pp. 622-637, 2018.
and Toby P Breckon. GANomaly: Semi-supervised

---

Published as a conference paper at ICLR 2020
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In ICLR, 2017.
Alexander A Alemi, Ben Poole, Ian Fischer, Joshua V Dillon, Rif A Saurous, and Kevin Murphy. Fixing a broken ELBO. ICML, volume 80, pp. 159-168, 2018.
Jerone T A Andrews, Edward Morton, and Lewis D Griffin. Detecting Anomalous Data Using Auto-Encoders. IJLLC, 6(1):21, 2016.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization.. ICML, volume 80, pp. 244-253, 2018.
Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to under- stand kernel learning. In JCML, pp- 540-548, 2018.
Anthony J Bell and Terrence J Sejnowski. An information-maxxmmzzation approach to blind separa- tion and blind deconvolution. Neural computation, ((6):112--1555) 1995.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Semi-supervised novelty detection. Journal of Machine Learning Research, II(Nov)297333000, 2010.
Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: A survey.
arXiv:1901.03407, 2019.
Raghavendra Chalapathy, Aditya Krishna Menon, and Sanjay Chawla. Anomaly detection using one-class neural networks. arXiv:1802.06360, 2018.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly Detection: A Survey. ACM Com- puting Surveys, 1153)11-58, 2009.
Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-supervised learning. IEEE Trans- actions Neural Networks, 20(3):542 5:222242 2009.
Jinghui Chen, Saket Sathe, Charu C Aggarwal, and Deepak S Turaga. Outlier Detection with Au- toencoder Ensembles. In SDM, pp. 90-98, 2017.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS, pp. 2172-2180, 2016.
Nadav Cohen, Or Sharir, and Amnon Shashua. On the expressive power of deep learning: A tensor analysis. In COLT, volume 49, pp. 698-728, 2016.
Thomas M Cover and Joy A Thomas. Elements of Information Theory. John Wiley & Sons, 2012.
Zihang Dai, Zhilin Yang. Fan Yang, William W Cohen, and Ruslan Salakhutdinov. Good semi- supervised learning that requires a bad gan. In NIPS, pp. 6510-6520, 2017.
Lucas Deecke, Robert A Vandermeulen, Lukas Ruff, Stephan Mandt, and Marius Kloft. Image anomaly detection with generative adversarial networks. In ECML-PKDD, 2018.
François Denis. PAC learning from positive statistical queries. In International Conference on Algorithmic Learning Theory, pp. 1 112-126, 1998.
Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In COLT, volume 49, pp. 907-940, 2016
Sarah M Erfani, Sutharshan Rajasegarar, Shanika Karunasekera, and Christopher Leckie. High- dimensional and large-scale anomaly detection using linear one-class SVM with deep learning. Pattern Recognition, 18:1211134, 2016.
Tolga Ergen, Ali Hassan Mirza, and Suleyman Serdar Kozat. Unsupervised and semi-supervised anomaly detection with LSTM neural networks. arXiv: 1710.02710 2017.
10

---

Published as a conference paper at ICLR 2020
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, pp. 249-256, 2010.
Izhak Golan and Ran EI- Yaniv. Deep anomaly detection using geometric transformations. NeurIPS, pp. 9758-99888 9769, 2018.
In
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
w/www. www.ceeeppeermmmmpoooooo
Nico Görnitz, Marius Kloft, Konrad Rieck, and Ulf Brefeld. Toward supervised anomaly detection. Journal of Artificial Intelligence Research, 46:235-262 2013.
Simon Hawkins, Hongxing He, Graham Williams, and Rohan Baxter. Outlier Detection Using Replicator Neural Networks. In DaWaK, volume 2454, pp. 170-180, 2002.
Dan Hendrycks, Mantas Mazeika, and Thomas G Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019a.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. NeurlPS, pp. 15637-15648, 2019b.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the Dimensionality of Data with Neural Networks. Science, 313(5786):504-507, 2006.
Devon Hjelm, Alex Fedorov, Samuel Marccilnn,, Karan Grewal, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximiza- tion. In ICLR, 2019.
Matthew D Hoffman and Matthew J Johnson. ELBO surgery: yet another way to carve up the varia- tional evidence lower bound. In NIPS Workshop Advances in Approximate Bayesian Inference, 2016.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML, pp.
2015.
Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information distillation for unsupervised image segmentation and clustering. 187.00680..06653 2018.
JooSeuk Kim and Clayton D Scott. Robust kernel density estimation. Journal of Machine Learning Research, 235$52)255,, 2012.
Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2013.
Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Supi-supervised learning with deep generative models. In NIPS, pp. 3581-3589, 2014.
B Kiran, Dilip Thomas, and Ranjith Parakkal. An overview of deep learning based methods for unsupervised and semiisupervised anomaly detection in videos. Journal of Imaging, 4(2):36, 2018.
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robed Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10(1):1096 2019.
Ralph Linsker. Self-organization in perceptual network. IEEE Computer, 21(3):105-117 117, 1988.
Fei T Liu, Kai M Ting, and Zhi-Hua Zhou. Isolation Forest. In ICDM, pp. 413 422, 2008
Yi Liu and Yuan F Zheng. Minimum enclosing and maximum excluding machine for pattern de- scription and discrimination. In ICPR, pp. 129-132, 2006.
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. In ICI.R 2014.
11

---

Published as a conference paper at ICLR 2020
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, lan Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv:1511.066w 2015.
Erxue Min, Jun Long, Qiang Liu, Jianjing Cui, Zhiping Cai, and Junbo Ma. SU-IDS: A semi- supervised and unsupervised framework for network intrusion detection. In International Con- ference on Cloud Computing and Security, pp. 322-334, 2018.
Grégoire Montavon, Mikio Braun, and Klbes-Robert Müller. Kernel analysis deep networks. Journal of Machine Learning Research, 12(Sep):2563-25811 2011.
Mary M Moya, Mark W Koch, and Larry D Hostetler. One-class classifier networks for target recognition applications. In Proceedings World Congress on Neural Networks, pp. 797-801, 1993.
Jordi Muñoz-Mari, Francesca Bovolo, Luis Gómez-Chovar Lorenzo Bruzzone, and Gustavo Camp- Valls. Semi-Supervised One-Class Support Vector Machines for Classification of Remote Sensing Sata. IEEE Transactions on Geoscience and Remote Sensing, 48(8))1188-3193 2010.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization in deep learning. In NIPS, pp. 5947-5956, 2017.
Miguel Nicolau, James McDermott, et al. A hybrid autoencoder and density estimation model for anomaly detection. In International Conference on Parallel Problem Solving from Nature, pp. 717-726, 2016.
Augustus Odena.
Semi-supervised
learning
with generative adversarial networks.
arXiv:1606.01583, 2016.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin D Cubuk, and Ian J Goodfellow. Realistic evalu- ation of deep supe-supe learning algorithms. In Neur/PS, pp. 3235-33333330 1336-324,,11 2018.
Guansong Pang, Chunhua Shen, and Anton van den Hengel. Deep anomaly detection with deviation networks. In KDD, pp. 353-362, 2019.
E. Parzen. On Estimation a Probability Density Function and Mode. The annals of mathematical statistics, 33(3):1065-1076, 1962.
Marco AF Pimentel, David A Clifton, Lei Clifton, and Lionel Tarassenko. A review of novelty detection. Signal Processing, 99:215-249, 2014.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl Dickstein. On the expressive power of deep neural networks. In ICML, volume 70, pp. 2847-2854, 2017.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. In NIPS, pp. 3546-340 -5544,355,,,4 2015.
Shebuti Rayana. ODDS library, 2016. URL http::/ http///ddds..m/ cs wtw.ystooooooooccooom..... edu
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In ICML, volume 32, pp. 1278 1288, 2014.
Lukas Ruff, Robert A Vandermeulen, Nico Görnitz, Lucas Deecke, Shoaib A Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In ICML, volume 80, pp. 4390-4399, 2018.
Lukas Ruff, Yury Zemlyanskiy, Robert Vandermeulen, Thomas Schnake, and Marius Kloft. Self- attentive, multi-context one-class classification for unsupervised anomaly detection text. In ACL, pp. 4061-4071, 2019.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by eиror propagation. In Parallel Distributed Processing Explorations in the Microstructure of Cognition, chapter 8, pp. 318-362, MIT Press, 1986.
Mayu Sakurada and Takehisa Yairi. Anomaly detection using autoencoders with nonlinear dimen- sionality reduction. In Proceedings of the 2nd MLSDA Workshop, pp. 4, 2014.
12

---

Published as a conference paper at ICLR 2020
Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Bren- dan Damiel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. In ICLR, 2018.
Bernhard Schölkopf and Alex J Smola. Learning with Kernels. MIT press, 2002.
Bernhard Schölkopf, John C Platt, John Shawe- Mayaor, Alex J Smola, and Robert Williamson. Estimating the Support of High-Dimensional Distribution. Neural computation, 13(7):1443- 1471, 2001.
Clayton D Scott and Robert D Nowak. Learning minimum volume sets. Journal of Machine Learn- ing Research, 7(Apr):665-704, 2006.
Claude Elwood Shannon. mathematical theory of communication. Bell system technical journal, 27(3):379-423, 1948.
Ravid Shwartz-Ziv and Naftali Tishby, Opening the black box of deep neural networks via informa- tion. 1733:07003.00100 2017.
Noam Slonim, Gurinder Singh Atwal, Gašper Tkačik, and William Bialek. Information-based clus- tering. Proceedings of the National Academy of Sciences, 102(51):18297-18300 2005.
Hongchao Song, Zhuqing Jiang, Aidong Men, and Bo Yang. A hybrid semi-supervised anomaly detection model for high-dimensional data. Computational Intelligence and Neuroscience, 2017.
Ingo Steinwart, Don Hush, and Clint Scovel. A classification framework for anomaly detection. Journal of Machine Learning Research, (tteb):211-2322 2005.
David M J Tax and Robert P W Duin. Support Vector Data Description. Machine learning, 54(1): 45 45,66,22 2004.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5, 2015.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. In The 37th annual Allerton Conference on Communication, Control and Computing, pp. 368-377, 1999.
Robert A Vandermeulen and Clayton Scott. Consistency of robust kernel density estimators. In COLT, pp. 591.591,220 2013.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. ICML, pp. 1096-1 103, 2008.
Jigang Wang, Predrag Neskovic, and Leon N Cooper. Pattern classification via single spheres. In International Conference on Discovery Science, pp. 241-252. Springer, 2005.
Thomas Wiatowski and Helmut Böleskei. A mathematical theory of deep convolutional neural networks for feature extraction. IEEE Transactions on Information Theory, 6443):885-1186, 2018.
Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-33, 1945.
Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang, Deep structured energy based models for anomaly detection. In ICML, volume 48, pp. 1100-1109, 2016.
Bangzuo Zhang and Wanli Zuo. Learning from positive and unlabeled examples: A survey. In Proceedings of the IEEE International Symposium on Information Processing, pp. 650-654, 2008.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLRR 2017.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Info VAE: Information maximizing variational autoencoders.
2017.
Xiaojin Zhu. Semi-super learning literature survey. Computer Sciences TR 1530, University of Wisconsin Madison, 2005.
13

---

Published as a conference paper at ICLR 2020
A
ADDITIONAL RESULTS ON MNIST, FASHION-MNIST, AND CIFAR-10
A.1
SENSITIVITY ANALYSIS W.R.T REPRESENTATION DIMENSIONALITY
MNIST
Pashico-MNIST
CIFAR-10
1.0
SSAD Hyhrid
Deep SAD
0.9
AUC
0.8
0.7
32
129
512
16
64
256
32
128
Figure 6: Sensitivity analysis the network representation dimensionality d for Deep SAD method and the closest competitor hybrid SSAD. We report avg. AUC with st. dev. over 90 experi- ments for various values of d.
A.2 AUC SCATTERPLOTS OF BEST VS. SECOND BEST METHODS ON CIFAR-10
We provide AUC scatterplots in Figures 7 of the best (1s) vs. second best (2nd) performing meth- ods in the experimental scenarios (ii-(ii) on the most complex CIFAR-10 dataset. If most points fall above the identity line, this very strong indication that the best method indeed significantly outperforms the second best, which often is the case for our Deep SAD method.
AUC
AUC
AUC
1.0
1.0
1.0
14 SSAD Row
1d Deep SAD
Deep SAD
0.8
0.8
0.8
2od Deep SAD
2nd SSAD Hybrid
200
SSAD Hybeid
(a) 7 = 0.01
(b) 7: 0.05
(c) = 0.1
AUC
1.0
Desp SAD
0.8
0.6
0.3
0.6
0.7
11.
0.9
1.0
Supervised
(d) 0.2
Figure 7: AUC scatterplots of best (19) second best (2nd) performing methods in experimental scenario (i) on CIFAR-10, where we increase the ratio of labeled anomalies n in the training set.
14

---

Published as a conference paper at ICLR 2020
AUC
AUC
AUC
1.0
1.0
1.0
1 Deep SAD
1 Deep SAD
1 Doep SAD
0.8
0.8
0.8
0.8
0.6
0.5
0.6
0.7
0.8
0.9
0.5
3.6
0.7
11.
0.9
0.5
0.6
I1.7
0.8
0.9
1.0
SSAD Raw
2 SSAD Raw
2d SSAD
Raw
(a) PpP
0
(b) 7p
0.01
(c) TPp
0.05
AUC
AUC
Deep SAD
0.8
Doep
0.6
0.5
0.6
0.7
0.8
0.9
1.0
0.7
1.
2ul SSAD Row
2nd SSAD Hybrid
(d) 7pp = 0.1
(c) 79 0.2
Figure 8: AUC scatterplots of best (1s) vs. second best (2nd) performing methods in experimental scenario (ii) on CIFAR-10, where we pollute the unlabeled part of the training set with (unknown) anomalies various ratios 7/p.
AUC
AUC
AUCC
1.0
14 Deep SAD
Deep SAD
Deep SAD
0.8
0.8
0.8
0.6
0.6
0.9
0.5
0.8
20d SSAD Raw
24d SSAD Raw
20d SSAD Raw
(a) ky
(b) ku
(c) =
AUC
1f Deep SAD
0,8
0.6
0.3
0.6
0.8 0.9
2nd SSAD Raw
(d) k = 5
Figure 9: AUC scatterplots of best (1*) vs. second best (2nd) performing methods in experimental scenario (iii) on CIFAR-10, where we increase the number of anomaly classes ky included in the labeled training data.
15

---

Published as a conference paper at ICLR 2020
B RESULTS ON CLASSIC ANOMALY DETECTION BENCHMARK DATASETS
In this experiment, we examine the detec- tion performance on some well-established AD benchmark datasets (Rayanal (0166 listed in T We do this evaluate the deep against the shallow approaches also on non mon-image, tab- ular datasets that are rarely considered in the deep AD literature. For the evaluation, we con- sider random train-to-test set splits of 60:40 while maintaining the original proportion of anomalies each set. We then run experiments for 10 seeds with " 0.01 and % = 0, i.e. 1% of the training set are labeled anomalies and the unlabeled training data is unpolluted. Since there are specific different anomaly classes in these datasets, we have k = 1. We standardize features to have zero mean and unit variance as the only pre-processing step.
Table 1: Anomaly detection benchmarks.
Dataset
N
D
#outliers (%)
arrhythmia
452
274
66 (14.6%)
cardio
1,831
21
176 (9.6%)
satellite
6,435
36
2,036 (31.6%)
satimage-2
5,803
36
71 (1.2%)
shuttle
49,097
3,511 (7.2%)
thyroid
3,772
93 (2.5%)
Table 2 shows the results of the competitive methods. We observe that the shallow kernel methods seem to perform slightly better on the rather small, low-dimensional benchmarks. Deep SAD proves competitive though and the small differences might be explained by the strong advantage we grant the shallow methods in the selection of their hyperparameters. We provide the complete table with the results from all methods in Appendix F
Table 2: Results on classic AD benchmark datasets in the setting with no pollution 7,- 0 and a ratio of labeled anomalies of 7z = 0.01 in the training set. We report avg. AUC with st. dev. computed over 10 seeds. A "*" indicates a statistically significant (a 0.05) difference between 1st and 2nd
OC-SVM
OC-SVM Hybrid
Deep SVDD
SSAD Raw
SSAD Hybrid
Supervised Classifier
Deep SAD
Dataset
Raw
arrhythmia
84.513.
76.716.2
74.6-9..0
86.7.14.0'
78.345.11
39.2.19.5
75.9
cardio
98.5
82.8.19.3
84.8-3.66
98.8-10.3
86.3155.8
83.2419.6
95.0
satellite
95.1+0.2
68,6=t.88
79.8=4.1
96.2:10.*
86.942.8
87.242.1
91.5#1111
satimage-2
99.4+10.8
96.7=2.1
98.3=1.4
99.9110.1
96.8+2.1
99.911.11
99.910.11
shuttle
99.4:0.9
94.119.5
86.3=7.5
99.6110.5
97.7+1.0
95.118.0
98.42:099
thyroid
98.3+00.9
91.2++4.0
72.0=9977
97.9++1.9
95.3+...1
97.8+22.6
98.6-+0.9
16

---

Published as a conference paper at ICLR 2020
C OPTIMIZATION OF DEEP SAD
Our Deep SAD objective 7 is generally non-convex in the network weights W which usually the case in deep learning. For computationally efficient optimization, we rely on (mini-batch) SGD to optimize the network weights using backpropagation. For improved generalization, we add L weight decay regularization with hyperparameter À > 0 to the objective. Algorithm summarizes the Deep SAD optimization routine.
Algorithm Optimization of Deep SAD Input:
Unlabeled data: Fi,..., . In
Labeled data: (æ/ 1/), (I'm V')
Hyperparameters: 7),
SGD learning rate: 8
Output:
Trained model: W
I: Initialize:
Neural network weights: W Hypersphere center:
2: for each epoch do
3: for each mini-batch do
4: Draw mini-batch
5: W t W (W)(W;;)
6: end for
7:
end for
Using SGD allows Deep SAD to scale with large datasets the computational complexity scales linearly in the number of training batches and computations in each batch can be parallelized (e.g., by training GPUs). Moreover, Deep SAD has low memory complexity as a trained model is fully characterized by the final network parameters W and no data must saved or referenced for prediction. Instead, the prediction only requires forward pass the network which usually is just a concatenation of simple functions. This enables fast predictions for Deep SAD.
Initialization of the network weights W We establish autoencoder pre-training routine for initialization. That is, we first train an autoencoder that has an encoder with the same architecture as network ó on the reconstruction loss (mean squared error cross-entropy! After training, then initialize W with the converged parameters of the encoder. Note that this is in line with the Infomax principle for unsupervised representation learning (Vincent et al. 2008).
Initialization of the center c After initializing the network weights W, we fix the hypersphere center e the mean of the network representations that obtain from an initial forward pass on the data (excluding labeled anomalies). We found SGD convergence to be smoother and faster by fixing center e the neighborhood of the initial data representations also observed by by|Ruf' al. (2018). If sufficiently many labeled normal examples are available, using only those examples for mean initialization would be another strategy minimize possible distortions from polluted unlabeled training data. Adding center e as free optimization variable would allow a trivial "hypersphere collapse" solution for the fully unlabeled setting, i.e. for unsupervised Deep SVDD.
Preventing a hypersphere collapse A "hypersphere collapse" describes the trivial solution that neural network ø converges to the constant function @ = c. i.c. the hypersphere collapses a single point. Ruff et al. (018)) demonstrate theoretical network properties that prevent such col- lapse which we adopt for Deep SAD. Most importantly, network 6 must have no bias terms and no bounded activation functions. We refer to Ruff al. (2018) for further details. If there sufficiently many labeled anomalies available for training. however, hypersphere collapse is not a problem for Deep SAD due to the opposing labeled and unlabeled objectives.
17

---

Published as a conference paper at ICLR 2020
D NETWORK ARCHITECTURES
We employ LeNet-type convolutional neural networks (CNNs) on MNIST, Fashion-MNIST, and CIFAR-10, where each convolutional module consists of a convolutional layer followed by leaky ReLU activations with leakiness a 0.1 and with two modules, 8x (xx55)- -filterr followed by 4x (6x6)-fflters) and final dense layer of 32 units. On Fashion-MNIST, we employ a CNN also with two modules, 16x (5x5)--ilten) and 32x (5x5)- filters, followed by dense layers of 64 and 32 units respectively. On CIFAR-10, we employ CNN with three modules, 32x (5x 5))fiitrrrrs 64x (5x5)-filters and 128 (5x5)-filters, followed by final dense layer of 128 units.
On MNIST, we employ a CNN
On the classic AD benchmark datasets, we employ standard MLP feed-forward architectures. On arrhythmia, 3-layer MLP with 128-64-32 units. On cardio. satellite, satimage-2, and shuttle 3-layer MLP with 32-16-8 units. On thyroid a 3-layer MLP with 32-16-4 units.
For the (convolutional) autoencoders, we always employ the above architectures for the encoder networks and then construct the decoder networks symmetrically, where we replace max-pooling with simple upsampling and convolutions with deconvolutions.
E DETAILS ON COMPETING METHODS
The OC-SVM and SVDD are equivalent for the Gaussian/RBF kernel we em- ploy. As mentioned in the main paper, we deliberately grant the OC-SVM/SVDD an unfair advan- tage by selecting its hyperparameters to maximize AUC on subset (10%) of the test set to estab- lish a strong baseline. To do this, we consider the RBF scale parameter y E and select the best performing one. Moreover, we always repeat this over v-parameter {0.01, 0.05, 0.1, 0.5} and then report the best final result.
OC-SVM/SVDD
Isolation Forest (IF) We set the number of trees to t 100 and the sub-sampling size to 1b 256, as recommended in the original work (Liu et al. 2008).
Kernel Density Estimator (KDE) We select the bandwidth h of the Gaussian kernel from h E (20.5 21
2-} via 5-fold cross-validation using the log-likelihood score following (Ruff et al.
2018
We also deliberately grant the state-of-the-an cemi-supervised AD kernel method SSAD the unfair advantage of selecting its hyperparameters optimally to maximize AUC on subset (10%) of the test set. To do this, we again select the scale parameter & of the RBF kernel we use from y 2 .... 22) and select the best performing one. Otherwise we set the hyperparameters as recommend by the original authors to K = 1, K 1, Th = 1, and = 1 (Görnitz et al. 2013).
SSAD
(Convolutional) Autoencoder ((C)AE) To create the (convolutional) autoencoders, we symmet- rically construct the decoders w.r.t. the architectures reported in Appenidx D which make up the encoder parts of the autoencoders. Here, we replace max-pooling with simple upsampling and con- volutions with deconvolutions. We train the autoencoders on the MSE reconstruction loss that also serves as the anomaly score.
Hybrid Variants To establish hybrid methods, we apply the OC-SVM, IF, KDE, and SSAD as outlined above to the resulting bottleneck representations given by the respective converged autoen- coders.
Unsupervised Deep SVDD We consider both variants, Soft- boft-Boundary Deep SVDD and One-Class Deep SVDD as unsupervised baselines and always report the better performance as the unsupervised result. For Noft-Boundary Deep SVDD, we optimally solve for the radius R on every mini-bateh and run experiments for (0.01, 0.1}. We set the weight decay hyperparameter to À 10 For Deep SVDD, we always remove all the bias terms from a network to prevent a hypersphere collapse as recommended by the authors in the original work (Ruff et al.) 2018).
Deep SAD We set = 10-6 and equally weight the unlabeled and labeled examples by setting 17 1 if not reported otherwise.
SS-DGM We consider both the M2 and M1+M2 model and always report the better performing result. Otherwise we follow the settings recommended in the original work (Kingma al. 2014).
18

---

Published as a conference paper at ICLR 2020
Note that the latent class probability estimate (normal anomalous) of semi- cemi-supervised DGM as natural choice for the anomaly score, and not the reconstruction error as used for un- supervised autoencoding models such as the (convolutional) autoencoder consider. Such deep semi-supervised models designed for classification as the downstream task have no notion of out- of-distribution and again implicitly make the cluster assumption (Zhu, 2005 Chapelle et al. (009) we refer to. Thus, semi-supervised DGM also suffers from overfitting to previously seen anomalies training similar the supervised model which explains its bad AD performance.
Supervised Deep Binary Classifier To interpret AD as a binary classification problem, we rely on the typical assumption that most of the unlabeled training data is normal by assigning y = +1 to all unlabeled examples. Already labeled normal examples and labeled anomalies retain their assigned labels of ý +1 and = respectively. We train the supervised classifier on the binary cross- entropy loss. Note that in scenario (i), in particular, the supervised classifier has perfect, unpolluted label information but still fails generalize as there are novel anomaly classes at testing.
SGD Optimization Details for Deep Methods We use the Adam optimizer with recommended default hyperparameters (Kingma Ba 2015) and apply Batch Normalization (loffe Szegedy) 2013) in SGD optimization. For all deep approaches and all datasets, we employ two-phase ("searching" and "fine-tuning") learning rate schedule. In the searching phase we first train with a learning rate 10 for 50 epochs. In the fine-tuning phase train with 10 for another 100 epochs. We always use a batch size of 200. For the autoencoder, SS-DGM, and the supervised classifier, initialize the network with uniform Glorot weights (Glorot & Bengio (2000). For Deep SVDD and Deep SAD, we establish an unsupervised pre-training routine via autoencoder as explained in Appendix C where we the network be the encoder of the autoencoder that train beforehand.
F COMPLETE TABLES OF EXPERIMENTAL RESULTS
The following Tables 366 list the complete experimental results of all the methods in all our experi- ments.
19

---

Published as conference paper at ICLR 2020
Table 3: Complete results of experimental scenario (i), where increase the ratio of labeled anomalies y in the training st. dev. computed over 90 experiments at various ratios ?..
We report the avg. AUC with
OC-SVM
OC-SVM Hybrid
KDE Raw
KDE Hybrid
Deep SVDD
SSAD Raw
SSAD Hybrid
Deep SAD
Supervised ised Classifier
Data
71 Raw
Raw
Hybrid
CAE
SS-DGM
96.0:12.9
96.34:2.5
85.4:8.77
90.5:::.3
95.04333
87.8:5.66
92.955.7
92.8:4499
96.0:122.9
96.3:12255
42.81:499
.01
96.6.12.4
96.8.12.3
89.919.2
96.4
92.8
93.34-3.6
97.42200
92.245.6
96.7422.4
94.544.66
.10
90.7+4444
97.6+1.7
91.6#5,5
96.9+2.3
95,0=-4,7
87.24:566
97,8:1155
91.2+5.6
96.922244
95.624444
F-MNIST
.00
92.8+4.7
91.244.7
91.6+t.5
82.5
92.0+449
69.7
90.24:.88
89.2466.2
92.8+t-77
91.2:14.7
89.2
89.4-6600
.05
89.3-662
90.5=
71.4 12.7
90,5=6,55
76.8=13.2
85.5+7.11
91.0:5:.6
72.9+
91.3+6600
79.0=
20
82.018.00
89.716.66
74.7::
91.01:555
81.4::1..0
CIFAR-10
62.0=100.
63.8:9000
60.0=10.0
59.926.7
19.9
56.1+10.2
56.24:3.2
60.9994
62.0:::0.6
63.8+99.0
60.99944
.01
73.0:18.0
70.5:88.3
4..7
72.61744
$5.61.
73.3188.
77.917.2
63.5
20
.10
70.1
74.0=28.1
52.0
798827.1
67.729.6
67.4.18888
74.5
53.2.16.7
81.91..0
8..5

---

Published as conference paper at ICLR 2020
Table 4: Complete results of experimental scenario (ii), where pollute the unlabeled part of the training with (unknown) anomalies. We report the avg. AUC with st. dev. computed over 90 experiments at various ratios 7/p-
OC-SVM Raw
OC-SVM Hybrid
IF Raw
IF Hybrid
KDE Raw
KDE Hybrid
Deep SVDD
SSAD Raw
SSAD Hybrid
Deep SAD
Supervised Classifier
Data
7
CAE
SS-DGM
96.0:2299
96.3:::55
85.418.7
90.51:5.3
95.0::3.3
87.81:5.6
92.9-:5.77
92.814999
97,9-1188
97.41:200
92.2:1:5.6
96.7=224
94.5:1-4.6
.01
94.31.3.9
95.6.12..5
85.2.88.8
90.6.15.0
91.2.14.9
87.91..33
91.3.6.11
92.11.511
96.6.1244
95.2.12.3
92.0.16.0
95.5
91.5.15.99
.05
91.455.2
83.949.2
89.7-6.0
85.547.11
87.37700
87.247.1
89445.8
934433
89.5-1399
91.0++6.9
93.5-4.1
86.47..4
.10 88.8=6.0
91.455.1
82.3=9.5
88.2=6.5
82.1
85.9-6.6
83.748.4
66.5468
90.744444
86.0+-466
89.7+7.5
91.2=4.9
83,6#2..2
84.1=7.66 200
8.
10.7
85.3:7.9
17.4+10.99
82.6:28.6
18.6:100.3
8154844
87.44:566
82.1::544
87.4
86.6:6.6
79.7499.4
F-MNIST
.00
92,8=2477
91.2+:4.77
91.6#5.55
82.5=8.1
92.0#t4.9
69.7:1444
90.24::88
89.246.2
94.0:1444
90.5::599
71.412277
90.5=6,5
76.8=13.2
91.7.45.0
91.5455.5
7394124
87.2-7.11
.05
90,7=5,5
90.74-4.9
90.9=5.9
85.5=7.2
85,249.1
15.44229
61.6
80.6
88.3+662
82.7=7.88
71.9114.3
81.5=8.5
59.8:44466
.10
89.5=6.11
69.3+
90.2:6.3
85.577.7
81,8+11.2
718+1200
77.411111
76.247.3
85.6+7700
79.8:19.00
72.5
78.2=9.1
56.74-4.1
20
8..3
68.1
88.417.6
86.317.4
13.4:11..6
82.119.8
72.51:26
69.31..3
81948.1
74.3::10.6
70.8
74.8:19.4
53.91:299
CIFAR-10
.00
62.0:::00.6
63.8:19.0
10.0:10.00
59.9=6.77
19.9:111.7
16.1
56.24:3.2
60.9994
73.8+7.66
73.3+28.4
50,8++4.7
77.9=7.2
63.5
.01
61.9:1:0.6
63.8:19.3
59.9110.11
59.916.77
59.2:112.3
16.3:110.4
36.2111.1
60.5.994
73.0:18.0
72.8:8.11
$1.144.77
76,5
62.9:1..3
61.41:0077
59.6.6.4
58.1112.9
21
.10 60.8
62.9:8822
$8.8
59.1=6.6
57.3+13.5
54.911.11
55.4113.3
$8.6210..0
69.8488.4
09.3
50.5433.6
718827.0
60.6
.20
60.3.110.3
61.911.11
10.1
58.3
56.2.113.9
$4.2.11111
54.6.13.3
$7.0.110.6
67.8.88.66
67.918.11
50.1
68.5
58.5166.7

---

Published as conference paper at ICLR 2020
Table 5: Complete results of experimental scenario (iii), where increase the number of anomaly classes ky included in the labeled training data. We report the avg. AUC with st. dev. computed over 100 experiments at various numbers kj.
OC-SVM
OC-VVM Hybrid
KDE Raw
KDE Hybrid
Deep SVDD
SSAD Raw
SSAD Hybrid
Deep SAD
Supervised Clussifier
Data
Raw
Raw
Hybrid
CAE
SS-DGM
88.816.0
91445..1
82.3:9955
88.2:16.5
82.148.5
65.911.66
83.7=8.4
86.5:16.8
88.8116.0
91.4:::511
86.5416.88
90.7
86.0.14.6
89.7.17.5
91.2.14499
83.618.2
92.541..6
87.74:388
92.043366
93.9+3.3
89.8++3.3
94,944422
94,7+22.8
93.9+2.8
95.5:22.5
91.9:3300
96.7#22.3
97.3
96.9
F-MNIST
89.5:6.11
89.3
90.2:
85.5+7.7
8188111.
77.8=12.
774+1L1
76.2#1.3
89,3:6
89.3-6622
76.247.3
79.8:19.0
78.2449.1
87,8
80.1=1066 10.5
74.3+154
80,548.2
62.3+2.9
89.455.5
83,84994
17.5=14.7
83.947.4
67.3+33.0
91.2:14.8
8..8:17.7
13.9
87.316644
75.3
CIFAR-10
10.8
62.9
58.8+10.1
59.1
57.3+:3.5
$4.91111
55.4133.
18.6
60.8=10.7
62.9:18.2
58,6210.0
69.8 8.4
69.3:18..5
30.5:13.6
71.8:7700
60.6:88.3
73.0..7.1
72.3:17.55
75.2.16.4
22
73.8::6.6
73.3:27000
50.0=
77.545.9
62.7+688
75.1
74,2.16.5
$0.0.11.0
80.4.1466
60.9114.6

---

Published as a conference paper at ICLR 2020
Table 6: Complete results classic AD benchmark datasets in the setting with pollution Tp 0 and ratio of labeled anomalies of x 0.01 in the training set. We report the avg. AUC with st. dev. computed over 10 seeds.
OC-SVM Raw
OC-SVM Hybrid
Deep SVDD
SSAD Raw
SSAD Hybrid
Deep SAD
Supervised Clussifier
Data
CAE
SS-DGM
arrhythmmia
84.5+:3.9
76.716.2
74.0:1:55
74.6=9.0
86.71:4.0
78.3:15.1
50.3=9.8
75.948.7
39.2=9.5
cardio
98.51.0.3
828819.3
94.31120
8..8..3.6
98.8.10.3
86.31558
66.2.1:4.3
9..0.1..6
83.2 9.6
satellite
95.1--0.2
68.644888
80.0
79.8=4.11
96.2440.3
86.9428
574=6.4
91.541.11
87.2=2.1
satimago-2
99.4=0.8
96.7=2.1
99.9+00.0
98.3=1.4
99.9+t0.1
96.++2.1
99.2=0.6
99.9+0.1
99.9=0.1
shuttle
99.4::0.9
94.1+19.5
98.2:11.2
86,3=7.55
99.6=t0.5
97.721100
97.9=0.3
98.4+0999
95.1=8.0
thynoid
98.31.0.9
91.214.0
75.2:110.2
72.0:1..7
97.91:19
95.313..1
72.7.120
98.6.10.99
2.68
23


Reviews:
[{"review_id": "BylBYZcAFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\nThe paper proposes an abnormal detection (AD) framework under general settings where 1) unlabeled data, 2) labeled positive (normal) data, and 3) labeled negative (abnormal) data are available (with the last two optional), denoted as semi-supervised AD. Starting from the assumption that abnormal data are sampled from background unpredicted distribution, rather than “cluster” assumption, it is argued that conventional discriminative formulation is not applicable. Motivated by the recent deep AD methods (e.g., deep SVDD), the paper proposes to approach semi-supervised AD from the information theoretic perspective where 1) mutual information between raw data and learnt representation should be maximized (infomax principle), 2) entropy of labeled positive data should be minimized (“compactness” constraint), and 3) enrtropy of labeled negative data should be maximized to reflect the uncertainty assumption of anomaly. The solution is implemented by the encoder of a pre-trained autoencoder that is further fine tuned to enforce entropy assumption on all types of training data. Extensive experiments on benchmarks suggests promising results on the proposed framework versus other state-of-the-arts.\n\n[Comments]\nThe paper is well written and easy to follow (the presentation is especially pleasant to read). The problem is well defined and of interest to the community under fairly general and practical conditions. Despite the fact that the implementation is only marginally tweaked from previous work (deep SVDD), the theoretical motivation, nevertheless, is sound and well justified, and the empirical evaluation is extensive to reveal the behaviors of the proposed method. It would be better if complexity analysis can also be provided for all concerning methods. Overall, the value of the paper is worth circulation in the community. \n\n[Area to improve]\nThe manuscript could be further improved by exploring the training process more. In the current format, the solution follows the strategy of deep SVDD that learns the model in two separate stages: pre-training the autoencoder, and then fitting the encoder to enforce compactness and entropy minimization/maximization. What if these are implemented in an end-to-end fashion? Will this help to achieve a better result?   \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Byl7mCRpFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary of the work\n- The work proposes a new method two find anomaly (out of distribution) data when some labeled anomalies are given. \n- The authors apply information theory-derived loss based on that the normal (in distribution) data usually have lower entropy compared to that of the abnormal data. \n- The paper conducts extensive experiments on MNIST, Fashion-MNIST, and CIFAR 10, with varying the number of labeled anomlies.\n\nI think the paper is well written and the experiment seems to support the authors argument. Unfortunately, this field is not overlapped to my research field, and it is hard for me to judge this paper.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xf_hzkqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose in this paper a variant of Deep SVDD which brings semi-supervision to this model. The paper is well written and contains a thorough experimental evaluation (even disregarding the supplementary material). As far as I know, the proposed method is new and improve anomaly detection. The modification of Deep SVDD are in a way minimalistic, but they do the job.\n\nThe only negative aspects, in my opinion, is the \"information-theoretic view\" which is close to hand waving. The authors are indeed 1) disregarding the regularization term 2) considering an upper bound of the entropy 3) pretending the results on the pre-trained NN hold after post training. Putting everything together, I do not see how this reasoning could accepted. In fact, its extension to Deep SVDD is even more problematic as the discussion in the paper contradicts the reasoning. The authors emphasize the fact that anomalies should not fulfill the clustering assumption (which is indeed an important remark). But then the distribution of phi(x,W) cannot be approximated by a Gaussian for anomalies and thus the bound on the entropy is not valid. \n\nI strongly recommend to remove this part of the paper and to derive Deep SAD from Deep SVDD from heuristics consideration (which is fine!). This will provide an opportunity to remove the cute sentence \"We are happy to now introduce Deep SAD\".", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Issues raised by the reviewers have been addressed by the authors, and thus I suggest the acceptance of this paper.
Decision: Accept (Poster)