Deep Variational Semi-Supervised Novelty Detection
Authors: Tal Daniel
Thanard Kurutach
Aviv Tamar
Venue:  (2020)
Keywords: anomaly detection
semi-supervised anomaly detection
variational autoencoder

Abstract: In anomaly detection (AD), one seeks to identify whether a test sample is abnormal,  given a data set of normal samples.   A recent and promising approach to AD relies on deep generative models, such as variational autoencoders (VAEs),for unsupervised learning of the normal data distribution. In semi-supervised AD (SSAD), the data also includes a small sample of labeled anomalies. In this work,we propose two variational methods for training VAEs for SSAD. The intuitive idea in both methods is to train the encoder to ‘separate’ between latent vectors for normal and outlier data. We show that this idea can be derived from principled probabilistic formulations of the problem, and propose simple and effective algorithms.  Our methods can be applied to various data types, as we demonstrate on SSAD datasets ranging from natural images to astronomy and medicine, and can be combined with any VAE model architecture. When comparing to state-of-the-art SSAD methods that are not specific to particular data types, we obtain marked improvement in outlier detection.

Under review as a conference paper at ICLR 2020
DEEP VARIATIONAL SEMI-SUPERVISED NOVELTY DE- TECTION
Anonymous authors Paper under double-blind review
ABSTRACT
In anomaly detection (AD), one seeks identify whether test sample is abnor- mal, given a data set of normal samples. A recent and promising approach to AD relies on deep generative models, such as variational autoencoders (VAEs), for unsupervised learning of the normal data distribution. In semi-supervinn (SSAD), the data also includes a small sample of labeled anomalies. In this work. we propose two variational methods for training VAEs for SSAD. The intuitive idea in both methods to train the encoder to 'separate' between latent vectors for normal and outlier data. We show that this idea can be derived from principled probabilistic formulations of the problem, and propose simple and effective algo- rithms. Our methods can be applied to various data types, as we demonstrate on SSAD datasets ranging from natural images to astronomy and medicine, and can be combined with any VAE model architecture. When comparing to state-of sifee-of-tll of-the- art SSAD methods that are not specific to particular data types, we obtain marked improvement in outlier detection.
AD
INTRODUCTION
Anomaly detection (AD) - the task of identifying samples that are abnormal with respect to some normal data has important applications in domains ranging from health-care, to security, and robotics (Pimentel et al.] (014)). In its common formulation, training data is provided only for nor- mal samples, while at test time, anomalous samples need to be detected. In the probabilistic AD approach, a model of the normal data distribution is learned, and the likelihood of a test sample under this model is thresholded for classification as normal or not. Recently, deep generative mod- els such variational autoencoders (VAEs, Kingma & Welling 2013) and generative adversarial networks (Goodfellow et al. 2014) have shown promise for learning data distributions in AD (An & Cho 2015 (uh et al. (chleeg al., 2017 Wang al. 2017).
Here, we consider the setting of semi-supervised AD (SSAD), where in addition to the normal samples, small sample of labeled anomalies is provided (Görnitz et al. (2013). Most importantly, this sei is too small to represent the range of possible anomalies, making classification methods superriseeee
(either supervised or semi-supervised) unsuitable. Instead,
approaches are based on 'fixing' an unsupervised AD method to correctly classify the labeled anomalies, while still maintaining AD capabilities for unseen outliers (c.g., Görnitz et al. 2013 Muñoz-Marí al. 2010 Ruff al. 2019).
In this work, we present a variational approach for learning data distributions in the SSAD problem setting. We base our method on the VAE, and modify the training objective account for the labeled outlier data. We propose two formulations for this problem. The first maximizes the log- lig-iiklihood of normal samples, while minimizing the likeliiid of outliers, and we effectively optimize this objective by combining the standard evidence lower bound (ELBO) with the x upper bound (CUBO, Dieng al. 2017). The second method is based on separating the VAE prior between normal and outlier samples. Effectively, both methods have similar intuitive interpretation: they modify the VAE encoder to push outlier samples away from the prior distribution (see Figure []]. Importantly, method does not place any restriction the VAE architecture, and can be used to modify any VAE to account for outliers. As such, it can be used for general types of data.
We evaluate our methods in the comprehensive SSAD test-suite Ruff al. (2019), which includes both image data and dow-dimensional data sets from astronomy, medicine, and other domains, and report marked improvement in performance compared to both shallow and deep methods. In

---

Under review as a conference paper at ICLR 2020
addition, demonstrate the flexibility of our method by modifying conditional VAE used for generating sampling distributions for robotic motion planning (Ichter et al. 2018 to not generate way points that collide with obstacles.
normal
Figure 1: t-SNE of the latent space of a conventional VAE (left), and our proposed dual-prior VAE (right), trained on the Thyroid dataset, and evaluated on an unseen test sample. Note that latent vectors for outliers (red crosses) in DP-VAE are pushed away from the normal samples.
2 RELATED WORK
Anomaly detection, a.k.a. outlier detection or novelty detection, is an active field of research. Shal- low methods such one-class SVM (OC-SVM, Schölkopp et al. 2001) support vector data de- scription (SVDD, Tax & Duin 2004) have shown great success, but rely on hand-crafted features.
Recently, there is growing interest deep learning methods for AD (Chalapathy & Chawla 2019).
Most studies on remi-supervised AD also require hand designed features. proposed S*OC-SVM, modification of OC-SVM that introduces labeled and unlabeled samples, Görnitz et al. (2013) proposed an approach based on SVDD, and Blanchard et al.] (2010) base their approach on statistical testing. Deep SSAD has been studied recently in specific contexts such as videos (Kiran et al. 2018), network intrusion detection (Min et al.] 2018), or specific neural network architectures (Ergen et 2017). The most relevant prior work to our study is the recently proposed deep semi-supervised anomaly detection (Deep SAD) approach of Ruf et al. (2019). Deep SAD is general method based on deep SVDD (Ruff al.) 2018). which learns a neural-netword mapping of the input that minimizes the volume of data around predetermined point. While it has been shown to work on a variety of domains, Deep SAD must place restrictions on the network architecture, such bias terms (in all layers) and bounded activation functions, to prevent degeneration of the minimization problem to a trivial solution. The methods we propose here do not place any restriction the network architecture, and can be combined with any VAE model. In addition, show improved performance in almost all domains compared to Deep SAD's state-of-the-ar results.
Saaiuurres,Müoz---ee Muñoz et al. (2010)
While focus here is SSAD, for completeness review recent work on deep AD. Erfani et al. (016); Andrews et al. 2 Cao et al. (2016); Chen et al. (2017) follow a hybrid approach where used learn features, which then used within shallow AD method. Specifically for image domains, Golan & Yaniy (018)) learn features using a self- supervised paradigm - by applying geometric transformations to the image and learning classify which transformation was applied. Lee et al. (2018) learn a distance metric in feature space, for networks pre-trained on image classification. Both of these methods display outstanding AD per- formance, but are limited to image domains, while our approach does not require particular data or pre-trained features. Several studies explored using deep generative models such as GANs and VAEs for AD (e.g., An & Cho. 2015; Suh et al. 22166 /Schlegl et al.) 2017 Wang et al.) 2017), and more recently, deep energy- ameegy-based models (Zhai et al.) 2016) Song & Ou 2018) have reported outstanding results. Our work extends the VAE approach to the SSAD setting. Our results show that even a 1% fraction labelled anomalies can improve over the state-of-the-an AD scores of deep energy based models, demonstrating the importance of the SSAD setting.
deep unsupervised learning
2

---

Under review as a conference paper at ICLR 2020
3 BACKGROUND
We consider deep unsupervised learning under the standard variational inference setting (Kingmal & Welling 2.13). Given some data 2, one aims fit the parameters 0 of latent variable model (o( ppt) Pe [z|=2)], where the prior p(z) is known. For general models, the typical maximum- likelihood objective max log (x) is intractable due to the marginalization over 2, and can be approximated using the following variational inference methods.
3.1 EVIDENCE LOWER BOUND (ELBO)
The evidence lower bound (ELBO) states that for some approximate posterior distribution g(=[>r):
log Pe (x) 2 Ex((ll)) [logre DKL (<<xx)))))
where the Kullback-Leible divergence is DKL (g((g()))) (p(=)) Bx(=)) [log a(>l>)
p(z) In the varia- tional autoencoder (VAE, Kingma & Welling 2013). the approximate posterior is represented as go(=|²) ( (x) Eg
for some neural network with parameters @, the prior is p(z) (Ho, Eo), and the ELBO can be maximized using the reparameterization trick. Since the resulting model resembles an autoencoder, the approximate posterior (s(2)z) )) is also known as the encoder, while po(x is termed the decoder.
3.2 X UPPER BOUND (CUBO)
Recently, Dieng et al. (2017) derived variational upper bounds to the data log-likelihood. The x p(slX)
divergence given by Dx- (pllg) Ex(=:0)
- 1 For n > 1, Dieng et al. (2017)
((2;0)
propose the following x upper bound (CUBO):
1 log pa (x) S log Ex(=(()
po(zl=)p(z)) (((2x)
)"] = CU
n
For a family of approximate posteriors (x(2)z), one can minimize the CUBO using Monte Carlo (MC) estimation. However, MC gives a lower bound to CUBO and its gradients are biased. As an alternative, Dieng et al. (207)) proposed the following optimization objective: By monotonicity of exp, this objective reaches the same optima
== exp(n CU as CU BO, (z)(x)))) Moreover, this produces an unbiased estimate, and the number of samples only affects the variance of the gradients.
4 DEEP VARIATIONAL
ANOMALY DETECTION
In the anomaly detection problem, the goal is to detect whether sample was generated from some normal distribution (2)) or not making it an anomaly. In semi-supervised anomaly detection (SSAD), addition, we are given Nouttier examples of anomalous data, denoted by Xouttiers and we assume that N Nouttier < Nnormal- In particular, Xoutlier does not cover the range of possible anomalies, and thus classification methods (neither supervised nor semi-supervised) are not applicable.
given Normall samples from Pnormai which denote X Nooraal- In
Our approach for SSAD is to approximate Pnormat(z) (z) using deep latent variable model po(x;, and to decide whether sample is anomalous or not based on thresholding its predicted likelihood. In the following, we propose we term max-min likelihood VAE (MML-VAE), maximizes the likelihood of normal samples while minimizing the likelihood of outliers. The second method, which we term dual-prior VAE (DP- VAE), assumes different priors for normal and outlier samples.
variational methods for learning (x)). The first method, which
4.1 MAX-MIN LIKELJHOOD VAE
In this approach, we seck to find model parameters based on the following objective:
max 8
log Po (X y log po ( Noutlser):
(1)
3

---

Under review as a conference paper at ICLR 2020
where y 2 0 is a weighting term. Note that, in the absence of outlier data when y 0, Equation is just maximum likelihood estimation. For y > O, however, we take into account the knowledge that outliers should not be assigned high probability.
We model the data distribution using a latent variable model po (z) = ppp)) [Po(z|))], where the prior p(z) is known. To optimize the objective in Equation effeetivelly, we propose the following variational lower bound:
log pe nXoomm))))) y log po(Xoutterr 2 ELBOQ, (Xnormat) -CU (Xouttier) (2)
where Qı (z|rr), Q2(=|x) are the variational auxiliary distributions. In principle, the objective in Equation 2an be optimized using the methods of Kinggaa &
Dieng et al. 2011
which would effectively require training two encoders: separately on the two datasets. However, it is well-known that training deep generative models requires abundant data. Thus, there is little hope for learning an informative Q2 using the small dataset X Noutierr- To account for this, our main idea is to use the same variational distribution Q(=[z) for both loss terms, which effectively relaxes the lower bound as follows:
Q1 and Q2, and one decoder pe (a
(3) In other words, we use the same encoder for both normal and anomalous data. Finally, the loss function of the MML-VAE is: L ?CU (2
log Po (Xnormat) y log pe (Xoullier) 2 ELBO (0000((nnnii) normai) CCU B
- ELBOQ (x000m())m)a))
To gain intuition about the effect of the CUBO term in the loss function, it is instructive to assume that pe (z(2) is fixed. In this case, maximizing the lower bound only affects the variational distri- bution (z(xx). Note that the ELBO term seeks to minimize the KL distance between g( ((=]Xnormat) and ((=[Xnormat) which 'pushes' q(2 (xnoraa) toward high- likelihood regions of p(2), while the
CUBO term is proportional to a log Ex(((()) [
(plelXovttser) ([(x|Xottter))
and thus seeks to maximize the X"
distance between q(2 Xouttier) and p(= [x|X oouttier). Now, assume that for some outlier, falls within high-likelihood region of p(z). In this case, the CUBO term will 'push' q22|X (outtier)) away from p(z). Thus, intuitively, the CUBO term seeks to separate the latent distributions for normal samples, which will concentrate on high-likelihood regions of p(z), and outlier samples.
For the common choice of Gaussian prior and approximate posterior distributions KKigmma & Welling 2013), the CUBO component of the loss function is given by:
Lou exp -2 LR + log Egg + 1lg Eg 1y+
22 log F.g (exp(=2²= - x
Where Ln log po (xrl2) is the reconstruction error. The expectation is approximated with MC. In our experiments, we found that computing exp(log E [exp(-)) is numerically more stable, as we can employ the log-sum-exp trick.
In practice, we found that updating only the encoder according to the CUBO loss (i.e., ignoring the CR term) results in better performance and stabler training. The reason for this is that CUBO seeks to maximize the reconstruction error, which does not produce informative updates for the decoder. Since this decoder is shared with the ELBO term, the CUBO update can decrease performance for normal samples as well. Complete algorithm details and derivation of the CUBO objective are provided Appendix A..4
4.2 DUAL PRIOR VAE
The second method propose is a simple yet effective modification of the VAE for SSAD, which we term dual-prior VAE (AP-VAE). Here, we assume that both normal and outlier data are generated from single latent variable model po (a) Ep(x..) [po v)], where y is additional Bernoulli variable that specifies whether the sample is normal or not. Additionally, we assume the following conditional prior for
([Pnormal: Enormat), = y y=0
p(=ly)
M(poutier: Eouttier),
BBased on the ELBO and CUBO definitions, the decoder (x) is the same for hoth terms. "This method can casily be extended to non-Gaussian priors, so long as the two priors different.
4

---

Under review as a conference paper at ICLR 2020
Since data is labeled, y is known for every x and does not require inference. Deriving the ELBO for normal and outlier samples in this case gives:
ELBO (x) CR(x) DKL D 1)],
LB outlier (x) LR(2)) DRL. (( 1p((/y 0)]-
Here, similarly to the MML-VAE above, we assume the same encoder Q=2|x) for normal and outlier samples. The loss function we minimize
-= [ELBO (Xnormat) + ELBO outtier (Xoutlie)]]
In our implementation, we set Enorma! Coutlier = I, Pnormal = O, and Houttier = ad where * O. In this case, this loss function has the intuitive interpretation of training encoder that separates between latent variables for normal and outlier data.
As C simply a combination of VAE loss functions, optimization using the reparametrization trick straightforward? Similarly to the MML-VAE method, we freeze the decoder when minimizing ELLOO outlier. do not expect for informative reconstruction of the outlier distribution from the limited data X outlier-
4.3 ANOMALY DETECTION WITH MML-VAE AND DP-VAE
We next deseribe an SSAD algorithm based on MML VAE and DA-, After training an MML-- VAE, the ELBO provides an approximation of the normal data log-likelihood. Thus, given novel sample x, we can score its (log) likelihood of belonging to normal data as:
score(x) EL)O
After training a DP-VAE, ELBO normal provides a similar approximation, and in this case we set score(x) ELBO F.LBOnormal (z). Finally, for classifying the sample, we threshold its score.
At this point, the reader may wonder why we pick the ELBO and not the CUBO for scoring sample. The reason is that most of our data is used for training the ELBO, while the CUBO (and ELBOmutiin in DP-VAE) is trained with much smaller outlier data, which is not informative enough to accurate approximate the normal data likelihood.
4.4 VAE ARCHITECTURES AND ENSEMBLES
Both MML-VAE and DP-VAE simply add loss terms to the objective function of the conventional VAE. Thus, they can be applied to any type of VAE without any restriction on the architecture. In addition, they be used with ensembles of VAEs, popular technique robustify VAE training (Chen et al. 2017: Mirsky et al.) 2018). We train K different VAEs (either MML-VAE or DP-VAE) with random initial weights, and the score be the average over the ELBOs in the ensemble. We found that ensembles significantly improve the SSAD performance of our methods, as demonstrated in our experiments.
EXPERIMENTS
In our experiments, we follow the evaluation methods and datasets proposed in the extensive work of Ruff et al.) (2019). These evaluations include strong baselines of state-of-the of-the-art shallow, deep, and hybrid algorithms for AD and SSAD. Hybrid algorithms are defined as shallow SSAD methods that are trained on features extracted from deep autoencoder trained on the raw data. A brief overview of the methods provided in Appendix A.2, and we refer the reader to Ruff et al. (011) for more detailed description. Performance is evaluated by the under the curve of the receiver operating characteristic curve (AUROC), a commonly used criterion for AD. There are two types of datasets: (1) nigh-dimensional datasets which were modified be semi-supervised and (2) classic anomaly detection benchmark datasets. The first type includes MNIST, Fashion-MNIST and CIFAR- 10, and
We also experimented with a hybrid model, trained by combining the loss functions for MML-VA MML-VAE and DP-VAF, and can be seen as DP-VAR with the additional loss of minimizing the likelihood of outlier samples in F.T. B0 (z). This model obtained similar performance, as report in the appendix.
5

---

Under review as a conference paper at ICLR 2020
the second includes datasets from various fields such as astronomy and medicine. For strengthen- ing the baselines, Ruff et al. (2019) grant the shallow and hybrid methods an unfair advantage of selecting their syper-parametery also follow this approach. When comparing with the state-of-the-art Deep SAD method, we used the same network architecture but include bias terms in all layers. For the MNIST dataset, we found that this architecture did not work well, and instead used a standard convolutional neural network (CNN) architecture (see appendix for full details). Our implementation is done in PyTorch (Paszke et al. 2017) and we run our experiments on machine with an Nvidia RTX 2080 GPU. For all of the experiments, our methods use an ensemble of size K 5.
maximize AUROC on subset (10%) of the test set. Here
5.1 IMAGE DATASETS
MNIST, FaShion-MNIST and CIFAR-10 datasets all have ten classes. Similarly to Ruff et al. 2019), we derive ten AD setups on each dataset. In each setup, one of the ten classes is set to be the normal class, and the remaining nine classes represent anomalies. During training, we set X nnrraa as the normal class samples, and Xoutlier as a small fraction of the data from only one of the anomaly classes. At test time, we evaluate on anomalies from all anomaly classes. For pre- processing, pixels are scaled to [o, 1]. Unlike Ruff et al. (2019) which did not report on using validation set tune the hyper-parameters, we take a validation set (20%) out of the training data to tune hyper-parameters.
The experiment we perform, similarly to Ruff al. (2019), is evaluating the model's detection ability as a function of the ratio of anomalies presented to it during training. We set the ratio of labeled training data to be ?/- N Nouttser/ ((N (NNpormal + N o, and we evaluate different values of nI in each scenario. In total, there are 90 experiments per each value of y. Note that for " 0, no labeled anomalies are presented, and we revert to standard unsupervised AD, which our approach amounts to training a standard VAE ensemble.
The complete table with all of the competing methods can be found in the appendix A.I.I Note that even small fraction of labeled outliers (?- 0.01) leads to significant improvement compared to the standard unsupervised VAE ('1 = 0) and compared to the best- pest-performing unsupervised methods MNIST and CIFAR-10. Also, our methods outperform other SSAD baselines in most domains.
Our results are presented in Table [
OC-SVM Raw
OC-SVM Hlybrdd
Inclusive NRF
SSAD Raw
SSAD Hybrid
Deep SAD
Supervised Clasiffer
MML VAE
DP VAE
Data
MNIST
.00
96.0229
96.-t2.5
95.2
960
96.32255
92.8449
942430
942430
.01
966:22
9682223
964227
928855
97.3:21
97.0::333
.05
913333
97.42:0
967724
94.51446
97
97.51:20
-10
9077444
91.6177
969-23
95.0::47
.20
87 872156
9.
96.9224
95.6144
97.91116
97.9-1
F-MNIST
.00
92.81:477
91,241477
918:14.77
912214.77
89:11622
90.8144.6
90.8114.6
.01
92.11:5.0
89-41.0
900164
14411366
91.216666
909116
.05
883---622
905-1-59
905--66.
763+-132
91.646633
92.21-466
85.51:7.11
91.0::566
9133660
7901122
91.7
91.7160
.20
82.01:8.0
89.71:66
91005:5
B1A1120
91.91600
92.1::577
CIFAR-10
.00
62.01106
63.8290
70.01:499
62.04:066
638890
609494
10.74:107
$2.74
.01
7300:80
7266774
58.650
73.74:3
74.512444
.05
715+-:81
73.3+884
779912
635580
79341722
79.1
.10
70 10:11:
7401::1
79.17:1
67.7196
$1.11811
.20
674488
745580
$19970
805159
$2.62722
$28:7.33
Table 1: Results for image datasets. We report the average and standard deviation of AUROC over 90 experiments, for various ratios of labeled anomalies in the data %/
CatsSsDogs Dataset In addition to the test domains of Ruff al. (2019). we also evaluate on the
Vstsss dataset, which is notoriously difficult for anomaly detection (Golan & El- Yaniv (018)) This dataset contains 25,000 images of cats and dogs in various positions, 12,500 in each class. Following Golan & EI- Yaniv (0018), We split this dataset into training set containing 10,000 images, and a test set of 2,500 images in each class. We also rescale each image to size 64x64. We follow similar experimental procedure described above, and average results over the two classes. We chose a VAE architecture similar to the autoencoder architecture in Golan & El- Yaniv 2018). and for the Deep SAD baseline, we modified the architecture not bias terms and

---

Under review as a conference paper at ICLR 2020
bounded activations. We report results in Table 2 along with the numerical scores for baselines taken from Golan & Yaniy (2018). Note that without labeled anomalies, our method is not informative, predicting roughly at chance level, and this aligns with the baseline results reported by Golan & EI- Yaniv 2). However, even just 1% labeled outliers is enough to significantly improve predictions and produce informative results. This demonstrates the potential of the SSAD approach. However, in this domain, using the geometric transformations of Golan & El- -aniv (2018) allow for significantly better performance even without labeled outliers. While this approach domain specific, incorporating similar self-supervised learning ideas into probabilistic AD methods is an interesting direction for future research.
OC&VV
OC-SVM
Deep
MML
72
Hybrid
Vasgs
51.7
52.5
49
01
54.1
59.1
64.0
05
60.1
68.4
70.3
10
64.4
71.5
75.3
73.3
Table 2: Results of the experiment where we increase the ratio of labeled anomalies ") in the training set, on the Cats VsDogs dataset.
5.2 CLASSIC ANOMALY DETECTION DATASETS
AD is well-studied field of research, and as a result, there are many publicly available AD bench- marks (Shebuti, (016) with well-established baselines. These datasets are lower-dimensional than the image datasets above, and by evaluating our method on them, we aim to demonstrate the flexibil- ity of our method for general types of data. We follow Ruff et al. (2019), and consider random train- test split of 60:40 with stratified sampling to ensure correct representation classes. We evaluate over ten random seeds with 1% of anomalies, i.e., 7 = 0.01. There are no specific anomaly classes, thus we treat all anomalies as one class. For pre- pre-processing we preform standardization of the features to have zero mean and unit variance. As most of these datasets have very small amount of labeled anomalous data, is inefficient to take a validation set from the train- ing set. To tune the we also follow Ruff et al. (2019) and set a 150 training epochs limit. The complete hyper-parameters table can be found in Appendix A.3.2 A.3.2
normal and anomaly
we measure AUROC performance the training data, and
In Table 3 we present the results of the best-performing algorithms along with our methods. The complete table with all the competing methods can be found in Appendix A.I.2 Our methods outperform all of the other methods on most datasets, demonstrating the flexibility of our approach for general types of data.
OC-SVM
Supervised Classifeer
Deep SAD
MML VAE
DP VAE
Dataset
Raw
Raw
VAE
arthythmia
8..5:1:399
86.7:14.00
39.2:19.5
7..9:88.7
85.6
2.4
85.7
1 2.3
86.7
1.7
cardio
98.5
98.8.10033
83.2.19.6
95.0:1
95.5
0.8
99.2
1 0.4
99.1
0.4
satellite
9..1
9..2-1..3
87.2422.1
91.5
77.7
92.0
1.5
89.2
1.6
satimage-2
99.4-20.8
99.9:10.1
99.9:10.1
99.9--0.1
99.6
10.6::00 0.6
99.7
0.3
99.9
+ 0.1
shuttle 99.4::099
99.6:10.5
95.1-18.0
98.4::099
98.1
thyroid
98.3=009
91.9:1199
97.8+22.6
98 98.6
86.9
+ 3.7
99.9
- 0.04
99.9 + 0.04
Table 3: Results on classic AD benchmark datasets in the setting with a ratio of labeled anomalies
of 7 = 0.01 in the training set. We report the avg. AUROC with st.dev. computed over 10 seeds.
5.3 ABLATIVE ANALYSIS
We perform an ablative analysis of DP-VAE method the cardio, satellite and CIFAR-10 datasets (with 1% outliers). We evaluate our method with the following properties: (1) Frozen and unfrozen decoder. (2) separate and same encoder for normal data and outliers and (3) the effect of using ensembles of VAEs instead of one. Tummarisess the analysis; the complete table can be found in Appendix A.2.I It can be seen that using the same encoder is necessary we expected, since
7

---

Under review as a conference paper at ICLR 2020
Env 1
Env 2
Env 3
VAE
DP-VAE
VAE
DP-VAE
VAE
DP-VAE
Figure 2: Motion planning with VAF. Following the work of lchter et al. (2018), a conditional VAE is trained to generate robot configurations to used within a sampling based motion planning algorithm. The VAE generates 6D robot configurations (robot position and velocity; here projected onto the 2D plane) given an image of obstacles, an initial state (blue) and goal state (red). Using VAE method, added to the VAE training negative samples obstacle boundaries. We compare samples from the standard VAE (left) and DP- VAE (right) on three unseen obstacle maps. Note that our method results in much more informative samples for planning obstacle avoidance.
training a neural network requires sufficient amount of data. Moreover, when dealing with a small pool of outliers, the effect on the decoder is minimal. Hence, freezing the decoder contributes little to the improvement. Ensembles improve our methods result by approximately 2-4% on average, which makes sense VAE training is stochastic process. For fair comparison with the state-of- the-art, we have also evaluated an ensemble of Deep SAD models. These results, which are detailed in Appendix I.I.I show that ensembles have little effect Deep SAD. This can be explained as follows. In Deep-SAD, confidence is measured according to distance to an arbitrary point c in the vector space. Thus, the scores from different networks in the ensemble are not necessarily calibrated (they have different e points). In our VAE approach, on the other hand, the score of each network is derived from the same principled variational bound, and therefore the scores are calibrated, giving rise to the benefit of the ensemble.
Cardio AUROC
Satellite AUROC
CIFAR-10 AUROC
Encoder
Ensemble
Separntic
Unfreese
96.6 36.62:0.6
65.31:1.1
19,8:110,7
Separne
Freere
96.6220.
66.92:12
50.02109
Same
Unfreesse
S
98. 98.8-0.06
8..91:17
73.918899
Same
Freeze
19.1
1.
74.52%844
97.8
87.62117
72.5299.0
Table 4: Ablative analysis of the Dual Prior method. AUROC is reported over average of 10 seeds for the satellite and cardio. For CIFAR-1O, results are reported for the experiment with I 1% of anomalies, averaged over 90 experiments.
5.4 SAMPLE-BASEE MOTION PLANNING APPLICATION
While our focus is on SSAD, our methods can be used to enhance any VAE generative model when negative samples available. We demonstrate this idea in motion planning domain (Latombe) 2012]. Sampling-based planning algorithms search for path for a robot between obstacles on a graph of points sampled from the feasible configurations of the system. In order accelerate the planning process, lchter et al. (2018) suggest to learn non-uniform sampling strategies that concen- trate in regions where an optimal solution might lie, given training set of planning problems to learn from. Ichter et al. (2018) proposed conditional VAE that samples configurations conditioned on an image of the obstacles the domain, and is trained using samples of feasible paths on a set of training domains. Here, we propose to enhance the quality of the samples by including a few outliers (5% of the normal data) during training, which we choose to be points on the obstacle boundaries as such points clearly do not belong to the motion plan. We used the publicly available code base of Ichter et al. (0188), and modified only the CVAE training using our DP- VAE method. Exemplary generated samples for different obstacle configurations (unseen during training) are shown in Figure 2] It can be seen that adding the outliers led to a VAE that is much more focused on the feasible parts of the space, and thus generates significantly less useless points in collision with obstacles.
8

---

Under review as a conference paper at ICLR 2020
6 CONCLUSION
We proposed two VAE modifications that account for negative data examples, and used them for semi- cemi-supervised anomaly detection. We showed that these methods can be derived from natural probabilistic formulations of the problem, and that the resulting algorithms are general and effective they outperform the state-of-the-art on diverse datasets. We further demonstrated that even a small fraction of outlier data can significantly improve anomaly detection on various datasets, and that our methods can be combined with VAE applications such as in motion planning.
We see great potential in the probabilisticc approach to AD using deep generative models: it has a principled probabilistic interpretation, is agnostic to the particular type of data, and it can be implemented using expressive generative models. For specific data such images, however, dis- criminative approaches that exploit domain specific methods such as geometric transformations are currently the best performers. Developing similar self-supervised methods for generative approaches an exciting direction for future research. Another promising direction would be incorporating SSAD within energy-based models.
REFERENCES
Jinwon An and Sungzoon Cho. Variational autoencoder hased anomaly detection using reconstruc- tion probability. Special Lecture IE, 2(1), 2015.
Jerone Andrews, Edward Morton, and Lewis Griffin. Detecting anomalous data using auto-encoders. International Journal of Machine Learning and Computing, 6:21, 01 2016.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Semi-supervised novelty detection. J. Mach. Learn. Res., 11:2973 3009, December 2010.
Van Loi Cao, Miguel Nicolau, and James Medermott. A hybrid autoencoder and density estimation model for anomaly detection. In PPSN, volume 9921, pp. 717-726, 09 2016.
Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: A survey. CoRR, abs/1901.03407, 2019.
Jinghui Chen, Saket Sathe, Charu Aggarwal, and Deepak Turaga. Outlier detection with autoencoder ensembles. Proceedings of the 2017 SIAM International Conference on Data Mining, pp. 90- 98. SIAM, 2017.
Lucas Deecke, Robert Vandermeulen, Lukas Ruff, Stephan Mandt, and Marius Kloft. Anomaly detection with generative adversarial networks, 2018.
Adji Bousso Dieng, Dustin Tran, Rajesh Ranganath, John Paisley, and David Blei. Variational in- ference via x upper bound minimization. In Advances in Neural Information Processing Systems, pp. 2732-2741, 2017.
Sarah M. Erfani, Sutharshan Rajasegarar, Shanika Karunasekera, and Christopher Leckie. High- dimensional and large-scale anomaly detection using a linear one-class svm with deep learning. Pattern Recogn., 121C)11 October 2016.
Tolga Ergen, Ali Hassan Mirza, and Suleyman Serdar Kozat. Unsupervised and semi-supervised anomaly detection with 1stm neural networks. arXiv preprint urXiv: 2017.
Izhak Golan and Ran Yaniv.. Deep anomaly detection using geometric transformations. In Ad- vances Neural Information Processing Systems, pp. 9758-9769, 2018
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672 2680, 2014.
Nico Gömitz, Marius Kloft, Konrad Rieck, and Ulf Brefeld. Toward supervised anomaly detection. Journal of Artificial Intelligence Research, 46:235-263 2013.

---

Under review as a conference paper at ICLR 2020
B. Ichter, J. Harrison, and M. Pavone. Learning sampling distributions for robot motion planning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 7087-7094, May 2018.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Soren Kaae Sønderby, and Ole Winther. Ladder Variational Autoencoders. arXiv e-prints, art. 1xXi:1602.02282, Feb 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes.
arXiv preprint
arXiv:1312.6114,200 2013.
Diederik P. Kingma, Danilo Jimenez Rezende, Shakir Mohamed, and Max Welling. Semi- supervised learning with deep generative models. CoRR, abs// 406.5298, 2014.
Kiran, Dilip Thomas, and Ranjith Parakkal. An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos. Journal of Imaging, 4(2):36, 2018.
Latombe. Robot motion plamning, volume 124. Springer Science & Business Media,
2012.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Process- ing Systems, pp. 7167-7177, 2018.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 Eighth IEEE Interna- fional Conference on Data Mining, pp. 413-422. IEEE, 2008
Erxue Min, Jun Long, Qiang Liu, Jianjing Cui, Zhiping Cai, and Junbo Ma. Su-ids: A semi- supervised and unsupervised framework for network intrusion detection. In International Con- ference on Cloud Computing and Security, pp. 322-334. Springer, 2018.
Yisroel Mirsky, Tomer Doitshman, Yuval Elovici, and Asaf Shabtai. Kitsune: An ensemble of autoencoders for online network intrusion detection. CoRR, abs/ (bs/1882.09089, 2018.
Jordi Muñoz-Mari, Francesca Bovolo, Luis Gómez-Chova, Lorenzo Bruzzone, and Gustavo Camp-
Valls. Semisupervised one-class support vector machines for classification of remote sensing data. IEEE transactions on geoscience and remote sensing, 48(8):3188-31) 2010.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Marco AF Pimentel, David A Clifton, Lei Clifton, and Lionel Tarassenko. A review of novelty detection. Signal Processing, 99:215-249, 2014.
Lukas Ruff, Robert A. Vandermeulen, Nico Görmitz, Lucas Deecke, Shoaib A. Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. Proceedings of the 35th International Conference Machine Learning, volume 80, pp. 4393 4402, 2018.
Lukas Ruff, Robert A. Vandermeulen, Nico Gömitz, Alexander Binder, Emmanuel Müller, Klaus- Robert Müller, and Marius Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.
0269999960064,,. 02694, 2019.
Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International Conference on Information Processing in Medical Imaging, pp. 146 157. Springer, 2017.
Bernhard Schölkopf, John C. Platt, John C. Shawe-Taylaee Alex J. Smola, and Robert C. Williamson. Estimating the support of a high-dimensional distribution. Neural Comput., 13(7):1443-1477 July 2001.
10

---

Under review as a conference paper at ICLR 2020
Rayana Shebuti. Odds library, 2016. [tttt::://oddss http://dd...c..tttnooooooo....
Yunfu Song and Zhijian Ou. Learning neural random fields with inclusive auxiliary generators. ArXiv, abss/180.00027,, 2018.
Suwon Suh, Daniel H Chae, Hyon-Goo Kang, and Seungjin Choi. Echo-state conditional variational autoencoder for anomaly detection. In 2016 International Joint Conference on Neural Networks (IJCN), pp. 1015-1022. IEEE, 2016.
David M. J. Tax and Robert P. W. Duin. Support vector data description. Mach. Learn., 54(1): 45-66, January 2004. ISSN 0885-6125.
William Wang, Angelina Wang. Aviv Tamar, Xi Chen, and Picter Abbeel. Safer classification by synthesis. CoRR, abs/1711.08534, 2017.
Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang, Deep structured energy based models for anomaly detection. CoRR, http//005007717 2016.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Dacki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection, In Inter- national Conference on Learning Representations, 2018.
A APPENDIX
A.1 COMPLETE RESULTS
A.1.1 MNIST, FASHION-MNIST. CIFAR-10
Table includes the complete results. The results for an ensemble of Deep SAD models are in parenthesis (CIFAR-10). The ensemble method is implemented the same way K = separate models (i.e., each model has its own c), and the score (which is the distance from c) the average of scores from all models in the ensemble.
train
A.1.2 CLASSIC ANOMALY DETECTION
Table 6 includes the complete results. MML-DP VAE combines both suggested objectives when training the VAE, i.e. MML and DP, and as reflected in the results, its performance is on par with them.
A.2 COMPETING METHODS
We compare methods to reported performance of both deep and shallow learning approaches, as detailed by Ruff et al. (2019) and Golan & El- Yaniv (2018). For completeness, we give brief overview of the methods.
The one-class support vector machine (OC-SVM) is kernel based method for novelty detection (Schölkopf al., 2001). is typically employed with an RBF kernel, and learns a collection of closed sets in the input space, containing most of the training samples. SVDD (Tax & Duin 2004) is equivalent to OC-SVM the RBF kernel. OC-SVM are both granted an unfair advantage by selecting its hyper-parametert to maximize the AUROC on subset (10%) of the test set to establish a strong baseline.
OC-SVM/SVDD
Isolation Forest (IF) Proposed by Liu
2008). IF tree-based method that explicitly isolates anomalies instead of constructing a profile of normal instances and then identify instances that do not conform the normal profile as anomalies. As recommended in the original paper, the number of trees is set to f 100, and the sub-sampling size to e = 256.
Kernel Density Estimator (KDE) The bandwidth of the Gaussian Kernel is selected via 5-fold voss-validation using the log-likelihood following Ruff al. (2018).

---

Under review as conference paper at [CLR 2020
OC-SVM
OC-SVM
IF
KDE
KDE
Deep
Inclasive
SSAD
SSAD
Supervised
MML
DP
72
Hybrid
Raw
Hybrid
Hybrid
SVDD
Hyhrid
SS-DGM
Classieer
VAR
.00
96.0229
96.322.5
85 8574
90.525.3
95.0133
878456
92.95577
92.81:49
96.022.
96.12:
92.822499
942230
943130
.01
96.6-124
96.8 96.8+-23
89.91-922
96.4--2.77
92.8 22.8555
97.3+-11
97.0
13
97.4220
92:01:66
96.72224
94.5=4.6
97-5220
.10
90.7444.4
9.64+117
91.6-:555
96.9
95.0=4.7
97.8+-1.6
97.6--2.1
20
87.2
97.811
91:2556
969224
95.624.4
97.92:
97.21A
F-MNIST
.00
92.82:7
91.22:77
916655
82.528.1
92.0249
69.72144
90.2558
89.2262
92.824
91.24447
89.266
908244
908246
909466.7
.05
88.126
714
90.526.5
7682112
91.62263
92:11:46
.10
85.5
91.0:45666
7299:122
91.3:66.0
79.0:1-2.3
91.74644
91.74600
82.028.0
80.7166
74.7
91.02:555
81.4212.0
91.9460
92.1::77
CIFAR-IO
10.6
63.82990
6000:000
59.92667
10.911177
56.12002
56.2213.
609494
70.049
62.0=10.6
638899
60.9294 (60.79944
52.72
52.72:077
.01
23.0:18.0
70.5 4-.3
49.7
226677.4
55.6=5.0
73.24:7.33
74.5
71.52281
73.11:84
5082477
77.9772 (1777778)
63.5x800
79.327222
79.1
.10
70.1
74.0:88.1
$2.01:55
79.827.1 (99.6:44)))
67.7=9.6
$0.8:17.77
81.148.11
200
67.42888
74.5280
53.246.7
81.927.011.2225
80.5=1.9
82.6-27.2
828823
12
Table 5: Complete results of the experiment where increase the ratio of labeled anomalies "v in the training set. We report the avg. AUROC with st.dev. computed over 90 experiments at various ratios y. In parenthesis, the results of an ensemble of Deep SAD models (K = 5).

---

OC-SVM
Deep
Deep
Supervised Classilier
Under review as conference paper at [CLR 2020
Data
Raw
Hybrid
CAE
SVDD
Raw
Hybrid
SS-DGM
SAD
VAE
VAE
VAE
VAE
antyythmia
34.9
76.7+62
74.0+7.5
74.64-9.0
86.74400
78.3+...1
50.34:9.8
75.9+8.7
39.24:9.5
85.6 = 2.4
85.7 2.3
86.7
87.3 + 1.7
cardio
98.5:t:0.3
82.8:99.3
94.3+-2.0
84.8::3.6
98.8=0.3
86.341:3.8
66.22:14.3
95.0=116
83.2499.6
95.5 = 0.8
99.3 - 0.3
99.1
99.2 - 0.4
80.01:1.7
79.81:4.1
96.2:10.3
86.91:2.88
57.4:1.44
87.2:4:.11
1.0
satimago-2
99,4::0.88
96.7+2211
99.9=0.0
98.3+1144
99.9=0.1
96.8+2.1
99.2+t0066
99.9+0011
99.9+b0.1
99.6 = 0.6
99.7 0.3
99.9 0.1
99.8 - 0.1
shuttle
99.4:1-0.9
94.1499.5
98.2+122
86.3::7.5
99.6:00.5
97.74:1.0
98.4::099 72.7112.0
97.940.3
95.1-2..00
98.1 = 0.6
99.8 0.1
99.9 2: 0.04
99.9 0.09
thyrosd
98.3.10.9
91.2.14.0
75.2.110.2
72.0.19.7
97.9.1.9
95.3113.1
98.6.10.9
97.8.1..66
86.9 1: 3.7
99.9
0.04
99.9 11991000 0.04
99.9 0.03
Table 6: Complete results classic AD benchmark datasets in the setting with ratio of labeled anomalies of m 0.01 in the training set. AUROC with st.dev. computed over 10 seeds.
We report the avg.
13

---

Under review as a conference paper at ICLR 2020
Semi-Supervised Anomaly Detection (SSAD) A kernel method suggested by Görmitz al. (2013) which is a generalization to SVDD to both labeled and unlabled examples. Also granted the same unfair adavantage OC-SVMISVDD.
Convolutional Autoencoder (CAE) Autoencoders with convolution and deconvolution layers in the encoder and decoder, respectively. We use the same architectures described A.3.1 for our VAE.
Hybrid Methods In all of the the hybrid methods mentioned in the results, the inputs are repre- sentations from a converged autoencoder, instead of raw inputs.
Unsupervised Deep SVDD Two end-to-end variants OC-SVM methods called Soft-Boundary Deep SVDD and One-Class Deep SVDD proposed by Ruff et al. (2018). They use an objective similar to that the classic SVDD to optimize the weights deep architecture.
Deep Semi-supervised Anomaly Detection (Deep SAD) Recently proposed by Ruff et al. (011)). Deep SAD is general method based on deep SVDD, which learns a neural-network mapping of the input that minimizes the volume of data around predetermined point.
Semi-Supervised Deep Generative Models (SS-DGM) Kingma et al. (2014) proposed deep variational generative approach to semi-supervised learning. In this approach, classifier trained on the latent space embeddings of VAE, which are lower in dimension than the original input. They also propose a probabilistic model that describes the data using the available labels. These two models are fused together to stacked semi-supervised model.
Deep Structured Energy-Based Models (DSEBM) Proposed by Zhai et al. (2016), DSEBM is deep neural technique, whose output is the energy function (negative log probability) associated with an input sample. The chosen architecture, as decribed by Golan HI- Yaniv (2018) is the same that of the encoder part in the convolutional autoencoder used by OC-SVM Hybrid.
Deep Autoencoding Gaussian Mixture Model (DAGMM) Proposed by Zong et al. 2018 DAGMM is end-to-end deep neural network that leverages Gaussian Mixture Modeling to per- form density estimation and unsupervised anomaly detection in a low-dimensional space learned by deep autoencoder. simultaneously optimizes the parameters of the autoencoder and the mixture model in end-to-end fashion, thus leveraging separate estimation network to facilitate the pa- rameter learning of the mixture model. The architecture, as decribed by Golan El- Yaniv) (2018) is the same as of the autoencoder we used is similar to that of the convolutional autoencoder used in OC-SVM Hybrid.
Anomaly Detection with Generative Adversarial Network (ADGAN) A GAN-based model, proposed by Deecke et al. (2018). Anomaly detection is done with GANs by searching the gen- erator's ' latent space for good sample representations. In the experiments performed by Golan & El-Yaniy (2018), the generative model the ADGAN had the same architecture used by the authors of the original paper.
Deep Anomaly Detection using Geometric Transformations (DADGT) A stac-of-the-art deep anomaly detection in images method proposed by Golan & EI- El- 2018). In this method, features are learned using a self-supervised paradigm by applying geometric transformations to the image and learning classify which transformation was applied.
Inclusive Neural Random Fields (Inclusive-RRF) A state-of-the-art energy-based model pro- posed by Song & Ou (018). The inclusive-NRF learns neural random fields for continuous data by minimized auxiliary generators and stochastic gradient sampling. As this model directly provides density estimate, it is an efficient tool for AD, as the estimated density be used the decision criterion.
developing mnclusive-divergee
14

---

Under review as a conference paper at ICLR 2020
A.2.1
COMPLETE ABLATIVE ANALYSIS
The complete ablative analysis is reported in table
Cardio AUROC
Satellite AUROC
CIFAR-10
Encader
Decoder
Ensemble
AUROC
Separnic Sopurato
Unfreeze
5
96.6:20.6
65.3
19,8211077
Same
Unfreeze
0.08
8.
73.91:8.9
Same
Freeze
S
99.1::0.44
91.71:1.5
74.5:28.4
Soparate
Unfreese
32.5
65.14:5.3
50.4 10.2
Soparate
49.2::10.4
Same
Unfreese
96.8:21.5
90.01:2.99
72.8128.6
Same
Froeze
97,8:
87.6-1:.7
72.3+4.9.0
Table Complete ablative analysis of the Dual Prior method. AUROC is reported over average of 10 seeds for the satellite and cardio. For CIFAR-1O 10, results are reported for the experiment with I 1% of anomalies, averaged over 90 experiments.
A.3 IMPLEMENTATION DETAILS
We provide essential implementation information and describe how we tuned our model
A.3.1 NETWORK ARCHITECTURES
Our work is based on deep variational autoencoders which require encoder network and a decoder network. For the architectures, we follow Ruff et al. (2019) architectures for the autoencoders, with the only differences being the use of bias weights in our architectures and instead of autoencoders we use variational autoencoders (i.e. the encoder outputs mean and standard deviation of the latent variables).
For the image datasets, LeNet-type convolutional neural networks (CNNs) are used. Each convolu- tional layer is followed by batch normalization and leaky ReLU activations (a = 0.1) and 2 2-max pooling. On Fashion-MNIs" we use two convolutional layers, one the first with 16 x (5 x 5) filters and second with 32 x (5 5) filters. Following are two dense layers of 64 and 32 units respectively. On 10, three convolutional layers of sizes 32 x (5 x 5) ,64 x (5 x 5) and 128 x (5 x 5) filters are followed by a final dense layer of 128 units (i.e. the latent space of dimension 128). For MNIST, we use different architecture where the first layer is comprised of 64 x (4 x 4)-filters and the second 128 (4 x 4) -fitte with ReLU activations and no batch normalization. We use two dense layers, the first with 1024 units and the final has 32 units. For Cats we follow Golan El- Yaniv (2018) architecture for the autoencoder. We employ three convolutional layers, each followed by batch normalization and ReLU activation. The number of filters in each layer 128 x (3 x 3). 256 x (3 x 3) and 512 x (3 x 3) respectively. The final dense layer has 256 units, which is the latent space dimension.
For the classic AD benchmark datasets, we use regular multi-layer perceptrons (MLPs). On arrbyth- mia, a 3-layer MLP with 128-64-32 units. On cardio, satellite, satimage-2 and shuttle, use 3-layer MLP with 32-16-8 units. On thyroid a 3-layer MLP with 32-16-4 units.
For the motion planning, follow Ichter al. (2018) and two-layer MLP with 512 hidden units and latent space dimension of 256. We use dropout (p- = 0.5) as regularization and ReLU activation for the hidden layers.
A.3.2 HYPER- PAREERPARAMETERS AND TUNING
Tuning models semi-supervised setting is not trivial task, as usually there is abundance of data from one class and a small pool of data from the other classes. Thus, it is not clear whether one should allocate validation set out of the training set (and by doing that, reducing the number of available samples for training) or just evaluate the performance on the training set and hope for the best. Ruff al. (2019) didn't 't use validation set, but predetermined the total number of epochs to run and finally evaluated the performance on the test set. We, on the other hand, decided to take
*Our code will be published upon acceptance for publication
15

---

Under review as a conference paper at ICLR 2020
validation set out of the training set for the image datasets, have enough data. The validation set is composed of unseen samples from the normal class and samples from the current outlier class unlike the test set. which is composed of samples from all ten classes (9 outlier classes). On the classic AD benchmark datasets, as there is very few outlier data, we evaluate the performance during training the training performance on the test set. For all datasets, we used a batch size of 128 and an ensemble of 5 VAEs. For the image datasets, we run 200 epochs and for the classical AD benchmarks we run 150 epochs. For the MML method, we found = 1 to perform well on all datasets. Furthermore, similarly to the additional term as derived in Equation[4 The motivation for adding this balancing hyper-parameter is that since as explained earlier,
itself without taking validation set. Finally,
evaluate the
3KL the ELBO term, we add Scu BO to the CUBO
= ex CU (go (=[x)) reaches the same optima as CU BO.. (1 so does C = exp [3... n. CU (=)())())r))) Finding good value for ÅCU BO can contribute to the performance, as reflected
the results. Moreover, for the optimization of the CUBO component, used gradient clipping and learning rate scheduling (every 50 epochs, learning rate is multiplied by 0.1). Tables [ and summarize all the papee-parameters chosen for our model per dataset.
Data
ND Update Interval
Learning Rate
Bx
Hin
MNIST
2
0.001
0.005
10
Aashion-MNIST
CIFAR-ID 10
2
0.001
0.005
10
Cats VDngs
arrhythmia
0.0005
0.5
cardio
0.05
satellite
0.001
0.05
satimage-2
shuttle
0.001
0.05
thyroid
0.0001
0.05
Table 8: Hyper-parameters for the Dual Prior VAE
Data
ND Update Interval
Learning Rate
AgL
ACUBO
MNIST
0,0005
0.005
0.005
Fashion-MNIST
2
0.001
0,005
0.005
CIFAR-I0
0.005
Dogs
0.0005
0.005
0.005
arthythmia
0.0005
0.5
0.5
cardio
0.001
0.05
0.05
satellite
0.001
0.05
0.05
satimage-2
0.001
0.05
0.05
shuttle
thyroid
0.0001
0.05
0.05
Table 9:
for the Max-Min Likelihood VAE
For training, follow Kaae Sønderby et al. (016)) recommendations for training VAEs and use 20-epoch annealing for the KL-divergence component, that is, the KL-divergence coefficient, 3KL. is linearly increased from zero to its final value. Moreover, representation of the normal data in a warm-up period of 50 epochs, and then we begin applying the novelty detection updates. For optimization, rate schedule to stabilize training when outlier samples are fed after the warm-up period.
allow the VAE to first learn good
use Adam (Kingma & Bal (014) with a learning
A.4 CUBO LOSs DERIVATION
The x-divergence is defined as follows:
p((2(z) 2
(xl()) = Eg
1|
((2; 0)
The general upper bound as derived in Dieng et al. 20177:
p(zz22 2)) )²
(x² CUBO,
= 2
log IE 1y(=
[(
a(2;; 0)
16

---

Under review as a conference paper at ICLR 2020
The optimized CUBO:
L =
= E 1x()))) [(
p(x, 2)
)
((2; 0)
In the our VAE framework, we denote LR = - log (x)))) the reconstruction error between the output of the decoder and the original input. We assume that [x] ~ ~X(,((X)) Ex and that in general, Pouttier (z) ~ NNo,. T), but we note that in our case, we set t4a = 0, for both CUBO and hybrid methods. As the CUBO loss is derived only for the anomalous data, we omit the labels in the following. BOUBO is a balancing typer-parameter we add, as mentioned in A...2
LCUBO =Bg(z:)) [( (((e))2) "] Egy (2g (X + 2loog ((z;
p(z)
}}} =
{x|[X)
By exp( -2 Ln + 23CUBO (logp(=) - lo (=|x)}}]|
By eepp 2- (- 2 (²)= (a-Mo) -1 log | 2 (2- Mg) (3-Ma) E (2- (z-Hy)1)BB
2
2
[ [x[exp(22 -2 LR + BounO
(- 13-22- 2=- Mn + 2 [b} + log E5 +
z² En"
22 T Eng pug 14g Sig 1n)33
Bg exppp-2 LR + BCUBO (log + Mg
T b p4 - 1o Po
T
2² 2/22 + 2: 11 + 2/ 2 12- 22-22 E [k)3
exp(-2- La + SCUBO (log Egg + 1 2/ - 1 Ho) }.
[ expe(BBUUR
(-=² 22 Mo + =
Eg =- 2z 2i mn)
expl-2- LR + BCUBO (log Ei + u expflog By expl ( (- (-2²²= z + 22 Ho + 2td Lq - 22 2
- 2gg - 16 Ho)} }.
exp(-2- Ln + BCUBOO (log Exg + T -ー q 14 - 1/f Mo) +
log Eng
( 2T 2 + 2n Mo + 2 __=''=- 2- bg)1 (4)
The expectation is estimated with Monte Carlo and for numeric stability we employ the commonly used log-sum-exp trick.
A.5 ELBO WITH GAUSSIAN PRIOR
We provide the derivation for the ELBO objective function where the prior, p(2) is a Gaussian with non-zero mean, that is, z N (Poy T). Under the i.i.d. assumption in the VAE framework, assume that z; ~ (Poz 1) and 2 x ~ (( x3) Thus, it holds that:
Eg of + Mgg
We now derive the KL-divergence component of the ELBO:
log [(2[xx] Bggoog (x)xX]]] E,logp((2]] log((2)
D Eg
H. llg (=x[x)] = log Egg
- F.l( ((2 - m4)) "' (2 1
pnyl =
2
1
log Egg
+ 2 u
14g 1
5ー1
F
1 2] =
2
2
1 n
2
112
1
2 i=1
Dlog 07
+ 1 +
0
2
=
2
log 0 + 1]
i=1
17

---

Under review as a conference paper at ICLR 2020
1
E.glog p(=)] =
2 221u1(22 po) Noll
o 1g 24q + 1
Finally:
Dkulo(s[X)]l(((]]====
+ log 03 - -o - -Ma + 24qcHo. - 1
18


Reviews:
[{"review_id": "H1x83bU7cS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes two variational methods for training VAEs for SSAD (Semi-supervised Anomaly Detection). Experiments on benchmarking datasets show improvements over state-of-the-art SSAD methods.\n\nIn generally, the paper is well written. But I have some concerns.\n\n1. Some of the results have not yet been obtained.\n\n2. Missing some relevant references.\nIn addition to VAEs, there is another class of deep generative models - random fields (a.k.a. energy-based models, EBMs), which have been applied to anomaly detection (AD) recently. Particularly, the unsupervised AD results on MNIST and CIFAR-10 from [2] are much better than the proposed methods (MML-VAE, DP-VAE).\nThough semi-supervised AD is interesting, good performances on unsupervised AD can be a baseline indicator of the effectiveness of the AD models. The authors should add comments and comparisons.\n\n[1] S. Zhai, Y. Cheng, W. Lu, and Z. Zhang, “Deep structured energy based models for anomaly detection,” ICML, 2016.\n[2] Y. Song, Z. Ou. \"Learning Neural Random Fields with Inclusive Auxiliary Generators,\" arxiv 1806.00271, 2018.\n\n3. “For all of the experiments, our methods use an ensemble of size K = 5.”\nAre other methods also tested by using an ensemble?\n\n--------update after reading the response-----------\nThe updated paper has been improved to address my concerns.\n\nI partly agree with the authors that their results demonstrate the importance of the semi-supervised AD setting (a 1% fraction of labelled anomalies can improve over the state-of-the-art AD scores of deep energy based models). However, I think, the proposed methods in this paper will not be as competitive as semi-supervised deep energy based models.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgL_iMqKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The papers proposes to use VAE-like approaches for semi-supervised novelty detection. Two methods are describes:\n(1) the MML-VAE fits a standard VAE to the normal samples and add a repulsive term for the outliers -- this term encourages the encoder to map the outliers far from the latent-space prior distribution.\n(2) the DP-VAE fits a VAE with a mixture of Gaussian prior on the latent space -- one mixture component for normal samples, and one mixture component for outlier samples.\n\nThe described methods are simple, natural, and appear to work relatively well -- for this simple reason, I think that the text could be accepted.\n\nThere are several things that are still not entirely clear.\n(1) without the reconstruction term, the methods are small variations of supervised methods. Consequently, I feel that the authors should try to explain much more carefully why the introduction of a reconstruction term (which could be thought as an auxiliary task) helps.\n(2) given point (1), one could think of many auxiliary task (eg. usual colorisation, or rotation prediction, etc..) Would it lead to worse results?\n(3) proportion > 10% of anomaly is relatively standard for supervised-methods + few standard tricks to work very well. Although I understand that only one small subset of anomalies is presented during training, I think that it would still be worth describing in more details the efforts that have been spent to try to make standard supervised methods work. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJgu6JrjtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents two novel VAE-based methods for the (more general) semi-supervised anomaly detection (SSAD) setting where one has also access to some labeled anomalous samples in addition to mostly normal data. The first method, Max-Min Likelihood VAE (MML-VAE), extends the standard VAE objective that maximizes the log-likelihood for normal data by an additional term that in contrast minimizes the log-likelihood for labeled anomalies. To optimize the MML objective, the paper proposes to minimize the sum of the standard (negative) ELBO for normal samples and the so-called CUBO, which is a variational upper bound on the data log-likelihood, for anomalous samples. The second method, Dual Prior VAE (DP-VAE), modifies the standard VAE by introducing a second separate prior for the anomalous data, which is also Gaussian but has different mean. The DP-VAE objective then is defined as the sum of the two respective ELBOs which is optimized over shared encoder and decoder networks (with the adjustment that the outlier ELBO only updates the encoder). The anomaly score for both models then is defined as the (negative) ELBO of a test sample. Finally, the paper presents quite extensive experimental results on the benchmarks from Ruff et al. [2], CatsVsDogs, and an application of robotic motion planning which indicate a slight advantage of the proposed methods.\n\nI am quite familiar with the recent Deep SAD paper [2] this work builds upon and very much agree that the (more general) SSAD setting is an important problem with high practical relevance for which there exists little prior work. Overall this paper is well structured/written and well placed in the literature, but I think it is not yet ready for acceptance due to the following key reasons: \n(i) I think DP-VAE, the currently better performing method, is ill-posed for SSAD since it makes the assumption that anomalies are generated from one common latent prior and thus must be similar; \n(ii) I think the worse performance of MML-VAE, which I find theoretically sound for SSAD, is mainly due to optimization issues that should be investigated; \n(iii) The experiments do not show for the bulk of experiments how much of the improvement is due to meta-algorithms (ensemble and hyperparameter selection on a validation set with some labels).\n\n(i) DP-VAE models anomalies to be generated from one common latent distribution (modeled as Gaussian here) which imposes the assumption that anomalies are similar, the so-called cluster assumption [2]. This assumption, however, generally does not hold for anomalies which are defined to be just different from the normal class but anomalies do not have to be similar to each other. Methodologically, DP-VAE is rather a semi-supervised classification method (essentially a VAE with Gaussian mixture prior having two components) which the paper itself points out is ill-posed for SSAD: “... the labeled information on anomalous samples is too limited to represent the variation of anomalies ... .” I suspect the slight advantage of DP-VAE might be mainly due to using meta-algorithms (ensemble, hyperparameter selection) and due to the rather structured/clustered nature of anomalies in the MNIST, F-MNIST, and CIFAR-10 benchmarks.\n\n(ii) I find MML-VAE, unfortunately the worse performing method, to be a conceptually sound approach to SSAD following the intuitive idea that normal samples should concentrate under the normal prior whereas the latent embeddings of anomalies should have low likelihood under this prior. This approach correctly does not make any assumption on the latent structure of anomalies as DP-VAE does. I believe MML-VAE in its current formulation leads to worse results mainly to optimization issues that I suspect can be resolved and should be further investigated. I guess the major issue of the MML-VAE loss is that the log-likelihood for outlier samples has steep curvature and is unbounded from below. Deep networks might easily exploit this without learning meaningful representations as the paper also hints towards. This also results in unstable optimization. I think removing the reconstruction term for outliers, as the paper suggests, also helps for this particular reason but this is rather heuristic. These optimization flaws should be investigated and the loss adjusted if needed. Maybe simple thresholding (adding an epsilon to lower bound the loss), gradient clipping, or robust reformulations of the loss could improve optimization already?\n\n(iii) To infer the statistical significance of the results and to assess the effect of meta-algorithms (ensemble, hyperparameter tuning) an ablation study as in Table 4 (at least on the effect of ensembling) should be included also for the major, more complex datasets. Which score is used for hyperparameter selection (ELBO, log-likelihood, AUC)? How would the competitors perform under similar tuning?\n\n\n####################\n*Additional Feedback*\n\n*Positive Highlights*\n1. Both proposed methods can be used with general data types and VAE network architectures (the existing Deep SAD state-of-the-art method employs restricted architectures).\n2. The paper is well placed in the literature and all major and very recent relevant work that I am aware of are included.\n3. This is an interesting use of the CUBO bound which I did not know before reading this work. This might be interesting for the general variational inference community to derive novel optimization schemes.\n4. I found the robotic motion planning application quite cool. This also suggests that negative sampling is useful beyond the AD task.\n5. I appreciate that the authors included the CatsVsDogs experiment although DADGT performs better as it demonstrates the potential of SSAD. I very much agree that employing similar self-supervised learning ideas and augmentation is a promising direction for future research.\n\n*Ideas for Improvement*\n6. Extend the semi-supervised setting to unlabeled (mostly normal), labeled normal, and labeled anomalous training data. The text currently formulates a setting with only labeled normal and labeled anomalous samples. A simple general formulation could just assign different weights to the unlabeled and labeled normal data terms.\n7. There might be an interesting connection between MML-VAE and Deep SAD in the sense that MML-VAE is a probabilistic version of the latter. The $\\chi_n$ distance of the CUBO loss has terms similar to the inverse squared norm penalty of Deep SAD.\n8. Report the range from which hyperparameters are selected.\n9. Add the recently introduced MVTec AD benchmark dataset to your experimental evaluation [1].\n10. Run experiments on the full test suite of Ruff et al. [2]. At the moment only one of three scenarios are evaluated.\n\n*Minor comments*\n11. Inconsistent notation for the expected value ($\\mathbb{E}$ vs $\\mathbf{E}$)\n12. In Section 3, the parameterization of the variational approximate $q(z | x)$ is inconsistently denoted by $\\phi$ and $\\theta$ (which beforehand parameterizes the decoder).\n13. In Section 3.2, the current formulation first says that MC produces a biased, then an unbiased estimate of the gradients.\n14. First sentence in Section 4: I would not use “classify” but rather “detect” etc. for anomaly/novelty detection since the task differs from classification.\n15. In Section 4.2, there should be a minus in front of the KL-divergence terms of the $ELBO_{normal}$ and $ELBO_{outlier}$ equations.\n16. In the fully unsupervised setting on CIFAR-10 (Table 5), why is the VAE performance essentially at random (~50) in comparison to CAE and Deep SVDD although they use the same network architecture?\n17. Is the CUBO indeed a strictly valid bound if one considers the non-normal data-generating distribution?\n18. Are there any results on the tightness of the CUBO?\n\n\n####################\n*References*\n[1] P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9592–9600, 2019.\n[2] L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694, 2019.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents two novel VAE-based methods for semi-supervised anomaly detection (SSAD) where one has also access to a small set of labeled anomalous samples. The reviewers had several concerns about the paper, in particular completely addressing reviewer #3's comments would strengthen the paper.
Decision: Reject