Novelty Detection via Robust Variational Autoencoding
Authors: Chieh-Hsin Lai
Dongmian Zou
Gilad Lerman
Venue:  (2021)
Keywords: novelty detection
variational autoencoding
robustness
Wasserstein metric
one-class classification
semi-supervised anomaly detection

Abstract: We propose a new method for novelty detection that can tolerate high corruption of the training points, whereas previous works assumed either no or very low corruption. Our method trains a robust variational autoencoder (VAE), which aims to generate a model for the uncorrupted training points. To gain robustness to high corruption, we incorporate the following four changes to the common VAE: 1. Extracting crucial features of the latent code by a carefully designed dimension reduction component for distributions; 2. Modeling the latent distribution as a mixture of Gaussian low-rank inliers and full-rank outliers, where the testing only uses the inlier model; 3. Applying the Wasserstein-1 metric for regularization, instead of the Kullback-Leibler (KL) divergence; and 4. Using a least absolute deviation error for reconstruction. We establish both robustness to outliers and suitability to low-rank modeling of the Wasserstein metric as opposed to the KL divergence. We illustrate state-of-the-art results on standard benchmarks for novelty detection.

Under review as conference paper at ICLR 2021
NOVELTY DETECTION VIA ROBUST VARIATIONAL AUTOENCODING
Anonymous authors Paper under double-blind review
ABSTRACT
We propose new method for novelty detection that can tolerate high corruption of the training points, whereas previous works assumed either no very low corruption. Our method trains a robust variational autoencoder (VAE), which aims to generate a model for the uncorrupted training points. To gain robustness to high corruption, we incorporate the following four changes to the common VAE: 1. Extracting crucial features of the latent code by a carefully designed dimension reduction component for distributions: 2. Modeling the latent distribution as mixture of Gaussian low-rank inliers and full-rank outliers, where the testing only uses the inlier model: 3. Applying the Wasserstein- I metric for regularization, instead of the Kullback-Leible (KL.) divergence; and 4. Using least absolute deviation error for reconstruction. We establish both robustness to outliers and suitability to low-rank modeling of the Wasserstein metric as opposed to the KL divergence. We illustrate ofte-offthhearr results standard benchmarks for novelty detection.
INTRODUCTION
Novelty detection refers to the task of detecting testing data points that deviate from the underlying structure of a given training dataset (Chandola et al. 2009 Pimentel et al. [0014] Chalapathy & Chawla (019)) finds crucial applications. in areas such insurance and credit fraud al. mobile robots (Neto& & Nchmzow] (2077) and medical diagnosis (Wei et al. 2018). Ideally, novelty detection requires learning leauings the underlying distribution of the training data, where sometimes it sufficient to learn a significant feature, geometric structure or another property of the training data. One can then apply the learned distribution (or property) to detect deviating points in the test data. This is different from outlier detection (Chandola et al. 2009), in which one does not have training data and has to determine the deviating points in a sufficienlly large dataset assuming that the majority of points share the same structure or properties. We note that novelty detection is equivalent to the well-known one-class classification problem (Moya & Hush 1996). In this problem, one needs to identify members of a class in a test dataset, and consequently distinguish them from "novel" data points, given training points from this class. The points of the main class are commonly referred to as inliers and the novel ones as outliers.
Novelty detection is also commonly referred to as semi-supervised anomaly detection. In this terminology, the notion of being "semi-superrvset" " is different than usual. It emphasizes that only the inliers are trained, where there is no restriction on the fraction of training points. On the other hand, the unsupervised case has no training (we referred to this setting above as "outlier detection")) and in the supervised case there are training datasets for both the inliers and outliers. We remark that some authors refer to semi-supervised anomaly detection the setting where small amount of labeled data provided for both the inliers and outliers (Ruff et al. 2020
There are a myriad of solutions to novelty detection. Nevertheless, such solutions often assume that the training set is purely sampled from single class or that it has very low fraction of corrupted samples. This assumption is only valid when the area of investigation has been carefully studied and there are sufficiently precise tools to collect data. However, there are different important scenarios, where this assumption does not hold. One scenario includes new areas of studies, where it is unclear how to distinguish between normal and abnormal points. For example, in the beginning the COVID-19 pandemic it was hard to diagnose COVID- 19 patients and distinguish them from other patients with pneumonia. Another scenario occurs when it is very hard to make precise measurements, for example, when working with the highly corrupted images obtained cryogenic electron microscopy (cryo-EM).
Therefore, we study a robust version of novelty detection that allows a nontrivial fraction of corrupted samples, namely outliers, within the training set. We solve this problem by using a special variational

---

Under review as conference paper at ICLR 2021
autoencoder (VAE) KKigmm& & Welling 2014). Our VAE is able to model the underlying distribution of the uncorrupted data, despite nontrivial corruption. We refer to our new method as "Mixture Autoencoding with Wasserstein penalty", or "MAW". In order to clarify it, we first review previous works and then explain our contributions in view of these works.
1.1 PREVIOUS WORK
Solutions to one-class classification and novelty detection either estimate the density of the inlier
distribution (Bengio & Monperrus 2005 [Ione et al. (006) or determine a geometric property
of the inliers, such as their boundary set (Breunig et al. 2000) Schölkopf et al. [000] Xiao et al.)
2016 Wang & Lan 2020 Jiang et al. 2019). When the inlier distribution is nicely approximated by a low-dimensional linear subspace, (Shyu et al. (2003) proposes to distinguish between inliers and outliers via Principal Component Analysis (PCA). In order to consider more general cases of nonlinear low-dimensional structures, one may use autoencoders (or restricted Boltzmann machines), which nonlinearly generalize PCA (Goodfellow et al. [016 Ch. 2) and whose reconstruction error naturally provides score for membership in the inlier class. Instances of this strategy with various architectures include Zhai /hai et al. (2016); Zong et al. (2018); ()aoSkrrrorouu et al. (2018): [eerera et al. (2019); Pidhorskyi et al. (2018) In all of these works, but Zong et (2018). the training set is assumed to solely represent (2019) observed that interpolation of latent space, which was
the inlier class. In fact. |Perera et al.
fact,/Perera
trained using digit images of a complex shape, can lead to digit representation of a simple shape. If there are also outliers (with a simple shape) among the inliers (with complex shape), encoding the inlier distribution becomes even more difficult. Nevertheless, some previous works already explored the possibility of corrupted training set (Xiao et al. 2016 Wang & Lan 2020: Zong et al. (2018). In particular, Xiao et al. (216)) Zong et (2018) test artificial instances with at most 5% corruption of the training set and Wang & Lan (2020) considers ratios of 10,, but with very small numbers of training points. In this work we consider corruption ratios up to 30%, with a method that tries to estimate the distribution of the training set, and not n geometrie property.
VAES Kingma & Welling, 2014) have been commonly used for generating distributions with reconstruction scores and are thus natural for novelty detection without corruption. They determine the latent code of an autoencoder via variational inference (Jordan al. 1999 Blei al. 2017).
Altemaaively, they can be viewed as autoencoders for distributions that penalize the Kullback-Leible (KL) divergence of the latent distribution from the prior distribution. The first VAE-based method for
novelty detection was suggested by Ay & Cho (2015). It was recently extended by Daniel al. (2019)
who modified the training objective. A variety of VAE models were also proposed for special anomaly detection problems, which are different than novelty detection (Xu et al. 2018 Zhang et al. 2019 Pol al. 2019). Current VAB-based methods for novelty detection do not perform well when the training data is corrupted. Indeed, the learned distribution of any such method also represents the corruption, that is, the outlier component. To the best of our knowledge, no effective solutions were proposed for collapsing the outlier mode so that the trained VAE would only represent the inlier distribution.
An adversarial autoencoder (AAE) (Makhzani et al. 2016) and a Wasserstein autoencoder (WAE) (Tolstikhin et al. 2018) can be considered as variants of VAE. The penalty term of AAE takes the form o generative adversarial network (GAN) (Goodfellow et al. 2016), where its generator is the encoder. A Wasserstein autoencoder ( (WAE) (Tolstikhin et al. 2018) generalizes AAE with a framework that minimizes the Wasserstein metric between the sample distribution and the inference distribution. It reformulates the corresponding objective function so that it can be implemented in the form of an AAE.
There are two relevant lines of works on robustness to outliers in linear modeling that can be used in nonlinear settings via autoencoders VAEs. Robust PCA aims deal with sparse elementwise corruption of a data matrix (Candès et al. [201 De La Torre& & Black 2003 Wright et al. 2009 Vaswani
& Narayanamurthy 2018). Robust subspace recovery (RSR) aims to address general corruption of
selected data points and thus better fits the framework of outliers (Watson| 2001 De La Torre& & Black
De
2003: Ding et al. 2006 Zhang et al.] [209) McCoo & Tropp 2011 Xu et al. 2012) Lerman & Zhang
|
2014 2014 Zhang & Lerman L al. [013] Lerman& & Maunu 2017 Maunu et | Lerman
L
& Lerman]|2200 2019). Autoencoders that use robust PCA for anomaly detection tasks were proposed in Chalapathy et al.
& Maunu
& Paffenroth (217)) Dai et al. ( show that a VAE can be interpreted as nonlinear robust PCA problem. Nevertheless, explicit regularizaii is often required to improve robustness to sparse corruption in VAEs (Akrami al. 2019 Eduardo et al. 2020). RSR was successfully applied to outlier detection by Lai et al. (2020). One can apply their work to the different setting of novelty detection; however, our proposed VAE formulation seems to work better.
2

---

Under review as conference paper at ICLR 2021
1.2
THIS WORK
We propose a robust novelty detection procedure, MAW, that aims to model the distribution of the training data in the presence of nontrivial fraction of outliers. We highlight its following four features:
MAW models the latent distribution by Gaussian mixture of low-rank inliers and full-rank outliers, and applies the inlier distribution for testing. Previous applications mixture models for novelty de- tection were designed for multiple modes of inliers and used more complicated tools such as construct-
ing another network ZZong et al. 2018) applying clustering (Aytekin et al. 2018 Lee al. 2018). MAW applies a novel dimension reduction component, which extracts swwer-dimensiones of the latent distribution. The reduced small dimension allows using full covariances for both the outliers (with full rank) and inliers (with deficient rank); whereas previous VAE- based methods for novelty detection used diagonal covariances in their models (An & Chol 2015 Daniel et al. 2019). The new component is inspired by the RSR layer in Lai et al. different since the RSR layer is only applicable for data points and not for probability distributions.
features
| (2020); however, they are essentially
For the latent code penalty, MAW uses the Wasserstein-1 1 (W1) metric. Under a special setting, prove that the Wasserstein metric gives rise to outliers-robust estimation and is suitable the low-rank modeling of inliers by MAW. We also show that these properties do not hold for the KL divergence, which is used by VAE, AAE and WAE. We remark that the use of the Wasserstein metric in WAE is different than that of MAW. Indeed, in WAE it measures the distance between the data distribution and the generated distribution and it does not appear in the latent code. Our use of W can be viewed as a variant of AAE, which replaces GAN with Wasserstein GAN (WGAN) (Arjovsky ct al. (017). That is, replaces the minimization of the KL divergence by that of the W1 distance. MAW achieves state-of-the-arr results on popular anomaly detection datasets.
Additional two features are as follows. First, for reconstruction, MAW replaces the common least squares formulation with least absolute deviations formulation. This can be justified by the use of either a robust estimator (Lopuhaa & Rousseeuw 1991) or a likelihood function with a heavier tail. Second, MAW is attractive for practitioners. It is simple to implement in any standard deep learning library, and is easily adaptable to other choices of network architecture, energy functions and similarity scores.
We remark that since we do not have labels for the training set, we cannot supervisedly learn the Gaussian component with low-rank covariance by the inliers and Gaussian component with the full-rank covariance by the outliers. However, the use of two robust losses (least absolute deviation and the W1 distance) helps obtain a careful model for the inliers, which is robust to outliers. Note that in our testing, we only use the model for the inliers.
We explain MAW in $
We establish the advantage of its use of the Wasserstein metric in 3 We carefully test MAW in At last, we conclude this work in $
2 DESCRIPTION OF MAW
We motivate and overview the underlying model and assumptions of MAW in 2.T] We describe the simple implementation details of its components in 2.2 Fig. Tillssrrates the general idea of MAW and can assist in reading this section.
2.1 THE MODEL AND ASSUMPTIONS OF MAW
MAW aims to robustly estimate a mixture inlicr-outlier distribution for the training data and then use its inlier component to detect outliers in the testing data. For this purpose, it designs a novel variational autoencoder with an underlying mixture model and a robust loss function in the latent space. We find the variational framework natural for novelty detection. Indeed, it Icarns a distribution that describes the inlier training examples and generalizes to the inlier test data. Moreover, the variational formulation allows a direct modeling of Gaussian mixture model in the latent space, unlike standard autoencoder.
We assume L training points in IR, which we designate by {x/ Let be random variable on D with the unknown training data distribution that we estimate by the empirical distribution of the training points. We assume latent random variable low and even dimension 2 I d S D, where our choice d = 2. We further assume a standardized Gaussian prior, p(z), (z), so that z ~ N NN0 Taxa). The posterior distribution p(z p(z|x) is unknown. However, assume approximation to it. which we denote by ((]|x), such that z x is a mixture of two Gaussian distributions representing the inlier and outlier components. More specifically, z/x ~ (N En) + (1 - ( E²), where we ex explain next its parameters. We assume that 1) > 0.5, where our default value 1) 5/6, that the
3

---

Under review as conference paper at ICLR 2021
Testing set: Croissants
Inliers: Cats
Bockpropagation
LyAE
40 [16L
Training (X
Encoder
Dimension Reduction
[200
Decoder (TT))
Component
P
Discrtminatoo (Eir 2 Eix 2
Inlier Mode NC 20
Cutller Mode NCC m
2(2 Gonssian mixeturg Backgropugation
dittibation (sandded Cuusiiai
Outliers: Croissants
Lwys
MA W
Using Inlier Madie
Similarity Score
Figure 1: Demonstration of the architecture of MAW for novelty detection.
first mode of represents the inliers and the second one represents the outliers. The other parameters are generated by the encoder network and a following dimension reduction component. We remark that unlike previous works which adopted Gaussian mixtures to model the clusters of inliers (Reddy) al. 2017 Zong al. 2018). the Gaussian mixture model in MAW aims to separate between inliers and outliers. The dimension reduction component involves a mapping from a higher-dimensional space onto the latent space. It analogous the RSR layer in Lai et al. (2020) that projects encoded points onto the latent space, but requires a more careful design since we consider a distribution rather than sample points. Due to this reduction, we assume that the mapped covariance matrices of z x are full, unlike common single-mode VAE models that assume diagonal covariance (Kingma & Welling 2014, An & Cho 2015). Our underlying assumption is that the inliers lie on low-dimensional structure and we thus enforce the lower rank d/2 for Σ,, but allow Σg to have full rank d. Nevertheless, later describe necessary regularization of both matrices by the identity.
Following the VAE framework, we approximate the unknown posterior distribution p(z]x x) within the variational family Q {q(z|x)), which is indexed by 141. Σ, μ12 and Σg. Unlike standard VAE, which maximizes the evidence lower bound (ELBO), MAW maximizes the following ELBO-Ward B00--Waserstten. or ELBOW, function, which uses the W, distance (see also IA.I)
ELBOW(g) Bo(x)) log p( / W1 (1(g(z), p(z))
Following the VAE framework, we use Monte-Care approximation to estimate Eg(e))) (xx|(x(z) 1T=1,
iith samples, (z(0) from q(((x) as follows:
Ex(z(x) logp(x(z)
Σ log ((x
To improve the robustness of our model, we choose the negative log likelihood function log (( (2) to be constant multiple of the lg norm of the difference of the random variable and mapping of the sample ( from R.d to TR: by the decoder, DD that is,
- log -logp(((z(")) x x D(z (12
(3)
Note that we deviate from the common choice of the squared l2 norm, which corresponds to an underlying Gaussian likelihood and assume instead a likelihood with a heavier tail.
MAW trains its networks by minimizing -ELBOW(q). For any 1 S i L, it samples {z |r=1
(i.1)
from ((z)x(()), (.) where all samples are independent. Using the aggregation formula: ((x)
--1 اا ())) which is also used by an AAE, the approximation of p(x) ) by the empirical dis-
tribution of the training data, and (I-(3) (3). MAW applies the following approximation -EL-ELBOOO (q):
x(()
+ Wi
E L 1=1
((z)x())), p(zz)
(4)
LT
2
Details of minimizing are described in $2.2 independent of the multiplicative constant in 5
We remark that the procedure described in $2.2 is
and therefore this constant is ignored in (4)
4

---

Under review as conference paper at ICLR 2021
During testing, MAW identifies inliers and outliers according high or low similarity scores computed between each given test point and points generated from the learned inlier component of x.
2.2 DETAILS OF IMPLEMENTING MAW
MAW has a VAE-type structure with additional WGAN-type structure for minimizing the Wi loss in 4. We provide here details of implementing these structures. Some specific choices the networks are described in since they may depend on the type of datasets.
The VAE-type structure of MAW contains three ingredients: encoder, dimension reduction component and decoder. The encoder forms neural network & that maps the training sample x(9) E RDD
(() RP',, where our default choice is D' 128. The dimension reduction component then computes the following statistical quantities of the Gaussian mixture z (+) means (0) and fuz in Rd and covariance matrices 2 and Σ in RRdxd First, linear layer, represented by A E R xd, maps (via A the features Mo,1 and M0, in R2'' to the following respective vectors in Rd. didag(s)))) A. For 2, compute x0 M2 M j 1, we first need to reduce the rank of M For this purpose, we form M U d (')
(6) (i)
to Mo.1 M0,2: 80,1, 0,2
(s)
(i)
f4 (1) = A A7M0.11 1 and p12 M.2
(i) For = 1, 2. form M
the spectral decomposition of M and then truncate its bottom d/2 eigenvalues. That is, let i (i) IRd have the same entries as the largest d/2 entries of 1
(i) and zero entries otherwise. Then, compute
M ( U1" diag (ō (0) )
(6)
and 2 M, M, (S) Since the TensorFlow package requires numerically-signfiienn positive
definiteness of covariance matrices, add identity matrix both > and > Despite this, the low-rank structure of { is still evident. Note that the dimension reduction component only trains A.
The decoder, D Rd R.,. maps independent samples, (t.6) V
Jt- generated for each < L
by the distribution n, (N( ( + (1 - m)) (6) xx(" ), into the reconstructed data space.
(s)
The loss function associated with the VAE structure is the first term in 44.
We can write it as
L T
LVAR(E, (Ar((E,,,)) LT
ΣΣ
D
gen
The dependence of this loss on & and A is implicit, but follows from the fact that the parameters of
(5.t) the sampling distribution of each Zgen were obtained by & and A.
The WGAN-type structure seeks to minimize the second term in (4) using the dual formulation
Wi Σ [Z)xx ).
))p())))(())))))
sup
Ea (Zhyp)
E i Etsn a(z)xin) (Zagen)
(8)
|| 11
The generator of this WGAN-type structure is composed of the encoder and the dimension reduction component, which we represent by A. It generates the samples {zgen Jiml,t= described above. The discriminator, Dis, of the WGAN-type structure plays the role of the Lipschitz function f in 8
L.T
compares the latter samples with the i.i.d. samples (2(5g) ) from the prior distribution. In order make Dis Lipschitz, its weights are clipped to [-1, 1] during training. In the MinMax game of this WGAN-type structure, the discriminator minimizes and the generator (E and A) maximizes
(f.t)
Tw (Dis) = -
1
Σ i=1 t=1
Dis (is((((tt)) Dis (t.0) Payp
LT
We note that maximization of by the generator is equivalent to minimization of the loss function
LGEN(E,
(zs(z((e))
(10)
gen
During the training phase, MAW alternatively minimizes the losses a weighted sum. Therefore, any multiplicative constant in front of either term of 64) will not effect the optimization. In particular, okay omit the multiplicative constant of when deriving (4)
instead of minimizing
5

---

Under review as conference paper at ICLR 2021
For each testing point ( we sample ( } from the inlier mode of the learned latent Gaussian (1:0)
mixture and decode them as (5((.4) Y (D in
(14t ) Using a similarity measure S(;,) (our is the cosine similarity), compute 5(J) E S(yl), g(.))) If
(f is larger than a chosen threshold, then (i) is classified normal, and otherwise, novel. Additional details of MAW are in A
3 THEORETICAL GUARANTEES FOR THE W1 MINIMIZATION
Here and in SD we theoretically establish the superiority of using the Wi distance over the KL divergence. We formulate simplified setting that aims to isolate the minimization of the WGAN-type structure introduced in $2.2 while ignoring unnecessary complex components of MAW. We assume a mixture parameter 1y 172, separation parameter > 0 and denote by R the regularizing function, which can be either the KI. divergence or W1, and by S and S the sets of K x K positive semidefinite and positive definite matrices, respectively. For /A0 E RRK and Eo E SK we consider the minimization problem
min (1.MEER* ExESK s.t. ||]4: -eall226
T/R (M E1), En)) + (1 7) (M E2), Eo)o (11)
We further motivate in $D11 For MAW, 440 = 0 and Eg I, but our generalization helps clarify things. This minimization aims to approximate the "prior" distribution ((H, Eo) with a Gaussian mixture distribution. The constraint || - H2ll2 26 distinguishes between the inlier and outlier modes and it is realistic assumption long e is sufficiently small.
Our cleanest result when Eg. En and Σg coincide. It demonstrates robustness to the outlier com- ponent by the W1 (or W,, p 2 1) minimization and not by the KL minimization (its proof is in $D2)..
Proposition 3.1. If Puu E IRK,, Eo SS R Wp, p 2 and the additional constraint: En = Σ = Σg, satisfies μ = p4o, and thus the recovered inlier distribution coincides with the "prior distribution" However, the minimizer of [T with R- KL L and the same constraint satisfies p4 7/11 + (1 122
and
n > 1/2, then the minimizer of 11 with
In $ D.3 we analyze the case where Σ₁ is low rank and Σ₂ E S We show that 11 is ill-defined when R. L.. The R = Wi case is hard to analyze, but we can fully analyze the R = W2 case and demonstrate exact recovery of the prior distribution by the inlier distribution when v approaches 1.
4 EXPERIMENTS
We describe the competing methods and experimental choices in $4.1
We report on the comparison with the competing methods in 442 We demonstrate the importance of the novel features of MAW in $4.
4.1 COMPETING METHODS AND EXPERIMENTAL CHOICES
We compared MAW with the following methods (descriptions and code links are in Deep
Autoencoding Gaussian Mixture Model (DAGMM) (Zong et al. 2018). Deep Structured Energy- Based
Models (DSEBMs) (Zhai et al. 2016). Isolation Forest (IF) (Liu el al. 2008). Local Outlier Factor
(LOF) (Breunig et al. 2000), One-class Novelty Detection Using GANs (OCGAN) (Perera al. 2019)
One-Class SVM (OCSVM) (Heller et al. 2003) and RSR Autoencoder (RSRAE) (Lai et al. 2020). DAGMM, DSEBMs, OCGAN and OCSVM M were proposed for novelty detection. IF, LOF and RSRAE
were originally proposed for outlier detection and we thus apply their trained model for the test data.
For MAW and the above four reconstruction- based methods, that is, DAGMM, DSEBMs, OCGAN and RSRAE, we use the following structure of encoders and decoders, which vary with the type of data (images or non-images). For non-images, which are mapped feature vectors of dimension D, the encoder is a fully connected network with output channels (32, 64, 128, 128 x 4). The decoder is a fully connected network with output channels (128, 64, 32, D). followed by normalization layer at the end. For image datasets, the encoder has three convolutional layers with output channels (32, 64, 128). 2, kernel sizes (5 5. 5 X 3, x 3) and strides 2, 2). Its output is flattened to lie in 1 and then mapped into a 128 x 4 dimensional vector using dense layer (with output channels 128 x 4). The decoder of image datasets first applies a dense layer from IR² to R 128 and then three deconvolutional layers with output channels (64, 32, 3), kernel sizes (3 x 3, 5 X 5, 5) and strides (2, 2, 2).

---

Under review as conference paper at ICLR 2021
For MAW set the following parameters, where additional details are in $A Intrinsic dimension: d = mixture parameter: 7) = 5/6, sampling number: = 5, and size of A (used for dimension reduction): 128 x 2. For all experiments, the discriminator is a fully connected network with size (32, 64, 128, 1).
4.2 COMPARISON OF MAW WITH STATE-OF- METHODS
We use five datasets for novelty detection: KDDCUP-99 (Dua & Graft 2017 Reuters-21578 (Lewis 1997). COVID-19 Radiography database (Chowdhury et al. 2020). Caltech101 (Fei-Fei et al.
2004) and Fashion MNIST' (Xiao et al.) 2017). We distinguish between image datasets (COVID-19)
Catlech and Fashion MNIST) and non-image datasets (KDDCUP-99 and Reuters-21578) We de- scribe each dataset, common preprocessing procedures and choices of their largest clusters in $F Each dataset contains several clusters (2 for KDDCUP-99 99. 5 largest ones for Reuters-21578, 3 for COVID-19, 11 largest ones for 10 and 10 for Fashion MNIST). We arbitrarily fix class and uniformly sam- ple training inliers and Nus testing inliers from that class. We let N 6000, 350, 160, 100, 100,300 and Niest 1200, 140, 60, 100. 60 for KDDCUP-99 Reuters-21578. COVID-19, Caltech 101 and Fashion MNIST, respectively. We then fix in f0.1. 0.3, 0.4, 0.5), and uniformly sample e percentage of outliers from the rest of the clusters for the training data. We also fix Gust in (0.1, 0.3, 0.5, 0.7, 0.9} and uniformly sample Gest percentage of outliers from the rest of the clusters for the testing data.
Using all possible thresholds for the finite datasets, we compute the AUC (area under curve) and AP (average precision) scores, while considering the outliers as "positive". For each fixed c = 0.1, 0.2, 0.3, 0.4, 0.5 we average these results over the values of Gest, the different choices of inlier clusters (among all possible clusters), and three runs with different random initializations for each of these choices. We also compute the corresponding standard deviations. We report these results in Figs, 2 and 3 and further specify numerical values in $ We observe state-of-the-ar performance of MAW in all of these datasets. In Reuters-211 21578, DSEBMs performs slightly better than MAW and OCSVM has comparable performance. However, these two methods are not competitive in the rest of the datasets. In S we report results for a different scenario where the outliers of the training and test sets have different char- acteristics. In this setting, we show that MAW performs even better when compared to other methods.
MAW
DAGMM
DSEBMs
IF
LOF
OCGAN
OCSVM
RSRAE
1.0
0.9
0.7
0.8
KDDCUP-99
AUC score
0.6
0.7
AP soUre
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.1
0.2
0.3
0.1
0.2
0.3
0.4
The constant e
The constant c
1.0
0.8
0.9
0.7
Reuters-2 2158
0.8
AUC spore
0.7
AP score
0.6
0.5
0.6
0.5
0.4
0.4
0.3
0.3
0.1
0.2
0.1
0.2
0.3
The constant
The constant c
Figure 2: AUC (on left) and AP (on right) scores with training outlier ratios 0.5 for the two non-image datasets: KDDCUP-99 and Reuters-21578.
0.1, 0.2, 0.3, and
7

---

Under review conference paper at ICLR 2021
MAW LOF
DAGMM OCGAN
DSEBMs
OCSVM
RSRAE
0.70
0,45
COVID-I9
0.60
0,40
AP score
AUC
0.55
0.35
0.50
0.30
0.45
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
The
The constant
0.8
0.6
0.7
Caltech 101 AUC
0.5
Al
0.4
0.3
0.2
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0,5
constant
The constant
0.9
0.8
Fashion MNIST
0.8
0.6
AUC
0.6
AP
0.5
0.4
0.4
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
The constant
The constant
Figure 3: AUC (on left) and AP (on right) scores with training outlier ratios 0.2, 0.3, and 0.5 for the three image datasets: COVID-19, Caltech101 and Fashion MNIST.
4.3 TESTING THE EFFECT OF THE NOVEL FEATURES OF MAW
We experimentally validate the effect of the following five new features of MAW: the least absolute deviation for reconstruction, the W1 metric for the regularization the latent distribution, the Gaussian mixture model assumption, full covariance matrices resulting from dimension reduction component and the lower rank constraint for the inlier mode. The following methods respectively replace each of the above component of MAW with a traditional one: MAW-MSE, MAW-KL, divergence, MAW-same rank, MAW-single Gaussian and Aiidooago cov., respectively. In addition, consider standard variational autoencoder (VAB). Additional details for the latter six methods are B
We compared the above six methods with MAW using two datasets: KDDCUP-99 and COVID-19 with training outlier ratio 0.1, 0.2 and 0.3. We followed the experimental setting described in $4.1 Fig. 4 Preports the averages and standard deviations of the computed AUC and AP scores, where the corresponding numerical values are further recorded in H. The results indicate a clear decrease of accuracy when missing any of the novel components of MAW using standard VAE.
8

---

Under review conference paper at ICLR 2021
MAW
MAW-KL divergence
MAW single Gaussiun
VAE
MAW-MSE
MAW-same
MAW-diagonal
0.8
0.95
0.90
0,7
KDDCUP-99
AUC 500re
AP scorc
0.6
0.75
0.70
0,5
0.1
0.2
0.3
0.1
0.2
0,3
0.70
0.46
0.65
0.44
COVID-19
AUC soure
0.60
0.42
0.55
0.38
0.50
0.36
0,1
0.2 The
0.3
0.1
0,2
0.3
Figure 4: AUC (on left) and AP (on right) scores for variants of MAW (missing a novel component) with training outlier ratios e = 0, 0.2, 0.3 using the KDDCUP-99 and COVID-19 datasets.
5 CONCLUSION AND FUTURE WORK
We introduced MAW, a robust VAE-type framework for novelty detection that can tolerate high corruption of the training data. We proved that the Wasserstein regularization used in MAW has better robustness to outliers and is more suitable to a low-dimensional inlier component than the divergence. We demonstrated sate-of-the--al performance of MAW with variety of datasets and experimentally validated that omitting any of the ideas results in significant decrease of accuracy.
We hope to extend our proposal the following ways. First of all, we plan to extend and test some of our ideas for the different problem of robust generation, in particular, for building generative networks which robust against adversarial training data. Second of all, would like carefully study the virtue our idea of modeling the most significant mode in a training data. In particular, when extending the work to generation, one has to verify that this idea does not lead to mode collapse. Furthermore, we would like to explore any tradeoff of this idea, as well our setting of robust novelty detection, with fair- ness. At last, we hope to further extend our theoretical guarantees. For example, two problems that cur- rently seem intractable are the study of the W1 version of Proposition D.I and of the minimizer of (14).

---

Under review as conference paper at ICLR 2021
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, lan Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowizz, Lukasz Kaiser, Manjunath Kudlur. Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore. Derek Murray, Chris Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URLhttps: /www wwwsorensoroooo org/ Software available from tensorflow.org.
Olah,
Martial Agueh and Guillaume Carlier. Barycenters in the Wasserstein space. SIAM Journal on Mathematical Analysis, 43(2):904 900)29949249220000 2011.
Haleh Akrami, Anand A Joshi, Jian Li, and Richard M Leahy. Robust variational autoencoder. arXiv preprint arXiv: /9wp:://050.991.. 2019.
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruction probability. Special Lecture IE, 2(1), 2015.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, pp. 214-223, 1144-22.MMLR,, PMLR, 2017.
Caglar Aytekin, Xingyang Ni, Francesco Crieri, and Emre Aksu. Clustering and unsupervised anomaly detection with 1 normalized deep auto-encoder representations. In 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1 1-6. IEEE, 2018.
Yoshua Bengio and Martin Monperrus. Non-local manifold tangent learning. In Advances in Neural Information Processing Systems, pp. 129-136, 2005.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mytec ad-a comprehensive the IEEE Conference
real-world dataset for unsupervised anomaly detection. Proceedings the on Computer Vision and Pattern Recognition, pp. 9592-9600, 2019.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 11((518):55--777 877,
Markus M Breunig, Hans- PPter Kriegel, Raymond T Ng, and Jörg Sander. LOF: identifying density-based based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pp. 93-104, 2000.
Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Arnad Joly, Brian Holt, and Gaël Varoquaux. API design for machine learning software: experiences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Mining and Machine Learning. pp. 108-122, 2013.
Emmanuel J. Candès, Xiaodong Li, Yi Ma. and John Wright. Robust principal component analysis? J. ACM, 58(3), June 2011. ISSN 0004-5411, doi: 10.1145/1970392.1977 1970395. URL https::/doi..rg/// 11t..//9/......1111.11..........
Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: A survey. arXiv preprint 1xXiv:1901.03-077 2019.
Raghavendra Chalapathy, Aditya Krishna Menon, and Sanjay Chawla. Robust, deep and inductive anomaly detection. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 36-51. Springer, 2017.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey, ACM computing surveys (CSUR) (13): 2009.
Yongxin Chen. Tryphon T Georgiou, and Allen Tannenbaum. Optimal transport for gaussian mixture models. IEEE Access, 1:6269-6278 2018.
10

---

Under review as conference paper at ICLR 2021
Muhammad EH Chowdhury, Tawsifur Rahman, Amith Khandakar, Rashid Mazhar, Muhammad Abdul Kadir, Zaid Bin Mahbub, Khandakar Islam, Muhammad Salman Khan, Atif Iqbal, Nasser Al-Emadi, et al. Can Al help in screening viral and COVID-19 pneumonia? arXiv preprint 1rXiv:2003.13145, 2020.
Bin Dai, Yu Wang, John Aston, Gang Hua, and David Wipf. Connections with robust pea and the role of emergent sparsity in variational autoencoder models. The Journal of Machine Learning Research, 19(1):1573-166 1614, 2018.
Tal Danicl, Thanard Kurutach, and Aviv Tamar. Deep variational semi-supervised novelty detection. arXiv preprint arXiv: 1,.977,,,0 2019.
Fernando De La Torre and Michael J Black. A framework for robust subspace learning. International Journal Computer Vision, 54(1-3): 117-142 2003.
Chris Ding, Ding Zhou, Xiaofeng He, and Hongyuan Zha. RI-PCA: rotational invariant ( -nnrm principal component analysis for robust subspace factorization. In Proceedings of the 23rd international conference on Machine learning, pp. 281-288. ACM, 2006.
Dheeru Dua and Casey Graff.
UCI machine learning repository, 2017. URL http:
//archive. ics. wcw.uui.edu/mmmmmmccccccm
Simão Eduardo, Alfredo Nazábal, Christopher K. I. Williams, and Charles Sutton. Robust variational autoencoders for outlier detection and repair of mixed-type data. In AISTATS, 2020.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop. pp. 178-178. IEEE, 2004.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Izhak Golan and Ran Yaniv.. Deep anomaly detection using geometric transformations. In Advances in Neural Information Processing Systems, pp. 9758 9769, 2018.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Katherine Heller, Krysta Svore, Angelos Keromytis, and Salvatore Stolfo. One class support vector machines for detecting anomalous windows registry accesses. 2003.
John R Hershey and Peder A Olsen. Approximating the Kullback Leibler divergence between gaussian mixture models. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSS ICASSP 07, volume 4, pp. IV-317. IEEE, 2007.
Jarmo Ilonen, Pekka Paalanen, J-K Kamarainen, and H Kalviainen. Gaussian mixture pdf in one-class classification: computing and utilizing confidence values. In /8th International Conference Pattern Recognition (ICPR'06) volume 2, pp. 577-580. IEEE, 2006.
Hansi Jiang, Haoyu Wang. Wenhao Hu, Deovrat Kakde, and Arin Chaudhuri. Fast incremental SVDD learning algorithm with the Gaussian kernel. Proceedings of the AAAI Conference Artificial Intelligence, 33(01)33911-300 -39989 Jul. 2019. doi: 10. wtl11669aaii.v.33 v33101 33013991.. URI.
https::/ojj...
aaai. org/ index.niix.////i php/ /ttp///A/aaaicee/eee//////////
Michael Jordan, Zoubin Ghahramani. Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Muchine learning, 37(2):183-233, 1999.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference for Learning Representationss 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014. URL
33x9rd2
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.

---

Under review as conference paper at ICLR 2021
Chieh-Hsin Lai, Dongmian Zou, and Gilad Lerman. Robust subspace recovery layer for unsupervised anomaly detection. In International Conference on Learning Representations, 2020. URL
https://openreviee net forum?id-ryyb3ee
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, pp. 7167-7177, 2018.
Gilad Lerman and Tyler Maunu. Fast, robust and non-convex subspace recovery. Information and Inference: Journal the IMA, 7(2):277-336, 2017.
Gilad Lerman and Tyler Maunu. An overview of robust subspace recovery. Proceedings of the IEEE, 2018.
Gilad Lerman and Teng Zhang. 1p-recovery of the most significant subspace among multiple subspaces with outliers. Constructive Approximation, 40(3):329 -0099385570 2014.
Gilad Lerman, Michael B McCoy, Joel A Tropp, and Teng Zhang. Robust computation of linear models by convex relaxation. Foundations of Computational Mathematics, 15(2):363 410, 2015.
David Lewis. Reuters-21577 text categooriin test collection. Distribution 1.0, AT&T Labs-Research, 1997.
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 Eighth IEEE International Conference on Data Mining, pp. 413-421222 IEEE, 2008
Hendrik P. Lopuhaa and Peter J. Rousseeuw. Breakdown points of affine equivariant estimators of multivariate location and covariance matrices. Ann. Statist., 19(1):22-2248, 03 1991. doi: 10.1214/aos/1 1a11214/axx/1166377999 176347978. URL/attps://dol. 0tt.://...1111//0/1777759939939
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and lan Goodfellow. Adversarial autoen- coders. In International Conference on Learning Representations Workshop, 2016. URL
httpps://peereeviww http://www......o...o.oooo/o//ea
Tyler Maunu and Gilad Lerman. Robust subspace recovery with adversarial outliers. arXiv preprint arXiv:/904.03275. 2019.
Tyler Maunu, Teng Zhang, and Gilad Lerman. A well-tempered landscape for non-convex robust subspace recovery. Journal of Machine Learning Research, 20(37):1-59, 2019.
Michael McCoy and Joel A Tropp. Two proposals for robust PCA using semidefinite programming. Electronic Journal of Statistics, 1211123-166 2011
Mary Moya and Don Hush. Network constraints and opt.iiiieciiieeeeiiiiiiiii for one-class classification. Neural Networks, 9(3):463447,, 1996.
Hugo Vieira Neto and Ulrich Nehmzow. Real-time automated visual inspection using mobile robots. Journal of Intelligent and Robotic Systems, 49(3):293-307, 2007.
Victor M Panaretos and Yoav Zemel. Statistical aspects of wasserstein distances. Annual review of statistics and its application, 6:405-431 2019.
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. OCGAN: One-class novelty detection using gans with constrained latent representations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2898-2906 2019.
Gabriel Peyré, Marco Cuturi, al. Computational optimal transport: With applications to data science. Foundations and Trends R in Machine Learning, 11(5-6):355-60777 207,
Stanislav Pidhorskyi, Ranya Almohsen, and Gianfranco Doretto. Generative probabilistic novelty detection with adversarial autoencoders. In Advances in neural information processing systems, pp. 683226833,.0 2018
Marco AF Pimentel, David A Clifton. Lei Clifton, and Lionel Tarassenko. A review of novelty detection. Signal Processing, 99:215-249, 2014.
12

---

Under review as conference paper at ICLR 2021
Adrian Pol, Victor Berger, Gianluca Cerminara, Cecile Germain, and Maurizio Pierini. Anomaly detection with conditional variational autoencoders. HAL archive preprint oa2-09296779, 2019.
Anand Rajaraman and Jeffrey David Ullman. Mining of massive datasets. Cambridge University Press, 2011.
Aarthi Reddy, Meredith Hrdway-Wesst Melissa Lee, Matt Dugan, Joshua Whitney, Ronen Kahana, Brad Ford, Johan Muedsam, Austin Henslee, and Max Rao. Using gaussian mixture models to detect outliers in seasonal univariate network traffic. In 2017 IEEE Security and Privacy Workshops (SPW). pp. 229 234. IEEE, 2017.
Lukas Ruff, Robert A. Vandermeulen. Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus-Robert Müller, and Marius Kloft. Deep semi-supervised anomaly de- tection. In International Conference on Learning Representations, 2020. URL
htttps://ppenrev/
net. htt.//ooummieeeee Fo
Mohammad Sabokrou, Mohammad Khalooei, Mahmood Fathy, and Ehsan Adeli. Adversarially learned one-class classifier for novelty detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3379-3388 2018.
Bernhard Schölkopf, Robert C Williamson, Alex J Smola, John Shawe -aylor, and John C Platt. Support vector method for novelty detection. In Advances in neural information processing systems, pp. 582-588, 2000.
Mei-Ling Shyu, Shu-Ching Chen, Kanoksri Sarinnapakorn, and LiWu Chang. A novel anomaly detection scheme based on principal component classifier. In ICDM Foundation and New Direction of Data Mining workshop, 2003.
Ilya Tolstikhin, Olivier Bousquet. Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. In International Conference on Learning Representations, 2018. URL
https::/oppereevlt
net htte///ooummmmdkkkkmm
Namrata Vaswani and Praneeth Narayanamurthy. Static and dynamic robust PCA and matrix completion: A review. Proceedings of the IEEE, 106(8):135--1379 2018.
Kunzhe Wang and Haibin Lan. Robust support vector data description for novelty detection with contaminated data. Engineering Applications of Artificial Intelligence, 91:103554, 2020. ISSN 0952-1976. doi: http///eweeeaatttee/////////////
2020. 103554. URL http:
www. www.scieecedi rect. com
G. Alistair Watson. Some Problems in Orthogonal Distance and Non-Orthogonal Distance Regression. Defense Technical Information Center, 2001.
URL http:
bbooks.google.. google.
l/bbook.
Qi Wei, Yinhao Ren, Rui Hou, Bibo Shi, Joseph Y Lo, and Lawrence Carin. Anomaly detection for medical images based on a one-class classification. In Medical Imaging 2018: Computer-A Diagnosis, volume 10575, pp. 105751M. International Society for Optics and Photonics, 2018.
John Wright, Arvind Ganesh, Shankar Rao, Yigang Peng, and Yi Ma. Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization. In Advances neural information processing systems. pp. 2080-2088, 2009.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist novel image dataset for benchmarking machine learning algorithms. arXiv preprint (70.: 2017."
Yingchao Xiao, Huangang Wang, Wenli Xu, and Junwu Zhou. Robust one-class SVM for fault detection. Chemometrics and Intelligent Laboratory Systems, 151:15 - 25, 2016. doi:
ISSN 0169-7439.
URL http:
7/ 7/www. wwt.e//cwieeee..ooo/ceeeeeeaaaaa
Haowen Xu, Wenxiao Chen, Nengwen Zhao, Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, et al. Unsupervised anomaly detection via variational auto-encoder for seasonal KPIs in web applications. In Proceedings of the 2018 World Wide Web Conference, pp. 187-196, 2018.
Huan Xu, Constantine Caramanis, and Sujay Sanghavi. Robust PCA via outlier pursuit. IEEE Trans. Information Theory, 3844:3307733044 2012. doi: 10.1 1099//091 22773156.
13

---

Under review as conference paper at ICLR 2021
Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. Proceedings of The 33rd International Conference on Machine Learning. volume 48, pp. 1100-1109 109. PMLR, 2016.
Chunkai Zhang, Shaocong Li, Hongye Zhang, and Yingyang Chen. VELC: A new variational autoencoder based model for time series anomaly detection. arXiv preprint arXiv:1907.01702., 2019.
Teng Zhang and Gilad Lerman. A novel M-estimator for robust PCA. Journal of Machine Learning Research, (1(1)))49-800)) 2014.
Teng Zhang, Arthur Szlam, and Gilad Lerman. Median K-flats for hybrid linear modeling with many outliers. Computer Vision Workshops (ICCV Workshops), 2009 IEEE 12th International Conference on, pp. 234-241. IEEE, 2009.
Chong Zhou and Randy C Paffenroth. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 665-674, 2017.
Xun Zhou, Sicong Cheng, Meng Zhu, Chengkun Guo, Sida Zhou, Peng Xu, Zhenghua Xue, and Weishi Zhang, A state of the art survey of data mining-based fraud detection and credit scoring. In MATEC Web of Conferences, volume 189, pp. 03002. EDP Sciences, 2018.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Dacki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International Conference on Learning Representations, 2018.
14

---

Under review as conference paper at ICLR 2021
A
ADDITIONAL EXPLANATIONS AND IMPLEMENTATION DETAILS OF MAW
In A.1 we review the ELBOO function and explain how ELBOW is obtained from ELBO. Additional implementation details of MAW are in $A.2 At last, AA. provides algorithmic boxes for training MAW and applying it for novelty detection.
A.1 REVIEW OF ELBO AND ITS RELATIONSHIP WITH ELBOW
standard VAE framework would minimize the expected KL-divergence from x(z|x)) to g()xx) in Q, where the expectation is taken over p(x). By Bayes' rule this is equivalent to maximizing the evidence lower bound (ELBO):
(q) = B p(xp z)) - pt K [L((()))))p((()
The first term of ELBO is the reconstruction likelihood. Its second term restricts the deviation of q(2(x)
from p( z) and can be viewed as regularization term. ELBOW is a more robust version of ELBO
with different regularization. That is, it replaces Ex(x) K (g( 22xx [[)) with Wa (g(zz, p(2)). We remark that the Wi distance cannot be computed between (z) x) and p(z) and ELBOW thus practically replaces (z)) x) with its expected distribution, g()) pxp)) / x) (or discrete approximation of this).
A.2 ADDITIONAL. IMPLEMENTATION DETAILS OF MAW
The matrix A and the network parameters for encoders, decoders and discriminators are initialized by the Glorot uniform initializer (Glorot & Bengio 2010).
The neural networks within MAW are implemented with TensorFlow (Abadi al.) 2013) and trained for 100 epochs with batch size 128. We apply batch normalization to each layer of any neural network. The neural networks were optimized by Adam (Kingma & Ba (015)) with learning rate 0.00005. For the VAE-structure of MAW, we use Adam with learning rate 0.00005. For the WGAN-type structure discriminator of MAW, we perform RMSprop (Bengio & Monperrus 2005) with learning rate 0. 0.0005, following the recommendation of Arjovsky
2017 () for WGAN.
A.3 ALGORITHMIC BOXES FOR MAW
Algorithms and 2 describe training MAW and applying MAW for novelty detection, respectively. In these descriptions. we denote by 0, and 8 the trainable parameters of the encoder E, decoder D and discriminator Dis, respectively. Recall that includes the trained parameters of the dimension reduction component.
Algorithm 1 Training MAW
Input: Training data {x() Ke initialized parameters 0, &o and 8 of E, D and Dis, respectively; initialized A; weight 7y: number of epochs; batch size 7; sampling number learning rate
Output: Trained parameters 0, &p and A
1: for each epoch do
2: for each batch (x(() Jiez v do
3: Mu.1 M0,21 9.11 802 + {x(())
(5) (8) (6) (4)
4:
1 AT Pog M( ATagg( s 0.1
(t)
A, j 1,2
Compute M( according to and
5:
6:
x) M M Σ
(+)T
M
(3)
M
(OT
7:
fort 1,,.. do hiez ~ nN
sample a batch { (1.))
Σ + (1 (1- W) )
8:
9:
sample a batch (z Jie N(O,1) end for
11:
(0, A. (p) + (0, a (0. A Lvar (f. A, (p) according to
+ 8 1 (8) according to
13:
8 6 c (1.1)
14:
A) (0, A) a (0.A) T.GEN( (0, A) according to 10
15: end for
16: end for
15

---

Under review conference paper at ICLR 2021
B MORE DETAILS ON TESTING THE EFFECT OF THE NOVEL FEATURES OF MAW
In $4.3 we experimentally validated the essential components of MAW by implementing variants of MAW that replace each novel component of MAW with a standard one. We notice that the AUC and AP scores in Figs. and consistently decrease when the outlier ratios increase, and thus the chosen training outlier ratios (c 0.1, 0.2 and 0.3) are sufficient to demonstrate the effectiveness of MAW over its variants. We provide additional details on each of these variants of MAW.
MAW-MSE replaces the least absolute deviation loss LVAE with the common mean squared error (MSE). That is, it replaces
xx())
Dizgen (6. in 77 with x D(zgen) (8.0)
2
MAW-KL divergence replaces the Wasserstein regularization Lw, with the KL-divergence. This is implemented by replacing the WGAN-type structure of the discriminator with a standard GAN.
MAW-same rank uses the same rank dl for both the covariance matrices 2 and x instead of
forcing Σ) to have lower rank d/2.
MAW-single Gaussian replaces the Gaussian mixture model for the latent distribution with single Gaussian distribution with full covariance matrix.
MAW-diagona cov. replaces the full covariance matrices resulting from the dimension reduction component by diagonal covariances. Its encoder directly produces 2-dimensional means and diagonal covariances (one of rank for the inlier mode and one rank for the outlier mode).
VAE has the same encoder and decoder structures MAW. Instead of dimension reduction compo- nent, it uses dense layer which maps the output the encoder to 4 dimensioonll vector composed of a 2-dimensional mean and 2-dimensional diagonal covariance. This is common for traditional VAE.
C SENSITIVITY OF HYPERPARAMETERS
We examine the sensitivity of some of the reported results to changes of some hyperparameters, In C. report the sensitivity choices of the intrinsic dimension. In CC22 to choices of the mixture parameter.
report the sensitivity
Algorithm 2 Applying MAW to novelty detection
Input: Test data ( أ=
sampling number T; trained MAW model; threshold er; similarity S(::) Output: Binary labels for novelty for each j = 1....,A N
for j L....,N do
2:
Po.1: 0,1 + E(y)))
(3)
3: 4 + H. ( +- A diag so. )
4: Compute MD) according to 5 and 6
5: Σ(" MM M
6:
fort = 1.... T do
sample z (1.0) N( x(1))
7:
8:
g(.1) +- in
(1.0)
9: compute 5(y (i), ((it))
end for
11: ((i) t T E (1) (0.4
12:
SU) 2 cT then
y6) normal example
14: else
15:
yb) is novelty
end if
17:
end for
18:
return Normality labels for 1, 1,...,
16

---

Under review conference paper at ICLR 2021
C.I SENSITIVITY TO DIFFERENT INTRINSIC DIMENSIONS
In all of the other experiments in this paper the default value of the intrinsic dimension is d = 2. Here we study the sensitivity of our numerical results to the following choices intrinsic dimensions: d = 2. 8. 16, 32 and 64, while using the KDDCUP-9 and COVID-19 datasets. We fix the intermediate training outlier ratio c = 0.3 for demonstration purpose. We compute the AUC and AP scores averaged over testing outlier ratios Ceest 0.1, 0.3, 0.5, 0.7 and 0.9 with three runs setting. Fig. Fig-5 reports the averaged results and their standard deviations, which are indicated by error bars.
AUC scores
AP
0.65
0.85
0.60
0.80
0.55
AUC/ scores
KDDCUP-99
0.75
COVID-19
0.50
AUC/ AUC/AP AP
0.70
0.45
0.65
0.40
0.60
0.35
0.55
0.30
8
16
32
64
2
4
16
32
64
Dimensions
Dimensions
Figure 5: AUC and AP scores with intrinsic dimensions d = 2, 4, 8, 16, 32 and 64 for KDDCUP-99 (on the left) and COVID-19 (on the right), where 0.3.
We can see from Fig. 5 that our default choice of intrinsic dimension 2 results in the best performance. For COVTD-19 we see a clear decrease of accuracy with the increase of the intrinsic dimension. For KDDCUP-99 dimensions is not so noticeable in COVID-19, These experiments confirm our default choice and indicate that the accuracy may decrease when the intrinsic dimension is not sufficiently small.
still
preference for 2, but the decrease with higher
C.2 SENSITIVITY TO MIXTURE PARAMETERS
In the rest of our experiments the default value of the mixture parameter is 5/6. Namely, we assume that the inlier mode has larger weight among the Gaussian mixture. In this section, study the sen- sitivity of the accuracy of MAW to the mixture parameters: f0.1, 0.2, 0.3, 0.4, 0.5, 0.6. 0.7. 5/6, 0.9). We use 5/6 2 0.83, instead of the nearby value 0.8, since it was already tested for MAW. The training outlier ratios are 0.1, 0.2 and 0.3. We report results on both KDDCUP and COVID-19 in Fig. 6
We notice that the AUC and AP scores mildly increase as the mixture parameter rj increases (though they may slightly decrease at 0.9). It seems that MAW seems to learn the inlier mode better with larger weight for the inlier mode and consequently gain more robustness. Nevertheless, the variation in the accuracy as function of 1j is not large general.
D ADDITIONAL THEORETICAL GUARANTEES FOR THE W, MINIMIZATION
In D.I we fully motivate our focus studying [1] in order to understand the advantage of the of the Wasserstein distance over the KI. divergence in the framework of MAW. In $D.2 we prove Proposition 3.1 Additional and more technical proposition that involves low-rank inliers is stated and proved D.2
17

---

Under review conference paper at ICLR 2021
AUC scores
AP scores
KDDCUP-99
COVID-19
1.0
Constant 0.1
'AP scores
AUC AP seores
0.9
AUC /
0.5
0.7
0.2
0.3
0.4
0.7
5/6
0.9
0.2
0.3
0.4
0.5
5/6
0.9
0.7
Constant c 0.2
0.9
AUC/A /CC/AP scores
AUC AP seores
0.6
0.8
0.7
0.6
0.1
0.2
0.3
0.4
0.5
0.6
0.7
5/6
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
5/6
0.9
0.9
Constant с 0.3
0.6
AUC/AI AAP scores
0.8
/ AP senres
0.7
0.5
0.5
0.1
0.2
0.3
0.4
0.5
0.6
0.7
5/6
0.9 0.1
0.2
0.3
0.4
0.5
0.6
0.7
5/6
0.9
Mixture parameters f
Mixture parameters
Figure 6: AUC and AP scores with mixture parameters 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 5/6 and 0.9 for KDDCUP-99 (on the left) and COVID-19 (on the right). From the top the bottom row, training outlier ratio are 0.1, 0.2 and 0.3, respectively.
D.1 MOTIVATION OF STUDYING
The implementation any VAE or its variants, such as AAE, WAE and MAW, requires the optimization of regularization penalty R, which measures the discrepancy between the latent distribution and the prior distribution. This penalty is typically the KL divergence, though one may use appropriate metrics such as W2 or W,. Therefore, one needs to minimize
R
T. Σ (z[xx(0), p(p(z)
(12)
over the observed variational family {q(z[x)), which indexed by some parameters of q. Here, is the batch size of the input data and E ()) is its observed aggregated distribution.
Since the explicit expressions of the regularization measurements between aggregated distributions unknown, is not feasible study the minimizer of (12). We thus consider the following approximation of (12):
L E (1((((1)))) (((2)=()) ).
(13)
can minimize one term of this sum time, that, minimize R (g(x)x), over Q. This minimization strategy is common in the study of the Wasserstein barycenter problem (Aguch & Carlier
2011 Peyré al. 2019; Chen al. 2018).
18

---

Under review as conference paper at ICLR 2021
One of the underlying assumptions of MAW is that the prior distribution p(z) is Gaussian and ((z(x) a Gaussian mixture. That is, p(z) ~ (NHo Σn) and g(z)x) ) ~ (Hi, Σ1) + (1 n)W (42 En). This gives rise to the following minimization problem
min
R (n E1) + (1 ()) Σg), N (No: Eo)).
(14)
A Mig 2 E
Similarly to approximating 12) by 113) we approximate (4) by the following minimization problem:
min
n/R (N E1).N(H (mo, En))
R (N(H2, E2), Σo)).
Pa En E
Recall that in MAW (M,, ) and (u Σg) are associated with the inlier and outlier distribution of MAW. We further assume that there is sufficiently small threshold > for which || - P2 ll 2 c. This is a reasonable assumption since, in practice, if and μ22 are very close, the reconstruction loss
will be large. These assumptions lead to the optimization problem (11 proposed in
D.2 PROOF OF PROPOSITION
Recall that o40 E RRK is the mean of the prior Gaussian, > O is the fixed separation parameter for the means of the two modes and 17 1/2 the fixed mixture parameter. For i 0. 1, 2, we denote
the Gaussian probability distribution by MM,; Σ.). Since in our setting Σn- = ΣS = En, we denote
the common covariance matrix in SK by Σ. That is, Σ E, for i 1.2.
We first analyze the solution of with R = Wp, where p 2 1, and then analyze the solution of (1I) with R KL.
The case R Wp, p. p 2 1: We follow the next three steps prove that the minimizer of satisfies 141
140-
Step I: We prove that Wp(Pi, vo) = (WN N),N E)) = || pollz for 2 and i 1,2.
(15)
First, we note that using the definition of Pp 2 1 and the common notation II(;;, 2b) for the distribution on IR x IR with marginals v4 and vo inf E( - Wp vo) = =E))(
inf mEIl(P).Mo)
E( 12
(16)
1| moll
where the inequality follows the fact that 11-112 is convex and from Jensen's 's inequality.
On the other hand, for = 1 or = 2, let x" be an arbitrary random vector with distribution P; and let y* x le; + M4g The distribution of is Gaussian with mean μ4 and covariance Σ,, that is, this distribution is vo. Let x= be the joint distribution of the random variables x" and y". We note that n is in 11(2,, 1o) and that
B(xy)~ar ||x
y122- = (xxy)~x 1| 140ll = I|m: -mu0l1
Therefore,
inf (0), (e)
E Ex ||x y11 S (xy) ||x y112 = 11 Moll
(17)
The combination of (T6) and immediately yields (15).
Step II: We prove that with R = Wp. 2 1, is equivalent
min
" ||ml| Holl2 + (1 (1-1) |02 - poll2
#1 Aa.
(18)
s.t. Ma :22:colnear
We first note that
with R min
Wp p is equivalent to
n p1 Holla (1 m)) || pollz-
(19)
s.t. [p -all
Indeed, this is a direct consequence of the expression derived in step I for R in this case. is thus left to show that uúj ul E R minimize 19,, then we can construct μ4, 1h E R that are colinear with dg and also minimize [9).
For any μr and u42 in R with || H22ls Ze and for the given Mo R define po. fai and ji TR K and demonstrate them in Fig. The point jáo is the projection of μm onto in A42 and
19

---

Under review as conference paper at ICLR 2021
i := p4, M40 jg for 2. We observe the following properties, which can be proved by direct calculation, though Fig. aasso clarifies them:
p4: poll2 2 | Moll2 for i 1,
and consequently, " 11 Holl2 + (1 1)) l - -poll2 2 2 11 Mallz + (1 - 17) 1 -
(20)
| i = -11 12 2
(21)
and
R1 [2 and 140 are colinear.
(22)
Clearly, the combination of (20). 21 and (22) concludes the proof of step II. That is, it implies that if p
11l E RRK minimize (19 then j and fl₂ defined above colinear with oál and also minimize 19
Ho.
pa
H40
Project Po
Po
140
111
Figure 7: Illustration of the points Po. f1 and jı and their properties.
Step III: We directly solve (8) and consequently with Wp, p 2 1. Due to the colinearity
constraint in [1.. we can write
P4 (1 + t) tt for R.
(23)
The objective function in 18) can then be written as
||m 42 (n/l)] + (1 - 1)] + +|) 2 (n/l + (1 r)) +
where equality is achieved if and only if [| /22l22 = and note that
We thus define r(1) = 1/|4 + (1 - 1)) +
t t+ (1 - 17),
t20 (1-1), n),
r(t)
(1 - 2m)t + (1 (1-
02t2 -1 -12t
+ (17 - 1),
and its derivative is
1, t> 0
(t)
2m,
0>t> -1> t.
The above expressions for and r' and the assumption that r7 > 1/2 imply that r(f) is increasing when
t >> O, decreasing when / < 0 and = 1- 1/ < r(1). Thus r has global minimum at / 0.
Hence, it follows from 23) that the minimizer of (11 and equivalently (II) with R Wp. p. p 1
satisfies μ1 = 140.
The case R = K L: We prove that the solution of 11 with R = K L satisfies 140 = 7//1 + (1- - 1) We practically follow similar steps the proof above.
Step I: We derive an expression for K T. (1o). where = 2. We use the following general formula,
which holds for the case where Eg, Σ, and Σg are general covariance matrices in S (see e.g., (2) in Hershey & Olsen 2007)):
K KL (p))
(log det E;
log det Eo
K + tr(EE
(m:: " (μ -
mm)
(24)
Since in setting Σ1 Eg E, this expression has the simpler form:
KL(vi(bu) = (H: - - T (H4: - 14)).
20

---

Under review as conference paper at ICLR 2021
Step II: We reformulate the optimization problem. The above step imples that 11 with R = KL can
be written as
min
niI - 14) Mo)" - -HO)
+ (1 (22 - -40'' " (142 Mo),
1 A9 or equivalently,
" - (M) m || + (1 -11) 17) - (#2 -mo0 _
2
min | 12
(25)
1 as E-1 UAUT, where E S and U an We express the eigenvalue decomposition of Σ orthogonal matrix. Applying the change of variables μ4 = A U for = 0, 1,2, we rewrite (25 as 4o
77 U22 Mo| /
min
(26)
At last, applying the same colinearity argument as above (supported by Fig. 7 we conclude the following equivalent formulation of (26):
7 Mi mell (1 (1-n)
m)) mollas moll
min
Mo pe, are colinear
(27)
& [1 -2 Ze
Step III: We directly solve (27). Due to the colinearity constraint,
write
Po
t)l - tut fort E IR
(28)
and express the objective function of (27
as
f1 1 (nt + (1 + +))) 2 (17 + (1-m)(++t)))
where equality is achieved if and only if || و = c. We thus define r(1) = 1/22 + (1 (1- (1+
and note that (t) 2(t + (1 17)) and (t) 2, and thus conclude that r(t) obtains its global
minimum at = 7) - 1. This observation and (28) imply that the minimizers μ4 and μ22 of (I] with
R = KL satisfy oo 7 + (1 12
D.3 THEORETICAL GUARANTEES FOR WITH Σ₁
We study the minimization problem [1 when Σj has low rank and Σg E S and also when R = W2 or R. = K.. Unfortunately, the case where R W1 is hard to analyze and compute. We first formulate our result for R W2- In this case we assume that the prior distribution a standard Gaussian distribution on g That is, it has mean 140 0g and covariance Eg IKXK. We further denote by 1 the vector (1, 1) E RK Similarly, we may define for any E N, On, 1,, Inxn- When it is clear from the context we only use O, 1 and I. For vectors R and Rm we denote the concatenated vector in 1 by (a: b).
Proposition D.1. IfK, KEN,
1,e 0,1>1>1' = K-n+e2
-K K- u² = [K (K-
12(2n-1)
where one can note that 1f" > 3 and n²' E (0,1), then the minimizer of (17) with R = W2 and with the constraint that Σ, is of rank and Eg of rank K, or equivalently, the minimizer of
min
7W2(N(11, E, (0,, +1-)+(111)))) (1-1))22((1)) E2,,N((,, I)) (29)
s.t. |1 & rank(E) (x))= (Ea)
satisfies 0g u² μ1 + (1 u² 11 E₁ diag(L (Ix: OK-R) and Σg diag 1xi (u*) (2x)-21 11xx))
Moreover, || 1l u'e and || (1 ue
We next formulate our simple result on the ill ell-posedsess of I1 with R W₂ and with the same constraint as in Proposition D.I
Proposition D.2. If K E K. > 8 L 0, > O, po E R Eo E sk and Σ E Sk with rank Σ) = K, then
KL(N(H1, EDILA =o0
Therefore, the solution of E. = I is poospd
with R KT. with the additional constraint that En of rank n. and
Next we clarify the implications of both propositions. Note that Proposition D. implies that as + 1, u² + 0. Hence for the inlier component μ4 - Og as 17 + 1 and ΣS = diag 0g-.)) so in the limit the inlier distribution has the same mean the prior distribution and, independently of
21

---

Under review as conference paper at ICLR 2021
77, its covariance is obtained by an appropriate projection of the covariance En onto k-dimensional subspace. We similarly note that as 17 + 1, Στ - diag( 1,: k , so that the outliers will disperse.
We further note that Proposition D.2 implies that the KI. divergence fails is unsuitable for low-rank covariance modeling as it leads to infinite value in the optimization problem.
At last, we note that the inlier and outlier covariances, Σn and Σg, obtained by Proposition D. are diagonal. Furthermore, the proof of Proposition D. clariiffss that the underlying minimization problem of this proposition may assume without loss of generality that the inlier and outlier covariances are diagonal (see e-g.. (32p, which is formulated below). On the other hand, the numerical results in $4.33 support the use of full covariances, instead of diagonal covariance. Nonetheless, we claim that the full covariances matrices of MAW comes naturally from the dimension reduction component of MAW. This component also contains trainable parameters for the covariances and they will effect the weights of the encoder, that is, will effect both the W1 minimization and the reconstruction loss. Thus the analysis of the W1 minimization component not sufficient for inferring the whole behavior of MAW. For tractability purposes, the minimization in (11) ignores the dimension reduction component. For com- pleteness remark that there are two other differences between the use of (IT) in Proposition D.11 and the way arises in MAW that may possibly also result in the advantage of using full covariance in MAW. First all, the minimization in Proposition D.I uses R = W2. whereas MAW uses R. = W1. which we find intractable when using the rest of the setting of Proposition D.I] Second of all, the optimization
problem with R W is an approximation of the minimization of W1 (7 E q(z (() )(())
(see For explanation), which also intractable (even if one uses = W2).
In S D.4 we prove Proposition D.I and in DD5 we prove Proposition
D.2
D.4 PROOF OF PROPOSITION
D.I
We follow the same steps of the proof of Proposition 3.1
Step I: We immediately verify the formula
(WN,, No, 1)) = | 2 1
|/ for = 1,2.
r
(30)
We use the following general formula, which holds for the case where Eo, E₁ and Eg are general covariance matrices in SK (see e.g., (4) in Panaretos & Zemel (0019)):
(W W.o, En)) = || moll2 + tr( + Eg 2( Eo
1,2. (31)
Indeed, 30 is obtained as a direct consequence of (31) using the identity
tr Σ + I - 2 = (
2
Step II: We reformulate the underlying minimization problem in two different stages. We first claim that the minimizer of [T] with R W22 and the constraint that Σ₁ is of rank x and Eg is of rank K can be expressed as the minimizer of
1 T/ (32)
min 141 M42 ER s.t. poau-uull2- En M2 diagomal in x
mV |
+
I + (1 - my 11212 +
& rank( ES )=K, rank E>)=K
In view of 1I] and (30) we only need to prove that the minimizer of (22) is the same if one removes the constraint that Σn and E₂ are both diagonal matrices and require instead that they are in e S This is easy to show. Indeed, if for = 1 or = 2, E, E SK then it can be diagonalized as follows:
Σ, = U.A.U,. where A, E SK is diagonal and U, is orthogonal. Hence, Σ = U. U.A.U, and 2 F = |UT - I)U, 2 = | - 1 Consequently, (2(N((,, E.).N D) (2 (O) I)) for 1,2,
2 = U.A.U. -
Σ - F
and the above claim is concluded.
Next, we vectorize the minimization problem in 332 as follows. We denote by R, the set of positive real numbers. Let b be general vector in R a' be general vector in R and = (a': (0=x) E KR
Given, the constraints Σ₁ and Σg, we can parametrize the diagonal elements of Σ and Σ by and
22

---

Under review as conference paper at ICLR 2021
b, that is, we set 1 diag( () and Σ diag(b). The objective function of (2) can then be written as
V |n 1 + ||a 1x 2 + (1 - / 122 12 ||b 1x 2
Combining this last expression and the same colinearity argument as in $D.2 (supported by Fig. 7)
(32) is equivalent to
min
7 al(()aa)) - (0x: 1x) (1(e; b) (0xi 1x)ll2
P1.MgER" BERR f. E. a=(aa=((00))) 0x ()x:))))
colinena klil -2all2
(33)
By the colinearity constraint, we can write (0xi 1x) = u( b) - (2u ()); where E IR. We thus obtain that
Step III: We solve (33
(H22 b) - (0x: 1x) = (u - 1) ((1) a) - (J22: b)) ((11 a) (122 b))
(34)
(μ1:a) a) (0x: 1x)
Furthermore, denoting the coordinates of a' and b by {a: [a=)z=1 and {bi we similarly obtain that
0g up2 - (u - 1)
1 ub; (u 1)a;, 1 1Sisk.
(35)
ubi: + 1 Siis K
The last two of equations imply that
1 - a' a'112 42
(0 1,)3
and
Σ
62 K
R u2
i=x+1
Combining 30
34
and the above two equations, rewrite the objective function of (33)
follows:
(7//a) + 11((1 (1)) ((uia) (H2z b)ll2
K
= (n]ual + lu 1/(1 - 1))) -)))
|| - 12l1 + Σ
(a: - 0,)2 +
1=1
1=
1k a 112
K
(n/uul + lu -1[(1 - 7)) V
2 +
+
(36)
u²
u2
= (K- xs) (a-
1
+ +n
+ + ( + lu- ([(1-))) m))
u
u 1
1/2
+ || - al11
(1
(1-w))
17
where equality is achieved if and only if [M 42 e. One can make the following two obser-
vations: u 0 does not yield minimizer of (33), and for any # 0, (36) obtains its minimum at a' 1R, In view of these observations and the derivation above, we define
f(u) = (K xx) (1-m)
+1)
f + lu- 11(1 - -n))²
(37)
and note that 33 is equivalent to
min u²0
Vf()).
(38)
We rewrite ((u) as
(K - x) p-1 - 1) + +1) + (nu + (1- (1-1)(u- 1))², u 1
f(u) =
(K- -- x) 1-= n) +1) 2 (nu + (1-1)(1 =))²
12 u > 0
(K' x)) ("-1
n) +n) 2 (nu + (1 (1)n)(u- 1))²,
0>u u
We denote
ri(w)= == (K x)
u ---1 (1 n) +
+ (nm + - (1- - 1))2
23

---

Under review as conference paper at ICLR 2021
and
(1-1(1 11) +11 + (1 n)( u))
r2(u)= == (K -x)
u
Their derivatives are
2
(u) 23
(u (1 - 1))) (e + (K -n)(1- - 11))
and
(2 n ((2 - 1) + (1 1j)) (2m 1)u2 - (K- - x) - n))
These expressions for r and r2 imply that the critical points for 11 are
(K x) 11)
$
x 1 7) and u
2²
and the critical points for 12
(1)
2n and u/2 (K x1] -1)
1 - 11
(2(2m --
$
We note that rı is increasing (uri the other hand, r2 is increasing on (urz (1) 0) U (u/ oo) and decreasing on (-o0, upz (1) U (0, 22 Since 17 > n7t
(2) 0) U (1) 00) and decreasing on (-o0, th:
(2) U (0, uf""). On
K- K-x+2c2 urg (0, 1). The derivative of with respect is
K K-m+e² (2)
ri(,), > 0
(n) = ru(,,, 1>6>0
ri(,,,
0>
So f()) is increasing on (uri 0) (urg (2) 20) and decreasing on (-o0, ur?)) ( U (0, ) of f at urz and urr are
). The values
(K n1((1 (2 1)2)
1
+ (1 ((1-n) 2
(IK (KK-m))
( 1) 3
f(u(2)
+ + +
6²
(1 (1-1)
11) ri((1 ((K-n))
$
+ (1 (1-mm) m) 2
(K x)) (
2
f(u!))=
+ +
22
(1 m
Consequently, the minimum of is obtained u² := Urz
(2) By
34 and 355.. the means fl, P2
and the covariance matrices E, En satisfy: 0x = u" μ12 + (1 u 11 Σ
diag( 1,; 0x-x) and
Σg diag( (13 ki (u*) " x-x=) Moreover, the norms of μrl and μ2 can be computed from 35 as 2'6
and (1 u² )e, respectively.
D.5 PROOF OF PROPOSITION D.2
Notice that since Eo E SK ++ det(Eo) rank(2.) = K < K, det(eet)) En) = 0. Therefore,
0. On the other hand, since Ei E SK with
log det Eg)
= log det( det((e)) - det(E)) = 00
det(En) Σn)
and this observation and (4 imply that KL(N(H1, ET)l En)) = Co.
E ADDITIONAL. DETAILS ON THE BENCHMARK METHODS
We overview the benchmark methods compared with MAW, where we present them according to alphabetical order of names. We will include all tested codes in supplemental webpage.
For completeness, we mention the following links (or papers with links) we used for the different codes. For DSEBMs and DAGMM we used the codes Golan & El- Yaniv (2018). For LOF, OCSVM and IF we used the seikit-learn (Buitinck al. 2013) packages for novelty detection. For OCGAN used its TensorFlow implementation from https:::...........p...ttt ppppi orgg httg://jjec//oooooeo we adapted the code of Lai et al. (2020) to novelty detection.
For RSRAE,
All experiments were executed on Linux machine with 64GB RAM and four GTX10800 GPUs.
24

---

Under review as conference paper at ICLR 2021
We remark that for the neural networks based methods (DAGMM, DSEBMs, OCGAN and RSRAE), we followed similar implementation details as the one described in $ for MAW.
Deep Autoencoding Gaussian Mixture Model (DAGMM) Zong et al. (0188) is a deep autoencoder model. It optimizes end-to-end structure that contains both an autoencoder and estimator for a Gaussian mixture model. Anomalies are detected using this Gaussian mixture model. We remark that this mixture model is proposed for the inliers.
Deep Structured Energy-Based Models (DSEBMs) Zhai et al. (216)) makes decision based an en- ergy function which the negative log probability that a sample to follows the data distribution. The energy based model is connected to an autoencoder in order to avoid the need of complex sampling methods.
Isolation Forest (IF) Liu et al. (008 iteratively constructs special binary trees for the training set and identifies anomalies in the test set as the ones with short average path lengths in the trees.
Local Outlier Factor (LOF) Breunig et al. (2000) measures how isolated a data point is from its surrounding neighborhood. This measure based on estimation of the local density of data point using its & nearest neighbors. In the novelty detection setting, it identifies novelties according to low density regions learned from the training data.
One-class Novelty Detection Using GANs (OCGAN) Perera et al. (019)) is composed of four neural networks: a denoising autoencoder, two adversarial discriminators, and a classifier. aims to adversarially push the autoencoder to learn only the inlier features.
One- One-Class SVM (OCSVM) Heller al. (2003) estimates the margin of the training set, which is used as the decision boundary for the test set. Usually it utilizes a radial basis function kernel to obtain flexibility.
Robust Subspace Recovery Autoencoder (RSRAF) Lai et al. 2020) an autoencoder structure together with a linear RSR layer imposed with a penalty based on the 2 energy. The RSR layer extracts features of inliers in the latent code while helping to reject outliers. The instances with higher reconstruction errors are viewed as outliers. RSRAE trains model using the training data, We then apply this model for detecting novelties in the test data.
F ADDITIONAL DETAILS ON THE DIFFERENT DATASETS
Below we provide additional details on the five datasets used in our experiments. We remark that each dataset contains several clusters (2 for KDDCUP-99, 5 for Reuters-21578, 3 for COVID-19, 11 largest ones for Caltech 101 and 10 for Fashion MNIST), We summarize the number of inliers and outliers per dataset (for both training and testing) in Table[]
KDDCUP-99 a classic dataset for intrusion detection. It contains feature vectors of connections between internet protocols and a binary label for each feature vector identifying normal vs. abnormal ones. The abnormal ones are associated with an "attack" or "intrusion".
Reuters-21578 contains 21, 21.578 documents with 90 text categories having multi-labels Following Lai et al. (2020). we consider the five largest classes with single labels. We utilize the scikit-learn packages: TFIDF and Hashing Vectorizer (Rajaraman & Ullman 2011) preprocess the documents into 16, dimensional vectors.
COVID-19 (Radiography) contains chest X-ray RGB images, which are labeled according to the following three categories: COVID-19 positive, normal and bacterial Pneumonia cases. We resize the images to size 64 x 64 and rescale the pixel intensities to lie in [-1,
Caltech101 contains RGB images of objects from 101 categories with identifying labels. Following Lai et al. (200)) we use the largest 11 classes and preprocess their images to have size 32 x 32 and rescale the pixel intensities to lie in [-1, 1].
Fashion MNIST is an image dataset containing 10 categories of grayscale images of clothing and accessories items. Each image is of size 28 x 28 and we rescale the pixel intensities lie in [1 1,
We remark that COVID-19, Caltech101 and ceuters-21577 separate between training and testing datapoints, For KDDCUP-99, we randomly split it into training and testing datasets of equal sizes.
25

---

Under review as conference paper at ICLR 2021
Table 1: Numbers of inliers and outliers for training and testing used in the five datasets.
Training
Testing
Datasets
# #Inliers (N)
#Outliers
#Inliers (Nues)
# Outliers
(N x c)
lest Crest)
KDDCUP-99
6000
6000 Xc
1200
1200 X Ctest
Reuters-21 215788
350
350 xc
140
140 Clest
COVID-19 (Radiography)
160
160 xc
60
60 Xtest
Caltech101
100
100 xc
100
100 Ciest
Fashion MNIST
300
300 x
60 X Clest
G EXPERIMENTS WITH DIFFERENT OUTLIER TYPES
In this section, we test the performance of MAW and the benchmark methods when the training and test sets are corrupted by outliers with different structures. We generate a dataset, which we call "Mix Caltech1 101" the following way. We fix the largest class of Caltech101 (containing airplane images) the inlier class and randomly split it into the training inlier class (68.75 %) and testing inlier class (31.25 %). We form the training set by corrupting the training inlier class with random samples from the ten classes of CIFAR1O (Krizhessky et al. (009) with training outlier ratio {0.1, 10.1,0.2 0.2, 0.3, 0.4. 0.5}. For the test set, we corrupt the testing inlier class by "tile images" from MVTTech dataset (Bergmann al. 2019) with testing outlier ratio Gea {0.1. 0.3, 0.5, 0.7, 0.9}. The rest of the settings of the experiments are identical to the description in 4.2
We present the AUC
and AP scores and their standard deviations in Fig.
MAW
DAGMM
DSEBMs OCSVM
IF RSRAE
LOF
OCGAN
1.0
0.8
0.8
0.7
Mix Caltech101
AUC score
0.7
0.6
0.6
0.5
AP
0.5
0.4
0.3
0.3
0.2
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
The constant
The constant
Figure 8: AUC and AP scores with training outlier ratio (0.1,0.2,0.3 0.5) for the Mix Caltech101 dataset.
The competitive advantage of MAW in comparison to the rest of the methods is also noticeable in this setting. We note that OCSVM, the traditional distance- based method, and IF, the traditional density- based method, perform poorly in this scenario, whereas they performed well in our original setting.
H NUMERICAL RESULTS OF EXPERIMENTS
We present tables the numerical values depicted in Figs. and in H.1 and those in Fig. 4
H.2
26

---

Under review conference paper at ICLR 2021
H.1 TABLE REPRESENTATION FOR FIGS. AND
Tables report the averaged AUC and AP scores with training outlier ratios E 0..1,0.2, 0.3, 0.4, 0.5) that were depicted in Figs. and 3 Each table describes one of the averaged scores (AUC or AP) for one of the five datasets (KDDCUP-99, Reuters-21576 COVID-19, Caltech101 and Fashion MNIST) and also indicates the standard deviation of each value. The outperforming methods are marked in bold.
Table 2: AUC scores of KDDCUP-99
Training outlier ratios
Methods
0.1
0.2
0.3
0.4
0.5
MAW
0.935 +: 0.028
0.888 + 0.026
0.832 + 0.016
0.764 +: 0.023
0.724 + 0.012
DAGMM
0.614 0.083
0.660 + 0.109
0.584 = 0.133
0.457 = 0.099
0.521 = 0.089
DSEBMs
0.514 0.000
0.499 0.000
0.497 0.000
0.496 +: 0.000
0.496 + 0.000
IF
0,811
0.527
0.516
0.750
0.706
LOF
0.480
0.527
0.516
0.527
0.530
OCGAN
0.651 = 0.157
0.552 :
0.617 0.19
0.517 0.146
0.628 + 0.155
OCSVM
0.502
0.568
0.567
0.555
0.534
RSRAE
0.815 0.031
0.839 0.059
0.778 0.086
0.735 + 0.066
0.740 + 0.056
Table 3: AP scores of KDDCUP-99.
Training outlier ratios
Methods
0.1
0.2
0.3
0.4
0.5
MAW
0.740=0.022
0.698 + 0.033
0.647 + 0.012
0
0
DAGMM
0.446 0.047
0.506
0.064
0.459
0.087
0.373 0.109
0.464 + 0.998
DSEBMs
0.450 + 0.000
0.447 + 0.000
0.446 + 0.000
0,444 + 0.000
0,444 + 0.000
IF
0.636
0.6331
0.562
0.493
0.457
LOF
0.391
0.407
0,392
0.394
0.391
OCGAN
0.582 4
0.472 + 0.163
0.525 + 0.133
0.418 0.136
0.535 + 0.133
OCSVM
0.543
0.598
0.595
0.438
0.426
RSRAE
0.048
0.698 0.050
0.606 = 0.065
0.584 0.034
+ +
Table 4: AUC scores of Reuters-21577
Training outlier ratios c
Methods
0.1
0.2
0.3
0.4
0.5
MAW
0.885
0.028
0.830
0.013
0.770 + 0.017
0.700
0.002
0.648 0.016
DAGMM
0.500 + 0.000
0.511 + 0.027
0.566
0.110
0.559 + 0,087
0.570
DSEBMs
0.887 0.012
0.825 0.012
0.790 0.015
1690=
0.648 - 0.010
IF
0.544
0.535
0.520
0.453
0.452
LOF
0.757
0.612
0.579
0.631
0.616
OCGAN
0.648 + 0.127
0.477
0.498 0. 140
0.519 + 0.132
0.502 + 0.099
OCSVM
0.882
0.817
0.785
0.673
0.640
RSRAE
0.786 + 0.042
0.755 + 0.034
0.716 0.033
0.605 + 0.001
+ +
27

---

Under review conference paper at ICLR 2021
Table 5: AP scores of Reuters-21578.
Training outlier ratios
Methods
0.1
0.2
0.3
0.4
0.5
MAW
0.755 0.041
0.677 0.026
0.627 = 0.029
0.518 = 0.004
0.014
DAGMM
0.316 0.000
0.316 +.011
0.365
0.020
0.362
0.015
0.372 0.012
DSEBMs
0.763 0.012
0.697 0.011
0.666 + 0.007
0.515 + 0.003
0.473 + 0.003
IF
0.368
0.372
0.365
0.301
0.298
LOF
0.580
0.438
0.421
0.498
0.486
OCGAN
0,408 + 0.045
0.334 + 0.098
0.365 = 0.1
0.504 + 0.083
0,497 + 0.094
OCSVM
0.746
0.681
0.637
0.467
0.438
RSRAE
0.563 0.035
0.488 + 0.036
0.403 0.001
0.405
Table 6: AUC scores of COVID-19.
Training outlier ratios
Methods
0.1
0.2
0.3
0.4
0.5
MAW
0.682 + 0.021
0.639 + 0.018
0.606 + 0.020
0.551 + 0.030
0.534 0.010
DAGMM
0.547 0.068
0.068
0.565 0.051
0.538 + 0.062
0.524
0.060
0.523 + 0.057
DSEBMs
0.471 0.000 0.000
0.471 0.000
0.471 0.000
0.471 0.000
0
IF
0.604
0.571
0.555
0.523
0.499
LOF
0.672
0.618
0.572
0.580
0.589
OCGAN
0.492 0.000
0.492:
0.492 0.000
0,485 + 0,000
0,491 + 0.000
OCSVM
0.528
0.528
0.528
0.535
0.521
RSRAE
0.565 0.031
0.027
0.476 : 0.023
0.454 + 0.018
0.427 + 0.011
Table 7: AP scores of COVID-19.
Training outlier ratios
Methods
0.1
0.2
0.3
0.4
0.5
MAW
0,455 0.014
0.442 + 0.011
0.424 + 0.018
0.368 + 0.015
0.376 + 0.008
DAGMM
0.354+0.055
0.390 + 0.057
0.316 + 0.052
0.357 + 0.050
0.368 + 0.047
DSEBMs
0.372 0.000
0.372 0.000
0.372 : 0.000
0.372 = 0.000
0.372 + 0.000
IF
0.425
0.404
0.392
0.380
0.373
LOF
0.463
0.422
0.402
0.397
0.393
OCGAN
0.381 0.000
0.381 0.000
0.381 + 0.000
0.383 + 0.000
0.376 + 0.376 0.000
OCSVM
0.315
0.315
0.315
0.372
0.365
RSRAE
0.388+0.015 0.018
0.377 0.016
0.355 + 0.011
0.352 0.010
0
Table 8: AUC scores of Caltech 101.
Training outlier ratios c
Methods
0.1
0.2
0.3
0.4
0.5
MAW
0.801 0.017
0.760
0.028
0.700 = 0.038
0.608 - 0.031
0.570 0.021
DAGMM
0.684 0.100
0.588 +: 0.115
0.500
0.100
0.509 + 0.101
0.514
DSEBMs
0.536
0.612+
0.577 + 0.030
0.564 0.021
0.536 0.021
IF
0.755
0.694
0.626
0.575
0.540
LOF
0.674
0.593
0.495
0.436
0.411
OCGAN
0.494 + 0.000
0.494 0.000
0.494 0.000
0.500 - 0.000
0.500 + 0.000
OCSVM
0.682
0.618
0.577
0.538
0.516
RSRAE
0.774 0.027
4.22
0.664 + 0.082
0.579 + 0.047
0.568 0.036
28

---

Under review conference paper at ICLR 2021
Table 9: AP scores of Caltech101.
Training outlier ratios
Methods
0.1
0.2
0.3
0.4
0.5
MAW
0.634 0.027
0.572 + 0.039
0.531 + 0.064
0
0.024
DAGMM
0.574: 0.088
0.422
0.112
0.308
0.102
0.351
0.074
0.363 + 0.076
DSEBMs
0.385 0.003
0,472 0.051
0.398=1.019
0.383 + 0.023
0.365 + 0.028
IF
0.545
0.486
0.430
0.304
0.371
LOF
0,460
0,400
0.337
0,304
0.290
OCGAN
0.362 0.000
0.362 0.000
0.362 + 0.000
0.362 + 0.000
0.362 0.000
OCSVM
0.472
0.419
0.380
0.352
0.339
RSRAE
0.595 0.038
0.551 0.045
0,495 #0.073
0.425 0.040
0.027
Table 10: AUC scores of Fashion MNIST
Training outlier ratios c
Methods
0.1
0.2
0.3
0.4
0.5
MAW
0.897 0.013
0.879
0.011
0.852
0.022
0.830
0.017
0.801 + 0.016
DAGMM
0.607 + 0.093
0.376 + 0.070
0.427 + 0.090
0,401 + 0.078
0,411 0.081
DSEBMs
0.730 + 0.092
0.729 0.
0.739 + 0.086
0.723 + 0.106
0.687 + 0.096
IF
0.893
0.875
0.843
0.834
0.827
LOF
0.569
0.507
0.476
0.468
0.458
OCGAN
0.542 = 0.006
0.538 + 0.004
0.544 0.014
0.531 + 0.003
0.525 + 0.004
OCSVM
0.895
0.874
0.848
0.831
0.814
RSRAE
0.860 + 0.022
0.848 + 0.022
0.829 + 0.042
0.831 + 0.028
0.808 + 0.028
Table 11: AP scores of Fashion MNIST
Training outlier ratios
Methods
0.1
0.2
0.3
0.4
0.5
MAW
0.788 +0.013
0.754 +: 0.014
0.723-1.029
0,686 + 0.025
0,672 #0.021
DAGMM
0.482 +00051
0.303
+0.57
0.334 113
0.318 +0.056
0.330 + 0.038
DSEBMs
0.600 + 0.045
0.609 120
0.613+10.089
0.605 #0,086
0.565 + 0.072
IF
0.768
0.724
0.693
0.665
0.642
LOF
0.382
0.331
0.308
0.301
0.294
OCGAN
0.504 + 0.002
0.503 0.003
0.500 - 0.059
0.495 + 0.001
0.493 + 0.001
OCSVM
0.801
0.768
0.735
0.696
0.664
RSRAE
0.749 + 0.029
0.736 0.032
0.716 + 0.048
0.683 + 0.036
0.680 + 0.042
H.2 TABLE REPRESENTATION FOR FIG.
Tables 12 15 record the averaged AUC and AP scores with training outlier ratios c 0.1, 0.2, 0.3 that were depicted in Fig. 4 Each table describes one of the averaged scores (AUC or AP) for one of the two representative datasets (KDDCUP-99 and COVID-19 119) and also indicates the standard deviation of each value. The outperforming methods are marked in bold.
29

---

Under review conference paper at ICLR 2021
Table 12: AUC scores of KDD-99 for variations of MAW Training outlier ratios с
Methods
0.1
0.2
0.3
MAW
0.936 + 0.028
0.888 + 0.030
0.832 +00016
MAW-MSE
0.844 + 0.039
0.812 +0.032
0.746 + 0.044
MAW-KL divergence
0,905 0.026
0.863 0.028
0.801
MAW-same rank
0.912 + 0.023
0.868 + 0.011
0.797 + 0.022
MAW-single Gaussian
0.862 0.016
0.796
MAW-diagonal cov.
0.918 + 0.023
0.858 0.020
0.801 + 0.044
VAE
0.821 + 0.048
0.785 0.027
0.732 0.046
Table 13: AP scores of KDDCUP-99 for variations of MAW
Training outlier ratios с
Methods
0.1
0.2
0.3
MAW
0.740 + 0.025
0.698 = 0.033
0.647 = 0.012
MAW-MSE
0.715 + 0.079
0.589
MAW- KL divergence
0.735 0.028
0.618 0.028
MAW-same rank
0.725 + 0.028
0.681 +0.015
0.622 0.024
MAW-single Gaussian
0.737
0.675
0
MAW-diagonal cov.
0.724+:0.02
0.678 + 0.035
0.589 + 0.064
VAE
0.642 + 0.030
0.555 +
+ 0.028
Table 14; AUC scores of COVID-19 for variations of MAW
Training outlier ratios c
Methods
0.1
0.2
0.3
MAW
0.682 + 0.021
0.639 + 0.018
0.606 + 0.029
MAW-MSE
0.642 + 0.022
0.584
+0.063
0.548 + 0.041
MAW-KL divergence
0.654 0.025
0.610 0.026
0.528 0.064
MAW-same rank
0.034
0.604 - 0.048
= 0.557
MAW-single Gaussian
0.651 0.022
0.61611.029
0.537 0.047
MAW-diagone cov.
0.630
0.036
0.545
- 0.035
VAE
(629:00007
0.585
0.532 + 0.049
Table 15: AP scores of COVID-19 for variations of MAW
Training outlier ratios e
Methods
0.1
0.2
0.3
MAW
0.459 + 0.014
0.442
+ 0.011
0.424 0.018
MAW-MSE
0).41 0.015
0.395
+ 0.025
-377+00.0
MAW-KE divergence
0.427 0.016
0.40
0.020
MAW-same rank
0).42 0.001
0.413+0.022
0.375
MAW-single Gaussian
0.425
0.409
0.074
MAW-diagonal cov.
0.412 0.016
0.397 0.018
0.369 0.012
VAE
0.412 0.030
0.043
0,379 + 0.028
30


Reviews:
[{"review_id": "rPJaUq2FdB1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a robust novelty detection method (\"MAW\") to model the distribution of the training data in the presence of high fraction (corruption ratios up to 30\\%) of outliers. The method add new features to the variational autoencoder (VAE), to detect and isolate the outlier so that the learned distribution only represent the inlier distribution:\n1. Uses a carefully designed dimension reduction component to extracts latent lower-dimensional features of the latent distribution.\n2. Model the distribution of latent representation as a mixture of Gaussian low-rank inliers and full-rank outliers, both using full covariances instead of diagonal covariances as commonly used in previous VAE-based methods for novelty detection.\n3. Penalizes the Wasserstein-1 distance between the data distribution and the latent distribution from the prior distribution. Under a special setting, it theoretically proves that using the Wasserstein-1 metric for regularization yields outliers-robust estimation and is suitable to the low-rank modeling of inliers, while the commonly used Kullback-Leibler (KL) divergence does not.\n4. Using the least absolute deviation error for reconstruction.\nExperiments on popular anomaly detection datasets demonstrate state-of-the-art results of MAW on standard benchmarks for novelty detection.\n\n================== After rebuttal ==================\nSeveral typos:\nIn Figure 1, to be consistent the X and Z may be written in lower case.\nIn Eq. (6), the transpose should be applied to the right $U_1^{(i)}$ instead of the left one.\nIn Appendix C.1, ``It seems that MAW seems to learn\" should be ``It seems that MAW learns\" or ``MAW seems to learn\".\n\nIn Appendix D.3:\nIn line 2 above Proposition D.2, ``the ill-posedness of (11) with $\\mathcal{R}= W_2$\": The $\\mathcal{R}= W_2$ should be $\\mathcal{R}= KL$.\nPlease rephrase ``the KL divergence fails is unsuitable for low-rank covariance modeling\", by e.g., removing ``fails\" or inserting ``and\" between ``fails is\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "n1qRsuWsQ-e", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This study proposes a novel method that can work well even the training data is corrupted by partial data from the unknown domain. Though it deals with the well-known problem called 'Noisy data/label', its approach is not the same thing as the previous works as it focuses on variational autoencoder on the task of novelty detection. And its arguments and statistical assumptions are followed by mathematical proofs. \n\nOverall, it is an interesting approach and I believe it would give a good way to ML practitioners who are struggling with noisy datasets in real-world applications. However, there some questions/comments about the article which may make the study more consolidate:\n\nQuestions\n- In the description of the proposed method, MAW, Discriminator generates Loss(Lw1) by comparing between Zgen and Zhyp. And Zhyp is unimodal distribution while Zgen follows MoG. I wonder whether there is a risk that inlier and outlier distributions are mixed(combined) as the loss makes the generator generates just the same mu/sigma regardless of the domains. If so, is there any equilibrium trick required so that the generator would not be strong too much?\n\n- Though it is hard to pre-estimate how the outlier distribution looks like, it is more common to assume the outlier distribution has multi-modal than uni-modal. However, the proposed method approximates the outlier distribution as unimodal Gaussian distribution. Is it possible to model the outliers as multi-modal distribution such as MoG?\n\n- In the experiment with the multiclass dataset, the number of possible inlier domains is the same as the number of classes in the dataset. And the characteristic of 'training data' may be different by each combination. I wonder the experiment of this study covered all possible sets. \n And the corrupted data is sampled randomly from the other classes. Is there any deviation in the performance by each sampling?\n\n- This study aims to generate the model to be robust to corrupted training data. However, in the result, it is not clear that the proposed method is more robust than others as the AUC/AP from MAW falls (maybe greater than others) as the outlier ratio increases. The authors may give explanations about the result in detail. \n\n\nAdditional Comments\n- The readability of Figure 2, 3 is not good. How about showing them on the tables?\n- This study shows the superiority from four datasets (image, non-image). However, there is more dataset widely used for novelty detection such as (Fashion) MNIST or MVTech. The authors may consider doing the same experiments on the other dataset.\n-The authors may compare the method not only to the novelty detection methods, but many previous works which also aims to be robust to noisy data(or label) in the training process. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iQsvxb6Ugbp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper seeks to address the problem of novelty detection under the circumstance of having high corruptions in the training data. This is different from most previous work, which often assumes that training dataset is pure. To address this issue, a VAE-based approach is adopted in this paper, with several modifications made to the vanilla VAE to promote the robustness of VAE in detecting outliers in the corruption circumstance. Among the modifications, the paper assumes the posterior is approximated by a two-component Gaussian mixture distribution, with each having a low-rank and full-rank covariance matrix, respectively. The paper hopes that the posterior of inliers (normal data points) can be represented by the low-rank covariance matrix, while that of outliers cannot. Another notable modification is that the Wasserstein-1 regularization is used to replace the KL-regularization in the ELBO, which is claimed to be more suitable to the low-rank modeling. Some experiments are conducted to evaluate the outlier detection performance of the proposed method under corrupted circumstance.\n\nStrength:\n1. The problem of detecting outliers under highly corrupted environment is of practical importance and is not investigated under the extremely corrupted circumstance. \n\n2. The idea of proposing to use low-rank and full-rank geometry characteristics to separate inliers and outliers is interesting. \n\n\nWeakness:\n1. Although the idea of using low/full-rank geometric characteristic to detect outliers is interesting, the paper barely states how to implement this idea, that is, how to enforce the inliers resides in the low-rank covariance matrix, while the outlier will not. Without any specific design, we cannot believe this will be realized automatically.\n\n2. The theoretical result (Section 3) established to argue the superiority of Wasserstein regularization in robustness is better than the original KL regularization in vanilla VAE is not convincing, or is not practically meaningful. That is because the result only reveals that under a very special case, the Wasserstein may learn the true distribution under corrupted environment. We cannot see this result has any implication under a broader or more general circumstance.\n\n3. I guess the true reason to choose W_1 over KL distance is that because a Gaussian mixture posterior is employed, the KL cannot be evaluated in close-form. By resorting to W_1 distance, the distance can be estimated with the samples, but the KL distance cannot be estimated in this way. Overall, I cannot buy the argument that the authors proposed to use the W_1 distance/regularization here.\n\n4. I also have some concerns over the choosing of hyper-parameters. The paper sets the dimension of latent representation (d) to be 2, which, I think, is too small. But if it is set to be an appropriate value (e.g. 100), the posterior with a full-rank covariance matrix will be computationally expensive. Moreover, I also think that it is not a good idea to set the mixture coefficient $\\eta$ to be a fixed value. Generally, this should be learned from data, because you cannot know the ratio of different components in advance.\n\n5. I also have some doubts over the experiment settings. At the training stage, the training data is corrupted by a fraction of outliers. But if the data is corrupted by ‘outliers’, that means the model has already made use of the information of outliers. So, if the ratio of outlier in the training dataset is higher, it may be more favorable to the proposed method. For example, if the ratio is large, e.g. c=0.3, this problem under this setting is more like a clustering problem. So, this kind of settings may be not fair to the comparing methods. Maybe, the better setting should be let the training data corrupted by some data that is not in both the training and testing datasets. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y0CC9uT3vL2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Compared with conventional VAE, this paper incorporates the following strategies to improve VAE for novelty detection from corrupted training data. 1) By considering that outliers tend to have more complex structures, this paper assumes inliers and ourliers lie on Gaussian distributions with different ranks, and proposes a Gaussian mixture model for posterior q(z|x) including two component: a low-rank multivariate Gaussian distribution component for modeling inliers and a full-rank multivariate\nGaussian distribution component for outliers.  2) Applying the Wasserstein distance (W distance) between q(z|x) and p(z) instead of the KL divergence in the original VAE. The paper also proves the superiority of using W distance over KL divergence, by showing that the minimized value exists in W distance between a Gaussian mixture distribution q(z|x) and Gaussian prior p(z). 3) 3)For Rrconstruction loss, this model adopts ||x-D(z)||_2 as described in (3) instead of ||x-D(z)||^2_2, which is used in conventional VAE. This practice helps to alleviate the problem that the loss item in conventional VAE being too small when data point deviates from the center of the Gaussian distribution.\nThe experiments show that proposed method achieves good result among 4 datasets, which domonstrate the effectiveness of the three strategies.\nAdvantages of this paper: although Gaussian mixture model has been used in previous work of VAE, this model innovatively propose Gaussian components with different ranks for modeling inlier and outliers to improve the performance for training data with corrupted samples. This work also gives a theoretical guarantee for the advantage of W distance minimization in given setting.\nPotential drawbacks: Problem setting is confusing. Outliers/novelties are supposed to be rare in the dataset, while the paper assumes that a nontrivial fraction of outliers existing in the data. In addition, more recent work is not compared, such as Ruff, L., Vandermeulen, R. A., Görnitz, N., Binder, A., Müller, E., Müller, K. R., Kloft, M. (2019). Deep semi-supervised anomaly detection. arXiv preprint arXiv:1906.02694", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novelty detection method when training data is itself noisy. A VAE-based approach is developed that promotes robustness of the VAE. The paper assumes that the encoder a two-component Gaussian mixture distribution, individual components denoting inliers and outliers.

The paper hopes that the posterior of the inliers (normal data points) can be represented by a low-rank covariance matrix, while the outliers need a full covariance. Another notable modification is that the Wasserstein-1 regularization is used to replace the KL-regularization in the ELBO, which is claimed to be more suitable to the low-rank modeling. 

While this is a relevant problem, and the idea is perhaps interesting, some concerns have been raised.
* The details how to fit the model with the desired mixture posterior in practice is unclear.
* The arguments of section 3 to illustrate the superiority of Wasserstein were found unconvincing, with limiting/unclear assumptions
* The ultra-low latent space dimension (2) is not sufficiently justified 
* The experimental section and the selected datasets are small scale, it would be good to include a free larger scale datasets (at least cifar10).
* Comparisons to the open set recognition, or out-of-distribution (OOD) detection would have been a plus.

Overall, this is an OK paper but not yet of sufficient quality.
Decision: Reject