Novelty Detection via Robust Variational Autoencoding

1 INTRODUCTION . Novelty detection refers to the task of detecting testing data points that deviate from the underlying structure of a given training dataset ( Chandola et al. , 2009 ; Pimentel et al. , 2014 ; Chalapathy & Chawla , 2019 ) . It finds crucial applications , in areas such as insurance and credit fraud ( Zhou et al. , 2018 ) , mobile robots ( Neto & Nehmzow , 2007 ) and medical diagnosis ( Wei et al. , 2018 ) . Ideally , novelty detection requires learning the underlying distribution of the training data , where sometimes it is sufficient to learn a significant feature , geometric structure or another property of the training data . One can then apply the learned distribution ( or property ) to detect deviating points in the test data . This is different from outlier detection ( Chandola et al. , 2009 ) , in which one does not have training data and has to determine the deviating points in a sufficiently large dataset assuming that the majority of points share the same structure or properties . We note that novelty detection is equivalent to the well-known one-class classification problem ( Moya & Hush , 1996 ) . In this problem , one needs to identify members of a class in a test dataset , and consequently distinguish them from “ novel ” data points , given training points from this class . The points of the main class are commonly referred to as inliers and the novel ones as outliers . Novelty detection is also commonly referred to as semi-supervised anomaly detection . In this terminology , the notion of being “ semi-supervised ” is different than usual . It emphasizes that only the inliers are trained , where there is no restriction on the fraction of training points . On the other hand , the unsupervised case has no training ( we referred to this setting above as “ outlier detection ” ) and in the supervised case there are training datasets for both the inliers and outliers . We remark that some authors refer to semi-supervised anomaly detection as the setting where a small amount of labeled data is provided for both the inliers and outliers ( Ruff et al. , 2020 ) . There are a myriad of solutions to novelty detection . Nevertheless , such solutions often assume that the training set is purely sampled from a single class or that it has a very low fraction of corrupted samples . This assumption is only valid when the area of investigation has been carefully studied and there are sufficiently precise tools to collect data . However , there are different important scenarios , where this assumption does not hold . One scenario includes new areas of studies , where it is unclear how to distinguish between normal and abnormal points . For example , in the beginning of the COVID-19 pandemic it was hard to diagnose COVID-19 patients and distinguish them from other patients with pneumonia . Another scenario occurs when it is very hard to make precise measurements , for example , when working with the highly corrupted images obtained in cryogenic electron microscopy ( cryo-EM ) . Therefore , we study a robust version of novelty detection that allows a nontrivial fraction of corrupted samples , namely outliers , within the training set . We solve this problem by using a special variational autoencoder ( VAE ) ( Kingma & Welling , 2014 ) . Our VAE is able to model the underlying distribution of the uncorrupted data , despite nontrivial corruption . We refer to our new method as “ Mixture Autoencoding with Wasserstein penalty ” , or “ MAW ” . In order to clarify it , we first review previous works and then explain our contributions in view of these works . 1.1 PREVIOUS WORK . Solutions to one-class classification and novelty detection either estimate the density of the inlier distribution ( Bengio & Monperrus , 2005 ; Ilonen et al. , 2006 ) or determine a geometric property of the inliers , such as their boundary set ( Breunig et al. , 2000 ; Schölkopf et al. , 2000 ; Xiao et al. , 2016 ; Wang & Lan , 2020 ; Jiang et al. , 2019 ) . When the inlier distribution is nicely approximated by a low-dimensional linear subspace , Shyu et al . ( 2003 ) proposes to distinguish between inliers and outliers via Principal Component Analysis ( PCA ) . In order to consider more general cases of nonlinear low-dimensional structures , one may use autoencoders ( or restricted Boltzmann machines ) , which nonlinearly generalize PCA ( Goodfellow et al. , 2016 , Ch . 2 ) and whose reconstruction error naturally provides a score for membership in the inlier class . Instances of this strategy with various architectures include Zhai et al . ( 2016 ) ; Zong et al . ( 2018 ) ; Sabokrou et al . ( 2018 ) ; Perera et al . ( 2019 ) ; Pidhorskyi et al . ( 2018 ) . In all of these works , but Zong et al . ( 2018 ) , the training set is assumed to solely represent the inlier class . In fact , Perera et al . ( 2019 ) observed that interpolation of a latent space , which was trained using digit images of a complex shape , can lead to digit representation of a simple shape . If there are also outliers ( with a simple shape ) among the inliers ( with a complex shape ) , encoding the inlier distribution becomes even more difficult . Nevertheless , some previous works already explored the possibility of corrupted training set ( Xiao et al. , 2016 ; Wang & Lan , 2020 ; Zong et al. , 2018 ) . In particular , Xiao et al . ( 2016 ) ; Zong et al . ( 2018 ) test artificial instances with at most 5 % corruption of the training set and Wang & Lan ( 2020 ) considers ratios of 10 % , but with very small numbers of training points . In this work we consider corruption ratios up to 30 % , with a method that tries to estimate the distribution of the training set , and not just a geometric property . VAEs ( Kingma & Welling , 2014 ) have been commonly used for generating distributions with reconstruction scores and are thus natural for novelty detection without corruption . They determine the latent code of an autoencoder via variational inference ( Jordan et al. , 1999 ; Blei et al. , 2017 ) . Alternatively , they can be viewed as autoencoders for distributions that penalize the Kullback-Leibler ( KL ) divergence of the latent distribution from the prior distribution . The first VAE-based method for novelty detection was suggested by An & Cho ( 2015 ) . It was recently extended by Daniel et al . ( 2019 ) who modified the training objective . A variety of VAE models were also proposed for special anomaly detection problems , which are different than novelty detection ( Xu et al. , 2018 ; Zhang et al. , 2019 ; Pol et al. , 2019 ) . Current VAE-based methods for novelty detection do not perform well when the training data is corrupted . Indeed , the learned distribution of any such method also represents the corruption , that is , the outlier component . To the best of our knowledge , no effective solutions were proposed for collapsing the outlier mode so that the trained VAE would only represent the inlier distribution . An adversarial autoencoder ( AAE ) ( Makhzani et al. , 2016 ) and a Wasserstein autoencoder ( WAE ) ( Tolstikhin et al. , 2018 ) can be considered as variants of VAE . The penalty term of AAE takes the form of a generative adversarial network ( GAN ) ( Goodfellow et al. , 2016 ) , where its generator is the encoder . A Wasserstein autoencoder ( WAE ) ( Tolstikhin et al. , 2018 ) generalizes AAE with a framework that minimizes the Wasserstein metric between the sample distribution and the inference distribution . It reformulates the corresponding objective function so that it can be implemented in the form of an AAE . There are two relevant lines of works on robustness to outliers in linear modeling that can be used in nonlinear settings via autoencoders or VAEs . Robust PCA aims to deal with sparse elementwise corruption of a data matrix ( Candès et al. , 2011 ; De La Torre & Black , 2003 ; Wright et al. , 2009 ; Vaswani & Narayanamurthy , 2018 ) . Robust subspace recovery ( RSR ) aims to address general corruption of selected data points and thus better fits the framework of outliers ( Watson , 2001 ; De La Torre & Black , 2003 ; Ding et al. , 2006 ; Zhang et al. , 2009 ; McCoy & Tropp , 2011 ; Xu et al. , 2012 ; Lerman & Zhang , 2014 ; Zhang & Lerman , 2014 ; Lerman et al. , 2015 ; Lerman & Maunu , 2017 ; Maunu et al. , 2019 ; Lerman & Maunu , 2018 ; Maunu & Lerman , 2019 ) . Autoencoders that use robust PCA for anomaly detection tasks were proposed in Chalapathy et al . ( 2017 ) ; Zhou & Paffenroth ( 2017 ) . Dai et al . ( 2018 ) show that a VAE can be interpreted as a nonlinear robust PCA problem . Nevertheless , explicit regularization is often required to improve robustness to sparse corruption in VAEs ( Akrami et al. , 2019 ; Eduardo et al. , 2020 ) . RSR was successfully applied to outlier detection by Lai et al . ( 2020 ) . One can apply their work to the different setting of novelty detection ; however , our proposed VAE formulation seems to work better . 1.2 THIS WORK . We propose a robust novelty detection procedure , MAW , that aims to model the distribution of the training data in the presence of nontrivial fraction of outliers . We highlight its following four features : • MAW models the latent distribution by a Gaussian mixture of low-rank inliers and full-rank outliers , and applies the inlier distribution for testing . Previous applications of mixture models for novelty detection were designed for multiple modes of inliers and used more complicated tools such as constructing another network ( Zong et al. , 2018 ) or applying clustering ( Aytekin et al. , 2018 ; Lee et al. , 2018 ) . • MAW applies a novel dimension reduction component , which extracts lower-dimensional features of the latent distribution . The reduced small dimension allows using full covariances for both the outliers ( with full rank ) and inliers ( with deficient rank ) ; whereas previous VAE-based methods for novelty detection used diagonal covariances in their models ( An & Cho , 2015 ; Daniel et al. , 2019 ) . The new component is inspired by the RSR layer in Lai et al . ( 2020 ) ; however , they are essentially different since the RSR layer is only applicable for data points and not for probability distributions . • For the latent code penalty , MAW uses the Wasserstein-1 ( W1 ) metric . Under a special setting , we prove that the Wasserstein metric gives rise to outliers-robust estimation and is suitable to the low-rank modeling of inliers by MAW . We also show that these properties do not hold for the KL divergence , which is used by VAE , AAE and WAE . We remark that the use of the Wasserstein metric in WAE is different than that of MAW . Indeed , in WAE it measures the distance between the data distribution and the generated distribution and it does not appear in the latent code . Our use ofW1 can be viewed as a variant of AAE , which replaces GAN with Wasserstein GAN ( WGAN ) ( Arjovsky et al. , 2017 ) . That is , it replaces the minimization of the KL divergence by that of theW1 distance . • MAW achieves state-of-the-art results on popular anomaly detection datasets . Additional two features are as follows . First , for reconstruction , MAW replaces the common least squares formulation with a least absolute deviations formulation . This can be justified by the use of either a robust estimator ( Lopuhaa & Rousseeuw , 1991 ) or a likelihood function with a heavier tail . Second , MAW is attractive for practitioners . It is simple to implement in any standard deep learning library , and is easily adaptable to other choices of network architecture , energy functions and similarity scores . We remark that since we do not have labels for the training set , we can not supervisedly learn the Gaussian component with low-rank covariance by the inliers and Gaussian component with the full-rank covariance by the outliers . However , the use of two robust losses ( least absolute deviation and the W1 distance ) helps obtain a careful model for the inliers , which is robust to outliers . Note that in our testing , we only use the model for the inliers . We explain MAW in §2 . We establish the advantage of its use of the Wasserstein metric in §3 . We carefully test MAW in §4 . At last , we conclude this work in §5 .

Summary: This study proposes a novel method that can work well even the training data is corrupted by partial data from the unknown domain. Though it deals with the well-known problem called 'Noisy data/label', its approach is not the same thing as the previous works as it focuses on variational autoencoder on the task of novelty detection. And its arguments and statistical assumptions are followed by mathematical proofs.