Robust Subspace Recovery Layer for Unsupervised Anomaly Detection

1 INTRODUCTION . Finding and utilizing patterns in data is a common task for modern machine learning systems . However , there is often some anomalous information that does not follow a common pattern and has to be recognized . For this purpose , anomaly detection aims to identify data points that “ do not conform to expected behavior ” ( Chandola et al. , 2009 ) . We refer to such points as either anomalous or outliers . In many applications , there is no ground truth available to distinguish anomalous from normal points , and they need to be detected in an unsupervised fashion . For example , one may need to remove anomalous images from a set of images obtained by a search engine without any prior knowledge about how a normal image should look ( Xia et al. , 2015 ) . Similarly , one may need to distinguish unusual news items from a large collection of news documents without any information whether a news item is usual or not ( Kannan et al. , 2017 ) . In these examples , the only assumptions are that normal data points appear more often than anomalous ones and have a simple underlying structure which is unknown to the user . Some early methods for anomaly detection relied on Principal Component Analysis ( PCA ) ( Shyu et al. , 2003 ) . Here one assumes that the underlying unknown structure of the normal samples is linear . However , PCA is sensitive to outliers and will often not succeed in recovering the linear structure or identifying the outliers ( Lerman & Maunu , 2018 ; Vaswani & Narayanamurthy , 2018 ) . More recent ideas of Robust PCA ( RPCA ) ( Wright et al. , 2009 ; Vaswani & Narayanamurthy , 2018 ) have been considered for some specific problems of anomaly detection or removal ( Zhou & Paffenroth , 2017 ; Paffenroth et al. , 2018 ) . RPCA assumes sparse corruption , that is , few elements of the data matrix are corrupted . This assumption is natural for some special problems in computer vision , in particular , background subtraction ( De La Torre & Black , 2003 ; Wright et al. , 2009 ; Vaswani & Narayanamurthy , 2018 ) . However , a natural setting of anomaly detection with hidden linear structure may assume instead that a large portion of the data points are fully corrupted . The mathematical framework that addresses this setting is referred to as robust subspace recovery ( RSR ) ( Lerman & Maunu , 2018 ) . While Robust PCA and RSR try to extract linear structure or identify outliers lying away from such structure , the underlying geometric structure of many real datasets is nonlinear . Therefore , one ∗Equal contribution . needs to extract crucial features of the nonlinear structure of the data while being robust to outliers . In order to achieve this goal , we propose to use an autoencoder ( composed of an encoder and a decoder ) with an RSR layer . We refer to it as RSRAE ( RSR autoencoder ) . It aims to robustly and nonlinearly reduce the dimension of the data in the following way . The encoder maps the data into a high-dimensional space . The RSR layer linearly maps the embedded points into a low-dimensional subspace that aims to learn the hidden linear structure of the embedded normal points . The decoder maps the points from this subspace to the original space . It aims to map the normal points near their original locations , and the anomalous points far from their original locations . Ideally , the encoder maps the normal data to a linear space and any anomalies lie away from this subspace . In this ideal scenario , anomalies can be removed by an RSR method directly applied to the data embedded by the encoder . Since the linear model for the normal data embedded by the encoder is only approximate , we do not directly apply RSR to the embedded data . Instead , we minimize a sum of the reconstruction error of the autoencoder and the RSR error for the data embedded by the encoder . We advocate for an alternating procedure , so that the parameters of the autoencoder and the RSR layer are optimized in turn . 1.1 STRUCTURE OF THE REST OF THE PAPER . Section 2 reviews works that are directly related to the proposed RSRAE and highlights the original contributions of this paper . Section 3 explains the proposed RSRAE , and in particular , its RSR layer and total energy function . Section 4 includes extensive experimental evidence demonstrating effectiveness of RSRAE with both image and document data . Section 5 discusses theory for the relationship of the RSR penalty with the WGAN penalty . Section 6 summarizes this work and mentions future directions . 2 RELATED WORKS AND CONTRIBUTION . We review related works in Section 2.1 and highlight our contribution in Section 2.2 . 2.1 RELATED WORKS . Several recent works have used autoencoders for anomaly detection . Xia et al . ( 2015 ) proposed the earliest work on anomaly detection via an autoencoder , while utilizing large reconstruction error of outliers . They apply an iterative and cyclic scheme , where in each iteration , they determine the inliers and use them for updating the parameters of the autoencoder . Aytekin et al . ( 2018 ) apply ` 2 normalization for the latent code of the autoencoder and also consider the case of multiple modes for the normal samples . Instead of using the reconstruction error , they apply k-means clustering for the latent code , and identify outliers as points whose latent representations are far from all the cluster centers . Zong et al . ( 2018 ) also use an autoencoder with clustered latent code , but they fit a Gaussian Mixture Model using an additional neural network . Restricted Boltzmann Machines ( RBMs ) are similar to autoencoders . Zhai et al . ( 2016 ) define “ energy functions ” for RBMs that are similar to the reconstruction losses for autoencoders . They identify anomalous samples according to large energy values . Chalapathy et al . ( 2017 ) propose using ideas of RPCA within an autoencoder , where they alternatively optimize the parameters of the autoencoder and a sparse residual matrix . The above works are designed for datasets with a small fraction of outliers . However , when this fraction increases , outliers are often not distinguished by high reconstruction errors or low similarity scores . In order to identify them , additional assumptions on the structure of the normal data need to be incorporated . For example , Zhou & Paffenroth ( 2017 ) decompose the input data into two parts : low-rank and sparse ( or column-sparse ) . The low-rank part is fed into an autoencoder and the sparse part is imposed as a penalty term with the ` 1-norm ( or ` 2,1-norm for column-sparsity ) . In this work , we use a term analogous to the ` 2,1-norm , which can be interpreted as the sum of absolute deviations from a latent subspace . However , we do not decompose the data a priori , but minimize an energy combining this term and the reconstruction error . Minimization of the former term is known as least absolute deviations in RSR ( Lerman & Maunu , 2018 ) . It was first suggested for RSR and related problems in Watson ( 2001 ) ; Ding et al . ( 2006 ) ; Zhang et al . ( 2009 ) . The robustness to outliers of this energy , or of relaxed versions of it , was studied in McCoy & Tropp ( 2011 ) ; Xu et al . ( 2012 ) ; Lerman & Zhang ( 2014 ) ; Zhang & Lerman ( 2014 ) ; Lerman et al . ( 2015 ) ; Lerman & Maunu ( 2017 ) ; Maunu et al . ( 2017 ) . In particular , Maunu et al . ( 2017 ) established its well-behaved landscape under special , though natural , deterministic conditions . Under similar conditions , they guaranteed fast subspace recovery by a simple algorithm that aims to minimize this energy . Another directly related idea for extracting useful latent features is an addition of a linear selfexpressive layer to an autoencoder ( Ji et al. , 2017 ) . It is used in the different setting of unsupervised subspace clustering . By imposing the self-expressiveness , the autoencoder is robust to an increasing number of clusters . Although self-expressiveness also improves robustness to noise and outliers , Ji et al . ( 2017 ) aims at clustering and thus its goal is different than ours . Furthermore , their selfexpressive energy does not explicitly consider robustness , while ours does . Lezama et al . ( 2018 ) consider a somewhat parallel idea of imposing a loss function to increase the robustness of representation . However , their goal is to increase the margin between classes and their method only applies to a supervised setting in anomaly detection , where the normal data is multi-modal . 2.2 CONTRIBUTION OF THIS WORK . This work introduces an RSR layer within an autoencoder . It incorporates a special regularizer that enforces an outliers-robust linear structure in the embedding obtained by the encoder . We clarify that the method does not alternate between application of the autoencoder and the RSR layer , but fully integrates these two components . Our experiments demonstrate that a simple incorporation of a “ robust loss ” within a regular autoencoder does not work well for anomaly detection . We try to explain this and also the improvement obtained by incorporating an additional RSR layer . Our proposed architecture is simple to implement . Furthermore , the RSR layer is not limited to a specific design of RSRAE but can be put into any well-designed autoencoder structure . The epoch time of the proposed algorithm is comparable to those of other common autoencoders . Furthermore , our experiments show that RSRAE competitively performs in unsupervised anomaly detection tasks . RSRAE addresses the unsupervised setting , but is not designed to be highly competitive in the semisupervised or supervised settings , where one has access to training data from the normal class or from both classes , respectively . In these settings , RSRAE functions like a regular autoencoder without taking an advantage of its RSR layer , unless the training data for the normal class is corrupted with outliers . The use of RSR is not restricted to autoencoders . We establish some preliminary analysis for RSR within a generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ; Arjovsky et al. , 2017 ) in Section 5 . More precisely , we show that a linear WGAN intrinsically incorporates RSR in some special settings , although it is unclear how to impose an RSR layer . 3 RSR LAYER FOR OUTLIER REMOVAL . We assume input data { x ( t ) } Nt=1 in RM , and denote by X its corresponding data matrix , whose t-th column is x ( t ) . The encoder of RSRAE , E , is a neural network that maps each data point , x ( t ) , to its latent code z ( t ) = E ( x ( t ) ) ∈ RD . The RSR layer is a linear transformation A ∈ Rd×D that reduces the dimension to d. That is , z̃ ( t ) = Az ( t ) ∈ Rd . The decoder D is a neural network that maps z̃ ( t ) to x̃ ( t ) in the original ambient space RM . We can write the forward maps in a compact form using the corresponding data matrices as follows : Z = E ( X ) , Z̃ = AZ , X̃ = D ( Z̃ ) . ( 1 ) Ideally , we would like to optimize RSRAE so it only maintains the underlying structure of the normal data . We assume that the original normal data lies on a d-dimensional “ manifold ” in RD and thus the RSR layer embeds its latent code into Rd . In this ideal optimization setting , the similarity between the input and the output of RSRAE is large whenever the input is normal and small whenever the input is anomalous . Therefore , by thresholding a similarity measure , one may distinguish between normal and anomalous data points . In practice , the matrix A and the parameters of E and D are obtained by minimizing a loss function , which is a sum of two parts : the reconstruction loss from the autoencoder and the loss from the RSR layer . For p > 0 , an ` 2 , p reconstruction loss for the autoencoder is LpAE ( E , A , D ) = N∑ t=1 ∥∥∥x ( t ) − x̃ ( t ) ∥∥∥p 2 . ( 2 ) In order to motivate our choice of RSR loss , we review a common formulation for the original RSR problem . In this problem one needs to recover a linear subspace , or equivalently an orthogonal projection P onto this subspace . Assume a dataset { y ( t ) } Nt=1 and let I denote the identity matrix in the ambient space of the dataset . The goal is to find an orthogonal projector P of dimension d whose subspace robustly approximates this dataset . The least q-th power deviations formulation for q > 0 , or least absolute deviations when q = 1 ( Lerman & Maunu , 2018 ) , seeks P that minimizes L̂ ( P ) = N∑ t=1 ∥∥∥ ( I−P ) y ( t ) ∥∥∥q 2 . ( 3 ) The solution of this problem is robust to some outliers when q ≤ 1 ( Lerman & Zhang , 2014 ; Lerman & Maunu , 2017 ) ; furthermore , q < 1 can result in a wealth of local minima and thus q = 1 is preferable ( Lerman & Zhang , 2014 ; Lerman & Maunu , 2017 ) . A similar loss function to ( 3 ) for RSRAE is LqRSR ( A ) = λ1LRSR1 ( A ) + λ2LRSR2 ( A ) : = λ1 N∑ t=1 ∥∥∥∥∥∥z ( t ) −ATAz ( t ) ︸ ︷︷ ︸ z̃ ( t ) ∥∥∥∥∥∥ q 2 + λ2 ∥∥AAT − Id∥∥2F , ( 4 ) where AT denotes the transpose of A , Id denotes the d × d identity matrix and ‖·‖F denotes the Frobenius norm . Here λ1 , λ2 > 0 are predetermined hyperparameters , though we later show that one may solve the underlying problem without using them . We note that the first term in the weighted sum of ( 4 ) is close to ( 3 ) as long as ATA is close to an orthogonal projector . To enforce this requirement we introduced the second term in the weighted sum of ( 4 ) . In Appendix C we discuss further properties of the RSR energy and its minimization . To emphasize the effect of outlier removal , we take p = 1 in ( 2 ) and q = 1 in ( 4 ) . That is , we use the l2,1 norm , or the formulation of least absolute deviations , for both reconstruction and RSR . The loss function of RSRAE is the sum of the two loss terms in ( 2 ) and ( 4 ) , that is , LRSRAE ( E , A , D ) = L 1 AE ( E , A , D ) + L 1 RSR ( A ) . ( 5 ) We remark that the sole minimization of L1AE , without L 1 RSR , is not effective for anomaly detection . We numerically demonstrate this in Section 4.3 and also try to explain it in Section 5.1 . Our proposed algorithm for optimizing ( 5 ) , which we refer to as the RSRAE algorithm , uses alternating minimization . It iteratively backpropagates the three terms L1AE , LRSR1 , LRSR2 and accordingly updates the parameters of the RSR autoencoder . For clarity , we describe this basic procedure in Algorithm 1 of Appendix A . It is independent of the values of the parameters λ1 and λ2 . Note that the additional gradient step with respect to the RSR loss just updates the parameters in A . Therefore it does not significantly increase the epoch time of a standard autoencoder for anomaly detection . Another possible method , which we refer to as RSRAE+ , is direct minimization of LRSRAE with predetermined λ1 and λ2 via auto-differentiation ( see Algorithm 2 of Appendix A ) . Section 4.3 and Appendix I.2 demonstrate that in general , RSRAE performs better than RSRAE+ , though it is possible that similar performance can be achieved by carefully tuning the parameters λ1 and λ2 when implementing RSRAE+ . We remark that a standard autoencoder is obtained by minimizing only L2AE , without the RSR loss . One might hope that minimizing L1AE may introduce the needed robustness . However , Section 4.3 and Appendix I.2 demonstrate that results obtained by minimizing L1AE or L 2 AE are comparable , and are worse than those of RSRAE and RSRAE+ .

Summary: This paper adapts the concept of Robust Subspace Recovery (RSR) as a layer in an auto-encoder model for anomaly detection. A loss function is proposed that combines reconstruction error and a regularizer that enforces robustness against outliers. The reconstruction error expresses the accuracy of the nonlinear dimensionality reduction imposed by the autoencoder. The regularizer is the sum of absolute deviations from the latent subspace that represents a linear structure robust against outliers. An alternative procedure is applied where the loss terms are applied iteratively during training. Once trained, the reconstruction error is used directly for anomaly detection with a threshold. The AUC is used as a performance measure. The method is compared against 6 other methods (LOF, OCSVM, IF, DESBM, GT, DAGMM). The setting is fully unsupervised, meaning that the training data contains various amounts of anomalies, and the results are parametrized with the amount of corruption. The results show that the proposed approach outperforms the other methods in most cases, especially for larger amounts of corruption. An ablation study compares the approach with auto-encoder-only and a non-alternating gradient descent (fixed factors for each part of the loss function) and shows that the alternating method outperfroms all by a wide margin.