Deep Semi-Supervised Anomaly Detection

1 INTRODUCTION . Anomaly detection ( AD ) ( Chandola et al. , 2009 ; Pimentel et al. , 2014 ) is the task of identifying unusual samples in data . Typically AD methods attempt to learn a “ compact ” description of the data in an unsupervised manner assuming that most of the samples are normal ( i.e. , not anomalous ) . For example , in one-class classification ( Moya et al. , 1993 ; Schölkopf et al. , 2001 ) the objective is to find a set of small measure which contains most of the data and samples not contained in that set are deemed anomalous . Shallow unsupervised AD methods such as the One-Class SVM ( Schölkopf et al. , 2001 ; Tax & Duin , 2004 ) , Kernel Density Estimation ( Parzen , 1962 ; Kim & Scott , 2012 ; Vandermeulen & Scott , 2013 ) , or Isolation Forest ( Liu et al. , 2008 ) often require manual feature engineering to be effective on high-dimensional data and are limited in their scalability to large datasets . These limitations have sparked great interest in developing novel deep approaches to unsupervised AD ( Erfani et al. , 2016 ; Zhai et al. , 2016 ; Chen et al. , 2017 ; Ruff et al. , 2018 ; Deecke et al. , 2018 ; Ruff et al. , 2019 ; Golan & El-Yaniv , 2018 ; Pang et al. , 2019 ; Hendrycks et al. , 2019a ; b ) . ∗Majority of the work was done while RV was at TU Kaiserslautern , Germany . †Part of the work was done while MK was a sabbatical visitor of the DASH Center at the University of Southern California , United States . 1 Published as a conference paper at ICLR 2020 Unlike the standard unsupervised AD setting , in many real-world applications one may also have access to some verified ( i.e. , labeled ) normal or anomalous samples in addition to the unlabeled data . Such samples could be hand labeled by a domain expert for instance . This leads to a semi-supervised AD problem : given n ( mostly normal but possibly containing some anomalous contamination ) unlabeled samples x1 , . . . , xn and m labeled samples ( x̃1 , ỹ1 ) , . . . , ( x̃m , ỹm ) , where ỹ = +1 and ỹ = −1 denote normal and anomalous samples respectively , the task is to learn a model that compactly characterizes the “ normal class. ” The term semi-supervised anomaly detection has been used to describe two different AD settings . Most existing “ semi-supervised ” AD methods , both shallow ( Muñoz-Marí et al. , 2010 ; Blanchard et al. , 2010 ; Chandola et al. , 2009 ) and deep ( Song et al. , 2017 ; Akcay et al. , 2018 ; Chalapathy & Chawla , 2019 ) , only incorporate the use of labeled normal samples but not labeled anomalies , i.e . they are more precisely instances of Learning from Positive ( i.e. , normal ) and Unlabeled Examples ( LPUE ) ( Denis , 1998 ; Zhang & Zuo , 2008 ) . A few works ( Wang et al. , 2005 ; Liu & Zheng , 2006 ; Görnitz et al. , 2013 ) have investigated the general semi-supervised AD setting where one also utilizes labeled anomalies , however existing deep approaches are domain or data-type specific ( Ergen et al. , 2017 ; Kiran et al. , 2018 ; Min et al. , 2018 ) . Research on deep semi-supervised learning has almost exclusively focused on classification as the downstream task ( Kingma et al. , 2014 ; Rasmus et al. , 2015 ; Odena , 2016 ; Dai et al. , 2017 ; Oliver et al. , 2018 ) . Such semi-supervised classifiers typically assume that similar points are likely to be of the same class , this is known as the cluster assumption ( Zhu , 2005 ; Chapelle et al. , 2009 ) . This assumption , however , only holds for the “ normal class ” in AD , but is crucially invalid for the “ anomaly class ” since anomalies are not necessarily similar to one another . Instead , semi-supervised AD approaches must find a compact description of the normal class while also correctly discriminating the labeled anomalies ( Görnitz et al. , 2013 ) . Figure 1 illustrates the differences between various learning paradigms applied to AD on a toy example . We introduce Deep SAD ( Deep Semi-supervised Anomaly Detection ) in this work , an end-to-end deep method for general semi-supervised AD . Our main contributions are the following : 2 Published as a conference paper at ICLR 2020 • We introduce Deep SAD , a generalization of the unsupervised Deep SVDD method ( Ruff et al. , 2018 ) to the semi-supervised AD setting . • We present an information-theoretic framework for deep AD , which can serve as an inter- pretation of our Deep SAD method and similar approaches . • We conduct extensive experiments in which we establish experimental scenarios for the general semi-supervised AD problem where we also introduce novel baselines . 2 AN INFORMATION-THEORETIC VIEW ON DEEP ANOMALY DETECTION . The study of the theoretical foundations of deep learning is an active and ongoing research effort ( Montavon et al. , 2011 ; Tishby & Zaslavsky , 2015 ; Cohen et al. , 2016 ; Eldan & Shamir , 2016 ; Neyshabur et al. , 2017 ; Raghu et al. , 2017 ; Zhang et al. , 2017 ; Achille & Soatto , 2018 ; Arora et al. , 2018 ; Belkin et al. , 2018 ; Wiatowski & Bölcskei , 2018 ; Lapuschkin et al. , 2019 ) . One important line of research that has emerged is rooted in information theory ( Shannon , 1948 ) . In the supervised classification setting where one has input variable X , latent variable Z ( e.g. , the final layer of a deep network ) , and output variable Y ( i.e. , the label ) , the well-known Information Bottleneck principle ( Tishby et al. , 1999 ; Tishby & Zaslavsky , 2015 ; Shwartz-Ziv & Tishby , 2017 ; Alemi et al. , 2017 ; Saxe et al. , 2018 ) provides an explanation for representation learning as the trade-off between finding a minimal compression Z of the input X while retaining the informativeness of Z for predicting the label Y . Put formally , supervised deep learning seeks to minimize the mutual information I ( X ; Z ) between the input X and the latent representation Z while maximizing the mutual information I ( Z ; Y ) between Z and the classification task Y , i.e . min p ( z|x ) I ( X ; Z ) − α I ( Z ; Y ) , ( 1 ) where p ( z|x ) is modeled by a deep network and the hyperparameter α > 0 controls the trade-off between compression ( i.e. , complexity ) and classification accuracy . For unsupervised deep learning , due to the absence of labels Y and thus the lack of a clear task , other information-theoretic learning principles have been formulated . Of these , the Infomax principle ( Linsker , 1988 ; Bell & Sejnowski , 1995 ; Hjelm et al. , 2019 ) is one of the most prevalent and widely used principles . In contrast to ( 1 ) , the objective of Infomax is to maximize the mutual information I ( X ; Z ) between the data X and its latent representation Z : max p ( z|x ) I ( X ; Z ) + βR ( Z ) . ( 2 ) This is typically done under some additional constraint or regularizationR ( Z ) on the representation Z with hyperparameter β > 0 to obtain statistical properties desired for some specific downstream task . Examples where the Infomax principle has been applied include tasks such as independent component analysis ( Bell & Sejnowski , 1995 ) , clustering ( Slonim et al. , 2005 ; Ji et al. , 2018 ) , generative modeling ( Chen et al. , 2016 ; Hoffman & Johnson , 2016 ; Zhao et al. , 2017 ; Alemi et al. , 2018 ) , and unsupervised representation learning in general ( Hjelm et al. , 2019 ) . We observe that the Infomax principle has also been applied in previous deep representations for AD . Most notably autoencoders ( Rumelhart et al. , 1986 ; Hinton & Salakhutdinov , 2006 ) , which are the predominant approach to deep AD ( Hawkins et al. , 2002 ; Sakurada & Yairi , 2014 ; Andrews et al. , 2016 ; Erfani et al. , 2016 ; Zhai et al. , 2016 ; Chen et al. , 2017 ; Chalapathy & Chawla , 2019 ) , can be understood as implicitly maximizing the mutual information I ( X ; Z ) via the reconstruction objective ( Vincent et al. , 2008 ) under some regularization of the latent code Z . Choices for regularization include sparsity ( Makhzani & Frey , 2014 ) , the distance to some latent prior distribution , e.g . measured via the KL divergence ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) , an adversarial loss ( Makhzani et al. , 2015 ) , or simply a bottleneck in dimensionality . Such restrictions for AD share the idea that the latent representation of the normal data should be in some sense “ compact. ” As illustrated in Figure 1 , a supervised ( or semi-supervised ) classification approach to AD only learns to recognize anomalies similar to those seen during training , due to the class cluster assumption ( Chapelle et al. , 2009 ) . However , anything not normal is by definition an anomaly and thus anomalies do not have to be similar . This makes supervised ( or semi-supervised ) classification learning principles such as ( 1 ) ill-defined for AD . We instead build upon principle ( 2 ) to motivate a deep method for general semi-supervised AD , where we include the label information Y through a novel representation learning regularization objectiveR ( Z ) = R ( Z ; Y ) that is based on entropy . 3 Published as a conference paper at ICLR 2020 3 DEEP SEMI-SUPERVISED ANOMALY DETECTION . In the following , we introduce Deep SAD , a deep method for general semi-supervised AD . To formulate our objective , we first briefly explain the unsupervised Deep SVDD method ( Ruff et al. , 2018 ) which we then generalize to the semi-supervised AD setting . 3.1 UNSUPERVISED DEEP SVDD AND ENTROPY MINIMIZATION . For input space X ⊆ RD and output space Z ⊆ Rd , let φ ( · ; W ) : X → Z be a neural network with L hidden layers and corresponding set of weights W = { W 1 , . . . , WL } . The objective of Deep SVDD is to train the neural network φ to learn a transformation that minimizes the volume of a data-enclosing hypersphere in output space Z centered on a predetermined point c. Given n ( unlabeled ) training samples x1 , . . . , xn ∈ X , the One-Class Deep SVDD objective is min W 1 n n∑ i=1 ‖φ ( xi ; W ) − c‖2 + λ 2 L∑ ` =1 ‖W ` ‖2F , λ > 0 . ( 3 ) Penalizing the mean squared distance of the mapped samples to the hypersphere center c forces the network to extract those common factors of variation which are most stable within the dataset . As a consequence normal data points tend to get mapped near the hypersphere center , whereas anomalies are mapped further away ( Ruff et al. , 2018 ) . The second term is a standard weight decay regularizer . Deep SVDD is optimized via SGD using backpropagation . For initialization , Ruff et al . ( 2018 ) first pre-train an autoencoder and then initialize the weightsW of the network φ with the converged weights of the encoder . After initialization , the hypersphere center c is set as the mean of the network outputs obtained from an initial forward pass of the data . Once the network is trained , the anomaly score for a test point x is given by the distance from φ ( x ; W ) to the center of the hypersphere : s ( x ) = ‖φ ( x ; W ) − c‖ . ( 4 ) We now argue that Deep SVDD may not only be interpreted in geometric terms as minimum volume estimation ( Scott & Nowak , 2006 ) , but also in probabilistic terms as entropy minimization over the latent distribution . For a latent random variableZ with covariance Σ , pdf p ( z ) , and supportZ ⊆ Rd , we have the following bound on entropy H ( Z ) = E [ − log p ( Z ) ] = − ∫ Z p ( z ) log p ( z ) dz ≤ 1 2 log ( ( 2πe ) d det Σ ) , ( 5 ) which holds with equality iff Z is jointly Gaussian ( Cover & Thomas , 2012 ) . Assuming the latent distribution Z follows an isotropic Gaussian , Z ∼ N ( µ , σ2I ) with σ > 0 , we get H ( Z ) = 1 2 log ( ( 2πe ) d detσ2I ) = 1 2 log ( ( 2πeσ2 ) d · 1 ) = d 2 ( 1 + log ( 2πσ2 ) ) ∝ log σ2 , ( 6 ) i.e . for a fixed dimensionality d , the entropy of Z is proportional to its log-variance . Now observe that the Deep SVDD objective ( 3 ) ( disregarding weight decay regularization ) is equivalent to minimizing the empirical variance and thus minimizes an upper bound on the entropy of a latent Gaussian . Since the Deep SVDD network is pre-trained on an autoencoding objective that implicitly maximizes the mutual information I ( X ; Z ) ( Vincent et al. , 2008 ) , we may interpret Deep SVDD as following the Infomax principle ( 2 ) with the additional “ compactness ” objective that the latent distribution should have minimal entropy .

Summary: The paper proposes an abnormal detection (AD) framework under general settings where 1) unlabeled data, 2) labeled positive (normal) data, and 3) labeled negative (abnormal) data are available (with the last two optional), denoted as semi-supervised AD. Starting from the assumption that abnormal data are sampled from background unpredicted distribution, rather than “cluster” assumption, it is argued that conventional discriminative formulation is not applicable. Motivated by the recent deep AD methods (e.g., deep SVDD), the paper proposes to approach semi-supervised AD from the information theoretic perspective where 1) mutual information between raw data and learnt representation should be maximized (infomax principle), 2) entropy of labeled positive data should be minimized (“compactness” constraint), and 3) enrtropy of labeled negative data should be maximized to reflect the uncertainty assumption of anomaly. The solution is implemented by the encoder of a pre-trained autoencoder that is further fine tuned to enforce entropy assumption on all types of training data. Extensive experiments on benchmarks suggests promising results on the proposed framework versus other state-of-the-arts.