RaPP: Novelty Detection with Reconstruction along Projection Pathway

1 INTRODUCTION . How can we characterize novelty when only normality information is given ? Novelty detection is the mechanism to decide whether a data sample is an outlier with respect to the training data . This mechanism is especially useful in situations where a proportion of detection targets is inherently small . Examples are fraudulent transaction detection ( Pawar et al. , 2014 ; Porwal & Mukund , 2018 ) , intrusion detection ( Lee , 2017 ; Aoudi et al. , 2018 ) , video surveillance ( Ravanbakhsh et al. , 2017 ; Xu et al. , 2015b ) , medical diagnosis ( Schlegl et al. , 2017 ; Baur et al. , 2018 ) and equipment failure detection ( Kuzin & Borovicka , 2016 ; Zhao et al. , 2017 ; Beghi et al. , 2014 ) . Recently , deep autoencoders and their variants have shown outstanding performances in finding compact representations from complex data , and the reconstruction error has been chosen as a popular metric for detecting novelty ( An & Cho , 2015 ; Vasilev et al. , 2018 ) . However , this approach has a limitation of measuring reconstruction quality only in an input space , which does not fully utilize hierarchical representations in hidden spaces identified by the deep autoencoder . In this paper , we propose RAPP , a new method of detecting novelty samples exploiting hidden activation values in addition to the input values and their autoencoder reconstruction values . While ordinary reconstruction-based methods carry out novelty detection by comparing differences between input data before the input layer and reconstructed data at the output layer , RAPP extends these comparisons to hidden spaces . We first collect a set of hidden activation values by feeding the original input to the autoencoder . Subsequently , we feed the autoencoder reconstructed input to the autoencoder to calculate another set of activation values in the hidden layers . This procedure does not need additional training of the autoencoder . In turn , we quantify the novelty of the input by aggregating these two sets of hidden activation values . To this end , we devise two metrics . The first metric measures the total amount of reconstruction errors in input and hidden spaces . The second metric normalizes the reconstruction errors before summing up . Note that RAPP falls back to the ordinary reconstruction-based method if we only aggregate input values before the input layer and the reconstructed values at the output layer . Also , we explain the motivations that facilitated the development of RAPP . We show that activation values in a hidden space obtained by feeding a reconstructed input to the autoencoder are equivalent to the corresponding reconstruction in that hidden space for the original input . We refer the latter quantity as a hidden reconstruction of the input . Note that this is a natural extension of the reconstruction to the hidden space . Unfortunately , we can not directly compute the hidden reconstruction as in the computation of the ordinary reconstruction because the autoencoder does not impose any correspondence between encoding-decoding pairs of hidden layers during the training . Nevertheless , we show that it can be computed by feeding a reconstructed input to the autoencoder again . Consequently , RAPP incorporates hidden reconstruction errors as well as the ordinary reconstruction error in detecting novelty . With extensive experiments , we demonstrate using diverse datasets that our method effectively improves autoencoder-based novelty detection methods . In addition , we show by evaluating on popular benchmark datasets that RAPP outperforms competing methods recently developed . Our contributions are summarized as follows . • We propose a new novelty detection method by utilizing hidden activation values of an input and its autoencoder reconstruction , and provide aggregation functions for them to quantify novelty of the input . • We provide motivation that RAPP extends the reconstruction concept in the input space into the hidden spaces . Precisely , we show that hidden activation values of a reconstructed input are equivalent to the corresponding hidden reconstruction of the original input . • We demonstrate that RAPP improves autoencoder-based novelty detection methods in diverse datasets . Moreover , we validate that RAPP outperforms recent novelty detection methods on popular benchmark datasets . 2 RELATED WORK . Various novelty detection methods with deep neural networks rely on the reconstruction error ( Sakurada & Yairi , 2014 ; Hoffmann , 2007 ; An & Cho , 2015 ) , because discriminative learning schemes are not suitable for highly class-imbalanced data which is common in practice . Unsupervised and semi-supervised learning approaches handle such imbalance by focusing on the characterization of normality and detecting samples out of the normality . Variational Autoencoders ( VAE ) ( Kingma & Welling , 2014 ) were reported to outperform vanilla autoencoders for novelty detection based on reconstruction error ( An & Cho , 2015 ) . To carry out the novelty detection outlined in this approach , an autoencoder needs to be trained only with normal data . The autoencoder encodes the training data , which comprises of only normal data in this case , into a lower-dimensional space and decodes them to the input space . To test novelty , an input value is fed to the autoencoder to produce a reconstructed value and calculate the distance between the input and reconstructed values . This distance is the reconstruction error . A higher reconstruction error means that the input value can not be encoded onto the lower-dimensional space that represents normal data . Therefore , the input value can be marked as a novelty if its reconstruction error exceeds a certain threshold . Instead of autoencoders , Generative Adversarial Networks ( GAN ) have been also suggested to model a distribution of normal data ( Sabokrou et al. , 2018 ; Schlegl et al. , 2017 ) . Despite the same purpose of discovering a simpler , lower-dimensional representation , the training criterion for GAN is focusing on the quality of data generation rather than the reconstruction quality of training data . Recently , several pieces of research have combined autoencoders and adversarial learning to meet both criteria in dimension reduction and data generation ( Haloui et al. , 2018 ; Pidhorskyi et al. , 2018 ; Zenati et al. , 2018 ) . One limitation of these methods based on the ordinary reconstruction error is that they do not exploit all the information available along the projection pathway of deep autoencoders . We will explain how to leverage this information for novelty detection in the next section . From the viewpoint of the diversity and ratio of the normal data in novelty detection , there are two cases available . The first case is when a small fraction of classes are normal . This case has been studied in a one-class classification context , and usually evaluated by organizing training data into a collection of samples belonging to a small number of normal classes ( Ruff et al. , 2018 ; Perera & Patel , 2018 ; Sabokrou et al. , 2018 ; Golan & El-Yaniv , 2018 ) . The second case is when a majority of classes are assigned as normal ( An & Cho , 2015 ; Schlegl et al. , 2017 ; Haloui et al. , 2018 ; Zenati et al. , 2018 ) . In this case , normal data is more diverse , and the training data is consist of samples of a relatively large number of normal classes : e.g. , nine digits of MNIST . One setup does not dominate the other , but depending on applications , either can be more suitable than the other . Different methods may perform differently in both cases . In this paper , we evaluate RAPP and other competing methods with experiments in both setups . 3 PROPOSED METHOD : RAPP . In this section , we describe the proposed novelty detection method RAPP based on an autoencoder . The main idea is to compare hidden activations of an input and its hidden reconstructions along the projection pathway of the autoencoder . To be precise , we project the input and its autoencoder reconstruction onto the hidden spaces to obtain pairs of activation values , and aggregate them to quantify the novelty of the input . For the aggregation , we present two metrics to measure the total amount of difference within each pair . 3.1 RECONSTRUCTION BASED NOVELTY DETECTION . An autoencoder A is a neural network consisting of an encoder g and a decoder f , responsible for dimension reduction and its inverse mapping to the original input space , respectively : i.e . A = f ◦g . For the purpose , training the autoencoder aims to minimize difference between its input x and output A ( x ) . The space that the encoder g constitutes is called the latent space , and provides more concise representation for data than the input space . Due to this unsupervised representation learning property , the autoencoder has been widely used for novelty detection . Specifically , training an autoencoder on normal data samples , novelty of a test sample x is measured by the following reconstruction error : = ‖x−A ( x ) ‖2 . The test sample x is more likely to be novel as the error ( x ) becomes larger , because it means that x is farther from the manifold that the autoencoder describes . Although this approach has shown promising results in novelty detection , the reconstruction error alone does not fully exploit information provided by a trained autoencoder especially when its architecture is deep . In other words , hierarchical information identified by the deep architecture is being ignored . This is rather unfortunate because hierarchical representation learning is one of the most successfully proven capabilities of deep neural networks . To fully leverage that capability , below we will describe the way to exploit hidden spaces to capture the difference between normal and novel samples in more detail . 3.2 RECONSTRUCTION ERROR IN HIDDEN SPACES . Let A = f ◦ g be a trained autoencoder where g and f are an encoder and a decoder , and ` be the number of hidden layers of g. Namely , g = g ` ◦ · · · ◦ g1 . We define partial computation of g as follows : g : i = gi ◦ · · · ◦ g1 , for 1 ≤ i ≤ ` . Let x be an input vector , and x̂ be its reconstruction by A : i.e. , x̂ = A ( x ) . In addition to comparing x and x̂ in the input space , as the ordinary approach does , we examine them in hidden spaces along a projection pathway of A . More precisely , feeding x and x̂ into A , we obtain pairs ( hi , ĥi ) of their hidden representations where hi ( x ) = g : i ( x ) , ĥi ( x ) = g : i ( x̂ ) = g : i ( A ( x ) ) . Figure 1a illustrates the procedure of computing hi and ĥi . As a result , novelty of the sample x is quantified by aggregating H ( x ) = { ( hi ( x ) , ĥi ( x ) ) : 1 ≤ i ≤ ` } . The overall procedure of RAPP is summarized in Algorithm 1 . To clearly state the required variables to construct H , we write the algorithm with the for loop in Lines 3–5 , but in practice , all of them Algorithm 1 : RAPP to compute a novelty score . Input : Sample x , trained autoencoder A = f ◦ g , the number of layers ` , and aggregation s. Output : Novelty score S. 1 x̂ = A ( x ) . 2 H = ∅ . 3 foreach i in 1 to ` do 4 H = H ∪ { ( g : i ( x ) , g : i ( x̂ ) ) } . 5 end 6 S = s ( H ) . can be computed by feed-forwarding one time each of x and x̂ to g. Note that RAPP is indeed a generalization of the ordinary reconstruction method with defining g0 as the identity function and sord as follows . sord ( H ( x ) ) = ‖h0 ( x ) − ĥ0 ( x ) ‖22 , where h0 ( x ) = g0 ( x ) = x and ĥ0 ( x ) = g0 ( x̂ ) = x̂ . In this paper , we provide two metrics sSAP and sNAP which more extensively utilize H than sord . Those are especially suited when no prior knowledge exists for the selection of layers to derive a novelty metric , which commonly happens when modeling with deep neural networks . Note that , however , more elaborate metrics can be designed if we have knowledge on or can characterize the spaces .

Summary: This paper proposes a novelty detection method by utilizing latent variables in auto-encoder. Based on this, this paper proposes two metrics to quantifying the novelty of the input. Their main contribution is the NAP metric based on SVD. Their method is empirically demonstrated on several benchmark datasets, and they compare their proposed metrics with other competing methods using AUROC and experiments results are encouraging.