Iterative energy-based projection on a normal data manifold for anomaly localization

1 INTRODUCTION . Automating visual inspection on production lines with artificial intelligence has gained popularity and interest in recent years . Indeed , the analysis of images to segment potential manufacturing defects seems well suited to computer vision algorithms . However these solutions remain data hungry and require knowledge transfer from human to machine via image annotations . Furthermore , the classification in a limited number of user-predefined categories such as non-defective , greasy , scratched and so on , will not generalize well if a previously unseen defect appears . This is even more critical on production lines where a defective product is a rare occurrence . For visual inspection , a better-suited task is unsupervised anomaly detection , in which the segmentation of the defect must be done only via prior knowledge of non-defective samples , constraining the issue to a two-class segmentation problem . From a statistical point of view , an anomaly may be seen as a distribution outlier , or an observation that deviates so much from other observations as to arouse suspicion that it was generated by a different mechanism ( Hawkins , 1980 ) . In this setting , generative models such as Variational AutoEncoders ( VAE , Kingma & Welling ( 2014 ) ) , are especially interesting because they are capable to infer possible sampling mechanisms for a given dataset . The original autoencoder ( AE ) jointly learns an encoder model , that compresses input samples into a low dimensional space , and a decoder , that decompresses the low dimensional samples into the original input space , by minimizing the distance between the input of the encoder and the output of the decoder . The more recent variant , VAE , replaces the deterministic encoder and decoder by stochastic functions , enabling the modeling of the distribution of the dataset samples as well as the generation of new , unseen samples . In both models , the output decompressed sample given an input is often called the reconstruction , and is used as some sort of projection of the input on the support of the normal data distribution , which we will call the normal manifold . In most unsupervised anomaly detection methods based on VAE , models are trained on flawless data and defect detection and localization is then performed using a ∗Equal contributions . distance metric between the input sample and its reconstruction ( Bergmann et al. , 2018 ; 2019 ; An & Cho , 2015 ; Baur et al. , 2018 ; Matsubara et al. , 2018 ) . One fundamental issue in this approach is that the models learn on the normal manifold , hence there is no guarantee of the generalization of their behavior outside this manifold . This is problematic since it is precisely outside the dataset distribution that such methods intend to use the VAE for anomaly localization . Even in the case of a model that always generates credible samples from the dataset distribution , there is no way to ensure that the reconstruction will be connected to the input sample in any useful way . An example illustrating this limitation is given in figure 1 , where a VAE trained on regular grid images provides a globally poor reconstruction despite a local perturbation , making the anomaly localization challenging . In this paper , instead of using the VAE reconstruction , we propose to find a better projection of an input sample on the normal manifold , by optimizing an energy function defined by an autoencoder architecture . Starting at the input sample , we iterate gradient descent steps on the input to converge to an optimum , simultaneously located on the data manifold and closest to the starting input . This method allows us to add prior knowledge about the expected anomalies via regularization terms , which is not possible with the raw VAE reconstruction . We show that such an optimum is better than previously proposed autoencoder reconstructions to localize anomalies on a variety of unsupervised anomaly localization datasets ( Bergmann et al. , 2019 ) and present its inpainting capabilities on the CelebA dataset ( Liu et al. , 2015 ) . We also propose a variant of the standard gradient descent that uses the pixel-wise reconstruction error to speed up the convergence of the energy . 2 BACKGROUND . 2.1 GENERATIVE MODELS . In unsupervised anomaly detection , the only data available during training are samples x from a non-anomalous dataset X ⊂ Rd . In a generative setting , we suppose the existence of a probability function of density q , having its support on all Rd , from which the dataset was sampled . The generative objective is then to model an estimate of density q , from which we can obtain new samples close to the dataset . Popular generative architectures are Generative Adversarial Networks ( GAN , Goodfellow et al . ( 2014 ) ) , that concurrently train a generator G to generate samples from random , low-dimensional noise z ∼ p , z ∈ Rl , l d , and a discriminator D to classify generated samples and dataset samples . This model converges to the equilibrium of the expectation over both real and generated datasets of the binary cross entropy loss of the classifier minG maxD [ Ex∼q [ log ( D ( x ) ) ] + Ez∼p [ log ( 1 − D ( G ( z ) ) ) ] ] . Disadvantages of GANs are that they are notoriously difficult to train ( Goodfellow , 2017 ) , and they suffer from mode collapse , meaning that they have the tendency to only generate a subset of the original dataset . This can be problematic for anomaly detection , in which we do not want some subset of the normal data to be considered as anomalous ( Bergmann et al. , 2019 ) . Recent works such as Thanh-Tung et al . ( 2019 ) offer simple and attractive explanations for GAN behavior and propose substantial upgrades , however Ravuri & Vinyals ( 2019 ) still support the point that GANs have more trouble than other generative models to cover the whole distribution support . Another generative model is the VAE ( Kingma & Welling ( 2014 ) ) , where , similar to a GAN generator , a decoder model tries to approximate the dataset distribution with a simple latent variables prior p ( z ) , with z ∈ Rl , and conditional distributions output by the decoder p ( x|z ) . This leads to the estimate p ( x ) = ∫ p ( x|z ) p ( z ) dz , that we would like to optimize using maximum likelihood estimation on the dataset . To render the learning tractable with a stochastic gradient descent ( SGD ) estimator with reasonable variance , we use importance sampling , introducing density functions q ( z|x ) output by an encoder network , and Jensen ’ s inequality to get the variational lower bound : log p ( x ) = log Ez∼q ( z|x ) p ( x|z ) p ( z ) q ( z|x ) ≥ Ez∼q ( z|x ) log p ( x|z ) −DKL ( q ( z|x ) ‖p ( z ) ) = −L ( x ) ( 1 ) We will use L ( x ) as our loss function for training . We define the VAE reconstruction , per analogy with an autoencoder reconstruction , as the deterministic sample fV AE ( x ) that we obtain by encoding x , decoding the mean of the encoded distribution q ( z|x ) , and taking again the mean of the decoded distribution p ( x|z ) . VAEs are known to produce blurry reconstructions and generations , but Dai & Wipf ( 2019 ) show that a huge enhancement in image quality can be gained by learning the variance of the decoded distribution p ( x|z ) . This comes at the cost of the distribution of latent variables produced by the encoder q ( z ) being farther away from the prior p ( z ) , so that samples generated by sampling z ∼ p ( z ) , x ∼ p ( x|z ) have poorer quality . The authors show that using a second VAE learned on samples from q ( z ) , and sampling from it with ancestral sampling u ∼ p ( u ) , z ∼ p ( z|u ) , x ∼ p ( x|z ) , allows to recover samples of GAN-like quality . The original autoencoder can be roughly considered as a VAE whose encoded and decoded distributions have infinitely small variances . 2.2 ANOMALY DETECTION AND LOCALIZATION . We will consider that an anomaly is a sample with low probability under our estimation of the dataset distribution . The VAE loss , being a lower bound on the density , is a good proxy to classify samples between the anomalous and non-anomalous categories . To this effect , a threshold T can be defined on the loss function , delimiting anomalous samples with L ( x ) ≥ T and normal samples L ( x ) < T . However , according to Matsubara et al . ( 2018 ) , the regularization term LKL ( x ) = DKL ( q ( z|x ) ‖p ( z ) ) has a negative influence in the computation of anomaly scores . They propose instead an unregularized score Lr ( x ) = −Ez∼q ( z|x ) log p ( x|z ) which is equivalent to the reconstruction term of a standard autoencoder and claim a better anomaly detection . Going from anomaly detection to anomaly localization , this reconstruction term becomes crucial to most of existing solutions . Indeed , the inability of the model to reconstruct a given part of an image is used as a way to segment the anomaly , using a pixel-wise threshold on the reconstruction error . Actually , this segmentation is very often given by a pixel-wise ( An & Cho , 2015 ; Baur et al. , 2018 ; Matsubara et al. , 2018 ) or patch-wise comparison of the input image , and some generated image , as in Bergmann et al . ( 2018 ; 2019 ) , where the structural dissimilarity ( DSSIM , Wang et al . ( 2004 ) ) between the input and its VAE reconstruction is used . Autoencoder-based methods thus provide a straightforward way of generating an image conditioned on the input image . In the GAN original framework , though , images are generated from random noise z ∼ p ( z ) and are not conditioned by an input . Schlegl et al . ( 2017 ) propose with AnoGAN to get the closest generated image to the input using gradient descent on z for an energy defined by : EAnoGAN = ||x−G ( z ) ||1 + λ · ||fD ( x ) − fD ( G ( z ) ) ||1 ( 2 ) The first term ensures that the generation G ( z ) is close to the input x . The second term is based on a distance between features of the input and the generated images , where fD ( x ) is the output of an intermediate layer of the discriminator . This term ensures that the generated image stays in the vicinity of the original dataset distribution . 3 PROPOSED METHOD . 3.1 ADVERSARIAL PROJECTIONS . According to Zimmerer et al . ( 2018 ) , the loss gradient with respect to x gives the direction towards normal data samples , and its magnitude could indicate how abnormal a sample is . In their work on anomaly identification , they use the loss gradient as an anomaly score . Here we propose to use the gradient of the loss to iteratively improve the observed x . We propose to link this method to the methodology of computing adversarial samples in Szegedy et al . ( 2014 ) . After training a VAE on non-anomalous data , we can define a threshold T on the reconstruction loss Lr as in ( Matsubara et al. , 2018 ) , such that a small proportion of the most improbable samples are identified as anomalies . We obtain a binary classifier defined by A ( x ) = { 1 if Lr ( x ) ≥ T 0 otherwise ( 3 ) Our method consists in computing adversarial samples of this classifier ( Szegedy et al. , 2014 ) , that is to say , starting from a sample x0 with A ( x0 ) = 1 , iterate gradient descent steps over the input x , constructing samples x1 , . . .xN , to minimize the energy E ( x ) , defined as E ( xt ) = Lr ( xt ) + λ · ||xt − x0||1 ( 4 ) An iteration is done by calculating xt+1 as xt+1 = xt − α · ∇xE ( xt ) , ( 5 ) where α is a learning rate parameter , and λ is a parameter trading off the inclusion of xt in the normal manifold , given by Lr ( xt ) , and the proximity between xt and the input x0 , assured by the regularization term ||xt − x0||1 .

Summary: The paper proposes to use autoencoder for anomaly localization. The approach learns to project anomalous data on an autoencoder-learned manifold by using gradient descent on energy derived from the autoencoder's loss function. The proposed method is evaluated using the anomaly-localization dataset (Bergmann et al. CVPR 2019) and qualitatively for the task of image inpainting task on CelebA dataset.