Iterative energy-based projection on a normal data manifold for anomaly localization

1 INTRODUCTION . Automating visual inspection on production lines with artificial intelligence has gained popularity and interest in recent years . Indeed , the analysis of images to segment potential manufacturing defects seems well suited to computer vision algorithms . However these solutions remain data hungry and require knowledge transfer from human to machine via image annotations . Furthermore , the classification in a limited number of user-predefined categories such as non-defective , greasy , scratched and so on , will not generalize well if a previously unseen defect appears . This is even more critical on production lines where a defective product is a rare occurrence . For visual inspection , a better-suited task is unsupervised anomaly detection , in which the segmentation of the defect must be done only via prior knowledge of non-defective samples , constraining the issue to a two-class segmentation problem . From a statistical point of view , an anomaly may be seen as a distribution outlier , or an observation that deviates so much from other observations as to arouse suspicion that it was generated by a different mechanism ( Hawkins , 1980 ) . In this setting , generative models such as Variational AutoEncoders ( VAE , Kingma & Welling ( 2014 ) ) , are especially interesting because they are capable to infer possible sampling mechanisms for a given dataset . The original autoencoder ( AE ) jointly learns an encoder model , that compresses input samples into a low dimensional space , and a decoder , that decompresses the low dimensional samples into the original input space , by minimizing the distance between the input of the encoder and the output of the decoder . The more recent variant , VAE , replaces the deterministic encoder and decoder by stochastic functions , enabling the modeling of the distribution of the dataset samples as well as the generation of new , unseen samples . In both models , the output decompressed sample given an input is often called the reconstruction , and is used as some sort of projection of the input on the support of the normal data distribution , which we will call the normal manifold . In most unsupervised anomaly detection methods based on VAE , models are trained on flawless data and defect detection and localization is then performed using a ∗Equal contributions . distance metric between the input sample and its reconstruction ( Bergmann et al. , 2018 ; 2019 ; An & Cho , 2015 ; Baur et al. , 2018 ; Matsubara et al. , 2018 ) . One fundamental issue in this approach is that the models learn on the normal manifold , hence there is no guarantee of the generalization of their behavior outside this manifold . This is problematic since it is precisely outside the dataset distribution that such methods intend to use the VAE for anomaly localization . Even in the case of a model that always generates credible samples from the dataset distribution , there is no way to ensure that the reconstruction will be connected to the input sample in any useful way . An example illustrating this limitation is given in figure 1 , where a VAE trained on regular grid images provides a globally poor reconstruction despite a local perturbation , making the anomaly localization challenging . In this paper , instead of using the VAE reconstruction , we propose to find a better projection of an input sample on the normal manifold , by optimizing an energy function defined by an autoencoder architecture . Starting at the input sample , we iterate gradient descent steps on the input to converge to an optimum , simultaneously located on the data manifold and closest to the starting input . This method allows us to add prior knowledge about the expected anomalies via regularization terms , which is not possible with the raw VAE reconstruction . We show that such an optimum is better than previously proposed autoencoder reconstructions to localize anomalies on a variety of unsupervised anomaly localization datasets ( Bergmann et al. , 2019 ) and present its inpainting capabilities on the CelebA dataset ( Liu et al. , 2015 ) . We also propose a variant of the standard gradient descent that uses the pixel-wise reconstruction error to speed up the convergence of the energy . 2 BACKGROUND . 2.1 GENERATIVE MODELS . In unsupervised anomaly detection , the only data available during training are samples x from a non-anomalous dataset X ⊂ Rd . In a generative setting , we suppose the existence of a probability function of density q , having its support on all Rd , from which the dataset was sampled . The generative objective is then to model an estimate of density q , from which we can obtain new samples close to the dataset . Popular generative architectures are Generative Adversarial Networks ( GAN , Goodfellow et al . ( 2014 ) ) , that concurrently train a generator G to generate samples from random , low-dimensional noise z ∼ p , z ∈ Rl , l d , and a discriminator D to classify generated samples and dataset samples . This model converges to the equilibrium of the expectation over both real and generated datasets of the binary cross entropy loss of the classifier minG maxD [ Ex∼q [ log ( D ( x ) ) ] + Ez∼p [ log ( 1 − D ( G ( z ) ) ) ] ] . Disadvantages of GANs are that they are notoriously difficult to train ( Goodfellow , 2017 ) , and they suffer from mode collapse , meaning that they have the tendency to only generate a subset of the original dataset . This can be problematic for anomaly detection , in which we do not want some subset of the normal data to be considered as anomalous ( Bergmann et al. , 2019 ) . Recent works such as Thanh-Tung et al . ( 2019 ) offer simple and attractive explanations for GAN behavior and propose substantial upgrades , however Ravuri & Vinyals ( 2019 ) still support the point that GANs have more trouble than other generative models to cover the whole distribution support . Another generative model is the VAE ( Kingma & Welling ( 2014 ) ) , where , similar to a GAN generator , a decoder model tries to approximate the dataset distribution with a simple latent variables prior p ( z ) , with z ∈ Rl , and conditional distributions output by the decoder p ( x|z ) . This leads to the estimate p ( x ) = ∫ p ( x|z ) p ( z ) dz , that we would like to optimize using maximum likelihood estimation on the dataset . To render the learning tractable with a stochastic gradient descent ( SGD ) estimator with reasonable variance , we use importance sampling , introducing density functions q ( z|x ) output by an encoder network , and Jensen ’ s inequality to get the variational lower bound : log p ( x ) = log Ez∼q ( z|x ) p ( x|z ) p ( z ) q ( z|x ) ≥ Ez∼q ( z|x ) log p ( x|z ) −DKL ( q ( z|x ) ‖p ( z ) ) = −L ( x ) ( 1 ) We will use L ( x ) as our loss function for training . We define the VAE reconstruction , per analogy with an autoencoder reconstruction , as the deterministic sample fV AE ( x ) that we obtain by encoding x , decoding the mean of the encoded distribution q ( z|x ) , and taking again the mean of the decoded distribution p ( x|z ) . VAEs are known to produce blurry reconstructions and generations , but Dai & Wipf ( 2019 ) show that a huge enhancement in image quality can be gained by learning the variance of the decoded distribution p ( x|z ) . This comes at the cost of the distribution of latent variables produced by the encoder q ( z ) being farther away from the prior p ( z ) , so that samples generated by sampling z ∼ p ( z ) , x ∼ p ( x|z ) have poorer quality . The authors show that using a second VAE learned on samples from q ( z ) , and sampling from it with ancestral sampling u ∼ p ( u ) , z ∼ p ( z|u ) , x ∼ p ( x|z ) , allows to recover samples of GAN-like quality . The original autoencoder can be roughly considered as a VAE whose encoded and decoded distributions have infinitely small variances . 2.2 ANOMALY DETECTION AND LOCALIZATION . We will consider that an anomaly is a sample with low probability under our estimation of the dataset distribution . The VAE loss , being a lower bound on the density , is a good proxy to classify samples between the anomalous and non-anomalous categories . To this effect , a threshold T can be defined on the loss function , delimiting anomalous samples with L ( x ) ≥ T and normal samples L ( x ) < T . However , according to Matsubara et al . ( 2018 ) , the regularization term LKL ( x ) = DKL ( q ( z|x ) ‖p ( z ) ) has a negative influence in the computation of anomaly scores . They propose instead an unregularized score Lr ( x ) = −Ez∼q ( z|x ) log p ( x|z ) which is equivalent to the reconstruction term of a standard autoencoder and claim a better anomaly detection . Going from anomaly detection to anomaly localization , this reconstruction term becomes crucial to most of existing solutions . Indeed , the inability of the model to reconstruct a given part of an image is used as a way to segment the anomaly , using a pixel-wise threshold on the reconstruction error . Actually , this segmentation is very often given by a pixel-wise ( An & Cho , 2015 ; Baur et al. , 2018 ; Matsubara et al. , 2018 ) or patch-wise comparison of the input image , and some generated image , as in Bergmann et al . ( 2018 ; 2019 ) , where the structural dissimilarity ( DSSIM , Wang et al . ( 2004 ) ) between the input and its VAE reconstruction is used . Autoencoder-based methods thus provide a straightforward way of generating an image conditioned on the input image . In the GAN original framework , though , images are generated from random noise z ∼ p ( z ) and are not conditioned by an input . Schlegl et al . ( 2017 ) propose with AnoGAN to get the closest generated image to the input using gradient descent on z for an energy defined by : EAnoGAN = ||x−G ( z ) ||1 + λ · ||fD ( x ) − fD ( G ( z ) ) ||1 ( 2 ) The first term ensures that the generation G ( z ) is close to the input x . The second term is based on a distance between features of the input and the generated images , where fD ( x ) is the output of an intermediate layer of the discriminator . This term ensures that the generated image stays in the vicinity of the original dataset distribution . 3 PROPOSED METHOD . 3.1 ADVERSARIAL PROJECTIONS . According to Zimmerer et al . ( 2018 ) , the loss gradient with respect to x gives the direction towards normal data samples , and its magnitude could indicate how abnormal a sample is . In their work on anomaly identification , they use the loss gradient as an anomaly score . Here we propose to use the gradient of the loss to iteratively improve the observed x . We propose to link this method to the methodology of computing adversarial samples in Szegedy et al . ( 2014 ) . After training a VAE on non-anomalous data , we can define a threshold T on the reconstruction loss Lr as in ( Matsubara et al. , 2018 ) , such that a small proportion of the most improbable samples are identified as anomalies . We obtain a binary classifier defined by A ( x ) = { 1 if Lr ( x ) ≥ T 0 otherwise ( 3 ) Our method consists in computing adversarial samples of this classifier ( Szegedy et al. , 2014 ) , that is to say , starting from a sample x0 with A ( x0 ) = 1 , iterate gradient descent steps over the input x , constructing samples x1 , . . .xN , to minimize the energy E ( x ) , defined as E ( xt ) = Lr ( xt ) + λ · ||xt − x0||1 ( 4 ) An iteration is done by calculating xt+1 as xt+1 = xt − α · ∇xE ( xt ) , ( 5 ) where α is a learning rate parameter , and λ is a parameter trading off the inclusion of xt in the normal manifold , given by Lr ( xt ) , and the proximity between xt and the input x0 , assured by the regularization term ||xt − x0||1 .

Summary: This paper discusses an important problem of solving the visual inspection problem limited supervision.  It proposes to use VAE to model the anomaly detection. The major concern is how the quality of f_{VAE} is estimated. From the paper it seems f_{VAE} is not updated. Will it be sufficient to rely a fixed f_{VAE} and blindly trust its quality?