Iterative Image Inpainting with Structural Similarity Mask for Anomaly Detection

1 INTRODUCTION . Anomaly detection ( AD ) is the identification task of the rarely happened events or items that differ from the majority of the data . In the real world , there are many applications , such as the medial diagnosis ( Baur et al. , 2018 ; Zimmerer et al. , 2019a ) , defect detection in the factories ( Matsubara et al. , 2018 ; Bergmann et al. , 2019 ) , early detection of plant disease ( Wang et al. , 2019 ) , and X-Ray security detection in public space ( Griffin et al. , 2018 ) . Because manual inspection by humans is slow , expensive , and error-prone , automating visual inspection is the popular application of artificial intelligence . In transferring knowledge from humans to machines , there is a lack of anomalous samples due to their low event rate and difficulty annotating and categorizing various anomalous defects beforehand . Therefore , AD methods typically take unsupervised approaches that try to learn compact features of data from normal samples and detect anomalies by thresholding anomaly score to measure the deviation from learned features . To deal with high-dimensional images and learn their features , it is popular to use deep neural networks ( Goodfellow et al. , 2016 ) . In this work , we focus on the reconstruction-based unsupervised AD . This attempts to reconstruct only the normal dataset and classify the normal or anomalous data on thresholding reconstruction errors ( An & Cho , 2015 ) . The architectures are based on deep neural networks such as deep autoencoders ( Hinton & Salakhutdinov , 2006 ) , variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) , or autoencoders with generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) . These models compress the high-dimensional information into the data manifold in lower-dimensional latent space by reconstructing input data under certain constraints for latent space , such as a prior distribution or an information bottleneck ( Alemi et al. , 2016 ) . The reconstruction-based AD approach issue is that autoencoders fail to model small details and yield blurry image reconstruction . This is especially the case for the high-frequency textures , such as carpet , leather , and tile ( Bergmann et al. , 2019 ) . Dehaene et al . ( 2020 ) also pointed out that there is no guarantee of the generalization of their behavior for out-of-samples , and local defects added to normal images could deteriorate whole images . In the viewpoint of the signal-to-noise ratio ( SNR ) , it is interpreted that blurry reconstruction makes anomaly signals ( reconstruction errors in anomaly pixels ) unclear and increases normal noises ( reconstruction errors in normal pixels ) . Since the SNR explains the feasibility of AD by thresholding a sample-wise reconstruction error , the low SNR makes AD challenges . We point out an additional issue about the gap between optimized function at training and evaluated function at testing . Rethinking our goal in unsupervised AD , it is concluded not to minimize reconstruction errors merely but to maximize the SNR . Although models are trained to minimize a sample-wise reconstruction error , they are expected to have a large deviation of anomaly pixels and a small deviation of normal pixels at testing . In this paper , we propose I3AD ( Iterative Image Inpainting for Anomaly Detection ) . As Figure 1 , our method utilizes an inpainting model that only encode unmasked regions and reconstruct masked regions instead of vanilla autoencoders . Once computed reconstruction errors , it is recycled as an inpainting mask for the next iteration . We show that the iterative update enhances the reconstruction quality and satisfies the expected objective to maximize the expected SNR at testing . Through experiments and analysis on the MVTecAD dataset shows that our I3AD outperforms existing methods on nine categories and has ave. +11.6 % improvement on texture category . 2 METHODOLOGY . 2.1 HIGH-LEVEL IDEA . We think of unsupervised AD using autoencoders . Here , we implicitly assume anomalies show up in partial regions , and pixels in surrounding regions obey by the distribution of normal datasets . Therefore , the sample-wise anomaly score based on reconstruction errors is the summation of two types of pixels : ( 1 ) pixels of normal regions ( normal noises ) and ( 2 ) pixels of anomalous regions ( anomaly signals ) . We expect an ideal model with zero errors on normal regions and distinguishable per-pixel scores on anomalous regions , leading to the high SNR . Inheriting vanilla autoencoders ’ architecture does not help to resolve the low SNR issue mentioned by Bergmann et al . ( 2019 ) and Dehaene et al . ( 2020 ) . Indeed , autoencoders are forced to encode whole images with anomalous pixels of local defects and attempt to decode whole images with background normal pixels of fine structures . They do not learn their behavior to encode unseen anomalous pixels . That anomalous information can affect the whole image decoding . One approach to resolve the issue is a combination of the per-pixel identity function and conditional autoencoder . Compared to vanilla autoencoders , conditional autoencoders can encode only normal regions and decode only anomalous regions . The per-pixel identity function could copy the remaining unreconstructed regions . This model architecture is the same as an image inpainting model . Actually , deep inpainting models are designed by conditional autoencoders that encode unmasked regions and fill in masked regions under a certain mask matrix ( Yu et al. , 2019 ) . However , the image inpainting method for AD falls into the tautology trap that we do not know a perfect inpainting mask matrix in advance , and detecting anomalous regions is the main goal to achieve . The key ideas to disentangle this tautology are mask generation by the anomaly score and iterative update of a mask matrix . Updating the inpainting mask matrix dynamically controls encoding and decoding information balances with a pixel-wise confidence level of anomaly scores . A generator gradually receives more information on potentially normal pixels and focuses on the suspected pixels during iterations . Furthermore , this process not only reduces background noises but also improves the SNR directly . 2.2 ITERATIVE IMAGE INPAINTING FOR ANOMALY DETECTION . Following the above discussion , we construct our I3AD method by an inpainting generator and a mask generation module . As a mask generation module , we explain the detail in the next subsection . Our model overview is depicted in Figure 2 . We construct an inpainting generator using conditional generative adversarial networks ( cGANs ) ( Isola et al. , 2017 ) and train its networks by a general image inpainting task under a normal dataset . We feed normal images partially hidden by randomly generated masks into generator networks and train them to decode the masked pixels from the unmasked pixels and corresponding bool mask matrix . A discriminator network distinguishes generated images from normal images . A generator is rewarded for fooling a discriminator , while a discriminator is rewarded for detecting the generated images , respectively . This training can be considered as the two-player min-max game that a generator and a discriminator compete . As a result , the inpainting model tries to find the optimal point on the loss function as below : min G max D Ex∼Pdata ( x ) [ logD ( x ) ] +Ex∼Pdata ( x ) , M∼P ( M ) [ log ( 1−D ( G ( x̂ =M x , M ) ) ) ] , where x and x̂ are real samples from data distribution Pdata ( x ) and their masked real samples . is an element-wise product . M is the corresponding bool mask matrix generated from the random distribution P ( M ) . G ( x̂ , M ) is an image inpainting network that takes an incomplete image and masked matrix . D ( x ) denotes a binary classifier whether an image is generated or real . We borrow and customize Spectral-Normalized Markovian GAN ( SN-PatchGAN ) architecture following by Yu et al . ( 2019 ) . The network consists of two networks : a coarse-to-fine generator network with attention module and gated convolution , and a spectral normalized Markovian ( Patch ) discriminator network . Our I3AD expects to handle more fine structured masks than usual free-form masks . Therefore , to better handle more irregular masks , we apply the self-attention module ( Zhang et al. , 2018 ) instead of the contextual attention module originally designed for large rectangular masks as described in Yu et al . ( 2018 ; 2019 ) . To stabilize the training of GAN , we adapted proposed spectral normalization ( Miyato et al. , 2018 ) for discriminator ’ s layers . As approximation of objective min-max loss function ( Miyato et al. , 2018 ) , we also derived loss functions respectively for generator LG and discriminator LD below : LG = −Ex∼Pdata ( x ) , M∼P ( M ) [ D sn ( G ( x̂ =M x , M ) ) ] LD = Ex∼Pdata ( x ) [ ReLU ( 1−D sn ( x ) ) ] + Ex∼Pdata ( x ) , M∼P ( M ) [ ReLU ( 1 +D sn ( G ( x̂ =M x , M ) ) ) ] , where Dsn ( x ) denotes spectral-normalized discriminator . ReLU is the abbreviation of Rectified Linear Units activation function , defined by ReLU ( x ) = max ( 0 , x ) . For a generator network , we use a spatially discounted l1 reconstruction loss ( Yu et al. , 2018 ) . At the test step , we fix all trainable parameters in the I3AD generator . The I3AD generator receives test images that are normal or anomalous images and adaptive mask matrices . Mask matrices are constructed from the pixel-wise reconstruction errors between original images and generated images at the previous iteration step . Since mask matrices are dynamically updated and shrunken during iterations , the I3AD generator generates masked regions intensively , leveraging the gradually increasing information of the surrounding unmasked regions . 2.3 INPAINTING MASK OF STRUCTURAL SIMILARITY ( SSIM MASK ) . In anomaly segmentation tasks , structural similarity measure ( SSIM ) index ( Wang et al. , 2004 ) sharply measures the small anomalous change ( Bergmann et al. , 2018 ) . Details of SSIM calculation is in Appendix A . We propose the structural similarity mask ( SSIM-Mask ) to mask anomalous pixels during test iterations . SSIM-Mask Mi is a binary mask thresholding the pixel-wise SSIM Index between an input image x0 and the reconstructed image x̃i at ith iteration step , defined as Mi = { 1 if ai ( x ) ≥ u 0 otherwise , ai ( x ) = SSIM ( x0 , x̃i ) , x̃i = G ( x̂i−1 , Mi−1 ) where u denotes the threshold level for binary classification . After N iterations , we use the N th SSIM anomaly score aN ( x ) for AD evaluation . 2.4 MASK INITIALIZATION . We have no mask information at the first iteration step during testing . We use four checkerboard matrices to initialize masks . Figure 6 shows initialized mask examples . A generator encodes pixels of test images in white regions and decodes pixels in black boxes . These black regions are mutually exclusive between four masks and cover target images collectively . Therefore , we combine four generated images into a single whole reconstruction . 2.5 ITERATION STEPS AND STOP CRITERIA . We expect no masked region for normal images and some local masked region for anomalous images . Therefore , applying iterative inpainting for some samples on the training dataset could estimate iteration steps enough to reduce masks on normal pixels . Since I3AD decodes only masked regions , Mi+1 will be almost all subset of Mi . Therefore , we could set early stop criteria whether the difference between Mi and Mi+1 is small against masked regions of Mi .

Summary: This paper presents a method for contrastive anomaly detection (AD) using an iterative masked conditional autoencoder inpainting approach. An autoencoder network is trained using an adversarial approach to reconstruct a randomly masked part of the input image. At test time a mask is derived from the generated anomaly map (using SSIM index between the input and reconstructed image) and used to mask the input so that the process is repeated N times. The method is shown to produce SOT results on the MVTec AD benchmark.