Iterative Image Inpainting with Structural Similarity Mask for Anomaly Detection

1 INTRODUCTION . Anomaly detection ( AD ) is the identification task of the rarely happened events or items that differ from the majority of the data . In the real world , there are many applications , such as the medial diagnosis ( Baur et al. , 2018 ; Zimmerer et al. , 2019a ) , defect detection in the factories ( Matsubara et al. , 2018 ; Bergmann et al. , 2019 ) , early detection of plant disease ( Wang et al. , 2019 ) , and X-Ray security detection in public space ( Griffin et al. , 2018 ) . Because manual inspection by humans is slow , expensive , and error-prone , automating visual inspection is the popular application of artificial intelligence . In transferring knowledge from humans to machines , there is a lack of anomalous samples due to their low event rate and difficulty annotating and categorizing various anomalous defects beforehand . Therefore , AD methods typically take unsupervised approaches that try to learn compact features of data from normal samples and detect anomalies by thresholding anomaly score to measure the deviation from learned features . To deal with high-dimensional images and learn their features , it is popular to use deep neural networks ( Goodfellow et al. , 2016 ) . In this work , we focus on the reconstruction-based unsupervised AD . This attempts to reconstruct only the normal dataset and classify the normal or anomalous data on thresholding reconstruction errors ( An & Cho , 2015 ) . The architectures are based on deep neural networks such as deep autoencoders ( Hinton & Salakhutdinov , 2006 ) , variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) , or autoencoders with generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) . These models compress the high-dimensional information into the data manifold in lower-dimensional latent space by reconstructing input data under certain constraints for latent space , such as a prior distribution or an information bottleneck ( Alemi et al. , 2016 ) . The reconstruction-based AD approach issue is that autoencoders fail to model small details and yield blurry image reconstruction . This is especially the case for the high-frequency textures , such as carpet , leather , and tile ( Bergmann et al. , 2019 ) . Dehaene et al . ( 2020 ) also pointed out that there is no guarantee of the generalization of their behavior for out-of-samples , and local defects added to normal images could deteriorate whole images . In the viewpoint of the signal-to-noise ratio ( SNR ) , it is interpreted that blurry reconstruction makes anomaly signals ( reconstruction errors in anomaly pixels ) unclear and increases normal noises ( reconstruction errors in normal pixels ) . Since the SNR explains the feasibility of AD by thresholding a sample-wise reconstruction error , the low SNR makes AD challenges . We point out an additional issue about the gap between optimized function at training and evaluated function at testing . Rethinking our goal in unsupervised AD , it is concluded not to minimize reconstruction errors merely but to maximize the SNR . Although models are trained to minimize a sample-wise reconstruction error , they are expected to have a large deviation of anomaly pixels and a small deviation of normal pixels at testing . In this paper , we propose I3AD ( Iterative Image Inpainting for Anomaly Detection ) . As Figure 1 , our method utilizes an inpainting model that only encode unmasked regions and reconstruct masked regions instead of vanilla autoencoders . Once computed reconstruction errors , it is recycled as an inpainting mask for the next iteration . We show that the iterative update enhances the reconstruction quality and satisfies the expected objective to maximize the expected SNR at testing . Through experiments and analysis on the MVTecAD dataset shows that our I3AD outperforms existing methods on nine categories and has ave. +11.6 % improvement on texture category . 2 METHODOLOGY . 2.1 HIGH-LEVEL IDEA . We think of unsupervised AD using autoencoders . Here , we implicitly assume anomalies show up in partial regions , and pixels in surrounding regions obey by the distribution of normal datasets . Therefore , the sample-wise anomaly score based on reconstruction errors is the summation of two types of pixels : ( 1 ) pixels of normal regions ( normal noises ) and ( 2 ) pixels of anomalous regions ( anomaly signals ) . We expect an ideal model with zero errors on normal regions and distinguishable per-pixel scores on anomalous regions , leading to the high SNR . Inheriting vanilla autoencoders ’ architecture does not help to resolve the low SNR issue mentioned by Bergmann et al . ( 2019 ) and Dehaene et al . ( 2020 ) . Indeed , autoencoders are forced to encode whole images with anomalous pixels of local defects and attempt to decode whole images with background normal pixels of fine structures . They do not learn their behavior to encode unseen anomalous pixels . That anomalous information can affect the whole image decoding . One approach to resolve the issue is a combination of the per-pixel identity function and conditional autoencoder . Compared to vanilla autoencoders , conditional autoencoders can encode only normal regions and decode only anomalous regions . The per-pixel identity function could copy the remaining unreconstructed regions . This model architecture is the same as an image inpainting model . Actually , deep inpainting models are designed by conditional autoencoders that encode unmasked regions and fill in masked regions under a certain mask matrix ( Yu et al. , 2019 ) . However , the image inpainting method for AD falls into the tautology trap that we do not know a perfect inpainting mask matrix in advance , and detecting anomalous regions is the main goal to achieve . The key ideas to disentangle this tautology are mask generation by the anomaly score and iterative update of a mask matrix . Updating the inpainting mask matrix dynamically controls encoding and decoding information balances with a pixel-wise confidence level of anomaly scores . A generator gradually receives more information on potentially normal pixels and focuses on the suspected pixels during iterations . Furthermore , this process not only reduces background noises but also improves the SNR directly . 2.2 ITERATIVE IMAGE INPAINTING FOR ANOMALY DETECTION . Following the above discussion , we construct our I3AD method by an inpainting generator and a mask generation module . As a mask generation module , we explain the detail in the next subsection . Our model overview is depicted in Figure 2 . We construct an inpainting generator using conditional generative adversarial networks ( cGANs ) ( Isola et al. , 2017 ) and train its networks by a general image inpainting task under a normal dataset . We feed normal images partially hidden by randomly generated masks into generator networks and train them to decode the masked pixels from the unmasked pixels and corresponding bool mask matrix . A discriminator network distinguishes generated images from normal images . A generator is rewarded for fooling a discriminator , while a discriminator is rewarded for detecting the generated images , respectively . This training can be considered as the two-player min-max game that a generator and a discriminator compete . As a result , the inpainting model tries to find the optimal point on the loss function as below : min G max D Ex∼Pdata ( x ) [ logD ( x ) ] +Ex∼Pdata ( x ) , M∼P ( M ) [ log ( 1−D ( G ( x̂ =M x , M ) ) ) ] , where x and x̂ are real samples from data distribution Pdata ( x ) and their masked real samples . is an element-wise product . M is the corresponding bool mask matrix generated from the random distribution P ( M ) . G ( x̂ , M ) is an image inpainting network that takes an incomplete image and masked matrix . D ( x ) denotes a binary classifier whether an image is generated or real . We borrow and customize Spectral-Normalized Markovian GAN ( SN-PatchGAN ) architecture following by Yu et al . ( 2019 ) . The network consists of two networks : a coarse-to-fine generator network with attention module and gated convolution , and a spectral normalized Markovian ( Patch ) discriminator network . Our I3AD expects to handle more fine structured masks than usual free-form masks . Therefore , to better handle more irregular masks , we apply the self-attention module ( Zhang et al. , 2018 ) instead of the contextual attention module originally designed for large rectangular masks as described in Yu et al . ( 2018 ; 2019 ) . To stabilize the training of GAN , we adapted proposed spectral normalization ( Miyato et al. , 2018 ) for discriminator ’ s layers . As approximation of objective min-max loss function ( Miyato et al. , 2018 ) , we also derived loss functions respectively for generator LG and discriminator LD below : LG = −Ex∼Pdata ( x ) , M∼P ( M ) [ D sn ( G ( x̂ =M x , M ) ) ] LD = Ex∼Pdata ( x ) [ ReLU ( 1−D sn ( x ) ) ] + Ex∼Pdata ( x ) , M∼P ( M ) [ ReLU ( 1 +D sn ( G ( x̂ =M x , M ) ) ) ] , where Dsn ( x ) denotes spectral-normalized discriminator . ReLU is the abbreviation of Rectified Linear Units activation function , defined by ReLU ( x ) = max ( 0 , x ) . For a generator network , we use a spatially discounted l1 reconstruction loss ( Yu et al. , 2018 ) . At the test step , we fix all trainable parameters in the I3AD generator . The I3AD generator receives test images that are normal or anomalous images and adaptive mask matrices . Mask matrices are constructed from the pixel-wise reconstruction errors between original images and generated images at the previous iteration step . Since mask matrices are dynamically updated and shrunken during iterations , the I3AD generator generates masked regions intensively , leveraging the gradually increasing information of the surrounding unmasked regions . 2.3 INPAINTING MASK OF STRUCTURAL SIMILARITY ( SSIM MASK ) . In anomaly segmentation tasks , structural similarity measure ( SSIM ) index ( Wang et al. , 2004 ) sharply measures the small anomalous change ( Bergmann et al. , 2018 ) . Details of SSIM calculation is in Appendix A . We propose the structural similarity mask ( SSIM-Mask ) to mask anomalous pixels during test iterations . SSIM-Mask Mi is a binary mask thresholding the pixel-wise SSIM Index between an input image x0 and the reconstructed image x̃i at ith iteration step , defined as Mi = { 1 if ai ( x ) ≥ u 0 otherwise , ai ( x ) = SSIM ( x0 , x̃i ) , x̃i = G ( x̂i−1 , Mi−1 ) where u denotes the threshold level for binary classification . After N iterations , we use the N th SSIM anomaly score aN ( x ) for AD evaluation . 2.4 MASK INITIALIZATION . We have no mask information at the first iteration step during testing . We use four checkerboard matrices to initialize masks . Figure 6 shows initialized mask examples . A generator encodes pixels of test images in white regions and decodes pixels in black boxes . These black regions are mutually exclusive between four masks and cover target images collectively . Therefore , we combine four generated images into a single whole reconstruction . 2.5 ITERATION STEPS AND STOP CRITERIA . We expect no masked region for normal images and some local masked region for anomalous images . Therefore , applying iterative inpainting for some samples on the training dataset could estimate iteration steps enough to reduce masks on normal pixels . Since I3AD decodes only masked regions , Mi+1 will be almost all subset of Mi . Therefore , we could set early stop criteria whether the difference between Mi and Mi+1 is small against masked regions of Mi .

Summary: This paper presents an impainting-based method for anomaly localization on images. In the training time, a conditional GAN-based generative modeling approach is adopted. In the test time,  a mask matrix is adaptively estimated by thresholding the structural similarity index measure (SSIM) between the original images and reconstructed images. The idea is very intuitive and experiments demonstrate improved performance (especially on textures) over two recent baselines methods.