Iterative Image Inpainting with Structural Similarity Mask for Anomaly Detection
Authors: Hitoshi Nakanishi
Masahiro Suzuki
Yutaka Matsuo
Venue:  (2021)
Keywords: anomaly detection
unsupervised learning
structural similarity
generative adversarial network
deep learning

Abstract: Autoencoders have emerged as popular methods for unsupervised anomaly detection. Autoencoders trained on the normal data are expected to reconstruct only the normal features, allowing anomaly detection by thresholding reconstruction errors. However, in practice, autoencoders fail to model small detail and yield blurry reconstructions, which makes anomaly detection challenging. Moreover, there is objective mismatching that models are trained to minimize total reconstruction errors while expecting a small deviation on normal pixels and a large deviation on anomalous pixels. To tackle these two issues, we propose the iterative image inpainting method that reconstructs partial regions in an adaptive inpainting mask matrix. This method constructs inpainting masks from the anomaly score of structural similarity. Overlaying inpainting mask on images, each pixel is bypassed or reconstructed based on the anomaly score, enhancing reconstruction quality. The iterative update of inpainted images and masks by turns purifies the anomaly score directly and follows the expected objective at test time. We evaluated the proposed method using the MVTec Anomaly Detection dataset. Our method outperformed previous state-of-the-art in several categories and showed remarkable improvement in high-frequency textures.

Under review as a conference paper at ICLR 2021
ITERATIVE IMAGE INPAINTING WITH STRUCTURAL SIMILARITY MASK FOR ANOMALY DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Autoencoders have emerged as popular methods for unsupervised anomaly detec- tion. Autoencoders trained on the normal data are expected to reconstruct only the normal features, allowing anomaly detection by thresholding reconstruction errors. However, in practice, autoencoders fail model small detail and yield blurry reconstructions, which makes anomaly detection challenging. Moreover, there is objective mismatching that models are trained to minimize total recon- struction errors while expecting small deviation normal pixels and large deviation on anomalous pixels. To tackle these two issues, we propose the iter- ative image inpainting method that reconstructs partial regions in an adaptive in- painting mask matrix. This method constructs inpainting masks from the anomaly score of structural similarity. Overlaying inpainting mask on images, each pixel is bypassed or reconstructed based on the anomaly score, enhancing reconstruction quality. The iterative update of inpainted images and masks by turns purifies the anomaly score directly and follows the expected objective at test time. We eval- uated the proposed method using the MVTec Anomaly Detection dataset. Our method outperformed previous state-of-the-art in several categories and showed remarkable improvement high-frequency textures.
INTRODUCTION
Anomaly detection (AD) the identification task of the rarely happened events or items that differ from the majority of the data. In the real world, there are many applications, such as the medial diagnosis (Baur et al. 2018 Zimmerer al. 2019a), defect detection the factories (Matsubara et al. 2018 Bergmann et al. 2019). early detection of plant disease (Wang et al. 2019). and X-Ray security detection in public space (Griffin et al. 2018). Because manual inspection by humans is slow, expensive, and error-prone, automating visual inspection is the popular application of artificial intelligence. In transferring knowledge from humans to machines, there is a lack of anomalous samples due to their low event rate and difficulty annotating and categorizing various anomalous defects beforehand. Therefore, AD methods typically take unsupervised approaches that try to learn compact features of data from normal samples and detect anomalies by thresholding anomaly score to measure the deviation from learned features. To deal with high-dimensional images and learn their features, it popular use deep neural networks (Goodfellow et al. 2016).
In this work, we focus on the reconstruction-bant based unsupervised AD. This attempts to reconstruct only the normal dataset and classify the normal or anomalous data on thresholding reconstruction errors (An & Cho. 2015). The architectures are based on deep neural networks such as deep au- toencoders (Hinton Salakhutdinov 2006). variational autoencoders (VAEs) (Kingma Welling 2013 Rezende et 20114) or autoencoders with generative adversarial networks (GANs) -Goodfel- low et al. 4). These models compress the high-dimensional information into the data manifold in lower-dimensional latent space by reconstructing input data under certain constraints for latent space, such as a prior distribution or an information bottleneck (Alemi et al. 2016).
The icconstruction-based AD approach issue is that autoencoders fail to model small details and yield blurry image reconstruction. This is especially the case for the high- figh-frequency textures, such carpet, leather, and tile (Bergmann et al. 2019). Dehaene et al. (2020) also pointed out that there is no guarantee of the generalization of their behavior for sut-ol-samples, and local defects added to normal images could deteriorate whole images. In the viewpoint of the signal-to-noise ratio (SNR),

---

Under review as a conference paper at ICLR 2021
Autoencoder
Iterative Image Inpainting (Ours)
Anomaly Image
Masking
bolizaatio on
Reconstructed Image
Residual Map (Anomaly Score)
Encoder
Decoder
unmasked region
Decoder of masked region
Figure Autoencoder vs. BAD method (Ours). Autoencoder fails to reconstruct the high- frequency texture of "carpet" and yields large residual map. Ours utilizes the residual map as an inpainting mask and iteratively updates the reconstructed image and the residual map.
it is interpreted that blurry reconstruction makes anomaly signals (reconstruction errors in anomaly pixels) unclear and increases normal noises (reconstruction errors in normal pixels). Since the SNR explains the feasibility of AD by thresholding sample-wise reconstruction error, the low SNR makes AD challenges.
We point out an additional issue about the gap between optimized function at training and evalu- ated function testing. Rethinking our goal in unsupervised AD, concluded not to minimize reconstruction errors merely but to maximize the SNR. Although models are trained to minimize sample-wise reconstruction error, they are expected to have large deviation of anomaly pixels and small deviation of normal pixels at testing.
In this paper, we propose I3AD (Iterative Image Inpainting for Anomaly Detection). As Figure [ method utilizes inpainting model that only encode unmasked regions and reconstruct masked regions instead of vanilla autoencoders. Once computed reconstruction errors, it is recycled as an inpainting mask for the next iteration. We show that the iterative update enhances the reconstruction quality and satisfies the expected objective to maximize the expected SNR at testing. Through exper- iments and analysis the MVTecAD dataset shows that our I3AD outperforms existing methods on nine categories and has ave. +1.6% improvement on texture category.
METHODOLOGY
2.1 HIGH-LEVEL IDEA
We think of unsupervised AD using autoencoders. Here, we implicitly assume anomalies show up in partial regions, and pixels in surrounding regions obey by the distribution of normal datasets. Therefore, the sample-wise anomaly score based reconstruction errors is the summation of two types of pixels: (1) pixels of normal regions (normal noises) and (2) pixels of anomalous regions (anomaly signals). We expect ideal model with zero errors normal regions and distinguishable per-pixel scores on anomalous regions, leading to the high SNR.
Inheriting vanilla autoencoders' architecture does not help to resolve the low SNR issue mentioned by |errmmann et al. (2019) and Dehaene et al. (2020). Indeed, autoencoders are forced to encode whole images with anomalous pixels of local defects and attempt decode whole images with background normal pixels of fine structures. They do not learn their behavior to encode unseen anomalous pixels. That anomalous information affect the whole image decoding.
2

---

Under review as a conference paper at ICLR 2021
Image Inpainting (Train)
Free-form random mask
Train Image (Normal)
Reconstructed
Image
Encoder (unmasked region)
Decoder (masked region)
Real
minator
or
Fake
Anomaly Detection (Test)
Residual Map
Test Image
Reconstructed Image
Residual Map
Anomaly Score
(Normal Anomaly)
(N Anomaly Score
Finally to measure
utilize mask matrix
Figure 2: Model overview of iterative image inpainting method for AD (I3AD). At the training step, we solve a general image inpainting task in normal datasets. At the testing step, generator receives test images with an adaptive mask and only reconstructs masked regions. This adaptive mask is updated from previous reconstruction results during iterations.
One approach to resolve the issue is combination of the per-pixel identity function and conditional autoencoder. Compared vanilla autoencoders, conditional autoencoders can encode only normal regions and decode only anomalous regions. The per-pixel identity function could copy the remain- ing unreconstructed regions. This model architecture is same as an image inpainting model. Actually, deep inpainting models designed by conditional autoencoders that encode unmasked regions and fill masked regions under a certain mask matrix Yu et al., (019). However, the image inpainting method for AD falls into the tautology trap that do not know perfect inpainting mask matrix in advance, and detecting anomalous regions is the main goal to achieve.
The key ideas disentangle this tautology mask generation by the anomaly score and iterative update of mask matrix. Updating the inpainting mask matrix dynamically controls encoding and decoding information balances with a pixel-wise confidence level of anomaly scores. A generator gradually receives more information on potentially normal pixels and focuses on the suspected pixels during iterations. Furthermore, this process not only reduces background noises but also improves the SNR directly.
2.2 ITERATIVE IMAGE INPAINTING FOR ANOMALY DETECTION
Following the above discussion, we construct our I3AD method by an inpainting generator mask generation module. As mask generation module, explain the detail in the next subsection.
Our model overview depicted in Figure We construct an inpainting generator using conditional generative adversarial networks (cGANs) (Isola et al. 2017) and train its networks by a general image inpainting task under a normal dataset. We feed normal images partially hidden by ran- domly generated masks into generator networks and train them decode the masked pixels from the unmasked pixels and corresponding bool mask matrix. A discriminator network distinguishes generated images from normal images. A generator is rewarded for fooling discriminator, while a discriminator is rewarded for detecting the generated images, respectively. This training can be considered as the two-player min- max game that a generator and discriminator compete. As result, the inpainting model tries to find the optimal point on the loss function as below:
min max (gn-Pann(w)) D
(w) flog
MP. [Ig((1 D(G = Mo M))),,
where v and are real samples from data distribution (a)ta(x) and their masked real samples. O an dementtwise product. M is the corresponding bool mask matrix generated from the random distribution P(M). G(ă, .M) is an image inpainting network that takes an incomplete image and masked matrix. (2x)) denotes binary classifier whether an image is generated real.
3

---

Under review as a conference paper at ICLR 2021
We borrow and customize Spectral-Normale Markovian GAN (SN-PatchGAN) architecture fol- lowing by] et et al. (2019). The network consists of two networks: a scoare--t-finn generator network with attention module and gated convolution, and spectral normalized Markovian (Patch) discrim- inator network. Our I3AD expects to handle more fine structured masks than usual free-form masks. Therefore, to better handle more irregular masks, we apply the self-attention module (Zhang al.) 2018) instead of the contextual attention module originally designed for large rectangular masks as described in Yu al. (2018) 2019).
To stabilize the training of GAN, we adapted proposed spectral normalization (Miyato et al. 2018 for discriminator's 's layers. As approximation of objective min-max loss function (Miyato et al,, 2018), we also derived loss functions respectively for generator Lg and discriminator Lp below:
LG (E ( M ox,,
LD E
+
[ReLU(1 + D (G( = M o x, M))),
where (z) denotes spectral-normalized discriminator. ReLU is the abbreviation of Rectified Linear Units activation function, defined by ReLU(()) max z). For generator network, we use a spatially discounted I reconstruction loss (Yu et al. 2018).
At the test step, we fix all trainable parameters in the I3AD generator. The I3AD generator ceives test images that are normal or anomalous images and adaptive mask matrices. Mask matrices constructed from the pixel-wise reconstruction errors between original images and generated images at the previous iteration step. Since mask matrices are dynamically updated and shrunken during iterations, the BAD generator generates masked regions intensively, leveraging the gradually increasing information of the surrounding unmasked regions.
2.3 INPAINTING MASK OF STRUCTURAL SIMILARITY (SSIM MASK)
In anomaly segmentation tasks, structural similarity measure (SSIM) index (aag et al. 2004) sharply measures the small anomalous change (Bergmann et al. 2018). Details of SSIM calculation in Appendix A
We propose the structural similarity mask (SSIM-Mask) to mask anomalous pixels during test iter- ations. SSIM-Mask M; is binary mask thresholding the pixel-wise SSIM Index between input image 2o and the reconstructed image i; at 5th iteration step, defined as
M; {
if ax(x) 2 otherwise,
(x) SSIM R:), 8 = G(
where u denotes the threshold level for binary classification. After N iterations, we use the Ntth SSIM anomaly score aN (x) for AD evaluation.
2.4 MASK INITIALIZATION
We have no mask information at the first iteration step during testing. We use four checkerboard matrices to initialize masks. F initialized mask examples. A generator encodes pixels of test images in white regions and decodes pixels in black boxes. These black regions are mutually exclusive between four masks and cover target images collectively. Therefore, combine four generated images into a single whole reconstruction.
2.5 ITERATION STEPS AND STOP CRITERIA
We expect masked region for normal images and some local masked region for anomalous im- ages. Therefore, applying iterative inpainting for some samples on the training dataset could esti- mate iteration steps enough to reduce masks normal pixels. Since IAAD decodes only masked regions, MH+1 will be almost all subset of M,. Therefore, we could set early stop criteria whether the difference between M, and M is small against masked regions of M,-
4

---

Under review as a conference paper at ICLR 2021
3 RESULTS
Table 1: Results for anomaly detection on MVTecAD dataset, expressed in the AUROC on sample-wise reconstruction errors for different autoencoders and datasets. We compare the vanilla L and SSIM autoencoders(Bergnnee Bergmaan et al. 2018) and their iterative projection method (Dehaene cet al., 2020) as baselines. Bold font is the best AUC in each category and lightblue background indicates the best method measured by the average AUC on I and SSIM anomaly score.
Bergmann et al. (2019)
Dehaene et al. (2020)
Ours
Autoencoders
Iterative projection
Inpainting
Model
L
DSAE
LAAE
DSAE
IADD
Score
L
SSIM
L
SSIM
L
SSIM
L
SSIM
L
SSIM
carpet
0.476
0.531
0.401
0.538
0.447
0.513
0.402
0.449
0.602
0.605
Textures
grid
0.947
0.764
0.849
0.665
0.947
0.936
0.891
0.840
0.998
0.947
leather
0.778
0.694
0.720
0.382
0.826
0.831
0.772
0.752
0.823
0.659
tile
0.763
0.706
0.533
0.380
0.810
0.815
0.684
0.662
0.978
0.983
wood
0.939
0.861
0.894
0.875
0.952
0.935
0.915
0.896
0.938
0.936
bottle
0.978
0.921
0.966
0.921
0.958
0.980
0.967
0.983
0.966
0.905
cable
0.758
0.827
0.751
0.807
0.564
0.725
0.630
0.717
0.767
0.773
capsule
0.728
0.689
0.738
0.699
0.792
0.803
0.819
0.813
0.708
0.731
hazelnut
0.902
0.841
0.898
0.845
0.956
0.942
0.957
0.912
0.930
0.895
Objects
metal nut
0.613
0.569
0.597
0.557
0.583
0.623
0.547
0.590
0.658
0.605
pill
0.825
0.838
0.780
0.818
0.741
0.746
0.721
0.725
0.783
0.780
screw
0.015
0,670
0.264
0.659
0.768
0.773
0.772
0.718
0.980
0.842
toothbrush
0.981
0.886
0.972
0.875
0.989
0.978
0.983
0.975
0.958
0.903
transistor
0.820
0.881
0.818
0.879
0.746
0.813
0.728
0.792
0.864
0.876
zipper
0.873
0.785
0.833
0.755
0.818
0.864
0.772
0.810
0.994
0.998
Texture
0.781
0.711
0.680
0.568
0.796
0.806
0.733
0.720
0.868
0.826
Ave.
Object
0.749
0.791
0.762
0.782
0.792
0.825
0.790
0.804
0.861
0.831
Total
0.760
0.764
0.734
0.710
0.793
0.818
0.771
0.776
0.863
0.829
3.1 EXPERIMENT CONFIGURATION
We performed experiments with the industrial high-resolution dataset named MVTee Anomaly De- tection (MVTecAD) (Bergmann et al., 2019). Please see Appendix B for further details on the dataset. [Bergmann et al. (2019) applies different subsampling and evaluations for each model and each category. Dehaene et al. (2020) resizes 128 128 pixels for objects and extract patches of 128 128 pixels from resized 512 x 512 pixels for textures. In contrast to these previous works, we resize all images to 256 x 256 pixels in 15 categories for training and testing, and evaluated all models by same experiment setting.
We train I3AD generator by 500,000 iterations with 10 images per batch. We two Adam optimizers (Kingma & Ba 2014) with learning rates of 0.0001 for the generator and 0.0004 for the discriminator, respectively. Random blushing stroke masks are generated by the algorithm proposed by Yu et al.] (2019). Regarding mask generation, we apply by 6 patched four checker- board matrices masks initialization. We find the threshold hyperparameter for generator to achieve low L1 reconstruction errors on training normal images. We apply the 13AD method on samples of training datasets with u from 0. 0.50 by 0.05. Then, we find minimal threshold to achieve small 7, reconstruction errors less than the order of 10 after 10 iteration steps. We apply 30 iterations for test images in all categories.
I3AD
Baselines
As baseline models, compare two autoencoders that are trained to minimize different reconstruction loss of L and SSIM. Similar to Bergmann et al. (2018). both autoencoder models are parameterized by the same architecture with latent space dimensionality (set to 100) and optimized
5

---

Under review as a conference paper at ICLR 2021
Normal
Anomalous
L-AE
IADD (Ours) L'AE-grad
Figure 3: First Normal samples of tile, carpet, transistor, screw, wood, and zipper in MVTecAD dataset; Second row: anomalous samples from the aforementioned dataset categories; Third row: Anomaly map by L autoencoder (Bergmann et al. 2019); Fourth row: our proposed anomaly map by our I3AD method. Ground truth is represented by red contour, and each estimated anomaly score highlighted by green.
by Adam with the same learning rate (set to 0.0001). These models are trained by 20,000 itera- tions on 10 images per batch. For the evaluation iterative projection method, both L and SSIM autoencoders applied have additional 50 iterations with step size 0.5 and L regulation weight À = 0.05, same parameter setting with Dehaene et al. (2020).
We compute the Area Under the Receiver Operating Characteristics (AUROC) get performance independent of the determined threshold for evaluation. The AUROC scores on the sample-wise L and SSIM reconstruction errors are computed for anomaly detection, and pixel-wise errors computed for anomaly localization. For the SSIM Index, their hyperparameters are selected as window size 11, = 7 performed on a graphics processing unit NVIDIA Quadro RTX 8000 based on system running Python 3.7, PyTorch library version 1.5, and CUDA 10.1.
same with Bergmann al. (2018). All experiments were
3.2 QUALITATIVE AND QUANTITATIVE RESULTS
Table shows the AUROC result for anomaly detection. Table 5 in Appendix E shows the AU- ROC result for anomaly localization. From both detection and localization, the baselines of vanilla L and SSIM autoencoders perform well for object datasets and challenge the high-frequency tex- tures such as carpet, leather, tile, or zipper. Despite different resolution from original experiments, Dehaene et al. (2020)'s 's approach works and improves AUROC of both localization and detection on many datasets. However, there remains room to improve in the categories where the base au- toencoders had weak result, especially in textures. In anomaly detection, I3AD outperforms baselines on nine categorics and has ave. +11.1% on textures, ave. +8.85% on objects and ave. +13.01% in total from vanilla autoencoders.
Figure shows the qualitative results baseline AE and our I3AD method. Figure 7 in Ap- pendix G shows the results of the remaining categories. SSIM anomaly score is used for highlight- ing. Our IAD has clearly a large improvement in high-frequency categories such as tile, carpet, and zipper.

---

Under review as a conference paper at ICLR 2021
3.3 HYPERPARAMETER SENSITIVITY
In this section, we investigate the hyperparameter sensitivity for the AD results. Compared to typical inpainting tasks to fill various types of images, the inpainting task for AD is relatively easy only to leam every single category. Thus, skip hyperparameters of the inpainting networks and focus the hyperparameters regarding mask generation.
Mask initialization We changed the patch size on four checkerboard matrices. Figure showed that different patch size initialization has performance differences in the first few steps while con- verging similar results after enough iterations. We verified that no mask initialization generates anomalous pixels directly and leads to a meaningless result.
(a]
(b)
(c)
0.80
0.80
0.88
0.86
0.86
0.86
0.84
0.82
0.82
0.82
L2: patch number
SSIM: patch number
0.78
indicator
0.78
0.78
0.76
SSIM
0.76
12
0.76
12
10
20
10
20
10
20
Iteration
iteration
iteration
Figure 4: AUC change of different patch-size mask initialization during iteration. Both I,2 and SSIM anomaly scores computed. (a) average AUC result of each anomaly score, (b) AUC result measured by 2 (c) AUC result measured by SSIM
Mask generation threshold We determine mask generation threshold u naively from the Lı construction levels on training datasets. Figure shows the AUC score on different threshold se- lection. Even though the best parameters unknown and inaccessible from training datasets, could find threshold parameter naively. Moreover, since this fitting does not perform equally across data categories, mask generation has to be optimized in semi-supervised approach.
1.0
0.9
0.7
mask generation
0.6
L1 loss 30.0
L1 lass 1.0
0.5
bottle
cable
capsule
carpet
grid
hazeinut
leather
metal, nut
Aill
screw
teathrussh
transistor
wood
Zlpper
Uile
Figure 5: AUC change of different threshold for mask generation
Iteration speed measure the iteration speed for AnoGAN, Iterative Projection, and our I3AD. Table 2 shows 13AD runs at compatible speed with Iterative Projection method and.
4 RELATED WORKS
We related works about anomaly detection in this section. Appendix H summarizes the architec- tures of reconstruction bassed approach in unsupervised AD. Appendix [ shortly summarizes related works about image inpainting.

---

Under review as a conference paper at ICLR 2021
models
time per image (seconds)
iterations
iteration
batch calculation
AnoGAN
63.4 + 13.4
500
back-prop
not available
Iterative Projection
0.70 + 0.11 0.58 0.01
50
back-prop
not available
I3AD
30
forward
available
Table 2: Comparison of iteration speed
Anomaly detection overview |Bergmann et al. (2019) introduced the MVTecAD dataset and con- ducted a thorough evaluation of traditional shallow models and recent state-of-the-ar deep neural networks for unsupervised AD and segmentation tasks. They showed the evaluated methods do not perform equally across data categories, and there is still room for improvement.
Autoencoders Convolutional autoencoders are commonly used as base architecture in unsu- pervised AD. Autoencoders are customized by various reconstruction and regularization losses. Bergmann et al. (2018) applies SSIM Index as reconstruction loss and anomaly map. ZZimeerer et alall (2019b) shows that loss gradients are useful anomaly scores to improve the AUC score on unsupervised pixel-wise tumor detection. Zimmerer al. (2019a) proposes Context-encoding VAE (ceVAE) for unsupervised AD, Context-encoding (Pathak et al. 2016) is special class of denoising autoencoders (Vincent al., 2010) learn inpainting random masks instead of additive Gaussian noise denoising.
Both context-encoding and our I3AD solve inpainting task during training. Motivation and usage are different during testing. Since context- -entext-nncodin is data augmentation for feature learning, model architecture used as vanilla autoencoders. On the other hand, our I3DD uses inpainting masks and encodes the unmasked regions during testing. The iterative process realizes conditional encoding and decoding to estimate unknown masks.
VAEs VAEs is class of autoencoders to constrain lower-dimensional latent space by Kullback- Leibler divergence, allowing us to conduct anomaly detection by the reconstruction errors and es- timated likelihood. On the other hand, Nalisnick et al. (2018); [Shafaei et al. (2018): [Hendrycks) et al. (2018) provide evidence that deep generative models might fail to assign a high likelihood for the out-of-distribution dataset. Several approaches such training auxiliary dataset of outliers (Hendrycks et al., 2018), training a background model (Ren et al., 2019), and Likelihood Regret score by fine-tuning test images Xiao al. (2020). are proposed mitigate this issue. Since VAEs tend to generate blur images, vanilla autoencoders are popular for local defect detection. Indeed, Matsubara et al. (2018) showed the KI. divergence term deteriorates the anomaly sensitivity and proposes to use only reconstruction term for anomaly map. Baur et al. (2018) uses VAEs for un- supervised anomaly segmentation in brain MR seans, and the improvement from autoencoders to VAEs was limited.
Adversarial training Adversarial training is one approach to generate high-resolution images. Schlegl et al. (2117) proposed AnoGAN, which utilizes GANs for unsupervised AD. We do not know a random noise to generate an anomalous-free image close to a targeted image. AnoGAN finds latent noise sample by iterative update under minimizing the reconstruction errors and the semantic similarity of a discriminator's outputs. Since there was a drawback of long runtime for this calculation, several studies had efforts to reduce its runtime. ADGAN (Deecke al.) 2018) updates the input images and the generator's parameters. Several studies propose to train an encoder network to find an expected random noise from an input image. Their architectures are similar to autoencoders or VAEs with an adversarial term (here referred to as "AEGAN" and "VAEGAN") (Larsen et al. 2016 Dumoulin et al. 2017, Donahue et al. 2017). Their difference is around train- ing processes and loss terms. Efficient-GGr GAN (Zenati et al. 2018) adopted ALI training (Dumoulin ctt al. 2017) that The discriminator receives the joint pair of latent features and images. GANomaly (Akaa et al.) 2018) combines three losses about L1 loss between images, L2 encoder loss between latent features, and adversarial loss. f-AnoGAN (Schlegl et al.) 2019) trains an encoder and a gen- erator separately. They examined three architectures regarding loss terms and showed that "izif" architecture best performed of them. Venkataramanan et al.) added attention expansion loss for unsupervised and weakly-supervised AD.
8

---

Under review as a conference paper at ICLR 2021
Skip-connection Sxip-connection is another approach to model small details. U-Net (Ron- neberger et al.] 2013) is widely applied for cardiac MR, brain tumors, and abdominal CT in su- pervised segmentation tasks. Akçay al. (2019); Sabokrou et al. (2018) replace autoencoder by U-Net with AEGAN for unsupervised anomaly detection. Skip-GAANmmll (Akçay et al. 2019)
inherits GANomaly and uses the same loss terms and anomaly scores. AVID (Sabokrou al. 2018)
is trained as same as vanilla AEGANs. It applies Fully convolutional neural networks (FCNs) for discriminator capture the regional information and define each region's 's regularity likelihood.
Oktay et al. (2018) added attention modules into U-Net to be specific to local regions and tested 3D
abdominal CT scans.
We tested U-Net architecture for the MVTecAD dataset and verified several models by switching U-Net's four skip-connections on or off. As a result, We faced learning trivial identity function, and anomalous pixels are reconstructed if models have first or second skip-connection close to input images. I3AD considered as an application of per-pixel skip-connections that only transport high convinced regions as normal pixels from anomaly score.
Iterative methods In addition to the aforementioned iterative methods such as Likelihood Regret, AnoGAN, and ADGAN, Dehaene et al. (2020) proposed the iterative projection method on trained autoencoders make blurred images clear testing time. It iteratively updates an input sample's pixels to minimize reconstruction errors under the constraint on the distance from the original image.
Dehaene et al. 2020)'s 's method has underlying autoencoders that encode and decode the whole image with back proppaaatot update. Our I3AD has an underlying of conditional autoencoders with forwarding iterations. Thus, our approach efficiently updates high-resolution pixels and avoids reconstructing complex patterns in background normal pixels.
summarize where differentiates previous AD methods and our proposed method below.
Though aforementioned autoencoder-base methods encode and decode whole images, 13AD encodes unmasked regions and decodes masked regions.
Skip connection passing whole images potentially leads to trivial function. 13AD could learn per-pixel skip-connections where connects local regions the model assigns a high possibility normal pixels.
Previous iterative methods used to minimize total reconstruction errors. I3DD targets to minimize the reconstruction errors on normal pixels and maximize ones on anomalous pixels. attempts to fill in the gap of train and test objectives on unsupervised AD.
Previous iterative methods require back-propagation with long iteration steps. Ours only uses forward iteration and efficiently updates high-resolution pixels.
DISCUSSIONS AND FUTURE WORKS
As future works, our I3AD should leverage GAN's unsupervised AD technique. For example, like AVID, PatchGAN discriminator score could be useful to generate iterative masks. 13AD could be tested on different inpainting models. The coarse-to-fine network with an attention module has a remarkable performance, requiring large network structure. To reduce the model parameters for speeding up the inference and saving hardware costs, Sagong et al. (2019) proposes weight sharing on coarse and fine generator networks while maintaining performance.
CONCLUSION
In high-resolution images, autoencoders fail to model small details, which yields blurry image re- constructions. This is especially the case for high-frequency textures, such as carpet, leather, and tile. To tackle this issue, propose the iterative image inpainting method reconstruct partial re- gions adaptively. Our method utilizes the structural similarity measure for inpainting regions, which modifies their structural difference and enhances their reconstruction quality iteration steps. Our method outperforms state-of-the-art results on several categories in MVTecAD datasets and shows an especially large improvement in the texture categories.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Samet Akcay, Amir Atapour-Abarghouel and Toby P Breckon. Ganomaly: Semi-supervised anomaly detection via adversarial training. In Asian conference on computer vision, pp. 622- 637. Springer, 2018.
Samet Akçay, Amir Atapour-Abarhhoci. and Toby P Breckon. Skip-ganomaly: Skip connected and adversarially trained encoder-decoder anomaly detection. In 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2019.
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 2016.
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruc- tion probability. Special Lecture on IE, 2(1):1-18, 2015.
Christoph Baur, Benedikt Wiestler, Shadi Albarqouni, and Nassir Navab. Deep autoencoding models for unsupervised anomaly segmentation in brain mr images. In International MICCAI Brainlesion Workshop, pp. 161-169. Springer, 2018.
Paul Bergmann, Sindy Löwe, Michael Fauser, David Sattlegger, and Carsten Steger. Improving un- supervised defect segmentation by applying structural similarity to autoencoders. arXiv preprint arXiv:1807.0200 2018.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mytec ad comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9592-9600, 2019.
Lucas Deecke, Robert Vandermeulen, Lukas Ruff, Stephan Mandt, and Marius Kloft. Image anomaly detection with generative adversarial networks. In Joint european conference on ma- chine learning and knowledge discovery in databases, pp. 3-17. Springer, 2018.
David Dehaene, Oriel Frigo, Sébastien Combrexelle, and Pierre Eline. Iterative meergy-based pro- jection on normal data manifold for anomaly localization. International Conference on Learning Representations, 2020.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. International Conference Learning Representations (ICLR 2017), 2017.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropictro, Alex Lamb, Martin Ar- jovsky, and Aaron Courville. Adversarially learned inference. International Conference Learning Representations (ICLR 2017), 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Lewis D Griffin, Matthew Caldwell, Jerone TA Andrews, and Helene Bohler. "unexpected item in the bagging area": Anomaly detection in x-ray security images. IEEE Transactions on Informa- fion Forensics and Security, 14(6):1539-1553 2018.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint
2018.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-5077 2006.
Satoshi lizuka, Edgar Simo- Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions Graphics (ToG), 36(4):1-14, 2017.
10

---

Under review as a conference paper at ICLR 2021
Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to -mmge translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and patter recognition, pp. 1125-1134, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther, Autoen- coding beyond pixels using a learned similarity metric. In International conference on machine learning, pp. 1558-1566, PMLR, 2016.
Guilin Liu, Fitsum Reda, Kevin Shih, Ting-Chur Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 85-100, 2018.
Takashi Matsubara, Ryosuke Tachibana, and Kuniaki Uehara. Anomaly machine component detec- tion by deep generative model with unregularized score. In 2018 International Joinl Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXi::880..055 2018.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint arXiv: 2018.
Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. Medical Imaging with Deep Learning (MIDL 2018), 2018.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2536-2544, 2016.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems, pp. 14680-14691, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv: 2014.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Mohammad Sabokrou, Masoud Pourreza, Mohsen Fayyaz, Rahim Entezari, Mahmood Fathy, Jürgen Gall, and Ehsan Adeli. Avid: Adversarial visual irregularity detection. In Asian Conference on Computer Vision, pp. 488-505 Springer, 2018.
Min-cheol Sagong, Yong-gov goo Shin, Seung-wood wook Kim, Scung Park, and Sung-jea Ko. Pepsi: Fast image inpainting with parallel decoding network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11360-11368, 2019.
Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Sehnidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International conference on information processing in medical imaging, pp. 146- 157. Springer, 2017.
Thomas Schlegl, Philipp Seeböck. Sebastian M Waldstein, Georg Langs, and Ursula Schmidt- Erfurth. f-anogan: Fast unsupervised anomaly detection with generative adversarial networks. Medical image analysis, 54:30 44, 2019.
11

---

Under review as a conference paper at ICLR 2021
Alireza Shafaci, Mark Schmidt, and James J Little. A less biased evaluation of out-of-distribution sample detectors. arXiv preprint 1809.0:800.4400 2018.
Shashanka Venkataramanan, Kuan-Chuan Peng, Rajat Vikram Singh, and Abhijit Mahalanobis. At- tention guided anomaly detection and localization in images. Computer Vision ECCV 2020.
Lecture Notes in Computer Science, vol /2362.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 1 11(12), 2010.
Dongyi Wang, Robert Vinson, Maxwell Holmes, Gary Seibel, Avital Bechar, Shimon Nof, and Yang Tao. Early detection of tomato spotted wilt virus by hyperspectral imaging and outlier removal auxiliary classifier generative adversarial nets (or-ac-gan). Scientific reports, 9(1):1-14, 2019.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. 34th Conference on Neural Information Processing Systems (NeurIPS 2020), 2020.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505-5514, 2018.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE International Conference on Com- puter Vision, pp. 4471 14800, 2019.
Houssam Zenati, Chuan Sheng Foo, Bruno Lecouat, Gaurav Manek, and Vijay Ramaseshan Chan- drasekhar. Efficient banebased anomaly detection. arXiv preprint arXiv: 2018.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint arXiv:1805.083/6 2018.
David Zimmerer, Fabian Isensee, Jens Petersen, Simon Kohl, and Klaus Maier-Hein. Unsupervised anomaly localization using variational auto-encoders. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 289-297. Springer, 2019a.
David Zimmerer, Jens Petersen, Simon AA Kohl, and Klaus Maier-Hein. A case for the score: Identifying image anomalies using variational autoencoder gradients. arXiv preprint 19X10:19/2.00003,, 2019b.
12

---

Under review as a conference paper at ICLR 2021
A ANOMALY SCORE OF STRUCTURAL SIMILARITY (SSIM INDEX)
For AD, some per-pixel error measure such as L'-distance utilized as anomaly score. The struc- tural similarity (SSIM) metric (Wang et al. 2004) could be also employed to capture perceptual similarity (Bergmann et al. 2018; 2019). The SSIM Index defines a structural similarity measure between two K x K image patches p and q, taking into account their similarity in luminance [(p, q), contrast c(p, q). and structure spp q):
SSIM(p,9) lp. c(p s(p.
where a, 7 E R are hyperparameters. From the patches' mean intensities Up, Myg the patches' variances 0p: Sg. and their covariance pqq the three measures are defined as
2pppa +CL
20,0g 0
20 pa + C2
1(p q) 10 + +2 +
c(p.g)
s(p, q)
o3 o + 02
20,0g + 02
The constants Cj and C2 ensure numerical stability and are typically set to c₁ 0.01 and Co 0.03. By substituting three measures under equally weighting = 3 by
1, the SSIM Index is derived
(2141/4 ci (20 pq c2)
SSIM(p,
(13 + p + c1) o + c2)
We have that Sp 9) E [-1, and SSIM(p, q) if and only if p and q are identical.
B MVTEC ANOMALY DETECTION DATASET
We performed experiments with the industrial dataset named MVTec Anomaly Detection (MVTecAD) (Bergmann et (019). It contains 5,354 high-resolution color images of ten different objects and five texture categories. All image resolutions are set in the range between 700 x 700 and 1024 x 1024 pixels. It splits 3,629 images for training and validation and 1,725 images for testing. At test time, there are 467 defect-free images and 1,258 defect images. This dataset includes 73 different defect types, such as defects on the object's surface, structural defects, or the absence of certain parts. These anomalies are manually generated to produce realistic anomalies as they would occur real-world industrial inspection scenarios. |Bergmann al. (2019) use several resolution for objects and textures on each model. As AnoGAN, both training and testing images are zoomed to 128 128 pixels for object categories. For textures, 128 128 patches are extracted from zoomed 512 x 512 pixels. As L2 and SSIM autoencoders, they reconstruct patches of 128 128 pixels for textures and of 256 x 256 pixels for objects.
13

---

Under review as a conference paper at ICLR 2021
C ANOMALY LOCALIZATION
Table 3: Results for anomaly localization on the MVTecAD dataset, expressed in the AUROC pixel-wise reconstruction errors for different autoencoders and datasets. Same anomaly detection, we compare the vanilla I and SSIM autoencoders(BBrrgmann et al. 2018) and their iterative projection method (Dehaene et al. 2020) baselines. Bold font the best AUC in each category and lightblue background indicates the best method measured by the average AUC on L and SSIM anomaly score.
Bergmann al. (2019)
Dehaene al. (2020)
Ours Inpainting
Autoencoders
Itrattive projection
Model
L'AE
DSAE
LEAE
DSAE
I3AD
Score
L
SSIM
L
SSIM
L
SSIM
L
SSIM
L
SSIM
carpet
0.598
0.736
0.579
0.733
0.596
0.768
0.581
0.750
0.809
0.850
Textures
grid
0.741
0.817
0.719
0.815
0.719
0.859
0.705
0.841
0.959
0.987
leather
0.739
0.758
0.775
0.730
0.735
0.812
0.779
0.923
0.833
0.938
tile
0.513
0.513
0.564
0.509
0.529
0.581
0,566
0.603
0.630
0.788
wood
0.650
0.652
0.647
0.661
0.664
0.767
0.656
0.740
0.665
0.776
bottle
0.886
0,924
0.883
0.931
0.884
0.930
0.875
0.926
0.923
0.950
cable
0.773
0.779
0.800
0.793
0.740
0.822
0.710
0.784
0.819
0.795
capsule
0.839
0.919
0.815
0.898
0.843
0.914
0.819
0.909
0.733
0.854
hazelnut
0.934
0.971
0.929
0.967
0.911
0.949
0.893
0.928
0.664
0.756
Objects
metal nut
0.857
0.855
0.854
0.850
0.861
0.898
0.860
0.894
0.515
0.526
pill
0.871
0.911
0.859
0.917
0.881
0.916
0.866
0.916
0.649
0.725
screw
0.824
0.974
0.870
0.974
0.834
0.975
0.877
0.975
0.852
0.959
toothbrush
0.935
0.970
0.935
0.968
0.941
0.978
0.940
0.976
0.946
0.969
transistor
0.817
0.867
0.802
0.862
0.831
0.875
0.852
0.889
0.596
0.651
zipper
0.739
0.741
0.739
0.750
0.727
0.860
0.726
0.857
0.854
0.962
textures
0.648
0.695
0.657
0.690
0.649
0.757
0.657
0.771
0.779
0.868
Ave.
objects
0.847
0.891
0.849
0.891
0.845
0.912
0.842
0.905
0.755
0.815
total
0.781
0.826
0.785
0.824
0.780
0.860
0.780
0.861
0.763
0.832
14

---

Under review as a conference paper at ICLR 2021
D ADDITIONAL COMPARISON BETWEEN GAN MODELS (1)
Table 4: Results for anomaly detection on the MVTecAD dataset, expressed the AUROC on sample-wise reconstruction errors for different autoencoders and datasets. We compared AnoGAN (Deecke et al. 2018). f-AnoGAN (Schlegl et al. 2019), and AEGAN that is the base architecture for several models Zenati et al. 2018 Akcay et al. 2018) Bold font is the best AUC each category and lightblue background indicates the best method measured by the average AUC on L and SSIM anomaly score. We also test L Lp anomaly scores that combines reconstruction errors and discriminator features as mentioned Deecke et (2088)
Model
AnoGAN T,2
f-AnoGAN SSIM 1,2 + L.
AEGAN
I3AD
Score
SSIM
1,2
I?
SSIM 1,2 + L.D
T,2
SSIM
carpet
0.360 0.399
0.311
0.512
0.433
0.488
0.565
0.132
0.602
0.605
Textures
grid
0.661 0.517
0.451
0.733
0.236
0.937
0.826
0.942
0.998
0.947
leather
0.393 0.298
0.385
0.271
0.288
0.648
0.290
0.624
0.823
0.659
tile
0.627 0.476
0.724
0.514
0.735
0.540
0.410
0.591
0.978
0.983
wood
0.820 0.779
0.849 0.940
0.905
0.933
0.875
0.930
0.938
0.936
bottle
0.839 0.722
0.829
0.690
0.294
0.965
0.910
0.892
0.966
0.905
cable
0.660 0.636
0.611
0.623
0.565
0.785
0.849
0.704
0.767
0.773
capsule
0,491 0.538
0.548
0.608
0.205
0.557
0.566
0,470
0.708
0.731
hazelnut
0.632 0.617
0.579
0.684
0.436
0.761
0.739
0.507
0.930
0.895
Objects
metal nut
0,460 0.478
0.354
0.429
0.467
0.634
0.653
0.572
0.658
0.605
pill
0.549 0.543
0.756
0.729
0.690
0.700
0.642
0.485
0.783
0.780
screw
0.227 0.415
0.226
0.496
0,419
0.142
0.192
0.041
0.980
0.842
toothbrush
0.708 0.775
0.622
0.644
0.550
0.981
0.931
0.461
0.958
0.903
transistor
0.641 0.623
0.866
0.901
0.897
0.866
0.901
0.773
0.864
0.876
zipper
0,587 0,506
0.695
0.566
0.291
0.901
0.780
0.875
0.994
0.998
textures
0.572 0.494
0.562
0.576
0.519
0.709
0.593
0.644
0.868
0.826
Ave.
objects
0.580
0.585
0.609
0.637
0.481
0.729
0.716
0.578
0.861
0.831
total
0.577 0.555
0.593 0.617
0.494
0.722
0.675
0.600
0.863
0.829
We compared GAN based anomaly detection and localization models. We use same network ar- chitecture for our I3AD model and there is only difference that input images masked or not. AnoGAN takes 500 iterations at test time to find latent random noise. f-AnoGAN has two training steps and encoder is optimized after training the generator.
15

---

Under review as a conference paper at ICLR 2021
E ADDITIONAL COMPARISON BETWEEN GAN MODELS (2)
Table Results for anomaly localization on the MVTecAD dataset, expressed in the AUROC pixel-wise reconstruction errors for different autoencoders and datasets. We compared AnoGAN (Deecke et 2018). f-AnoGAN (Schlegl al.
2019 and AEGAN that is the base architecture for several models (Zenati et al. 2018 Akcay et al. 2018). Bold font is the best AUC in each category and lightblue background indicates the best method measured by the average AUC on L and SSIM anomaly score.
Model
AnoGAN L²
f-AnoGAN L
AEGAN L
I3DD
Score
SSIM
SSIM
SSIM
L
SSIM
carpet
0.553
0.576
0.487
0.504
0.571
0.695
0.809
0.850
Textures
grid
0.537
0.548
0.561
0.532
0.777
0.863
0.959
0.987
leather
0.624
0.712
0.557
0.721
0.798
0.754
0.833
0.938
tile
0.544
0.508
0.536
0.513
0.539
0.495
0.630
0.788
wood
0.598
0.629
0.657
0.655
0.627
0.617
0.665
0.776
bottle
0.775
0.796
0.773
0.772
0.880
0.922
0.923
0.950
cable
0.711
0.754
0.684
0.723
0.741
0.739
0.819
0.795
capsule
0.778
0.859
0.740
0.869
0.857
0.913
0.733
0.854
hazelnut
0.885
0.946
0.849
0.955
0.884
0.955
0.664
0.756
Objects
metal nut
0.747
0.733
0.738
0.749
0.849
0.854
0.515
0.526
pill
0.805
0.883
0.692
0.752
0.872
0.922
0.649
0.725
screw
0.789
0.925
0.786
0.898
0.857
0.947
0.852
0.959
toothbrush
0.873
0.931
0.881
0.925
0.925
0.956
0.946
0.969
transistor
0.704
0.768
0.814
0.843
0.814
0.843
0.596
0.651
zipper
0.710
0.734
0.702
0.727
0.734
0.743
0.854
0.962
textures
0.571
0.594
0.560
0.585
0.662
0.685
0.779
0.868
Ave.
objects
0.778
0.833
0.766
0.821
0.841
0.879
0.755
0.815
total
0.709
0.753
0.697
0.742
0.782
0.815
0.763
0.832
16

---

Under review as a conference paper at ICLR 2021
F MASK INITIALIZATION
2 by 2 patched masks
4 by 4 patched masks
6 by 6 patched masks
Figure 6: Mask initialization in test iteration step. Images masked by four masks that includes by X patched boxes.
A conditional generator in I3AD encodes unmasked pixels and decodes masked pixels. More spe- cially, it decodes whole pixels and uses spatial discounted reconstruction loss to learn to decode only for the masked region. Therefore
as conditional-encoding autoencoder Pathak et al, (2016) During test iterations, the generative model needs the initialized mask at the first iteration step. To cover whole images and segment few batches, we split images into by X patches and aggregated them into four masks.
17

---

Under review as a conference paper at ICLR 2021
G
QUALITATIVE RESULT
Normal
Anomalous
LAAE
L-AE-grad
I3AD (Ours)
gOs
Faseh
Fales
Giben
Figure 7: Other qualitative results
18

---

Under review as a conference paper at ICLR 2021
H
ARCHITECTURE COMPARISON
Base Architecture
Model Name
AutoEncoder
Image
SSIM-AutoEncoder
DAE
Encoder
Decoder
VAE
Gradient anomaly scores*
ceVAE coontex-enncoiin)
Generator (Decoder)
Discri- minator
Z
AnoGAN
Backprop Iteration
Generator (Decoder)
ADGAN
minator
Backprop Iteration
Image
U-Net
Attention U-Net
Image
AVID Skip-GANomaly
U-Net
minator
Image
(Z)
AEGAN / VAEGAN
AnoVAEGAN
Encoder
Generator (Decader)
Discri- minator
f-AnoGAN
Efficient-GAN (ALI)
GANomaly CAVAE
Image
(Z)
Iterative projection
Backprop Iteration
Image
Encoder
Decoder
Likelihood Regret
Backprop Iteration
Masked Image
I3AD (Ours)
Conditional Encoder
Conditional Decoder
Discri- minator
Forward Iteration
Referred to these names in this paper
Figure 8: Architecture comparison
19

---

Under review as a conference paper at ICLR 2021
We summarize the base model architecture used in recent reconstruction based approach. This is not an all-inclusive list and several studies proposed additional losses and modules with different anomaly score measures in the above base architectures.
20

---

Under review as a conference paper at ICLR 2021
I IMAGE INPAINTING RELATED WORKS
Image inpainting (a.k.a. image completion or image filling) is one of the Image-to-Image trans- lation tasks cover damaged, deteriorating, missing parts from remaining parts alternative contents guided by users. In computer vision, recently, there is much research about the approach based on feed-forwart generative models with convolutional networks, especially using conditional generative adversarial networks (cGANs)(Isola et al. 2017).
lizuka al. (2017) proposed the model with global and local consistency to handle high-resolution images. To handle irregular masks, Liu et al. (2018) proposed a partial convolution layer that the weights in the convolutional layers are re-normalized by the number of valid pixels from the mask matrix. To produce higher-quality image inpainting, et al. (2018) proposed two-stage architecture with coarse and refinement networks and a contextual attention module refinement network. The contextual attention module captures long-range spatial dependencies between masked regions and their surroundings, allowing models to fill large rectangular masks. To handle free-form masks, YYu et al. 2019) proposed the gated convolution that extends the partial convolution and works pixel normalization and mask updates as trainable weights in neural network layers.
J ANALOGY OF SSIM-MASK WITH PARTIAL CONVOLUTION LAYER
In inpainting task, the convolutional layer in a generator network assigns the weight balanced on both valid pixels in unmasked region and invalid pixels in masked region. (Liu et al. 2018) proposed partial convolution layer to adapt an irregular shaped mask and re-normalize the dependence of valid pixels. It is computed by the following equations:
W (X
M
sum (M)
0 =
Sum(M)
otherwise,
where let W be the convolution filter weights for the convolution layer, X be the feature values for the sliding window, M be the corresponding binary mask, and Oz.g be the output features. The corresponding binary mask M has also the mleebbsed update in each layer following by
{ 1
sum(M') >
M
otherwise.
Our I3AD passes through the forward model iteratively, which is interpreted a very deep gener- ative model. Our SSIM-Mask corresponds to kind of partial convolutions, which extends to the non-linear structural similarity rule.
21


Reviews:
[{"review_id": "kB1MEM7vjsc", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for contrastive anomaly detection (AD) using an iterative masked conditional autoencoder inpainting approach. An autoencoder network is trained using an adversarial approach to reconstruct a randomly masked part of the input image. At test time a mask is derived from the generated anomaly map (using SSIM index between the input and reconstructed image) and used to mask the input so that the process is repeated N times. The method is shown to produce SOT results on the MVTec AD benchmark.\n \n\n PROS:\n \n  * The method is novel and interesting. The iterative masked approach does provide a principled way to increase the signal-to-noise ratio in the reconstructed images.\n  \n \n CONS:\n \n  * The numbers given in table 1 for the baseline SOT methods are not from the literature. The authors reproduced the numbers but they appear to be much lower than those published. For example, the published value for the average AUC over 15 classes in Bergmann-2019 are L2=0.82;SSIM=0.86, while on table 1, these values are: L2=0.76;SSIM=0.76. This is a substantial difference, which is not explained at all. It makes the proposed method look much better than SOT, while in reality it appears to have similar performance.\n \n  * The number of iterations is not discussed. This is an important hyperparameter as it affects the overall speed of the approach. Only fig 4 shows the progression of the AUC during iterations but no discussion are provided. Also the actual number of iterations used to generate table 1 AUC scores is not given. Assuming the complexity is higher than other SOT methods due to the iterative aspect, and given that the performance is similar to SOT (see above point), this approach now looks a lot less convincing.\n  \n  * The paper is not well written and suffers from grammatically wrong and overly complicated sentences, to the point where it becomes distracting and confusing.\n \n \n Overall, this is an interesting and novel approach to contrastive anomaly detection. However, I think the paper is not ready for publication. The baseline SOT numbers need to be explained or changed to reflect previously published numbers. The computational complexity of the approach needs to be compared to these baselines. And finally the paper needs to be seriously proof-read.\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "j1qiOvZkjb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposed a novel learning strategy for unsupervisedly anomaly detection. Particularly, authors propose to use an iterative mask generation process based on image impainting and reduction of a structural similarity metric (SSMI) between the input image and its reconstructed version. For evaluation purposes, authors resort to the public MVTec benchmark, showing better results than the baselines. Please find me comments below:\n\nStrengths\n-The paper is generally well written and easy to follow.\n-Results show that the proposed method outperforms the baselines.\n\nWeaknesses.\n-The methodological contribution is marginal/incremental. Similarly to several methods in weakly supervised segmentation (see [1] for example) authors use iterative steps to refine the initial segmentation mask. The only difference is that instead of mine regions based on classification activation maps and classification scores the authors use a structural similarity pixel-wise metric. Beyond that there is nothing novel in the proposed methodology.\n\n-Some ideas/motivations are unclear. For example, I don’t really understand why the mask initialization is needed at test time. Further, authors also mention in the Appendix. D that to cover whole images, they are split into X by X patches and then aggregated into four masks. Please provide more details since this is unclear. \n\n-Literature is poorly conducted with many relevant recent papers missing (furthermore, the literature comes late in the paper). In addition to the previous paper in weakly supervised segmentation, the works in [2-6] are recent works in anomaly detection, just to name a few. Authors should include all this papers, discuss their limitations and show how the proposed work can overcome these drawbacks. Authors mention that auto-encoders produce blurry images, which is true. Nevertheless, works including an adversarial discriminator have somehow addressed the issue of blurred reconstructed images. Given all this, authors should better motivate their work. \n\n-Among previous missing papers, there is the work in [2], which also employs an impainting strategy coupled with an adversarial model. Which are the differences with respect to this work?\n\n-Experiments need to be significantly extended. First, authors merely include two methods in their evaluation, while there exist more than those used in the comparisons. This is particularly important since some recent methods which have been omitted in the literature review made by the authors (e.g., [4]) significantly outperform the proposed method (0.863 vs 0.90 in AUC). Second, authors report results in terms of AUC, while I strongly suggest that they use the accuracy for individual classes, and the AUC as average of the classes. The reason for this is to better compare to related work (See for example Table 6 in [4]).\n\n\nReferences:\n[1] Wang et al. Weakly-supervised semantic segmentation by iteratively mining common object features. CVPR 2018.\n[2] Sabokrou et al. Avid: Adversarial visual irregularity detection. ACCV 2018 \n[3] Perera et al. Ocgan: One-class novelty detection using gans with constrained latent representations. CVPR 2019.\n[4] Venkataramananet al. Attention Guided Anomaly Localization in Images. ECCV 2020.\n[5] Deecke et al. Image anomaly detection with generative adversarial networks. In Joint european conference on machine learning and knowledge discovery in databases 2018.\n[6] Li et al. Exploring deep anomaly detection methods based on capsule net. In Canadian Conference on Artificial Intelligence 2020\n\n\nMinor: The paper in David Dehaene, Oriel Frigo, S´ebastien Combrexelle, and Pierre Eline. Iterative energy-based projection on a normal data manifold for anomaly localization. arXiv preprint arXiv:2002.03734 is an ICML 2020 published paper,\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "SOKoPizrwwH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an impainting-based method for anomaly localization on images. In the training time, a conditional GAN-based generative modeling approach is adopted. In the test time,  a mask matrix is adaptively estimated by thresholding the structural similarity index measure (SSIM) between the original images and reconstructed images. The idea is very intuitive and experiments demonstrate improved performance (especially on textures) over two recent baselines methods.  \n\nStrong points: \n\n- This paper improves the test time in unsupervised anomaly detection with an iteration scheme. Although similar ideas of iteratively focusing on refining the low-error regions and leaving alone the high-error regions have been proposed before, the residual-thresholding scheme proposed in this paper looks even simpler. \n\n- Another novelty is in the training time, the proposed approach using a conditional and GAN-based approach. This is in contrast with autoencoder-based reconstruction approaches which may lead to blurry image reconstruction. \n\nWeak points: \n\n-  The experiment evaluation is not quite convincing. Some of these numbers look worse than what is reported by Bergmann et al., 2018 and Dehaene et al., 2020. Are there any settings different when reproducing the baseline results?  \n\n- The qualitative comparison with Dehaene et al. 2020 is missing in Figure 3. \n\n- Figure 4 clearly shows the AUC improves with iterations. Would this SSIM-threshold scheme also benefit other approaches (e.g., a conditional autoencoder)? Are these performance improvements over baselines mainly attributed to the choices of free-form random mask and GAN, or the proposed SSIM-threshold scheme? Would it be better than other iterative schemes such as iterative projection with the same trained model? More ablation studies would be helpful. \n\nOther points to clarify: \n\n- I guess Table 1 AUROC is for pixel-wise segmentation rather than image classification, right?\n\n- How does the checkboard mask initialize the test? Applying different masks to 4 copies of images and then merge? \n\n- Stopping rule of iteration \n\nOverall, I think the idea in this paper is very interesting. I still have some conservations about experimental evaluation (as explained above) against me voting for acceptance. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "R457kGhcb8V", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": " This approach relies on learning merely normal training sample and then\ndistinguishing between normal and abnormal events regarding the threshold of the reconstruction error.\nI3AD exploits a combination of per-pixel identity function and conditional auto-encoder which is\ncapable of encoding only normal regions and decoding just anomalous regions.\nWeaknesses:\n-Although this paper has attempted to propose an efficient algorithm for anomaly detection, the\nthe general idea of this paper is not sufficiently innovative. \n-The most important works on the topics are not cited. \n-The proposed method is not comprehensively compared with the other solution.\n-A comprehensive analysis of the complexity of the proposed method is required. It seems the proposed method achieves the state-of-the-art performed at the expense of complexity. \n\nStrengths:\nThis paper is well written so that the structure is easy to follow. The majority of the obtained results\nare better than the state-of-the-art has been reported in this paper. The proposed method has tried\nto tackle two noticeable issues of using auto-encoders for the task of anomaly detection including\ntheir defect inappropriately modeling small detail as well as the existing object mismatching that\nthe models are trained to minimize total reconstruction errors\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The initial reviews were a bit split. R4 was slightly positive, R3 was slightly negative, and both R1 and R2 voted for rejection. The main issue was lack of proper comparisons with the SOTA methods and missing references. In the rebuttal, the authors added additional experiments as requested, but R1 and R2 were not convinced by the new results. In particular, R1 pointed out that even the unsupervised setup in [4] achieved 0.89 AUC, outperforming 0.86 as reported in the paper. The AC agrees with R1 and R2 that the paper cannot be published until more thorough comparisons are conducted.

Decision: Reject