RGI: robust GAN-inversion for mask-free image inpainting and unsupervised pixel-wise anomaly detection
Authors: Shancong Mou
Xiaoyi Gu
Meng Cao
Haoping Bai
Ping Huang
Jiulong Shan
Jianjun Shi
Venue: ICLR 2023 poster (2023)
Keywords: Robust GAN-inversion
Mask-free Semantic Inpainting
Unsupervised Pixel-wise Anomaly Detection

Abstract: Generative adversarial networks (GANs), trained on a large-scale image dataset, can be a good approximator of the natural image manifold. GAN-inversion, using a pre-trained generator as a deep generative prior, is a promising tool for image restoration under corruptions. However, the performance of GAN-inversion can be limited by a lack of robustness to unknown gross corruptions, i.e., the restored image might easily deviate from the ground truth. In this paper, we propose a Robust GAN-inversion (RGI) method with a provable robustness guarantee to achieve image restoration under unknown \textit{gross} corruptions, where a small fraction of pixels are completely corrupted. Under mild assumptions, we show that the restored image and the identified corrupted region mask converge asymptotically to the ground truth. Moreover, we extend RGI to Relaxed-RGI (R-RGI) for generator fine-tuning to mitigate the gap between the GAN learned manifold and the true image manifold while avoiding trivial overfitting to the corrupted input image, which further improves the image restoration and corrupted region mask identification performance. The proposed RGI/R-RGI method unifies two important applications with state-of-the-art (SOTA) performance: (i) mask-free semantic inpainting, where the corruptions are unknown missing regions, the restored background can be used to restore the missing content. (ii) unsupervised pixel-wise anomaly detection, where the corruptions are unknown anomalous regions, the retrieved mask can be used as the anomalous region’s segmentation mask.

Published as a conference paper at ICLR 2023
RGI: ROBUST GAN-INVERSION FOR MASK-FREE IM- AGE INPAINTING AND UNSUPERVISED PIXEL-WISE
ANOMALY DETECTION
Shancong Mou", Xiaoyi Gu', Meng Cao', Haoping Bai', Ping Huang', Jiulong Shan', Jianjun Shi'
* Georgia Institute of Technology. AApple
(shancong mou, xi soyy! gu)egatech. edu, jianjun. shi @ mengcao, haoping bai, huang ping, @ishan]eapple com
ABSTRACT
Generative adversarial networks (GANs), trained on large-scale image dataset, can be a good approximator of the natural image manifold. GAN-inversion, using pre-trained generator as deep generative prior, is promising tool for im- age restoration under corruptions. However, the performance of GAN-inversion can be limited by lack of robustness unknown gross corruptions, i.e., the re- stored image might easily deviate from the ground truth. In this paper, we propose Robust GAN-inversion (RGI) method with provable robustness guarantee achieve image restoration under unknown gross corruptions, where a small frac- tion of pixels are completely corrupted. Under mild assumptions, we show that the restored image and the identified corrupted region mask converge asymptoti- cally the ground truth. Moreover, we extend RGI to Relaxed-RGi (R (R-RGI) for generator fine-tuning to mitigate the gap between the GAN learned manifold and the true image manifold while avoiding trivial overfitting the corrupted input image, which further improves the image restoration and corrupted region mask identification performance. The proposed RGI/R-RGI method unifies two impor- tant applications with ofte--of-thh-ar (SOTA) performance: (i) mask-free seman- tic impainting, where the corruptions are unknown missing regions, the restored background can be used to restore the missing content. (ii) unsupervised pixel- wise anomaly detection, where the corruptions are unknown anomalous regions, the retrieved mask can be used the anomalous region's 's segmentation mask.
INTRODUCTION
When trained large-scale natural image datasets, GAN (Goodfellow al. 2020) is good ap- proximator of the underlying true image manifold. It captures rich knowledge of natural images and can serve as image prior. Recently, utilizing the learned prior through GANS shows impressive
results in various tasks, including the image restoration (Yeh et al.] 2017 Pan et al. 2021 Gu et al., 2020), unsupervised anomaly detection (Schlegl et al., 2017 Xia al.
(022b) and on. In those applications, GAN learns a deep generative image prior (DGP) to approximate the underlying true image manifold. Then, for any input image, GAN-inversion (Zhu et al., 2016 used search for the nearest image on the learned manifold, i.e., recover the d'dimensional latent vector by
arg min Lrec(I, G(z)). =ERd
where G(-) is the pre-trained generator, x is the input image, and Lree(;)) :) is the loss function mea- suring the distance between and the restored image á G(2), such 1 12-ormm distance and perceptual loss (Johnson et af.] 2016), or combinations thereof. However, this approach may fail when x is grossly corrupted by unknown corruptions, i.c., a small fraction of pixels are completely corrupted with unknown locations and magnitude. For example, in semantic image inpainting (Ych) ct al. 2017). where the corruptions are unknown missing regions, a pre-configured missing regions' segmentation mask is needed to exclude the missing regions' influence on the optimization proce- dure. Otherwise, the restored image will easily deviate from the ground truth image (Figure [].

---

Published as a conference paper at ICLR 2023
For another example, in supervised anomaly detec- tion (Schlegl et al. 2017), where the anomalies nat- urally occur unknown gross corruptions and the residual between the in- put image and the re- stored image is adopted as the anomaly segmenta- tion mask, i.e., x - G(2), such deviation will de- teriorate the segmentation performance. However, the assumption of know- ing pre-configured cor- rupted region mask can be strong (for semantic in- painting) even invalid (for unsupervised anomaly detection). Therefore, im- proving the robustness of GAN-inversion under un- known gross corruptions important.
learned
manifold
Approximation
comupted input image
GAN-inversion
True mamifold
RGI restored image
True image
R-RGI restored image
Figure 1: Inverting corrupted test image in Stanford cars dataset (Krause et al., 2013) (i) the GAN-inversion restored background can significantly deviate from the ground truth; contrast, the RGI method achieves robust background restoration under unknown gross corruptions; (ii) due to the GAN approximation gap. the true clean image may not live the GAN learned image manifold; R- RGI can further fine tune the learned manifold toward the true image manifold.
Another problem is the GAN approximation gap between the GAN learned image manifold and the true image manifold, i.e., even without corruptions, the restored image from Equation D can contain significant mismatches to the input image x. This limits the performance of GAN-based methods for semantic inpainting and, especially for unsupervised anomaly detection since any mis- match between the restored image and the input image will be counted towards the anomaly score. When segmentation mask of the corrupted region is known, such approximation gap can be mitigated by fine-tuning the generator (Pan et al. 2021). However, adopting such a technique under unknown gross corruptions can trivially overfit the corrupted image and fail restoration. There- fore, mitigating GAN approximation gap under unknown gross corruptions is important.
To address these issues, we propose an RGI method and further generalize it to R-RGI. For any corrupted input image, the proposed method can simultancously restore the corresponding clean image and extract the corrupted region mask. The main contributions of the proposed method are:
Methodologically. RGI improves the robustness of GAN-inversion in the presence of unknown gross corruptions. We further prove that, under mild assumptions, (i) the RGI restored image (and identified mask) asymptotically converges to the true clean image (and the true binary mask of the corrupted region) (Theorems | and 2); (ii) in addition to asymptotic results, for properly selected tuning parameter, the true mask of the corrupted region is given by simply thresholding the RGI identified mask (Theorem 2). (iii) Moreover, we generalize the RGI method to R-RGI for meaning- ful generator fine-tuning to mitigate the approximation gap under unknown gross corruptions.
Practically (i) for mask-free semantic inpainting. where the corruptions are unknown missing re- gions, the restored background can be used to restore the missing content; (ii) for unsupervised pixel-wise anomaly detection, where the corruptions are unknown anomalous regions, the retrieved mask be used as the anomalous region's 's segmentation mask. The RGI/R-RGI method unifies these two important tasks and achieves SOTA performance in both tasks.
RELATED LITERATURE
GAN-inversion Xia et al. 2022a) aims to project any given image to the latent space of a pre- trained generator. The inverted latent code can be used for various downstream tasks, including GAN-based image editing (Wang et al., 2022a), restoration (Pan et al. (02T), and so on. GAN- inversion be categorized into learning-based, optimization-based, and hybrid methods. The
2

---

Published as a conference paper at ICLR 2023
objective of the learning- based inversion method is to train encoder network to map image into the latent space based on which the reconstructed image closely resembles the original. Despite its fast inversion speed, learning- baanin inversion usually leads to poor reconstruction quality (Zhu et al. 2020 Richardson et al., 2021; Creswell & Bharath 2018). Optimization-basae methods directly solve a latent code that minimizes the reconstruction loss in Equation [ through backpropagation (which can be time-consuming), with superior image restoration quality. Hybrid methods balance the trade-off' between the aforementioned two methods (Xia et al.) 2022a). There are also different latent spaces to be projected on, such as the 2 space applicable for inverting all GANs, m 2 space (Gu al.) 2020). W and W* spaces for StyleGANs (Karras et al. 2019 Abdal et al. 2019 2020) and so on, All these methods do not have explicit robustness guarantees with respect to gross corruptions To improve the robustness of GAN-inversion, MimieGAN (Anirudh et al. 2018) uses surrogate network to mimic the unknown gross corruptions at the test time. However, this method requires multiple test images with the same corruptions to learn a surrogate network. Here, we focus on developing robust commonly used 2 space, with a provable robustness guarantee. The proposed method can be applied to single image with unknown gross corruptions, and has the potential to be applied to learning- based as well as hybrid methods, even for different latent spaces, to increase their robustness.
P
for optimization-basee
methods, projecting onto the most
As mentioned in the Introduction, DGP plays important role in corrupted image restoration. GAN-inversion is an effective way of exploiting the DGP captured by a GAN. Therefore, GAN- inversion gains popularity in two important applications of corrupted image restoration: semantic image inpainting and unsupervised anomaly detection. (Comprchensive reviews on semantic image inpainting and unsupervised anomaly detection are provided in Appendix A. and A.2
Mask-free free Semantic inpainting aims to restore the missing region of an input image with little or no information on the missing region in both the training and testing stages. GAN-inversion for semantic inpainting was first introduced by Ych et al. (2017) and was further developed by Gu et al. (2020) Pan et al. (20TT; Wang et al. | (2022b); El Helou Süsstrunk (2022) for improving inpainting quality. Current GnN-inversion based methods have the advantage of inpainting a single image with arbitrary missing regions, without any requirement for missing region mask information in the training stage. However, they do require a pre-configured missing region mask for reliable inpainting of a corrupted test image. Otherwise, the restored image can deviate from the true image. Moreover, the pre-configured corrupted region mask is also the key in mitigating the GAN approximation gap (Pan et al.) (02T) through generator fine-tuning. Such a pre-configured corrupted region mask requirement hinders the application of GAN-inversion in mask-free semantic inpainting.
Unsupervised pixel-wise anomaly detection aims to extract a pixel-level segmentation mask for anomalous regions, which plays an important role in industrial cosmetic defect inspection and med- ical applications (Yan et al., 2017 |Burr et al., 2022). Unsupervised pixel-wise anomaly detection extracts the anomalous region segmentation mask through a pixel-wise comparison of the input im- age and corresponding normal background, which requires high-quality background reconstruction based on the input image (Cui et al. 2022). GANs have the advantage of generating realistic images from the learned manifold with sharp and clear detail, which makes GAN-inversion promising tool for background reconstruction in pixel-wise anomaly detection. GAN-inversion for unsupervised anomaly detection (Xia et al.) 2022b) was first introduced by (Schlegl et al. (017)) and various fol- lowup works have been proposed (Zenati et al. 2018 [chleel et al. 2019 Baur et al. 2018 Kimura ct al. 2020; [Akcay et al. 2018). Counter-intutivell, instead of pixel-wis -ixel-wise anomaly detection, the applications of GAN-based anomaly detection methods mainly focus on image-leve//localzatoo level (Xia et al., 2022a) with less satisfactory performance. For example, as of the benchmark methods on the MVTec dataset (Bergmann et al. (011)). AnoGAN (Schlegl et al.. 2017) performs the worst on image segmentation (even localization) compared vanilla AE, not mention the state-of-the-art methods (Yu et al. 2021 Roth et al. 2022). This is due to two intrinsic issues of GAN-inversion under unknown gross corruptions: (i) Lack of robustness: due the existence of the anomalous region, the reconstructed normal background can easily deviate from the ground truth background (Figure D: (ii) Gap between the approximated and actual manifolds (Pan et al., 202T): even for clean input image, it is difficult to identify latent representation that can achieve perfect reconstruction. When the residual is used for pixel-wise anomaly detection, those two issues will easily deteriorate its performance.
Note that the "robustness to defects" mentioned in Abdal et al. 2019) means that the image together with the defects can be faithfully restored in the latent space, instead of restoring defect-free image
3

---

Published as a conference paper at ICLR 2023
3 ROBUST GAN-INVERSION
In this section, we first give problem overview. Then, we present the RGI method with theoretical justification of its asymptotic robustness properties. A simulation study is conducted verify the robustness. Next, we generalize the proposed method to R-RGI to mitigate the GAN approximation gap. Finally, we give a discussion that connects the proposed method with existing methods.
Overview. Given pre-trained GAN network on a large-scale clean image dataset, such that the generator learns the image manifold. For any input image from the same manifold with unknown gross corruptions, we aim to restore a clean image and a corrupted region mask (Figure 2
Training
learned
Clean image dataset
image manifold
train GAN
Gemerator
Testing
A
lire
Apsparse
Corrupted imput image
Restored image
Corrupted regioni nusk
Mask-frec Semantic inpsinting
Applications
Unsupervised pixel-wise anomaly dotection
Figure 2: RGI/R-RGI for mask semantic inpainting and unsupervised anomaly detection: Given a pre-trained GAN network on large-scale clean image dataset, such that the generator learns the image manifold. For any imput image from the same manifold with unknown gross corruptions, the proposed RGI/R-RGI can restore clean image G(z) and and identify the corrupted region mask M by solving the optimization problems thereof. Therefore, we unifies two important applications: (i) mask-free semantic inpainting; and (ii) unsupervised pixel-wise anomaly detection
Notation. Before we introduce the RGI method, first introduce the following notations: For any positive integer k. we use [k] to denote the set {1,2....
k}: .1.2....,k.: For any index set A C [m] x [n]. use |A| denote the cardinality of A; For any matrix T. the I-norm |[T]| (e.g. |[ (o)) is calculated by treating T as vector, and we use I to denote the non-zero mask of T, i.e. (Jr); = 0
if Tú = 0 and 1 otherwise; For any two sets A and B, we use dl (A. B) = suPa5 inf ||e blle denote the one-sided [-norm Hausdorff distance between and B. noting that when A singleton, it becomes the standard (-norm distance di(a, B) = inf lla bll:: For any two matrices and B, we use @ to denote element-wisg product, i.e., (A & B).g A.g B.j-
3.1 ROBUST GAN-INVERSION
Assume that the GAN learns accurate image manifold, i.e., there approximation gap between the GAN learned image manifold and true image manifold, such that any input image x E Rmx with gross corruptions s* m follows:
R is the true latent code and G(-) is pre-trained generator, i.c., G(-) Rd + pu Further, assume that s* admits sparsity property, i.c., ||s* llu S ng. where no is the number of corrupted pixels. Given æ, we aim to restore G(=*) (or z*), and consequently. achieve (i) semantic image inpainting, i.e., G(z*). ). or (ii) pixel-wise anomaly detection, i.e., M In-G(==)- To achieve so, we propose learn the latent representation and the corrupted region mask M at the same time, i.e.,
G(z²) ({2²))+ s* where
min GERd,MRRmss
Terec(11 M) 2 x. M) @ G(z))
s.t.
MM S no-
(2)

---

Published as a conference paper at ICLR 2023
The reconstruction loss term Lye((;-)) measures the distance between the input image and the gen- erated one outside the corrupted region, which guides the optimization process to find the latent variable 2. Intuitively, when solving for the mask along with the latent variable, the TAp mask elements such that reconstruction loss is minimized. It is easy to check that the true solution G(=*) and M is optimal. Moreover, if we assume that =" is the only latent code such that || - G(z>)llo S no, then we have uniqueness.
aim to allocate
However, Equation 2 with || llo is hard to solve. To address this issue, we relax Equation 2 to an unconstrained optimization problem that can be solved directly using gradient decent algorithms:
min =ER SERA,MER' M.
[ree((1 - M) ④ x (1 - M) の G(z)) + M|I
Equation is named RGI, where the second term penalizes the mask size avoid a trivial solution with the mask expanding to the whole image. Intuitively, the second term encourages small mask; however, the reconstruction loss will increase sharply once the learned mask cannot cover the cor- rupted region. By carefully selecting the tuning parameter X, we will arrive at solution with (i) high-quality image restoration with negligible reconstruction error; (ii) an accurate mask that covers the corrupted region. The following two theorems justify our intuition.
Theorem (Asymptotic optimality of 2) Assume (i) GAN learns an accurate image manifold, i.e.,
there exists z" such that || G(= loo no. (ūi is bounded for both Equation 2 and Equation 3
or equivalently there exists R > such that || < R, i.e., 2 E Sd with Sd == I-R, Rt)
(in)
(rre(+) = -1:: and (iv) (2)) is continuous. Let ( be any optimal 2 solution of Equation with
tuning parameter x. and z" be the optimal solution set of Equation 2 we have do (2(A), 2-) 4
as X + 0. Moreover, denote ñ = minges* || -Gz))lo and 2 = {= Sd || G(2)llo n). we
have dic (2(A), 2) $ 0 as X $ O. we further assume a unique 2t arg min || G(2) llo. i.e.
2 = }="}. then 2(2) - 2" as X O.
Note that Assumption (ii) is only for the proof purpose. We could always choose a large enough R to include all possible optimal solutions so that the optimality of Equation and Equation 3 remains. Remark:
Theorem | states that the optimal solution of the RGI method, G(2), converges to the true background G(z*) as + O, regardless of the corruption magnitude, which proves the robustness of the RGI method to unknown gross corruptions, and is the key to image restoration.
Theorem 2 (Asymptotic optimality of M) Follow the same assumptions and notations in Theo-
rem Let M be any optimal M solution of (3), and M == {{ E Σ} C {M e
{0, (0.1)mx m || S n]. We have dee (A), & as X 4 O. Moreover, there is a finite Ā such that for any S X, there an M M such that M = Jryay If further assume 2 = (2").
then (i) M(A - M as X 4 O, and for any X < X, M((x) M
Remark:
Theorem states that the optimal M solution of the RGI method, AZ, converges to the true corrupted region mask M as X O, regardless of the corruption magnitude. Moreover, there is fixed X, if we choose tuning parameter À I X, the true corrupted regions mask can be identified MAy. which is the key for pixel-wise anomaly detec-
by simply thresholding the M, i.e., M tion. The proof of Theorems and 2 are provided in Appendix B A simulation study to verify the
robustness of proposed RGI method is provided in Appendix
3.2 RELAXED ROBUST GAN-INVERSION
In traditional GAN-inversion methods (Yeh et al., 2017: Pan et al. 202I), without mask informa- tion, fine-tuning the generator parameters will lead to severe overfitting towards the input image. However, fine-tuning is the key step to mitigate the gap between the learned image manifold and any specific input image (Pan al. 202T). The proposed approach makes fine-tuning possible, i....
min ZERd ReVERmxn.EBB
Lrec (11 - M) @ x (1 - M) o G(2; 0)) + X||M[]
(4)
Equation 4 is named R-RGI. This problem can also be solved directly using gradient decent types of algorithm with carefully designed parameters for learning parameter 0. We found the following
5

---

Published as a conference paper at ICLR 2023
strategy gives better performance: at the beginning of the solution process, fix 9. When get a stable reconstructed image as well as the mask, then we optimize 0 together with all the other decision variables with small step size for limited iterations.
In this section, we address the robustness of GAN-inversion methods and show the asymptotic opti- mality of the RGI method. Moreover, the R-RGI enables fine-tuning of the learned manifold towards a specific image for better restoration quality and thus improves the performance of both tasks.
3.3 DISCUSSIONS
Connection to robust machine learning methods. The RGI method roots in robust learning meth- ods (Caramanis et al. 2012 Gabrel et al. 2014), which aims to restore a clean signal (or achieve robust parameter estimationn in the presence of corrupted input data. Robust machine learning meth-
ods, including robust dimensionality reduction (Candès et al., 2011 Xu et al. 2010; Peng et al, 2012). matrix completion (Candes & Recht [209]] Jain et al.] 2013). use statistical priors to model the signal to be restored, such as low rank, total variation, etc. Those statistical priors limit its appli- cations involving complex natural images, e.g., the restoration of corrupted human face images.
Similarly, we also aim for signal restoration from a corrupted input signal, but with two key differ- ences: (i) instead of the restrictive statistical priors, we adopt a learned deep generative prior (Pan) et al. 2021), i.e., G(z), which plays key role in modeling complex natural images. (ii) instead of recovering the corruptions, we learn a sparse binary mask M that covers the corrupted region, which is much easier than learning corruptions itself. The RGI method significantly extends the traditional robust machine learning methods to a wider range of applications.
Connection to robust statistics. The proposed method also has deep connection with traditional robust statistics (Huber
2011]: when adopting an 12-norm reconstruction loss, the loss function of Equation 3 be simplified Enp A2; where (2;
(xr G(2))
2 G()))
<>
(Equation in the proof of Theorem 1t. which shares
A G(=),, otherwise
similar spirit as M-estimators, e.g., metric Winsorizing and Tukey's biweight, Thus inherits the robustness with respect to outliers. Moreover, Equation [ allows a flexible way of incorporating robustness to reconstruction loss functions beyond convex formulations, such as the perceptual loss and discriminator loss (Pan et al. 2021
CASE STUDY
4.1 MASK-FREE SEMANTIC INPAINTING
Semantic inpainting is important task in image editing and restoration. (Please Appendix A. for a comprehensive literature review on this topic.) Among all the methods, GAN-inversion based methods have the advantage of inpainting single image with arbitrary missing regions with- out any requirement for mask information in the training stage. However, the requirement of a pre-configured corrupted region mask during testing hinders its application in mask-free semantic inpainting. In this section, we aim to show that the RGI method can achieve mask-free seman- tic inpainting by inheriting the mask-free training nature of GAN-inversion based methods, while avoiding the pre-configured mask requirement during testing. Therefore, we will compare with the sitie-of-the-aas art -nN-iiierooon based image inpainting methods that projecting onto the Z space, including (a) Yeh et al. (2017) without a pre-configured mask (Yeh et al. (2017) w/o mask) baseline; (b) Yeh et al. (2017) with a pre-configured mask ((Yeh et 2017) w/ mask); and (c) Pan et al. 2021 with a pre-configured mask (Pan et al. (2021) w/ mask).
Datasets and metrics. We evaluate the proposed methods on three datasets, the CelebA (Liu et al., 2015), Stanford car (Krause et al. (001)) and LSUN bedroom (Yu et al. 2015), which are commonly used for benchmarking image editing algorithms. We consider two different cases: (i) central block missing, (ii) random missing and (iii) irregular shape missing (see Appendix[]. We fill in the missing entry with pixels from (--1, 1). PSNR and SSIM are used for performance evaluation. Please see Appendix D for implementation details.

---

Published as a conference paper at ICLR 2023
Table 1: Semantic inpainting performance of Yeh et (2017) (w/ and w/o mask), Pan et al. (021
(w/ mask), RGI and R-RGI
methods
Datasets
Cases
metrics
Yeh et al.
Yeh et al.
Pan et al.
RGI
R-RGI
w/o mask
w/ mask
w/ mask
PSNR 1
11.50
20.82
19.70
21.74
20.05
Case (i)
SSIM T
0.358
0.492
0.451
0.570
0.509
CelebA
PSNR
19.64
22.63
21.52
27.63
23.73
Case (ii)
SSIM T
0.440
0.536
0.490
0.766
0.655
PSNR 1
16.57
17.50
16.89
20.98
19.31
Case (i)
SSIM
0.359
0.377
0.363
0.636
0.618
Cars
PSNR 1
17.36
17.71
17.52
21.61
21.18
Case (ii)
SSIM
0.361
0.382
0.363
0.650
0.588
PSNR T
16.15
19.27
17.67
21.36
18.72
Case
LSUN
SSIM
0.405
0.428
0.416
0.587
0.567
bedroom
PSNR 1
19.26
19.66
19.72
22.30
22.29
Case (ii)
SSIM
0.419
0.433
0.420
0.599
0.557
Comparison Results. The PSNR and SSIM of image restoration are shown in Table We can observe that (i) the RGI outperforms the Yeh et al. (2017) w/o mask baseline, and achieves a com- parable performance with Yeh et al. (017) w/ mask the best possible result without fine-tuning the generator. However, there is no pre-configured mask requirement in the RGI method, which demonstrates RGI's robustness to unknown gross corruptions. Such performance improvement is significant, especially CelebA dataset, where GAN learns high quality face manifold (high SSIM/PSNR value Yeh et al. (017)) w/ mask). (ii) the R-RGI further improves the image restora- tion performance with fine-tuning the generator, which achieves comparable performance with the (Pan et al. (202T) w/ mask the best possible result with fine-tuning the generator. Such perfor- mance improvement is significant, especially on Stanford cars and LSUN bedroom datasets, where et al. (2017) w/ mask is limited, indicating large GAN approximation gap. As shown Figure 3 the mask mask-free generator finettnnig by R-RGI guarantees a high-quality image restoration. More qualitative results are in Appendix D
even the performance of Yeh
Input
Yeh et al. sulo maok
at. we musk
RGI
Pan et al. maok
R R-RGi
Ground tnith
Stamford cars
LSUN
bedroom
Figure Case (i): Restored images on Cars Krause al. 2013) and LSUN Yu et al. 2015).
4.2 UNSUPERVISED PIXEL- WISE ANOMALY DETECTION
Unsupervised pixel-wise anomaly detection is becoming important in product cosmetic inspection. The extracted pixel-wise accurate defective region masks are then used for various downstream tasks, including aiding pixel-wise annotation, providing precise defect specifications (i.e. diameter,

---

Published as a conference paper at ICLR 2023
area) for product surface quality screening, and so on, which cannot be achieved by current sample level/localization level algorithms. The RGI/R-RGI method is developed for such an unsupervised fine-grained surface quality inspection task in data-rich but defect//annottiion which is common for mass production such as consumer electronics, steel manufacturing, and so on. In those applications, it is cheap collect large number of defect-free product images, while expensive and time-consuming to collect and annotate defective samples due to the super-high yield rate and expert annotation requirements.
environment,
There are three categories of unsupervised pixel-wise anomaly detection methods, including robust optimization based methods, deep reconstruction based methods (including GAN-inversion based methods), and deep representation based methods (Cui et al.] (002) (Please see Appendix |A.2 for a comprehensive literature review). In addition to the AnoGAN (Schlegl et al. (017) baseline, we will compare with the SOTA method in each category, including the RASI. (Peng et al.] 2012). the SOTA method in robust optimization method, which improves the RPCA (Candes et al. 201 1) to solve the linear misalignment issue; DRAEM (Zavrtanik et al., 2021) which is the SOTA method in deep-reconstruction based methods; and PatchCore (Roth et 2002), representative deep repre- sentation based method that performs the best on the MVTTc (Bergmann et al. 2019) dataset. We aim to show that with a simple robustness modification, the RGI/R- will significantly improve the baseline AnoGAN's performance and outperform the SOTA. We use PGGAN (Karras al.) 2017) as the backbone network and a l2 norm reconstruction loss term (Iree). For AnoGAN, the pixel-wise reconstruction residual | x used as the defective region indicator. We apply simple thresholding of the residual and report the Dice coefficient of the best performing threshold.
Datasets. We notice that the popular benchmark datasets, including MVTec AD (Bergmann et al. 2019) and BTAD (Mishra et al. 202T) for industrial anomaly detection, are not suitable for this task due to the following reasons: (i) the annotation of those datasets tends to cover extra regions of the real anomaly contour, which favors localization level methods. (ii) The number of clean images in most of the categories is small (usually 200 300 images), which may not be sufficient for E To gain
GAN training. A detailed discussion of the MV'Tec dataset can be found in Appendix a better control of the defect annotation and better reflect the data-rich but defect/anmntation rare application scenario, we generate synthetic defect dataset based on Product03 from the BTAD (Mishra et al. (02T) dataset. The synthetic dataset contains 900 defect-free images for training and 4 types of defects for testing, including crack, scratch, irregular, and mixed large (100 images in each category). Qualitative and quantitative comparisons with SOTA methods will be conducted on this dataset. The synthetic defect generation process are provided in Appendix
Metrics. We use the Dice coefficient to evaluate the pixel-wise anomaly detection performance, which is widely adopted for image segmentation tasks. Dice coefficient is defined as (2]l O (IIo)/(II) Mlo + ||M Io), where M E R is the predicted binary segmentation mask for the anormalous region and M E Rm the true binary segmentation mask with indicating the defective pixels and 0 otherwise. Notice that pixel-wise AUROC score (Bergmann et al., [2019) is sensitive to class imbalance, which may give misleading results in defective region segmentation tasks when the defective region only covers small portion of pixels in the whole image (This is often the case in industrial cosmetic inspection or medical applications (Baur et al. 2021 Mou et al., 2022 Zavrtanik et al. 2021). We mainly compare the Dice coefficients for different methods.
Compare with the AnoGAN baseline on on synthetic defect dataset. The results are shown in AnoGAN(Schless) et 2017), the only modification in the RGI method is the additional sparsity penalty term of the anomalous region mask M to en- hance its robustness. However, with such simple modification, RGI significantly outperforms the AnoGAN's performance under large defects ('mix large'), where the large anomalous region can easily lead deviation between the AnoGAN restored image and the real background. (ii) The R-RGI achieves significant and consistent performance improvement over the RGI and AnoGAN methods. The generator fine-tuning process closes the gap between the GAN learned normal back- ground manifold and the specific test image, which leads to better background restoration and mask refinement. The implementation details and more qualitative results can be found in Appendix G
Table 2 and Figure 4 (a). (i) Compared
Compare with the SOTA methods on on synthetic defect dataset. The results are shown in Table 2 and Figure 4 (b). R-RGI method performs the best in all defect types. The limited modeling capability of the low-rank prior used in RASL (Peng et al. (2122) leads to its bad performance; As localization level method, PatchCore (Roth et al. (022) can successfully localize the defect (Figure
8

---

Published as a conference paper at ICLR 2023
Input
R-RGI
RGI
AnoGAN
Input
True mask
R-RGI
DRAEM
RASL
PatchCore
(a) Comparison Among R-RGI, RGI and AnoGAN:
(b) Comparison Among R-RGI, DRAEM, RASL and PatchCore: The first column indicates the input images, and the following columns are detected defect mask.
The first row: input image and restored backgrounds; The second row: true mask and detected defect mask.
Figure 4: Performance comparison among different methods.
Table 2: Pixel-wise anomaly detection performance on synthetic defect dataset
Defects
methods
metries
crack
irregular
scratch
mixed large
Dice f
0.293 (0. (01)
0.246 (0.181)
0.280 (0.167)
0.355 (0. 100)
RASL
AUROC 1
0.868 (0.044)
0.817 (0.061)
0.834 (0.079)
0.784 (0.066)
Dice f
0.277 (0.157)
0.448 (0.167)
0.350 (0.103)
0.683 (0.138)
PatchCore
AUROC
0.936 (0.040)
0.973 (0.036)
0.966 (0.033)
0.972 (0.029)
Dice f
0.457 (0.117)
0.422 (0.208)
0,436 (0.212)
0.540 131)
AnoGAN
AUROC
0.954 (0.021)
0.956 (0.030)
0.947 (0.030)
0.949 (0.023)
Dice f
0.458 (0.119)
0,420 (0.194)
0.430 (0.192)
0.617 (0.110)
RGI
AUROC
0.951 (0.021)
0.952 (0.027)
0.944 (0.029)
0.949 (0.022)
Dice I AUROC
(0.2122
0.697 (0.152)
0.639 (0.169)
0.825 108)
DRAEM
0.951 (0.035)
0.988 (0.013)
0.982 (0.017)
0.989 195)
Dice I
0.809 (0.109)
0.758 (0.135)
0.745 (0.167)
0.810 (0.064)
R-RGI
AUROC
0.977 (0.037)
0.988(0.015)
0.980 (0.020)
0.968 (0.015)
(Roth et al. 2022 ). However, the loss of resolution deteriorates its pixel-level anomaly detection performance; The DRAEM (Zavrtanik et al. (2021) jointly trains reconstructive sub-network and discriminative sub-network with additional simulated anomaly samples on top of the clean training images. Its performance highly relies on the coverage of the simulated anomaly samples and is more sensitive large anomalies. More importantly, by incorporating the so-called mask free fine-tuning, the R-RGI method successfully improves the baseline AnoGAN method's performance over those SOTA methods on this task. More qualitative results can found Appendix G
5 CONCLUSION
Robustness has been a long pursuit in the field of signal processing. Recently, utilizing GAN- inversion for signal restoration gains popularity in various signal processing applications, since demonstrates strong capacity in modeling the distribution of complex signals such as natural images. However, there is no robustness guarantee the current GAN-inversion method.
To improve the robustness and accuracy of GAN-inversion in the presence of unknown gross corrup- tions, we propose an RGI method. Furthermore, we prove the asymptotic robustness of the proposed method, i.c. (i) the restored signal from RGI converges to the true clean signal (for image restora- tion); (ii) the identified mask converges to the true corrupted region mask (for anomaly detection). Moreover, generalize RGI method R-RGI method close the GAN approximation gap, which further improves the image restoration and unsupervised anomaly detection performance.
The RGI/R-RGI method unifies important tasks under the same framework and achieves SOTA performance: (i) Mask-free semantic inpainting, and (ii) Unsupervised pixel-wish wise anomaly detec- tion.

---

Published as a conference paper at ICLR 2023
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432 4441, 2019.
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan+++ How edit the embedded images? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8296-8305, 2020.
Manya V Afonso, José M Bioucas-Dias, and Mário AT Figueiredo. An augmented lagrangian approach to the constrained optimization formulation of imaging inverse problems. IEEE transactions on image processing. 20(3):681665,, 2010.
Samet Akcay, Amir Atapour-Abarghoue and Toby P Breckon. Ganomaly: Semi-supervised anomaly detection via adversarial training. In Asian conference on computer vision, pp. 622- 637. Springer, 2018.
Rushil Anirudh, Jayaraman Thiagarajan, Bhavya Kailkhura, and Timo Bremer. Mimic- gan: Corruption-mimicking for blind image recovery & adversarial defense. arXiv preprint A.KV:1811.08484. 2018.
Christoph Baur, Benedikt Wiestler, Shadi Albarqouni, and Nassir Navab. Deep autoencoding models for unsupervised anomaly segmentation in brain mr images. International MICCAI brainlesion workshop, pp. 161-169. Springer, 2018.
Christoph Baur, Stefan Denner, Benedikt Wiestler, Nassir Navab, and Shadi Albarqouni. Autoen- coders for unsupervised anomaly segmentation in brain mr images: a comparative study. Medical Image Analysis, 69:101952, 2021.
Paul Bergmann, Sindy Löwe, Michael Fauser, David Sattlegger, and Carsten Steger. Improving un- supervised defect segmentation by applying structural similarity to autoencoders. arXiv preprint 1, 2018.
Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehen- sive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1592-96000 2019.
Thierry Bouwmans and El Hadi Zahzah. Robust pca via principal component pursuit: A review for comparative evaluation in video surveillance. Computer Vision and Image Understanding, 122: 22-34. 2014.
Emmanuel Candès and Benjamin Recht. Exact matrix completion via convex optimization. Foundations of Computational mathematics, (6):717777, 2009.
Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1-37, 2011.
Constantine Caramanis, Shie Mannor, and Huan Xu. 14 robust optimization in machine learning. Optimization for machine learning, pp. 369, 2012.
Antonia Creswell and Anil Anthony Bharath. Inverting the generator of a generative adversarial network. IEEE transactions neural networks and learning systems, 30(7):1967-1974, 2018.
Yajie Cui, Zhaoxiang Liu, and Shiguo Lian. A survey on unsupervised industrial anomaly detection algorithms. arXiv preprint arXiv:2204. 1161, 2022.
Thomas Defard, Aleksandr Setkov, Angelique Loesch, and Romaric Audigier. Padim: patch distri- bution modeling framework for anomaly detection and localization. In International Conference on Pattern Recognition, pp. 475-489. Springer, 2021.
Majed El Helou and Sabine Süsstrunk. Bigprior: Toward decoupling learned prior hallucination and data fidelity in image restoration. IEEE Transactions Image Processing, 31:1628-11 1640, 2022.
10

---

Published as a conference paper at ICLR 2023
Virginie Gabrel, Cécile Murat, and Aurélie Thicle. Recent advances in robust optimization: An overview. European journal of operational research, 235(3):471-483 2014.
Dong Gong, Lingqiao Liu, Vuong Le. Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1705-1714, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1):139-1444 144, 2020.
Jinjin Gu, Yujun Shen, and Bolei Zhou. Image processing using multi-code gan prior. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3012-3021, 2020.
Denis Gudovskiy, Shun Ishizaka, and Kazuki Kozuka. Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizing flows. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 98-107, 2022.
Kaiming He and Jian Sun. Statisties of patch offsets for image completion. In European conference on computer vision, pp. 16-29. Springer. 2012.
Yao Hu, Debing Zhang, Jieping Ye, Xuelong Li, and Xiaofei He. Fast and accurate matrix comple- tion via truncated nuclear norm regularization. IEEE transactions on pattern analysis and machine intelligence, 35(9):21772210) 2012.
Peter Huber. Robust statistics. In International encyclopedia of statistical science, pp. 1248-1251 Springer, 2011.
Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics (ToG), 36(4):1-14, 2017.
Prateek Jain, Praneeth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alter- nating minimization. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing. pp. 065-674, 2013.
Justin Johnson, Alexandre Alahi, and Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference computer vision, pp. 694-711. Springer, 2016.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint #XXv:11100110196 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401 4410- 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Jaakko generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:12104-12114, 2020.
Jaakko Lehtinen, and Timo Aila. Training
Daiki Kimura, Subhajit Chaudhury, Minori Narita, Asim Munawar, and Ryuki Tachibana. Adver- sarial discriminative attention for robust anomaly detection. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 2172 117222181, 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint
arXiv:141m 4412.6980. 2014.
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- mne-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
11

---

Published as a conference paper at ICLR 2023
Jingyuan Li, Ning Wang. Lefei Zhang. Bo Du, and Dacheng Tao. Recurrent feature reasoning for image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7760-7768, 2020.
Yijun Li, Sifei Liu, Jimei Yang, and Ming- Hsuan Yang. Generative face completion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 391 191199, 2017.
Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European conference on computer vision (ECCV), pp. 85-100, 2018.
Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang. Coherent semantic attention for image inpainting. In Proceedings of the IEEE/CVT International Conference on Computer Vision, pp. 4170-4179, 2019a.
Yang Liu, Jinshan Pan, and Zhixun Su. Deep blind image inpainting. In International Conference on Intelligent Science and Big Data Engineering, pp. 128 141. Springer, 2019b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Pankaj Mishra, Riccardo Verk, Daniele Fornasier, Claudio Piciarelli, and Gian Luca Foresti. Vt-adl: A vision transformer network for image anomaly detection and localization. In 2021 IEEE 30th International Symposium on Industrial Electronics (ISIE), pp. 01-06. IEEE, 2021.
Shancong Mou, Meng Cao, Haoping Bai, Ping Huang, Jianjun Shi, and Jiulong Shan. Paedid: Patch autoencoder based deep image decomposition for pixel-level defective region segmentation. arXiv preprint arXiv:2203, 14457, 2022.
Kamyar Nazeri, Eric Ng. Tony Joseph, Faisal Z Qureshi, and Mchran Ebrahimi. Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv preprint 100.:1900.00212, 2019.
Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM Computing Surveys (CSUR), 54(2):1-38, 2021.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition. pp. 2536-2544, 2016.
Yigang Peng, Arvind Ganesh, John Wright, Wenli Xu, and Ma. Rasl: Robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE transactions on pattern analysis and machine intelligence, 34(1) (((11):2200 (1:2:)332222244 2012.
Rui Qian, Robby T Tan, Wenhan Yang. Jiajun Su, and Jiaying Liu. Attentive generative adversarial network for raindrop removal from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2482-2491, 2018.
Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li. Structureflow: Image inpainting via structure-aware appearance flow. In Proceedings of the IEEE/CVF International
Conference on Computer Vision, pp. 181-190, 2019.
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: stylegan encoder for image-to-image translation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2287-2296, 2021.
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1-318-14328, 2022.
12

---

Published as a conference paper at ICLR 2023
Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International conference on information processing in medical imaging, pp. 146- 157. Springer, 2017.
Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Georg Langs, and Ursula Schmidt- Erfurth. f-anogan:
Fast unsupervised anomaly detection with generative adversarial networks. Medical image analysis, 54:30 44, 2019.
Jianhong Shen and Tony F Chan. Mathematical models for local nontexture inpaintings. SIAM Journal Applied Mathematics, 62(3):1019-043,, 2002.
Yuhang Song, Chao Yang, Zhe Lin, Xiaofeng Liu, Qin Huang. Hao Li, and C-C Jay Kuo. Contextual-based image inpainting: Infer, match, and translate. Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the |EEE/CVF Winter Conference on Applications Computer Vision, pp. 2149-2155 2022.
Guodong Wang, Shumin Han, Errui Ding, and Di Huang. Student-teache feature pyramid matching for unsupervised anomaly detection. arXiv preprint arXiv:2103.04257, 2021.
Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen. High-fidelity gan inversion for image attribute editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11379 1179--13880 11388, 2022a.
Wentao Wang, Li Niu, Jianfu Zhang, Xue Yang, and Liqing Zhang. Dual-path image inpainting with auxiliary gan inversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14221-11430, 2022b.
Yi Wang, Ying-Cong Chen, Xin Tao, and Jiaya Jia. Venet: A robust approach to blind image inpainting. In European Conference Computer Vision, pp. 752-768. Springer, 2020.
Ryan Webster, Julien Rabin, Loic Simon, and Frédéric Jurie. Detecting overfitting of deep generative networks via latent recovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11273-11281 2019.
Fuzhang WU, Yan Kong, Weiming Dong, and Yanjun Wu. Gradient-aware blind face inpainting for deep face verification. Neurocomputing, 331:301-311, 2019.
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022a.
Xuan Xia, Xizhou Pan, Nan Li, Xing He, Lin Ma, Xiaoguang Zhang, and Ning Ding. Gan-based anomaly detection: review. Neurocomputing, 2022b.
Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, and Jiebo Luo. Foreground- aware image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1840-5848, 2019.
Huan Xu, Constantine Caramanis, and Sujay Sanghavi. Robust pea via outlier pursuit. Advances in neural information processing systems, 23, 2010.
Hao Yan, Kamran Paynabar, and Jianjun Shi. Anomaly detection in images with smooth background via smooth-sparse decomposition. Technometrics, 59(1):102-1144 2017.
Zhaoyi Yan, Xiaoming Li, Mu Li, Wangmeng Zuo, and Shiguang Shan. Shift-net: Image inpainting via deep feature rearrangement. In Proceedings of the European conference on computer vision (ECCV), pp. 1-17, 2018.
Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. High-resolution image inpainting using multi-scale neural patch synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6721-6729, 2017.
13

---

Published as a conference paper at ICLR 2023
Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Husegawa-Johnson. and Minh Do. Semantic image inpainting with deep generative models. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5485-5493, 2017.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop, arXiv preprint arviv: 1atii::1506.0335 506.03365, 2015.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505 5514, 2018.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4471-4480, 2019.
Jiawei Yu, Ye Zheng, Xiang Wang. Wei Li, Yushuang Wu, Rui Zhao, and Liwei Wu. Fast- flow: Unsupervised anomaly detection and localization via 2d normalizing flows. arXiv preprint arXiv:2111.07677, 2021.
Vitjan Zavrtanik, Matej Kristan, and Danijel Skočaj. Draem-a discriminatively trained reconstruc- tion embedding for surface anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8330-8339, 2021.
Houssam Zenati, Chuan Sheng Foo, Bruno Lecouat, Gaurav Manek, and Vijay Ramaseshan Chan- drasekhar. Efficient gan-based anomaly detection. arXiv preprint arXiv: FrXxv:1802.0622) 2018.
Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Learning gyramid-contes en- coder network for high-quality image inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1486-1494 2019.
Yu Zeng, Zhe Lin, Jimei Yang, Jianming Zhang, Eli Shechtman, and Huchuan Lu. High-resolution image inpainting with iterative confidence feedback and guided upsampling. In European conference on computer vision, pp. 1-17. Springer, 2020.
Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. arXiv preprint 1XXi:22103.10428 2021.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In European conference on computer vision, pp. 592-608, 2020.
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipu- lation on the natural image manifold. In European conference on computer vision, pp. 597-613 Springer, 2016.
Manyu Zhu, Dongliang He, Xin Li, Chao Li, Fu Li, Xiao Liu, Errui Ding, and Zhaoxiang Zhang- Image inpainting by end-to-end cascaded refinement with mask awareness. IEEE Transactions Image Processing, 0944855-4866 2021.
A
COMPREHENSIVE LITERATURE REVIEW
This section provides comprehensive literature reviews of mask free semantic inpainting and unsu- pervised pixel-wise anomaly detection, including but not limited to
based methods.
A. COMPREHENNSIVE LITERATURE REVIEW FOR MASK-FREE SEMANTIC INPAINTING
Mask-free Semantic inpainting aims to restore the corrupted region of an input image with little or no information the corruptions. To achieve this goal, multiple traditional single image semantic
inpainting methods exploit fixed image priors, including total variation (Afonso et al. 2010 Shen & Chan 2002), low rank (Hu al. 2012), patch off set statistics (He & Sun 2012 and
14

---

Published as a conference paper at ICLR 2023
However, due to the fixed image prior, those method have strong assumptions the input image, such as smoothness, containing similar structure or patches, which may fail when dealing with large missing regions with novel content, i.e., recovering the nose or mouth in facial images (Yeh et al. 2017].
Notice that various convolutional neural network based methods for semantic inpainting has been proposed (Pathak et al.
2016; lizuka et al. 2011 Li et al.] 2017 Yu et al. 2018 Liu et al.] 2018
Yu et al.
2019: Li et 2020, Suvorov et al. [022] Song et 2018; Yan al. 2018 Liu al.
201 1019al b Nazeri et al. 2019 Ren et al. 2019 Xiong et al. 2019 Zeng et al. 2019 2020; Zhao et al,
2021 Zhu et al. 2021). In addition to the requirement of pre-confggured mask for impainting an input image, they usually need mask information in the training stage, either the same fixed mask as the region to be inpainted (Pathak et al., 2016), or trying to cover irregular missing regions by randomly sampling rectangular mask with random location (Yang et al., 2017), or a fixed set of irregular masks (Liu et al. 2018) or generating masks following a set of rules ( (Yu et al., 2019 /Zhoo
et al. 1021 Those methods cannot fulfill the mask-free semantic inpainting goal.
Another closely related field is named blind image inpainting, which aims to inpaint a single cor- rupted image without the need for the corrupted regions mask (Liu et al. 2009b;; Wang et al. 2020) Qian et al., 2018 Wu et al. 2019; El Helou & Süsstrunk] 2022). However, most of them need training set of possible comuptiinn (and/or corrupted region masks), which again restricts their ability to generalize to unknown gross corruptions. Thus, they are not mask-free methods.
GAN-inversion for semantic inpainting was first introduced by Yeh et al. (2017) and further de- veloped by Gu et al. (2020); Pan et al. (202T); Wang et al.] (20226) for improving inpainting quality. They have the advantage of inpainting single image with arbitrary missing regions, without any requirement for mask information in the training stage. However, they do require a pre-configured corrupted region mask for reliable inpainting during testing.
The RGI method inherits the mask-free training nature of GAN-inversion based semantic inpainting methods, while avoiding the pre-configured mask requirement during testing. Thus, we can achieve mask-free semantic inpainting for single test image with arbitrary gross corruptions.
A.2 COMPREHENSIVE LITERATURE REVIEW FOR UNSUPERVISED PIXEL WISE ANOMALY DETECTION
Unsupervised pixel-wise anomaly detection aims to extract a pixel-level segmentation mask for anomalous regions, which plays an important role in industrial and medical applications (Yan al.,
2017 Baur et al. 2021). Unlike image-level (identify anomalous samples) or localization level (i.e., localize anomaly) anomaly detection, unsupervised pixel-wise anomaly detection extracts the anomalous region segmentation mask through pixel-wise comparison of the input image and corre- sponding normal background. Therefore, it requires a high-quality background reconstruction based on the input image. To achieve this goal, robust optimization methods rely on the statistical prior knowledge of the background (such as low-rank (Bouwmans & Zahzah, 2014] and smoothness (Yan) et al. 2017)), which is effective when the true background satisfies those assumptions. However, such assumptions can be restrictive and highly dependent on the properties of background image for specific applications. In contrast, the deep reconstruction based methods (Pang et al. 202T methods reconstruct the normal background from a learned subspace and assume such a subspace does not generalize anomalies. Autoencoder (AB) (Bergmann et al.) 2018), variational AE (VAE) (Kingmal & Welling, 2013) and its variants (please see the review paper (Baur et al., (02T)) are popular tools. However, such assumptions may not always hold, i.c., an AE that achieves a satisfactory reconstruc- tion of normal regions of the input image also "generalize" so well that it can always reconstruct the abnormal inputs as well (Gong et af.) (201)). Some solutions, such as MemAE (Gong et al, 2019) and PAEDID (Mou et al.) 2022) restrict this generation capability by reconstructing the back- ground from a memory bank of clean training images, DRAEM Zavrtanik D generation capability by integrating approaches are deep representation-bue based methods, which learns the discriminate embeddings of normal images from clean training set and achieve anomaly detection by comparing the embed- ding of a test image and the distribution of the normal image embeddings, such as PatchCore(Roli et al. 2022), Padim (Defard al. 2021), Cflow (Gudovskiy et al. 2022) STFPM (Wang et al.
et al. (2021) restrict this discriminated network. Another category of unsupervised
15

---

Published as a conference paper at ICLR 2023
2021). Those methods usually serves as localization tools since the comparison in the embedded space will lead to the loss of resolution.
Recently GAN-based anomaly detection methods have gained popularity in reconstruction based methods (Xia et al. 2022b). GANs have the advantage of generating realistic images from the learned manifold with sharp and clear detail, regardless of image type (Pang et al. 2021), which make GAN a promising tool for background reconstruction for pixel-wise anomaly detection. In- spired by this idea. Schleel et al. (2017) borrowed the GAN-inversion for unsupervised anomaly detectionand various followup works has been proposed, including EBGAN (Zenati et al. 2018), f- AnoGAN (Schlegl al. 2019) and GANomaly (Akeay et al. 2018). which mainly focus on improv- ing inference speed. Counterintuitieely, instead of pixel-wise anomaly detection, the applications of GAN-based anomaly detection methods mainly focus on image-leve//localizttoo level (Xia et al. 2022a) with less satisfactory performance. For example, as one of the benchmark methods on the MVTec dataset (Bergmann et al. 2019), AnoGAN (Schlegl et al., 2017) performs the worst on im- age segmentation (even localization) compared to vanilla AE, not to say the state-of-the-art methods (Yu et al.) 202T; Roth et al. 2022). This is due to the intrinsic issues GAN-inversion: (i) Lack of robustness: due to the existence of the anomalous region, the reconstructed normal background can easily deviate from the ground truth background (Figure I (ii) Gap between the approximated and actual manifolds (Pan al. 202T): even for clean input image, is difficult identify la- tent representation that can achieve perfect reconstruction. When the residual is used for pixel-wise anomaly detection, those issues will easily deteriorate its performance.
We aim to demonstrate the performance improvement of GAN-based anomaly detection method by RGI, which makes the GAN-inversion-basse methods practical in pixel-wise anomaly detection tasks.
B PROOF TO THEOREMS AND
B.1 PROOF TO THEOREMS
Under the assumption, there exists 2" such that ||x-G(==)lo I no- Thus (z", M*) solves Equation to its optimal value of O.
Denote n* = {a E gd | ||x- G())]l S no). Note that for any = E n we could set M = G() with || llo S no and then (z. M) also solves Equation 2 to its optimal value of 0. On the other hand, for any d g i.e., | G(2)llo 2 no + 1, T(1 M) @ (x G(=))
O unless
[|M] > x G(z)llo 2 no + 1, which renders such M is infeasible. Thus, we conclude that -" Z
The same optimality arguments apply to every E E E as 2 c z*. We next prove that Equation asymptotically converges to 2, which will complete the proof.
Denote f(z, f(, A: = I[(1 M) @ (z G(=)) + M|| the objective function of Equation Select any E and let M- (()) and we note that f(2,, M; = Ań.
Now consider for any given 2, we could next calculate M2) which minimizes f(z, M; A). Note that
M(2, = Σ ((1 My) G(=)); AlM&l)
Σ Sis(2, Mcs
with
05 0fg 2(x G(=)) - 1) + M09
JM, OM,
where a| Mayl is the partial differential of Mayl- It is clear that aM, Within the interval (0, 1), we in addition have
ahs < 0 for M,j < 0 and aM,
85. > 1 for Mij > 1, and thus the optimal My E |0, 1).
0h = 2(x - ((=))),(M)) 1) + A, aM,;
(5)
16

---

Published as a conference paper at ICLR 2023
and
0 solves to aM.y
8fg
Mi = 1 2(x - G(2))
Discussion:
. If 2(x G(=)) 2 X:
8M,; < 0 for Mij E (0, M,j) and ورم aMg > 0 for Mj E (My, 1},
afg
proving the optimality of Mij Mij.
If G(=)) < X:
afu aMG
0, for Mij E (0, 1), thus pointing the optimal My 0.
Combining those two cases, introduce (-)+ : maxf. 0} and we have
A
My 2(x G(=))
We now take the optimal M back to f and get
(xr - G(>))²
2 G(=)) < X,
M1:)
1 4(x G(E))
otherwise.
Define
m(A): == ñ 2 X A/2,
4
while also noting that μ 4 0 as 4 For any E $d, define the index set
Ax(2) {(i.)) | (a 14--Gi G(=)); m(A)}
and
2(A) = {z E sd |1Ax S
We next show that any Φ 2(A) cannot be optimal. For any & Z(),, we have [Ax(2)] 2 hn+ +11 1.
We also note that when (x G(2))² ((A) 2 A/2, we have
X²
1
fız(z;)) x
(1
7-2 ).
4n(A)
Therefore
f(z, ((,) (ft ()(1
nd = 5(2, M; al
ñ +
proving the non-optimality of such
As we limit the optimal solution to 2()), we now show that d (2(A), 2) 0 as 4
Assume the statement is not true, i.e., there exists en > 0 such that d (()), 2) > for any
O. Denote E {> C [m] x [n] | s R.. which is clearly finite. For any A E E, denote
z z²(A) = f2 gd (x G(2))² = p(A), v(i.)) EA}
and we have the following decomposition of 2(A):
z(A) UAEEZ*(A))
which is mathematically saying that Z()) can be decomposed by enumerating all the possible cases of choosing ñ elements from [m] x [m]. Combined with the fact that 3 is finite, we have
d (Z()), 2) maxAez d (Z)
60-
Note that Z(A) decreasing with respect to X, (Z(A), Z) is also decreasing, and the same ap- plies to (^( for any A E. Therefore, there exists particular v E such that dl (z" (A), 2) > to for any À (If not, for any A, there exists X(A) such that d (2*(A))) 2)
17

---

Published as a conference paper at ICLR 2023
co. and taking X = minAe= X(A) get (z 2) maxAes dm (2)),, 1
MaxĄE= a (ZA() 2) < co. contradicting the assumption).
Denote for this particular V, (A) == {a E (A) dz) 2) Ea} + 0. Notice that Uº (A) is
compact as it is both closed and bounded, and decreasing with respect to A. Therefore, let X L 0 be any decreasing series to 0, and from Cantor's intersection theorem, we have
(Ax) # 0.
Note that for any E E U (Aa), it clear that (x- - G(=)) = 0. i.., || G(z)llo S ñ thus 2 E 2, a contradiction to d (2, 2) 2 Ev.
Finally, as we have for any (A) optimal to Equation 3 as 2(A E Z()) we have 0 < dx (2(A), 2) S d (A),,, 2) 4 0 as X & 0, or d ($(A), 2) 4 0 as X $ O. which completes the proof.
B.2 PROOF TO THEOREM 2
M S {M E [0,1] mm || vi} comes straightforward from the definition of 2. We now
decompose 2 in the same fashion as 2()). For any E E let 2d := {2 E Sa | (z 1 G(=)), O, V(i.j) & A), and we have UAez = {a E S || G(z)llo I n} = Z, the last quality from the minimality of n. For the same reason, 2 is empty unless |A| = n. Note that 2 is closed and thus compact following the continuity of G. and thus the compactness of 2.
For any non-empty AA and (f,j) E A we have inf 1(x G()))yl > 0. If not, following the continuity of G, there exists E Σ such that (z - G(z)){ = 0, so || G(z)llo ñ - 1, contradiction the minimality of n. (Note that in the case of A 0, inf e (r G()));y1 +o0). Denote s minAeE min(GJ) Ze Ix G(=)ly O, which is independent from X.
Given the continuity of G, for any 0, there exists r > 0 such that for any 2, Sd with || - = < we have [|G(2) - (xx')llo < e. Specifically, we consider s/2 as and have the corresponding Ta/2, and we select À > 0 satisfying
and á 3(n + 1) Notice such a Å exists, since d (A)), 2) 0 as 1
d" (2 2) <
Ts/2
5²
s²
2
2
For any À < I, consider any optimal solution of Equation as (a))), (A)) and we have there
exists E 2 such that [|²(A) - =| = do (E(A), E) from the compactness of 2. Note that [|=(A) =0 d (z(A), 3) < (2(A), 2) < d (²²), 2) S rs/2/22
and thus ||G(2) - G( < s/2. As E غ, there exists A E E such that ă E ΣA, noting that AA = n. For any (i.j) E A, is clear that (I G()(A))) > [ G(3))al ((G(B(A) G()))gg 2 s/2 8/2. Therefore, we have X 2(x G(
Therefore, for any (i, A,
X
2A
1 > MA) = =1 2(x G(E(A)))
>
1
11,
as A 4.
Note that (A)ss > 0. We also have
X²
X2
A
(y)() (A): > 4(x G(2())))
2
2 x
3
+1)
where the last inequality from X A < 82 32/3(n + 1).
Next, prove that for any (i.j) & A, we have Mi (A)is = 0. Assuming there exists (E' $ such that ()) # O, we have 2(x G(E(A))) 2 > and thus
x²
(A)) = A 4(x G(E())))
18

---

Published as a conference paper at ICLR 2023
Therefore,
(())A)
= E M(5))) Σ M())))
(iS)AA
(i.)EA
X
x
A(A
3( + 1) 2
which is contradiction to the optimality of (3(X), M M since f(2, M; >ñ. Therefore, for any (i.j) & A, we have (A) O.
a(a) In conclusion, let M I a(a) E M and we have dx (A)( = (M(A),Mll I 22/s2 4 as À O. It is also clear that M ())))) long as A < A, which completes the proof.
C SIMULATION STUDY
In this section, we verify the robustness the RGI method under gross corruptions using simulation.
Data Generation. A Progressive GAN Karras et al. 2017 network is trained on the training set of 200599 aligned face images of size 128 x 128 from CelebFaces Attributes dataset (CelebA (Liu ct al. 2015h) and the pre-trained generator G(-) is extracted. Then we generate test image with central block corruptions (== 25% pixels) by: (i) Sample 2 E 12500 from the multivariate standard
Cij- normal distribution, i.e., N(O, f); (ii) Generate x by Tij = G(2) otherwise
if i, E (33....,9900
where Eij [ee, 1) and e is the mean corruption level. The pixel values of images generated by G(-) are approximately between 1- 1]. To verify the robustness of the RGI method, we vary the mean corruption level in the range of {-1, -0.5, O, 0.5, 13. The process is repeated to generate 100 input corrupted images for each mean corruption level.
Solution Procedure For each mean corruption level, we use three methods to restore G(z): (i) 12:
Solving Equation [ with (2 reconstruction loss L-re()) = 11:: (ii) 4: Solving Equation with 11 reconstruction loss L.ee(.) || |l (iii) RGI: Solving Equation 3 with l2 reconstruction loss (.) = || 1/ All methods are solved by ADAM Kingma & Ba 2014) for 1000 iterations. The root mean squared image restoration error (RMSE) of 100 input images recorded, i.e..
100
RMSE
Σ
[G(z,) - G(
100
where G(z) and G(8) are the true and restored backgrounds, respectively.
The RMSE of image restoration results under different corruption levels from methods (i)- (iii) is shown in Figure 5 The RGI method demonstrated superior robustness with an RMSE close to zero with respect to all five different corruption levels. 12 and reconstruction losses perform significantly worse under large corruption magnitude, which is expected since they seek an image on the learned manifold that is close the input image (even though I reconstruction loss adds to the robustness a little bit), which can lead to significant deviation of the image restoration.
Results:
D DETAILED DISCUSSION ON SEMANTIC IMAGE INPAINTING
D.1 IMPLEMENTATION DETAILS
A Progressive GAN (Karras et al. 2017) network is used the backbone network. Notice that the discriminator network usually used to regularize the generated image, such that the generated image looks real. Different methods incorporate the discriminator differently. It can either be in- corporated as a separate penalty term in the objective function (Yeh et al. 2017). incorporated as a modified reconstruction loss term (Pan et al. 2021) ignored in the loss function (Gu et al.) 2020). For a fair comparison, we use a weighted combination of an l norm (with weight 1) and a dis- criminator penalty term (with weight 0.1) the reconstruction loss for comparison methods. In all
19

---

Published as a conference paper at ICLR 2023
0.8
1
0.7
RGI
0.6
RMSE
0.4
0.3
0.2
0.1
-1.0
-0.5
0.0
0.5
1.0
Corruption level
Figure 5: RMSE of image restoration by RGI and GAN-inversion with 4, l2 loss.
experiments, the optimization problems Equation I (GGN-iinerssoon Equation 3 (RGI) and Equa- tion (R- are solved by ADAM (Kingma Bal 2014) for 2000 iterations, with learning rate of 0.1 for both 2 and M. For Equation (R- we use the last 500 iterations for mask-free fine- tnning with the learning rate of le for 0. The tuning parameter À is selected by cross-validation. Notice that using SSIM and PSNR gives different corss-validation results and here we report both:
(i) CelebA: block missing RGI and R-RGI: ÀSSIM APSNR 0.07; random missing RGI:
Assra = 0.2, ÄPSNR 0.5, R-RGI: ÀSSIM 0.25, APSNR 0.6.
(ii) Stanford cars: block missing RGI Aşsras 0.9, APSNN 1.0, R-RGI: ASSIM APSNR=
0.9; random missing RGI: ÄSSIM ÄPSNR R-RGI: ÀSSIM APSNR = 0.8.
(iii) LSUN bedroom: block missing RGI: AssrM = APSNR = 0.8, R-RGI: AsSTM APSNR =
0.7; random missing RGI: ÄSSTM 0.8, APSNR 1.0, R-RGI: Asstar = 0.6, ÄPSNR = 0.9.
D.2 DATASETS DETAILS
CelebA (Liu et al. 2015) contains a training set of 200,599 aligned face images. We resize them
the size of 128 128. We use the remaining 2000 images as the test set. Missing regions are
generated as follows: (i) central block missing of size 32 x 32 and (ii) random missing (= 50% pixels). We fill the missing entry with pixels from (-) We randomly select 100 test images to evaluate algorithm performance.
Stanford cars (Krause et al., (013) contains 16,185 images of 196 classes of cars and is split into 8,144 training images and 8,041 testing images. We crop the image based on the provided bounding boxes and resize them to the size 128 x 128. Missing regions are generated as follows: (i) central block missing of size 16 x 16 and (ii) random missing (=s 25% pixels). We fill in the missing entry with pixels from (--1, 1). The training and test set partitions provided by the dataset are used. We randomly select 100 test images evaluate algorithm performance.
LSUN bedroom Yuu et al., 2015) contains 3,033,042 images for training and 300 images for vali- dation. We resize the images the size of 128 128. Missing regions generated as follows: (i) central block missing of size 16 x 16 and (ii) random missing ( 25% pixels). We fill in the missing entry with pixels from (N-1, 1). We randomly select 100 images from the validation set to evaluate algorithm performance.
Next, will show the qualitative image restoration results these datasets. Notice that avoid showing the CclebA result due to copyright/rivacy concerns.
D.3 QUALITATIVE IMAGE RESTORATION RESULTS
Osshwss the qualitative image restoration results on Stanford cars Krause et al. 2013)) dataset. From columns 2-4, we can observe that the RGI method has comparable performance (Yeh)
20

---

Published as a conference paper at ICLR 2023
et al. 2017) w/ mask, which improves the image restoration performance of (Yeh al. 2017 w/o
mask. However, the performance of Yeh et al. (2017) even with mask information is not satisfactory (for example, the second row of Figure 6), this mainly due to the GAN approximation gap (Pan et al. 2021 In this case, further generator fine-tuning will significantly improve the faithfulness of restored images, which can be observed from columns 5-6.
Figure 7hows the qualitative image restoration results on the LSUN bedroom dataset. similar conclusion can also be drawn.
Yu et al. 2015)
No fine-tune
Fine-tune
Yoh
Ground truth
Lagut
who musk
mark
w: musk
(a) Case (i): block missing
No fine-tune
Fine-tune
Imput
Yel al. (21117)
Yeh et all (2017)
RGI
Pan (2021)
R-RGE
Ground truth
Weo miskk
mask
mask
(b) Case (ii): random missing
Figure 6: Restored images by Ych et (2017) (w/ and w/o mask), Pan (2021 (w/ mask), RGI
and R. R-RGI on Stanford cars (Krause et al. 2013 dataset.
21

---

Published as a conference paper at ICLR 2023
No fine-tune
Fine-tune
Imput
(2017)
Yoh
RGI
(2021)
RGI
wio mask
w/ misk
w: mask
truth
Case (i): block missing
No fine-tune
Fine-tune
Trput
(2017) wi mark
R-RGI
wio mask
w/ mask
truth
(b) Case (ii): random missing
Figure 7: Restored images by Yeh et al. 2017) (w/ and w/o mask), Pan et al. (2021) (w/ mask), RGI
and R-II on LSUN bedroom Yu et al. 2013) dataset.
E DETAILED RESULTS ON THE MVTEC AD DATASET
Annotation issues of the MVTec AD dataset. Figure 8 shows example images from the MVTTec dataset as well as the corresponding annotations. It is clear that the annotation covers a larger area than the exact defect contour, which will favor localization level methods such PatchCore (Roth) et al. 2022). However, this level of annotation is neither sufficient to fulfill the fine- ome-grained surface quality inspection goal, such providing precise defect specifications (i.e. diameter, length, area) for product surface quality screening, nor serve as an effective dataset for training/evaluating pixel- wise anomaly detection algorithms.
22

---

Published as a conference paper at ICLR 2023
Figure 8: Example defective products and their annotations from the MVTec AD dataset: The blue circle indicates the defects. we can see that the annotation tends to cover extra regions of the real anomaly contour.
Qualitative assessment on MVTTc AD. Figure shows the qualitative results of the wood product. We can observe that both the performance of RGT and AnoGAN are poor, where the restored images far from the true background, which leads to a noisy anomalous segmentation mask. The main reason is the small size of the training set, where the learned generator tends to overfit (memorize) the training set (Karras et al., 2020: Webster et al. 2019) rather than generalize to images in the test set. This will lead to huge gap between the learned training image manifold and the testing image manifold. By generator fine-tuning, the R-RGI can mitigate this gap to improve both the background reconstruction and anomalous region identification performance.
Input
R-RGI
RGI
AnoGAN
Input
R-RGI
RGI
AnoGAN
Figure 9: Comparison between the restored backgrounds and detected anomalies wood product: the first row indicates the input image and reconstructed background by various methods; the second row indicates the defective region segmentation mask annotation from the MVTec AD dataset and the detected anomalous region segmentation mask by various methods.
However, the success of the RGI/R-RGI method is built upon the assumption of a large training dataset such that the generator can learn a reasonable manifold which can generalize to unseen test samples. Mask-free fine-tuning can then mitigate the GAN approximation gap to further improve its performance. When the training set size is too small, where the generator tends overfit (memorize) the training set, merely relying the mask-free fine-tuning lead to unstable result.
F SYNTHETIC DEFECT GENERATION ON BTAD PRODUCTO3
The detailed defect generation process is discussed in this section. Product03 has 1000 defective free images, from which randomly select 100 images for defect generation. To improve the
23

---

Published as a conference paper at ICLR 2023
faithfulness of generated defective images, collect the binary defective region masks (equals to for defective pixels and otherwise) from the annotations of the MVTec AD (Bergmann et al. 2019) dataset and organize them into 4 categories, including crack, irregular, scratch, and mixed large (defective region area larger than 400 pixels) type of defective region masks. Then, we generate the synthetic defective image x; yyss by:
2. 1 = (1 - Mt) Of) M o Cf e (1,..0000,, E {crack, irregular, scratch, mixed large),
where 2; the ith input defect-free image, M? the ith randomly selected mask from the jth category. C/ E Rmxxx3 an image with constant channel values to fill in the defective region. To avoid trivial anomaly detection, we set C² as the average pixel value of the defective region, i.e.,
Cla:;,113 Epre(m),pe(l)) Mt [p1, P2.
kl (p1, P2, kc] k E [3].
Eple(m],pxs(n)) M P2.
Finally we have 4 categories of synthetic defects with 100 defective images in each category. Ex- amples of generated defective images are shown in Figure TO We can observe that the defects close to the background color, which makes them hard to distinguish even with human eyes. This avoids trail defect detection.
Cruck
Imreuluur
W
Scratch
Mixed
Figure 10: Generated synthetic defective images.
G DETAILED RESULTS ON THE BTAD DATASET
G.1 IMPLEMENTATION DETAILS
We use a PGGAN Karras al. 2017) as the backbone network and a la norm reconstruction loss term (Iree) for RGI and R-RGI methods. The tuning parameter À is selected via cross validation by using Dice coefficient as the metric. For RGI, the following values are selected: Aorack Airregular
Ascratch Amized large 0.4. For R-RGI, the following values are selected: Acract = 0.12. Airregutar = 0.1, Ascrateh = 0.14, Amized large = 0.06. All optimiza- solved by ADAM (Kingma & Ba, 2014) for 2000 iterations, with learning rate
tion problems of 0.1 for both a and M. For R-RGI, we use the last 1500 iterations for mask-free fine-tuning with the learning rate of le for 0.
24

---

Published as a conference paper at ICLR 2023
G.2
QUALITATIVE RESULTS
Crack
Imeegllar
Scratch
Input/ masks
ADoGAN
RGI
R-RGI
Figure 11: Comparison between the restored backgrounds and detected anomalies (after optimal thresholding) among AnoGAN (Schlegl al., 2017), RGI and R-RGI: The first indicates the input images and true anomalous region segmentation masks. The following rows are restored backgrounds and detected anomalous region masks by different methods.
Imeegular
Input/ masks
DRAEM
RASL
PutchCore
Figure 12: Raw anomaly scores and detected anomalies (after optimal thresholding) among DRAEM
(Zavrtanik et al.) 2021). RASL (Peng et al.) 2012) and PatchCore (Roth et al. (022): The first row indicates the input images and true anomalous region segmentation masks. The following rows are raw anomaly scores and detected anomalous region masks (after optimal thresholding) by different methods.
25

---

Published as a conference paper at ICLR 2023
H APPLICATION ASPECT OF THE PROPOSED METHOD
In this section, we will discuss the application aspect of the proposed method.
H.1 COMPUTATIONAL ASPECT
As mentioned in Section RGI/R-RGI is first attempt address the current robustness and ap- proximation gap issue of GAN-inversion. Therefore, we choose to study the optimization- GAN-inversion methods, because of its popularity and significant advantage of superior image restoration quality (Zhu et al., 2020; Richardson et al. 2021 Creswell & Bharath, 2018). However, the proposed RGI/R-RGI method faces similar computational challenge as optimization-ban bbasd GAN-inversion methods: the inversion process of each input image requires solving optimiza- tion problem similar size, which can computationally expensive. Generalizing the proposed method to learning-baset as well as hybrid GAN-inversion methods for better computational effi- ciency is an immediate extension, and we leave it for future work.
H.2 REDUCING THE TRAINING SAMPLE SIZE REQUIREMENT
The proposed R-RG R-RGI method can reduce the training data requirement for GAN- anomaly de- tection methods (Schleg] et al. 2017 2019). GAN-based methods (Xia et al. 2022b) are important directions in anomaly detection, which usually needs large dataset of cleaning training images. Insufficient training samples is one of the major sources of the GAN approximation gap. Failing to address such gap will lead the poor performance (in addition the robustness issue) of most of current G GAN-inversion based anomaly detection methods (Xia et al. 2022b; Zenati et al. 2018 Schlegl et al. 2019 Baur et al. 2018 Kimura al. 2020 Akcay et al. 2018).
As can be seen from the experiment result in unsupervised anomaly detection (Table 2), the per- RGI over the AnoGAN still limited, which indicates a large GAN approximation gap, i.e., there is still large mismatch between the closest image in the learned im- age manifold and the ground truth background of the input test image, beyond the robustness issue. By addressing such gap in R-RGL, we observed significant performance improvement in all different defect scenarios, and the R-RGI outperforms the SOTA method (DRAEM) on this task.
formance improvement
I MASK-FREE IMAGE INPAINTING OF IRREGULAR MISSING REGIONS
In this section, we include the mask-free semantic image inpainting result when the missing re- gion is of irregular shapes, missing case (iii). The irregular missing masks are from Irregular Mask Dataset (Liu et al., 2018). The qualitative result is shown Figure [3 and Figure [4] The quantitative result is shown in Table We can observe that the inpainting performance of irregular missing in case (iii) and block missing in case (i) are fairly close (Tables and IT. This is expected since there is mask information involved in both the training and testing stage of the proposed method, which avoids potential overfitting to any specific missing mechanism. In other words, the irregular missing, block missing RGI/R-RGI method.
random missing all 'unknown gross corruptions' the
Table 3: Case (iii) irregular missing region: semantic inpainting performance of Yeh et al. 2017
(w/ and w/o mask). Pan et 2021) (w/ mask), RGI and R-RGI
methods
Datasets
Cases
metrics
Yeh et al.
Yeh et al.
Pan et al.
RGI
R-RGI
w/o mask
w/ mask
w/ mask
PSNR 1
13.72
21.25
19.86
22.71
20.42
CelebA
Case (iii)
SSIM
0.358
0.503
0.592
0.570
0.547
PSNR
17.07
17.57
17.04
21.34
20.20
Cars
Case (iii)
SSIM +
0.359
0.361
0.357
0.633
0.613
LSUN
PSNR
17.52
19.00
18.20
21.30
19.77
Case (iii)
bedroom
SSIM 1
0.393
0.424
0.412
0.599
0.575
26

---

Published as a conference paper at ICLR 2023
No fine-tune
Fine-tune
Expul
Yoher moak
E-BGi
Groed
Figure 13: Case (iii) irregular missing region: restored images by Ych et al. 2017 (w/ and w/o mask), Pan et al. (2021) (w/ mask), RGI and R-RGI Stanford cars (Kause et al. 2013) dataset.
fine-tune
Fine-tune
Iopat
Yeh al. (2011)
weo moak
Yoh et al, (2017) wi mask
RGE
Pan al (2101) mask
R--GG
Greend Falk
Figure 14: Case (iii) irregular missing region: restored images by Yeh et al. 2017 (w/ and w/o
mask), Pan et al. (202)) (w/ mask), RGI and R-RGI on LSUN bedroom Yuu et al. 2015) dataset.
J SENSITIVITY ANALYSIS OF X
The tuning parameter RGI/R-RG (equation and equation 4 is important both mask- free semantic inpainting and unsupervised pixel-wise anomaly detection applications. In this section, provide the empirical sensitivity analysis using examples from both applications.
Sensitivity analysis of À in mask-free semantic image inpainting. Figure 15 shows the SSIM value change with respect to À mask-free semantic inpainting of block missing experiment on CelebA (Liu et al. 2015) dataset. In this experiment, vary the tuning parameter À in the
27

---

Published as a conference paper at ICLR 2023
0.55
R RGI
0.50
SSIM
0.45
0.40
0.10
Figure 15: SSIM value change with respect to in mask-free semantic inpainting of block missing experiment CelebA (Liu et al. 2015) dataset.
0.8
R-RGI
0.7
0.6
0.5
0.06
0.20
0.40
0.60
0.80
1.00
Figure 16: Dice value change with respect to in unsupervised pixel-wise anomaly detection of crack defect experiment.
range of [0.05, 0.25). A plateau region can be observed for both the RGI ((0.03, 0.10)) and R-RGI (00.05, 0.15)) method, where similar inpainting performance as the optimal X can obtained.
Sensitivity analysis of À in unsupervised pixel-wise anomaly detection. Figure T6 shows the Dice value change with respect in in unsupervised pixel-wise anomaly detection of crack defect experiment the synthetic dataset. For RGI, we vary the tuning parameter À in the range of [0.05, 1.00]. A large plateau region (0.220, 0.60) can be observed, where similar semantic inpainting performance the optimal X can be obtained. For R- R-RGI, vary the tuning parameter À in the range of [0.06, 0.6]. Compared to RGI. R-RGI has an additional generator fine-tuing step when optimizing the mask M, which makes R-RGI more sensitive X. However, a plateau region (00006 0.2), where accepttale semantic inpainting performance, compared to the optimal X", can be obtained. The existence of such plateau region in both applications makes it easier for the tuning of X.
still observe
28


Reviews:
[{"review_id": "sQE0vDPm2_e", "reviewer": "Reviewer_TJ5g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a robust GAN-Inversion (RGI) method for mask-free image inpainting and unsupervised pixel-wise anomaly detection. Furthermore, they further propose a relaxed RGI method for achieving better results. Promising experimental results on image inpainting and anomaly detection by using the proposed method are shown with comparison with some previous methods.", "summary_of_the_review": "In summary, the paper has nice novelty in tern of the proposed new algorithms for solving the mask-free semantic inpainting and pixel-level anomaly detection problems. The proposed methods are accompanied by two theorems for robust GAN-Inversion. However, there are some concerns with the paper in terms of lack of discussion of the critical  limitations for the GAN-Inversion approach and the insufficient experimental justification, especially on the anomaly detection experiment.", "strength_and_weaknesses": "Strengths:\n1. This paper resents a robust GAN Inversion (RGI) method and a relaxed RGI (R-RGI) method for mask-free image inpainting and anomaly localization problems.\n2. This paper presents two theorems for robust RGI that provides the theoretical foundation for the optimization formulation of the proposed algorithm.\n3. Experimental results by applying the proposed RGI and R-RGI methods to two application problems on some public datasets are shown with promising results.\n\nWeaknesses:\n1. The proposed RGI or R-RGI methods require to solve the large optimization problems during the inference. There is no discussion on the computational cost involved in solving the large optimization problems during the inference. It could be very computationally infeasible for practical application.\n\n2. Another limitation with the GAN inversion approach is that it requires a large number of normal images to train a GAN model for the specific task. In many cases, there are not enough numbers of normal images for training the GAN models. In their experiments on image inpainting, the datasets used in the experiments are CelebA, Standard cars, and LSUN bedroom all contain very large numbers of images for the GAN model training. But for MVTec and BTAD datasets, the numbers of training images are not a lot. There is no discussion on the numbers of images used in the model training for the implementation on anomaly detection experiments. There should be more discussion on the issue of training data requirement for the proposed method.\n\n3. The experiments on pixel-wise anomaly detection are only performed on their synthetic defect dataset based on BTAD dataset. The experimental evaluation is not sufficient to demonstrate the proposed RGI or R-RGI achieve SOTA performance for anomaly detection. They should perform experiments with the standard protocol on the commonly used datasets, such as MVTec and BTAD datasets and include experimental comparison with some recent SOTA methods.\n\n4. The discussion in section 3.3 claims the proposed RGI method is connected to robust statistics and the loss function in eq. 3 can be simplified by using the robust error function, which avoids introducing M in the loss function. However, there is no justification of this claim. If this is the case, is the resulting optimization problem also simpler to solve? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oU4cMnEbEkb", "reviewer": "Reviewer_sD9p", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the GAN-Inversion problem. In particular, the GAN-inversion problem when the image suffers from missing or some pixels are no longer reliable. The proposed method is very simple and straightforward. A mask is introduced to the inversion optimization, and the mask has a constraint on the 0-norm. The results seem to be good, and this simple method works. The author also provided theoretical proof for this method.\n", "summary_of_the_review": "My rating is based on: 1) there are no major technical omissions, 2) the problem studied is important, and 3) the proposed method works. But since 1) the method is simple, 2) the proof is of limited value. I don't rate it too high.", "strength_and_weaknesses": "Strength:\nThe problem is important. The solution is simple, and the results are good.\nIt's actually surprising that such a simple method can work. As far as I know, before the stylegan came along, this kind of method was very difficult to work.\n\nWeaknesses:\n1. The trade-off parameter $\\lambda$ seems to be an important value, but the article does not show the sensitivity of the method to lambda.\n2. The proof may not make much sense. This paper provides some theoretical guarantees. I briefly checked the proof. Its biggest problem is that some assumptions basically do not appear in real-world applications. But the final experimental result was good. Probably this does not experimentally affect the performance of the proposed method. But on the other hand, this reduces the contribution of this paper, because it seems that this guarantee is not important.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jP_26KHSN8", "reviewer": "Reviewer_XsAp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a GAN-Inversion method to solve the image restoration problems. From the results shown in this paper, this author emphazie a lot on the image inpainting problem, and also show the potential in solving the anomaly detection problems. ", "summary_of_the_review": "Well. I am also very busy recently, thus I may not have enough time to go through every detail of this paper. However, based on my experiences in the past several years, this paper still needs some time to be better refined. ", "strength_and_weaknesses": "The strength and weakness of this problem are all very obvious. \n1. By using the GAN-Inversion, the author achieves better results on non-mask image inpainting scenarios, which help the classical image inpainting problems overcome a challenging obstacle. \n2. I know that for image inpainting problems, the guys always overfit their module some specified scenarios, and show very promising results on some test data with similar properties. Yeh etal is just like this. I am not sure whether this method is still in this way. If they still solve this problem in this manner and show poor generalization ability to other scenarios, I donot think the GAN-Inversion is a big plus. \n3. To relive my concern, please provide more details in the rebuttal about the details of training data, for example, if you want to get the result on CELEBA dataset, do you need to use similar data to train your network for the generative image prior?\n4. Is it ok to show more results beyond current established dataset? \n5. Is it ok to provide more details about Figure 2? From the figure, this work still relies on some kinds of mask to achieve the results, though this mask is not explicitly obtained. However, considering the examples in this figure, where the mask is with significant differences to its surrounding regions, it may not be very difficult to estimate such a mask. \n6. Image inpainting has already been a rat race. Many methods have been proposed. Why do you only compare with so limited methods?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "d24YNN63vmX", "reviewer": "Reviewer_fpcW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a robust GAN-inversion method to cope with restoring images with unknown corruption and detecting unknown defects. They addressed the problem by involving the mask in the optimization process, and encouraging it to be sparse. Besides, to further close the GAN gap, they proposed to finetune the GAN generators while adapting new datasets. The authors showed the experimental results on both mask-free inpainting and unsupervised defect detection tasks, and claim it outperforms SOTA, and can be applied to other SOTA framework easily. ", "summary_of_the_review": "The major concerns of this paper:\n- The synthetic dataset (for both tasks) fail to convince the reviewer that the proposed method is effective enough to recover the unknown corruption in the wild. If the author claimed only synthetic dataset can be used due to the lack of clean images for training GANs, then there are additional concerns on how the proposed methods can be generalized to more real-world tasks. Or the problems are not about the synthetic dataset, but the synthetic data (center block / random pixel missing) in this paper is not that realistic and diverse to reveal the advantages of the proposed method.\n- The evaluation is poor, especially for the mask-free inpainting tasks. Baselines are weak, and the metrics are not advanced to demonstrate the visual quality. For GAN-based generation, at least LPIPS/FID should also be reported. \n- More interesting topics like partial GAN-inversion should be explored to prove the robustness. ", "strength_and_weaknesses": "Strength:\n- The paper is well-written and easy to follow. The two tasks are closely related and the proposed methods make sense.\n- The two-stage optimization further closes the GAN gaps. The generalization of the method is intuitive.\n\nWeaknesses:\n- The idea is simple, effective and intuitive, but the problem setting is not that practical. \n    (1) Center block is too easy to be overfitted, and it cannot reveal the advantages of the mask learning. \n    (2) Random missing is also not practical enough, and a simple denoising method may be more effective in resolving the problem.\n    (3) Synthetic defects are limited, and cannot be proven to be generalized to real tasks. \n- The sparsity assumption is limited. Similar ideas can also be found in paper [1], while the authors of that inpainting paper proposed to jointly optimize the mask and reconstruction loss during training.  \n- It would be interesting to study whether we can learn the regional GAN-inversion by matching partial generated images with the ground truth. For example, whether we can optimize the inversion network to only extract the foreground objects or salient objects. It may be an interesting extension of this work. \n- Finetunning the generator may be not that practical for real applications. \n\n[1] Zeng, Yu, et al. \"High-resolution image inpainting with iterative confidence feedback and guided upsampling.\" European conference on computer vision. Springer, Cham, 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Qqz-2j2upMM", "reviewer": "Reviewer_pBCc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a Robust GAN-inversion method for restoring gross corrupted images. They also extend RGI to Relaxed RGI for generator fine-tuning to mitigate the gap between the GAN learned manifold and the true image manifold. Their result achieves sota performance in two applications: mask-free semantic inpating and unsupervised pixelwise anomaly detection.", "summary_of_the_review": "I think the paper's idea is novel, and the topic of this paper is also very important, I think the paper should add more experiments to demonstrate their method.", "strength_and_weaknesses": "Strength: 1. Paper is well written and easy to understand.\n2. The research problem is important and has practical value.\n3. Their proposed method seems novel.\n\nWeakness: The experiment seems not thorough, For example, In Table 1, I'm curious about why they only compare limited cases in the experiment. I think the method should compare with more previous works and datasets to demonstrate its efficiency. Besides, I think the paper should add an ablation study to demonstrate the method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster