Large Scale Image Completion via Co-Modulated Generative Adversarial Networks
Authors: Shengyu Zhao
Jonathan Cui
Yilun Sheng
Yue Dong
Xiao Liang
Eric I-Chao Chang
Yan Xu
Venue: ICLR 2021 Spotlight (2021)
Keywords: image completion
generative adversarial networks
co-modulation

Abstract: Numerous task-specific variants of conditional generative adversarial networks have been developed for image completion. Yet, a serious limitation remains that all existing algorithms tend to fail when handling large-scale missing regions. To overcome this challenge, we propose a generic new approach that bridges the gap between image-conditional and recent modulated unconditional generative architectures via co-modulation of both conditional and stochastic style representations. Also, due to the lack of good quantitative metrics for image completion, we propose the new Paired/Unpaired Inception Discriminative Score (P-IDS/U-IDS), which robustly measures the perceptual fidelity of inpainted images compared to real images via linear separability in a feature space. Experiments demonstrate superior performance in terms of both quality and diversity over state-of-the-art methods in free-form image completion and easy generalization to image-to-image translation. Code is available at https://github.com/zsyzzsoft/co-mod-gan.

Published as a conference paper at ICLR 2021
LARGE SCALE IMAGE COMPLETION VIA CO-MODUL- ATED GENERATIVE ADVERSARIAL NETWORKS
Shengyu Zhao
Jonathan Cui
III., Tsinghua University and Microsoft Research
Vacaville Christian Schools
Yilun Sheng
Yue Dong IIIS, Tsinghua University
IIIS, Tsinghua University and Microsoft Research
Xiao Liang
Eric I-Chao Chang Microsoft Research
The High School Affiliated to Renmin University of China
Yan Xu"
School Biological Science and Medical Engineering and Beijing Advanced Innovation Centre for Biomedical Engineering, Beihang University
ABSTRACT
Numerous task-specific variants of conditional generative adversarial networks have been developed for image completion. Yet, a serious limitation remains that all existing algorithms tend to fail when handling large-scale missing regions. To overcome this challenge, we propose a generic new approach that bridges the gap between image-conditional and recent modulated unconditional generative archi- tectures via co-modulation of both conditional and stochastic style representations. Also, due to the lack of good quantitative metrics for image completion. we propose the new Paired/ (niirerd Inception Discriminative Score (P-IDS/U-IDS)) which robustly measures the perceptual fidelity of inpainted images compared real images via linear separability in feature space. Experiments demonstrate superior performance in terms of both quality and diversity over state-of-the-art methods free-form image completion and easy generalization to image-to-image translation. Code is available at attttps: //tthp wttt:/szzooo..ttccccoo co mod- gan.
INTRODUCTION
Generative adversarial networks (GANs) have received a great amount of attention in the past few years, during which a fundamental problem emerges from divergence of development be- tween image-conditional and unconditional GANs. Image-conditional GANs have wide variety of computer vision applications (Isola et al. (017). As vanilla U-Net-like generators cannot achieve promising performance especially in free-forme form image completion (Liu et al. 2018 Yu et al. 2019), multiplicity of task-specific approaches have been proposed to specialize GAN frameworks, mostly focused on hand-engineeree multi-stage architectures, specialized operations, intermediate struc- tures like edges or contours (Altinel et al. 2018 Ding et al. 2018 lizuka et al. 20177; Jiang et al.) 2019 Lahiri et al.] 2020 Li et al.) 2020
Liu al. 2018 2019a 2020; Nazeri al.) 2019 Ren
00 2019 Yu et al. 2019: Zeng al.] 2019 Zhao et al. 2020al Zhou al. 2020). These branches
et al. 2019 Wang et al. 2018 Xie et al. 2019) Xiong et al. 2019 Yan et al. 2018 Yu et al. 2018
of works have made significant progress in reducing the generated artifacts like color discrepancy and blurriness. However, serious challenge remains that all existing algorithms tend to fail when handling large-scale missing regions. This is mainly due to their lack of the underlying generative capability - one can never learn to complete a large proportion of an object so long as it does not have the capability of generating a completely new one. We argue that the key to overcoming this challenge is to bridge the gap between image-conditions and unconditional generative architectures.
"Corresoonding author

---

Published as a conference paper at ICLR 2021
Mask
Original
Figure 1: Our image completion results w.r.t. different masks. Our method successfully bridges differently conditioned situations, from small-scale inpainting large-scale completion (left to right). The original images are sampled at 512 resolution from the FFHQ dataset (Karras et al. 201 9a) within 10k validation split (top examples) and the Places2 validation (Zhou 2017 (bottom two examples). We refer the readers to the appendix for extensive qualitative examples.
Recently, the performance of unconditional GANs has been fundamentally advanced, chiefly owing to the success of modulation approaches (Chen et al. 2019 Karras et al.) 2019a b) with learned style representations produced by latent vector. Researchers also extend the application of modulation approaches to image-conditional GANs with the style representations fully determined by an input image (Park et al. 2019; Huang et al. 2018 Liu et al. 2019b); however, the absence of stochasticity makes them hardly generalizable to the settings where only limited conditional information is avail- able. This limitation fatal especially in large scale image completion. Although some multi-modal unpaired image-to tongeett--mage translation methods propose to encode the style from another reference image (Huang al. 2018 Liu et 2019b), this unreasonably assumes that the style representations are entirely independent of the conditional input and hence compromises the consistency.
Therefore, propose co-modulated generative adversarial networks, generic approach that leverages the generative capability from unconditional modulated architectures, embedding both conditional and stochastic style representations co-modulation. Co-modulated GANs are thus able to generate diverse and consistent contents and generalize well not only small-scale inpainting but also extremely large-scale image completion, supporting both regular and irregular masks even with only little conditional information available. See Fig. for qualitative examples. Due to the effectiveness of co- co-modulation, we do not encounter any problem suffered in the image completion literature (Liu al. 2018 Yu et al. 2019). successfully bridging the long-existing divergence.
Another major barrier in the image completion literature the lack of good quantitative metrics. The vast majority of works in this literature seek improve their performance in terms of similarity-based
2

---

Published as a conference paper at ICLR 2021
metrics that heavily prefer blurry results, e.g., L1, L2, PSNR, and SSIM, among which many state that there are yet no good quantitative metrics for image completion (Liu et al., 2018: Yu et al.) 2018 2019). The only gold standard in this literature is the user study, which conducts real vs. fake test giving a pair of images to subjects (i.e., the users). However, the user study is subject to large variance and costly, therefore lacking reproducibility. Inspired by the user study, we propose the new Paired/Unpaired Inception Discriminative Score (P-IDS/U-IDS) Besides its intuitiveness and scalability, we demonstrate that P-IDS/U-IDS is robust to sampling size and effective of capturing subtle differences and further correlates well with human preferences.
Our contributions are summarized as follows:
We propose co-modulated GANx, a generic approach that bridges the gap between image- conditional and recent modulated unconditional generative architectures.
We propose the new P-IDS/U-IDS for robust assessment of the perceptual fidelity of GANs. Experiments demonstrate superior performance in terms of both quality and diversity in free-form image completion and casy generalization to image-to-ime -mage translation.
2 RELATED WORK
Image-Conditionas GANs. Image-conditional GANs can be applied to variety of image-to-image
translation tasks (Isola et al. 2017). The unpaired setting is also investigated when paired training
data not available (Choi et al. 2018 Huang et al. 2018 Kim et al., 2019 Lazarow al. 2017: Liu
it al. [017]] Yi et al. 2017: Zhao et al. 2020b Zhu et al. (017a). Recent works exploit normalization
layers with learned style representations embedded from the conditional input or another reference
image to enhance the output fidelity (Huang et al. 2018 Kim et al. 2019 Liu et al.) 2019b] Park et al.)
2019). They can be regarded as set of conditional modulation approaches, but still lack stochastic generative capability and hence poorly generalize when limited conditional information is available. Isola ct al. (2017) initially find that the generator tends to ignore the noise input although they try to feed it, in contrast to unconditional or class-conditional GANs. A branch of works aims enforce the intra-conditioning diversity using VAE- based latent sampling strategies (Zhu et al. 20176) or imposing distance-based loss terms (Huang al.. 2018 Mao al. 2019 Qin al. 2018)). Wang itt al. (2019b) also propose to decompose the convolution kernels into stochastic basis. However, the
enforcement of diversity conversely results in the deterioration of image quality. Our co- co-modulation approach not only learns the stochasticity inherently but also makes the trade-off casily controllable.
Image Completion. Image completion, also referred to image inpainting when incapable of completing large-scale missing regions, has received a significant amount of attention. It is a constrained image-to pongge-tt-immn image translation problem but exposes more serious challenges. Traditional methods (Ballester et al. 2001 Baanes et al. 2009 Darabi et al. 2012 Efros & Freeman, 2001 Efros
& Leung. 1999) utilize only low-level features and fail to generate semantically consistent contents.
Then, (Köhler et al. 2014 Ren et al. 2015 Xie et al. 2012) adopt deep neural networks for image completion; (Pathak et al. 2016) first exploits conditional GANs. Numerous follow-up works focus on the semantic context and texture, edges and contours, or hand-engineered architectures (Altinel al.
[0ig:Ding et al.
2018 lizuka al.) 2017]Jiang et al.) 20119) & Park] 2019)/ahin et al.
2017
Liu et al., 2019a| Nazeri et al. [0199] Ren et al. 2019)) (agonng et al. 2019 Wang et al. [2018]
Xie et al. 2019 Xiong et 2019 Yan et al. 2018 Yang et al. al. 2017; 2019a] Yu et al. | [208]] Yu
et al. 2019 Zeng et al. 2019: Lahiri et al. 2020:
Zhao et al. 2020a Li et al. 2020 Zhou et al.
2020). among which (Liu et al. 2018: Yu et al.
2019) introduce partial convolution and gated
convolution to address free-form form image completion The lack of stochasticity is also observed in
image completion (Cai & Wei 2019 Ma et al. 2019: Zheng et al. 2019). Other works address the
so-called outpainting subtasks (Sabin & Rusak 2018 Wang et al. [019a]] Yang et al. (201bb). To our knowledge, none of these methods produce promising results in the presence of free-form large-scale missing regions.
Evaluation Metrics. Great research interest has been drawn on the evaluation of GANs (DeVsies
et al. 2019 Gurumurthy et al. 2017 Sajjadi et al., 2018 Snell et al., 2017: Xiang & Li, 2017).
Inception Score (IS) (Salimans et al. 2016), and some other metrics like FCN Score (Isoaa a et al. 2017), are specialized to the pre- trained task thus cannot generalize. While FID (Heusel al. (017)) is generally acceptable, few promising metries for image completion exist. Previous works heavily rely on similarity-base metrics such L1, T2z, PSNR, and SSIM, which fail to capture stochastic
3

---

Published as a conference paper at ICLR 2021
const
y
y
y
M
M
D
D
D
D
unconditional
conditional
co-modulation
modulation
modulation
(a)
(b)
(c)
(d)
Figure 2: Illustration from modulation to co-modulation: (a) unconditional modulated generator; (b) vanilla
generator; (c) conditional modulated generator; and (d) co-modulated generator. y, represent the conditional input and the latent vector respectively; & D.M represent the conditional encoder, the generative decoder, and the mapping network, respectively.
regions and are ill-fitted for GANs. Our proposed metric is also related the classifier basee tests (Blau & Michaeli 2018 Lopez-Paz & Oquab) 2016). However, previous classifier based metrics require separate
for training and testing the classifier, making them sensitive the underlying generalizability of the trained classifier. We formulate the discriminability as a simple scalable metric for both the paired and unpaired versions without relying on the generalizability.
3 CO-MODULATED GENERATIVE ADVERSARIAL NETWORKS
Image-conditional GANs address the problem of translating an image-form conditional input to an output image æ (Isola et al. 2017). We assume for the setting where paired correspondence between input conditions and output images is available in the training data. The generator takes as input an image y along with the latent vector z and produces the output æ;; the discriminator takes as input pair of (æ, y) and seeks to distinguish fake generated pairs from the real distribution. Image completion can be regarded as a constrained mmage-conditionn pixels restricted be unchanged. In contrast the extensive literature specialized image completion frameworks, we introduce a generic approach that bridges between image-conditions GANs and recent success of unconditional modulated architectures.
generation problem where known
3.1 REVISITING MODULATION APPROACHES
Modulation approaches emerge from the style transfer literature (Dumoulin et al. 2016 Huang & Belongie 2017) and well exploited in state-of-the-art unconditional or class-conditional GANs. They generally apply scalar denormalization factors (c.g., bias and scaling) to the normalized feature maps, while the learned denormalization factors are conditioned on the side information such as class label (Odena et al., 2018) or the latent vector (Chen et al., 2019). Typical normalization layers used in the modulation blocks include batch normalization (Chen et al. 2019 Odena et al.) 2018), adaptive instance normalization (Huang & Belongie 2017] / Karras et al. 2019a), and weight
demodulation (Karras et al. 2019b) referred to the weight normalization (Salimans & Kingma 2016).
Here we take StyleGAN2 Karras et al., 2019b) an example to show how intermediate activations are modulated as function of the latent vector. As illustrated Fig. 2(a) the decoder simply originates from learned constant, while the latent vector is passed through multi-layer fully connected mapping network M. The mapped latent vector linearly generates a style vector for each subsequent modulation via learned affine transformation A (i.e., dense layer without activation):
A(M(z)).
(1)
Consider a vanilla convolutional layer with kernel weights Wijk. where i, j. & enumerate the input channels, the output channels, and the spatial footprint of the convolution, respectively. Given the style vector s, the input feature maps are first channel-wise multiplied by s, passed through the convolution, and finally channel-wise multiplied by where s/ = 11 Σi, (saW)a)2 acts as the weight demodulation step that normalizes the feature maps into statistically unit variance.
4

---

Published as a conference paper at ICLR 2021
While modulation approaches have significantly improved the performance of unconditional or class-conditional generators, we wonder whether they could similarly work for aanee-coniitioal generators. An intuitive extension to the vanilla image-conditional generator (Fig. 2 would be the conditional modulated generator (see Fig. [())..)) where the modulation is conditioned on the learned flattened features from the image encoder & Similar structures also exist in the well-conditioned -ongeeto- translation tasks (Huang et al. 2018 Liu et al. 2019b Park et al. 2019). In this case,
the style vector can be rewritten
8 = A(E()))
(2)
However, a significant drawback of the conditional modulation approach would be the lack of stochastic generative capability. This problem emerges more apparently in respect of large scale image completion. In most cases, the outputs should be weakly conditioned, i.e., they are not sufficiently determined by the conditional input. As result, it not only cannot produce diverse outputs but also poorly generalizes to the settings where limited conditional information is available.
3.2 CO-MODULATION
To overcome this challenge, we propose co-modulation, a generic new approach that easily adapts the generative capability from the unconditional modulated generators to the image-conditions generators. We rewrite the co-modulated style vector as (see Fig. ((d)
8 = M(()),
i..., a joint affine transformation conditioning on both style representations. Generally, the style vector can be non-linear learned mapping from both inputs, but here we simply assume that they can be linearly correlated in the style space and already observe considerable improvements. The linear correlation facilitates the inherent stochasticiry as will see in $5.1 that co-modulated GANs can easily trade-off between quality and intra-conditioning diversity without imposing any external losses, and moreover, co-modulation contributes to not only stochasticity but also visual quality especially at large-scale missing regions. Co-modulated GANs are encouraged to be trained with regular discriminator losses, while not requiring any direct guidance like the L1 term (Isola et al.) 2017), to fully exploit their stochastic generative capability.
4 PAIRED/UNPAIRED INCEPTION DISCRIMINATIVE SCORE
Our proposed Paired/Unpaired Inception Discriminative Score (P-IDS/U-IDS) aims to reliably measure the linear separability in pre-trained feature space, inspired by the "human discriminators" in the user study. Let I(.) be the pre-trained Inception v3 model that maps the input image to the output features of 2048 dimensions. We sample the same number of real images and their correspondingly generated fake images (drawn from the joint distribution (æ, x') E X, where a corresponds the real image and x' corresponds to the fake image), from which the features extracted and then fitted by a linear SVM. The linear SVM reflects the linear separability in the feature space and is known to be numerically stable training. Let f(-) be the (linear) decision function of the SVM, where (I(())) > 0 if and only if x is considered real. The P-IDS is given by
P-IDS(X) (a,=')eX
Pr
(f(I(z)) f(I(x)))
ie., the probability that fake sample is considered more realistic than the corresponding real sample.
We also provide an umpaired alternative that could generalize to the settings where no paired informa- tion is available. We similarly sample the same number of real images (drawn from distribution X) and fake images (drawn from distribution X') and fit the linear SVM f(-). We directly calculate the misclassification rate instead:
U-IDS(X,X)) x') = Pr (S([(=)) <0
Pr (S(((²')) > 0}. 2 x'Ex'
(5)
2 #ex
In addition to the super intuitiveness of P-IDS/U-IDS, we would like to emphasize three of their major advantages over FID: the robustness to sampling size, the effectiveness of capturing subtle differences, and the good correlation to human preferences.
5

---

Published as a conference paper at ICLR 2021
50% W=1
We1
40%
40%
20
20%
20%
10
Wed
W-B
0%5
0%
sampling size
sampling size
sampling size
Figure 3: Robustness to sampling size. Dashed convergence lines are measured using 50k samples. P-IDS/U-IDS converges fast; FID fails to converge within 10k samples. Results are averaged over runs; error bars indicate standard deviations.
14e44
2.5
3e-4
50%
12e-4
2e-4
10e-4
45%
P-ID//U-IDS
KID
6e-4
4e-4
2e-4
-1e.4
0-
30%
U-IDS P-IDS
8 FID,
-2e-4
KID
4e-4
25%
-304
1k
2k
3k
4k
5k
6k
7k
8k
9k
10k
2
2
2 2
2
2 2' 2 2
sampling size
amount of noise
Figure 4: Kernel Inception Distance (KID) still suffers from large variance. Although it achieves unbiased estimates, the huge variance even makes them often negative and hardly dis- tinguishable. Results are averaged over runs; error bars indicate standard deviations.
Figure Effectiveness of capturing subtle dif- ferences. All metrics are measured using 10k samples. P-IDS effectively reflects the amount of noise; FID and KID fail to respond within 2 noisy pixels. Results are averaged over runs; enror bars indicate standard deviations.
Robustness Sampling Size. We test the response of P-IDS, U-IDS, FID, and KID four manipulation strategies: masking the image (to zeros) with a random square of width w = 1, 2, 4, 8, respectively. Images are sampled from the FFHQ dataset (Karras al. 2019a) 512 512 resolution. The reference distribution for calculating FID is measured using 50k samples. As plotted in Fig. both P-IDS and U-IDS converge fast within a small number of samples and successfully distinguish the manipulation strategies; FID fails to converge within 10k samples, while the highest convergence line (1.13 when w = 8, measured using 50k samples) even below the lowest FID at 10k samples (1.63 when w = 1). Although KID addresses the "biased" problem of FID (Birikowski et al.) 2018), we find that the estimates are still subject to huge variance like FID especially when the two distributions close. KID requires fixed block size (Binkowski et al.) 2018) to achieve unbiased estimates; even with a block size 1000 that minimizes its variance, the estimates are still hardly distinguishable especially between
and 2 as plotted in Fig. 4
Effectiveness Capturing Subtle Differences. Capturing subtle differences particularly im- portant in image completion, since the difference between inpainted and real images only exists in partial region. We construct subtle image manipulation strategies by masking n random pixels which then nearest-poine interpolated by the neighboring pixels, using the same environment as the last experiment. As plotted in Fig. 5 P-IDS successfully distinguishes the number of manipulated pixels, while FIDD and KID fail respond within 29 noisy pixels. We note that U-IDS is still more robust in this case since the central tendency of FID and KID is significantly dominated by the variance.
Correlation to Human Preferences. P-IDS imitates the "human discriminators" and is expected
to correlate well with human preferences. While it seems clear in Fig. 6 we quantitatively measure the correlation using these data points (20 in total): the correlation coefficient is 0.870 between P-IDS and human preference rate, significantly better than -0.765 of FID. Table 3 further provides a case analysis where P-IDS/U-IDS coincides with clear human preferences opposed to FID.

---

Published as a conference paper at ICLR 2021
Table 1: Quantitative results for large scale image completion. Our method is compared against DeepFillv2 (Yu et al. 2019) and RFR (Li ct al. 2020). Results are averaged over 5 runs.
FFH
Places2
Method
P-IDS (%)
U-IDS (%)
P-IDS (%)
U-IDS (%)
FID
RFR (official)
0.0 0.0
0.0 0.0
48.7 0.5
0.3 0.3
4.6 0.0
49.6
DeepFillv2 (official)
0.0
0.1
0.6
0.8
8.4
3.,
DeepFillv2 (retrained)
0.9 0.1
8.6 0.2
0.4
1.4
11.4 0.0
22.1
Ours
16.6 + 0.3
29.4 =: 0.3
3.7 + 0.0
13.3 0.1
27.4
7,9 - 0.0
50%
humann proferance rate
Dur (F-M0)
Ours sca2)
40%
40%
Cus 20
Deeplitve (FF10)
Despriint (Maccaa)
2esphilg (hoces2)
30%-
30%-
20%
(Pacee2)
10%-
Desphind (FFH0)
0%
Deepili v2 (Pioxe))
(0.2. 0.4) (..4...)) 10.6. 10.8. 1.0)
0%-
0.2) (0.2. (0.4,
10.0.0.2) 10.2.
(0.6, 0.81 1.0) masked ratio
masked
masked
Figure 6: User study results, P-IDS and FID plots of DeepFillv2 (retrained) and ours w.r.t. different masked ratios. P-IDS and FID are averaged over runs; error bars indicate standard deviations.
Computational Analysis. The time complexity of training linear SVM is between (n)) and O(n)) (Bottou Lin 2007). compared to O(dd² + d of FID (Heusel al., 2017) and (n of KID BBnkowwki et al. 2018), where is the sampling size and d is the dimension of feature space. In practice, P-IDS/U-IDS incurs mild computational overhead in addition to the feature extraction process. For example, with 10k samples, extracting the Inception features
NVIDIA P100 GPU takes 221s, and fitting the SVM (which only uses CPU) takes an extra of 88s; with 50k samples, the feature extraction process and the SVM take 1080s and 886s respectively.
EXPERIMENTS
5.1
IMAGE COMPLETION
We conduct image completion experiments at 512x512 resolution on the FFHQ dataset (Karras ct al. 2019a) and the Places2 dataset (Zhou et al. 2017). Implementation details are provided in Appendix
A FFHQ augmented with horizontal flips; Places2 central cropped or padded. The sampling strategy of free-form masks for training and evaluating is specified in the appendix. We preserve 10k out of 70k images from the FFHQ dataset for validation. Places2 has its own validation set of 36.5k images and a large training set of 8M images. We train our model for 25M images on FFHQ and 50M images on Places2. Our model is compared against RFR (Li et al., 2020) and DeepFillv2 Yu et al. 2019), the oftte-of-the-aad algorithms for free-form image completion, using both their official pre-trained models and our retrained version of DeepFillv2 (using the official code, datasets, and sampling strategy) 1M iterations (i.e., 32M images). We sample the output once per validation image for all the metrics (P-IDS, U-IDS, and FID). The overall results are summarized in Table I Fig. plots the user study results, P-IDS, and FID of DeepFiillv2 (retrained) and ours w.r.t. different masked ratios. See Fig. 7 for a qualitative comparison. All these results demonstrate our superior performance. More qualitative examples, numerical user study results and complete tables w.r.t. the masked ratio, and details of the user study are provided in the appendix.
The Inherent Stochasticity. Co-modulated GANs are inherently stochastic, i.e., they naturally learn to utilize the stochastic style representations without imposing any external losses, and they are able to produce diverse results even when both the input image and the input mask are fixed. Furthermore, by tuning the truncation zb (Karras et al. 2019b; Kynkäänniemi et al. 2019) that explicitly amplifies the stochastic branch by t times, co co-modulated GANs can easily trade-off between quality and diversity (see Fig. 8)
7

---

Published as a conference paper at ICLR 2021
Masked
Original
PatchMatch
DeepFillva (offica))
DeepFillva (retrained)
Ours
Figure 7: Qualitative examples of state-of-the-art methods on large scale image completion: PatchMatch (Barnes et al. 2009). DeepFillv2 (Yu et al., (2019), and ours. The original images sampled at 512x512 resolution from the FFHQ dataset (Karras et al. (2099)) within a 10k validation split (top) and the Places2 validation (Zhou al. 2017) (bottom). We refer the readers to the appendix for extensive qualitative examples.
Ablation Study. Co-modulation promotes only stochasticity but also image quality. We com- pare vanilla, conditional modulated, and co-modulated GANs as illustrated in Figs. 2(b) to 2(d) Experiments are
the FFHQ dataset with the same setting as $5.1 While the vanilla version completely fails, our co-modulation approach dominates the conditional modulated version and especially when the masked ratio becomes large (see Fig. [0). We refer the readers the appendix for the complete results (Table 6). Qualitatively, we often observe some unusual artifacts of the
conditional modulated one in the large missing regions (see Fig. 9), which we hypothesize due to the lack of stochastic generative capability.
5.2 IMAGE-TO-IMAGE TRANSLATION
Edges to Photos. Co-modulated GANs are generic image-conditional models that can be easily adopted to image-to-image translation tasks. We follow the common setting (DeVries et al.) 2019 Wang et al. 201 9b) on the edges to photos datasets (Isola et al. 2017) at 256x256 resolution, where FID samples once per validation image (200 in total) and the training set is used as the reference distribution; LPIPS measures the intra- conditioning diversity for which we sample 2k pairs. As summarized in Table 2 our approach easily achieves superior fidelity (FID) over otate-of-thh-ar methods (Huang et al., 2018: Isola et al. 2017 Wang et al.) 2019h /hu et al. 7b) despite the fact that MUNIT assumes for the different unpaired setting (Huang et al. 2018). and also superior diversity on the HanebaHa dataset by simply tuning the truncation & as well as in the trade-off
view (see Fig. Our model does learn produce diverse outputs the Edges2Shoes dataset
Oriinnal
Ours
Original
(C-Mod)
(Co-Mod)
Ours
Orighnal
Ours
Masked
Ours (Vanlia)
Ours (C
Ours (Co-((ooMo))
Figure 8: The inherent stochasticity. Co- modulated GANs can easily trade-off between quality and diversity by tuning the truncation v.
Figure Qualitative ablation study among Vanilla, conditional modulation (C-Mod), and our co-modulation (Co-Mod) as in Figs. 2 to 2
8

---

Published as a conference paper at ICLR 2021
P- DS
FID
12% 14% 16% 18% 20% 22%
24% 26% 26% 30% 32%
100
90
70
60
50
(0.8, 1.0)
19th Ours (C: Mod)
0.4
MUN T
- Durs
(0.6, 0.8)
0.3
BicycleGAN
masked
00.4, 0.6)
00.2. 0.4]
PxxPFix
(0.0. 0.2)
0.0
Co-modulation dominates condi- tional modulation (C-Mod) at all masked ratios, especially when the masked ratio becomes large.
Figure 10:
Figure 11: Trade-off curve of method be- tween quality (FID) and diversity (LPIPS) on the Edges2Handhags dataset.
Table Image-to to-e translation results on the edges to photos (Isola al. 2017
datasets.
Edges2Shoes
Edges2Handbags
Method
FID
LPIPS
FID
LPIPS
Pix2Pix (Tsola ct al. 2017
74.2
0.040
95.6
0.042
BicycleGAN (Zhu al. 2017b)
47.3
0.191
76.0
0.252
MUNIT (Huang
2018
56.2
0.229
79.1
0.339
BasisGAN (Wang el al. 2019b
64.2
0.242
88.8
0,350
Ours Ours (eb 3)
38.5
0.036
56,9
0.143
38.5
0.038
71.1
0.379
despite its high fidelity, which we hypothesize is due to the learned strong correspondence between the input edge map and the color information extracted from the limited training set.
Labels Photos (COCO-Stuff). We further experiment on the COCO-Stufi dataset (Caesar et al. 2018) at 256 256 resolution following the experimental setting of SPADE (Caesar et al. 2018). The real images are resized to a short edge of 256 and then random cropped. The input label map has 182 classes; embedding layer used before feeding it into the network. We sample the output once per validation image (5k in total) for all the evaluation metrics. Table 3 shows that our method matches the FID of SPADE but significantly outperforms its P-IDS and U-IDS, without any direct supervision like the perceptual loss used in SPADE. We further conduct a user study between SPADE and ours. The user study indicates consistent human preference of ours over SPADE in accordance with proposed P-IDS/U-IDS. Qualitative results and the user study details are provided in the appendix.
Table Image-to-imag translation results on the COCO-Stuff dataset (labels to photos).
Method
User study (SPADE vs. ours)
P-IDS (%)
U-IDS (%)
FID
SPADE Parr et al. 2019
41,0%
1.1
5.3
22.6
Ours
59.0%
11.3
22.5
CONCLUSION
We propose the co-modulated generative adversarial networks, a generic approach that bridges the gap between conditional and unconditional modulated generative architectures, significantly improves free-form large scale image completion, and easily generalizes to mmage-to-mmage translation. We also propose the intuitive new metric P-IDS/U-IDS for robustly assessing the perceptual fidelity for GANs. We expect our approach to be a fundamental solution to the image completion literature and contribute as reliable quantitative benchmarks.
ACKNOWLEDGEMENTS
This work is supported by the National Science and Technology Major Project of the Ministry of Science and Technology in China under Grant 2017YFC01 10903, Microsoft Research under the eHealth program, the National Natural Science Foundation in China under Grant 81771910, the

---

Published as a conference paper at ICLR 2021
Fundamental Research Funds for the Central Universities of China under Grant SKLSDE-2017ZA from the State Key Laboratory of Software Development Environment in Beihang University in China, the 111 Project in China under Grant B13003.
REFERENCES
Fazil Altinel, Mete Ozay, and Takayuki Okatani. Deep structured energy- based image inpainting. In 2018 24th International Conference on Pattern Recognition (ICPR), pp. 423-428. IEFE, 2018.
Coloma Ballester, Marcelo Bertalmio, Vicent Caselles, Guillermo Sapiro, and Joan Verdera. Filling-in by joint interpolation of vector fields and gray levels. IEEE transactions on image processing. 100:11200-12111
2001.
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized corre- spondence algorithm for structural image editing. ACM Trunsactions on Graphics (ToG), pp. 24. ACM, 2009.
Mikolaj Binkowski, Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: 000401, 2018.
Yochai Blau and Tomer Michaeli. The perception-distortie tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6228 6237, 2018.
Léon Bottou and Chih-Jen Lin. Support vector machine solvers. Large scule kernel machines, 3(1::301-320, 2007.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1209-1218, 2018.
Weiwei Cai and Zhanguo Wei. Diversity-generated image inpainting with style extraction. arXiv preprint (rXiv:1912.01834 2019.
Ting Chen, Mario Lucic, Neil Houlsby, Sylvain Gelly. self modulation generative adversarial networks. In International Conference on Learning Representations, 2019. URL httpth::///ppeeeeeeeeeee
net / forum i d= 15oR55m
Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Woo Ha, Sunghun Kim, and Jacgul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8789-8797, 2018.
Soheil Darabi, Eli Shechtman, Connelly Barnes, Dan Goldman, and Pradeep Sen. Image melding: Combining inconsistent images using patch-based synthesis. ACM Trans. Graph., 3 (4)::22--, 2012.
Terrance DeVries, Adriana Romero, Luis Pineda, Graham Taylor, Michal Drozdzal. On the evaluation conditional gans. arXiv preprint wxXiv:190.008133 2019.
Ding Ding, Sundaresh Ram, and Jeffrey J Rodriguez. Image inpainting using nonlocal texture matching and nonlinear filtering. IEEE Transactions on Image Processing. (8(44)1705-11 1719, 2018.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. arXiv preprint 1rX.v26610.07629, 2016.
Alexei A Efros and William T Freeman. Image quilting for texture synthesis and transfer. In Proceedings of the
28th annual conference on Computer graphics and interactive techniques, pp. 341-346. ACM, 2001.
Alexci A Efros and Thomas K Leung. Texture synthesis by non- parametric sampling. In Proceedings of the seventh IEEE international conference computer vision, volume 2, pp.
IEEE, 1999.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672 2680, 2014.
Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R Venkatesh Babu. Deligan: Generative adversarial networks for diverse and limited data. In Proceedings of the IEEE conference computer vision and puttern recognition, pp. 166-174, 2017.
Martin Heusel, Hubert Ramsauer. Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
10

---

Published as a conference paper at ICLR 2021
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1501-1510, 2017.
Xun Huang, Ming-Yu Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 172-189 2018.
Satoshi Tizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics (ToG), 36(4):107, 2017.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image- to to-image translation with conditional adversarial networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Jianmin Jiang. Hossam M Kasem, and Kwok-Was Hung. Robust image completion via deep feature transforma- tions. IEEE Access, 7:113916-119900 2019.
Youngjoo Jo and Jongyoul Park. Sc-fegan: Face editing generative adversarial network with user's sketch and color. arXiv preprint (Xiv:1902.06885 2019.
Tero Karras, Samuli Laine, and Timo Aila. A style-hased generator architecture for generative adversarial networks. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 4401- 4410, 2019a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. arXiv preprint arXiv:1912.04958 2019b.
Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwanghee Lee. U-gat-it: unsupervised generative atten- tional networks with adaptive layer-instance normalization image-to-image translation. arXiv preprint 1xXiv:190,,10800 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Rolf Köhler, Christian Schuler, Bernhard Schölkopf, and Stefan Harmeling. Mask-specific inpainting with deep neural networks. In German Conferencee on Pattern Recognition, pp. 523-534. Springer, 2014.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, pp- 3929-3938, 2019.
Avisek Lahiri, Arna Jain, Prabir Kumar Biswas, and Pabitra Mitra, Improving consistency and correctness of se- quence inpainting using semantically guided generative adversarial network. arXiv preprint arXiv:1711.066000 2017.
Avisek Lahiri, Arnay Kumar Jain, Sanskar Agrawal, Pabitra Mitra, and Prabir Kumar Biswas. Prior guided gan based semantic inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13696-13705, 2020.
Justin Lazarow, Long Jin, and Zhuowen Tu. Introspective neural networks for generative modeling. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2774-2783, 2017.
Jingyuan Li, Ning Wang, Lefei Zhang, Bo Du, and Dacheng Tao. Recurrent feature reasoning for image inpainting. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 7760-7768, 2020.
Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference Computer Vision (ECCV), pp. 85-100, 2018
Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang. Coherent semantic attention for image inpainting. arXiv preprint arXiv: 12384, 2019a.
Hongyu Liu, Bin Jiang, Yibing Song, Wei Huang, and Chao Yang. Rethinking image inpainting via a mutual encoder-decoder with feature equalizations. arXiv preprint arXiv:2007.06929, 2020.
Ming-Yo Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Advances in neural information processing systems, pp. 700-708, 2017.

---

Published as a conference paper at ICLR 2021
Ming-Yu, Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-image translation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 115510, 2019b.
David Lopez-Paz and Maxime Oquab, Revisiting classifier two-sample tests. arXiv preprint 111.v:16 2016.
Yuqing Ma, Xianglong Liu, Shihao Bai, Lei Wang, Aishan Liu, Dacheng Tao, and Edwin Hancock. Region-wise generative adversarial imagcinpainting for large missing areas. arXiv preprint 19Xiv:199..12555 2019.
Qi Mao, Hsin-Ying Lee, Hung-Yi Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking generative adversarial networks for diverse image synthesis. In Proceedings of the IEFF Conference Computer Vision and Pattern Recognition, pp. 1429-1437, 2019.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? arXiv preprint arXiv:1801, 1801.04406. 2018.
Kamyar Nazeri, Eric Ng. Tony Joseph, Faisal Qureshi, and Mchran Ebrahimi. Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv preprint 1901:0000 2019.
Augustus Odena, Jacob Buckman, Catherine Olsson, Tom Brown, Christopher Olah, Colin Raffel, and Ian Goodfellow. Is generator conditioning causally related to gan performance? arXiv preprint arXiv: 1802.08888. 2018.
Taesung Park, Ming- Yu Liu, Ting-Chur Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2337-2346, 2019.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
Pengda Qin, Weiran Xu, and William Yang Wang. Dsgan: Generative adversarial training for distant supervision relation extraction. arXiv preprint 185.:0:900509999 2018.
Jimmy SJ Ren, Li Xu, Qiong Yan, and Wenxiu Sun. Shepard convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 901-909, 2015.
Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li. Structureflow: Image inpainting via structure- aware appearance flow. In Proceedings of the IEEE International Conference on Computer Vision, pp. 199, 2019.
Mark Sabini and Gili Rusak. Painting outside the box: Image outpainting with gans. arXiv preprint 108.:08808.0888. 2018.
Min-cheol Sagong, Yong-goo Shin, Seung-wook Kim, Seung Park, and Sung-jea Ko. Pepsi: Fast image inpainting with parallel decoding network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11360-11332 2019.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems, pp. 1228-5237, 2018.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization accelerate training of deep neural networks. Advances in neural information processing systems, pp. 901-909, 2016.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, In Advances in neural information processing systems, pp. 2234-2242, 2016.
Jake Snell, Karl Ridgeway, Renjie Liao, Brett D Roads, Michael C Mozer, and Richard Zemel. Learning to generate images with perceptual similarity metrics. In 2017 IEEE International Conference on Image Processing (ICIP), pp. 4277 -2811 IEEE, 2017.
Yi Wang, Xin Tao, Xiaojuan Qi. Xiaoyong Shen, and Jiaya Jia. Image inpainting via generative multi-column convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 331-340, 2018.
Yi Wang, Xin Tao, Xiaoyong Shen, and Jiaya Jia. Wide-context semantic image extrapolation. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 1399-1408, 2019a.
12

---

Published as a conference paper at ICLR 2021
Ze Wang. Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. Stochastic conditional generative networks with basis decomposition. arXiv preprint arXi::1909.1/286 2019b.
Sitao Xiang and Hao Li, On the effects batch and weight normalization in generative adversarial networks. arXiv preprint XXi::700.009771 2017.
Chaohao Xie, Shaohui Liu, Chao Li, Ming-Ming Cheng, Wangmeng Zuo, Xiao Liu, Shilei Wen, and Errui Ding, Image inpainting with learnable bidirectional attention maps. In Proceedings of the IEEE International Conference on Computer Vision, pp. 8858-8867, 2019.
Junyuan Xie, Linli Xu, and Enhong Chen. Image denoising and inpainting with deep neural networks. In Advonces neural information processing systems, pp. 341-349, 2012.
Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, and Jicbo Luo. Foreground-aware image inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5840-5848, 2019.
Zhaoyi Yan, Xiaoming Li, Mu Li, Wangmeng Zuo, and Shiguang Shan. Shift-net: Image inpainting via deep feature rearrangement. Proceedings of the European Conference on Computer Vision (ECCV), pp. 1 1-17, 2018.
Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. High-resolution image inpainting using multi-scale neural patch synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6721-6729, 2017.
Yang Yang. Xiaojie Guo, Jiayi Ma, Lin Ma. and Haibin Ling. Lafin: Generative landmark guided face inpainting. arXiv preprint arXiv:1911.1 1194, 2019a.
Zongxin Yang, Jian Dong. Ping Liu. Yang, and Shuicheng Yan. Very long natural scenery image prediction by outpainting. In Proceedings of the IEEE International Conference on Computer Vision, pp. 10561-10570, 2019b.
Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE international conference on computer vision, pp. 2849-2857, 2017.
Jiahui Ya, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pp. 5505-5514, 2018.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4471-4480, 2019.
Tao Yu, Zongyu Guo, Xin Jin, Shilin Wa, Zhibo Chen, Weiping Li, Zhizheng Zhang, and Sen Liu. Region Normalization for Image Inpainting. arXiv e-prints, art. 1911: Nov 2019.
Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Learning pyramid- pyramid-contex context encoder network for high-quality image inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2019.
Lei Zhao, Qihang Mo. Sihuan Lin. Zhizhong Wang, Zhiwen Zuo, Haibo Chen, Wei Xing, and Dongming La. Uctgan: Diverse image inpainting based on unsupervised cross-space translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5741-5750, 2020a.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. Advances in Neural Information Processing Systems, 33, 2020b.
Chuanxia 7heng, Tat-Jen Cham, and Jianfei Cai. Pluralistic image completion. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, pp. 1438-1447, 2019.
Bolci Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: 10 million image database for scene recognition. JEEE transactions on pattern analysis and machine intelligence, 40(6): 1452-1464, 2017.
Tong Zhou, Changxing Ding. Shaowen Lin, Xinchao Wang. and Dacheng Tao. Learning oracle attention for high-fidelity face completion. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pottern Recognition, pp. 7680-7689, 2020.
13

---

Published as a conference paper at ICLR 2021
Jun-Yan Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp.
2017a.
Jun- Van Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in neural information processing systems, pp. 465-476, 2017b.
APPENDIX A IMPLEMENTATION DETAILS
We mostly borrow the network details and hyperparameters from StyleGAN2 (Karras et al. 2019b), including the number of convolutional layers (2) at each level, the number of channels (64 at 5 x12x112 resolution, doubled each coarser level with a maximum of 512), architecture of the mapping network M (8-layer MLP), layer-wise noise injection, style mixing regularization (with probability of 0.5 instead), non-saturating logistic loss (Goodfellow et al. 2014) with R1 regularization (Mescheder al. 2018 of y 10, and the Adam optimizer (Kingma & Ba 2014) with a learning rate of 0.002.
Our conditional encoder imitates a similar architecture as the discriminator but without the cross- level residual connections. Skip residual connections are used between each level of & and D. To produce the conditional style representation, the final 4x feature map of is flattened and passed through fully connected layer of 1024 channels with dropout rate of 0.5. The dropout layer keeps enabled during testing since we observe that it partially correlates to the inherent stochasticity.
Our model has 109M parameters in total. All the experiments V100 GPUs The batch size is per GPU, 32 in total. The training length is 25M images unless specified, which takes about week 512x512 resolution.
on cards of NVIDIA Tesla
APPENDIX B FREE-FORM MASK SAMPLING
Figure 12: Random samples of free-form masks.
We sample free-form masks for training by simulating random brush strokes and rectangles. The algorithm of generating brush strokes is borrowed from DeepFillv2 Yu et al. 2019), while the width of the brush is uniformly sampled within [12, 48), the number of vertices is uniformly sampled within [4, 18) and the number of strokes uniformly sampled within [0, 20]. We then generate multiple rectangles with uniformly random widths, heights, and locations, while the number of up to full-size rectangles is uniformly sampled within [0, 5] and the number of up to half-size rectangles is uniformly sampled within [o, 10). See Fig. 12 for the sampled free-form masks. During evaluation, we use the same sampling strategy of free-form masks used in training if no masked ratio is specified; otherwise, we repeatedly apply the same algorithm until the specified range is satisfied.
APPENDIX C USER STUDY
For the user study of image completion, we randomly sample the same number (256) of validation images, free-form masks (using the algorithm above), and the corresponding outputs from each
14

---

Published as a conference paper at ICLR 2021
(wrs(r= =1)
Ours (w 3)
Masked
Original
Figure 13: Samples on Edges2Handbags.
Figure 14: Failure cases.
method, for each dataset and each range of masked ratio. The user is given a pair of fake and the corresponding real images each round and has seconds to decide which one is fake or "don't know": overtime rounds are also treated "don't 't know". No user will To compute the user preference rate of fakes over reals, we regard a correct answer as 0, an incorrect answer I, and "don't know" 0.5. We have received totally 14336 rounds of answers from 28 participants. See Table 4 for the numerical results.
real for more than once.
We adopt similar protocol for the user study COCO In each round, the user is given a pair of generated images of SPADE (Park et al.] 2019) and ours using the same validation input. The user has 5 seconds to decide which
preferred or "don't know": overtime rounds also treated as "don't t know". We regard "don't know" as 0.5. We have received 720 rounds answers from 12 participants, among which 319 prefer ours, 189 prefer SPADE, and 212 "don't know".
APPENDIX D MORE QUANTITATIVE RESULTS
Table presents the quantitative results for image completion across methods and masked ratios. Table presents the quantitative results of the ablation experiment. Experiments demonstrate our superior performance at all masked ratios.
APPENDIX E MORE QUALITATIVE RESULTS
Fig. 13 presents our generated samples for image-to-image to-image translation on the Edges2Handbags dataset under both v (which achieves superior fidelity) and 1 = 3 (which achieves superior diversity).
See Fig. [5] for a qualitative comparison for image-to-image translation on the COCO-Stuff dataset.
Extensive examples for free-form image completion are presented in Figs. T8 23
APPENDIX F DISCUSSION
Large scale image completion is challenging task that requires not only generative but also recogni- tion capability. Although our model generates promising results in most of the cases, sometimes fails to recognize the semantic information in the surrounding areas hence produces strange artifacts (see Fig. [14,, especially in the challenging Places2 dataset that contains millions of scenes under various style and quality. The readers encouraged to discover examples from interactive demo.
15

---

Published as a conference paper at ICLR 2021
Table 4: User study results for image completion at different masked ratios among PatchMatch (Barnes et al. 2009), DeepFillv2 (retrained) (Yu et al. 20119), and ours.
FFH
Places2
Method
(0, .2
(.2, .4)
(4).6)
(.,, (8)
(.8,
(2)
(.2, .4)
(.4,
(.6, .8)
(.8, 1)
PatchMatch
10.7%
2.7%
3.2%
1.9%
2.1%
15.7%
4.1%
3.1%
1.7%
3.1%
DeepFillv2 (rotrained)
40.6%
28.0%
17.9%
7.6%
2.7%
32.3%
19.1%
9.8%
3.4%
4.3%
Ours
47.9%
44.8%
37.8%
36.0 %
33.6%
40.3%
32.7%
18.7%
14.5%
13.8%
Table 5: Quantitative comparison for image completion different masked ratios among DeepFillv2 (official), DeepFillv2 (retrained) (Yu et al. 2019), and ours. Results are averaged over runs.
Masked
FFH
Places2
Method
Ratio
P-IDS (%)
U-IDS (%)
FID
P-IDS (%) U-IDS (%)
FID
DeepFillv2 (official)
6.4 0.1
+..
1.78
21.3
42.6
0.51 + 0,01
(0, .2
DeepFillv2 (etrained)
18.3 = 0.3
4.2
0.00
21.2
0.47 0.00
Ours
31.3 + 0.3
44.4
+ 0.3
0.54
33.3
46.2 16.2 0.0
0.33
DeepFillv2 (official)
0.0
0.1
1.11
3.1
0.1
0.003
(2,.4)
DeepFillv2 (mttainnd))
3.8
22.0
0.03
5.1
0.1
0.00
Ours
22.1 0.3
37.5
0.2
20.6 0.1
1.74 0.02
DeepFilly2
(officiall
0.0
0.0
0.01
0.5
0..
14.85 0.06
(6)
DecpFillv2 (retrained)
0.8
1.11
0.09
0.0
0.1
11.16 + 0.05
Ours
0.5
32.1 0.2
14.6 0.2
3.22
4.77 = 0.03
DeepFilly2 (official)
0.0
0.0 0.0
06.5810.11
0.1 0.0
3.0
35.81 0.14
.8)
DeepFillv2 (nttained)
0.1
3.2
18.02
0.5
25.75 0.0 0.06
Ours
15.2 0.3
27.9 +
4.13 0.06
11.5 0.1
24.9
9.64 + 0.04
DeepFillv2
0.0
0.0 0.0
179.12
0.0
0.0
71.72 0.16
(88, I) DeepFillv2 (xtrained)
0.0
0.0 0.0
44.63
0.1
51.52 2.0
0.27
Ours
13.7 - 0.2 24.1 + 0.2 5.52 - 0.08
9.8 0.1
19,8 - 0.0 15.27 0.06
Table 6: Ablation study among Vanilla, conditional modulation (C-Mod), and our co-modulation (Co-Mod), as illustrated in Figs. [(b) to 2[d] for image completion on the FFHQ dataset. Vanilla completely fails; our co-modulation dominates the conditional modulated version at all masked ratios and especially when the masked ratio becomes large. Results are averaged over 5 runs.
Masked Ratio
Method
P-IDS
U-IDS (%)
Vanilla
0.6 + 0.1
3.0
71.72 + 0.65
(0.0, 0.2)
C-Mod
30.7
4.2
0.55 = 0.01
Co-Mod
31.3 + 0.3
44.4 0.3
0.54 + 0.01
Vanilla
0.0
0.0
152.79 + 0.37
(0.2, 0.4)
C-Mod
21.1 0.2
3..9
1.78 + 0.02
Co-Mod
22.1 + 0.3
37.5 + 0.2
1.69 + 0.01
Vanilla
0.0 + 0.0
0.0 0.0
192,24 + 0.44
(0.4, 0.6)
C-Mod
16.3 i 0.3
30.9 0.3
3.03 + 0.02
Co-Mod
17.9 + 0.5
32.1 + 0.2
2.88 + 0.03
Vanilla
0.0 0.0
0.0 0.0
247.68 + 1.28
(0.6, 0.8)
C-Mod
13.6 1 0.2
26.2 0.1
4.42 1 0.02
Co-Mod
15.2 + 0.3
27.9
4.13 + 0.06
Vanilla
0.0 +:
0.0 0.0
286.53 + 0.93
(0.8, 1.0)
C-Mod
11.9 4
22.2 0.1
5.96 + 0.03
Co-Mod
13.7 +: 0.2
24.1 + 0.2
5.52 + 0.08
16

---

Published as a conference paper at ICLR 2021
Input
Real
SPADE
Ours
Input
Real
SPADE
Ours
Figure 15: Qualitative comparison between SPADE Park al. 2019) and ours for image-to-image
translation the COCO-Stuff validation set (Caesar et al. 2018) (labels to photos).
17

---

Published as a conference paper at ICLR 2021
Mask
original
ti
Figure 16: Our image completion results w.r.t. different masks. Our method successfully bridges differently conditioned situations, from small-scale inpainting to large-scale completion (left to right). The original images are sampled 512x512 resolution from the FFHQ dataset (Karras et al. 2019a) within a 10k validation split.
18

---

Published as a conference paper at ICLR 2021
Mask
Original
Figure 17: Our image completion results w.r.t. different masks. Our method successfully bridges differently conditioned situations, from small-scale inpainting to large-scale completion (left to right). The original images are sampled at 512x512 resolution from the Places2 validation (Zhou et 2017).
19

---

Published as a conference paper at ICLR 2021
Masked
Original
Ours
Qualitative examples for image completion at 1024 x 1024 resolution. The original images are sampled from the FFHQ dataset (Karras et al.) (019a) within a 10k validation split.
Figure 18:
20

---

Published as a conference paper at ICLR 2021
Masked
Original
Ours
Qualitative examples for image completion at 1024 x 1024 resolution. The original images are sampled from the FFHQ dataset (Karras et al.) (019a) within 10k validation split.
Figure 19:
21

---

Published as a conference paper at ICLR 2021
Masked
Original
PatchMatch
DeepFillvz (offica))
DeepFillv2 (ratralned)
Figure 20: Qualitative examples for image completion among PatchMatch (Barnes et al. 2009) DeepFillv2 (Yu et al. 2019), and ours. The original images are sampled at 512x512 resolution from the FFHQ dataset [Karras al. 2019a) within 10k validation split.
22

---

Published as a conference paper at ICLR 2021
Masked
Original
PatchMatch
DeepFillyz (offical)
DeepFillv2 (ratrabne)
Figure 21: Qualitative examples for image completion among PatchMatch (Barnes et al. 2009), DeepFillv2 (Yu et al. 2019), and ours. The original images are sampled at 512x512 resolution from the FFHQ dataset (Karras al. 2019a) within 10k validation split.
23

---

Published as a conference paper at ICLR 2021
Masked
Original
PatchMatch
DeepFillva (offca))
DeepFillv2 (retraked)
Ours
Figure 22: Qualitative examples for image completion among PatchMatch (Barnes et al. 2009). DeepFillv2 (Yu et al. 2019). and ours. The original images are sampled at 512x512 resolution from the Places2 validation set (Zhou et al. (017).
24

---

Published as a conference paper at ICLR 2021
Masked
Original
PatchMatch
DeepFillva (offca))
DeepFillv2 (revakned)
Figure 23: Qualitative examples for image completion among PatchMatch (Barnes et al. 2009); DeepFillv2 (Yu al. 2019), and ours. The original images are sampled at 512x512 resolution from the Places2 validation set (Zhou et al. (017).
25


Reviews:
[{"review_id": "xfRyj61TeYi", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an image completion method that can deal with large-scale missing regions. The proposed method employs a co-modulation technique to bridge the gap between conditional and unconditional GANs. It can then take the advantages from both sides with the consistency offered by conditional GANs and stochasticity provided by the unconditional GANs. The paper also proposes new image quality metrics, Paired/Unpaired Inception Discriminative Scores (P-IDS/U-IDS), for measuring the image quality of the inpainted images. Experiments show that the proposed method significantly outperforms DeepFillv2, a state-of-the-art freeform image completion method, on examples with large missing regions. Overall, I like the idea and the paper’s results. It represents a clear progress to the image completion problem with large missing regions. \n\n**Strengths**\n+ The idea is novel and makes sense. It is simple yet effective to the target problem. The idea of extending modulation to co-modulation is interesting. The co-modulation architecture design allows the proposed method to explore the generation capability of the unconditional GANs while maintaining the consistency of the completed image.\n+ The results are excellent. The paper and the appendix's visual results show that the proposed method can inpaint images with large missing regions very well. \n+ The paper is generally well-written. Figure 2 illustrates the basic idea very well. \n\n**Weaknesses**\n- Although the experiments show a good correlation of the proposed P-IDS/U-IDS metrics to the user study results, it is not clear how well the metrics reflect perceptual fidelity. Do these metrics work best for measuring the quality of inpainted images? How were the fake images generated for fitting the linear SVM? Is it possible to generalize to other image restoration problems?   \n- Although the main target is image completion, the paper claims that the proposed co-modulated GANs also works well on image-to-image translation. However, the paper only demonstrates the application on a simple edge2image problem, from edge images to commodity images. It would be better if the paper presents more image-to-image translation examples for validating how well the proposed method can handle image-to-image translation.\n\n**Post-rebuttal**\n\nThe rebuttal addresses the raised issues. An experiment supports the generalizability of the proposed metric. A new image-to-image translation experiment on COCO-Stuff is added. It shows the proposed method's potential for other translation tasks (the results are in Fig. 16, not Fig. 15). Taking the rebuttal and other reviews into account, I would like to recommend accepting the paper for its excellent results and simple yet effective idea. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "pmn_AxLb2Fz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes multiple contributions.\n\nThe paper identifies the following problem: current inpainting methods are suitable for small missing regions, but do not do well for large missing regions. I think the exposition is outdated and does not consider new work published at CVPR 2020, ECCV 2020, and possibly other venues.\nThe paper \"Recurrent Feature Reasoning for Image Inpainting\" explicitely points out \"However, filling in large continuous holes remains difficult due to the lack of constraints for the hole center.\" and the results seem to show many examples of inpaiting for large missing regions.\n\"Image2StyleGAN++ ...\" show the inpainting of large regions as application and specifically mention the large-scale inpainting problem.\n\"Image Fine-grained Inpainting\" states that \"Benefited from the property of this network, we can more easily recover large regions in an incomplete image\"\n\"DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting\" even mentions the problem of large missing regions in the title of the paper.\nThere are also other inpainting papers that one should look at. I didn't check the publication date or the relationship in detail at this point in time: \"Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations\", \"Deep Generative Model for Image Inpainting with Local Binary Pattern Learning and Spatial Attention\", \"Deep Generative Model for Image Inpainting with Local Binary Pattern Learning and Spatial Attention\", \"Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation\".\nMain concern for this submission is the literature research and I would like to see this addressed in the rebuttal.\n\nThe paper proposes an architecture modification the authors call co-modulation. The idea is to have the normalization of the generator layers not only controlled by either a random vector, or an input image, but by both. I think this overall idea is clear, but the details (including Fig. 3) are not that clear. I would say it's a nice, but smaller idea, that is suitable for publication if it leads to good results.\n\nThe paper also proposes a new way to evaluate GANs using the proposed P-IDS and U-IDS score. The main idea here is to use a pre-trained feature transformation (the inception network) on real and fake images and then to evaluate the images using a linear classifier.\nI do not think the evaluation strategy of masking a random square of size wxw is that convincing. A square of 1x1 is a single pixel and this is not a meaningful image manipulation. Even changing a square of 8x8 pixels is not especially meaningful. The fact that your metric can distinguish between these manipulations is not an indication that the metric can distinguish between high quality and low quality results.\nThe user study is more meaningful and gives some indication that the new metric is better. This looks promising and I like this result.", "extra_scores": {}, "extra_text": {}}, {"review_id": "j9J-CWUtDKf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a general approach for image completion with large-scale missing regions. The key is to combine image-conditional and modulated unconditional generative architectures via co-modulation. The presented approach has demonstrated strong performance in the image painting with large-scale missing pixels and some image-to-image translation tasks. A new metric P-IDS/U-IDS is proposed to evaluate the perceptual fidelity of inpainted images.\n\nStrength:\n- The idea of co-modulation is quite interesting and has demonstrated strong results in various tasks.\n- The paper is well-written and well-motivated.\n- The solution combines the best of two worlds in image-conditional and unconditional image generation.\n- A new metric P-IDS/U-IDS is proposed for perceptual evaluation, verified by the correlation to human preferences.\n\nWeakness:\n- Only one image inpainting DeepFillv2 is compared in the experiment. Other image inpainting methods such as gated convolution, partial convolution can be also evaluated.\n- The results in the main paper are mainly faces that have structured information. It would be good to move some outdoor results in the supplement to the main paper. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "NKC3GNljd_9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this article, authors proposes to bridge the gap between image-conditional and recent modulated unconditional generative architectures with a generic co-modulated gan architecture.\nThey also proposes a new inception score based on linear SVM in order to measure linear separability in a pre-trained feature space.\n\nThe paper is easy to read and the experimentation part supports the claims made in the theoritical sections.\n\nI have one question though: Why using a SVM in the computation of Paired/Unpaired Inception Discriminative Score (P-IDS/U-IDS) and not a simpler linear model ?\nHow the training of the SVM goes especially on large datasets ? (not in term of accuracy but in term of computation power)", "extra_scores": {}, "extra_text": {}}, {"review_id": "0Etby1AuhGL", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: Thanks for the response from the authors. The comments 3/4/5 from authors convincingly address my concerns. Regarding other classifier-based metrics, note that not all of them requires separate training and testing procedure. For example, the leave-one-out 1-NN accuracy [1] does not. Also, I'm still not sure about the technical novelty. Therefore, I keep my original rating.\n\n[1] Xu, Qiantong, et al. \"An empirical study on evaluation metrics of generative adversarial networks.\" arXiv preprint arXiv:1806.07755 (2018).\n\n---------\n\nThis paper proposes generator architecture for image inpainting using co-modulation, which is similar to the weight modulation in StyleGAN2 but is conditioned on both the input image and the stochastic variable instead of only the stochastic variable.\n\n\nPros:\n+ The paper is well-written and easy to read.\n+ The proposed method significantly outperforms the baseline DeepFillv2. The results look qualitatively convincing.\n\nCons: \n- The novelty of this paper is a bit limited. It seems to me that the proposed co-modulation is a straightforward extension of conditional modulation with stochasticity. \n- It seems most of the improvement over DeepFillv2 is from the better and larger network backbone (StyleGAN2). The improvement from conditional modulation is a bit small (Table 5 in the appendix).\n- In Figure 5, the authors argue that KID is subject to huge variance. However, it is not clear to me if KID really has a larger variance than FID. Figure 6 shows that FID and KID have very similar (relative) variance. It is argued that KID can hardly distinguish w=1 vs w=2. But from Figure 4 right, it seems to me that FID has the same problem too.\n \nQuestions for the rebuttal:\n- What is the running time of the proposed classification-based metrics?\n- The proposed co-modulation follows the “late-fusion” strategy: processing the image and random information independently, then concatenate them before modulation. Is it helpful to fuse them earlier, using an MLP to process the concatenated information before modulation?\n- The paper argues that \"however, to the best of our knowledge, we are the ﬁrst to formulate the discriminability as ψ a simple scalable metric\" when discussing other classifier-based metrics. I'm not totally convinced why the proposed metric is more scalable than other ones, e.g., the two-sample test. It would be great if you could elaborate on that.\n\nOverall, I find the empirical results of the paper are pretty impressive. The technical novelty is however a bit lacking. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received two clear accept, one accept, one borderline accept and one reject review. R4 identified that the paper falls short in discussing recent works from CVPR and ECCV 2020 on the image inpainting and completion tasks which also tackle challenging scenarios in these tasks. The authors improve their related work section with these more recent works while pointing out that the task still remains unsolved and they propose an effective technique towards the solution. The meta reviewer recommends acceptance based on the following observations. 

The submission proposes a GAN architecture for image inpainting using co-modulation, which is similar to the weight modulation in StyleGAN2 but is conditioned on both the input image and the stochastic variable instead of only the stochastic variable. The main novelty of co-modulation appears to be interesting as well as being generalisable to different tasks. The approach is shown to perform well in the image painting with large-scale missing pixels and some image-to-image translation tasks. Furthermore a new metric P-IDS/U-IDS is proposed to evaluate the perceptual fidelity of inpainted images. 
Decision: Accept (Spotlight)