Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

1 INTRODUCTION . Generative adversarial networks ( GANs ) have received a great amount of attention in the past few years , during which a fundamental problem emerges from the divergence of development between image-conditional and unconditional GANs . Image-conditional GANs have a wide variety of computer vision applications ( Isola et al. , 2017 ) . As vanilla U-Net-like generators can not achieve promising performance especially in free-form image completion ( Liu et al. , 2018 ; Yu et al. , 2019 ) , a multiplicity of task-specific approaches have been proposed to specialize GAN frameworks , mostly focused on hand-engineered multi-stage architectures , specialized operations , or intermediate structures like edges or contours ( Altinel et al. , 2018 ; Ding et al. , 2018 ; Iizuka et al. , 2017 ; Jiang et al. , 2019 ; Lahiri et al. , 2020 ; Li et al. , 2020 ; Liu et al. , 2018 ; 2019a ; 2020 ; Nazeri et al. , 2019 ; Ren et al. , 2019 ; Wang et al. , 2018 ; Xie et al. , 2019 ; Xiong et al. , 2019 ; Yan et al. , 2018 ; Yu et al. , 2018 ; 2019 ; Yu et al. , 2019 ; Zeng et al. , 2019 ; Zhao et al. , 2020a ; Zhou et al. , 2020 ) . These branches of works have made significant progress in reducing the generated artifacts like color discrepancy and blurriness . However , a serious challenge remains that all existing algorithms tend to fail when handling large-scale missing regions . This is mainly due to their lack of the underlying generative capability — one can never learn to complete a large proportion of an object so long as it does not have the capability of generating a completely new one . We argue that the key to overcoming this challenge is to bridge the gap between image-conditional and unconditional generative architectures . ∗Corresponding author Recently , the performance of unconditional GANs has been fundamentally advanced , chiefly owing to the success of modulation approaches ( Chen et al. , 2019 ; Karras et al. , 2019a ; b ) with learned style representations produced by a latent vector . Researchers also extend the application of modulation approaches to image-conditional GANs with the style representations fully determined by an input image ( Park et al. , 2019 ; Huang et al. , 2018 ; Liu et al. , 2019b ) ; however , the absence of stochasticity makes them hardly generalizable to the settings where only limited conditional information is available . This limitation is fatal especially in large scale image completion . Although some multi-modal unpaired image-to-image translation methods propose to encode the style from another reference image ( Huang et al. , 2018 ; Liu et al. , 2019b ) , this unreasonably assumes that the style representations are entirely independent of the conditional input and hence compromises the consistency . Therefore , we propose co-modulated generative adversarial networks , a generic approach that leverages the generative capability from unconditional modulated architectures , embedding both conditional and stochastic style representations via co-modulation . Co-modulated GANs are thus able to generate diverse and consistent contents and generalize well to not only small-scale inpainting but also extremely large-scale image completion , supporting both regular and irregular masks even with only little conditional information available . See Fig . 1 for qualitative examples . Due to the effectiveness of co-modulation , we do not encounter any problem suffered in the image completion literature ( Liu et al. , 2018 ; Yu et al. , 2019 ) , successfully bridging the long-existing divergence . Another major barrier in the image completion literature is the lack of good quantitative metrics . The vast majority of works in this literature seek to improve their performance in terms of similarity-based metrics that heavily prefer blurry results , e.g. , L1 , L2 , PSNR , and SSIM , among which many state that there are yet no good quantitative metrics for image completion ( Liu et al. , 2018 ; Yu et al. , 2018 ; 2019 ) . The only gold standard in this literature is the user study , which conducts real vs. fake test giving a pair of images to subjects ( i.e. , the users ) . However , the user study is subject to large variance and costly , therefore lacking reproducibility . Inspired by the user study , we propose the new Paired/Unpaired Inception Discriminative Score ( P-IDS/U-IDS ) . Besides its intuitiveness and scalability , we demonstrate that P-IDS/U-IDS is robust to sampling size and effective of capturing subtle differences and further correlates well with human preferences . Our contributions are summarized as follows : • We propose co-modulated GANs , a generic approach that bridges the gap between imageconditional and recent modulated unconditional generative architectures . • We propose the new P-IDS/U-IDS for robust assessment of the perceptual fidelity of GANs . • Experiments demonstrate superior performance in terms of both quality and diversity in free-form image completion and easy generalization to image-to-image translation . 2 RELATED WORK . Image-Conditional GANs . Image-conditional GANs can be applied to a variety of image-to-image translation tasks ( Isola et al. , 2017 ) . The unpaired setting is also investigated when paired training data is not available ( Choi et al. , 2018 ; Huang et al. , 2018 ; Kim et al. , 2019 ; Lazarow et al. , 2017 ; Liu et al. , 2017 ; Yi et al. , 2017 ; Zhao et al. , 2020b ; Zhu et al. , 2017a ) . Recent works exploit normalization layers with learned style representations embedded from the conditional input or another reference image to enhance the output fidelity ( Huang et al. , 2018 ; Kim et al. , 2019 ; Liu et al. , 2019b ; Park et al. , 2019 ) . They can be regarded as a set of conditional modulation approaches , but still lack stochastic generative capability and hence poorly generalize when limited conditional information is available . Isola et al . ( 2017 ) initially find that the generator tends to ignore the noise input although they try to feed it , in contrast to unconditional or class-conditional GANs . A branch of works aims to enforce the intra-conditioning diversity using VAE-based latent sampling strategies ( Zhu et al. , 2017b ) or imposing distance-based loss terms ( Huang et al. , 2018 ; Mao et al. , 2019 ; Qin et al. , 2018 ) . Wang et al . ( 2019b ) also propose to decompose the convolution kernels into stochastic basis . However , the enforcement of diversity conversely results in the deterioration of image quality . Our co-modulation approach not only learns the stochasticity inherently but also makes the trade-off easily controllable . Image Completion . Image completion , also referred to as image inpainting when incapable of completing large-scale missing regions , has received a significant amount of attention . It is a constrained image-to-image translation problem but exposes more serious challenges . Traditional methods ( Ballester et al. , 2001 ; Barnes et al. , 2009 ; Darabi et al. , 2012 ; Efros & Freeman , 2001 ; Efros & Leung , 1999 ) utilize only low-level features and fail to generate semantically consistent contents . Then , ( Köhler et al. , 2014 ; Ren et al. , 2015 ; Xie et al. , 2012 ) adopt deep neural networks for image completion ; ( Pathak et al. , 2016 ) first exploits conditional GANs . Numerous follow-up works focus on the semantic context and texture , edges and contours , or hand-engineered architectures ( Altinel et al. , 2018 ; Ding et al. , 2018 ; Iizuka et al. , 2017 ; Jiang et al. , 2019 ; Jo & Park , 2019 ; Lahiri et al. , 2017 ; Liu et al. , 2019a ; Nazeri et al. , 2019 ; Ren et al. , 2019 ; Sagong et al. , 2019 ; Wang et al. , 2018 ; Xie et al. , 2019 ; Xiong et al. , 2019 ; Yan et al. , 2018 ; Yang et al. , 2017 ; 2019a ; Yu et al. , 2018 ; Yu et al. , 2019 ; Zeng et al. , 2019 ; Lahiri et al. , 2020 ; Zhao et al. , 2020a ; Li et al. , 2020 ; Zhou et al. , 2020 ) , among which ( Liu et al. , 2018 ; Yu et al. , 2019 ) introduce partial convolution and gated convolution to address free-form image completion . The lack of stochasticity is also observed in image completion ( Cai & Wei , 2019 ; Ma et al. , 2019 ; Zheng et al. , 2019 ) . Other works address the so-called outpainting subtasks ( Sabini & Rusak , 2018 ; Wang et al. , 2019a ; Yang et al. , 2019b ) . To our knowledge , none of these methods produce promising results in the presence of free-form large-scale missing regions . Evaluation Metrics . Great research interest has been drawn on the evaluation of GANs ( DeVries et al. , 2019 ; Gurumurthy et al. , 2017 ; Sajjadi et al. , 2018 ; Snell et al. , 2017 ; Xiang & Li , 2017 ) . Inception Score ( IS ) ( Salimans et al. , 2016 ) , and some other metrics like FCN-Score ( Isola et al. , 2017 ) , are specialized to the pre-trained task thus can not generalize . While FID ( Heusel et al. , 2017 ) is generally acceptable , few promising metrics for image completion exist . Previous works heavily rely on similarity-based metrics such as L1 , L2 , PSNR , and SSIM , which fail to capture stochastic Published as a conference paper at ICLR 2021 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formula : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 const uncondit ional modulat ion Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 For ula : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formula : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 ( a ) Fo mulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue ong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formula : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 3 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formula : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 ( b ) condit ional modulat ion Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 0 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Form las : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 0 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 3 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 0 2017011407 x y z D E M E ( y ) ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Form la : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on N v , 4 2019 Yue Dong YaoClass 70 2 17 1407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 3 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011 07 x y z D E M E ( y ) ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 ( c ) co-modulat ion Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 201 011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 201 011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Form la : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 3 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Form las : Instructed by Due n Nov , 4 2019 Yue Dong Ya Class 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Form la : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instruct d by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on N v , 4 2019 Yue Dong YaoClass 70 2 17 1407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 3 5 Formulas : Instructed by Due on Nov , 4 2019 Yue ong YaoClass 70 2017011 07 x y z D E M E ( y ) ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) M ( z ) s s′ A ψ = 1 3 5 Formulas : Instructed by Due on Nov , 4 2019 Yue Dong YaoClass 70 2017011407 x y z D E M E ( y ) ( z ) s s′ A ψ = 1 ψ = 3 ψ = 5 ( d ) Figure 2 : Illustration from modulation to co-modulation : ( a ) unconditional modulated generator ; ( b ) vanilla image-conditional generator ; ( c ) conditional modulated generator ; and ( d ) co-modulated generator . y , z represent the conditional input and the latent vector respectively ; E , D , M represent the conditional encoder , the generative decoder , and the mapping network , respectively . regions and are ill-fitted for GANs . Our proposed metric is also related to the classifier-based tests ( Blau & Michaeli , 2018 ; Lopez-Paz & Oquab , 2016 ) . However , previous classifier-based metrics require separate sets for training and testing the classifier , making them sensitive to the underlying generalizability of the trained classifier . We formulate the discriminability as a simple scalable metric for both the paired and unpaired ver ions without relying on the generalizability .

Summary: In this paper, the authors propose a general approach for image completion with large-scale missing regions. The key is to combine image-conditional and modulated unconditional generative architectures via co-modulation. The presented approach has demonstrated strong performance in the image painting with large-scale missing pixels and some image-to-image translation tasks. A new metric P-IDS/U-IDS is proposed to evaluate the perceptual fidelity of inpainted images.