High Resolution and Fast Face Completion via Progressively Attentive GANs

1 INTRODUCTION . Image completion is a technique to replace target regions , either missing or unwanted , of images with synthetic content so that the completed images look natural , realistic and appealing . Two types of methods have been used : data similarity driven methods and data distribution based generative methods . In the first paradigm , texture synthesis or patch matching are usually used ( Efros & Leung , 1999 ; Kwatra et al. , 2003 ; Criminisi et al. , 2003 ; Wilczkowiak et al. , 2005 ; Komodakis , 2006 ; Barnes et al. , 2009 ; Darabi et al. , 2012 ; Huang et al. , 2014 ; Wexler et al. , 2007 ) . The second paradigm learns the underlying distribution governing the data generation with respect to the context and is able to synthesize novel content . Much progress ( Iizuka et al. , 2017 ; Yeh et al. , 2017 ; Li et al. , 2017 ; Yang et al. , 2016 ; Denton et al. , 2016 ; Pathak et al. , 2016 ; Yu et al. , 2018 ; Liu et al. , 2018 ) has been made since the generative adversarial network ( GAN ) was proposed ( Goodfellow et al. , 2014 ) . We adopt the data distribution based generative method and focus on human face completion in this paper . Three important issues are addressed . First , previous methods are only able to complete faces at low resolutions ( e.g . 176 × 216 ( Iizuka et al. , 2017 ) and 256 × 256 ( Yu et al. , 2018 ) ) . Second , most approaches can not control the attributes of the synthesized content . Previous methods focus on generating random realistic content . However , users may want to complete the missing parts with certain properties ( e.g . expressions ) . Third , most existing approaches ( Iizuka et al. , 2017 ; Yeh et al. , 2017 ; Li et al. , 2017 ) require post processing ( e.g . Poisson Blending ( Pérez et al. , 2003 ) ) or complex inference process ( e.g . thousands of optimization iterations ( Yeh et al. , 2017 ) or repeatedly feeding an incomplete image to CNNs at multiple scales ( Yang et al. , 2016 ) ) . To overcome the above limitations , we propose a novel progressively attentive GAN to complete face images at high resolution with multiple controllable attributes in a single forward pass without any post processing . We utilize facial landmarks as backbone guidance of face structures and propose a straightforward method of integrating them in our system . The training methodology of growing GANs progressively ( Karras et al. , 2017 ) is used to generate high-resolution images end-to-end . To avoid distorting the learned coarse structures when the network is growing to a higher resolution , we design a novel Frequency-oriented Attentive Module ( FAM ) to encourage the model to attend on finer details ( i.e . higher-frequency structures , see Figure 1 ) . A conditional version of our network is designed so that the appearance properties ( e.g . male or female ) , and facial expressions of the synthesized faces can be controlled . Moreover , we design a set of loss functions inducing the network to blend the synthesized content with the contexts in a realistic way . Our method was compared with state-of-the-art approaches on a high-resolution face dataset CelebA-HQ ( Karras et al. , 2017 ) . Both the evaluations and a pilot user study showed that our approach completed face images significantly more naturally than existing methods . The main contributions of this paper are : ( i ) We propose a progressively attentive GAN architecture , which incorporates a novel frequency-oriented attention mechanism , to complete face images with random masks in much higher resolution than existing methods . ( ii ) A conditional version of our model is designed to control multiple attributes of the synthesized content . ( iii ) Our framework is able to complete images in a single forward pass , without any post-processing , and thus fast . 2 RELATED WORK . There is a large body of image completion literature . Early non-learning based algorithms ( Efros & Leung , 1999 ; Bertalmio et al. , 2000 ; 2003 ) complete missing content by propagating information from known neighborhoods , based on low level cues or global statistics ( Levin et al. , 2003 ) . Texture synthesis and patch matching based approaches ( Efros & Leung , 1999 ; Kwatra et al. , 2003 ; Criminisi et al. , 2003 ; Wilczkowiak et al. , 2005 ; Komodakis , 2006 ; Barnes et al. , 2009 ; Darabi et al. , 2012 ; Huang et al. , 2014 ; Wexler et al. , 2007 ) find similar structures from the context of the input image or from an external database ( Hays & Efros , 2007 ) and then paste them to fill in the holes . Recent learning based methods have shown the capability of CNNs to complete large missing content . Based on existing GANs , the Context Encoder ( CE ) ( Pathak et al. , 2016 ) encodes the contexts of masked images to latent representations , and then decodes them to natural content images , which are pasted into the original contexts for completion . However , the synthesized content of CE is often blurry and has inconsistent boundaries . Given a trained generative model , Yeh et al . ( Yeh et al. , 2017 ) propose a framework to find the most plausible latent representations of contexts to complete masked images . The Generative Face Completion model ( GFC ) ( Li et al. , 2017 ) and the Global and Local Consistent model ( GL ) ( Iizuka et al. , 2017 ) use both global and local discriminators , combined with post processing , to complete images more coherently . Built on GL , Yu et al . ( Yu et al. , 2018 ) design a contextual attention layer ( CTX ) to help the model borrow contextual information from distant locations . Liu et al . ( Liu et al. , 2018 ) incorporates partial convolutions to handle irregular masks . Unfortunately , these approaches can only complete face images in relatively low resolutions ( e.g . 176 x 216 ( Iizuka et al. , 2017 ) and 256× 256 ( Yu et al. , 2018 ) ) . Yang et al . ( Yang et al. , 2016 ) combine a global content network and a texture network , and the networks are trained at multiple scales repeatedly to complete high-resolution images ( 512× 512 ) . Like the patch matching based approaches , Yang et al . assume that the missing content always shares some similar textures with the context , which is improbable for the face completion task . Many methods try to generate high quality images and stabilize the training process . The Laplacian GAN ( Denton et al. , 2015 ) uses a a cascade of CNNs to synthesize images from coarse to fine by generating high frequency information at different layers . The Deep Recurrent Attentive Writer ( DRAW ) architecture ( Gregor et al. , 2015 ) , which is a spatial attention mechanism , generates images iteratively by learning to read and write parts of images at each time-step with a sequence of variational auto-encoders ( VAEs ) . Unfortunately , these techniques are unable to synthesize high-resolution images ( e.g . 64× 64 ( Gregor et al. , 2015 ; Denton et al. , 2015 ) ) . Karras et al . ( Karras et al. , 2017 ) put forward a progressive training methodology ( Progressive GAN ) to grow GANs from low to high resolution , and are able to generate realistic 1024× 1024 images . However , since all the parameters remain trainable at the growing stage , learned coarse structures can be altered and distorted , and thus the training process of Progressive GANs is usually unstable . Note that these generative models can not be applied to the image completion task directly because they aim at generating natural content which are not necessarily consistent with the image contexts . 3 APPROACH . 3.1 PROBLEM FORMULATION . Denoted by Λ an image lattice ( e.g. , 1024 × 1024 pixels ) . Let IΛ be an RGB image defined on the lattice Λ. Denote by Λt and Λc the target region to complete and the remaining context region respectively which form a partition of the lattice.Without loss of generality , we assume Λt is a single connected component region.IΛt is masked out with the same gray pixel value . Let MΛ be a binary mask image with all pixels in MΛt being 1 and all pixels in MΛc being 0 . For notational simplicity , we will omit the subscripts Λ , Λt and Λc when the text context is clear . The objective of image completion is to generate a synthesized image Isyn that looks natural , realistic and appealing for an observed image Iobs with the target region IobsΛt masked out in the ground-truth full image Igt . Furthermore , it is desirable to control the completion according to a set of attributes which are assumed to be independent from each other ( e.g. , to respect the underlying intrinsic ambiguities due to the loss of information in the target region ) . Let A = ( a1 , · · · , aN ) be a N -dim vector with ai ∈ { 0 , 1 } encoding if a corresponding attribute appears ( ai = 1 ) or not ( ai = 0 ) ( e.g . the “ Male ” attribute in Figure 5 ) . We define the generator as , Isyn = G ( Iobs , M , A ; θG ) , subject to I syn Λc ≈ IobsΛc ( 1 ) where θG collects all parameters of the generator , and ≈ represents the two context regions , IsynΛc and IobsΛc , need to kept very similar ( both to be elaborated later ) . 3.2 THE PROPOSED METHOD . The proposed method is built on progressive GANs with well-executed combination between the network architecture , appropriate receipt of loss functions and a novel FAM . Denote by Gr and Dr the generator and discriminator at resolution r respectively , where r ∈ { 1 , · · · , R } is the index of resolution ( e.g. , r = 1 represents 4 × 4 and r = R = 9 represents 1024× 1024 ) . Accordingly , we have the observed corrupted image , its corresponding binary mask and its ground-truth uncorrupted image ( in training only ) , Iobsr , Mr and I gt r at each resolution . The generator , Gr takes as input the observed data XGr and the attribute vector A , then outputs a completed image Isynr . It consists of two components , Isynr = Gr ( X G r , A ; θGr ) = G compl r ( G enc r ( X G r ; θGencr ) , A ; θGcomplr ) , ( 2 ) where Gencr ( · ) encodes the input XGr to a latent low dimensional vector . The latent vector is concatenated with the attribute vector . The concatenated vector plays the role of the noise random variable z in the original GAN . Then , Gcomplr ( · ) transforms the concatenated vector to a sample Isynr ( i.e. , the completed image ) . Gencr and G compl r are mirrored to each other in the form of U-shape networks ( Ronneberger et al. , 2015 ; Newell et al. , 2016 ) . θGr = ( θGencr , θGcomplr ) . The discriminator , Dr classifies its input XDr and has two output branches , the fake versus real classification branch and the attribute prediction branch . It consists of three components : a shared feature backbone and two head classifiers . We have , Dr ( X D r ; θDr ) = { Dclsr ( Fr ( XDr ; θFr ) ; θDclsr ) , D attr r ( Fr ( XDr ; θFr ) ; θDattrr ) } ( 3 ) Where the feature backbone , Fr ( XDr ; θFr ) computes the feature map . On top of the feature map , the first head classifier , Dcls ( · ) computes binary classification between real and fake , and the second one , Dattr ( · ) predicts an attribute vector . All the parameters of the discriminator are collected by θDr = ( θFr , θDclsr , θDattrr ) . We note that the discriminator is only needed in training . We will omit the notations for parameters in equations when no confusion is caused . Progressive Growing of Generators and Discriminators . Following the methodology proposed in ( Karras et al. , 2017 ) , we start with training G1 and D1 . Then , at the growing stage r ( r > 1 ) , Gr = ( Gr−1 , G fade−in r ) is first created on top of the previous trained stage Gr−1 with newly added layers Gfade−inr , and both of them are trainable in a fade-in process for smooth transition . The discriminators grow in the same way . The alternation of Gr−1 in training stage Gr may lead to the instability of the growing process and the loss of learned structures of the underlying image distribution , which motivates the proposed FAM in growing generators ( Section 3.2.1 ) . Inputs for Generators and Discriminators , XGr and X D r . We have , XGr = ( Î obs r , Mr , Lr ) and X D r = ( Ir , Lr ) ( 4 ) where Lr represents the facial landmarks . Recent works ( Isola et al. , 2016 ; Wang et al. , 2017 ; Zhu et al. , 2017 ; Sangkloy et al. , 2017 ; Xian et al. , 2017 ; Chen & Hays , 2018 ) have shown the capability of GANs to translate sketches or edges to photo-realistic images . Given a corrupted image , it is better if the model is able to “ draw ” a sketch of face first , which provides a backbone guidance for image completion . We utilize the following methods to compute landmarks ( Figure 2 ) . • In training , we extract landmarks from the uncorrupted image at 256 × 256 resolution using an off-the-shelf pre-trained Face Alignment Network ( FAN ) ( Bulat & Tzimiropoulos , 2017 ) which achieved very good results for faces in the wild . • In testing , to predict landmarks from corrupted images , we first train a single stage face completion network at 256× 256 resolution , denoted by GL , using reconstruction loss ( Section 3.2.2 ) only . For a testing image , we use GL to generate a blurry completed image from which the landmarks are extracted with FAN . L is up-sampled or down-sampled to Lr to match the size of networks at different resolutions . Ir in XDr represents either the uncorrupted image or the image synthesized by Gr . For Î obs r in X G r , we have Îobs1 = I obs 1 , and Î obs r is the output of FAM ( r > 1 ) as described in following section .

Summary: The paper proposes a complex generative framework for image completion (particularly human face completion). It aims at solving the following challenges: 1) complete the human face at both low and high resolution; 2) control the attribute of the synthetic content; 3) without the need of complex post-processing. To achieve so, this paper proposes a progressively attentive GAN to complete face image at high resolution with multiple controllable attributes in a single forward pass without post-processing. Particularly it introduces a frequency-oriented attentive module (FAM) to attend on finer details.