Towards Controllable and Interpretable Face Completion via  Structure-Aware and Frequency-Oriented Attentive GANs

Face completion is a challenging conditional image synthesis task . This paper proposes controllable and interpretable high-resolution and fast face completion by learning generative adversarial networks ( GANs ) progressively from low resolution to high resolution . We present structure-aware and frequency-oriented attentive GANs . The proposed structure-aware component leverages off-the-shelf facial landmark detectors and proposes a simple yet effective method of integrating the detected landmarks in generative learning . It facilitates facial expression transfer together with facial attributes control , and helps regularize the structural consistency in progressive training . The proposed frequency-oriented attentive module ( FOAM ) encourages GANs to attend much more to finer details in the coarse-to-fine progressive training , thus enabling progressive attention to face structures . The learned FOAMs show a strong pattern of switching their attention from low-frequency to high-frequency signals . In experiments , the proposed method is tested on the CelebA-HQ benchmark . Experiment results show that our approach outperforms state-of-the-art face completion methods . The proposed method is also fast with mean inference time of 0.54 seconds for images at 1024× 1024 resolution ( using a Titan Xp GPU ) . 1 INTRODUCTION . Conditional image synthesis aims to learn the underlying distribution governing the data generation with respect to the given conditions/context , which is also able to synthesize novel content . Much progress ( Iizuka et al. , 2017 ; Yeh et al. , 2017 ; Li et al. , 2017 ; Yang et al. , 2016 ; Denton et al. , 2016 ; Pathak et al. , 2016 ; Yu et al. , 2018 ; Liu et al. , 2018 ; Brock et al. , 2018 ; Karras et al. , 2018 ) has been made since the generative adversarial networks ( GANs ) were proposed ( Goodfellow et al. , 2014 ) . Despite the recent remarkable progress , learning controllable and interpretable GANs for high-fidelity image synthesis at high resolutions remain an open problem . We are interested in controllable and interpretable GANs . We take a step forward by focusing on high-resolution and fast face completion tasks in this paper . Face completion is to replace target regions , either missing or unwanted , of face images with synthetic content so that the completed images look natural , realistic , and appealing . State-of-the-art face completion approaches using GANs largely focus on generating random realistic content . However , users may want to complete the missing parts with certain properties ( e.g . expressions ) . Controllability is entailed . Existing face completion approaches are usually only able to complete faces at relatively low resolutions ( e.g . 176 × 216 ( Iizuka et al. , 2017 ) and 256 × 256 ( Yu et al. , 2018 ) ) . To facilitate high-resolution image synthesis , the training methodology of growing GANs progressively ( Karras et al. , 2017 ) is widely used . For face completion tasks , one issue of applying progressive training is how to avoid distorting the learned coarse structures when the network is growing to a higher resolution . Interpretability is thus entailed to guide GANs in the coarse-to-fine pipeline . In addition , most existing approaches ( Iizuka et al. , 2017 ; Yeh et al. , 2017 ; Li et al. , 2017 ) require post-processing ( e.g . Poisson Blending ( Pérez et al. , 2003 ) ) , complex inference process ( e.g . thousands of optimization iterations ( Yeh et al. , 2017 ) or repeatedly feeding an incomplete image to CNNs at multiple scales ( Yang et al. , 2016 ) ) during test . We present structure-aware and frequency-oriented attentive GANs that are progressively trained for high-resolution and fast face completion using a fast single forward step in inference without any post-processing . By controllable , it means that the completed face images can have different facial attributes ( e.g. , smiling vs not smiling ) and/or facial expressions transferred from a given source actor . By interpretable , it means that the coarse-to-fine generation process in progressive training is rationalized . We utilize facial landmarks as backbone guidance of face structures and propose a straightforward method of integrating them in our system . We design a novel Frequency-Oriented Attention Module ( FOAM ) to induce the model to attend to finer details ( i.e . higher-frequency content , see Fig . 1 ) . We observe significant improvement of the completion quality by the FOAM against the exactly same system only without FOAM . A conditional version of our network is designed so that the appearance properties ( e.g . male or female ) , and facial expressions of the synthesized faces can be controlled . Moreover , we design a set of loss functions inducing the network to blend the synthesized content with the contexts in a realistic way . Our method was compared with state-of-the-art approaches on a high-resolution face dataset CelebA-HQ ( Karras et al. , 2017 ) . Both the evaluations and a pilot user study showed that our approach completed face images significantly more naturally than existing methods . 2 RELATED WORK . Recent learning based methods have shown the capability of CNNs to complete large missing content . Based on existing GANs , the Context Encoder ( CE ) ( Pathak et al. , 2016 ) encodes the contexts of masked images to latent representations , and then decodes them to natural content images , which are pasted into the original contexts for completion . However , the synthesized content of CE is often blurry and has inconsistent boundaries . Given a trained generative model , Yeh et al . ( Yeh et al. , 2017 ) propose a framework to find the most plausible latent representations of contexts to complete masked images . The Generative Face Completion model ( GFC ) ( Li et al. , 2017 ) and the Global and Local Consistent model ( GL ) ( Iizuka et al. , 2017 ) use both global and local discriminators , combined with post-processing , to complete images more coherently . Built on GL , Yu et al . ( Yu et al. , 2018 ) design a contextual attention layer ( CTX ) to help the model borrow contextual information from distant locations . Liu et al . ( Liu et al. , 2018 ) incorporates partial convolutions to handle irregular masks . Unfortunately , these approaches can only complete face images in relatively low resolutions ( e.g . 176× 216 ( Iizuka et al. , 2017 ) and 256× 256 ( Yu et al. , 2018 ) ) . Yang et al . ( Yang et al. , 2016 ) combine a global content network and a texture network , and the networks are trained at multiple scales repeatedly to complete high-resolution images ( 512× 512 ) . But , they assume that the missing content always shares some similar textures with the context , which is improbable for the face completion task . 3 THE PROPOSED METHOD 3.1 PROBLEM FORMULATION , Denote by Λ an image lattice ( e.g. , 1024×1024 pixels ) . Let IΛ be a face color image defined on the lattice Λ. Denote by Λt and Λctx the target region to complete and the remaining context region respectively ( note that the target region is not necessarily a single connected component , and the two parts form a partition of the lattice ) . IΛt is masked out with the same gray pixel value . LetMΛ be a binary mask image with all pixels inMΛt being 1 and pixels inMΛctx being 0 . For simplicity , we will omit the subscripts Λ , Λt and Λctx when the text context is clear . Unlike existing approaches ( Pathak et al. , 2016 ; Li et al. , 2017 ; Iizuka et al. , 2017 ) which first utilize unconditional image synthesis to generate the target region image and then blend them with context using using sophisticated post-processing , we address the completion problem as a coherent conditional image generation process . As illustrated in Fig . 2 , given an observed image Iobs with the target region IobsΛt masked out from a ground-truth uncorrupted image Igt , the objective of the proposed face completion is to synthesize an image Isyn that looks natural and realistic , and to enable a controllable generation process in terms of a given facial attribute vector , denoted by A ( such as male vs female , and smiling vs not smiling and for simplicity we use binary attribute vector in this paper ) and/or a given facial expression encoded by facial landmark , denoted by L. Denote by XG = ( Iobs , M , A , L ) the input of the generator G ( · ) that realizes the completion . We have , Isyn = G ( XG ; θG ) , subject to I syn Λctx ≈ IobsΛctx , ( 1 ) where θG collects all parameters of the generator and ≈ represents that the two context regions IsynΛctx and IobsΛctx need to be kept very similar . Structure-Aware Completion . As illustrated in Fig . 3 ( left ) , to enable transferring facial expressions in completion , we leverage the off-the-shelf state-of-the-art facial landmark detector , Face Alignment Network ( FAN ) ( Bulat & Tzimiropoulos , 2017 ) which achieved very good results for faces in the wild . Motivated by this , we also want to integrate the landmark information in completion for faces without facial expression trans- fer required . Recent works ( Isola et al. , 2016 ; Wang et al. , 2017 ; Zhu et al. , 2017 ; Sangkloy et al. , 2017 ; Xian et al. , 2017 ; Chen & Hays , 2018 ) have shown the capability of GANs to translate sketches to photo-realistic images . We choose facial landmarks as an abstract representation of face structures in general . As illustrated in Fig . 3 ( right ) , we first train a simple face completion model at the resolution of 256 × 256 using reconstruction loss ( Section 3.3 ) only . Given an image 1 , we use the trained model to generate a blurry completed image from which the landmarks are extracted with FAN ( we observed that FAN can compute sufficiently good landmarks from blurry completed images ) . Not only can this unify the generation process for different controllable settings ( since the inputs to the generator are kept the same between with and without facial expression transfer ) , 1The coarse completion model is only needed for testing . In training , we can extract landmarks from uncorrupted face images at the same resolution . but it also makes the completion process structure-aware . Since faces have very regular structures ( e.g . the eyes are always above a nose ) , when some facial components are occluded , it is possible to predict which parts are missing . Given a corrupted image , the quality of synthesized image can be further improved if the model is able to “ draw ” a sketch of the face first , which provides backbone guidance for image completion . 3.2 LEARNING WITH THE FOAM BETWEEN PROGRESSIVE STAGES . On top of GANS ( Goodfellow et al. , 2014 ) , the framework of Context Encoder ( CE ) ( Pathak et al. , 2016 ) is adopted , so the generation process of our model is conditioned on the contextual information . The framework of training GANs progressively ( Karras et al. , 2017 ) is also adopted to facilitate a high-resolution completion model . This starts with the lowest resolution ( such as 4× 4 ) . After running a certain number of iterations , higher resolution layers are added to both the generator and discriminator simultaneously until the network is grown to a desired resolution ( such as 1024×1024 ) . We present details of the proposed FOAM to stabilize and rationalize the progressive training . Denote by Gr and Dr the generator and discriminator at a resolution level r , respectively , where r ∈ { 1 , · · · , R } is the index of resolution ( e.g. , r = 1 represents 4 × 4 and r = R = 9 represents 1024 × 1024 ) . The final stage generator GR ( ) will be used as the generator G in Eqn . 1 in testing . The observed masked image , its corresponding binary mask , and the facial landmarks are re-sized to Iobsr , Mr and Lr for each resolution respectively . In our model , both Gr and Dr are conditioned on facial landmarks . We attach the resolution index to the input and rewrite Eqn . 1 as , Isynr = Gr ( X Gr ; ΘGr ) , subject to I syn r , Λctx ≈ Iobsr , Λctx , ( 2 ) where XGr = ( Iobsr , Mr , A , Lr ) . For the discriminator Dr , its input is X Dr = ( Ir , Lr ) where Ir represents either an uncorrupted image or a image synthesized by Gr . Dr has two branches which share a common backbone and predict the fake vs real classification and the attribute estimation Â respectively . The loss functions for training are defined in Section 3.3 . During progressive training , to avoid sudden changes to the trained parameters of Gr−1 , the added layers ( i.e . the higher resolution components ) need to be faded into the networks smoothly during a growing stage . Since the parameters of added layers are initialized randomly , these layers may generate noise that distorts the coarser structures learned by Gr−1 if they are merged with Gr−1 directly . To reduce this effect , Karras et al . ( Karras et al. , 2017 ) use a linear combination of the higher and lower resolution branches . The synthesized image Îsyn is computed by Îsyn = αIsynr + ( 1− α ) Ĩ syn r−1 , ( 3 ) in which Isynr and Ĩ syn r−1 are the output images from the higher and lower resolution branches respectively ( Ĩsynr−1 is up-sampled from I syn r−1 to match the resolution of r ) . α is a weight increasing linearly from zero to one during the growing stage . Therefore , at the beginning , the added layers have no impact on the network . During training , the influence of the higher resolution branch increases linearly while the weight of the lower-resolution branch decreases . In the end when α = 1 , the synthesized image depends only on the higher resolution branch ( i.e . Îsyn = Isynr ) and the lower resolution branch can simply be removed . Because of this , once the training is complete , a cor- rupted image only needs to be fed to a single branch for image completion , and this process does not depend on any inputs or networks of lower resolutions . The FOAM . Eqn . 3 is equivalent to applying “ all-pass filters ” to the higher and lower resolution branches , since all the pixels in images are assigned the same weight ( i.e . α or 1− α ) regardless of their locations . Although this linear combination ( Eqn . 3 ) has been shown effective for reducing the impact of noise generated during the growing stage , we observe that it does not work well for highresolution face completion , as shown in Fig . 4 . The coarse structures that have been learned well at lower resolutions are still vulnerable to being distorted during the joint training ( i.e. , 0 < α < 1 ) . The intuitive idea of the proposed FOAM is to encourage the generator to focus more on learning finer details during the growing stage , which is enabled by changing the “ all-pass filters ” reflected in Eqn . 3 to attentive “ band-pass filters ” that learn to protect what has been learned well in the previous stages and to update finer details as needed under the guidance of the loss functions . Existing approaches ( Gregor et al. , 2015 ; Yu et al. , 2018 ) use spatial attention mechanisms to encourage networks to attend to selected parts of images ( e.g . a rectangular region ) . As illustrated in Fig . 1 , we observe that the FOAM filters indeed act like “ band-pass filters ” and show a strong pattern of switching its attention from coarse structures ( i.e . the low-frequency information ) to finer details ( i.e . the high-frequency information ) as the resolution increases . But , we note that different from regular band-pass filters , the filters learned by FOAM are predicted based on image semantics through the objective function ( see Equation 14 ) . This makes them sensitive to locations inferred on-the-fly in a coarse-to-fine manner . For instance , the model learns to pay more attention to eye regions where the rich details aggregate , especially at high resolutions . With the help of FOAM , the model is capable of learning meaningful and interpretable filters automatically . As illustrated in Fig . 5 , the proposed FOAM consists of a read and a write operation . In the read operation , only information that is important in Iobsr but does not exist in I obs r−1 will be allowed to enter the network . Similarly , in the write operation , only when the added layers produce information that can help reduce the overall loss , will it be allowed to add to the synthesized image Îsyn . The read and write operations , which are like two gates in a circuit , are controlled by the read and write filters learned by our model , respectively ( denoted by Fread and Fwrite ) . Fread is predicted from the lower resolution branch and computed by , Fread = ToFilter ( Gfixedr−1 ( X Gr−1 ) ) , ( 4 ) using a trained generator Gfixedr−1 with fixed weights and a small trainable network ToFilter . Similarly , Fwrite is predicted from the last feature maps of the higher resolution branch . The value in the filters represents the weight . Fread helps extract the most valuable information in the contexts of Iobsr and Iobsr−1 . The read operation is implemented by , Îobsr = Fread ( 1−Mr ) Iobsr , Îobsr−1 = Downsample ( ( 1− Fread ) ( 1−Mr ) Ĩobsr−1 ) , ( 5 ) where denotes element-wise multiplication . Ĩobsr−1 is up-sampled from Iobsr−1 to match the resolution of level r. Similar to Eqn . 3 , Fread and ( 1 − Fread ) are assigned to the higher and lower resolution branches , respectively . The write filter Fwrite combines the outputs from two branches ( i.e . I syn r and the up-sampled Ĩsynr−1 ) to generate the final completed image Î syn r . Fwrite helps extract the most valuable information in the contexts of Isynr and Ĩ syn r−1 . The write operation is defined by , Îsynr = ( I syn r · α+ Ĩ syn r−1 · ( 1− α ) ) ( 1−Mr ) + ( Fwrite Isynr + ( 1− Fwrite ) Ĩ syn r−1 ) Mr , ( 6 ) so , only the target region of Îsynr is controlled by Fwrite . The context region is a linear combination of the contexts of Isynr and Ĩ syn r−1 . To facilitate fast face completion in testing , we further design transformation functions to adjust the value ranges of Fread and Fwrite , so the lower resolution branches and FOAMs can both be safely removed when the growing process is done . Similar to the vanilla progressive training method , a testing image only needs to go through the final stage for completion . To that end , a transformation function ( Eqn . 7 ) is used to adjust the upper and lower bounds of the dynamic value ranges of the read and write filters . For instance , the transformed F̂read starts as an all-zero filter , is adjusted by a trainable ToFilter at the growing stages , and eventually increased to all ones . The transformed filters F̂read and F̂write are defined by , F̂read = β · Fread + γ , F̂write = β · Fwrite + γ , ( 7 ) where the parameters are computed by β : { 2α , 2− 2α , γ : { 0 , α ≤ 0.5 2α− 1 , 0.5 < α ≤ 1.0 ( 8 ) in which α is a weight increasing linearly from zero to one proportional to the number of seen images during growing . Eqn . 7 will be actually used in the read operation , Eqn . 5 and the write operation , Eqn . 6 .

Summary: This paper proposes a face completion network that synthesizes the missing part in the face images with GANs. Using facial landmarks and facial attributes, face completion became controllable as both are used as conditional information (input) for the generation (synthesis). Moreover, the proposed Frequency-Oriented Attention Module (FOAM) enables an interpretable coarse-to-fine progressive generative process. The proposed methods show significant improvement in the completion quality.