FREQUENCY-ORIENTED ATTENTIVE GANS
Year: 2019

Abstract:
Face completion is a challenging conditional image synthesis task. This paper proposes controllable and interpretable high-resolution and fast face completion by learning generative adversarial networks (GANs) progressively from low resolution to high resolution. We present structure-aware and frequency-oriented attentive GANs. The proposed structure-aware component leverages off-the-shelf facial landmark detectors and proposes a simple yet effective method of integrating the detected landmarks in generative learning. It facilitates facial expression transfer together with facial attributes control, and helps regularize the structural consistency in progressive training. The proposed frequency-oriented attentive module (FOAM) encourages GANs to attend much more to finer details in the coarse-to-fine progressive training, thus enabling progressive attention to face structures. The learned FOAMs show a strong pattern of switching their attention from low-frequency to high-frequency signals. In experiments, the proposed method is tested on the CelebA-HQ benchmark. Experiment results show that our approach outperforms state-of-the-art face completion methods. The proposed method is also fast with mean inference time of 0.54 seconds for images at 1024× 1024 resolution (using a Titan Xp GPU).

text: Face completion is a challenging conditional image synthesis task. This paper proposes controllable and interpretable high-resolution and fast face completion by learning generative adversarial networks (GANs) progressively from low resolution to high resolution. We present structure-aware and frequency-oriented attentive GANs. The proposed structure-aware component leverages off-the-shelf facial landmark detectors and proposes a simple yet effective method of integrating the detected landmarks in generative learning. It facilitates facial expression transfer together with facial attributes control, and helps regularize the structural consistency in progressive training. The proposed frequency-oriented attentive module (FOAM) encourages GANs to attend much more to finer details in the coarse-to-fine progressive training, thus enabling progressive attention to face structures. The learned FOAMs show a strong pattern of switching their attention from low-frequency to high-frequency signals. In experiments, the proposed method is tested on the CelebA-HQ benchmark. Experiment results show that our approach outperforms state-of-the-art face completion methods. The proposed method is also fast with mean inference time of 0.54 seconds for images at 1024× 1024 resolution (using a Titan Xp GPU).

heading: 1 INTRODUCTION
text: Conditional image synthesis aims to learn the underlying distribution governing the data generation with respect to the given conditions/context, which is also able to synthesize novel content. Much progress (Iizuka et al., 2017; Yeh et al., 2017; Li et al., 2017; Yang et al., 2016; Denton et al., 2016; Pathak et al., 2016; Yu et al., 2018; Liu et al., 2018; Brock et al., 2018; Karras et al., 2018) has been made since the generative adversarial networks (GANs) were proposed (Goodfellow et al., 2014). Despite the recent remarkable progress, learning controllable and interpretable GANs for high-fidelity image synthesis at high resolutions remain an open problem.
We are interested in controllable and interpretable GANs. We take a step forward by focusing on high-resolution and fast face completion tasks in this paper. Face completion is to replace target regions, either missing or unwanted, of face images with synthetic content so that the completed images look natural, realistic, and appealing. State-of-the-art face completion approaches using GANs largely focus on generating random realistic content. However, users may want to complete the missing parts with certain properties (e.g. expressions). Controllability is entailed. Existing face completion approaches are usually only able to complete faces at relatively low resolutions (e.g. 176 × 216 (Iizuka et al., 2017) and 256 × 256 (Yu et al., 2018)). To facilitate high-resolution image synthesis, the training methodology of growing GANs progressively (Karras et al., 2017) is widely used. For face completion tasks, one issue of applying progressive training is how to avoid distorting the learned coarse structures when the network is growing to a higher resolution. Interpretability is thus entailed to guide GANs in the coarse-to-fine pipeline. In addition, most existing approaches (Iizuka et al., 2017; Yeh et al., 2017; Li et al., 2017) require post-processing (e.g. Poisson Blending (Pérez et al., 2003)), complex inference process (e.g. thousands of optimization iterations (Yeh et al., 2017) or repeatedly feeding an incomplete image to CNNs at multiple scales (Yang et al., 2016)) during test.
We present structure-aware and frequency-oriented attentive GANs that are progressively trained for high-resolution and fast face completion using a fast single forward step in inference without any post-processing. By controllable, it means that the completed face images can have different facial attributes (e.g., smiling vs not smiling) and/or facial expressions transferred from a given source actor. By interpretable, it means that the coarse-to-fine generation process in progressive training is rationalized. We utilize facial landmarks as backbone guidance of face structures and propose a straightforward method of integrating them in our system. We design a novel Frequency-Oriented Attention Module (FOAM) to induce the model to attend to finer details (i.e. higher-frequency content, see Fig. 1). We observe significant improvement of the completion quality by the FOAM against the exactly same system only without FOAM. A conditional version of our network is designed so that the appearance properties (e.g. male or female), and facial expressions of the synthesized faces can be controlled. Moreover, we design a set of loss functions inducing the network to blend the synthesized content with the contexts in a realistic way. Our method was compared with state-of-the-art approaches on a high-resolution face dataset CelebA-HQ (Karras et al., 2017). Both the evaluations and a pilot user study showed that our approach completed face images significantly more naturally than existing methods.

heading: 2 RELATED WORK
text: Recent learning based methods have shown the capability of CNNs to complete large missing content. Based on existing GANs, the Context Encoder (CE) (Pathak et al., 2016) encodes the contexts of masked images to latent representations, and then decodes them to natural content images, which are pasted into the original contexts for completion. However, the synthesized content of CE is often blurry and has inconsistent boundaries. Given a trained generative model, Yeh et al. (Yeh et al., 2017) propose a framework to find the most plausible latent representations of contexts to complete masked images. The Generative Face Completion model (GFC) (Li et al., 2017) and the Global and Local Consistent model (GL) (Iizuka et al., 2017) use both global and local discriminators, combined with post-processing, to complete images more coherently. Built on GL, Yu et al. (Yu et al.,
2018) design a contextual attention layer (CTX) to help the model borrow contextual information from distant locations. Liu et al. (Liu et al., 2018) incorporates partial convolutions to handle irregular masks. Unfortunately, these approaches can only complete face images in relatively low resolutions (e.g. 176× 216 (Iizuka et al., 2017) and 256× 256 (Yu et al., 2018)). Yang et al. (Yang et al., 2016) combine a global content network and a texture network, and the networks are trained at multiple scales repeatedly to complete high-resolution images (512× 512). But, they assume that the missing content always shares some similar textures with the context, which is improbable for the face completion task.
3 THE PROPOSED METHOD 3.1 PROBLEM FORMULATION
,
Denote by Λ an image lattice (e.g., 1024×1024 pixels). Let IΛ be a face color image defined on the lattice Λ. Denote by Λt and Λctx the target region to complete and the remaining context region respectively (note that the target region is not necessarily a single connected component, and the two parts form a partition of the lattice). IΛt is masked out with the same gray
pixel value. LetMΛ be a binary mask image with all pixels inMΛt being 1 and pixels inMΛctx being 0. For simplicity, we will omit the subscripts Λ, Λt and Λctx when the text context is clear. Unlike existing approaches (Pathak et al., 2016; Li et al., 2017; Iizuka et al., 2017) which first utilize unconditional image synthesis to generate the target region image and then blend them with context using using sophisticated post-processing, we address the completion problem as a coherent conditional image generation process.
As illustrated in Fig. 2, given an observed image Iobs with the target region IobsΛt masked out from a ground-truth uncorrupted image Igt, the objective of the proposed face completion is to synthesize an image Isyn that looks natural and realistic, and to enable a controllable generation process in terms of a given facial attribute vector, denoted by A (such as male vs female, and smiling vs not smiling and for simplicity we use binary attribute vector in this paper) and/or a given facial expression encoded by facial landmark, denoted by L. Denote by XG = (Iobs,M,A,L) the input of the generator G(·) that realizes the completion. We have,
Isyn = G(XG; θG), subject to I syn Λctx ≈ IobsΛctx , (1)
where θG collects all parameters of the generator and ≈ represents that the two context regions IsynΛctx and IobsΛctx need to be kept very similar.
Structure-Aware Completion. As illustrated in Fig. 3 (left), to enable transferring facial expressions in completion, we leverage the off-the-shelf state-of-the-art facial landmark detector, Face
Alignment Network (FAN) (Bulat & Tzimiropoulos, 2017) which achieved very good results for faces in the wild. Motivated by this, we also want to integrate the landmark information in completion for faces without facial expression trans-
fer required. Recent works (Isola et al., 2016; Wang et al., 2017; Zhu et al., 2017; Sangkloy et al., 2017; Xian et al., 2017; Chen & Hays, 2018) have shown the capability of GANs to translate sketches to photo-realistic images. We choose facial landmarks as an abstract representation of face structures in general. As illustrated in Fig. 3 (right), we first train a simple face completion model at the resolution of 256 × 256 using reconstruction loss (Section 3.3) only. Given an image 1, we use the trained model to generate a blurry completed image from which the landmarks are extracted with FAN (we observed that FAN can compute sufficiently good landmarks from blurry completed images). Not only can this unify the generation process for different controllable settings (since the inputs to the generator are kept the same between with and without facial expression transfer),
1The coarse completion model is only needed for testing. In training, we can extract landmarks from uncorrupted face images at the same resolution.
but it also makes the completion process structure-aware. Since faces have very regular structures (e.g. the eyes are always above a nose), when some facial components are occluded, it is possible to predict which parts are missing. Given a corrupted image, the quality of synthesized image can be further improved if the model is able to “draw” a sketch of the face first, which provides backbone guidance for image completion.

heading: 3.2 LEARNING WITH THE FOAM BETWEEN PROGRESSIVE STAGES
text: On top of GANS (Goodfellow et al., 2014), the framework of Context Encoder (CE) (Pathak et al., 2016) is adopted, so the generation process of our model is conditioned on the contextual information. The framework of training GANs progressively (Karras et al., 2017) is also adopted to facilitate a high-resolution completion model. This starts with the lowest resolution (such as 4× 4). After running a certain number of iterations, higher resolution layers are added to both the generator and discriminator simultaneously until the network is grown to a desired resolution (such as 1024×1024). We present details of the proposed FOAM to stabilize and rationalize the progressive training.
Denote by Gr and Dr the generator and discriminator at a resolution level r, respectively, where r ∈ {1, · · · , R} is the index of resolution (e.g., r = 1 represents 4 × 4 and r = R = 9 represents 1024 × 1024). The final stage generator GR() will be used as the generator G in Eqn. 1 in testing. The observed masked image, its corresponding binary mask, and the facial landmarks are re-sized to Iobsr , Mr and Lr for each resolution respectively. In our model, both Gr and Dr are conditioned on facial landmarks. We attach the resolution index to the input and rewrite Eqn. 1 as,
Isynr = Gr(X Gr ; ΘGr ), subject to I syn r,Λctx ≈ Iobsr,Λctx , (2) where XGr = (Iobsr ,Mr, A, Lr). For the discriminator Dr, its input is X
Dr = (Ir, Lr) where Ir represents either an uncorrupted image or a image synthesized by Gr. Dr has two branches which share a common backbone and predict the fake vs real classification and the attribute estimation Â
respectively. The loss functions for training are defined in Section 3.3.
During progressive training, to avoid sudden changes to the trained parameters of Gr−1, the added layers (i.e. the higher resolution components) need to be faded into the networks smoothly during a growing stage. Since the parameters of added layers are initialized randomly, these layers may generate noise that distorts the coarser structures learned by Gr−1 if they are merged with Gr−1 directly. To reduce this effect, Karras et al. (Karras et al., 2017) use a linear combination of the higher and lower resolution branches. The synthesized image Îsyn is computed by
Îsyn = αIsynr + (1− α)Ĩ syn r−1, (3)
in which Isynr and Ĩ syn r−1 are the output images from the higher and lower resolution branches respectively (Ĩsynr−1 is up-sampled from I syn r−1 to match the resolution of r). α is a weight increasing linearly from zero to one during the growing stage. Therefore, at the beginning, the added layers have no impact on the network. During training, the influence of the higher resolution branch increases linearly while the weight of the lower-resolution branch decreases. In the end when α = 1, the synthesized image depends only on the higher resolution branch (i.e. Îsyn = Isynr ) and the lower
resolution branch can simply be removed. Because of this, once the training is complete, a cor-
rupted image only needs to be fed to a single branch for image completion, and this process does not depend on any inputs or networks of lower resolutions.
The FOAM. Eqn. 3 is equivalent to applying “all-pass filters” to the higher and lower resolution branches, since all the pixels in images are assigned the same weight (i.e. α or 1− α) regardless of their locations. Although this linear combination (Eqn. 3) has been shown effective for reducing the impact of noise generated during the growing stage, we observe that it does not work well for highresolution face completion, as shown in Fig. 4. The coarse structures that have been learned well at lower resolutions are still vulnerable to being distorted during the joint training (i.e., 0 < α < 1).
The intuitive idea of the proposed FOAM is to encourage the generator to focus more on learning finer details during the growing stage, which is enabled by changing the “all-pass filters” reflected in Eqn. 3 to attentive “band-pass filters” that learn to protect what has been learned well in the previous stages and to update finer details as needed under the guidance of the loss functions. Existing approaches (Gregor et al., 2015; Yu et al., 2018) use spatial attention mechanisms to encourage networks to attend to selected parts of images (e.g. a rectangular region). As illustrated in Fig. 1, we observe that the FOAM filters indeed act like “band-pass filters” and show a strong pattern of switching its attention from coarse structures (i.e. the low-frequency information) to finer details (i.e. the high-frequency information) as the resolution increases. But, we note that different from regular band-pass filters, the filters learned by FOAM are predicted based on image semantics through the objective function (see Equation 14). This makes them sensitive to locations inferred on-the-fly in a coarse-to-fine manner. For instance, the model learns to pay more attention to eye regions where the rich details aggregate, especially at high resolutions. With the help of FOAM, the model is capable of learning meaningful and interpretable filters automatically.
As illustrated in Fig. 5, the proposed FOAM consists of a read and a write operation. In the read operation, only information that is important in Iobsr but does not exist in I obs r−1 will be allowed to enter the network. Similarly, in the write operation, only when the added layers produce information that can help reduce the overall loss, will it be allowed to add to the synthesized image Îsyn. The read and write operations, which are like two gates in a circuit, are controlled by the read and write filters learned by our model, respectively (denoted by Fread and Fwrite). Fread is predicted from the lower resolution branch and computed by,
Fread = ToFilter (Gfixedr−1(X Gr−1)), (4)
using a trained generator Gfixedr−1 with fixed weights and a small trainable network ToFilter. Similarly, Fwrite is predicted from the last feature maps of the higher resolution branch. The value in the filters represents the weight. Fread helps extract the most valuable information in the contexts of Iobsr and Iobsr−1. The read operation is implemented by, Îobsr = Fread (1−Mr) Iobsr , Îobsr−1 = Downsample ((1− Fread) (1−Mr) Ĩobsr−1), (5) where denotes element-wise multiplication. Ĩobsr−1 is up-sampled from Iobsr−1 to match the resolution of level r. Similar to Eqn. 3, Fread and (1 − Fread) are assigned to the higher and lower resolution branches, respectively. The write filter Fwrite combines the outputs from two branches (i.e. I syn r and the up-sampled Ĩsynr−1) to generate the final completed image Î syn r . Fwrite helps extract the most valuable information in the contexts of Isynr and Ĩ syn r−1. The write operation is defined by,
Îsynr = (I syn r · α+ Ĩ syn r−1 · (1− α)) (1−Mr) + (Fwrite Isynr + (1− Fwrite) Ĩ syn r−1) Mr, (6) so, only the target region of Îsynr is controlled by Fwrite. The context region is a linear combination of the contexts of Isynr and Ĩ syn r−1.
To facilitate fast face completion in testing, we further design transformation functions to adjust the value ranges of Fread and Fwrite, so the lower resolution branches and FOAMs can both be safely removed when the growing process is done. Similar to the vanilla progressive training method, a testing image only needs to go through the final stage for completion. To that end, a transformation function (Eqn. 7) is used to adjust the upper and lower bounds of the dynamic value ranges of the read and write filters. For instance, the transformed F̂read starts as an all-zero filter, is adjusted by a trainable ToFilter at the growing stages, and eventually increased to all ones. The transformed filters F̂read and F̂write are defined by, F̂read = β · Fread + γ, F̂write = β · Fwrite + γ, (7) where the parameters are computed by
β : { 2α, 2− 2α, γ : { 0, α ≤ 0.5 2α− 1, 0.5 < α ≤ 1.0 (8)
in which α is a weight increasing linearly from zero to one proportional to the number of seen images during growing. Eqn. 7 will be actually used in the read operation, Eqn. 5 and the write operation, Eqn. 6.

heading: 3.3 LOSS FUNCTIONS
text: To induce high-fidelity face completion, we utilize the loss functions as follows.
Adversarial Loss Given an uncorrupted ground-truth image Igt, its attribute vector A, a mask M , landmarks L, and the corresponding corrupted image Iobs, we define the loss by,
ladv(I gt,M,L, Iobs, A|G,D) = log (1−Dcls(Isyn, L) + logDcls(Igt, L), (9)
where Isyn = G(Iobs,M,A,L) and Dcls represents the classification branch of the discriminator.
Attribute Loss Similar to the InfoGAN models (Chen et al., 2016; Choi et al., 2017), for the attribute prediction head classifier in the discriminator, we define the attribute loss based on crossentropy between the predicted attribute vectors, Âgt = Dattr(Igt, L) and Âobs = Dattr(Iobs, L), and the corresponding input attribute vectors A for both a ground-truth image and a synthesized image,
lattr(I gt, A,M, Iobs|G,D) = CrossEntropy (A, Âgt) + CrossEntropy (A, Âobs), (10)
where Dattr represents the attribute prediction branch of the discriminator.
Reconstruction Loss Since our method generates an entire completed face, we define a weighted reconstruction loss lrec to rebuild both the content and the context regions,
lrec(I gt,M,L, Iobs, A|G) = ‖κ M Idiff‖1 + ‖(1− κ) (1−M) Idiff‖1, (11)
where Idiff = Igt − Isyn and κ is the trade-off parameter. Feature Loss In additional to the reconstruction loss, we also expect a synthesized image to have similar feature representations (Johnson et al., 2016) to a ground-truth image. Let φ be a pre-trained deep neural network and φj be the activation of the jth layer of φ, the feature loss is defined by,
lfeat(I gt,M,L, Iobs, A|φ,G) = ‖φj(Igt)− φj(Isyn))‖22. (12)
In our experiments, φj is the relu2 2 layer of a 16-layer VGG network (Simonyan & Zisserman, 2014) pre-trained on the ImageNet dataset (Russakovsky et al., 2015).
Boundary Loss To make the generator learn to blend the synthesized target region with the original context region seamlessly, we further define a close-up reconstruction loss along the boundary of the mask. Similar to (Yeh et al., 2017), we first create a weighted kernel w based on the mask image M . w is computed by blurring the mask boundary in M with a mean filter so that the pixels closer to the mask boundary are assigned larger weights. The kernel size of the mean filters is seven in our experiments. We have,
lbdy(I gt,M,L, Iobs, A|G) = ‖w (Igt − Isyn)‖1. (13)
Our model is trained end-to-end by integrating the expected loss of the loss functions defined above under the minimax two-player game setting. The full objective is,
min G max D Ladv(G,D) + λ1Lattr(G,D) + λ2Lrec(G) + λ3Lfeat(G,φ) + λ4Lbdy(G), (14)
where λi’s are trade-off parameters between different loss terms. Fig. 4 shows an ablation study of the importance of the loss functions. Note that the ablation study was run at 256 × 256. Since the training of high-resolution models was very time consuming, an ablation study for 1024× 1024 networks is left for future work.

heading: 4 EXPERIMENTS
text: Datasets and Experiment Settings. We used the CelebA-HQ (Karras et al., 2017) dataset for
evaluation. It contains 30,000 aligned face images at 1024 × 1024 resolution. The dataset is split randomly while ensuring there is no identity overlap between test/training sets: 3,009 images for testing, and 26,991 for training. There were two types of masks in training: center and random. The center mask was a square region in the middle of the image with a side length of half the size of the image. The random masks, generated in a similar way to previous methods (Iizuka et al., 2017; Yu et al., 2018), were rectangular regions with random width-to-height ratios, sizes and locations covering about 5% to 25% of the original images. Network architectures, hyper-parameters and more results are provided in the Appendix.
Completion without Attribute Control. We first trained a high-
resolution (1024 × 1024) model with center masks (examples shown in Fig. 6) to test whether our model is capable of learning high-level semantics and structures of faces and synthesizing large missing regions. The second model was trained with random rectangular masks, but was able to handle various shaped masks (e.g. irregular hand-drawn masks). Fig. 6 shows that our model was able to capture the anatomical structures of faces and generate content that is consistent with the holistic semantics.
Completion with Attribute Control. Two attributes (“Male vs Female” and “Smiling vs Not Smiling”) were chosen. This model was trained from scratch and the result was run at a 512 × 512 resolution. Fig. 7 shows that the attributes of synthesized images were controlled by our model explicitly. Fig. 8 shows the facial expression transfers together with attribute control.
Quantitative Evaluation. In current literature (Yeh et al., 2017; Yu et al., 2018), reconstruction metrics such as mean L1, L2 errors and peak signal-to-noise ratio (PSNR) are commonly used for the evaluation of in-painting methods. We show the comparison between our method and state-
of-the-art models at their reported resolutions respectively (Table 1). The result shows that our model outperformed state-of-the-art approaches. In addition to comparing how well each model
Method Resolution L1 (%) L2 (%) PSNR GL (Iizuka et al., 2017) 128× 128 9.34 1.75 18.22
Ours 128× 128 7.8 1.42 19.15 CTX (Yu et al., 2018) 256× 256 8.53 1.75 18.41
Ours 256× 256 7.05 1.21 19.97
user study, which is considered the “gold standard” to evaluate GAN models. Our method obtains significantly better results (Fig. 9, see detail in the Appendix).
Computation Time. We tested our model with a Titan Xp GPU by processing 3000 1024 × 1024 images with 512 × 512 holes. The mean completion time is 0.54 second per image. It takes about one minute for the model of Yang et al. (Yang et al., 2016) to complete a 512×512 scene image with a Titan X GPU.
Limitations Though our method has low inference time, the training time is long due to the progressive growing of networks. In our experiment, it takes about three weeks to train a 1024 × 1024 model on a Titan Xp GPU. By carefully zooming in our results, we find that our high-resolution model fails to learn lowlevel skin textures, such as furrows and sweat
holes. Moreover, the model could generate distorted content when removing large parts (e.g. hats) or synthesize some plausible but unnatural faces (Fig. 10). Furthermore, for facial expression transfer, our method requires that the head poses of the source and target faces are similar. These issues are left for future work.
5 CONCLUSION
We propose a progressive GAN with frequency-oriented attention modules (FOAM) for high resolution and fast controllable and interpretable face completion, which learns face structures from coarse to fine guided by the FOAM. By consolidating information across all scales, our model not only outperforms state-of-the-art methods by generating sharper images in low resolution (such as 256 × 256), but is also able to synthesize faces in higher resolutions (such as 512 × 512 and 1024 × 1024) than existing techniques. Our attribute and expression controller allows users to manipulate the appearance and facial expressions of generated images explicitly with at-
tribute vectors and landmarks. Our system is designed in an end-to-end manner, in that it learns to generate completed faces directly and more efficiently.

heading: A APPENDIX
text: In this section, we first show more results of high resolution face completion. Then, we present detail of the user study. We also provide detail of the network architectures and hyper-parameters used in training.

heading: A.1 MORE RESULTS OF FACE COMPLETION
text: Fig. 11, Fig. 12 and Fig. 13 show more results of high resolution face completion with masks of various shape. Our model is capable of learning high-level semantics and structures of faces and handling challenging mask types that were not included in the training set (e.g. hand-drawn masks in Fig. 13).
Fig. 14 and Fig. 15 show more results of hard and soft attribute control respectively. Fig. 16 show more results of joint attribute and expression control.

heading: A.2 DETAIL OF USER STUDY
text: We compared our method with CTX (Yu et al., 2018), which is a state-of-the-art CNN-based face completion approach capable of completing face images at 256 × 256 resolution, with a pilot user study at 256× 256 resolution with random masks. 27 subjects (15 male and 12 female participants, with ages from 22 to 32) volunteered to participate.
There were four sessions in the experiments. For each trial, a user was shown two images and asked to choose the more realistic one. In the first session, two images completed from the same image by different methods were chosen (one by our model and another by CTX). In sessions two to four, a real image and a corresponding synthesized image were shown. In the first session, time was unlimited. In session two to four, images were on display for 250ms, 1000ms, 4000ms respectively. We randomly chose 100 pairs of images [Ours, CTX] from the test set for session one and another 100 groups of images [Ours, CTX, GT] for session two to four. The display order of images was randomized.
The result (Fig. 9) shows that there was a significantly higher chance that images completed by our model looked more realistic than those completed by CTX in session one. While comparing with the ground-truth images in session two to four, the highest possible percentage that a method can achieve is 50%, which is a random guessing. There was a probability of about 40% that our method could fool a human observer when the display time was 250 ms. As the display time increased, users started to notice more detailed artifacts, so all the percentages dropped. Regardless, our model always significantly outperformed CTX. Statistical analysis was performed to confirm the significant differences between our method and CTX. The details of analysis are listed in the supplemental materials.
Some of the most-frequently-picked images by participants are shown in Fig. 17. Overall, our approach generated sharper images with more details and fewer distortions. Sometimes, the synthesized faces looked even more natural than the ground-truth images.
Statistical Analysis of the User Study. In order to confirm the intuition of our comparison results, we tested for statistical significance. To do this, we first collapsed each participant’s rankings into a frequency list. Once frequency lists were built for all participants, the frequencies for each method were again averaged over the 27 participants to produce a final list of averages from n = 27 samples. For session one, we performed the paired samples t-test to compare the means of frequencies for these two methods: t(26) = 46.368, p < 0.001. The results confirmed that our method was favored significantly more often than CTX. For session two through four, a two-way repeated measures analysis of variance (ANOVA) was used because there were two factors: Method (ours and CTX) and Time (250 ms, 1000 ms, and 4000 ms). Since the sphericity assumption was not met in our data, we used the correction of Huynh-Feldt for the Method and Time factors, and the GreenhouseGeisser correction for the Method×Time interactions. Not surprisingly, results (Table 2) showed a significant difference in means of different method groups for a standard α = 0.05, which denoted that there was a significantly higher probability that images completed by our method were classified as real samples versus those generated by CTX.

heading: A.3 NETWORK ARCHITECTURES AND HYPER-PARAMETERS
text: The generator G in our model is implemented by a U-shape network architecture consisting of the first component Genc transforming the observed image and its mask to a latent vector and the second component Gdec transforming the concatenated vector (latent code and input attributes) to a completed image. There are residual connections between layers in Genc and the counterpart in Gdec similar in the spirit to the U-Net (Ronneberger et al., 2015) and the Hourglass network (Newell
et al., 2016) to consolidate information across multiple scales. Fig. 19 illustrates the two structures of a layer in the generator for training without and with the attribute controller respectively, which are adapted from the U-Net and Hourglass network.
Every convolutional layer (Conv) is followed by an Instance Normalization (InsNorm) and a LeakyReLU layer, except that the Conv before the latent vector (i.e. the second Conv layer in Table 3) is not followed by an InsNorm. Additionally, the there are no InsNorms or LeakyReLUs after the last Convs of bothDcls andDattr. All Convs used in the residual block of the skip connections of our conditional model have a kernel size of three and a stride of one.
Since we use Instance Normalization rather than Batch Normalization, the batch size is not an important hyper-parameter. Technically, for faster computation, we use as large a batch size as possible so long as it does not exceed the GPU memory limit.
Tables 3 and 5 demonstrate the architecture of the components of the generator G while Tables 6 shows the components of the discriminator D. In Table 6, depending on the operation of the skip connection (Skip), the number of filters is either doubled (for a concatenation operation) or remains the same (for an addition operation).
The progressive training process is illustrated in Fig. 18. At a resolution lower than 1024 × 1024, the input face images, masks, landmarks and real images are all down-sampled with average pooling to fit the given scale. One of the major challenges of generating high resolution images is the limitation of Graphics Processing Unit (GPU) memory. Most completion networks use Batch Normalization (Ioffe & Szegedy, 2015) to avoid covariate shift. However, with the limited GPU memory, only a small number of batch sizes are supported at high resolution, resulting in low quality of generated images. We use the Instance Normalization (Ulyanov et al., 2016), similar to Zhu et al. (Zhu et al., 2017), and update D with a history of completed images instead of the latest generated one (Shrivastava et al., 2016) to stabilize training.
At the growing stage, new layers are added for both D and G and these layers are faded in with current networks smoothly. After the fade-in process, the network is trained on more images for stabilization. We used 300K, and 150K training images for resolution [8 × 8 to 256 × 256] and [512 × 512, 1024 × 1024] respectively at growing stage, and 600K, 430K images for 4 × 4 and [8× 8 to 1024× 1024] at stabilizing stage respectively. In the experiments, the reconstruction trade-off parameter was set to κ = 0.7 to focus more on the target region. To balance the effects of different objective functions, we used λattr = 2, λrec = 500, λfeat = 8, and λbdy = 5000. The Adam solver (Kingma & Ba, 2014) was employed with a learning rate of 0.0001.
r x r cubes in the figure represent convolutional layers that handle resolution r. For the conditional
version, attribute labels Aobs are concatenated to the latent vectors. The discriminator D splits into two branches in the final layers: Dcls that classifies if an input image is real, and Dattr that predicts attribute vectors. Note that XG and XD are both a set of inputs as defined in the paper. We use images in this Figure as a simplified illustration.

Review summaries:
This paper proposes a face completion network that synthesizes the missing part in the face images with GANs. Using facial landmarks and facial attributes, face completion became controllable as both are used as conditional information (input) for the generation (synthesis). Moreover, the proposed Frequency-Oriented Attention Module (FOAM) enables an interpretable coarse-to-fine progressive generative process. The proposed methods show significant improvement in the completion quality.

This paper proposes controllable and interpretable high-resolution and fast face completion by learning generative adversarial networks (GANs) progressively from low resolution to high resolution. It combines the masks, landmarks, corrupted images as inputs to generate completed images in high-resolution. The proposed frequency-oriented attentive module (FOAM) encourages GANs to highlight much more to finer details in the coarse-to-fine progressive training, thus enabling progressive attention to face structures.