Towards Controllable and Interpretable Face Completion via  Structure-Aware and Frequency-Oriented Attentive GANs
Authors: Zeyuan Chen
Shaoliang Nie
Tianfu Wu
Christopher G. Healey
Venue:  (2020)
Keywords: Face Completion
GANs
Conditional Image Synthesis
Interpretability
Frequency-Oriented Attention

Abstract: Face completion is a challenging conditional image synthesis task. This paper proposes controllable and interpretable high-resolution and fast face completion by learning generative adversarial networks (GANs) progressively from low resolution to high resolution. We present structure-aware and frequency-oriented attentive GANs. The proposed structure-aware component leverages off-the-shelf facial landmark detectors and proposes a simple yet effective  method of integrating the detected landmarks in generative learning. It facilitates facial expression transfer together with facial attributes control, and helps regularize the structural consistency in progressive training. The proposed  frequency-oriented attentive module (FOAM) encourages GANs to attend to only finer details in the coarse-to-fine progressive training, thus enabling progressive attention to face structures. The learned FOAMs show a strong pattern of switching its attention from low-frequency to high-frequency signals. In experiments, the proposed method is tested on the CelebA-HQ benchmark. Experiment results show that our approach outperforms state-of-the-art face completion methods. The proposed method is also fast with mean inference time of 0.54 seconds for images at 1024x1024 resolution (using a Titan Xp GPU).

Under review as a conference paper at ICLR 2020
TOWARDS CONTROLLABLE AND INTERPRETABLE FACE COMPLETION VIA STRUCTURE-AWARE AND FREQUENCY-ORIENTER ATTENTIVE GANs
Anonymous authors Paper under doublc-blind review
ABSTRACT
Face completion is a challenging conditional image synthesis task. This paper proposes controllable and interpretable high-resolution and fast face completion by learning generative adversarial networks (GANs) progressively from low res- olution to high resolution. We present structure awwae and reeuency-oriente attentive GANs. The proposed structure-wware component leverages off-the-shelff facial landmark detectors and proposes simple yet effective method of integrat- ing the detected landmarks in generative learning. It facilitates facial expression transfer together with facial attributes control, and helps regularize the structural consistency in progressive training. The proposed rrequency-oriente attentive module (FOAM) encourages GANs to attend much more to finer details the coarse-to-fine progressive training, thus enabling progressive attention to face structures. The learned FOAMs show a strong pattern of switching their atten- tion from low-frequency to high-frequency signals. In experiments, the proposed method is tested on the CelebA-HQ benchmark. Experiment results show that approach outperforms kate-of-thhearr face completion methods. The pro- posed method is also fast with mean inference time 0.54 seconds for images at 1024 1024 resolution (using a Titan Xp GPU).
INTRODUCTION
Conditional image synthesis aims to learn the underlying distribution governing the data generation with respect to the given conditions/context, which is also able to synthesize novel content. Much progress (lizuka et al.]]111 2017 Yeh et al., 2017: Li al. 2017: Yang et al. 2016|Denton Denton al. 2016] Pathak al. 2016 Yu et al. 2018:] Liu al. 2018 Brock et al. 2018 Karras et al.) 2018) has been made since the generative adversarial networks (GANs) were proposed (Goodfellow et al.) 2014]. Despite the recent remarkable progress, learning controllable and interpretable GANs for high-fidelity image synthesis at high resolutions remain an open problem.
We are interested in controllable and interpretable GANs. We take step forward by focusing high-resolution and fast face completion tasks this paper. Face completion is to replace target regions, either missing images look natural, realistic, and appealing. State-of-the-art face completion approaches using GANs largely focus on generating random realistic content. However, users may want complete the missing parts with certain properties (e.g. expressions). Controllability is entailed. Existing face completion approaches are usually only able complete faces at relatively low resolutions (e.g. 176 x 216 (lizuka et al. (017)) and 256 x 256 (Yu et al.) (018)). To facilitate high-resolution image synthesis, the training methodology of growing GANs progressively (Karras et al., (017) widely used. For face completion tasks, one issue of applying progressive training is how to avoid distorting the learned coarse structures when the network is growing to a higher resolution. Inter- pretability is thus entailed to guide GANs in the coarse-to-fin fine pipeline. In addition, most existing approaches (Tizuka et al., 2017: Yeh et al., 2017 Li et al. 2017) require post-processing (e.g. Pois- son Blending (Pérez al. 2003) complex inference process (e.g. thousands of optimization itera- tions YYh et 2017 or repeatedly feeding an incomplete image to CNNs at multiple scales (Yang et al. 2016)) during test.
unwanted, of face images with synthetic content that the completed

---

Under review as a conference paper at ICLR 2020
Masked
Synthesized
Figure 1: Face completion results of the proposed methed CelehA-HQ (Krrrs al. 2017) 1024 x 1024 resolution. The leftmost column masked images while the rightmost synthesized images. The learned FOAM filters are shown with higher intensities meaning more attention. lower resolutions, model focuses more on learning coarse structures (i.e. the lower-frequency signals). As the resolution increases, the model pays more attention to finer details (i.c. the higher-frequent and implicitly performs "band-pass filter" guiding the generation process. For instance, the model pays more attention regions with richer details, such as hair eyes, especially high resolutions. The learned FOAM also relatively stable when target regions are similar, see the last two rows. Best viewed color and magnification.
information). Therefore, the FOAM partially
We present structure-aware and frequency-orientedd attentive GANs that are progressively trained for high-resolution and fast face completion using a fast single forward step inference without any post-processing. By controllable, it means that the completed face images can have different facial attributes (e.g., smiling vs not smiling) and/or facial expressions transferred from a given source actor. By interpretable, it means that the coarse-to-fine generation process in progressive training rationalized. We utilize facial landmarks as backbone guidance of face structures and propose a straightforward method of integrating them in system. We design novel Trequency-Orientee Attention Module (FOAM) to induce the model to attend to finer details (i.e. higher-frequency content, Fig. [ We observe significant improvement of the completion quality by the FOAM against the exactly same system only without FOAM. A conditional version of our network is de- signed so that the appearance properties (e.g. male or female), and facial expressions of the synthe- sized faces can be controlled. Moreover, we design a set of loss functions inducing the network to blend the synthesized content with the contexts in realistic way. Our method was compared with state-of of-the-art approaches on high-resolutin face dataset CelebA-HQ (Karras et al.) (017). Both the evaluations and pilot user study showed that our approach completed face images significantly more naturally than existing methods.
RELATED WORK
Recent learning based methods have shown the capability of CNNs to complete large missing con- tent. Based on existing GANs, the Context Encoder (CE) Pathak et al.) (016) encodes the contexts of masked images to latent representations, and then decodes them to natural content images, which are pasted into the original contexts for completion. However, the synthesized content CE of- ten blurry and has inconsistent boundaries. Given a trained generative model, Yeh et al. (Yeh al.) 2017) propose framework to find the most plausible latent representations of contexts to complete masked images. The Generative Face Completion model (GFC) (Li et al. 2017) and the Global and Local Consistent model (GL) (lizuka et al.) (017) use both global and local discriminators, com- bined with processo.,,,,ng, complete images coherently. Built GL. Yu al. (Yu et al.
2

---

Under review as a conference paper at ICLR 2020
2018) design contextual attention layer (CTX) to help the model borrow contextual information from distant locations. Liu et al. (Liu et al. 2018) incorporates partial convolutions to handle ir- regular masks. Unfortumately, these approaches can only complete face images in relatively low resolutions (e.g. 176 x 216 (lizuka et al. 2017) and 256 x 256 Yu et al. (018)). Yang et al. (Yang et al., 2016) combine global content network and texture network, and the networks
trained at multiple scales repeatedly to complete high-resolution images (512 x 512). But, they assume that the missing content always shares some similar textures with the context, which is improbable for the face completion task.
3 THE PROPOSED METHOD
UNei
3.1 PROBLEM FORMULATION
Concoxiation
Denote by A an image lattice (e.g., 1024 1024 pixels). Ler IA be face color image defined on the lattice A. Denote by A, and Aexx the target region to complete and the remaining context region respectively (note that the target region is not necessarily a single connected compo- nent, and the two parts form a partition of the lattice). IA, is masked out with the same gray pixel value. Let MA binary mask image with all pixels in MA, being 1 and pixels MA being O. For simplicity, we will omit the subscripts A. A, and Actx when the text context is clear. Unlike existing approaches (Pathak et al. 2016 Li et al. (2017) lizuka et al. (017)) which first utilize uncon- ditional image synthesis generate the target region image and then blend them with context using using sophisticated post-processing, we address the completion problem as a coherent conditional image generation process.
1 Figure 2:
Overview of the proposed completion model. See text for details.
As illustrated in Fig. given an observed image Jobs with the target region 1 masked out from a ground-truth uncorrupted image 7 the objective of the proposed face completion is to synthesize an image / that looks natural and realistic, and enable a controllable generation process in terms of given facial attribute vector, denoted by A (such as male vs female, and smiling vs not smiling and for simplicity we use binary attribute vector in this paper) and/or a given facial expression encoded by facial landmark, denoted by L. Denote by generator G.) that realizes the completion. We have,
(robs M, M,A,I, L) the input of the
- = G(XC;0a) subject to 1 ~ 1
(I)
where AG collects all parameters the generator and = represents that the two context regions Am and Au need to be kept very similar.
Structure-Aware Completion. As illustrated in Fig. B| (left), to enable transferring facial expres- sions in completion, we leverage the uff-the-shelf state-of state-of-the-arr facial landmark detector, Face Alignment Network (FAN) (Bu-
lat & Tzimiropoulos) 2017) which achieved very good re- sults for faces in the wild. Mo- tivated by this, we also want to integrate the landmark infor- mation in completion for faces without facial expression trans- fer required. Recent works (Isola et al. 2016, Wang al.. 2017 Zhu et al. 2017 Sangkloy et al. 2017 Xian al. Chen Chen & Hays (0188) have shown the capability of GANs to translate sketches to photo-realistic images. We choose facial landmarks as an abstract representation of face struc- tures in general. As illustrated in Fig. 3 (right), first train simple face completion model the resolution of 256 x 256 using reconstruction loss (Section 3.3) only. Given an image [ we use the trained model to generate a blurry completed image from which the landmarks with FAN (we observed that FAN can compute sufficiently good landmarks from blurry completed images). Not only can this unify the generation process for different controllable settings (since the imputs to the generator are kept the same between with and without facial expression transfer),
X
FAN
Figure 3: Illustration of computing facial landmarks for structure-awe completion. See text for detail.
extracted
'The coarse completion model is only needed for testing. In training, can extract landmarks from uncorrupted face images the resolution.
3

---

Under review as a conference paper at ICLR 2020
but it also makes the completion process structure-aware. Since faces have very regular structures (e.g. the eyes are always above a nose), when some facial components are occluded, it is possible to predict which parts missing. Given corrupted image, the quality of synthesized image can be further improved if the model is able to "draw" sketch of the face first, which provides backbone guidance for image completion.
3.2 LEARNING WITH THE FOAM BETWEEN PROGRESSIVE STAGES
On top of GANS (Goodfellow et al. 2014), the framework of Context Encoder (CE) (Pathak al. 2016 is adopted, so the generation process our model conditioned on the contextual infor- mation. The framework of training GANs progressively (Karras et al., 2017) is also adopted to facilitate a high-resolution completion model. This starts with the lowest resolution (such as x 4). After running a certain number of iterations, higher resolution layers ator and discriminator simultaneously until the network is grown to a desired resolution (such as 1024x1024). We present details of the proposed FOAM stabilize and rationalize the progressive training.
added to both the gener-
Denote by G, and D, the generator and discriminator a resolution level r, respectively, where r E (1,....P is the index of resolution (e.g., = represents 4 X 4 and r = R 9 represents 1024 x 1024). The final stage generator GR() will be used the generator G in Eqn. | in testing. The observed masked image, its corresponding binary mask, and the facial landmarks are re-sized Jobs M, and Lyr for each resolution respectively. our model, both G, and D, are conditioned on facial landmarks. We attach the resolution index to the input and rewrite [qa.
1: (X OG.), subject to I
Jobs T,Anx
where X = (Irhs Mr. A, L,). For the discriminator D,, its input is X = (Ir, Lr) where I, represents either an uncorrupted image or a image synthesized by G,. D, has two branches which share a common backbone and predict the fake vs real classification and the attribute estimation Á respectively. The loss functions for training are defined in Section 3.3
During progressive training, to avoid sudden changes to the trained parameters of G,-1, the added layers (i.e. the higher resolution com- ponents) need to be faded into the networks smoothly during growing stage. Since the parameters of added layers are initialized ran- domly, these layers may generate noise that dis- torts the coarser structures learned by G,-1 1 if they are merged with Gy-1 directly. To reduce this effect, Karras et al. (Karras et al. (2017) use a linear combination of the higher and lower resolution branches. The synthesized image jsyn is computed by
Maked
E
E
TOAM
fsym aT + (1 a
(3)
the output images from the higher and lower resolution branches respectively (1,0h is up-sampled from 1"T to match the resolution of r). is a weight increasing linearly from zero to one during the growing stage. Therefore, at the begin- ning, the added layers have no impact on the network. During training, the influence of the higher resolution branch increases linearly while the weight of the lower-resolution branch decreases. In the end when sized image depends only on the higher reso- lution branch (i.e. jayn = 1et)) and the lower resolution branch simply be removed. Because of this, once the training is complete,
in which 7:2m and ioym
Figure 4: The ablation study shows the impact of es- sential components of our method. A model that trained with adversarial Lasv and regular reconstruction loss Li generates only blurry images. After adopt- ing the progressive training method and set of de- signed loss functions, the synthesized image quality improved. By incorporating FOAM, the model focuses on learning only finer details while growing, resulting in sharper images with fewer distortions. Best viewed in magnification.
1, the synthe-
4

---

Under review as a conference paper at ICLR 2020
TaFlie
Fe..
Til Clar
Figure 5: Illustration of the FOAM using an example of growing 32 32 network to 64 x 64. The proposed FOAM consists of read and a write operation to realize attentive "band-pass filters". See text for detail. rupted image only needs to be fed single branch for image completion, and this process does not depend on any inputs or networks of lower resolutions.
The FOAM. Eqn. 3 is equivalent to applying "all-pass filters" to the higher and lower resolution branches, since all the pixels in images are assigned the same weight (i.e. a or 1 a) regardless of their locations. Although this linear combination (Eqn.B) has been shown effective for reducing the impact of noise generated during the growing stage, we observe that it does not work well for high- resolution face completion, as shown Fig. 4 The coarse structures that have been learned well at lower resolutions are still vulnerable to being distorted during the joint training (i.e., 0 < < 1).
The intuitive idea of the proposed FOAM to encourage the generator to focus more on learning finer details during the growing stage, which is enabled by changing the "all-pass filters" reflected in Eqn.Btto attentive "band-pass filters" that learn to protect what has been learned well in the previous stages and to update finer details as needed under the guidance of the loss functions. Existing approaches (Gregor et al.) 2015; Yu et al. 2018) use spatial attention mechanisms to encourage networks to attend to selected parts of images (e.g. rectangular region). As illustrated in Fig, [ we observe that the FOAM filters indeed act like "band-pass filters" and show a strong pattern of switching its attention from coarse structures (i.e. the low-frequency information) to finer details (i.e. the high-frequency information) as the resolution increases. But, we note that different from regular band-pass filters, the filters learned by FOAM are predicted based image semantics through the objective function (see Equation [4]. This makes them sensitive to locations inferred on-the-fly in a coarse-to-fine manner. For instance, the model learns to pay more attention to eye regions where the rich details aggregate, especially at high resolutions. With the help of FOAM, the model is capable of learning meaningful and interpretable filters automatically.
As illustrated in Fig. 5] the proposed FOAM consists of a read and a write operation. In the read operation, only information that important in I but does not exist in 1oth will be allowed to enter the network. Similarly, in the write operation, only when the added layers produce information that can help reduce the overall loss, will it be allowed to add to the synthesized image jsym The read and write operations, which are like two gates in a circuit, are controlled by the read and write filters learned by model, respectively (denoted by Fread and Fwrtte.. Fread is predicted from the lower resolution branch and computed by.
Fread = ToFilter (
(4)
using a trained generator Gfsad with fixed weights and small trainable network ToFilter. Similarly, Fwitt is predicted from the last feature maps of the higher resolution branch. The value in the filters represents the weight. Fread helps extract the most valuable information in the contexts of Io and 1 The read operation is implemented by.
jo = Fread @ (1 - M2) @ Jobs johs = Downsample ((1 Fread) ② (1 M,) o jobs
(5)
where 3 denotes element- wise multiplication. johh, is up-sampled from 1ohe to match the resolution of level r. Similar to Eqn. 3 Fread and (1 - Fread) are assigned to the higher and lower resolution branches, respectively. The write filter Fwiit combines the outputs from two branches (i.e. rp
and the up-sampled ) to generate the final completed image 1:'t. Fwiite helps extract the most valuable information in the contexts of 1: and r-1- The write operation is defined by,
îsym (I a + 1 (1 (1-a)) a)) @ (1 M,) + (Fwine O I (1 Fwzee O ITm) Mp. (6) so, only the target region of iTmm is controlled by Fwriee The context region is linear combination of the contexts of 128 and P-1
5

---

Under review as a conference paper at ICLR 2020
To facilitate fast face completion in testing, we further design transformation functions to adjust the value ranges of Fred and Fwste so the lower resolution branches and FOAMs can both be safely removed when the growing process is done. Similar to the vanilla progressive training method, testing image only needs to go through the final stage for completion. To that end, a transformation function (Eqn. 7 is used to adjust the upper and lower bounds of the dynamic value ranges of the read and write filters. For instance, the transformed Pread starts as all-zero filter, is adjusted by trainable ToFilter at the growing stages, and eventually increased to all ones. The transformed filters Fread and F Friie are defined by,
Fren B Fread +7:
Pwite B F.wiite + 7+
where the parameters are computed by
2a,
0, 2a 1, 0.5 < a 1.0
a S 0.5
8 :::
2 2 2a,
y
(8)
in which is weight increasing linearly from zero to one proportional to the number of seen images during growing. Eqn. 7 will be actually used in the read operation, Eqn. S and the write
operation, Eqn. 6
3.3 LOSS FUNCTIONS
To induce high-fidelity face completion, we utilize the loss functions as follows.
Adversarial Loss Given an uncorrupted ground truth image 1 its attribute vector A. a mask M, landmarks L, and the corresponding corrupted image I we define the loss by,
Indv ( M, L,, /obs, A|G,, D) log (1 Dan(1 L) + log Das L),
(9)
where 132 ==m=Ga G( M, A, L) and Dets represents the classification branch of the discriminator.
Attribute Loss Similar the InfoGAN models (Chen al.] 2016 Choi al. 2017). for the at- tribute prediction head classifier in the discriminator, we define the attribute loss based on cross- entropy between the predicted attribute vectors, A = Dame [ L) and A = Daur (Joh,, L). and the corresponding input attribute vectors for both ground-truth image and a synthesized image, larr _ ; A, M, r IG, D) = CrossEntropy (A, A + CrussEntrupy (A, A where Danr represents the attribute prediction branch of the discriminator.
(10)
Reconstruction Loss Since our method generates an entire completed face, we define a weighted reconstruction loss Lree to rebuild both the content and the context regions,
lree( (8t M, L, jobs A[G) = || @ M o 1 diff [|1 + [|(1 - x) o (1 - M) O 1
(11)
where ddi = _ - 1 and x. is the trade-off parameter.
Feature Loss In additional to the reconstruction loss, we also expect synthesized image to have similar feature representations (Johnson et al. (016) to ground-truth image. Let Φ be a pre-trained deep neural network and Φ, be the activation of the jth layer of c. the feature loss is defined by,
4ear (1 M, 1 Alø, G) = | (1(")))
(12)
In our experiments, Φj is the relu2.2 layer of a 16-layer VGG network (Sionnyan & Zisserman 2014) pre- trairrd on the ImageNet dataset (Russakovsky al. 2015).
Boundary Loss To make the generator learn to blend the synthesized target region with the original context region seamlessly, we further define close-up reconstruction loss along the boundary of the mask. Similar to (Yeh et al.] 2017). we first create a weighted kernel w based on the mask image M. w is computed by blurring the mask boundary in M with mean filter so that the pixels closer to the mask boundary are assigned larger weights. The kernel size of the mean filters is seven in our experiments. We have,
( M, L. oos A|G) = w|w O (1 -
(13)
Our model is trained end-to-end by integrating the expected loss of the loss functions detined above under the minimax two-player game setting. The full objective is,
min max Laav D) + X₁ Camn + A2)rec + Aa )) + (G))
(14)
G where \ 's are trade-off parameters between different loss terms. Fig. 4 shows an ablation study of the importance of the loss functions. Note that the ablation study was at 256 x 256. Since the training of high-resolution models was very time consuming, an ablation study for 1024 x 1024 networks is left for future work.

---

Under review as a conference paper at ICLR 2020
C
C
C
Figure 6: Examples of high-resolution face completion results by our method at 1024 x 1024 resolution. For each group, from left to right: masked. synthesized and real images. Our model was able to capture the anatomical structures of faces and handle various shaped masks.
4 EXPERIMENTS
Datasets and Experiment Settings. evaluation. contains 30,000 aligned face images 1024 1024 resolu- tion. The dataset is split randomly while ensuring there is no identity overlap between test/training sets: 3,009 images for testing, and 26,991 for training. There were two types of masks in training: center and ran- dom. The center mask was a square region in the middle of the image with a side length of half the size of the image. The random masks, gen- erated in similar methods (Tizuka et al. [017]] Yu al.] 2018), were rectangular regions with random width-to-height ratios, sizes and locations covering about 5% to 25% the original images. Network architectures, hyper-parameter and more results are provided in the Ap- pendix.
We used the CelebA-HQ (Karras et al. 2017) dataset for
to previous
Merbed
Figure 7: Examples of controlling attributes. All images 512 x 512 resolution. The leftmost column are masked images, and the rest are generated faces. More Fig. and Fig. 15
Completion without Attribute Con- trol. We first trained high- resolution (1024 x 1024) model with center masks (examples shown in Fig. to test whether our model capable of learning high-level semantics and structures of faces and synthesizing large missing regions. The second model was trained with random rectangular masks, but was able to handle various shaped masks (e.g. irregular hand-drawn masks). Fig. 6 shows that our model was able to capture the anatomical structures of faces and generate content that is consistent with the holistic semantics.
Completion with Attribute Control. Two attributes ("Male Female" and "Smiling Not Smil- ing") were chosen. This model was trained from scratch and the result was run at 512 x 512 resolution. Fig. 7 shows that the attributes of synthesized images were controlled by our model explicitly. Fig. 8 shows the facial expression transfers together with attribute control.
Quantitative Evaluation. In current literature (Yhh al.) 2017: Yu et al. 2018), reconstruction metrics such as mean LI, L2 errors and peak signal-to-noise ratio (PSNR) are commonly used for the evaluation of in-painting methods. We show the comparison between method and state-

---

Under review as a conference paper at ICLR 2020
Syrthestredd
WinDened Tocttes
Figure 8: Examples joint attribute and expression control. All images are at 512 x 512 resolution. Though the source and synthesized faces have different identities, their expressions are very similar.
of-the-art models their reported resolutions respectively (Table D. The result shows that our
model outperformed state-of-the-art approaches. In addition to comparing how well each model reconstructs the missing regions, it is also important to evaluate the natural- ness of synthesized images since im- age completion aims to generate re- alistic and plausible content rather than restoring the original images per- fectly. Due to the lack of good met- rics for naturalness, we user study, which considered the "gold method obtains significantly better results (Fig. 2
Table 1:
The quantitative comparison between
method and state-of-the-art methods
Method
Resolution
LT
(2
PSNR
GL lizuka et al. 2017
128 x 128
9.34
1.35
18.22
Ous
128
C al. 2018
256 x 256
833
1.35
18.41
Ours
256 x 256
7,05
1.21
19.97
a pilot
standard" to evaluate GAN models.
Our
see detail in the Appendix).
Session One
Session Two Four
Computation Time. We tested our model with a Titan Xp GPU by processing 3000 1024 x 1024 images with 512 512 holes. The mean completion time is 0.54 second per image. It takes about one minute for the model of Yang et al. (Yang et al. 2016) to complete 512 512 scene image with Titan GPU.
Lm
300m 1800m
acbool
Figure 9: Comparisons the naturalness: ours and CTX (Yu et al. 2018 Left: There was significantly higher percentage of images completed by our model that looked more realistic than those completed by CTX. Right: The percentage that synthesized image con- sidered more realistic than a ground-true -rand (GT) one. There is a significantly higher probability that images completed by our method were classified as real sam- ples versus those generated by CTX.
Limitations Though our method has low infer- ence time, the training time is long due to the progressive growing of networks. In our ex- periment, it takes about three weeks to train 1024 1024 model on Titan Xp GPU. By carefully zooming in our results, we find that high-resolution model fails to learn low- level skin textures, such as furrows and sweat holes. Moreover, the model could generate distorted content when removing large parts (e.g. hats) or synthesize some plausible but unnatural faces (Fig. [0). Furthermore, for facial expression trans- fer, our method requires that the head poses of the source and target faces are similar. These issues are left for future work.
CONCLUSION
We propose progressive GAN with frequency-oriented attention modules (FOAM) for high resolution and fast controllable and interpretable face completion, which learns face structures from coarse to fine guided by the FOAM. By consolidating information across all scales, our model not only outperforms saae-off-hhe-an methods by generating sharper images in low resolution (such as 256 x 256), but is also able to synthesize faces in higher resolutions (such as 512 x 512 and 1024 x 1024) than existing techniques. Our attribute and expression con- troller allows users to manipulate the appearance and fa- cial expressions of generated images explicitly with at- tribute vectors and landmarks. Our system is designed in an end-to-end manner, in that learns to generate completed faces directly and more efficiently.
Figure 10: Some failure cases. From left to
right: masked, synthesized, and real images.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809. 2018.
Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face align- ment problem?(and a dataset of 230,000 facial landmarks). In International Conference on Computer Vision, volume 1, pp. 8, 2017.
Wengling Chen and James Hays. Sketchygan: Towards diverse and realistic sketch to image syn- thesis. arXiv preprint arXiv: 1801.0273,, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2172-21800 2016.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha. Sunghun Kim, and Jacgul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. arXiv preprint 1rXiv:1711.09000 2017.
Emily Denton, Sam Gross, and Rob Fergus. Semi-supervised learning with context-conditional generative adversarial networks. arXiv preprint arXiv: 1eeo:16611.0040, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- mation processing systems, pp. 2672-2680, 2014.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. arXiv preprint
2015.
Satoshi lizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Globally and locally consistent image completion. ACM Transactions on Graphics (TOG), 36(4):107, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456, 2015.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. arXiv preprint arXiv:161..07000 2016.
Justin Johnson, Alexandre Alahi, and Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference Computer Vision, pp. 694-711. Springer, 2016.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im-
proved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. arXiv preprint arXiv:1812.0499) 144481112 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Yijun Li, Sifei Liu, Jimei Yang, and Ming Hiugn Yang. Generative face completion. arXiv preprint 17X4v:1704.05838 2017.
Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. arXiv preprint 2018.
Andrew I. Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models. In Proc. ICML, volume 30, 2013.
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estima- tion. In European Conference Computer Vision, pp. 483 499.. Springer, 2016.

---

Under review as a conference paper at ICLR 2020
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536-2544, 2016.
encoders:
Patrick Pérez, Michel Gangnet, and Andrew Blake. Poisson image editing. In ACM Transactions on graphics (TOG), volume 22, pp. 313-318. ACM, 2003.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234-241. Springer, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252 2015.
Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. Seribbler: Controlling deep image synthesis with sketch and color. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, 2017.
Ashish Shrivastava, Tomas Pfister. Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb. Learning from simulated and unsupervised images through adversarial training. arXiv preprint 1rXiv:16/..078888, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.08022 2016
Ting- Ting-Chun Wang, Ming- Yu Liu, Jun- Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. arXiv preprint 17.1 2017.
Wenqi Xian, Patsorn Sangkloy, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays. Texturegan: Controlling deep image synthesis with texture patches. arXiv preprint 17Xivv1706.0266 2017.
Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. High-resolution image inpainting using multi-scale neural patch synthesis. arXiv preprint wxxi::/61..09900 2016.
Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh Do. Semantic image inpainting with deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3485-5493, 2017.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. arXiv preprint arXiv;
2018.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to tomiaagg translation using cycle-consistent adversarial networks. arXiv preprint arXiv: 7703. 10593, 2017.
A APPENDIX
In this section, we first show more results of high resolution face completion. Then, we present detail of the user study. We also provide detail of the network architectures and hyper-parameters used in training.
A.1
MORE RESULTS OF FACE COMPLETION
Fig. Fig. 12 and Fig. 13 show more results of high resolution face completion with masks of
various shape. Our model is capable of learning high-level semantics and structures of faces and handling challenging mask types that were not included in the training set (e.g. hand-drawn masks
in Fig. 13).
Fig. and Fig. 15 show more results of hard and soft attribute control respectively. Fig. show more results of joint attribute and expression control.
10

---

Under review as a conference paper at ICLR 2020
Figure 11:
High-resolution face completion results with center masks. All images are at 1024 x 1024 resolu- tion. For each group. from left to right: masked, synthesized, and real images.
11

---

Under review as a conference paper at ICLR 2020
Ilc
N
N
W
Figure 12: More examples High-resolution face completion results with center masks. All images are at 1024 x 1024 resolution. For each group, from left to right: masked, synthesized, and real images.
12

---

Under review as a conference paper at ICLR 2020
OCIE
OCIE
OCI
TES
TES
TES
stantte
Figure 13: More examples of High- reghhrresoluion face completion results with hand- drawn masks. All images are 1024 x 1024 resolution. For each group, from left to right: masked, synthesized, and real images.
13

---

Under review as a conference paper at ICLR 2020
Macked
10.1] Shilling
[1,0] Male, Mring
lniling
Smiling
Figure 14: Face completion results with attribute controller. Attribute "Male vs Female" is used to control the appearance. Landmarks from source used control the facial expressions of synthesized images. The leftmost column shows masked images and faces generated with ground truth attributes and landmarks.
Figure 15: Snapshots of face completion results with relative and soft attribute controller. The Demo video will be available as supplementary material.
14

---

Under review as a conference paper at ICLR 2020
Source Actors
Expression
Synthesized
Source Actors
Expeessor
Synthesized
With Ground Truth labels
Femac
Female
Male
Male
Figure 16: Face completion results with attribute controller. Attribute "Male Female" is used to control the appearance. Landmarks from source actors are used control the facial expressions of synthesized images. The leftmost column shows masked images and faces generated with ground-truth attributes and landmarks.
CTX
Ours
GT
CTX
GT
Ours
Figure 17: Examples of images used in the user study. The preferred images are marked with red boxes.
15

---

Under review as a conference paper at ICLR 2020
A.2 DETAIL OF USER STUDY
We compared our method with CTX (Yu et al. 2018), which is a state-of-the-art CNN-based face completion approach capable of completing face images at 256 x 256 resolution, with pilot user study at 256 x 256 resolution with random masks. 27 subjects (15 male and 12 female participants, with ages from 22 to 32) volunteered to participate.
There were four sessions in the experiments. For each trial, user was shown two images and asked to choose the more realistic one. In the first session, two images completed from the same image by different methods were chosen (one by our model and another by CTX). In sessions two four, a real image and a corresponding synthesized image were shown. In the first session, time was unlimited. In session two to four, images were on display for 250ms, 1000ms, 4000ms respectively. We randomly chose 100 pairs of images [Ours, CTX] from the test set for session one and another 100 groups of images [Ours, CTX, GT] for session two to four. The display order of images was randomized.
The result (Fig. 99) shows that there was a significantly higher chance that images completed by our model looked more realistic than those completed by CTX in session one. While comparing with the ground-trun
images in session two to four, the highest possible percentage that method can achieve is 50%, which is random guessing. There was probability of about 40% that our method could fool a human observer when the display time was 250 ms. As the display time increased, users started to notice more detailed artifacts, all the percentages dropped. Regardless, always significantly outperformed CTX. Statistical analysis was performed to confirm the significant differences between our method and CTX. The details of analysis are listed in the supplemental materials.
model
Some of the most-freqquntty-pickee images by participants are shown in Fig. 17 Overall, our approach generated sharper images with more details and fewer distortions. Sometimes, the synthe- sized faces looked even more natural than the ground-truth images.
The results of the two-way repeated measures ANOVA of the user study. There was a strong main effect for Method, which indicated that the images generated by our method were recognized as real ones hy the human observers significantly more frequently than those completed by CTX (Yu et al. 2018).
Table 2:
Source
p
Method
F(1,26) = 352.645
P<0.001
Time
F(1.657,43.09) 203.235
p=0.001
Method x Time
F(1.283,33.346) 1.7)) 1.760
p=0.194
Statistical Analysis of the User Study. In order to confirm the intuition of our comparison results, we tested for statistical significance. To do this, we first collapsed each participant's rankings into a frequency list. Once frequency lists were built for all participants, the frequencies for each method were again averaged over the 27 participants to produce final list of averages from n = 27 samples. For session one, we performed the paired samples t-test to compare the means of frequencies for these two methods: t(26)- = 46.368, p < 0.001. The results confirmed that our method was favored significantly more often than CTX. For session two through four, a two-way repeated measures analysis of variance (ANOVA) was used because there were two factors: Method (ours and CTX) and Time (250 ms, 1000 ms, and 4000 ms). Since the sphericity assumption was not met in our data, we used the correction of Huynh-Feldi for the Method and Time factors, and the Greenhouse- Geisser correction for the Method Time interactions. Not surprisingly, results (Table 2 showed significant difference in means of different method groups for a standard = 0.05, which denoted that there was significantly higher probability that images completed by our method were classified as real samples versus those generated by CTX.
A.3 NETWORK ARCHITECTURES AND
The generator G in our model is implemented by U-shape network architecture consisting of the first component Gene transforming the observed image and its mask to a latent vector and the second component Gdee transforming the concatenated vector (latent code and input attributes) to a completed image. There are residual connections between layers in G.ne and the counterpart in Ge similar in the spirit to the U-Net (Ronneherger al. 2015) and the Hourglass network (Newell
16

---

Under review as a conference paper at ICLR 2020
et al. 2016) to consolidate information across multiple scales. Fig. 19 illustrates the two structures of a layer in the generator for training without and with the attribute controller respectively, which adapted from the U-Net and Hourglass network.
Every convolutional layer (Conv) is followed by an Instance Normalization (InsNorm) and a LeakyReLUU layer, except that the Conv before the latent vector (i.e. the second Conv layer in T)bleB) is not followed by an InsNorm. Additionally, the there are no InsNorms or LeakyReLUl af- ter the last Convs of both Dets and Damr All Convs used the residual block of the skip connections of our conditional model have a kernel size of three and a stride of one.
Instance Normalization rather than Batch Normalization, the batch size is not im- Technically, for faster computation, we use as large a batch size as possible so long as it does not exceed the GPU memory limit.
Since
portant
Tables and 5 demonstrate the architecture of the components of the generator G while Tables shows the components of the discriminator D. In Table o depending on the operation of the skip connection (Skip), the number of filters is either doubled (for concatenation operation) or remains same (for an addition operation).
The progressive training process is illustrated in Fig. 18 At resolution lower than 1024 x 1024, the input face images, masks, landmarks and real images are all down-sampled with average pool- ing to fit the given scale. One of the major challenges of generating high resolution images is the limitation of Graphics Processing Unit (GPU) memory. Most completion networks use Batch Normalization (loffe & Szegedy 2015) to avoid covariate shift. However, with the limited GPU memory, only a small number of batch sizes are supported at high resolution, resulting in low qual- ity of generated images. We use the Instance Normalization (Ulyanov al. (2016), similar to Zhu al. (Zhu et al.] 2017), and update D with history of completed images instead of the latest generated one (Shrivastava al. 2016) to stabilize training.
At the growing stage, new layers are added for both D and G and these layers are faded in with current networks smoothly. After the fade-in process, the network is trained on more images for stabilization. We used 300K, and 150K training images for resolution [8 x 8 to 256 x 256] and [512 512, 1024 X 1024] respectively growing stage, and 600K, 430K images for 4 and [8 x 8 to 1024 x 1024] at stabilizing stage respectively.
In the experiments, the reconstruction trade-off parameter was set to K 0.7 to focus more on the target region. To balance the effects of different objective functions, we used Aattr 2, Arec 500, Aeeat 8, and Andy 5000. The Adam solver (Kingma & Ba 2014) was employed with learning rate of 0.0001.
17

---

Under review as a conference paper at ICLR 2020
yobs
Troat
q1
4x4
Training Progresses
Treat
Bx8
4p66
Innent
Ax4
868
10ZAK19Z4
preat
A
As4
Figure 18: The progressive training process of our approach. The training of the completion network (or the "generator" G) and the discriminator D starts at low resolution (4 x 4). Higher layers are added both G and D progressively to increase the resolution of the synthesized images. The rxr cubes in the figure represent convolutional layers that handle resolution r. For the conditional version, attribute labels A are concatenated to the latent vectors. The discriminator D) splits into two branches in the final layers: Dets that classifies an input image real, and Dame that predicts attribute vectors. Note that X and X are both set of inputs as defined in the paper. We use images in this Figure as a simplified illustration.
Conveutoes Intense
Inpu Temer
Concelenaion Aeddion
IH|
Oeeae Lapes
Dispen Lavta
Figure 19; Illustrations of single layer of our architecture. There are skip connections between mirrored encoder and decoder layers. Left: the structure of the completion network; the skip con- nection is copy-and-coneatemaie operation. This structure helps preserve the identity information between the synthesized images and real faces, resulting in little deformation. Right: the structure of the conditional completion network; residual connections are added to the encoder, and the skip connections are residual blocks instead of direct concatenation. The attributes of the synthesized contents can be manipulated more easily with this structure. Each blue rectangle represents set of Convolutional, Instance Normalization and Leaky Rectified Linear Unit (LeakyReLU) (Maas al. 2013) layers.
18

---

Under review as a conference paper at ICLR 2020
Table Top: the Encoding component of generator Genci Bottom: Latent Layer. N the length of an attribute vector. The attribute concatenation operation (AttrConcat) is only activated for our conditional model.
Type
Kernel
Stride
Output Shape
Input Image
4 x 1024 1024
Conv
1 x 1
1 x
16 1024 x 1024
Conv
3
1 x
32 1024 x 1024
Conv
x 3
1 x
32 x 1024 x 1024
Downsample
32 512
512
Conv
3
1x1
64 x 512
512
Conv
x
1x1
64 512
x 512
Downsample
64 x 256
256
Conv
3x3
1x1
128
256
x 256
Conv
3x3
1x1
128
256
256
Downsample
128
128 X 128
Conv
3
1x1
256 128 X 128
Conv
x
1x1
256
128
128
Downsample
256 x 64 x 64
3x3 Downsample
Conv
1x1
512 64
Conv
3 3x3
1x1
512 x 64 X 64
512 x 32
Conv
3x3 3x3
1X
512 x 32
32
Conv
1
512 x 32 x 32
Downsample
512 x
16
16
Conv
3x3
1 1
512 X 16
16
Conv
3x3
1x
512 X
16
16
Downsample
512 X
X 8
Conv
3x3
1x1
512 x
X
Conv
3x3
1x1
512 x
Downsample
512 x
X
Type
Kernel
Stride
Output Shape
Conv
3 3
1
512 X
X
4
Conv
x 4
x 1
512 x
X
AutrConcat
optional
512(+N)
X
1
Conv
4 x 4
1 1 1
512 x
4
X
4
Conv
x 3
x 1
512
X
19

---

Under review as a conference paper at ICLR 2020
Table 4: The completion component of generator Gd Depending on the particular operation of the skip connection (Skip), the number of filters is either doubled (for concatenation operations) or remains the same (for addition operations). practice, Gdee output feature map that can be used to generate RGB image (with ToRGB layers) or predict a read/write Filter (with ToFilter layers, see 5).
Type
Kernel
Stride
Output Shape
Upsample
512
8
8
Skip
1024 (512) x
8
x
8
Conv
3 x 3
1 x 1 1 x
512
8
8
Conv
3 x 3
512 x
8
8
Upsample
512 x
16 X
Skip
1024 (512) X 16 X 16
Conv
3
1x1 1
512 x
X
Conv
3
1 1
512 x 16 X 16
Upsample
512 x 32 x 32
Skip
1024 (512) x
x 32
Conv
3 3 1x 1
512 x 32 x 32
Conv
3x3
1
512 x X
Upsample
512 X 64 X 64
Skip
1024 (512) X 64 X 64
Conv
3
1 1
512 x 64 X 64
Conv
3x3
1
512 x 64 x 64
Upsample
512 x 128 128
Conv
3x3
1 1
256 x 128 x 128
Skip
512 (256) 128
Conv
3x3
1x1
256 x 128 x 128
Conv
3x3
1x1
256 x 128 x 128
Upsample
256 x 256 x 256
Conv
3x3
1
128 x 256 x 256
Skip
256
(128) 256 256
Conv
3
1x1
128 x 256 x 256
Conv
3x3
1x1
128 256 256
Upsample
128 x 512 x 512
Conv
3x3
1x 1
64 x 512 x 512
Skip
128 (64) 512 x 512
Conv
3x3
1x1
64 x 512 x 512
Conv
3x3
1x1
64 512 x 512
Upsample
64 x 1024 X 1024
Conv
3x3
1
32 1024 1024
Skip
64 (32) x 1024 X 1024
Table 5: Left: The ToRGB layers that convert feature maps to RGB images. Right: ToFilter layers that predict read/write filter from feature maps.
Cony
3
1
32 1024
1024
Conv
3
1 1
64 x 1024 x 1024
Conv
3 xx3 3
1 1x
32 x 1024 X 1024
Conv
3 3 3
1 1x 1
64 x 1024 X 1024
Cony
1x1
I
1024 1024
Conv
1x1
1x1
1 1024
1024
20

---

Under review as a conference paper at ICLR 2020
Table 6: Top: Feature Network (()) computes feature map for input image, which is later used
by Dets and Dair Middle: The real/fake head classifier Das: Bottom: The attribute network Daur- N is the length of an attribute vector. This network is only activated for the conditional model.
Type
Kernel
Stride
Output Shape
Input Image
3 x 1024 x 1024
Conv
1
1
x 1024 1024
Conv
3 x 3 3
x 1
16 x 1024 x 1024
Conv
x 1
32 x 1024 x 1024
Downsample
32 x 512 x 512
Conv
3 3
1 x 1 1 x 1
32 x 512 x 512
Conv
3x3
64 512 x 512
Downsample
64 256 x 256
Conv
3
X 256 X 256
Conv
3
1 1 1
128 X 256 X 256
Downsample
128 x 128 X 128
Conv
3x3 3
1 X 1
128 x 128
128
Conv
3
x 1
256 x 128 x 128
Downsample
256 64 X 64
Conv
3x3
1 x 1
256 X 64 X 64
Conv
3x3
512
X 64
Downsample
512 X 32 x 32
Conv
3x3
1 1
512 x 32 x 32
Conv
3x3
1x 1
512
32
Downsample
512 x
16 x 16
Conv
3x3
512
X
16
Conv
3
1 x
512 X
16 X
16
Downsample
512 X
8
X
8
Conv
3x3
1x1
512
8
8
Conv
3
1x1
512 X
8
X 8
Downsample
512
4
4
Type
Kernel
Stride
Output Shape
Conv Conv
3 x 3
x
512
4
4
4 x
x
1 x
1
X
Type Conv Conv
Kernel
Stride
Output Shape
3 xx33 3
1 x 1
512 4
4 x 4
X 1
Nx
21


Reviews:
[{"review_id": "ryxuOhO2cB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims at the problem of face synthesis. The authors propose a progressive GAN with frequency-oriented attention modules for high resolution and fast controllable and interpretable face completion, which learns face structures from coarse to fine guided by the FOAM. Experiments are conducted to verify the effectiveness of the proposed method. This paper is well written and is easy to understand. \n\n1. The most interesting idea is the frequency-oriented attention modules ,while the idea of structure-aware seems very common in the area.\n\n2. The experiments are unconvincing. There are so many works about face synthesis in recent years. Why do the authors only compare with GL and CTX? Also, the authors do not study how each component affects the final performance, which is very important for the reader to understand why it works.\n\n3. In general, the framework of the proposed method is very common. This paper only follow previous work and lacks new insights about the problem. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "rkxBCAd6YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a face completion network that synthesizes the missing part in the face images with GANs. Using facial landmarks and facial attributes, face completion became controllable as both are used as conditional information (input) for the generation (synthesis). Moreover, the proposed Frequency-Oriented Attention Module (FOAM) enables an interpretable coarse-to-fine progressive generative process. The proposed methods show significant improvement in the completion quality.\n\nOverall,  the method shows how the face completion can be controlled and how the face completion is done by improving details. The attentive framework makes possible to do kinds of band-pass filtering. The results are impressive but have some concerns as the following:\n\n- Have you tried to train models without using facial landmarks? Are facial landmarks only for controlling facial expressions?\n- As I carefully looked at the generated faces, many of them have asymmetric (unbalanced) eyes. Is it due to the predicted facial landmarks?\n- Is there any randomness (random input) involved during completion (generation)? Is this model possible to generate different faces from the same conditional input?\n- What happens if the attributes are interpolated rather than zero or one?\n- The order of Fig 4 and 5 seems weird.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxSxA-Rtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes controllable and interpretable high-resolution and fast face completion by learning generative adversarial networks (GANs) progressively from low resolution to high resolution. It combines the masks, landmarks, corrupted images as inputs to generate completed images in high-resolution. The proposed frequency-oriented attentive module (FOAM) encourages GANs to highlight much more to finer details in the coarse-to-fine progressive training, thus enabling progressive attention to face structures.\n\nIntegrating the Progressive growing GAN (PGGAN) for high-resolution face completion is an interesting step-up work after the success of PGGAN on high-resolution image generation.\n\nBasically the FOAM is proposed to merge the images from different resolution levels instead of a weighted summation. The novelty, however, is not strong as the simple summation provides reasonable performance as convinced in the progressive growing GAN paper. And there lacks quantitative analysis on ablation study as showed in Figure 4. Therefore, the contributions of each component are not convincing.\n\nUsing Progressive Growing GAN for high-resolution face completion is also studied in [1]. Figure 17, Figure 18, indicates the author used the same structure as used by [1]. However,  [1] is not cited and there is no comparison between these two models in the paper.\n\n[1] Zeyuan Liu, et al  \"High Resolution Face Completion with Multiple Controllable Attributes via Fully End-to-End Progressive Generative Adversarial Networks \"", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work performs fast controllable and interpretable face completion, by proposing a progressive GAN with frequency-oriented attention modules (FOAM).  The proposed FOAM encourages GANs to highlight more to finer details in the progressive training process. This paper is well written and is easy to understand. While reviewer #1 is overall positive about this work, the reviewer #2 and #141 rated weak reject with various concerns, including unconvincing experiments, very common framework, limited novelty, and the lack of ablation study. The authors provided response to the questions, but did not change the rating of the reviewers. Given the various concerns raised, the ACs agree that this paper can not be accepted at its current state.
Decision: Reject