LatentKeypointGAN: Controlling GANs via Latent Keypoints
Authors: Xingzhe He
Bastian Wandt
Helge Rhodin
Venue: ICLR 2022 Submitted (2022)
Keywords: Part Disentanglement
Unsupervised Learning

Abstract: Generative adversarial networks (GANs) have attained photo-realistic quality in image generation. However, how to best control the image content remains an open challenge. We introduce LatentKeypointGAN, a two-stage GAN which is trained end-to-end on the classical GAN objective with internal conditioning on a set of space keypoints. These keypoints have associated appearance embeddings that respectively control the position and style of the generated objects and their parts. A major difficulty that we address with suitable network architectures and training schemes is disentangling the image into spatial and appearance factors without domain knowledge and supervision signals. We demonstrate that LatentKeypointGAN provides an interpretable latent space that can be used to re-arrange the generated images by re-positioning and exchanging keypoint embeddings, such as generating portraits by combining the eyes, nose, and mouth from different images. In addition, the explicit generation of keypoints and matching images enables a new, GAN-based method for unsupervised keypoint detection.

Under review as a conference paper at ICLR 2022
LATENTKEYPOINTGAN: CONTROLLING GANS VIA LATENT KEYPOINTS
Anonymous authors Paper under double-blind review
ABSTRACT
Generative adversarial networks (GANs) have attained photo-realistic quality in image generation. However, how to best control the image content remains an open challenge. We introduce LatentKeypoinGGAN, trained end-to-end on the classical GAN objective with internal conditioning on set of space keypoints. These keypoints have associated appearance embed- dings that respectively control the position and style of the generated objects and their parts. A major difficulty that address with suitable network architec- tures and training schemes is disentangling the image into spatial and appearance factors without domain knowledge and supervision signals. We demonstrate that LatentKeypointGAn provides an interpretable latent space that can be used to re-arrange the generated images by re-positioning and exchanging keypoint em- beddings, such as generating portraits by combining the eyes, and mouth from different images. In addition, the explicit generation of keypoints and matching images enables a new, GAN-based method for unsupervised keypoint detection.
two-stage GAN which is
INTRODUCTION
It is a long-standing goal to build generative models that depict the distribution of example images faithfully. While photo-realism is reached for well-constraneed domains, such as faces, it remains challenging to make this image generation process interpretable and editable. Desired is latent space that disentangles an image into parts and their appearances, which would allow user to re- combine and re-imagine generated face interactively and artistically. There are promising attempts (Lorenz et al.) 2019) that use spatial image transformations, such as the thin plate spline (TPS) method, create pairs of real and deformed images and impose an equivariance loss discover keypoints and object appearance embeddings as the bottleneck of an autoencoder. Thereby, one can edit the image by moving the keypoints or modifying the appearance embedding. However, L et al. (2020) points out that the learned keypoints may be biased and only meaningful for the artificially introduced image transformations, the TPS transformation require careful parameter tuning for each domain, and Xu et al. (2020b) shows that some geometry and appearance entanglement remains.
+
Original
Keypoints
Moving
Switching
Removing
Adding
Figure 1: LatentKeypontGGAN generates images (first column) with associated keypoints (second column), which enables local editing operations, such as moving, switching, and adding parts.

---

Under review as a conference paper at ICLR 2022
To avoid these shortcomings of existing uutoencoder-basee techniques, propose alternative GAN- approach that does not rely on paired images and TPS transformation, thereby alleviat- ing their tomain-speciff tuning and bias. Instead of an autoencoder, keypoints as a latent embedding. Figure | shows how this Laten/KeypoinGGNN enables our main goal of editing the output image by changing the keypoint position, adding or removing points, and exchanging associated appearance embedding locally. Although entirely unsupervised, the learned keypoints meaningfully align with the image landmarks, such keypoint linked to the nose when generating images of faces. As byproduct, we can learn a separate keypoint detector on generated mmage-keypoinn pairs for unsupervised keypoint detection and to quantify localization accuracy.
GAN that includes
The difficulty of such unsupervised learning approaches lies in finding the right implicit bias to facilitate the desired disentanglement/ Locatello al.] (2019). LatentKeypoinGAAN is designed as a two-stage GAN architecture that is trained end-to-end on the standard GAN objective. In the first step, generator network turns the input values sampled from normal distribution into 2D) keypoint locations and their associated encoding. We ensure with suitable layer connectivity that some the encodings are correlated while others remain independent. These generated keypoints then mapped to spatial heatmaps of increasing resolution. The heatmaps define the position of the keypoints and their support sets the influence range of their respective encodings. In the second step, a SPADE-like (Park et al. 2019) image generator turns these spatial encodings into complete and realistic image.
We summarize contributions below:
1. Development of GAN-based framework for image manipulation and keypoint detection;
2. Design of a keypoint generator that models dependent and independent factors explicitly;
3. Empirical study comparing different editing methods in terms of perceptual quality;
4. A new methodology for keypoint detection that contests established autoencoder methods.
2 RELATED WORK
In the following, we discuss variants of deep generative models that learn to synthesize images from a collection of examples, focusing on methods modeling keypoints and those providing control on the image content.
GANs (Goodfellow et al. 2014) are trained to generate images from a distribution that resembles the training distributions. Recent approaches attain photo-realism for portrait images. Karras et al. 2018 train a progressive growth of the GAN's generator and discriminator. After that, Karras et al. 2019 2020) integrate multiplicative neural network layers to build StyleGAN and StyleGAN2. StyleGAN gains some control on the generated high-resolution, high-quality faces, by adopting ideas from style transfer (Gatys al. 2016; HHang & Belongie| 2017) and exchanging features of hidden layers between different samples. We build upon these network architectures and equip them with additional structure.
Editing GANs globally. More recently, efforts have been made on exploring the latent space of a
pre-trained StyleGAN for image editing (Shen et al., [2020] Jahanian al. 2020). To allow editing
real-world images, various encoders (Zhu et al. [020aa] Abdal et al. 2019 2020 Guan et al.) 2020
Wulff & Torralba 2020 Richardson et al.) 2020) have been trained to project images into the latent space of StyleGANs. These methods provide control over the image synthesis process, such as for changing age, pose, and gender. To enable rig-like controls over semantic face parameters that are interpretable in 3D, such as illumination, Tewari et al. (2020b a): Ghosh et al. (2000); Deng et al. (020)) integrate 3D face models (Blanz Vetter 1999 Li et al. (017) with GANs. Compared with these methods, our model focuses on detailed and local semantic controls. Instead of changing the face as a whole, our method is able to change a local patch without an obvious impact on other regions. Furthermore, keypoints provide control handles for animation without manual rigging. Therefore, LatentKeypointGAN can be applied to many different objects and image domains.
Editing GANs locally. Instead of exchanging entire feature maps for style editing, local modifica- tions are possible but require care to maintain consistent and artifact free synthesis. Collins et al. 2020) propose to cluster the middle-layer feature maps of StyleGAN and use the embedding to
2

---

Under review as a conference paper at ICLR 2022
edit local appearance but require human supervision to select suitable clusters. Instead of using isting feature maps. [Alharbi & Wonka (2020) and Kim et al. (202T) generate spatially-variabll maps by structured noise generators and then swap the pixel embedding of the maps edit the images. However, this requires to draw source and target regions by hand for every instance. By contrast, our model can explicitly change the pose by modifying keypoint locations that are consistent across images. Kwon & Yel (202I) manipulate attention maps to edit the pose and local shape, but they do not demonstrate local appearance editing. While enabling different forms of editing, all of the methods above require some form of pixel-level selection of regions at test time or manual cluster selection out of hundreds of candidates. In addition, they do not demonstrate on out-of ditto---istrbbttion editing, such moving parts relative to each other and adding removing parts, which requires stronger local disentanglement. (Zhan et al. (2019) 9 use a cycle consistency with a spatial transforma- tion network to fuse a foreground object on top of background reference. They can add a part but only if separate datasets with and without that part are available (c.g.. with and without glasses). In sum, our keypoint- based editing is both more intuitive and enables additional editing capabilities.
Conditional image synthesis methods synthesize images that resemble a given reference input, such as category labels (Mirza & Osindero 2014 Odena et al.. 2017). text (Zhang et al., 2017 Reed et al.] 2016). and layout (Zhao et al.] 2019; Sun & Wu 2019p. A variant of the conditional
GANs can do image-to-image translation (Isola et al. 2017 Huang et al. 2018 Liu et al. 2017 Zhu 201 a Wang et al.] 2018:: Park et al.] (219)) These approaches aim to transfer images from one domain to another while preserving the image structure, such as mapping from day night. Park et al. (201)) pioneered using ppatially-adaptive normalization to transfer segmentation masks to images, which we borrow and adapt to be conditioned on landmark position. To control individual aspects of faces. such as changing eye or nose shape, recent works condition on segmentation masks (Lee et al.. 2020 Yu et al. [2020] Richardson et al., 2020 Zhu et al., |020b] (Tn et al.) 2020), rough sketches Tseng et al.) Chen et al. 2020), landmarks ( (ang et al. 20 word labels such as with or w/o glasses (Cheng et al. 2020a), of faces as input. Hong et al. (2018) can superimpose new object without requiring exact mask, but still require one at training time. Compared with these methods, our model does not take any kind of supervision or other conditions at training time. It is trained in totally unsupervised manner. Still, our method allows the landmarks to learn meaningful location and semantic embedding that can be controlled at test time.
et al.
Unsupervised landmark discovery approaches aim to detect the landmarks from images without supervision. Most works train wwo-bannch autoencoders, where shape and appearance are disentan- gled by training on pairs of images where one of the two is matching while the other factor varies. The essence of these methods is to compare the pair of images to discover disentangled led landmarks. These pairs can stem from different views (Suwajanakorm et al., 2018, Rhodin et al. 2019). and frames of the same video (Siarohin et al. Kulkarni et al.) [019]] Minderer et al. 2019). How- ever, this additional motion information is not always available or is difficult to capture. Existing unsupervised methods trained on single images create pairs from spatial deformation and color shifts of a source image (Xing et al. 2018 (hu et al. 2018:: Thewlis et al. 2019 Li et al., 2020 Cheng let al. 2020b; Dundar et al. 2020). However, the parameters of augmentation strategies, such as the commonly used thin plate spline deformation model (Thewlis et al. 2017 Zhang et al. 2018 Jakab ee al.) 2018. Lorenz et al. 2019). are difficult to calibrate and the learned keypoints may be biased to be only meaningful for the transformation eLi ct al. 2020). Besides, the appearance information may be hidden in keypoint location (Xu et al. 2020b The cycle consistency-based methods (Wul Xu et al. (020b) cannot provide local editing ability. We show that our generator can well disentangle the appearance and keypoint location and control the images locally. Furthermore, the underlying idea of these methods, no matter whether they depend on multi-views, videos, or thin plate splines, is creating pairs of images to compare to find the clue of equivariant keypoints in comparison. On the other hand, our method does not need any paired correspondences. It is totally based on generation. Our GAN approach poses a viable alternative by training a keypoint detector on synthetic examples generated by LatentKeypoinGGAN.
let al. [019
3 LATENTKEYPOINTGA. METHOD AND METHODOLOGG
The unsupervised spatial disentanglement of image content that we target has been shown to be impossible without an implicit bias imposed by the network architecture (Locatello et al. 2019). To this end, our proposed LatentKeypointGAN architecture encodes position explicitly and appearance
3

---

Under review as a conference paper at ICLR 2022
Keypoint Generator
Spatial embedding layer
Image generator
Keypoints poordinates
MLP
Sap_poet
Healmaps
concel
SPADE 30%
MLP
W gichol
W
11.
Plg cab
MIPP
W
broadcast multppication
style
12
I:R38
Figure 2: Overview. Starting from noise z, LatentKeypoinGGAN generates keypoint coordinates, and their embeddings These turned into feature maps that localized around the keypoints, forming conditional maps for the image generation via SPADE blocks. The images are generated by toRGB blocks. At test time, the position and embedding of keypoints can be edited separately.
locally with two carefully designed sub-networks. First, the keypoint generator, IC, defines the spatial arrangement of parts and their embedding. Subsequently, spatial embedding layer, S, turns these sparse estimates into dense feature maps, and the image generator 9 up-scales these into a high- resolution image. Figure 2 shows the entire architecture. In the following, details and how we can train these networks end-to-end on a standard GAN objective, without any supervision on keypoints or masks. At inference time, the latent keypoints allow keypoint location and appearance interactively. Moreover, generated keypoint image pairs can be used train independent keypoint detector E, which enables unsupervised keypoint localization.
explain architectural
author the
3.1 KEYPOINT GENER ATOR
The keypoint generator, IC, learns the embeddings and spatial arrangement of image parts, such as eyes, nose, and mouth for describing a portrait image. takes three Gaussian noise vectors input Aup_pose Akp_app Zbg.emb ~ (aiee I x Dzix), where Dmoise is the dimension. Each vector is passed through a three-layer MLP to respectively generate the K keypoint coordinates L,... K, global style vector Wglobal Whg- Here K is pre-defined hyperparameter. Special is our embeddiing w that combines local and global factors,
kj E [-1, 1,3,
IR Dintee, and background embedding
w
Wgisbal & w const:
(1)
with the elementwise product 8 The constant embedding w w E R.Panss is designed to encode the keypoint semantics, e.g., left right eye. They are updated during the training but fixed during testing. The global style vector Welobal E 12 Pasted can be regarded as a noise vector with learned distribution; it ensures that different appearance is drawn for every generated image. We show in Appendix fthhat this factorization into dependent and independent factors is crucial and improves on straightforward variants that directly use an MLP. For instance, facilitates learning the correlation of the two eyes of a face while still covering the wide variety of eye variations.
3.2 SPATIAL EMBEDDING LAYER
With keypoint coordinates and embeddings generated, we now turn these point-wise estimates into discrete style maps that are further processed by the convolutional generator. The style map S R Deaest x Hx W for each keypoint j generated by broadcasting keypoint embedding w. E Datted at each pixel of the heatmap H R f and rescaling w by the heatmap value H (p).
(p()) (p) where H'(p) = exp (-llp- ke 1/1///)
has Gaussian shape. is centered at the keypoint location ki, and controls the influence range. We also define heatmap 4 for the background as the negative of all keypoint maps, H² (p) max, HH' [p)] The background heatmap is multiplied with the independent noise vector be generated from amb instead of keypoint emhedding, but treated equally otherwise. Then
4

---

Under review as a conference paper at ICLR 2022
concatenate all K keypoint style maps 1/- and the background style map She in the channel dimension to obtain the style map S E conditional variables at different levels of the image generator is explained in the next section.
How these style maps are used as
3.3
IMAGE GENERATOR
The image generator & follows the progressively growing architecture of StyleGAN Karras et al, 2019), combined with the idea of spatial normalization from SPADE (Park et al. 2019), which was designed to generate images conditioned on segmentation masks. Please refer to Appendix E for additional details. In the following,
explain how replace the role of manually annotated segmentation masks in the original with learned keypoints. Our generator starts from a learned 4 x 4 512 constant matrix and keeps applying convolutions and upsampling to obtain feature maps of increasing resolution. Following SPADE (Park et al. 2019), the original BatchNorm (Toffe) & Szegedy 2015) layers replaced with spatial adaptive normalization control the content. By contrast to SPADE, we do not condition on annotated segmentation masks but instead on the learned feature maps introduced Section [.1] and we do not use the residual links (He al. 2016) because we found it detrimental in combination with progressive training. The generator increases resolution layer-by-layer with multiple adaptive normalization layers, requiring differently-sizze feature maps. To create feature maps that match the respective spatial resolution of the generator, we apply Equation 2 multiple times.
3.4 Loss FUNCTIONS
Adversarial losses. We found it crucial for LatentKeypoinGGAN to use the non-saturating loss (Goodfellow et al. 2014).
C()) Een Be..v//l((exx(((D(((2)))
(3)
for the generator, and logistic loss
L(D)GAN Ez log(exp(D(G(z)) + Ex-ren log(exp(-D(x)) +
for the discriminator, with gradient penalty Mescheder et al. 2018) applied only on real data,
L(D) gp F Expppm VD((x.
Other losses such as hinge loss Park et al. (2019) failed.
Background loss. To further disentangle the background and the keypoints, and stabilize the key- point location, we introduce a background penalty,
(6) where , and 22 share the same keypoint location and appearance input noise, and only differ the background noise input. The H₃ and H2 are the background heatmaps generated by and Z2. With this penalty, we expect the keypoint location and appearance do not change with the background. The final loss for the discriminator and the generator are, respectively,
(( E,, (1 + @ G(z)) - (1 H1 @ G(22)).
L(D) = L(D)GAN + Ag))(D)er and L(9) = L(G)GAN + Ang (( mog.
EXPERIMENTS
We evaluate the improved quality and editing operations that our unsupervised LatentKeypoinIGAN approach for leamning disentangled representations brings about. We show the types of edits in Section 4.3 and quantitatively and qualitatively evaluate our improved editing quality compared with existing unsupervised and supervised methods in Section 4.4 and appendix, respectively. To test the keypoint consistency, that keypoints stick to particular object part, we compare to existing autoencoder frameworks for unsupervised landmark detection in Section 4.7 Even though not core focus, we show that by further tailoring the proposed architecture for precise localization, this new GAN method reaches state-of-the oftte-of-thearn art accuracy on a keypoint detection benchmark.
Since our main goal of enhanced editing quality is hard to quantify, we also conducted a user study to further validate these improvements. The appendix contains exhaustive ablation tests network architecture, hyperparameters, and training strategies to quantify the contributions of each individual component of our approach. The supplemental material contains additional video results.
5

---

Under review as a conference paper at ICLR 2022
Move
Edit location
Scalling
Keypolint exchange
Figure 3: Location and scale editing. The first column is the source and the last the target. The images in-between are the result of the following operations. First row: pushing the eye keypoint distance from 0.8x to 1.2x. Note that the marked eye keypoints in this row slightly shifted upward for better visualization. Second row: interpolating the hair keypoint to move the fringe from right to left. Third row: scaling the keypoint location and, therefore, the face from 1.15x to 0.85x. Fourth row: interpolating all keypoint locations, to rotate the head to the target orientation.
4.1 DATASETS
CelebA (Liu 2015 contains 200k celebrity faces. We this dataset
our model's
ability to discover the keypoints unsupervised. Following Thewliis et al. (2017). we use three subsets of this dataset: CelebA training set without MAFL (160k images), MAFL training set (19k images), MAFL test set (Ik images). MAFL (Zhang al. 2014) is subset of CelebA. More details can be found in Section
CelebA-HO (Lu et al. 2015) contains 30k celebrity face images selected from CelebA. Following Zhu al. 2020b). use the first 28k images training set and the rest 2k images testing set.
FlickrFaces-HO (FFHQ) (Karras al. 2019) consists of 70k high-quality portrait images, with more variation than CelebA (Liu et al. 2015). Therefore, we use this dataset to test our model's ability disentangle the local representations of images.
BBC Pose (Charles al. 2013) consists of 20 videos of different sign-language signers with various backgrounds. We use this dataset to test our model's ability to edit human appearance.
2015) consists of more than million images of bedrooms, with very diverse spatial structure. We use this dataset to test our model's generalization ability to edit entire indoor scenes.
LSUN bedroom YYu et al.
Hyperparameters. We use 512 512 images for face editing on FFHQ. 256 x 256 for FID cal- culation, comparison with SEAN (Zhu et al. (020b) on CelebA-HQ, and human pose experiments on BBCPose, and 128 128 for experiments on LSUN Bedroom, and comparison with unsuper- vised keypoint based methods on CelebA. Unless specified otherwise, we set 0.01 and use 10 keypoints. For editing experiments on FFHQ and detection experiments on CelebA, we augment by randomly cropping the training images size of 70-100%.
4.2 INTERACTIVE EDITING
We show the capabilities of LatentKeypoinGGAN by changing the keypoint locations and exchang- ing the keypoint embeddings between different images. As shown in Figure 3 we can thereby

---

Under review as a conference paper at ICLR 2022
Source
Difference
0.0
0.25
0.50
0.75
1.0
Target
Figure 5: Disentangled keypoint embeddings FFHQ. The leftmost images are the source and the rightmost images are the target. The cross landmarks on the first column denote the parts to be changed. The second column shows the difference between the original image and the changed image. The third to the second to last columns show the interpolation between the original image and the target image.
edit the face direction, face size, and individual key parts by changing the keypoint locations. If only subset of the keypoint embeddings changed, the other parts are not significantly affected. Figure [ shows heatmap of the area of interest. Since the GAN learns to generate a coherent face from its parts, global effects remain mostly unchanged, for instance, hairstyle and color. We discuss in Section 5] which degree this is desired and what lim- itations remain.
Modified Original
Additional ablation studies, editing, and interpolation amples are shown in Appendix Appendix and the supplemental video.
Figure 4: Disentangled Background. The background is changed while the faces fixed.
4.3 DISENTANGLED REPRESENTATIONS
We first analyze how background and keypoint embeddings disentangled and used for editing portrait images. Further examples are analyzed in the user study.
Disentangled keypoint embeddings. Figure 5 shows editing operations of independent facial re- gions. We fix the background noise, Zog_emb. and change some of the keypoint embeddings. This enables exchanging of eyes, mouth, or nose between persons. (eguuue]include heatmaps that visu- alize the difference between original and interpolated images. Their local activation highlights the spatial disentanglement of individual keypoint features.
Disentangled background. Figure 4 shows faithful change of backgrounds while keeping the face fixed. To this end, we fix the keypoint noise 7xp-pose> Akpapp- and change only the background noise input, Zog.omb- The local change in the three diverse examples shows that the background and keypoint encodings are disentangled well. The illumination and hair color is learned be part of the background, which makes sense global feature cannot be attributed to individual keypoints.
4.4
IMAGE QUALITY
We test our image quality on CelebA-HQ and compare it with an unsupervised autoencoder and segmentation condiconddti GANs, which require labeled training images. This comparison is biased in that the mask-conditioned GANs use the masks from the test set as input, while ours unconditional. To make an as fair as possible comparison, we add a version that uses a self- supervised keypoint detector (see Sec. 4.7 to condition the latent keypoint locations on the same test images as the baselines do. Scores are computed by using the Pytorch FID calculator (Seitzer) 2020). We followed Zhu al. (2020b) to the first 28k images the training and the last 2k images as the test set. We list the results in Table T Our approach attains better FID scores than those GANs conditioning segmentation masks. It gets close SEAN (Zhu al. 2020b).

---

Under review as a conference paper at ICLR 2022
Method
Conditioned segmentation masks
HID score 4
Pix2PixHD Wang et al. 2018
23.69
SPADE (Park ot al. 2019
segmentation masks
22.43
SEAN (Zuu el al. 20206
segmentation masks & image
17.66
Ours
keypoints (unsupervised, see Sec. 4.7
21.71
Ours (without detection)
(unsupervised)
20.12
Table 1: Image quality with respect supervision type on CelebA-HQ. Our FID scores improve on mask-based solutions while providing similar editing capabilities and being unsupervised.
Change individual keypoint embeddiings
Change individual keypoint locations
Original
Difference
Original
0.25
0.50
Target
Removing
Remaving light
Removing window
Adding
Adding another
Adding drawing
Figure 6: Editing on Bedroom. Left first row: interpolating the keypoint embedding on the curtain. Left second row: interpolating the keypoint embedding on the window. Right first row: changing the position of keypoint on the bed. Right second row: changing the position of the keypoints on the light. Left third row: removing parts from the bedroom. Right third row: adding parts.
which has auto-encoder architecture and their reported FID is reconstructed images instead of randomly generated appearances. addition, the user study shows that on editing tasks, our method also exceeds the image editing quality of SEAN significantly (preferred by 94. 94.78% of all responses).
4.5 GENERALIZATION TO OTHER DATASETS
LSUN Bedroom and BBC Pose. In Figure 6 we explore the editing ability of entire scenes on the LSUN bedroom dataset. No previous unsupervised part-based model has tried this difficult task before. We successfully interpolate the local appearance by changing the corresponding key- point embeddings and translating the local key parts (window, bed) by moving the corresponding keypoints. We can also remove or add some parts by removing or adding peaks in the Gaussian Heatmaps. Since our learned representation entirely. Figure exxlores the editing of individuals. Although artifacts remain due to the detailed background and motion blur in the datasets, pose and appearance can be exchanged separately.
two-dimensional, it is not possible to rotate objects
4.6 ABLATION TESTS
We test different variants of our architecture and demonstrate the effectiveness and necessity of our design. We test 1) removing the background; 2) removing the global style vector; 3) using additive global style vector instead of multiplicative ones; 4) using contrastive learned keypoint embeddings instead of multiplicative ones; 5) removing the keypoint embedding; 6) removing keypoints. We show the detection result on MAFL and FID score on FFHQ in Table 2 All contributions are crucial for the success and consistent for both detection and image quality tasks. The keypoint detection evaluation is explained the subsequent Section 4.7] For simplicity, the FID calculated resolution 256 256, between 50k generated images and the original dataset; slightly different from previous experiments.
4.7 UNSUPERVISED KEYPOINTS DISCOVERY
8

---

Under review as a conference paper at ICLR 2022
Appearance
Pose
Background
Figure 7: Editing on BBC Pose. The first row shows the source image and the second row the editing results. First three columns: the human appearance is swapped with the small target im- age. Middle three columns: changing the position to the one in the overlayed target. Last three column: changing the background (the bottom right corner shows the difference).
We demonstrated that our keypoints semantically meaningful by edit- ing the image via embeddings. In this part, we further demonstrate the consistency of our keypoints on the task of unsupervised keypoint detec- tion as introduced by [hewlls et al. (207)). We test whether keypoints consistently align to the same image parts, without randomly moving on the image when other factors such as color or texture are changed. To this end, we train standard ResNet de- tector (Xiao et al., 2018) supervised on 200,000 lentKeypiinGGAN is trained on the same number of training images as the other detectors, hence, results are comparable. For evaluation, we follow Thewlis al. (2017) and subsequent methods: As the order and semantics of unsupervised keypoints are undefined, a linear regressor from the predicted keypoints the 5 ground truth keypoints is learned on the MAFL training set. All images are resized to 128 x 128 and the training and test set of the MAFL subset from CelebA are excluded when training the LatentKeypoinGAAN The test error is the Euclidean distance to the ground truth on the MAFL test set, normalized by the inter-ocular distance in percent.
Method full model
T1 error %
FID
5.85%
23.50
w/o background w/o global style vector
6.43%
25.67
6.76%
28.75
adding global style vector
5.29%
42.02
contrastive keypoint embedding
7.53%
28.47
w/o keypoint embedding w/o keypoint
22.81%
32.41
34.69
Table 2: Quantitative ablation test on keypoint localiza-
tion (I₂) and image quality (FID). A lower number is better.
pairs generated on-the-fly by LatentKeypointGAN. Note, the La-
High accuracy can therefore only be obtained when estimated keypoints move consistently with the human-annotated ones, LatentKeypointGa.. the original 8.0% by Thewlis et al. (017)) and the most recent 2.8% by Dundar et al. (2020). We further tuned our architecture for precise localization by enforcing stronger translation equivariance of the keypoints (see Appendix B for details). This improved variant, LatentKeyponnGGANNo reaches 3.3%, which gets close Dunbar et al. (2.8%). On the in-the-wild variant of CelebA (more diverse head orientation, position, and scale), which is significantly more difficult, we reach an error of 5.6%. Here, outperform all existing methods reporting (0018) and 11.4% by Lorenz et al. (2019)). We use the same hyperparameters as before, showing the strong robustness new settings. Thereby, although not our main goal, LatentKeypoinGGAN also provides an alternative methodology for unsupervised keypoin//andmmak detection that avoids paired training and TPS deformation, which are notoriously difficult tune.
strikes low error of 5 5.9%, which lies between
this metrie (8.7% by Jakab et al.
4.8
USER STUDY ON EDITING QUALITY
We conducted user study to separately access image quality after editing, comparing against the unsupervised keypoint-basee method (Zhang et al. 2018 on CelebA (Liu et al. al. 2013) and the mask- conditioned method (Zhu et al. 2020b) CelebA-Ho HQ (Liu et al. 2015). The study confirms the main results: While Zhang et al. strike a high keypoint accuracy, its image quality (ours preferred by 92.17%% and editing results (our disentanglement preferred by 67.83% unsatisfactory. While

---

Under review as a conference paper at ICLR 2022
Zhu et al. reach a better FID score before editing, after editing, ours is the most realistic (preferred by 94.78%). Moreover, our disentanglement comes close to the one of Zhu et al. (49.57% voted equal quality). Zhu al. have better shape preservation because their masks provide silhouettes explicitly, but this likely also leads to the lower editing quality as masks and embeddings can become incompatible. For instance, the hair shape and strain direction need be correlated. The full details are in the appendix.
5 LIMITATIONS AND FUTURE WORK
For portrait images, the hair can mix with the background encoding, such that it is considered part of the background. Still, the hair can be changed by selecting a background embedding with the desired hairstyle. Moreover, the disentanglement into locally encoded features can lead to asymmetric faces, such as a pair of glasses with differently styled sides. For BBC Pose, the keypoints are not well localized. They are consistent across images with the same pose, which permits pose transfer, but are too inconsistent for keypoint detection. Limitations could be overcome by linking keypoints hicrarchically with skeleton and by reasoning about masks top of keypoints. While the face orientation portrait images can be controlled, we found that orientation changes on the bedroom images not reliable. The orientation is mostly baked into the appearance encoding. We believe that it will be necessary to learn a 3D representation and datasets with less-biased viewpoints.
6 CONCLUSION
We present a GAN-based framework that is internally conditioned on keypoints and their appearance encoding. providing an interpretable hidden space that enables intuitive editing. We provide the idea and implementation to avoid issues with TPS and related deformation models that are common in existing autoencoders. This LatentKeypoinGGAN also facilitates the generation of image-keypoint pairs, thereby providing a new methodology for unsupervised keypoint detection.
7
ETHICS STATEMENT
This research provides new unsupervised disentanglement and editing method. By contrast to existing supervised ones, e.g., those requiring manually annotated masks, ours can be trained on any image collection. This enables training on very diverse as well as personalized models for particular population and can thereby counter biases in the annotated datasets.
Photo-realistic editing tools have the risk of abuse via deep fakes. A picture of real person could be altered to express something not intended by the person. In that regard, our method has the advantage of only enabling the editing of generated images; it does not enable modifying real images; it only works in the synthetic to real direction. However, future work could extend it with an appearance encoder, which bears some risk.
Another risk is that the network could memorize appearances from the training set and therefore re- produce unwanted deformations of the pictured subjects. WhileBrock al. a (2018) and Karras al. (018) argue that GANs do not memorize training datasets, recently |Feng et al. (2021) empirically proved that whether GANs memorize training datasets depends on the complexity of the training datasets. Therefore, our model has some risk of leaking such identity information if the training set very small or the number of persons involved is limited, such as BBCPose. To mitigate the risk, only use established and publicly available datasets, in particular those that collect pictures of celebrities or other public figures and also those not containing any person (bedroom dataset).
Our IRB approved the survey and we collected explicit and informed consent at the beginning of the study. Results are stored on secure servers. This research did not cause any harm to any subject.
On the technical side, disclose all limitations in both the main paper and the supplementary. This research does not have conflicts with others. We credit the previous works the Introduction (Section TD and Related Work (Section 2)
10

---

Under review as a conference paper at ICLR 2022
8 REPRODUCIBILITY
We not only described our algorithm in Section but also provide detailed information of our archi- tecture, training strategy, and hyperparameters in the supplementary E All the datasets we used are publicly available and the pre-processing is also described in Section Furthermore, we provide the source code facilitate future work this direction of GAN-based generative image editing.
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE international conference computer vision, pp. 4432 4441, 2019.
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegant+: How to edit the embedded im- ages? Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8296-8305, 2020.
Yazeed Alharbi and Peter Wonka. Disentangled image generation through structured noise injection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 1134-5142, 2020.
Bilal Alsallakh, Narine Kokhlikyan, Vivek Miglani, Jun Yuan, and Orion Sbblit--Richardson. Mind the pad {(cnns can develop blind spots. In International Conference Learning Representa- tions, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1rXiv:1607.06455 2016.
Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pp. 187-194, 1999.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Representations, 2018.
James Charles, Tomas Pfister, Derek Magee, David Hogg, and Andrew Zisserman. Domain adapta- tion for upper body pose tracking in signed tv broadcasts. 2013.
Shu-Yo Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu. Deepfacedrawing: deep gener- ation of face images from sketches. ACM Transactions Graphics (TOG), 39(4)22--1, 2020.
Yen-Chi Cheng, Hsin- Ying Lee, Min Sun, and Ming-Hsuan Yang. Controllable image synthesis via segvae. In European Conference Computer Vision, 2020a.
Zezhou Cheng, long-Chy) Su, and Subhransu Maji. Unsupervised discovery of object landmarks via contrastive learning, arXiv preprint arXiv:2006. 14787, 2020b.
Edo Collins, Raja Bala, Bob Price, and Sabine Susstrunk. Editing in style: Uncovering the local semantics gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5771-5780, 2020.
Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong. Disentangled and controllable face image generation via 3d imitative-contrastiv learning. In IEEE Computer Vision and Pattern Recognition, 2020.
Aysegul Dundar, Kevin J Shih, Animesh Garg, Robert Pottorf, Andrew Tao, and Bryan Catanzaro. Unsupervised disentanglement of pose, appearance and background from images and videos. arXiv preprint arXiv:2001.09518 2020.
Qianli Feng, Chenqi Guo, Fabian Benitez- and Aleix M. Martinez. When do gans replicate? on the choice of dataset size. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6701-6710, October 2021.

---

Under review as a conference paper at ICLR 2022
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 2414-2423, 2016.
Partha Ghosh, Pravir Singh Gupta, Roy Uziel, Anurag Ranjan, Michael Black, and Timo Bolkart. Gif: Generative interpretable faces. arXiv preprint wrXiv:2009.00144 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Shanyan Guan, Ying Tai, Bingbing Ni, Feida Zhu, Feiyue Huang, and Xiaokang Yang. Collaborative learning for faster stylegan embedding. arXiv preprint arXiv:2007.01753. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026 1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Seunghoon Hong, Xinchen Yan, Thomas E Huang, and Honglak Lec. Learning hierarchical seman- tic image manipulation through structured representations. In Advances in Neural Information Processing Systems, pp. 2012-71122771 2723, 2018.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1501- 1510, 2017.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. Proceedings of the European Conference on Computer Vision (ECCV), pp- 172-189, 2018.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03166 2015.
Md Amirul Islam, Sen Jia, and Neil DB Bruce. How much position information do convolutional neural networks encode? arXiv preprint arXiv:2001.08246 2020.
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1 1125-1134, 2017.
Ali Jahanian Lucy Chai and Phillip Isola. On the "steerability" of generative adversarial networks. In International Conference on Learning Representations, 2020. URL https: //openrevi ew. net/fform?? d=
Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. In Advances in neural information processing systems, pp. 4016-4027, 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018. TRRLEtpp. http/opeevviiieeee net
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
12

---

Under review as a conference paper at ICLR 2022
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110-8119, 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. arXiv preprint urXiv:2106. /24233 2021.
Osman Semih Kayhan and Jan C van Gemert. On translation invariance in cnns: Convolutional layers can exploit absolute spatial location. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14274-14285, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning, arXiv preprint arXiv:2004.1136: 2020.
Hyunsu Kim, Yunjey Choi, Junho Kim, Sungjoo Yoo, and Youngjung Uh. Exploiting spatial dimen- sions of latent in gan for real-time image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Tejas D Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. In Advances in neural information processing systems, pp. 10724-10734, 2019.
Gihyun Kwon and Jong Chul Ye. Diagonal attention and style-based gan for content-style disentan- glement in image generation and translation. In Proceedings of the IEEE international conference on computer vision, pp. 13980-13300 13989, 2021.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5549-5558, 2020.
Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning model of facial shape and expression from 4D scans. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 19:1):194:1119:117
2017. URL
htt.d//...rgg11.1111111333333300 3130813
Weijian Li, Haofu Liao, Shun Miao, Le Lu, and Jiebo Luo. Unsupervised learning of landmarks based on inter-intra subject consistencies. arXiv preprint arXiv:2004.07936 2020.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image -omaggge translation networks. In Advances in neural information processing systems, pp. 700-708, 2017.
Ziwei Liu, Ping Luo. Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pp. 41244 PMLR, 2019.
Dominik Lorenz, Leonard Bereska, Timo Milbich, and Bjom Ommer. Unsupervised part-based dis- entangling of object shape and appearance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10955-10964, 2019.
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models. In in ICML Workshop on Deep Learning for Audio, Speech and Language Processing. Citeseer, 2013.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning. pp. 3481-3490. PMLR, 2018.
13

---

Under review as a conference paper at ICLR 2022
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole. Kevin P Murphy, and Honglak Lee.
Unsupervised learning of object structure and dynamics from videos. In Advances in Neural Information Processing Systems, pp. 2019.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv: 2014.
arXiv preprint
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2642-2651, 2017.
Taesung Park, Ming-Yi Yu Liu, Ting-Chun Wang, and Jun- Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings the Pattern Recognition, pp. 2337-2346, 2019.
the IEEE Conference on Computer Vision and
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. arXiv preprint arXiv 2016.
Helge Rhodin, Victor Constantin, Isinsu Katircioglu, Mathieu Salzmann, and Pascal Fua. Neural scene decomposition for multi multi-person motion capture. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), June 2019.
Elad Richardson, Yuval Alaluf, Or Patashník, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-image translation. arXiv preprint arXiv:2008.00955 00951, 2020.
Maximilian Seitzer. pytorch-fidt FID Score for PyTorch. pytorch-fid. August 2020. Version 0.1.1.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou, Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 9243-9252, 2020.
Zhixin Shu, Mihir Sahasrabudhe, Riza Alp Guler, Dimitris Samaras, Nikos Paragios, and Iasonas Kokkinos. Deforming autoencoders: Unsupervised disentangling of shape and appearance. In Proceedings of the European conference on computer vision (ECCV), pp. 650-665, 2018
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2377-2386, 2019.
Wei Sun and Tianfu Wu. Image synthesis from reconfigurable layout and style. In Proceedings of the IEEE International Conference on Computer Vision, pp. 10531-10540, 2019.
Supasorn Suwajanakorn, Noah Snavely, Jonathan Tompson, and Mohammad Norouzi. Discovery of latent 3d keypoints via end-to-end geometric reasoning. In Advances in neural information processing systems, pp. 2059-2070, 2018.
Zhentao Tan, Dongdong Chen, Qi Chu, Menglei Chai, Jing Liao, Mingming He, Lu Yuan, Gang Hua, and Nenghai Yu. Semantic image synthesis via efficient class-adaptive normalization. arXiv preprint (Xiv:2002.006644, 2020.
Ayush Tewari, Mohamed Elgharib, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt, et al. Pie: Portrait image embedding for semantic control. arXiv preprint arXiv:2009.094.0 2020a.
Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj. Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zöllhofer. and Christian Theobalt. Stylerig: Rigging stylegan for 3d control over portrait images, cvpr 2020. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEF, june 2020b.
14

---

Under review as a conference paper at ICLR 2022
James Thewlis, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks by factorized spatial embeddings. In Proceedings of the IEEE international conference on computer vision, pp. 5916-5922 2017.
James Thewlis, Samuel Albanie, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of land- marks by descriptor vector exchange. In Proceedings of the IEEE International Conference Computer Vision, pp. 6361-6371, 2019.
Hung-Yo Yu Tseng Matthew Fisher, Jingwan Lu, Yijun Li, Vladimir Kim, and Ming-Hsuan Yang. Modeling artistic workflows for image generation and editing. In European Conference on Com- puter Vision, 2020.
Ting- Chgun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8798 8798-8807, 2018.
Wayne Wu, Kaidi Cao, Cheng Li, Chen Qian, and Chen Change Loy. Transgaga: Geometry-avee -awar unsupervised image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 8012-8021, 2019.
Jonas Wulff and Antonio Torralba. Improving inversion and generation diversity in stylegan using a gaussianized latent space. arXiv preprint (XXiv:2009006525 2020.
Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In European Conference Computer Vision (ECCV), 2018
Xianglei Xing, Ruiqi Gao, Tian Han, Song-Chun Zhu, and Ying Nian Wu. Deformable gen- erator network: Unsupervised disentanglement of appearance and geometry. arXiv preprint arXiv: 06298, 2018.
Rui Xu, Xintao Wang, Kai Chen, Bolci Zhou, and Chen Change Loy. Positional encoding as spatial inductive bias in gans. arXiv preprint arXiv:2012.05217 2020a.
Yinghao Xu, Ceyuan Yang. Ziwei Liu, Bo Dai, and Bolei Zhou. Unsupervised landmark learning from unpaired data. arXiv preprint arXiv:2007.01053 2020b.
Xin Yang, Yuezun Li, Honggang Qi, and Siwei Lyu. Exposing gan-synthesized faces using landmark locations. Proceedings of the ACM Workshop on Information Hiding and Multimedia Security, pp. 113-118, 2019.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large- scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506. 03365, 2015.
Jun Yu, Xingxin Xu, Fei Gao, Shengjie Shi, Meng Wang, Dacheng Tao, and Qingming Huang. Toward realistic face photo-sketch synthesis via composition-aidee gans. IEEE Transactions on Cybernetics, 2020.
Fangneng Zhan, Hongyuan Zhu, and Shijian Lu. Spatial fusion gan for image synthesis. In Proceed- ings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3653-3662, 2019.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative ad- versarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 5907-5915, 2017.
Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, and Honglak Lee. Unsupervised dis- covery of object landmarks as structural representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2694-2703 2018
Zhanpeng Zhang. Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. 09 2014, ISBN 978-3-319-10598-1 doi: 10. 00.1007978---31111 10599.4 7.
15

---

Under review as a conference paper at ICLR 2022
Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal. Image generation from layout. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 0884-5593, 2019.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. arXiv preprint arXiv:2004. 000040 00049, 2020a.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017a.
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexci A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in neural information processing systems, pp. 465-476 2017b.
Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic -daative normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5104-5113, 2020b.
16

---

Under review as a conference paper at ICLR 2022
In this appendix, we present additional details on the neural network architectures, the progressive training. and hyperparameters. Furthermore, we show more qualitative results in the supplemental videos that are embedded and explained in the video/index.htm..
A KEYPOINTS ILLUSTRATION
Figure [8 shows the keypoints generated on all datasets, The keypoints are semantically meaningful and very consistent across different instances.
FFHQ
Bedroom
HelebA-HQ
BBC Pase
Figure 8: Keypoints. We show the keypoints on each dataset.
B TUNING LATENTKEYPOINTGAN FOR KEYPOINTS DETECTION
We desire an architecture that encodes the location of parts solely in the keypoint locations to im- prove keypoint localization and the subsequent learning detector. Even though the convolutional generator is translation invariant, additional care is necessary to prevent leakage of global position the image boundary and from the starting tensor. All these tuning steps are explained below.
Padding Margins. As pointed out by Islam et al. 20200; Alsallakh et al. 2021 Xu et al. (2020a);
Kayhan Gemert (2020), convolutions with zero padding are very good at implicitly encoding ab- solute grid position at deeper layers. To prevent this, we follow Karras et al. (202T). By maintaining a fixed margin around the feature map and cropping after each upsampling, we effectively prevent the leaking of absolute position and the bias to the center because none of the generated pixels ever reaches a boundary condition. We use a 10-pixel margin. Note that such measures do not directly apply autoencoders who are bound to the fixed resolution of the input.
Positional Encoded Starting Tensor. We remove the x 4 x 512 starting tensor because it can encode absolute grid position. We replace it with the positional encoding M of difference between keypoints k kj...., kx and the grid positions p.
M(p) = [sin(T Linear([p k1, ..... p - kk])), coS(TT * Linear((p - ki, p kxl))]- (8)
The underlying idea is only encode relative distances to keypoints but not the image boundary.
Larger Starting Tensor. We found that starting the feature map from 32 x 32 instead of 4 x 4 improves keypoint localization accuracy.
Background Handling. The background should be consistent across the whole image, but com- plete occlusion inaccurate to model with the sparse keypoints and their limited support in the intermediate feature maps. Hence, we introduce the explicit notion of foreground mask that blends in the background. The mask is generated as one additional layer in the image generator. To generate the background, we use separate network that is of equivalent architecture LatentKeypointGANN tuned. Note that in background generation we use AdaIN (Huang & Belongie 2017) instead of SPADE because there is spatial sensitive representation, such keypoints. Foreground and background are then blended linearly based on the foreground mask. We use an x convolution layer to generate the final RGB images.
17

---

Under review as a conference paper at ICLR 2022
Method
Aligned (K=10)
Wild (K=4)
Wild
Thewlis et al. (2017
7.95%
31.30%
Zhang et al.
2018
3.46%
40.82%
Lorenz al.
2019
3.24%
15.49%
11.41%
IMM (Jakab et al. 2018)
3.19%
19.42%
8.74%
Dundar et (2020)
2.76%
LatentkeypointCan-tuue
3.31%
12.1%
5.63%
larger starting tensor
3.50%
14.22%
8.51%
separated background generation
4.24%
19.29%
14.01 %
positional encoded starting tensor
5.26%
24. 12%
23.86%
margin
5.85%
25.81%
21.90%
Table 3: Landmark detection CelebA (lower is better). The metric is the landmark regression (without bias) error in terms of mean L2 distance normalized by inter-ocular distance. The bottom four rows shows our improvement step by step. We use the same number of keypoints as previous methods.
Simplification In addition, we remove the progressive training (Karras et al. 2018 and use the ResBlocks defined by Park al. (2019). This step is for simplification (quicker training results on medium-sized images) and does not provide a measurable change in keypoint accuracy.
As shown Table 3 each step of improvement contributes significantly the keypoint accuracy and consistent improvement on prior works on the two in-the-wild settings. Please see the main paper for the complete comparison. The FID of LatentKyyoinnGAANNN on CelebA resolution 128 x 128 is 18.34.
C IMAGE EDITING QUALITY COMPARISON
Figure validates the improved editing quality, showing comparative quality to conditional GAN (supervised by paired masks), and superior quality unsupervised methods (greater detail in the hair and facial features). Note that for SEAN (Zhu et al. (2200)) we use the edited images (com- bine mask with different face appearance) instead of reconstructed images (combine mask with the corresponding face appearance) for fair comparison with other GAN methods. The qualitative im- provement is further estimated in the subsequent user study.
The main paper reports FID scores for all those datasets where prior work does. To aid future comparisons we also report the remaining FID scores: 17.88 on FFHQ, 18.25 on CelebA, 30.53 on BBC Pose, and 18.89 on LSUN Bedroom. The FID is calculated by the generated 50k images and the resized original dataset.
We also qualitatively compare our method with publicly available versior Lorenz et al. (2019) on LSUN Bedroom, using the standard parameters that work well on other datasets. As shown in Figure 10 their model generates trivial keypoints and fails to reconstruct the bedroom images.
D EDITING QUALITY SURVEY
We designed comparisons in 4 different aspects to demonstrate our editing quality. We compare to two methods, unsupervised keypoint-based method (Zhang al. 2018) CelebA (Liu al.) 2015), and one mask-conditioned method (Zhu et al.) 2020b) on CelebA-HO (Liu et al. 2015). For both methods, we follow their experiment design in their papers to make fair comparison. For each question, we ask the participants to chose from 3 options: 1) ours is better; 2) theirs is better; 3) both methods have equal quality. The order of the pair to compare (ours in the first the second) in the questions is randomly generated by the function numpy. random. choice ce of the package Numpy in Python with random seed The results are illustrated in Table The entire study is in anonymous form in the supplemental document survey/index.hmm
com htwp:h/eeeeepppeeeerr////eeseeee sed di sentangl hg/tree/
reproducing ines
18

---

Under review as a conference paper at ICLR 2022
Cusprioc
Unopariiad heppeie
STADE Paaieral p012
MuAGAN Sararl
Sauaa (00R
Figure 9: Image editing quality comparison. We compare the image editing quality with both, supervised (left) and unsupervised (right). LatentKyypinnGAN improves on the methods in both classes.
Detected Keypoints
Reconstructed Images
Figure 10: Lorenz al. (219 on LSUN Bedroom. (Left) Detected keypoints, The keypoints are static and do not have semantic meaning. (Right) Reconstructed images. The reconstruction completely fails.
Study details. We invited 23 participants answering 35 questions in total, with 5 for each of the categories above and an additional 5 comparing to randomly selected real images from the celebA training set. The generated images are selected as follows. We generate 32 images with LatentKey- pointGAN. For the baselines we take the corresponding images from their paper (as implementations are not available). We then find our image (out of the 32) that best match the baseline images (in gender, pose, head size). If multiple images are similar, we use the first one.
Editing image quality. We edited the images by swapping the appearance embedding between different images. In each question, we show one image of ours and one image of theirs. We ask the participants to compare the resulting image quality.
Part disentanglement. To compare with Zhang et al. (201)), moved part of the keypoints, as they did in their paper. To compare with Zhu et al. (020b), we exchange the part embedding between different images, as they did in their paper. In each question, we show one pair of images of ours and one pair of images of theirs. We ask the participants to choose the one with better spatial disentanglement regardless the image quality.
Identity preservation while changing expression. We compare identity preservation with Zhang et al. (2018). Following their paper, we change part (or all) of the keypoints to change the expression show the two pairs of images. Each pair contains two images, one
of the face. In each question, original image, and one edited image. We ask the participants to choose the pair that can better preserve the identity of the face regardless of the image quality, as quality is assessed separately.
Shape preservation while changing appearance. We compare the shape preservation with iZu et al. 2020b). We edited the images by swapping the appearance embedding between different images. In each question, shape source image and one appearance image, and one combined image. We ask the participants to choose the triplet where the combined image has the more similar shape as the shape source image regardless of the image quality.
show the triplets of images. Each triplet contains three images, one
Interpretation comparative. This study confirms the findings of the main paper. Our method outperforms Zhang et al. in all metries in Table 4 We also outperform SEAN in image editing
19

---

Under review as a conference paper at ICLR 2022
Aspect
Method compare
In favour of ours
In favour of others
Equal quality
Editing image quality
Zhang et al. 2018
92.17%
0.87%
6.96%
Editing image quality
SHAN (Zhu ct al. 2020h
94.78%
2.61 %
2.61%
Part disentanglement
Zhang et al. 2018
67.83%
5.22%
26.95%
Part disentanglemmnt
SEAN (Zhu ct a 2020b
28.69%
21.74%
49.57%
Identity preservation
Zhang et al 2018
55.65%
14.78%
29.57%
Shape preservation
SEAN (Zhu et al.
2020b
33.91%
46.96%
19.13%
Table 4: Survey results.
We compare 4 different aspects with other methods. The first one is the editing image quality. The second one is part disentanglement. The third one is identity preservation while changing expression. The last one is shape preservation while changing appearance.
quality. This confirms our claims of superior editing capability but may be surprising on the first glance since they attain higher quality (better FID score) unedited images. However, it can be explained with the limited editing capabilities the mask-based approaches discussed next.
Participants give SEAN higher shape preservation quality (47% in favour and 19% equal), which is expected since it conditions on pixel-accurate segmentation masks that explicitly encode the feature outline. However, the masks have the drawback that they dictate the part outline strictly, which leads to inconsistencies when exchanging appearance features across images. For instance, the strain direction of the hair and their outline must be correlated. This explains why our mask-free method attaining significantly higher image quality after editing operations (95% preferred ours). Hence, the preferred method depends on the use case. F.g., for the fine editing outlines SEAN would be preferred while ours is better at combining appearances from different faces.
additional strong outcome is that our unsupervised approach has equal disentanglement scores compared to SEAN; 50% judge them equal, with 29% giving preference to ours and only 22% giving preference to SEAN. Validating that LatentKeypoinGGAN enables localized editing.
Interpretation realism. When comparing GAN (without modifying keypoint location or appearance) to real images at resolution 128 x 128 of the training set, 42% rate them as equal. Surprisingly 33% even prefer ours over the ground truth. This preference may be because the ground truth images have artifacts in the background due to the forced alignment that are smoothed out in any of the generated ones. Overall, these scores validate that the generated images come close to real images, even though minor artifacts remain at high resolutions.
E EXPERIMENTS DETAILS
SPADE Park et al. 2019) As shown in Figure II SPADE takes two inputs, feature map and style map, and use the style map calculate the mean and standard deviation, which used to denormalize the batch normalized feature map. Formally speaking, let F E NxCx H:xw; be a i-th feature map the network for batch of N samples, where C, is the number of channels.
Here we slightly abuse the notation to denote N batched style maps of size (H,. W: as St E R XXX++)) H; H::W; The same equation as for BatchNorm (loffe & Szegedy) 2015)
20155) is used to normalize the feature map, but now the denormalization coefficients stem from the conditional map, which in case is the processed style map. Specifically, the resulting value of the spatial adaptive normalization is
F n.c.p.x - HC (.).
A... (S, (S.
where E (1,..., N) is the index of the sample, () is the pixel index. The n¹ and of are the mean and standard deviation of
C). is the index of channels of the
feature map, and (y, channel The Te. and B (S,) are the parameters denormalize the feature map. They are obtained by applying two convolution layers on the style map S
Learning rate and initialization. We set the learning rate to 0.0001 and 0.0004 for the generator and discriminators, respectively. To let our model learn the coordinates reliably, we first set the learning rate of the ML.,, which generates keypoint coordinates to 0.05x the generator learning rate,
20

---

Under review as a conference paper at ICLR 2022
SPADE
Keypoint- ConvBlock
Aiy m
SNiDE
SPADE
malemaa 8
Figure 11: (Left) Spatially Adaptive Normalization. SPADE takes two inputs, feature map and style map. It first uses batch normalization to normalize the feature map. Then it uses the style map to calculate the new mean map and new standard deviation map for the feature map. (Right) Keypoint-based ConvBlock.
Le. 0.00005. We use Kaiming initialization (He et al. 2015) for our network. We initialize the weights of the last layer of the MLP that generates the keypoint coordinates to 0.05x the Kaiming initialization.
Progressive Growing Training we adopt progressive growing training (Karras et al. 2018) to al- low larger batch size, which helps both on keypoint localization and local appearance learning. This is likely due to the BatchNorm that is essential in SPADE. We also tried to replace the BatchNorm in SPADE with LayerNorm (Ba et al., 2016) and PixelNorm (Karras et al., 2019). but both of them cause mode collapse. We use scheduled learning rate for the Keypoint Generator K. As illustrated in Figure [2] at each resolution stage, the training is divided into adapting period and non-adapting period. We set the learning rate of I. zero in the adapting period and back to normal in the non- adapting period. In the adapting period, the training follows PGGAN (Karras et al., 2018) where the feature map is linear combination of the larger resolution RGB image and current resolution RGB image. The coefficient gradually increases from O to 1. At the end of the adapting period, the network is fully adapted generate higher resolution images. In the non-adapting period, the network generates high-resolution images without the linear combination. Following StyleGAN (Karras et al., 2019), we start from a x 4 x 512 learned constant matrix, which optimized during training and fixed during testing. We use the keypoint- ConvBlock (2) and bilinear upsampling to obtain feature maps with increasing resolutions. Unlike PGGAN (Karras et al. 2018) and Style- GAN Karras et al. 2019), who generating RGB images from feature maps of all resolutions (from x 4 to 1024 1024), we start generating RGB images from the feature maps of at least 64 x 64 resolution. This is possible with the keypint generator and its spatially localized embeddings taking over the role of low feature maps. It helps to locate the keypoints more accurately.
Generator.
We illustrate the LatentKeyponnGGAN generator in Figure 12 linearly combined by the output of toRGB block, where the weights depend on the training stage.
The output image is
Discriminator. We illustrate the discriminator in Figure 12 For each resolution, we use two convolutions followed by Leaky ReLU (Maas et al. 2013). The first convolution has a kernel size 4 4 and stride to downsample the feature map to 0.5x. The second convolutions have a kernel size 3 x 3 and stride to extract features.
Hyperparameter Setting For all experiments in image generation, we use leaky ReLU (Maas) ct al. 2013) with a slope 0.2 for negative values as our activation function. We use ADAM opti- mizer (Kingma& & Ba 2015) with 31 0.5 and 32 0.9. We set the learning rate to 0.0001 and 0.0004 for generator and discriminators, respectively (Heusel et al. (017). We start from generating 64 x 64 images in the progressive training. The batch size for 64 1282 256 5122 images
128, 64, 32, 8, respectively. We set Agp = 10 and Apg = 100. We set D Duoise 256 and Dembed 128 for all experiments unless otherwise stated (in ablation tests).
21

---

Under review as a conference paper at ICLR 2022
X4 64
S12-1-4
2x
eyporstear Dervicoors 4-1 Z
128 128 toAGB
128 129
seporsteed Bendiritii 1. Losempee
IoRo8
lnGGB
Coreded sett, Ooeeded (11), 10s 1986
Tum
Conviee Cues203 we1)
Kepoin beset Condlood111 Upirmped)
fromAGB
fromAGB
Hegusi we po Meuurgg
Convetel, 9421 Coen w-1), 2012 16.
fromAG8
0.5x
128 128 0.5x 1
0.5x
Genvtell sr0t Conveed (i1)) (512,8
Kepuose asdid Corelitaae
Conwiten rully Conadiak swil (515, 3.20
64
64x 64
Kepolin Somdlegglo
Plahago
Adapting period
Non-adipling pertod
Generator
Discriminator
Progressive Training
Figure 12: Detailed architecture. (Left) LatentKeypoinGGAN generator. The numbers in the parenthesis is the output dimension of the Keypoint-based ConvBlock. For example, (512, 4, 4) means the output feature map has resolution of 4 x 4 and the channel size is 512. The toRGB blocks are 1 convolutions to generate the RGB images with the same resolution as corresponding feature maps. (Middle) LatentKKypointCC.
discriminator. The number in the last parenthesis the output dimension. For example, (512, 4, 4) means the output feature map has a resolution of 4 x 4 and the channel size is 512. At each resolution, we apply two convolutions, downsample feature maps and one with stride to extract features. Leaky ReLU (Maas et al. 2013) used after all convolutions except the linear layer the last.. (Right) Progressive Training.. The adapting period is the same as PGGAN KKrrra et al. 2018) and StyleGAN (Karras et al. 2019). In the non-adapting period, we do not use the linear combination.
with stride 2 to
Dataset
background module and loss
Ce FFHQ
yes ycs
0.01
0.01
Bedroom
0.01
BBC Pose
yes
0.025
Table 5: Setting for different datasets. For the Bedroom dataset, we do not use the background module and loss. For the BBC Pose dataset, we use T 0.025.
We lists the different 7S and different background setting for all experiments in Table 5 In CclebA- HQ and FFHQ, the foreground is naturally disentangled from the background. The face can be freely moved on the image. However, in the Bedroom dataset, all objects and their parts are strongly correlated. For example, the bed cannot be moved the ceiling. and the window cannot be moved to the floor. Therefore, we treat every object in the bedroom the possible motion is restricted to plausible locations (see the supplementary video). separate background embedding does not make sense. Therefore, set the background (H 0) and the background loss Ang = for the experiments on the Bedroom dataset.
key part, even the floor, but
F ABLATION TESTS
F.1 ABLATION TEST THE NEURAL NETWORK ARCHITECTURE
We show the detection result on MAFL and FID score on FFHQ in Table for different architectures we tested below. The detection follows Section 4.7 The FID is calculated at resolution 256 x 256, by generated 50k images and the resized original dataset. Note that it is different from the main paper for simplicity since the goal of this part is ablation tests instead of comparing with others.
Removing background embedding. We remove the background embeddings from our architec- ture e2b and Whg in Figure 2 In this case, the keypoint embedding controls the whole appear-
22

---

Under review as a conference paper at ICLR 2022
ance of the image. In addition, as shown in Figure 14 the keypoints not exactly located at the key parts, though they are still consistent among different views.
Removing global style vector. We remove the global style vector Wglobal. Therefore, all the key- point embeddings are constant. Only keypoint location and background embedding are different among the images. In this case, the keypoint embedding works equivalent one-hot encoding, and cannot fully capture the variations on the key parts. Therefore, leads to inaccurate keypoints, as shown in Figure 14 Furthermore, we observed that without Wglobal, the network hides the appear- ance information in the keypoint location.
Changing keypoint embedding generation. We change the keypoint embedding generation in two ways. The first way is generating constant embedding wamnst and global style vector Wglohal just as before and then add them elementwisely instead of multiplying them. Formally speaking, for each keypoint j. its corresponding embedding
w global ( w
const:
(10)
where @ means clementwise addition. This gives slightly higher detection accuracy but lower image quality. We observe that in this case, the background controls the foreground appearance. However, different from Removing global style vector F.T the appearance information is not hidden in key- point locations. We believe this is because that Wgisbal works as noise to avoid the network from hiding foreground appearance information in keypoint location. As result of good disentanglement of appearance and keypoint location, the keypoint detection accuracy slightly increases. However, again, this setting, the keypoint embedding cannot fully capture the variations of the key parts. Therefore, the background takes the control of appearance.
The second way is generate [w=]]== together from Zip_app using MLP. In this case, there is no constant embeddings or global style vector. To force the embedding of the same keypoint to be similar, and the embedding for different keypoints to be different, we use Supervised Contrastive Losses (Khosla et al. 2020).
exp(wá w /T) EREAC) exp(w) S w w2/T)
Ccontrastive (9)= --
Σ JEJ
log KEKU)
(1I)
K()
where
A()) = (i: : w w are in the same batch}
K()) = {i w w belong to the same keypoint in the same batch}
J (indices of all keypoint embeddings in the same batch)
As shown in Figure 14 the keypoints are neither on the key parts nor consistent. We further visualize
the embeddings with T-SNE and PCA in Figure I3 Although the contrastive learned embedding has comparable T-SNE with our multiplicative design, the PCA shows that our multiplicative embedding is linearly separable while contrastive learned embedding is not. Hence, we demonstrate that our original design of elementwise production simple and effective.
Removing keypoint embedding. We remove the keypoint embedding w entirely. In this case, we only have background embedding Whg and the keypoint location. Thus, instead of generating the style map S, we directly concatenate the keypoint heatmaps {H*1K ] and the broadcasted back- ground style map to generate the style map without keypoint embedding. As shown in Figure 14 the keypoints are not meaningful or consistent. The keypoint location hides the appearance and is entangled with the background severely.
Removing keypoints. If we remove the keypoints, then SPADE (Pak et al. 2019) degenerates to AdaIN (Huang & Belongie, 2017). Instead of using style map S (2D). vector ( (ID), which is the background embedding. In this case, we do not have the power of local controllability on the generated images.
style
F.2 A ABLATION TESTS ON THE HYPERPARAMETERS
Ablation test on the dimension of embeddings. Different numbers of embedding dimensions make the expression power vary. As shown in Table 6 larger Dembed leads to larger error on MAFL.
23

---

Under review as a conference paper at ICLR 2022
T-SNE
PCA
Figure 13: Ablation study multiplicative embedding. We show the T-SNE and PCA visualiza- tion embeddings learned on FFHQ. The first two column shows keypoint embeddings and the last two column shows keypoint embeddings and background embedding.
Remeving background embedcing
Removing global style vecior
Changing contreetiie-esmmed embedding
Removing keypoint embedding
Figure 14: Ablation study on architecture. We show the keypoints for different architectures.
but lower (better) FID. We use Dembed 128 in our main paper because the increase error small but the decrease of FID is significant.
Ablation Test on r
A too-small value for does not influence the image and will cause artifacts A too-large value for will disable the background embedding and control
shown in Figure 15
the background.
Ablation test on the number of keypoints. By selecting different numbers of keypoints, we can 6 keypoints rather than
achieve different levels of control. In the second row of Figure 15 the default 10. Thereby, keypoints have a smaller area of effect. We observe that the background encoding then takes larger role and contains the encoding of hair and beard, while the keypoints focus only on the main facial features (nose, mouth, and eyes).
Ablation test on the combination of number of keypoints and т. The impact of keypoints de- pends the combination of number of keypoints and r. We test the pairwise combination between K 1, 6, 8, 12, 16, 32 and 0.002, 0.005, 0.01, 0.02. The FID is listed in Table and the detec-
Dimension of embeddings Dembed
Keypoint detection error on MAFL J
FID score on FFHQ
16
4.61 %
28.85
4.92%
26.
64
5.66%
27.64
128
5.85%
23.50
Table 6: Quantitative ablation test on dimension of embeddings. For both metrics, the lower
means better.
24

---

Under review as a conference paper at ICLR 2022
Ablation Test
Ablation Test number of keypoints
K-6
Original
Background
Whole face
Only eyes
Pose
Figure 15: Ablation study on hyperparameters. (Left) Face generation with FFHQ with 7 0.002. We use the red circle to mark the artifacts in the images. (Right) Face generation on FFHQ with number of keypoints 10 (top) and (bottom). More keypoints lead a stronger influence of the keypoint embedding. However, the 6-keypoint version still provides control, e.g., glasses, nose type, and pose. From left to right: original image, replaced background (difference map overlayed), replaced keypoint embeddings (target image overlayed), exchanged eye embeddings, and keypoint position exchanged.
K =11
K =
K = 8
K = 12
K = 16
K = 32
T 0.002
19.35
18.94
17.77
17.69
20.44
19.29
0.005
18.39
18.28
18.42
18.38
20.31
18.72
T 0.01
19.49
19.60
20.31
18.14
19.25
17.91
0.02
19.11
18.80
20.28
19.34
19.58
18.17
Table 7: FID scores of ablation tests on number of keypoints K and keypoint size r on CelebA of resolution 128 128. The lower means better. Neither K significantly influence the image quality. Interestingly, the small artifacts when т 0.002 in Figure [5 does not neither significantly influence the image quality.
tion error is listed in Table The image quality does not change much for different combinations. We illustrate samples of keypoints each combination in Figure aG editing results in Figure and Figure 18 If both the number of keypoints and 7 are small, e.g., K 1, and the = 0.002, then the background controls both foreground appearance and pose, and keypoints are trivial. If both, the number of keypoints and are large, e.g., K 32, and T 0.02, then the keypoint appearance controls background and pose. While the model degenerates in extreme cases, we found the model to be robust for wide range of values, i.c., K 8, 12, 16 and 0.005, 0.01. We summarize all the cases in Table
F.3 ABLATION TESTS ON THE TRAINING STRATEGY
Ablation Test GAN Loss If we replace equation and 5
with the spectral norm Miyato et al. 2018) and hinge loss Miyato 2019) used in the original SPADE architec-
al. [2018] Park ture, we get mostly static, meaningless latent keypoint coordinates. The object part location information entangled with the key part appearance. The comparison is shown in Figure 21
Figure 20: Ablation Test Background loss.
Ablation Test on Background Loss If we remove the background loss in Eq. 6 most keypoints are still reasonable positions while some move the background. As shown Figure 20 the
yellow keypoint is on the background while all the others are still on the foreground.
Removing Keypoint Scheduler. If we move the keypoint scheduler, i.e., updating keypoint gen- erator during resolution adaption, the keypoint locations diverge and the appearance collapses, as shown in Figure 21
25

---

Under review as a conference paper at ICLR 2022
K
K K=
K =
12
K
K
0.002
11.59%
7.86%
6.78%
7.09%
6.00%
5.35%
5 0.005
8.65%
7.28%
6.39%
5.24%
5.13%
4.11%
0.01
8.43%
7.91%
7.97%
6.06%
7.37%
8.84%
T 0.02
8.71%
6.26%
7.13%
5.16%
6.80%
8.26%
Table 8: Normalized Error ablation tests on number of keypoints and keypoint size on CelebA of resolution 128 x 128. For r = 0.002, 0.005, the error decreases as K increases while for r 0.01, 0.02, the error first decreases and then increases. If both of them are large, e.g. K 16, > 0.01, the appearance is entangled with the keypoints which results in larger enror.
K
K
=
6
K = 8
K
12
K
16
K
32
T 0.002
T
T
T
T
F 0.005
T
VV
V
/
VV
= 0.01
T
/ V
E
T 0.02
E
Table 9: Keypoint controllability. T denotes trivial keypoint, i.e., the background controls the en- tire image. E means entangled pose, appearance and background. V v means disentangled control and / means inferior disentanglement, where one of the pairs {(pose, appearance), (pose, back- ground), (appearance, background) entangled.). For a small keypoint size of T = 0.0002 model always gives trivial keypoints. With large number of keypoints and large keypoint size, ie., > and r > 0.01, our model gives entangled representations. Our model is robust in the range of K 16] and E [0.005, 0.01].
G FAILURE CASES AND LIMITATIONS
As described in the main text, model sometimes generates asymmetric faces as shown in the first two images in Figure 22 In addition, the hair sometimes is entangled with the background, especially long hair, shown in the right two images Figure 22
From the ablation tests in training strategy, we can see that this method can heavily depend on the state-of-the-art GAN training loss function and image-to-image translation architectures. In fact, we observed some image quality degeneration as the training goes on in the highest resolution (512x) 512). Therefore, we apply early stopping the highest resolution. We expect that researchers will push GAN and spatially adaptive image-to-inee to-image translation even further. Then people can plug in our keypoint generator to receive higher image quality and more accurate keypoints. In this paper, only provide the idea and focus unsupervised local image editing without using any loss pairs of images.
26

---

Under review as a conference paper at ICLR 2022
K=1
K=6
K=8
K=12
K=16
K=32
200
10,000
7=0.02
Visualization for different combinations of number of keypointe and keypoint size If both, the number of keypoints and the keypoint size are small (top left), the keypoint is trivial. If both of them are large (bottom right), the keypoints distribute uniformly over the images instead of focusing on parts.
Figure 16:
27

---

Under review as a conference paper at ICLR 2022
original
source
all kps
one kp
diff
adding
removing
K=1, 7=0.002
K=1, 7=0.005
K=1, T=0.01
K=1, 7=0.02
K=6, 7=0.002
K=6, 7=0.005
K=6, T=0.01
W
K=6,
7=0.02
Figure 17: Editing on different combinations of number of keypoints K and keypoint size K=1, 6. Column 1: original image; column 2: part appearance source image used to swap appearance; column the combined image with shape from the original images and the appearance from the part appearance source image; column4: we randomly swap single keypoint close to the mouth: column resulting difference map when changing the keypoint in the 4th column; column 6: move the face to the left and add another set of keypoints
the right; column 7: removing keypoints are trivial, and cannot be used to change appearance. When K the keypoint also only have limited control even if 0.02. The combination of K 6, = 0.005 gives good spatial disentanglement.
all keypoints. If T = 0.0002,
28

---

Under review as a conference paper at ICLR 2022
original
source
all kps
one kp
diff
adding removing
K=8, 7=0.002
K=8, T=0.005
K=8, 7=0.01
K=8, T=0.02
K=12, 7=0.002
K=12, 7=0.005
K=12,
T=0.01
K=12,
T=0.02
Figure 18: Editing on different combinations of number of keypoints K and keypoint size 7. K=8,12. For small 0.0002, the keypoint is trivial. When 7 large the background is entangled (K = 8, r 0.02) in some cases. We found the combinations of (K = =2, r = 0.0005)
and (K 12, 0.01) both give the best editing controllability.
29

---

Under review as a conference paper at ICLR 2022
original
source
all kps
one kp
diff
adding removing
K=16, 7=0.002
K-16, 7=0.005
K=16,
T=0.01
K=16, T=0.02
K=32, 7=0.002
K=32, T=0.005
K=32,
7=0.01
K=32,
T=0.02
Figure 19: Editing different combination of number of keypoints K and keypoint size K=16,32. Extreme small (r = 0.0002) constantly gives trivial keypoints even if K is large (K 32). When both, K and r, are large (K T 0.01), the keypoint embeddings control the background and the pose. We found the combinations of (K = 16, T = 0.0005) and (K 32, r 0.005) both give the best editing controllability.
2x resolution
resolution
2x resolution
grod ent ponally logistic loss
spectral nor i hinge 1o55
scheduled
constant
Figure 21: (Left) GAN Loss Importance. Without gradient penality logistic loss. as SPADE, keypoint coordinates remain static. (Right) Scheduling the keypoint generator learning rate. Reducing the learning rate after each progressive up-scaling step prevents mode collapse and enables high-resolution training.
30

---

Under review as a conference paper at ICLR 2022
Asymmetric eyes and face
Entangled hair
Entangled relative positions
(a) Training process
Fallure cases
Figure 22: (a) Training process. We visualize the image generated during the training. (b) Fail- ure cases. The left top two images show asymmetric faces: different eye colors for the man and different blusher for the woman. The middle top two images show the entanglement of hair and background. The right top two images show that the pose of head is hidden the relative positions of other keypoints than the keypoints the head. We visualize the process of removing parts the bottom. We sequentially remove the left eye, right eyc, mouth, nose, and the entire face. Due to the entanglement of hair and background, the hair remains even if we remove the whole face.
31


Reviews:
[{"review_id": "vnakjXi18Ri", "reviewer": "Reviewer_kpkc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "> Authors proposed an interesting idea that can detect keypoints of the object in the unsupervised way and offer the capability of manipulating images based on the detected keypoints.\n\n> Few related works are missing: Authors proposed the interesting idea and implemented it in the effective way. However, I could remember an approach that already provide the interactive capability for GAN by representing its intermediate representation as the segmentation map. I think authors need to add this reference and discuss their difference. Personally, in the sense that the segmentation map is similar; yet more advanced than the 2D heat maps, authors contribution could be significantly reduced by this paper.\n\n[1] Learning Hierarchical Semantic Image Manipulation through Structured Representations, NeurIPS'18.", "summary_of_the_paper": "Authors proposed the LatentKeypointGAN that detects keypoints and provides the editable capability using the keypoints. The overall training is performed in the unsupervised way without the explicit keypoint supervision. In the aspect that GAN provides the capability of generating the images with the photo-realistic quality while the ability to control attributes are limited, the proposed methods contribute to the related society.", "summary_of_the_review": "Overall, I am at the borderline for this paper; however towards more accept side. I could move towards the higher score, if authors could rebut my points effectively.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WT8UnTPkioU", "reviewer": "Reviewer_X57n", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The paper is well-written and easy to follow.\n2. The evaluation is thorough and convincing.\n\nWeakness:\n1. The proposed method is not quite novel. Although the paper emphasized key points, the incorporation of $w_{const}^j$ and the spatial embedding layer (section 3.2) that broadcasts $w^j$ according to a simple positional heatmap implicitly change the key points to a special kind of \"segmentation mask\". In other words, the core method (section 3.1 and section 3.2) is essentially a conversion from key points to a special kind of \"segmentation mask\" that is Gaussian-like and does not have sharp boundaries. This also explains why it can be easily incorporated into SPADE.\n2. As an unsupervised method, the performance of the proposed method depends heavily on two hyperparameters: the choice of the number of key points and their influence range $\\tau$. These are not highlighted in the main paper but are slightly discussed in F.2 of the appendix. I believe the choice of these two hyperparameters interferes with each other and can be difficult to tune, which can make the method less useful. I would suggest adding an in-depth analysis of how these two hyperparameters influence the performance.", "summary_of_the_paper": "This paper proposed a new way to disentangle and control the GAN synthesis process via key points positions, key points appearance and background appearance. Since the key points are learnt in an unsupervised way, the proposed method avoids the labour-intensive labelling process. Extensive experiments verify the effectiveness of the proposed method.", "summary_of_the_review": "As abovementioned, I think the paper is well-written and polished to a high level. However, its contribution is a bit thin and it lacks an in-depth analysis of the two deterministic hyperparameters: the number of key points and their influence range $\\tau$ (the discussion in appendix F.2 is not sufficient). Thus, I feel slightly negative on the submission.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "DIy9PtDpDig", "reviewer": "Reviewer_TTKS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think overall this idea is interesting and straighforward, where controlling the synthesis process via keypoints is pretty useful in practice.\nHowever, I think many related works that provide structural and local edits of GAN synthesis process are missed, making the experiments not very comprehensive.\n \nMajor concerns:\n1. Many related works are not included and discussed:\n\n[a] Kim, H., Choi, Y., Kim, J., Yoo, S., & Uh, Y. (2021). Exploiting Spatial Dimensions of Latent in GAN for Real-Time Image Editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 852-861).\n\n[b] Kwon, G., & Ye, J. C. (2021). Diagonal Attention and Style-based GAN for Content-Style Disentanglement in Image Generation and Translation. ICCV 2021\n\n[c] Edo Collins, Raja Bala, Bob Price, and Sabine Susstrunk. Editing in style: Uncovering the local semantics of gans. In CVPR, 2020\n\n[d]  Yazeed Alharbi and Peter Wonka. Disentangled image generation through structured noise injection. In CVPR, 2020.\n\n[e]  Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region-adaptive normalization. In CVPR, 2020.\n\n[f]  Fangneng Zhan, Hongyuan Zhu, and Shijian Lu. Spatial fusion gan for image synthesis. In CVPR, 2019.\n\n2. I consider latentkeypointGAN as a type of unconditional gan, thus it should not be compared to Pix2PixHD/SPADE/SEAN but recent works that also incorporate structural latent space for unconditional gans, e.g. the ones mentioned in point 1.\n\n3. Another question is that a background loss is adopted to disentangle background semantics from foreground semantics, will the semantics of these k keypoints be entangled with each other? Did you observe more than one keypoints control the same region (i.e. some kind of collapose) during training? If so, how should we resolve this? And if not, why?\n\n4. K is a pre-defined value? What do you mean by adding a keypoint in Figure 1? How should we choose the value of K? There is no ablation study about this.", "summary_of_the_paper": "This paper proposes to use a set of structured keypoints as the intermediate representation for image generation, so that ideally each keypoint with control the location and semantics of a certain part. For each generated image, three noise vectors will be sampled from a gaussian prior distribution, which respectively estimates the locations of k keypoints, the global image style, and the semantics of background. Training is achieved by the standard GAN loss plus a background loss that disentangles background semantics from foreground semantics.", "summary_of_the_review": "I think the idea is good. But related works are not discussed thouroughly. I also have concerns in terms of the baselines and ablation studies in experiments. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C14tirVhP-J", "reviewer": "Reviewer_Jmzu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n+ The paper is well-written and easy to follow. The main claim of allowing to edit images via keypoints is well-supported in the experimental section and in the videos in the supplementary material. \n+ The paper includes many ablations to validate the architectural choices made. Overall, the proposed method appears technically sound.\n+ The paper shows impressive results on LSUN beds which is a difficult dataset that none of the existing unsupervised approaches has tackled so far. \nQuantitative evaluation on celebA-HQ suggests that the method produces similar image fidelity to methods that require more supervision.\n\n### Weaknesses:\n- There are not enough qualitative and quantitative comparisons to the baseline approaches. The method claims that existing works struggle with the disentanglement of pose and appearance, but the experimental section does not validate that their method outperforms existing approaches on that matter. From the results, it seems that the method also struggles with the entanglement of pose and appearance in challenging cases, e.g. the video on LSUN in the supplementary or background and hair on FFHQ. It is unclear if auto-encoder-based methods perform similarly well in simpler cases or if the proposed method really improves the disentanglement.\n- The method might require as much tuning as the baseline approaches. As one key limitation of TPS transformation-based approaches the authors list that they are “notoriously difficult to tune”. But training a GAN also requires a lot of tuning for each dataset and can be very unstable, e.g., the proposed method is highly sensitive to the choice of GAN-objective. The proposed method further introduces two hyperparameters (tau and K) which might also require tuning for each dataset. Hence, it is questionable if the proposed method is really easier to tune than existing approaches.\n- The results in the paper and supplementary seem curated and no uncurated samples are shown. Therefore, it is difficult to predict if the shown results are cherry-picked.\n\n### Ethics statement: \n- Second paragraph: One could still optimize for the latent codes of a real image to manipulate it.\n- Third paragraph: The claim that GANs have a low risk of memorizing training images is not supported well enough. If there is work that clearly indicates that training data cannot be recovered from GANs it needs to be cited here, otherwise the claim does not hold.\n\n### Additional questions / comments:\n- Did you try to sample keypoints from a pretrained keypoint extractor instead of learning them end-to-end? As this might simplify the task it would be interesting to see if it can further improve your results.\n- What exactly makes TPS-based methods so difficult to tune?\n- I suggest adding the FID results for all datasets to the main paper, as well as the FID results for the compared baselines on those datasets so that the reader does not need to rely on a few qualitative results only.\n- Lorenz et al. also evaluate on Human3.6M and PennAction which are very interesting for keypoint-based editing as they consider more complex scenes than e.g. celebA. Evaluating these datasets would substantially strengthen the claim that the proposed method “can be applied to many different objects and image domains”.\n- On the BBC pose videos it looks like the GAN overfits to the appearance codes, because e.g. the head pose changes but keypoints remain at the same position. This is an interesting and reasonable failure case that should be added to the limitations.\n- Appendix B: why do you not always use the tuned LatentKeypointGAN? What are the downsides of this architecture? Please add the FID evaluation similar to Table 1. Or is this the ‘keypoints’ row? Then the naming needs to be more clear.\n- Adding more and especially uncurated samples on all datasets to the appendix, e.g. latent interpolations of single keypoints would be really useful to get a better intuition on how well your approach works. \n- Related work: your approach would be even better motivated if you could state more clearly why it is important to tackle this problem in an *unsupervised* setting. Is it difficult to obtain reliable segmentation masks/landmarks in the settings you consider?\n- Intro: Both works (Li, Xu) that list limitations of auto-encoder-based approaches are only arxiv publications in your references which is not particularly strong as an argument. (Li is actually accepted at ICPR though). It would be more convincing to illustrate the limitations (e.g. entanglement) of the approaches here.\n- FFHQ also has landmarks available. For completeness, it would be good to report the keypoint detection results on FFHQ as well.\n- Table 1: Why is FID without keypoint conditioning better than with?\n- Table 2: Why is FID so much worse with additive global style while L1 error gets better? (I think you mention this in the appendix, but it might be good to add it here already)\n- How useful is the additional loss on the background? You could add a (qualitative) ablation to show that it is needed.\n\n### Misc:\n- Throughout the paper: if you refer to experiments in the appendix, please also put the link to the corresponding section.\n- First paragraph sec 3.1 “MLP to respectively generate the”\n- Last paragraph sec 3.1 “We show in the evaluation ... ” Please add a link to the respective section in experiments.\n- Sec 3.1: Add information that K is a fixed hyperparameter\n- Sec 3.3: Please explain spatial adaptive normalization so that the paper is self-contained\n- Sec 4.1 celebA: please explain MAFL to be self-contained\n- Sec 4.1 celebA: celebA-HQ version missing that is used for sec 4.4, also why do you use celebA-HQ there and not just celebA?\n- Sec 4.2 “appendix document” → “appendix”\n- Sec 4.4 celebA-HQ: what’s the resolution?\n- Table 2: contrastive learned keypoint embeddings are not explained in the method section, not sure what you mean by this\n- Appendix B:  “the convolutional generator is translation invariant” this is not trivially true, see “Alias-free Generative Adversarial Networks (StyleGAN3)” by Karras et al.\n- Figure 9: (top) (bottom) should be (left) (right)", "summary_of_the_paper": "The work proposes a novel method for **locally controllable image synthesis via keypoints**. In contrast to existing auto-encoder frameworks, this work considers a GAN-based method to overcome the following limitations: \n- careful tuning of existing keypoint-based methods\n- better disentanglement of pose and appearance\n\nThe method operates in an **unsupervised** setting, i.e. training with rgb-images only.", "summary_of_the_review": "The paper shows **impressive results** for keypoint-based editing of synthesized images. However, there are **too few qualitative and quantitative comparisons to existing approaches** which makes it difficult to assess the quality of the proposed method in practice. In particular, it is not clear if the proposed method indeed outperforms existing approaches wrt. generalization, i.e. being easier to tune, and disentanglement of pose and appearance. Therefore, in its current state, I lean towards rejecting the paper.\n\n\n**Post-Rebuttal**:\nAfter the rebuttal, I lean towards accepting the paper as image fidelity improves over the baselines. However, my concerns regarding better performance wrt disentanglement and easier tuning than the baselines remain.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an unconditional GAN that learns a set of structured keypoints as the intermediate representation. It was shown that these learned keypoints may be used to control the image synthesis output. The paper received a mixed rating before the rebuttal, with one reviewer rating the paper marginally above the bar and three reviewers rating it marginally below the bar. While a couple of reviewers commented that the keypoint idea was interesting, several concerns were raised, including the seemingly challenging tuning requirement and the usability of the proposed method. Several missing related works were also pointed out. The rebuttal addressed some of the raised concerns but not fully. While Reviewer Jmzu raised the score from marginally below the bar to marginally above the bar, Jmzu still expressed concerns about the quality of the paper. Reviewer kpkc kept marginally above the bar rating but was not impressed with the contribution. Consolidating the reviews and rebuttals, the meta-reviewer found the raised concerns valid and would not recommend acceptance of the paper. The authors are encouraged to incorporate the feedback to strengthen the contribution.
Decision: Reject