PIVQGAN: Posture and Identity Disentangled Image-to-Image Translation via Vector Quantization
Authors: Bingchen Liu
Yizhe Zhu
Xiao Yang
Ahmed Elgammal
Venue: ICLR 2022 Submitted (2022)
Keywords: deep learning
generative model
image synthesis
generative adversarial network
self-supervised learning
image-to-image translation

Abstract: One popular objective for the image-to-image translation task is to independently control the coarse-level object arrangements (posture) and the fine-grained level styling (identity) of the generated image from two exemplar sources. To approach this objective, we propose PIVQGAN with two novel techniques in the framework of StyleGAN2. First, we propose a Vector-Quantized Spatial Normalization (VQSN) module for the generator for better pose-identity disentanglement. The VQSN module automatically learns to encode the shaping and composition information from the commonly shared objects inside the training-set images. Second, we design a joint-training scheme with self-supervision methods for the GAN-Inversion encoder and the generator. Specifically, we let the encoder and generator reconstruct images from two differently augmented variants of the original ones, one defining the pose and the other for identity. The VQSN module facilitates a more delicate separation of posture and identity, while the training scheme ensures the VQSN module learns the pose-related representations. Comprehensive experiments conducted on various datasets show better synthesis image quality and disentangling scores of our model. Moreover, we present model applications beyond posture-identity disentangling, thanks to the latent-space reducing feature of the leveraged VQSN module. 

Under review as a conference paper at ICLR 2022
PIVQGAN: POSTURE AND IDENTITY DISENTANGLED IMAGE-TO-IMAGE TRANSLATION VIA VECTOR QUANTIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
One popular objective for the image-to-imane -ommage translation task is independently control the coarse-level object arrangements (posture) and the fine-grained level styling (identity) of the generated image from two exemplar sources. To approach this objective, we propose PIVQGAN with two novel techniques in the frame- work of StyleGAN2. First, we propose Vector-Quantian (VQSN) module for the generator for better pose- unse-identity disentanglement. The VQSN module automatically learns to encode the shaping and composition infor- mation from the commonly shared objects inside the training-set images. Second, we design a joint-training scheme with self-supervision methods for the GAN- Inversion encoder and the generator. Specifically, we let the encoder and generator reconstruct images from two differently augmented variants of the original ones, defining the pose and the other for identity. The VQSN module facilitates a more delicate separation posture and identity, while the training scheme en- sures the VQSN module learns the pose-related representations. Comprehensive experiments conducted on various datasets show better synthesis image quality and disentangling scores of our model. Moreover, present model applications beyond posture-identity disentangling, thanks to the latent-space reducing feature of the leveraged VQSN module.
Spatial Normalization
Seif leamned object mask
dentty
Figure I: Unsupervised image-to-image translation results of PIVQGAN with disentangled posture and identity control. In each panel, the first row has input pose images, and the first column has referential identity images. The second and third rows are "segmentation- like"" masks automatically learned by PIVQGAN, and bottom-right are the synthesized images.
INTRODUCTION
We study sub-task of image-to-ing -o-iiaae translation, which we call posture-and aiiete--nnddddettty disentangled translation. We view an image as a combination of multiple objects with varied scales, then define posture as the shape and composition of the larger-scale objects, and identity as both the color/texture

---

Under review as a conference paper at ICLR 2022
of the objects and the shape of the finer-scale objects. Given two input images, one defines posture, and the other defines identity. Our goal is to develop a model that transfers the first image's 's stance to the second one's identity. This task has excellent potential in real-life applications such illustration
synthesis and motion driving for content-creation jobs. Note that a close task has been widely
studied as content-style disentangled image synthesis, e-g. Choi et al. (2020); [won & Ye (2021);
Lee et al. ) Liu et al. (2021; 2020a); Kotovenko et al. (2019); Kazemi et al. (2019). We find
the terminology of "content-style can be ambiguous sometimes, as both posture and identity has
content and style related properties. In contrast, posture-identing are relatively precise to the task
and datasets that we are working on, including human, animal, and animation faces.
Training an image-to-imane translation model usually requires image-level or set-level supervision, as represented by StarGAN-v2: Isola et al. (017)) Wang et al. (2018); Park et al. (2019): Zhu et al. (017): Li et al. (2018): Lee et al. (2018); 8): Liu et al. (2019); [ Choi et al. [l.[(220 (2020). However, it is ex- pensive to obtain paired data and extra labeling information real-world scenarios. More recently, fully unsupervised methods gain more research interests such as CLUIT: Back et al. (202I); Kwon & Ye (02T); Lee et al. (202T), where no labeling information but only a set of images are needed. However, these methods lack controllability
disentangled image attributes, and lack robustness on the degree of disentanglement among datasets. For our task of pose-identity translation, the limi- tations from prior works motivate us seek fully unsupervised method that is also controllable and robust. It is hard to train a model with precise control over posture and identity when there are no guidance labels of an image such as a segmentation or skeleton map. which are used by Nirkin al. (2018); Lee et al. (2020); Kocabas et al. (2020); Jin et al. (2020). Nonetheless, the need for such segmentation label inspires us to develop module that can learn segmentation-like representa- tion from the unlabeled image data in eelf-superviee manner, therefore boosting the translation performance, as shown in Fig. [
Our key idea is to let the image generator automatically learn a semantic mask of the posture im- age. Different regions in the mask represent a semantic object in the image. The synthesis then based on the objects' shape and composition in the mask. To facilitate the idea, we devise a Vector-Quantized Spatial Normalization layer (VQSN) that combines sector-uuantization (Oord iet al. (017) and apatially-adapiic
normalization (Park et al. 2019). Specifically, we first leverage the VQ mechanism to reduce a feature-map into a certain number of object embeddings. Each em- bedding corresponds to common object shared among images of the same domain (e.g. eye, mouth, and nose for face dataset; wall, window, and roof for building dataset). We then get spatial object mask that we refer when performing an affine transformation on the feature-map. Each spatial location is transformed differently according to its corresponding object.
We show that, with carefully designed self-supervision training scheme, VQSN can automatically learn meaningful object embeddings that provide useful posture information. The resulting model, which we call PIVQGAN, enables a precise control on posture and identity of the image generation, and outperforms existing baselines in terms of visual quality and translation accuracy. Besides, we show a robust performance of PIVQGAN even on unseen oft-of-domain images, where we further demonstrate its promising application beyond pose-identity translation, including zero-shot image style-transfer and semantic-aware image-editing.
2
RELATED WORK
2.1
UNSUPERVISED IMAGE TRANSLATION
Well established image-to to image translation models mostly require labeled data to be trained, e.g.
Isola et al. (2017) y Zhu et al. (2017); 7 Liu et al. (2017): Kim et al. (2017); Huang et al. (2018): Kim
et al. (2019) 9 : Liu et al. (2019): Choi al. (2020). Instance level paired images are required by
models with autoencoder training scheme like employed by Isola al. (2017): Park ct al. (2019);
Liu et al. (002)) D. and coarse domain labels are needed for uni- or multi-mode translation models
trained via sycle-consistency scheme as in Zhu al. (2017); HHang et al. (2018): Choi et al. 2018).
However, the need for labels often becomes a bottleneck in real-world applications, limiting both the model performance and the application scenarios. Therefore, fully unsupervised image translation has drawn more research attention recently. One kind of unsupervised methods acquire pseudo- labels by image clustering (Baek et al. 2021 Bahng al. 2020), another kind directly generates
2

---

Under review as a conference paper at ICLR 2022
paired images via style- tryee-transee from small portion of labeled data (Liu et al. 2021). These meth- ods lack robustness, as unintended translation results may occur when clustering algorithms fail to produce consistent clusters, or the synthesized pair data has low quality.
Focused on posture-and ann-iidetity image translation, methods without label-supervision are developed based on reliable prior provided by StyleGAN (Karras et al. 2019). [won & Ye (2021) proposed diagonal attention structure which aims to encode the object shaping information. Lee et al. (2021)
developed a contrastive self-supervised training scheme, where model is trained implicitly learn and transfer visual features without explicit domain separation. However, the disentangled latent features from these methods are not well interpretable and controllable, thus lacks practicality real
world scenarios. Our work combines the strength from both perspectives. We propose structural design that retains a more explicit object shaping representation, and leverages self-supervision to enforce the learning without labeling data.
2.2 DISENTANGLED IMAGE SYNTHESIS
A broader task of disentangled image synthesis usually aims to learn vector representation for the images in an image domain, where each vector entry controls one mutually independent visual attribute. antoencoder-based methods, e.g. [Higgins et al.] (2017); Burgess et al. (2018): [Kim & Mnih 20188;; [Emaeili et al.] (2018), approach disentanglement via the total-correlation objective, while GAN based models, e.e. Chen al. (2016); Lin al. (2020); [iu al. (2020b), rely on maximizing nntual-information loss. However, the vector representation has a limited capacity, and the per-attribute disentanglement still hard achieve. In this paper, we study the coarse- level disentanglement between two sets of attributes. which can be seen as the very first step in hierarchically srrechically-diiettangglle task as proposed by |Esmaeili al. (2188;; Singh et al. (2019).
The coarse-level level disentanglement is usually approached by the inductive biases of the layer designs and model structures. For example, Karras al. (2019); Karnewar Karreea Wang (2020); :020);[Liur et al. (202cc) showed that controlling the attributes of a generated image from multiple scales in the generator is effective. Park et al. (2019) developed SPADE which applies different affine transformations to different spatial regions of the feature-mat map, highlight different object shapes and diversify the object appearances. Kwon & Ye (2021) proposed to learn "content" branch beside the original "style" branch in StyleGAN, and uses a one-channel spatial mask to guide the content information. Kim et al. (2021) further replaced the vector representation the "style" branch with spatial masks embed better shaping information to the generated images.
We also bet on the structural bias to gain better control over the posture generation. However, unlike previous methods, our model learns the posture information in interpretable, transferable, and editable way while disentangled well from the identity attributes.
3 METHODS
In this section, we first describe PIVQGAN's synthesis flow. Then we introduce Vector-Quantizae Spatial Normalization (VQSN). Finally, we talk about the self-supervised training scheme and all the objective functions.
3.1 MODEL OVERVIEW
PIVOGAN consists of three models, the discriminator D and the generator G (modified with VQSN layers) from StyleGAN2, and an encoder E that converts images to vectors in G's latent space.
In each training iteration, D and G are first trained as GAN (Goodfellow et al. 2014). then G and E are trained for GAN inversion tasks (as in Xia et al. 2022)) like Autoencoder. A simi- lar joint-training schema without dedicated self-supervision also explored by Kim et al. (2021) in StyleMapGAN Such joint-training is required in supervisions as described in Sec 3.3.
to implement the proposed self-
We separate the generation of posture and identity attributes different convolutional layers of G. Specifically, given that G synthesizes an image at the resolution of 256 x 256, base feature-map of size 4 x 4 is up-scaled six times by 14 conv-layers, each conv-layer modulated by corresponding
3

---

Under review as a conference paper at ICLR 2022
VQSN
Find the nearest neighbor
Putit to corresponding
P
location
1X1
vg
muttiply
Image fcature-map
bota
512X16x116
Modulated Upsample
vo incox mask 180018
1XI conv
edd
512X4X4
Base
Hasted embeddings
(2)
512X16X166
embeddingg
512x16x166
genner elted latents
Generated image
predicted Intents
posture attributes
Real Image
controis the identity attributes
Come
prodicted latents
mixed latents
Reconstructed
mmage
3.
Image
Real
predicted latantss
Reconstructed mmge
Augmented image
G Cane
2.
G
MSE Loss LPIPS Loss
MSE Loss LPIPS Loss
Figure 2: Top: Illustration of the VQSN module. (1) base feature-map is quantized by a certain number (e.g. 2 or 6) of trainable embeddings. (2) The VQ embeddings are used to perform the spatial-wise affine transformation on image feature-maps, Bottom: Training scheme of PIVQGAN. Apart from the regular GAN training (omitted), we train encoder together with the convolution part the generator with self-supervision tasks. In (3), we latents-grafted reconstruction to help the model disentangle pose/identity attributes at different layers.
latent vector. We take 2 to 3 layers between 2nd to 7th layers (varied datasets) as the pose-layer. The pose-layers are forced to learn only posture-related information, which has to be independent of the other layers. Mcanwhile, given image, E is trained to generate the 14 latent vectors that can let G reconstruct the image. The goal of this setting is to let the latent vectors for pose-layers encode only the pose information, while the rest carry other attributes.
In inference time, E gets two latent sets 11:14:: (f:11] from two images i, 12. And we graft the latent sets by swapping the pose-layers' vectors to transfer the posture or identity features between the two images. For example, suppose layer 4 to are pose-layers, the grafted latent sets [/1 (a 4744] will let generate image with the identity features i,, while having the posture of i2. Moreover, we can also randomly sample / from the latent space of G, for more diverse translations.
3.2 VECTOR- VECTOR-QUANTIZER SPATIAL NORMALIZATION
With the hope of letting the selected pose-layers learn only pose-related features while not influ- encing the rest layers to maintain high synthesis quality, we propose the VQSN layer. VQSN is plug-in module that works on the output feature-map of common convolutional layers.
The VQSN module takes two inputs: a vector defines the posture attributes, and an image feature-map f cx e.g. c, h, w (512, 16, 16). And it outputs a revised feature-map f' in the same shape. There are two parts inside the VQSN module. First, a pose generator has trainable basis in the shape of 512 x 4 x 4 and three modulated convolutional layers (like tiny StyleGAN2 generator). This pose generator uses 1P to up-sample the basis into base feature-map S' E Rexbxx Second, shown in Fig. l fo is quantized into Feg according to a certain number of trainable embedding vectors. Finally, fog converted into alpha and beta via 1 x convolutional layers, and performs affine transformation f to get the output f""
5512 (from StyleGAN's latent space) that
Two intuitions guide our design of VQSN. First, we hope to compress the representation capability of the pose-layers so that it has no room to carry unwanted shape-invariant information. Vector- quantization fits well in this case, for it reduces the continuous representation space into few discrete embeddings. The VQ module categorizes each spatial location into one object. And the cat- egorization process naturally defines the shape of each object and the composition of all the objects.
4

---

Under review as a conference paper at ICLR 2022
Second, to highlight shape-variane information, we want treat each spatial location differently in a feature-map (c.g. the eye area has a value distribution that different from the mouth area). We apply unique difine-transformation at different pixel locations according to the quantized object mask to address such a difference.
3.3 SELF-SUPERVISED TRAINING SCHEME
The encoder E aims projecting an image into the latent space of G. where G can faithfully reconstruct the input image. There are two main training stages when we jointly train E and G. First, we sample latent vectors let gxx14x512 from the mapping network (Karras al. 2019) in G with random noise as input. Here we train the decoder network in G and E to reconstruct the sampled latent vectors:
Liatent = ((G(g)) - [g1ll2]
(1)
Llatent ensures E and G' to be well communicated, thus helps E always produce in-domain latent vectors in G's latent space. Second, we sample real images éreat and train and to reconstruct them with LPIPS loss (Zhang al. 2018):
Leee
E ireal - ((E(treat)) + LPIPS((real G(E((reat)))
Importantly, while the VQSN module captures large-scale posture features well (e.g. head direc- tion), it can hardly capture smaller-scale pose information, such as mouth and eyes openness, by itself. To help VQSN capture those detailed postures, we apply a series of augmentations to the real images and train E and G on the augmented image reconstruction task. As shown in Fig. 4 use shape-oriented augmentations, including random cutout, flipping, and re-scaling, to create pose and identity image pairs Borg, l. Then, as shown in Fig. b-bottom-(3), we train E and G on reconstruction tasks from the grafted latents (again suppose layer 4 to 6 are pose layers):
(org [1:14] E(iorg); youg = Eliang) [1:14]
Lang LPIPS(G((PT)) "1:6). (1:6)) (fria)) (ang) + LPIPS( [1:6]= (7 Rorg)
Laug makes sure that only the selected pose-layers encode these augmentation-rekutn shape infor- mation while the rest do not carry them. With this training scheme, VQSN automatically learns the proper semantics of the commonly shared objects between training images.
EXPERIMENTS
In this section, we describe the evaluation metrics and experiment results. We first compare PIVQ- GAN to the state-of-thc-an methods, then present additional analysis and demonstrate applications of it for further insights. In each experiment, we train all the models 256 256 resolution. Our model is trained five times each setting, and the median score is reported.
Baselines. We compare PIVQGAN with both latest and some earlier baseline models that perform examplar-basse -aase image-to-image translation. Including DATGAN by Kwon & Ye (202T), CLUIT by
Lee al. (202T). SNI by Alharbi & Wonka (2020), StarGAN-v2 by Choi al. 2020), MUNIT by Huang et al. (2018), and TUNIT by Back et al. (2021). For models that are not open-source, we use the scores reported from the authors.
Datasets. We show qualitative results of PIVQGAN on five datasets covering a wide range of image domains, including CelebA Liu et al. (2013), AFHQ Choi al. (2020), LSUN car and church Yu et al. (2016), and Anime-Girt Li et al. (2021). And we conduct quantitative comparisons with other models on CelebA and AFHQ datasets. We do not use any labeling data; only images are needed to train our model.
Metrics. For quantitative comparisons, we use Frechét inception distance (FII) Heusel et al. 2017 and learned perceptual image patch similarity (LPIPS, (.PIPS,Zhang et al. 2018) to evaluate the synthesized image quality and diversity. Perceptual path length (PPL, Karras et al. 2019) is used to measure the smoothness and linearity of the latent space of G. We also include a simple new metric called mix- back LPIPS. Given two images ia, ig. we exchange their pose by the models to get Lab. tha (lab has the identity of in and pose of is), then identity are perfectly disentangled, and the image synthesis quality is high, iaba should look exactly the same as i, so does ibab and ig. For all these metrics, lower score means better performance.
exchange their pose again get iaba, 4bab- If pose and
5

---

Under review as a conference paper at ICLR 2022
4.1
COMPARISON BASELINES
AFHQ
FID
m-LPIPS
r-LPIPS
PPL
AFHQ
FID
m-LPIPS
r-LPIPS
MUNIT
68.32
0.842
0.512
VQ-2
16.18
0.493
0.401
TUNIT
48.92
0.721
0.491
VQ-3
10.98
0.512
0.358
SNI
16.02
0.481
0.385
65.22
StarGAN-v2
19.93
0.671
0.436
97.83
w/o augment
12.32
0.532
0.366
DATGAN
16.09
0.472
0.378
63.44
w/o VQ
12.93
0.518
0,362
12.54
0.463
0,373
58.63
VQ-1
12.54
0.463
0.373
CelebA
CelebA
MUNIT
85.74
0.856
0.537
VQ-2
12.65
0.447
0.331
TUNIT
36.88
0.766
0,451
VQ-3
10.82
0.451
0.341
SNI
17.45
0.461
0.382
57.63
w/o augment
12.72
0.522
0.346
StarGAN-v2
22.35
0.671
0.405
72.26
DATGAN
18.11
VQ
13.53
0.538
0.352
12.65
0.447
0.331
51.32
VQ-1
12.46
0.449
0.339
Table 1: Comparison baselines.
Table 2: Ablation study.
StarGAN-v2 and CLUIT have the best visual quality among the baseline models, so we only show quality comparison with these two models on the AFHQ dataset in Fig. 3 We consider AFHQ data harder than CelebA because it has more prominent identity feature variance with different animal breeds. PIVOGAN outperforms its competitors in both translation accuracy and synthesis quality. Our model produces sharper image with more precise details and correct coloring in overall image quality. In terms of translation accuracy, PIVOGAN overcomes some common defects shared by prior methods. Specifically, models before CLUIT can hardly maintain good identity consistency. StarGAN-v2, TUNIT, and MUNIT even generate animals with the wrong species. CLUIT fails to translate the detailed posture attributes, including eye and mouth open degree and ear fold degree. In contrast, PIVOGAN not only translates these postures well but also captures the fine-grained identity features better, such as the texture leopard, the ear color of fox, and the eye color of the cats, where other models fail to capture.
To compute FID, we images from the testing set as pose and identity reference images, randomly match- ing them get 10000 input pairs and generate 10000 testing samples. These samples then used com- pute FID with all the training set images. The same input pairs also used to calculate the mix-back LPIPS. We also use LPIPS mea- sure the models' reconstruction per- formance, where we the same testing image as both pose and iden- tity reference, and compute the mean LPIPS of 10000 samples. To calcu- late PPI., we randomly sample two images from the testing set as pair and get latent vectors from them, then interpolate the two latent sets to compute the score. We also use 10000 sample pairs and report their mean value.
Poeture
Posture
CLUIT
Wrong
Low-contrast texture/pattem
Wrong eye openness
Whong operness tongue
Figure 3: Qualitative comparison of PIVQGAN and other
baselines on AFHQ dataset.
Table. shows the metrics comparison results. For both datasets, model stands out with almost 25% improvement on FID over the previous state-of-the-arts, indicating a better image quality and more diverse coverage identity and posture features. The better mix-back LPIPS shows that our model separates and preserves the pose and identity features well.
4.2 ANALYSIS OF MODEL COMPONENTS
VQSN vs Gumbel-softmax. To learn segmentation-lis like representation that captures the object shape and composition feature, Gumbel-softmax (GS, Jang et al.) is also a viable approach besides (GS,Jang
employed Vector-Quantization. We also implemented the GS version of spatial transfor-

---

Under review as a conference paper at ICLR 2022
Posture
Identity
Posture
Idonlity
Random cutout identity image
Pose image
image
Self-learmed
object mask
(a)
Flip, re-scaled identity image
Pose Image
Recon image
Self-learned
object mask
(b)
(c)
Figure 4: Left: Object-mass editing. Right: Various augmentation methods used on the model's
identity inputs, enabling the model to learn the augment- variant attributes from only posture inputs.
mation module. We use the differentiable soft-max trick in GS to classify each pixel location into one of the object labels. Unfortunately, with GS, the model is unable learn interpretable semantic masks as VQSN does. Moreover, the object mask from GS looks like random noise and leads to worse image synthesis quality of the model. Nonetheless, we think GS is worth more study; better parameter tuning and structure design may lead to viable performance.
VQSN structure design. explore multiple structural settings of our VQSN module to have better understanding of its performance. Among various trials, present three notable settings:
1) VQ-1: the basic design, where the object-mask is got from a base feature-mas map generated from the input latent vector and modulated conv-layers.
2) VQ-2: the base feature-map is generated from the previous object mask's VQ embeddings with modulated conv-layer and the current latent vector.
3) VQ-3: the VQ-embeddings will first concatenate to the image feature-map, then generate the affine transformation parameters.
The intuition for VQ-2 is to get hicrarchically coarse-to-fine object masks over multiple pose-layers. So the object-mask generated based the previous layer's output mask. For example, the first VQ layer learns the large-scale object information such as head region and background region; then, the second VQ layer can learn eye, mouth, and nose shapes inside the head region. For VQ-3, aim to smooth the affine transformation on object contour areas because the object mask has a sharp difference object boundary. Therefore, concatenate the spatially discrete VQ embeddings with the continuous image feature-map before using the VQ embeddings to revise the image feature-map.
Interestingly, VQ-2 does not do what we expected. Instead, it leads to worse visual semantics on object masks and worse performance across the metric. We hypothesize this depends on the image visual properties of the datasets. For example, as shown in Fig. right and Fig. Heeft, on AFHQ, the VQ layers first learn head shape then smaller objects. In reverse order, on CelebA, the model first learns eye, mouth, and nose objects, then captures head and hair in the next layer. We do not extend more on it in this paper but will further study the reason behind it.
Meanwhile, VQ-3 improves the image quality as expected. However, the pose translation perfor- mance slightly downgrades. The concatenated image fcature-map may introduce noise the VQ module and weaken its ability to learn object shaping features. The quantitative results on AFHQ are presented in Sec. 4.1 While VQ-3 has better FID and reconstruction LPIPS, it performs marginally worse than VQ-1 in disentangling and preserving the pose and identity features.
Shape-related image augmentation. Self-supervised reconstruction task with image augmentation (Fig. right) is the major reason that PIVQGAN captures the small-scale object semantics at VQSN layers. PIVQGAN can hardly capture the mouth and eye pose without this training scheme, which

---

Under review as a conference paper at ICLR 2022
Identity image
Pose image
Identity image
Identity image
Figure 5: Left: VQSN is controlled by the latent vectors, thus it has a smooth posture transaction when interpolating between two pose references. Right: When fixing the pose-latents and interpo- lating only the identity-latents, PIVQGAN shows a smooth transaction on identity-related attributes.
failed by other baseline models. Both the VQ module and the augmented training bring better translation performance, as be found according to the mix-back LPIPS score in Table. 2.
4.3 QUALITATIVE EVALUATION AND APPLICATIONS
Latent space smoothness. In the VQSN module, we still let vector control the synthesis of the object-mask. The design is to get a smooth latent space of the object-mask. Thus we can generate continuous transactions between different postures. 5 shows the interpolation between two posture vector sets. In each panel, the left-most image defines the identity and the initial posture, and the right-most image defines the target posture. Both inputs are real images from the testing set. We can see that both the generated images and the generated object-maske have smooth transaction. On the third panel, we show one problem of PIVQGAN that remains be solved. On CelebA, our model learns the hairstyle more of a shape-related property than shape-invariant one like color and texture. Such a problem is hard to address by neither the VQSN module nor the augmentation training scheme.
posture attribute rather than an identity attribute because the hairstyle
Fig. riight further shows the smoothness of the identity latent space PIVQGAN and also reflects the excellent disentanglement between posture and identity features of our model. The left-most image defines the pose and the initial identity in each row, while the right-most one defines the tar- get identity. Both inputs images from the testing set. All the synthesized images remain of high quality while maintains the original posture well. On CelebA, PIVQGAN generates expressive hu- man faces among the interpolations. On AFHQ, the interpolated images present the brendiblending visuals between the species. It shows the potential of our model for creative material synthesis.
Object-mask editing. After training, the encoder F combined with the VQSN layer in G becomes segmentation network. Given an image, the combined model generates object masks that highlight each component of the image. Of course, the produced masks are far less accurate than actual segmentation maps. However, these object-mask still get the essential semantics of the input image. And by editing the masks, we can create novel images from that follow our revisions.
Fig. left presents some editing examples with testing images as referential inputs. In Fig. 4 left- (a), we find that the red region represents the eyes, so if we enlarge the red parts, the eyes are also getting bigger in the generated images. Similarly, the green parts correlate well with the nose in the images, so we change the size of the green part, we achieve some funny pictures of tigers and cats having big noses. One last example is that the yellow part highlights the mouth area, if gradually remove the yellow part, we can see that the generated animals have their mouths closed.
Robust on out-domain images. We consider the robustness to unseen out-domain images to be an essential feature of PIVOGAN. The model performance is inevitably downgraded in prior works when the testing images have unseen texture, color, objects. However, high image synthesis quality as long as the out-domain images are not too far away from the training image domain. Two factors contribute such robustness. Firstly, the joint training scheme between
model can maintain its
8

---

Under review as a conference paper at ICLR 2022
(b)
(c)
(d)
Out-domain images with identity features the model never seen
PIVOGAN trained only on: (a.c) Anime-Giris, (b.d) AFHQ
In-domain images generated by the model from out-dommin images
Figure 6: PIVQGAN is robust to out-domain images with unseen textures, and can generate mean- ingful in-domain counterparts from these out-domain inputs. (a,b) Out-domain images as identity input. (c,d) Out-domain images as posture input.
GAN and the GAN-inversion encoder guarantees that the encoded latent vectors always lay in the latent space. Therefore, even for images with unseen visual attributes, E still projects the image into the closest vector inside the latent space. Secondly, the VQ module will always reduce the shape feature-map into a certain number of learned object embeddings. It means the produced object-mask always understandable G, even for input images with unseen objects.
With the factors mentioned above, we show the synthesized images of PIVOGAN with unseen image domains in Fig. [6 We train our model on the Anime-Girl dataset in (a) and (c), which only contains animated female portraits. When we input photo-realistic face as the identity image, the model automatically generates an "anime-girf girl" version of the input image. We can see that since the model has formed a solid prior bias on animated young girls, it translates all the faces into a young girl regardless of the input image's 's gender and age. On the other hand, translates the other facial attributes correctly, including hair color, skin tone, and posture, because all these attributes are already learned from the training dataset. When we input real human faces as the posture reference, the model transfers the pose without any problem. Similarly, in (b) and (d), the model is trained only on photo-realistic animal images while we fed toy animal pictures as pose or identity inputs. Again, the toy's identity is well translated into counterpart of the natural animal face, and our model captures the posture perfectly. In contrast, all the baseline models suffer from generating good quality results given the same training and input data. The out-domain performance reveals promising performance of PIVQGAN on a zero-shot image domain translation task, which has not gained much research attention before our knowledge.
CONCLUSION
We proposed the VQSN module and the self-supervised joint training scheme. The resulted model, PIVOGAN, tackles the problem of unsupervised fetture-andddeentiyy translation and improves the state-of-the-art performance on this task. The experimental results showed that our model outper- forms prior baselines in both image quality and translation accuracy.
Beyond posture-ane and identity translation, the proposed VQSN module also unveils the immense po- tential of fully unsupervised image segmentation and semantic understanding. Moreover, the robust performance of PIVQGAN on unseen-domain images gives hints to a new task of zero-shot image domain translation. We hope our model paves the way for future studies in this new direction.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Yazeed Alharbi and Peter Wonka. Disentangled image generation through structured noise injection. In CVPR, pp. 5134-5142, 2020.
Kyungjune Back, Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Hyunjung Shim. Rethinking the truly unsupervised image-to-image translation. In ICCV, 2021.
Hyojin Bahng, Sunghyo Chung, Seungjoo Yoo, and Jaegul Choo. Exploring unlabeled faces for novel attribute discovery. CVPR, pp. 5821-582111 2020.
Christopher P Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in f-vae. arXiv preprint arXiv:1804. 03599, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS, pp. 2172-2180, 2016.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In CVPR, pp. 8789-8797, 2018.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wop Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, pp. 8188-8197, 2020.
Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, Siddharth, Brooks Paige, Dana Brooks, Jennifer Dy, and Jan An--illemm van de Meent. Structured disentangled representations. star, 1050: 12, 2018
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. NIPS, 30, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basie visual concepts with constrained variational framework. In ICLR, 2017.
Xun Huang, Ming-Ye Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In ECCV, pp. 172-189, 2018.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, pp. 1125-1134, 2017.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR, 2017.
Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole-body human pose estimation in the wild. In ECCV, pp. 196-214. Springer, 2020.
Animesh Karnewar and Oliver Wang. Msg-gan: Multi-scale gradients for generative adversarial networks. In CVPR, pp. 7799-7808, 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pp. 4401-4410, 2019.
Hadi Kazemi, Seyed Mchdi Iranmanesh, and Nasser Nasrabadi. Style and content disentanglement in generative adversarial networks. In WACV, pp. 848-856. IEEE, 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint arXiv: arXiv:1802.05983 2018.
10

---

Under review as a conference paper at ICLR 2022
Hyunsu Kim, Yunjey Choi, Junho Kim, Sungjoo Yoo, and Youngjung Uh. Exploiting spatial dimen- sions of latent in gan for real-time image editing. In CVPR, pp. 852 8618 2021.
Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwanghee Lee. U-gat-it: unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation. arXiv preprint arXiv: 1907. 2019.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In ICML, pp. 1857-1865. PMLR, 2017.
Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. Vibe: Video inference for human body pose and shape estimation. In CVPR, pp. 5253-5263, 2020.
Dmytro Kotovenko, Artsiom Sanakoyeu, Sabine Lang. and Bjorn Ommer. Content and style disen- tanglement for artistic style transfer. In ICCV, pp. 1422-4433 2019.
Gihyun Kwon and Jong Chul Ye. Diagonal attention and style-based gan for content-style disentan- glement in image generation and translation. arXiv preprint 16Xiv:2103.16146, 2021.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In CVPR, 2020.
Hanbit Lee, Jinseok Seol, and Sang goo Lee. Contrastive learning for unsupervised image-to-image translation. arXiv, 2021.
Hsin- Hsin-Ying Lee, Hung- Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming- Yang. Diverse image-to-image translation disentangled representations. In ECCV, pp. 35-51, 2018.
Bing Li, Yuanlue Zhu, Yitong Wang, Chia- Wen Lin, Bernard Ghanem, and Linlin Shen. Anigan: Style-guided generative adversarial networks for unsupervised anime face generation, 2021.
Minjun Li, Haozhi Huang, Lin Ma, Wei Liu, Tong Zhang, and Yugang Jiang. Unsupervised image- to-image translation with stacked cycle-consistent adversarial networks. In ECCV, pp. 184-199, 2018.
Zinan Lin, Kiran Thekumparampil, Giulia Fanti, and Sewoong Oh. Infogan-cr and modelcentrality: Self-supervised model training and selection for disentangling gans. In ICML, pp. 6127-6139. PMLR, 2020.
Bingchen Liu, Kunpeng Song, Yizhe Zhu, and Ahmed Elgammal. Sketch-to-art: Synthesizing stylized art images from sketches. In ACCV, November 2020a.
Bingchen Liu, Yizhe Zhu, Zuohui Fu, Gerard De Melo, and Ahmed Elgammal. Oogan: Disen- tangling gan with one-hot sampling and orthogonal regularization. In AAAI, volume 34, pp. 4836-4843, 2020b.
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized gan training for high-fidelity few-shot image synthesis. In International Conference on Learning Representations, 2020c.
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Self-supervised sketch-to-image synthesis. In AAAI, volume 35-3, pp. 2073-2081, 2021.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Advances in neural information processing systems, pp. 700-708, 2017.
Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to- toggge translation. In ICCV, pp. (6551-10500, 2019.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, December 2015.
11

---

Under review as a conference paper at ICLR 2022
Yuval Nirkin, lacopo Masi, Anh Tran Tuan, Tal Hassner, and Gerard Medioni. On face segmentation, face swapping. and face perception. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 98-105. IEEE, 2018.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. arXiv preprint arXiv:/711..00 1.00977, 2017.
Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatiall--aapttive normalization. In CVPR, pp. 2337-2343 3337,2346, 2019.
Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. Finegan: Unsupervised hierarchical disen- tanglement for fine-grained object generation and discovery. In CVPR, June 2019.
Ting-Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In CVPR, pp. 8798- 8807, 2018.
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: A survey, arXiv preprint arXiv:2101.0522 2021.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction large-scale image dataset using deep learning with humans the loop, 2016.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features perceptual metric. In CVPR, 2018
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, pp. 2223-2232, 2017.
12


Reviews:
[{"review_id": "yzOi1o205DK", "reviewer": "Reviewer_SuTz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The paper addresses an important task that enables many creative applications such as retargeting of videos without any explicit tracking\n- The paper presents an unsupervised approach which makes it applicable to a much wider range of objects compared to supervised approaches which have to rely either on shape or identity annotations (although the latter is relatively easy to obtain from videos).\n- The presented approach is relatively simple but produces very good results compared to baselines as demonstrated by both qualitative and quantitative comparisons.\n- The model yields an interpretable shape representation in the form of a spatial mask that can be edited manually. This enables additional applications such as manual editing of these masks, which is also demonstrated in the experiments. It might also offer opportunities for  unsupervised segmentation of parts.\n\nWeaknesses\n- The related work discussion completely misses works on unsupervised keypoint learning. One of the main approaches in those works is to learn keypoints through image synthesis as a proxy task and many works such as [1-4] directly aim at disentangling shape (in the form of keypoints) from identity/appearance. They are even more relevant because this also provides an interpretable shape representation (keypoints) which can be used for other tasks such as manual shape manipulation and shape analysis, similar to what is presented in the work under review (and sometimes demonstrating additional capabilities such as swapping identity only on specific parts of objects, which would be a nice additional experiment so see how well this works with the presented approach). Additionally, there have also been works [5-7] which directly aim at disentangling shape and identity with a mask/part/segmentation-based shape representation similar to the one obtained in this work. Those works should be discussed and some of them also seem to be highly relevant for comparisons.\n- Since the work is about disentangling shape, it would be desirable to see results on data which contain more shape variations. The main evaluation is limited to centered faces (of humans and animals), where the biggest shape/pose variations are limited to gaze-direction. While Fig. 1 also shows results on cars and churches, there is no further evaluation on that data and the qualitative results already suggest that the model is struggling much more on the car dataset. What I would really like to see however, is the performance on articulated objects, especially full body human datasets such as DeepFashion which have also been considered in previous keypoint/mask based approaches for shape and identity disentanglement as mentioned above.\n- The experimental section is sometimes confusing and overall does not provide alot of insights. The text mentions that StarGAN-v2 and CLUIT have the best performance but CLUIT is not in Tab. 1 and both DATGAN and SNI are reported to have better FID scores than StarGAN-v2. Why does Tab. 1 report the scores for the VQ-1 variant on AFHQ but on CelebA it reports the number for the VQ-2 variant (Tab. 1 <-> Tab. 2)? Since the paper reports failure of Gumbel-Softmax, what about the argmax+straight-through estimator? The ablation study references Fig. 4 and 5 with respect to a failure of the VQ-2 variant but I fail to see how those figures connect to the ablation study. In fact, it is never mentioned with which variant those figures are produced (and as mentioned before, Tab. 1 reports number of different variants) and there is no qualitative comparison between the ablated versions. In general, there are too little qualitative comparisons and there should be a supplementary with qualitative results to be able to better judge the performance of the model and the different variants. In fact, the numbers reported in Tab. 2 obtained without using quantization (w/o VQ) outperform the baselines, so from this I really cannot judge whether the improvements over the baselines come from the proposed VQSN layer or from simply using a better architecture (StyleGAN) and training scheme (pretraining as GAN, training encoder to reconstruct latent but also encoder-decoder to reconstruct images). Given that the VQSN layer is the main contribution, its effect is discussed too little.\n\nMinor:\n- The reference keys are ambiguous, e.g. which of the works does Lee et al. refer to? There should be years/letters/etc. something to make them unambiguous and maybe follow a more common format. Also, putting reference keys after each of the method names instead of putting first all method names and then all reference keys in a list (Baselines paragraph in Sec. 4) would make it much more easy to find the correct references.\n- Why not use color augmentations for the pose image of the generated pose-identity pairs? That should simulate the actual test case closer, no?\n- The mixback LPIPS is a useful metric but also immediately suggests to use it for training similar to cycle consistency losses. Has that been tried?\n\nReferences:\n- [1] Jakab, Tomas et al. “Unsupervised Learning of Object Landmarks through Conditional Image Generation.” NeurIPS (2018).\n- [2] Zhang, Y. et al. “Unsupervised Discovery of Object Landmarks as Structural Representations.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018): 2694-2703.\n- [3] Lorenz, Dominik et al. “Unsupervised Part-Based Disentangling of Object Shape and Appearance.” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019): 10947-10956.\n- [4] Jakab, Tomas et al. “Self-Supervised Learning of Interpretable Keypoints From Unlabelled Videos.” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020): 8784-8794.\n- [5] Hung, Wei-Chih et al. “SCOPS: Self-Supervised Co-Part Segmentation.” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019): 869-878.\n- [6] Braun, Sandro et al. “Unsupervised Part Discovery by Unsupervised Disentanglement.” Pattern Recognition 12544 (2020): 345 - 359.\n- [7] Liu, Shilong et al. “Unsupervised Part Segmentation through Disentangling Appearance and Shape.” CVPR (2021).", "summary_of_the_paper": "The paper presents a generative model for unsupervised disentangling of shape and identity of images. The goal is to be able to combine two images of different identities such that the output depicts the person/animal/object of the first image in the shape/pose of the other image.\n\nOverall, this is achieved with a encoder-decoder like architecture, where the latent code is divided into a shape and a identity part. The desired mixed image is then obtained by \"grafting\" the latent codes of the two images, i.e. taking the latent identity code from the first image and the latent shape code from the second image.\n\nTo achieve the disentangling of the two codes during training, the paper relies on two main techniques: Self-supervised generation of shape-identity image-pairs, obtained by shape augmentations of a single image. A vector-quantized-normalization (VQSN) layer which introduces a bottleneck on the pose latent code with the hope that it won't have enough capacity to capture identity information and therefore distill the pose information from an image.\n\nExperiments compare the proposed model to previous approaches and ablated versions of the model and show applications enabled by features of the model such as smooth shape interpolation, shape editing via masks and robustness on out-of-domain images.", "summary_of_the_review": "The paper presents a simple approach for an important topic and demonstrates both gains over previous works as well as interesting applications that are possible with the model. However, I think it misses an important line of research on unsupervised disentangling of shape and identity based on interpretable shape representations such as keypoints and masks (e.g. [1-7]). Given these prior works, the self-supervised generation of pose-identity pairs is no new contribution. The significance of the remaining contribution with the VQSN layer is difficult to judge, since the ablations which do not use this contribution also seem to quantitatively outperform existing approaches and also do not seem very much worse than the full model. Since there are no qualitative comparisons between ablated version, its value remains unclear. So while the presented approach produces very good results, these issues make it impossible to judge where that improvement really comes from and thus it is not clear if the claims regarding the contributions and their significance is correct or not. Therefore I currently lean towards rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_Akl3EMVMSs", "reviewer": "Reviewer_dkA5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n1. This paper is overall well written with sufficient details. It feels like not too hard to reproduce while reading the paper. The main idea is clearly explained with proper illustrations.\n2. The main idea of combining vector quantization and spatially-varying normalization is intuitive and its effectiveness seems to be well supported by the experimental results when compared with related works and the visualizations. The learned masks do carry some semantic meanings as visualized.\n3. I found the Section 4.2 quite useful for understanding the model. It seems like the author explored a number of alternative solutions including Gumbel-softmax and network variations to land on the version being reported. \n4. Three applications nicely utilized the properties of the trained network with convincing results.\n\nCons:\n1. The image contents used in the experiments seem somewhat limited, e.g. aligned faces or animal heads. And the resolution is also not very high, so I would not say I'm very impressed by the results. It may be worth considering more challenging settings, such as 1024x1024, articulated objects (humans and animals) and large-scale datasets like places2. \n2. The proposed module is a nice combination of existing algorithms. While it seems to work well, but I would not be able to give a very high score for its originality. \n3. The learned (unsupervised) masks are very interesting but their quality is not good enough to be useful as intuitive control signals for interactive image editing. \n4. Typos: \"pose-layers encode only the post information\" --> \"pose-layers encode only the pose information\"\n\"We hypothesis this depends\" --> \"We hypothesize this depends\"", "summary_of_the_paper": "This paper presents a neural network module for achieving posture (shape) and identity (appearance) disentangling for image synthesis. The proposed module combines the vector quantization layer with the spatially-varying normalization layer so that the learned mask represents the shape while the \"object\" or the embedding for each mask label represents the shape-invariant features. Based on this module, this paper designs styleGAN-like generator with its discriminator and also an encoder that predicts the latent code from an input image. The training process is carefully managed with dedicated data augmentation methods to achieve the disentanglement. Results are reported on several commonly-used datasets with informative ablation study.", "summary_of_the_review": "A solid work with nice results and insightful analysis. It has limited originality but can benefit from more challenging experiments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UFb89VzgjeT", "reviewer": "Reviewer_fSLH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1.\tThe VQSN have is somewhat novel. The combination of vector quantization and spatially adaptive normalization is reasonable.\n2.\tThe self-supervised training scheme is also interesting. By applying a series of augmentations to the real images and switching the latent code for original and augmented images, the model is forced to learning the smaller-scale pose information.\n3.\tThe results look good.\n\nThe main concerns are in the experiment part which makes the reader quite confusing.\n\nCons:\n1.\tWhy the qualitative results of ablation study and some baselines (such as MUNIT, TUNIT SNI DATGAN) are missed. It is importance to visualize the changes in generated images made by different components. Why quantitative results of CLUIT are missed? \n2.\tOne important baseline is stylegan/stylegan2. Controlling the pose or identity in stylegan has been widely explored in recent works, such as GAN-inversion[3] stylegan-encoder[2], GAN latent disentanglement [1]. Comparing with stronger baseline may be better to show the advances of the proposed model.\n\n[1]Shen Y, Zhou B. Closed-form factorization of latent semantics in gans[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 1532-1540.\n[2]Richardson E, Alaluf Y, Patashnik O, et al. Encoding in style: a stylegan encoder for image-to-image translation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 2287-2296.\n[3]Pinkney J N M, Adler D. Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains[J]. arXiv preprint arXiv:2010.05334, 2020.\n3.\tImplementation details missed. Which generative architecture does the authors use for E, G, D? How does the VQSN module be plugged into the generator network?  It is very hard to re-produce results as the missed details. \n", "summary_of_the_paper": "This paper proposed a Vector-Quantized Spatial Normalization (VQSN) module for pose-identity disentanglement in GAN’s training. The module combines vector quantization and spatially adaptive normalization. Then a self-supervised training scheme is introduced to increment the feature disentanglement. The experiments show the model is able to separate pose and identity and performs better than existing baselines. ", "summary_of_the_review": "Overall, the paper is novel in method while experiments are somewhat weak to support the claims. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PpolmtSU767", "reviewer": "Reviewer_a2FQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[strength]\nThe main idea of this paper is that automatically learn the arrangement mask, and use the vector quantization with spatially-adaptive normalization. This paper presents the generalized image translation ability on various datasets. Experiments show that the proposed disentangled framework is able to translate image style into different arrangements and styles. The detailed experiment verifies the disentangled-ability of the proposed framework. Paper writing and structuring well.\n\n[weakness]\nHowever, user studies should be conducted to verify the performance and conduct experiments on the high-resolutions.\n\n", "summary_of_the_paper": "This paper focuses on the image translation problem, and proposes an arrangement and style disentangled framework that introduces the working logic by using vector quantization. It is a StyleGAN2-based method, with a vector-quantized spatial normalization. Besides, this paper also introduces a self-supervision method for the joint training between encoder and generator.  The experiments show effectiveness and interpretability. However, the experiments only use the 256p resolution, it is better to show more high-resolution results that compare with other methods.", "summary_of_the_review": "This paper proposes a disentangled method for the image translation task, which is meaningful for the computer vision community. The method sounds works but the experiments are insufficient, more experiments should be conducted on high-resolution images.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LHmhp4qIx8i", "reviewer": "Reviewer_SbZB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n- Experiments on 5 datasets \n- Good qualitative results \n**Weaknesses:**\nThe paper does not provide significant novelty on the generator architecture nor the training strategy. The proposed pipeline consists of using existing architecture (StyleGAN) and applying a series of feature operations without motivation. Below are my concerns about the work:\n- My major concern regarding the paper is the proposed Vector-Quantized Spatial Normalization (VQSN) module which is an incremental improvement of feature aggregation without properly explaining the rationale behind such operations. It is not clear why such a module can learn the pose because there is no direct evidence that the model learns a disentangled representation. \n- *Architecture choices:* It will be helpful if the authors justify the architectural choices. Specifically, justify the series of operations in the VQSN module over the “base feature map” that represents the pose. Additionally, the choice of taking 2 / 3 layers of a non-specified vector as a “pose-layer” is not motivated.\n- In the self-supervised loss, if the only examples of the identity and poses come from transformed images (rotated, zoomed, …), how to make sure that the model is not overfitting on these examples? Or do you use a small subset of image pairs?\n- Since the latent code combining the identity and the pose is obtained through the encoder E, then where is the inversion used?\n- *Resolution:* The model produces images of smaller resolution (256x256), while other state-of-the-art approaches can produce images of higher resolution e.g. DATGAN [1] has resolution of 1024x1024 in the CelebA-HQ dataset and 512x512 in the AFHQ dataset.\n- In the experiments, the variations VQ-1, VQ-2, and VQ-3 are not clearly explained and it’s hard to judge the benefit of such configurations\n\n[1] Kwon et al. Diagonal Attention and Style-based GAN for Content-Style Disentanglement in Image Generation and Translation. ICCV 2021\n\n**Minors:**\n- In the paper the number 14 is used twice. First, as a hyper-parameter to represent the number of convolutions over the “base feature map” and as the latent vector combining the pose and identify features. Does the number 14 represent the same concept in both cases? Why specifically 14, does it cover all the variation of the face orientation and identities? Or, is it dataset specific?\n- Can the authors clarify if the “Hosted embeddings” is a segmentation map? If so, this needs to be clearly specified in the paper\n- Have you tested if the proposed VQSN can be used with other architectures e.g. PGGAN [2]\n- Have you tried using other aggregations between $l^1_{[1:14]}$ and $l^2_{[1:14]}$ e.g. concatenation?\n- “such as illustration synthesis and motion driving”… Please add references \n\n[2] Karras et al. Progressive Growing of GANs for improved quality, stability, and variation. ICLR 2018\n\n**Writing:** The presentation of the paper needs some revision. Overall, the text is not self contained and the concepts used are briefly described. \n- Add reference when introducing concepts related to image-to-image translation e.g. has the pipeline consisting of an encoder E, generator G, and discriminator D been used before? What’s its advantage to just using G and D?\n\n- Try to have a problem formulation explaining clearly what each of the components (E, G, and D) is expecting as inputs and how they are related along with corresponding dimensions\n\n- The authors should specify if the method is specifically designed for StyleGAN or can be generalized to other models\n\n- Try to separate the concepts from the implementation details \n\n- Avoid using vague words such as: “certain number”, “we hope”, “ideally”. Be explicit and mention the values of each of your components. \n\n- Use simpler words, please review and change some unclear sentences as highlighted below:\n  - “we graft the latent sets” -> isn’t this a feature aggregation? \n  - “Revised feature map” \n  - “to be well communicated” \n\n- Fig.2: Is a mixture between detailed architecture (top) and abstract concepts (bottom). Have them both separated and defined the variables used e.g. $l^p, f_b, f_{vq}$", "summary_of_the_paper": "This paper presents PIVQGAN, a pipeline that tackles the image-to-image translation task. The contribution of the paper is two folds: (i) a module dubbed as Vector-Quantized Spatial Normalization (VQSN) that represents the pose and (ii) a self-supervised training scheme. Quantitative and qualitative results show good performances of the proposed pipeline.", "summary_of_the_review": "Overall, I am leaning towards rejecting the paper. The current version of the paper needs to be improved, in particular: the novelty, writing, and the experiments. \nThe novelty of the proposed VQSN is incremental, however, I would consider changing the rating of the paper if the authors can implement the suggested improvements. In particular, improve the clarity of the paper and show evidence of the performance of their method against other state-of-the-art methods at resolutions 1024x1024 and 512x512.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper offers a disentangled pose and identity representation for image to image translation.  The reviewers are borderline, but the AC finds the discussion by reviewer SuTz compelling, and agrees that the authors missed key references in the submitted manuscript.  In their rebuttal, the authors acknowledged the references were relevant, but believed their paper is not in the same area.  Overall this paper is borderline, but just below the threshold for acceptance in the opinion of this AC.
Decision: Reject