Learning Disentangled Representations for Image Translation
Authors: Aviv Gabbay
Yedid Hoshen
Venue:  (2021)
Keywords: disentanglement
image translation
latent optimization

Abstract: Recent approaches for unsupervised image translation are strongly reliant on generative adversarial training and architectural locality constraints. Despite their appealing results, it can be easily observed that the learned class and content representations are entangled which often hurts the translation performance. To this end, we propose OverLORD, for learning disentangled representations for the image class and attributes, utilizing latent optimization and carefully designed content and style bottlenecks. We further argue that the commonly used adversarial optimization can be decoupled from representation disentanglement and be applied at a later stage of the training to increase the perceptual quality of the generated images. Based on these principles, our model learns significantly more disentangled representations and achieves higher translation quality and greater output diversity than state-of-the-art methods.

Under review as a conference paper at ICLR 2021
LEARNING DISENTANGLED REPRESENTATIONS FOR IMAGE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
Recent approaches for unsupervised image translation are strongly reliant on gen- erative adversarial training and architectural locality constraints. Despite their appealing results, can be easily observed that the learned class and content rep- resentations are entangled which often hurts the translation performance. To this end, we propose OverL.ORD, for learning disentangled representations for the image class and attributes, utilizing latent optimization and carefully designed content and style bottlenecks. We further argue that the commonly used adver- sarial optimization can be decoupled from representation disentanglement and be applied at a later stage of the training to increase the perceptual quality of the generated images. Based these principles, our model learns significantly more disentangled representations and achieves higher translation quality and greater output diversity than state-of-the-ar methods.
INTRODUCTION
Learning disentangled representations from set of observations is fundamental problem in ma- chine learning. Such representations can facilitate generalization to downstream discriminative and generative tasks well improving interpretability (Hsu et al. 2017), reasoning (van Steenkiste jet al. 2019) and faimess (Creager et al.] 2211).. Recent advances have contributed to various tasks such novel image synthesis (Zhu al. 2018) and person re-identification (Eom & Ham 2019). Image translation is an extensively researched task that benefits from disentanglement. Its goal is to generate an analogous image in a target domain (e.g. cats) given an input image in a source domain (c.g. dogs). Although this task is generally poorly specified, it is often satisfied under the assumption that images in different domains share common attributes (e.g. head pose) which can be transferred during translation we name those content. In many cases, the class (domain) and common attributes do not uniquely specify the target image e.g. there are many dog breeds with the same head pose. This multi-modal translation motivates the specification of the particular class- specific attributes that we wish the target image to have - we name those style. The ability to transfer the content of source image to target class and style has been the objective of several methods e.g. MUNIT (Huang et al., 2018), FUNIT (Liu et al. 2019) and StartGAN-v2 (Choi et al.) 2020). Unfortunately, we show that despite their visually pleasing results, the translated images still retain many class-specific attributes of the original image resulting in limited translation quality. For ex- ample, when translating dogs to wild animals, current methods are prone transfer facial shapes which are unique to dogs and should not be transferred precisely to wild animals. As demonstrated in Fig. M our model transfers the semantic head pose more reliably.
In this work, we analyze the class-supervised setting and present a principled objective for disen- tangling image class and attributes. We explain why LORD, introduced by (Gabbay & Hoshen 2020). cannot be applied for multi-modal translation. We then show that introducing an additional style representation overcomes this issue and propose practical method for high-fidelity image translation by learning disentangled representations. Our method achieves this in two stages; i) Disentanglement: disentangled representation learning in a non-adversarial framework, leveraging latent optimization and well-motivated content and style bottlenecks. ii) Synthesis: the disentan- gled representations learned in the previous stage are used to "supervise" a synthesis network that generalizes to unseen images and classes. As synthesis network training is well-conditioned, can effectively incorporate a GAN loss resulting in a high-fidelity image translation model. Our approach illustrates that adversarial optimization, which is typically used for domain translation,

---

Under review as a conference paper at ICLR 2021
Content
Style
FUNIT
StarGAN-v2
Ours
Figure Examples of entanglement in ofate-of-the-art image translation models. Current ap- proaches and their architectural biases tightly preserve the original structure and generate unreliable translations. Our model disentangles the high level content and captures the target style faithfully.
is not necessary for disentanglement, and its main utility lies in generating perceptually pleasing images. Our model learns disentangled representations and achieves better translation quality and output diversity than current methods.
Our contributions are: i) Introducing non-adversarial disentanglement method that enables multi- modal solutions. ii) Learning statistically disentangled representations. iii) Extending domain trans- lation methods to cases with many (10k) domains. iv) Sfate-of-the-art results image-translation.
1.1 RELATED WORK
Image Translation Translating the content of images across different domains has attracted much attention. In the unsupervised setting, CycleGAN (Zhu et al., (017) introduces cycle consis- tency loss encourage translated images preserves the Comain--nvartei the source image. MUNIT (Huang et al. 2018) recognized that given content image could be transferred many different styles (e.g. colors and textures) in a target domain and extends UNIT (Huang & Belongie (2077) to learn multi-modal mappings by learning style representations. DRIT (Lee et al. 2018) tackles the same setting using an adversarial constraint at the representation level. MSGAN MMo al., 2019) added a regularization term to prevent mode collapse. StarGAN-v2 (Choi et al. 2020) and DMIT (Yu et al., 2019) extend previous frameworks to translation across more than two domains. FUNIT (Liu
attributes (e.g. pose) of
2019 further allows translation to novel domains.
Susse-Suee Disentanglement In this parallel line of work, the goal is to anchor the seman- ties of all the images within each class into separate class representation while modeling all the remaining class-independent attributes by a content representation. Several methods encourage dis- entanglement by adversarial constraints (Denton al.) 2017 Szabó et al.] 2018 Mathieu et al., 2016) while other rely on cycle consistency (Harsh Jha et al. 2018) or group accumulation (Boucha- court al. 2018). LORD (Gabbay & Hoshen 2020) takes a non-adversarial approach and trains a generative model while directly optimizing over class and content codes. Most works in this area demonstrate domain translation results on simple datasets but not in the multi-mod (many-to- many) settings. Moreover, their focus is to achieve disentanglement at the representation level rather than designing architectures for high-resolution image translation resulting weak performance on competitive benchmarks. In this work, we draw inspiration from LORD in relying the inductive bias conferred by latent optimization to learn a disentangled content representation. In contrast to LORD, we tackle the multi-modal image translation setting by modeling style. Moreover, we add an adversarial term in the synthesis network that increases the image quality and resolution.
BACKGROUND: REPRESENTATION LEARNING IN IMAGE-TRANSIATUE
Image translation takes as input a set of N images and corresponding class labels (I1, V1) ) (22, U2), ..., (IN, y.). Let us assume that image I; is fully specified by its class y, and attributes a;- As a motivational example, let us consider the images ø; to be animals, and the class label y; denotes the species. Attributes a, may include attributes af common all classes
2

---

Under review as a conference paper at ICLR 2021
such head angle, and class-specifie attributes of such as dog or cat breed. An unknown function G maps Yi, of and of to the image ry:
T3 = G* (yi, at af)
(I)
The goal of image translation is to replace the common attributes of of target æ; by aj of source zy;
Fij- G" G" (yis
aj. a7)
The main challenge is that during training. only the class yi of each image is given, but the attributes af and of are unknown. We define three representations corresponding to each of the physical prop- erties above - the class embedding Cy, represents the class yi, the contens code e represents the com- mon attributes of and the style code s; represents the class-specific attributes af. The computational task is to learn the representations ey, Si, such that they faithfully encode their corresponding physical properties of image 2; and are sufficient reconstruct the image using a generator G::
24 Gi 86, Gi)
This constraint however is insufficient to specify the representations uniquely. A popular constraint, utilizes the fact that common attributes of are independent of the class y,- In the motivating animals example, we assume that head pose acts similarly cats or dogs. It therefore requires independence between the content code e; and class yi-
Adversarial methods: Adversarial methods implement this constraint by training domain fusion discriminator on the translated image Zij = G(eg, $,, Cz). The discriminator attempts to distinguish translated images from original images of the target class. Unfortunately. we empiri- cally show that state-of-the-art methods do not learn content codes e; that are disentangled from the class yt- The consequence is that G transfers not only common attributes but also class-dependent attributes. We hypothesize that this failure is due to the challenging adversarial optimization.
In order constrain the style 8;, current methods (e.g. StarGAN-v2, FUNIT) rely on locality- preserving architectures that bias the content code to represent the structure, while the style op- relatively deep level within the generator
erates in a global manner per channel and is applied (typically from 16x x 16 spatial resolution). Unfortunately, such style is unable to model class-specifie attributes of of spatial nature e.g. the spatial facial shape of different dog breeds.
Non-adversarial methods: LORD is a non-adversarial approach for disentangled representation learning. It assumes that all attributes are common to all classes I(af;y) 0 and (aa) = where denotes mutual information and Hl denotes entropy. It therefore learns only class and content representations ey, and & but not style codes (s; = 0). In order to learn disentangled class and content representations it uses simple but effective constraint of minimal information in the content codes G. After learned disentangled representations Ey. and c,, LORD learns a synthesis network which generalizes to unseen images and classes.
When the assumptions of LORD are satisfied (H (H(") = 0), the content codes e; are indeed shown to recover the information of the common attributes a.. However, when class-specific attributes exist, LORD does not achieve its goals. The reason is that the content code C;, contains the information of both the common of and class specific at attributes (f(a";e) content code now contains class-specific information (e.g. cat breed), it is no longer disentangled from the class (we can predict the content of a cat); ii) when transferring content codes across classes (e.g. cats to dogs), we are missing the required class specific information in the target class (i.e. given the class code of edog and content of cat Cent: the generator is not provided with the information of what dog breed to generate).
0). This has two issues: i) as the
Due to its effective non-adversarial approach, LORD learns disentangled representations and achieves state-of-the-art results on uni-modal synthetic and low-res datasets. However, due to i) ignoring class-specific attributes as highlighted above; ii) using a non-adversarial synthesis network, it does not produce high fidelity images and cannot scale to high-resolution.
OVERLORD: PRINCIPLED DISENTANGLEMENT IN IMAGE TRANSLATION
Based on the above analysis, we present our disentangled image-translation method, OverLORD.
3

---

Under review as a conference paper at ICLR 2021
Stage 2: Synthesis with Advensarial Training
Stage Disentanglement with Latent Optimization
Class Eneader
Class
Embedding
Ey
Eyi
Content Rrabedding 62
Generator
Discrimiaor
Encoder
Ee
D
Style
Encoder
Ki
Transformation
Figure 2: A sketch of the entire method. In the disentanglement stage, we optimize over a single class embedding per class and single content embedding per image, as well as the parameters of the style encoder and the generator. In the synthesis stage, we optimize all modules in an amortized fashion, using the learned class and content embeddings as targets the new encoders. During this stage, additional adversarial discriminator is trained to increase the image fidelity.
3.1 LEARNING A DISENTANGLEMER MODEL
In order to learn disentangled representations for class, content and style, we propose non- adversarial framework with several principles.
Reconstruction: As the image should be fully specified by the representations Eyg 8i, Ca we employ a reconstruction term: Cree = E(G((e.. St, (,),)) where i is measure of similarity between images (we use the VGG perceptual loss).
Content bottleneck: In order to constrain the amount of information in each content variable c; we parameterize as a noisy channel consisting of vector d' and additive Normal noise = I), (0,
cc + z. Using the Shannon Shannon-Hartley Hartley theorem, the information capacity of the channel a function of the signal-to-noise ratio. For the noisy channel c,, the SNR ratio given by ||</||2. We therefore define the content bottleneck loss as L.s = Σ. ||]]]
Style: We train a style encoder F, : - infer the class-specific attributes a,. encourage variance to the common attributes ae, the input image first undergoes a random transformation. This creates a transformed version æ.fras the input image that shares the same class specific attributes a., but exhibits random common attributes ae (as style should be invariant to these attributes). The nature of the transformation is setting dependent e.g. if common attributes include spatial locations, then crop and rotation transformations will remove spatial attributes from the transformed image and make the style code invariant to them.
S₁ E. (. (rans)
(4)
Our complete objective can be summarized as follows:
min Ce F,G
Laisent
Σ E(G(en E. Ex (afrans) )
2), Aoolle/1 =~
(5)
There are several fundamental differences from the eVAE objective: i) the noise variance is fixed hyper-parameter and is learned; ii) there is content encoder, instead the latent content code for each variable is learned directly, in unamortized way often referred to latent optimization; iii) the input to the style encoder is a transformed version of the image. Gabbay Hoshen (2020) dis- covered that latent optimization improves disentanglement significantly over encoder-based meth- ods. The difference lies in the initialization as latent optimization starts with no class- class-content corre- lation (code initialization is IID) while content encoder starts with perfect correlation (class can be predicted from the output of a randomly initialized content encoder). We further elaborate on latent optimization and its inductive bias in Appendix A.3

---

Under review as a conference paper at ICLR 2021
Style
Content
Style
Content
StarGAN-v2
FUNIT StartiN--2
FUNIT
Ours
Ours
Figure 3: Comparison against best baselines on AFHQ (zoom in for better resolution). Class (i.e. cat, dog or wild) and style (e.g. breed) are guided by the reference image, while the content (e.g. head pose) of the source image should be preserved. StarGAN-v2 and FUNIT leak facial shapes that are unique to the source breeds leading to unreliable and inconsistent translations. Our method produces more disentangled results and captures the target style faithfully.
3.2 GENERALIZATION AND PERCEPTUAL QUALITY
After the disentanglement stage, we possess content code c, and shared class embedding ey, for every training image, and also trained style encoder E,. In order to generalize to unseen images and novel classes, we follow LORD (Gabbay & Hoshen 2020) and train content encoder F.g: X' ->
and class encoder E, X - Y to directly obtain content code C and class code V:
Lgen Σ ((() E.(,), (.( Lene
Σ -
-y, (e cs112 (6)
By optimizing the two terms, we ensure that the training set can be reconstructed in an amortized fashion (Igenn, without violating the disentanglement criteria established in the first stage (Lene)-
Note that the targets for Lene are the ones learned by our own model in the first stage.
To increase the visual quality, jointly train discriminator D and employ an adversarial loss:
Lado
log D(x;) + log(1 D(G(EE(a,)) Ex()))
Our objective of the second stage can he summarized as follows (the style encoder is unchanged):
min E..G
max D
Egen + Aenahonn + Dadto Lads
(8)
sketch of the entire method shown in Fig.
EXPERIMENTS
Our method evaluated against state-of-the-ait art methods for multi-modal domain translation and superviueer disentanglement. Note that we do not present comparisons unsupervised disen- tanglement methods such as betaVAE as they are not competitive on class-content disentanglement with methods that class-supervision. Implementation details provided in Appendix A.1
5

---

Under review as a conference paper at ICLR 2021
Table 1: Multi-modal domain translation on the three domains of AFHQ. Metries: classification ac- curacy of class labels from content codes (y <- c), classification of original class from the translated image (g) t- 2iy), translation fidelity (FID) and translation diversity (LPIPS).
Representation
Image
y f c
Y/ f- Fij 4
FID
LPIPS f
MUNIT
1.0
1.0
223.9
0.199
DRIT
1.0
1.0
114.8
0.156
MSGAN
1.0
1.0
69.8
0.375
LORD
0.74
0.47
97.1
LORD w/ style clustering
0.53
0,43
37.1
0.361
StarGAN-v2
0.89
0.75
19.8
0.432
FUNIT
0.94
0.85
18.8
0.436
Ours w/o style modeling
0.80
0.79
55.9
0
Ours w/o adversarial loss
0.33
0.38
29.1
0,448
Ours
0.33
0,42
16.5
0.511
Optimal
0.33
0.33
12.9
4.1 BASELINES
Multi-modal Domain Translation We compare StarGAN-v2 and FUNIT which the state- of-the-art domain translation methods and the established MUNIT, DRIT and MSGAN frameworks (which must be trained multiple times for every possible pair of domains). As standard LORD does not support multi-modal settings, we use its ImageNet K-MEANS augmented setting.
Plass-supervine Disentanglement We compare LORD and FUNIT as they can handle fine- grained class labels (i.e. 10k face identities), and generalize to unseen classes.
Single Attribute Manipulation We further assess the performance of our disentanglement approach in the task of translating Males to Females against FaderNetworks (Lample et al. (017) and mGAN- prior (Gu al. 2020) which operates in the latent space of StyleGAN.
4.2
DATASETS
AFHQ CCoi et al. 2020) 15, 000 high quality images categorized into three domains: cat, dog and wildlife. We follow the protocol used StarGAN-v2 and use the images at 256 x 256 resolution, holding out 500 images from each domain for testing.
CelebA (Liu et al. 2015) 202,599 images of 10,177 celebritics. We designate the person identity class. We crop the images 128 128 and 9,177 classes for training and 1, 1,000 for testing.
CelebA-HQ (Karras et al., 2018) 30,000 high quality images from CelebA. We set the gender as class. We resize the images to 256 256 and leave 1,000 images from each class for testing.
4.3 EVALUATION PROTOCOL
We assess the disentanglement at two levels: the learned representations and the generated images. At the representation level, we follow the protocol in LORD (Gabbay Hoshen 2020) and train a two-layer multi-layer perceptron to classify class labels from the learned content codes (lower accuracy indicates better disentanglement). In CelebA, where annotations of part of the common attributes are available (for evaluation only) such as 68-facial landmark locations, we train linear regression model to locate the landmarks given the learned class (identity) codes (higher error indi- cates better disentanglement). At the image level, we follow StarGAN-v2 and translate all images in the test set to each of the other domains multiple times, borrowing style codes from random ref- erence images in the target domain. We then train classifier to classify the original class of the content image from the generated image. A lower accuracy indicates better disentanglement as the source class does not leak into the translated image. We also compute FID (Heusel al. 2017 in

---

Under review as a conference paper at ICLR 2021
Class
Content
Class
Content
FUNIT
FUNIT
LORD
LORD
Ours
Ours
Figure 4: CelebA: Class (i.e. facial identity) and style (e.g. illumination) are guided by the reference image, while the content (e.g. head pose and expression) of the source image should remain intact. FUNIT preserves head pose but fails to model facial expression transfer the exact facial identity. LORD captures the facial identity but generates low fidelity images. Our method preserves head pose and facial expression well while transferring the identity and generating appealing images.
Table Class- suservsuee disentanglement of 10K classes on CelebA. Representation results: clas- sification accuracy of class labels from content codes (y + c) and the error of landmarks regression from class codes (R(ey) + e). Identity FaceNet cosine similarity, Expression RMSE on facial landmarks, Head Pose yaw, pitch, roll angle errors, Rec reconstruction error (LPIPS). Indicates loss of head pose and expression details.
Representation
Image
y + c4
R(eg) + et Identity f
Expression 4.
Head Pose 4
Rec
FUNIT
0.16
2.6
0.24
3.82
5.2, 6.9, 2.0
0.334
LORD
0.0001
3.6
0.48
3.17
4.2, 4.6, 1.8
0.325
Ours
0.001
3.6
0.63
2.71
2.8, 3.3, 1.4
0.310
Optimal
0.001
0
0, 0, 0
conditional manner to measure the discrepancy between the distribution of images in each target domain and the corresponding translations generated by the models. lower FID score indicates reliable and better fit the target domain. FID between real train and test images of the same class forms the optimal score for this metric. In order to assess the diversity of translation results, we measure the perceptual pairwise distances using LPIPS (Zhang between all translations of the same input image. Higher average distances indicate greater diversity in image translation. In cases where external annotation methods are available (for evaluation only), such face recognition (Cao et al. (0188) and head pose (Ruiz et al.] 2018) and landmark detection CelebA, we further measure similarity the identity of the generated face and the refer- ence, as well as expression (represented by landmarks) and head pose errors. While we do not have ground truth for translation CelebA as well, we present a reconstruction error (LPIPS similarity) by inferring the class (identity) representation from one image and reconstructing second of the same person by extracting the content representation from an image of a different identity which has the closest pose (nearest neighbour in the 68 facial-landmarks space). For the single attribute manipulation experiment on CelebA-HQ, we measure the accuracy of fooling a target classifier.
that the translations
2018)
4.4 RESULTS
We present a quantitative evaluation of multi modal domain translation on AFHQ in Tab. [ Our method outperforms all baselines achieving near perfect disentanglement the representation level

---

Under review as a conference paper at ICLR 2021
Female Male
Male Female
Toder
StyleGAN
Ours
Figure 5: CelebA-HQ Male/Female: Our method makes more significant changes to the face than Fader Networks while preserving the similarity to the original identity better than mGANprior.
Table 3: Female translation results on CelebA-HQ.
Target Classification
FID J
F2M
M2F
F2M
M2F
Fader Networks
0.82
0.80
119.7
81.7
mGANprior on StyleGAN
0.59
0.76
78.2
45.3
Ours Optimal (Real Images)
0.98
0.97
54.0
42.9
0.99
0.99
15.6
14.0
as the classifier could barely guess class labels from content codes by random chance. Similar results are obtained by trying to classify the original class of the translated image. We hypothe- size that the minor degradation in this metric due using deeper classifier the image level (VGG-based) It can be seen that the baselines leak a little bit less information at the output image compared the representation level, which might indicate that their generator ignores some parts of the representations. Moreover, we achieve both better visual quality (FID) and higher translation diversity (LPIPS). As can be seen the qualitative examples Fig. 3 StarGAN-v2 and FUNIT leak significant amount of details from the input image which results in unreliable translations. Our method generates much more disentangled translations and captures the exact style the refer- ence image. In order to assess the performance in cases where fine-grained class labels are available and the ability to generalize to unseen classes at inference time essential, we provide comparison on CelebA with 10K differentt identities in Tab. 2 It can be seen that our method achieves better expression and pose transfer while preserving the face identity more faithfully than the baselines. To validate that content is distributed evenly across identities, we landmark annotations together with pose-related attributes from CelebA (Open Mouth and Smiling) and train classifier to infer the identity. The accuracy of this classifier (0.001) forms the optimal result for the representation metric. Examples from this experiment are presented in Fig. 4 We also assess the performance of our method in the single attribute manipulation task of translating males to females on CelebA-HQ in Tab. 3 and Fig. [S] FaderNetworks makes minor changes to the given face which do not result in reliable translations. mGANprior modifies the input face significantly which reduces the sim- ilarity the original identity, many subtle facial attributes are lost when inverting real images within the StyleGAN latent space. Moreover, some of the images were not well reconstructed even though StyleGAN is trained on CelebA-HQ. Our model achieves near perfect classification score and generates visually pleasing results. More qualitative results are provided in Appendix A.2
8

---

Under review as a conference paper at ICLR 2021
4.5 ABLATION STUDY
In order to isolate the contribution of our main principles, we provide an assessment of several variants of our method.
Style modeling for multi-modal translation We train our model without style modeling (treating content and style as a single representation, both are complementary to the class), As can be seen in Tab. I this strategy violates the disentanglement criteria as class-specific attributes are now modeled by the content. As a consequence, the animals are translated in an unreliable and entangled manner between classes, as can be seen in Fig 11 in Appendix
A.4
Adversarial loss for perceptual quality We train our method without the adversarial loss in the synthesis stage. The results in Tab. suggest that the disentanglement criteria are achieved by our non-adversarial framework, while the additional adversarial loss contributes for increasing the output fidelity (FID). Qualitative evidence for this claim presented in Fig 11 in Appendix A.4
Limitations There are two main limitations of the proposed method; i) The disentanglement be- tween content and style (both complementary to the class) is achieved in an unsupervised manner and therefore relies on an inductive bias. In our framework it is driven by the design of a set of ran- dom transformations which the style needs to be invariant. While we show that common settings in image translation can be solved effectively with simple operators, other settings may require more sophisticated transformations. ii) Our model not well optimized for cases in which the object is not the major part of the image e.g. where the background contains other objects or large variation. We hypothesize that this can be better solved by introducing an unsupervised segmentation objective into the entire framework. We leave this direction to future research.
5 CONCLUSION
We present a principled approach for learning disentangled representations in a non-adversarial framework, leveraging latent optimization and well-motivated content and style bottlenecks. We show how adversarial optimization can be decoupled from representation disentanglement and be applied only to increase the fidelity of the generated images. Our proposed model learns disen- tangled representations and achieves state-of-the-art performance compared to both adversarial and non-adversarial disentanglement methods in different settings of image translation.
REFERENCES
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In Thirty-Second AAAI Con- ference on Artificial Intelligence, 2018.
Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). pp. 67-74. IEEE, 2018
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2020.
Elliot Creager, David Madras, Jörn-Henris Jacobsen, Marissa A Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. In Interna- tional Conference on Machine Learning. 2019.
Emily L Denton et al. Unsupervised learning of disentangled representations from video. In Ad- vances in neural information processing systems, pp. 4413,4423, 2017.
Chanho Eom and Bumsub Ham. Learning disentangled representation for robust person re- identification. In Advances Neural Information Processing Systems, pp. 5297-5308, 2019.
Aviv Gabbay and Yedid Hoshen. Demystifying inter-class disentanglement. In ICLR, 2020.

---

Under review as a conference paper at ICLR 2021
Jinjin Gu, Yujun Shen, and Bolei Zhou. Image processing using multi-code gan prior. In CVPR, 2020.
Ananya Harsh Jha, Saket Anand, Maneesh Singh, and VSR Veeravasarapu. Disentangling factors of variation with cycle-consistent variational auto-encoders. In ECCV, 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. In Advances in neural information processing systems, pp. 1878-1889, 2017.
2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1501- 1510, 2017.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In Proceedings of the European Conference Computer Vision (ECCV), pp- 172-189, 2018.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, and Marc' Aurelio Ranzato. Fader networks: Manipulating images by sliding attributes. In Advances in neural information processing systems, pp. 5967-5976, 2017.
Hsin- Hsin-Ying Lee, Hung- Yu Tseng, Jia-Bin Huang, Mancesh Kumar Singh, and Ming-Hsuan Yang. Diverse image-to-ing -ogeg translation via disentangled representations. In European Conference Computer Vision, 2018.
Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-image translation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 10551-10560, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. http://mmlabiieccom www..cckkkcckko.c htt..h///poojeets/cceeb.... In ICCV, 2015.
Qi Mao, Hsin- Ying Lee, Hung-Yi Yu Tseng, Siwei Ma, and Ming- Hsuan Yang. Mode seeking genera- tive adversarial networks for diverse image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1429-1437, 2019.
Michael F Mathieu. Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Disentangling factors of variation in deep representation using adversarial training. In NIPS, 2016.
Nataniel Ruiz, Eunji Chong, and James M. Rehg. Fine-grained head pose estimation without key- points. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
Attila Szab6, Qiyang Hu, Tiziano Portenier, Matthias Zwicker, and Paolo Favaro. Challenges in disentangling independent factors of variation. ICLRW, 2018.
Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disen- tangled representations helpful for abstract visual reasoning? In Advances in Neural Information Processing Systems, pp. 14245-14258, 2019.
Xiaoming Yu, Yuanqi Chen, Shan Liu, Thomas Li, and Ge Li. Multi-mapping image-to-iman to-image translation via learning disentanglement. In Advances in Neural Information Processing Systems, pp. 2994-3004, 2019.
10

---

Under review as a conference paper at ICLR 2021
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018.
Jun-Yan Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
Jun- Yan Zhu, Zhoutong Zhang, Chengkai Zhang. Jiajun Wu, Antonio Torralba, Josh Tenenbaum, and Bill Freeman. Visual object networks: Image generation with disentangled 3d representations. In Advances neural information processing systems, pp. 118-129, 2018.
A APPENDIX
A.I IMPLEMENTATION DETAILS
Architectures We borrow the generator and discriminator architectures from StyleGAN2 with two modifications; (i) The latent code is adapted our formulation and forms concatenation of three separate latent codes: content, class and style. (ii) We do not apply any noise injection during training. A brief summary of the architectures is presented in Tab. 44] and 5 for completeness. The architectures of the content, class and style encoders is influenced by StarGAN-v2 (Choi et al.) 2020) and presented Tab. 6 Note that we do not use any domain-specitn to generalize to unseen domains at inference time. For cases where the style is minimal and should mainly model low-level image features (e.g. the information that is not described by identity and pose in CelebA) we use a shallow style encoder with a small receptive field, as can be seen in Tab. 7
layers and our model is able
Optimization In the first stage, we optimize over single content embedding per image (content dim 644), single class embedding per class (class dim 512) and the parameters of the style encoder (style dim = 64) and the generator. We set the learning rate of the latent codes to 0.01, the learning rate of the generator to 0.001 and the learning rate of the style encoder to 0.0001. The penalty of the content bottleneck is set to Aet = 0.001. We train the disentanglement stage for 200 epochs. For each mini-batch, we update the parameters of the models and the latent codes with single gradient step each. In the synthesis stage, we add two encoders to infer the content and class codes learned in the first stage directly from an image. We also optimize a discriminator in an end- to-end manner to increase the perceptual fidelity of the images. This stage is trained for 100 epochs and the learning rate for all the parameters in this stage is set 0.0001, except for the style encoder whose weights are kept fixed.
Transformations As stated in Sec. 3 the design of the transformation applied on the input to the style encoder is setting dependent. For AFHQ. in order to encourage the style to be invariant to the head pose of the animal, we randomly flip the image horizontally and crop 0.6 to portion of the image. In CelebA, as we are given fine grained class labels (person identity). the desired style should ideally model only low level features and therefore we opt for a shallow style encoder with a small receptive field (5x5) except for the same augmentations as of AFHQ. The low receptive field avoids the style from encoding common-attrin attribbtes as facial expression (which might retain even after the transformation).
Baseline models For the evaluation of competing methods, use the following official publicly available pretrained models: StarGAN-v2 (for AFHQ) and StyleGAN (for mGANprior on CelebA- HQ). We train the rest of the baselines using the official repositories of their authors and make an effort to select the best configurations available for the target resolution (for example, FUNIT trained by us for AFHQ achieves similar results to the public StarGAN-v2 which was known the SOTA on this benchmark).
11

---

Under review as a conference paper at ICLR 2021
Generator architecture based StyleGAN2. StyleConv and ModulatedConv use the in- jected latent code composed of the content, class and style representations.
Table 4:
Layer
Kernel Size
Activation
Resample
Output Shape
Constant Input
4 x 4 512
StyledConv
Fusedl FusedLeakyReLLJ
4 x 512
StyledConv
3 3
FusedLeakyRedd ReUU
UpFirDn2d
8 x 8 512
StyledConv
3
FusedLeaky ReLU
8 x 512
StyledConv
3 3
FusedLeakyReLJ
UpFirDn2d
16 x 16 x 512
StyledConv
3
Fuse
16 x x 512
StyledConv
3 3
Fusedl Leaky
UpFirDDD2d
32 x 32 x 512
StyledConv
3 3
Fusedl ReayyRLU
32 x 32 x 512
StyledConv
3
FusedLeakyReLU
UpFirDn2d
64 64 x 512
StyledConv
3 3
FusedLcakyReeLU
64 X 64 512
StyledConv
3
FusedLeakyRell.
UpFirDn2d
128 X 128 256
StyledConv
3 3
FusedLeaky
128 x 128 x 256
StyledConv
3 3
FusedLeakyRelle
UpFirDn2d
256 x 256 128
StyledConv
3 x 3
FusedLeekyReLLU
256 x 256 128
ModulatedConv
256 256
Table 5: Discriminator architecture based on StyleGAN2.
Layer
Kernel Size
Activation
Resample
Output Shape
Input
256 x 256 x 3
Conv
3
ansedLeakyRel
256 256 128
ResBlock
3 X 3
Fusedl LeakyReI.U
UpFirDn2d
128 128 x 256
ResBlock
x 3
FusedLeaky Ue
UpFirDn2d
64 x 64 x 512
ResBlock
3 3 3
FusedLeakyRell
UpFirDn2d
32 32 512
ResBlock
3 3
FusedLeaky U
UpFirDn2d
16 16 x 512
ResBlock
3 3x33 3
FusedLeaky Laakleakk
UpFirDn2d
8 8 x 512
ResBlock
x 3
Fusedl LeakkyeLk
UpFirDn2d
4 x 512
Concat stddev
3
FusedLeaky ReLU
UpFirDn2d
4 4 x 513
Conv
3x3
Lakydle
4 4 4 x 512
Reshape FC
8192
FusedLeakyReLU
512
FC
Training resources Training takes approximately 3 days for 256x256 resolution single NVIDIA RTX-2080 TI.
A.2 ADDITIONAL RESULTS
We provide additional qualitative results on AFHQ Fig. on CelebA Fig and on CelebA-
HQ in Fig. T0
A.3
LATENT OPTIMIZATION
In this work, we opt for learning the content representation using latent optimization, similarly to LORD (Gabbay & Hoshen| 2020). Autoencoders assume parametric model, usually referred as the encoder, to compute vector a from image on the other hand, we jointly optimize the inputs and the model parameters. Since vector c; is learned directly and is unconstrained by parametric encoder function, our model can recover all the solutions that could be found by autoencoder, and reach some others. In order to justify this design choice, we validate the observation presented in LORD and train our disentanglement stage in amortized fashion using a content encoder. As can be seen in Fig. [4, amortized training fails to reduce the correlation between the content and class representations. We have experimented with several content decay factors (Deb in Eq. 5). Although
12

---

Under review as a conference paper at ICLR 2021
Table 6: Encoder architecture based StarGAN-v2. Note that do not use any domain-speciffe layers. D is the dimension of the content, class or style code respectively.
Layer
Kernel Size
Activation
Resample
Output Shape
Input
256 x 256 x 3
Conv
256 256 x 64
ResBlock
3 3 3
LeakyReLU (a 0.2)
Avg Pool
128 128 x 128
ResBlock
LeakyReLU (a = 0.2)
Avg Pool
64 64 x 256
ResBlock
3 3
LeakyReLU (a 0.2)
Avg Pool
32 32 256
ResBlock
3
LeakyReLU (a = 0.2)
Avg Pool
16 16 x 256
ResBlock
3x3
LeakyReL.U (a 0..)
Avg Pool
8 8 x 256
ResBlock
3x3
LeakyReL.U (a = 0.2)
Avg Pool
X 4 x 256
Conv
4x4
LeakyReLU (a 0.2)
1 X x 256
Reshape
256 D
FC
Table 7: Encoder shallow architecture with low receptive field for settings in which classes only exhibit low level intra style variation (e.g. CelebA).
Layer
Kernel Size
Activation
Resample
Output Shape
Input Conv
256 x 256
3
3
LeakyReLU (a = 0.2)
256 x 256 x 64
Conv
3 3
Global Avg Pool
64
FC
64
the disentanglement improves as Aeb increases, the reconstruction gets worse and the model fails converge with Aet > 0.1.
A.4 VISUALIZATION OF ABLATION ANALYSIS
Examples from the ablation analysis are provided in Fig.
Visualization of the three factors
modeled by method is provided in Fig. 12 and Fig.
A.5 FAILURE CASES
As our method strongly relies on the style image to infer the class-specific attributes c.g. identity and facial shape, we have found that when the animal in the style image present in an extreme pose such wide open mouth, the style encoder can sometime confuse the extreme pose as property of breed and transfer it from the style image. Examples are provided in Fig. [5
13

---

Under review as a conference paper at ICLR 2021
Content
Style
StarGAN-v2
FUNIT
Ours
StarGAN-v2
FUNIT
Ours
StarGAN-v2
PUNIT
Ours
Figure 6: More qualitative results on AFHQ.
14

---

Under review as a conference paper at ICLR 2021
Content
Style
StarGAN-v2
FUNIT
Ours
StarGAN-v2
FUNIT
Ours
StarGAN-v2
FUNIT
Ours
Figure 7: More qualitative results on AFHQ
15

---

Under review as a conference paper at ICLR 2021
(a) FUNIT
(b) LORD
(c) Ours
Figure 8: More qualitative results on CelebA. Top row: Original content images. Left column: Original class and style images.
16

---

Under review as a conference paper at ICLR 2021
(a) FUNIT
(b) LORD
(c) Ours
Figure 9: More qualitative results on CelebA. Top row: Original content images. Left column: Original class and style images.
17

---

Under review as a conference paper at ICLR 2021
V
Input
E de
FaderNerworks
E de
StyleGAN
Ours
(a) Female Male
Input
FaderNetworks
StyleGAN
Ours
(b) Male to Female
Figure 10: More qualitative results on CelebA-HQ.
18

---

Under review as a conference paper at ICLR 2021
Content
Style
9/A
Ours
w/ stylc
WA
Ours
w/o stylc
w/o wo GAN
Ouus
Figure II: Qualitative examples from the ablation analysis; (i) Only a single translation possi- ble without modeling style, which often yields entangled results. (ii) Disentanglement is achieved without adversarial loss. (iii) Adversarial loss contributes to visual fidelity.
19

---

Under review as a conference paper at ICLR 2021
Content
To Cat
Cat Style
To Dog
Dog Style
To Wild
Wild Style
Figure 12: Visualization of the three factors modeled by our method. Changing the class (e.g. To cat, dog, wild) while leaving the style intact affects high level semanties of the presented animal. Borrowing the style from reference image allows for specification of the exact target breed. Note that class and style are correlated and therefore transferring style across classes can generate unreliable translations (e.g. first two images in "To Wild".)
20

---

Under review as a conference paper at ICLR 2021
(a) Transferring class from images on the left column to images on the top row.
(b) Transferring class and style from images the left column to images on the top row.
Figure 13: Visualization of the three factors modeled by our method. Top row and left column present original images. Translating the images in each column to the class of the image on the left column (a) affects the person identity. Borrowing the style from the image on the left column (b) transfers illumination and class-specific attributes hair color. In both cases the head pose and the facial expression remain intact.
21

---

Under review as a conference paper at ICLR 2021
1.0
0.8
accurscy
0.4
Optiiization
Amortization (bottkerack - 0.0011
Amortizztion (bottlereck 0.01)
Amortization (bottkeneck
0.0
100
epoch
Figure 14: Evidence for the inductive bias conferred by latent optimization on AFHQ (a validation of the discovery presented in Gabbay & Hoshen (2020)). Latent optimization starts with random initialized content codes and preserves the disentanglement between content and class along the entire training. order to result in disentangled content codes, it is required to remove the class information from the content during the course of optimization. this plot we can see that the optimization is in practice unsuccessful at doing so.
contrast, a randomly initialized content encoder outputs entangled codes. In
Content
Style
Content
Style
Figure 15: Failure cases where the mouth and tongue are transferred over from the style image.
22


Reviews:
[{"review_id": "yIEyBCSMCn", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a novel approach named OverLORD to learn disentangled representations for image class and attributes. To tackle the problem of previous methods that the learned content and class are often entangled, the authors propose to disentangle image representations to class and attributes, and further disentangle attributes to common attributes among all classes (content), and class-specific attributes (style). It uses the idea from LORD to disentangle the image representations, and extends LORD to not only common attributes (content), but also class-specific attributes (style). In this way, it is able to transfer the common attributes while preserving the class and class-specific attributes. Experiments are conducted on animal faces and human faces datasets, and the proposed approach is able to preserve the class-specific attributes (e.g., shape or identity of the face) better than previous image-to-image translation methods that only disentangle style and content.\n \nPros: \n\n1. By separating common attributes and class-specific attributes, the model achieves better control for image-to-image translation. For example, when conducting image-to-image translation for animal faces, previous approaches tend to encode both pose and shape/structure information in the content code, so that then transferring the pose of an animal face, the shape may also change. The disentanglement space in this paper gives clearer definitions and better control for image translation.\n\n2. The proposed method uses a transformed version of the input images with random common attributes to force that the style encoder learns only class-specific style information. This is a smart design that successfully disentangles the class-specific attributes.\n\n3. The results are pretty good compared with previous approaches.\n\nCons: \n\n1. There are no ablation studies to validate the effectiveness of the architecture design. For example, (1) the effect of different loss functions. (2) What's the shape of the content embedding? Is it a feature vector or feature maps? Why do you choose the styleGAN2 structure? What if using a generator similar to the generator in FUNIT that uses AdaIN to combine the class and style information with the content information?\n\n2. The random transformations for transforming the images before the style encoder are manually defined, and the set of transformations is different for different datasets (Appendix A.1). This restricts the approach from generalizing to other various datasets with different types of class-specific attributes.\n\n3. The number of classes are predefined and fixed. It cannot generalize to unseen classes like FUNIT.\n\nReasons for score: \n \nThis paper solves a common problem in previous work for disentangling attributes and image translation. The method is quite novel and inspiring, and the result seems good compared with previous approaches.\n\nQuestions during the rebuttal period: \n\nPlease address and clarify the Cons above.\n\nPost-rebuttal:\n\nAfter reading the author feedback and other reviewers' comments, I would change my rating to 6. I partially agree with reviewer #3 and reviewer #5 that this work is an incremental extension of LORD. In addition, like I mentioned in the \"Cons\", the styles are decided by the pre-defined transformations that are dataset-specific. For different datasets the transformations are defined differently based on the properties of that dataset. This might restrict the method from being extended to new datasets with styles that are not easy to find corresponding transformations. However, I think this paper still have some inspiring aspects, like the idea of disentangling class-specific attributes and common attributes among all classes. And the results of this paper seems good compared with previous approaches. So I would give the rating of 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "xDT8mucrAxn", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes OverLORD for learning disentangled representations for image translation. The method employs a latent optimization framework and followed a GAN training process. This work also designs a style encoding framework by introducing transformation before style encoder. The results show good fidelity performance. \n\nApparently, this work is inspired by LORD, another non-adversarial method of learning disentangled representation. The concepts of class and content are borrowed from the LORD algorithm. This paper introduces an additional disentanglement dimension called style, which seems to be learned unsupervised. The learning method for this style is similar to learning general attributes that are robust to transformations.\n\nMy major concern is also about this ''style''. From Figure.3 and Figure.4, what is the different between ''style'' and ''class''? It seems that ''style'' and ''class'' are somewhat coupled. Could the author please explain what is the ''style'', ''common attributes'', ''class'' and ''class specific attributes'' for both cases in Figure.3 and 4. In order to enable readers to better understand, authors can consider visual transformations and the learned styles.\n\nThe second stage employs adversarial training, which greatly improved the visual effect. But the introduction of adversarial training is not a huge novelty. In the comparison, I am curious how much better OverLORD without adversarial training is than LORD. Is the introduction of style codes what makes OverLORD more suitable for image translation? If it is, please verify it experimentally.\n\nIf the authors are able to address the above questions, then I am happy to raise my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lRcqUOyr7PK", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a principled approach to style transfer by disentangling class-specific attributes from common (eq. class-independent) attributes. In order to do so, the paper leverages the formulation of a recently proposed disentangling approach called \"LORD\". The proposed approach is called OverLORD, and includes two main augmentations to LORD. The first is the introduction of a style encoder to learn a latent code for class-specific attributes, and the second is to introduce an adversarial learning in the second stage for high-quality style-transferred generation of images. The results are shown on three datasets: AFHQ (dog, cat, wildlife), CelebA (human faces), and CelebA-HQ (hi-res human faces), and are compelling in both qualitative and quantitative comparisons. \n\nStrengths: \n+ The weakness in LORD for style transfer applications has been identified and addressed (to an extent). \n+ The evaluations and comparisons presented are thorough and the results are very compelling, both qualitative and quantitative. Many different metrics have been explored for evaluating both disentanglement as well as style-transferred image generation. \n\nWeaknesses: \n- The main contributions of this work seems to be handling the style and content disentangling, and the quality of image generation. While the results are impressive, the exposition does not justify why such an extension to LORD is a nontrivial contribution. Consequently, the paper seems like a \"natural\" (or straightforward) extension of LORD, giving an impression of limited novelty. \n- LORD was ostensibly non-adversarial, but OverLORD is not. The necessity for an adversarial loss can be perhaps better justified by an ablation on with and without the adversarial loss to emphasize why a different kind of loss will not suffice. \n- The disentangling between class and content essentially comes from the design of LORD's latent optimization. However, it is not clear what is the key factor for disentangling the style and content, both of which are image specific. For instance, in Fig. 3 (left), the style from the leopard (jaguar?) image to the content of lion image is successfully transferred, despite both having the same class (wildlife). How are style and content being disentangled then? It appears that the choice (or family) of transformations at the input of the style encoder is important in disentangling style and content. I believe a deeper discussion of this aspect is warranted. If indeed the choice of transformation is crucial, how robust would the disentangling be to the inexact/inappropriate transformation?  \n- Style encoding requires transformations to be hand-picked for each dataset or setting. What kind of restrictions does this impose on the way the \"style\" component is defined? \n- I was curious about the failure cases of OverLORD, and I believe that a small set of qualitative examples would be a valuable addition. A short commentary on the limitations of the proposed approach would be very insightful.\n- Competing techniques & comparisons. \nHow were the results for competing techniques generated? Were the trained models available? Or were they retrained by the authors? Given the general difficulties in training the large adversarial learning based models, can the authors comment on the quality of the trained models and that of the results reported? Are the competing models of StyleGAN, FUNIT, etc., trained well enough to approximately achieve results reported in the respective papers of these competing methods? \n\nOther minor comments: \n- It will be great to include the training time and GPU infrastructure used. \n- In Table 1, the optima FID score is reported to be \"12.9\". While the classification accuracy was clear from the number of classes & random chance (also explained in text), the optimal FID score is not so obvious to me. It would be useful to add a short explanation in the text (or the caption of the table). \n \nI was torn between the scores 5 and 6. My choice is primarily due to the lack of clarity on the role of the transformations in learning the style encoder, and the limitations or restrictions on the style factors resulting from the choice of these transformations. I am looking for deeper insights from the authors and I am open to upgrading my rating.\n\nPost Rebuttal: \nThe authors have resolved my main concerns. However, the reliance on the nature of transformations applied is still an important factor. Nonetheless, I do see value in the approach presented in this paper. I am revising my rating to 6. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was near the borderline, but ultimately, calibrating with the acceptance criteria applied to submissions across the conference, we didn't find sufficient enthusiasm among the reviewers to accept the paper.  Two reviewers put it just above the bar for acceptance, on the strength of its results.  A third reviewer finds the results to be a small improvement over other work, and finds the definitions of class, content, and style used by the authors to be confusing.  The AC agrees with the 3rd reviewer that it is more natural to define (for the class of faces) the identity to be the content and the facial pose to the the style.  

Unfortunately, acceptance to ICLR required a stronger case than the reviewers presented for this paper.
The remaining concerns which swayed the AC's opinion included:

--concern that this was an incremental extension of LORD

--the reliance on the nature of transformations applied in the algorithm

--lack of any enthusiastic reviewer championing acceptance for the paper.


Decision: Reject