Rethinking Content and Style: Exploring Bias for Unsupervised Disentanglement
Authors: Xuanchi Ren
Tao Yang
Wenjun Zeng
Yuwang Wang
Venue:  (2021)
Keywords: Unsupervised Disentanglement
Content and Style Disentanglement
Inductive Bias
Representation Learning

Abstract: Content and style (C-S) disentanglement intends to decompose the underlying explanatory factors of objects into two independent latent spaces. Aiming for unsupervised disentanglement, we introduce an inductive bias to our formulation by assigning different and independent roles to content and style when approximating the real data distributions. The content embeddings of individual images are forced to share a common distribution. The style embeddings encoding instance-specific features are used to customize the shared distribution. The experiments on several popular datasets demonstrate that our method achieves the state-of-the-art disentanglement compared to other unsupervised approaches and comparable or even better results than supervised methods. Furthermore, as a new application of C-S disentanglement, we propose to generate multi-view images from a single view image for 3D reconstruction.

Under review as a conference paper at ICLR 2021
RETHINKING CONTENT AND STYLE: EXPLORING BIAS FOR UNSUPERVISED DISENTANGLEMENT
Anonymous authors Paper under double-blind review
ABSTRACT
Content and style (C-S) disentanglement intends to decompose the underlying explanatory factors of objects into two independent latent spaces. Aiming for supervised disentanglement, we introduce an inductive bias to our formulation by assigning different and independent roles to content and style when approximat- ing the real data distributions. The content embeddings of individual images are forced share a common distribution. The style embeddings encoding instance- specific features are used to customize the shared distribution. The experiments on several popular datasets demonstrate that our method achieves the state-of- tfte-of-the-- of-the- art disentanglement compared to other unsupervised approaches and comparable or even better results than supervised methods. Furthermore, as a new application of C-S disentanglement, we propose to generate multi-view images from single view image for 3D reconstruction.
INTRODUCTION
The disentanglement task aims to recover the underlying explanatory factors of natural images into different dimensions of latent space, and provide an informative representation for tasks like image translation (Wu et al.) [2019]] KKotovenko et al. 2019), domain adaptation (Li et al. [219 Zou et al.)
2020) and geometric attributes extraction (Wu et al. 2019c Xing et al. 2019), etc.
2018 Higgins et al., 2017 Burgess et al. 201 Sa; Kumar et al., 2017) learn disentangled factors by optimizing the total correlation in an unsupervised man- However, Locatello et al.) (2019) prove that unsupervised disentanglement is fundamentally impossible without inductive bias on both model and data.
The previous methods (Kim & Mnih
In this paper, focus on content and style (C-S) disentanglement, where content and style represent two separate groups of factors. The main novelty of our work is that we assign different roles to the content and style in modeling the image distribution instead of treating the factors equally, which is the inductive bias introduced in our method. Most of the previous C-S disentanglement works (Denton & Birodkar 2017 Tha et al. 2018, [Buchhcourt al.) 2018 Gabbay & Hoshen 2020) rely on supervision, which is hard to obtain for real data. E.g., Gabbay & Hoshen (2020) leverage group observation to achieve disentanglement by forcing images from the same group to share a common embedding. To our best knowledge, the only exception is Wu et al. (2019c). However, this method forces the content path learn geometric structure limited by 2D landmarks.
Our definition of content and style is similar to Gabbay & Hoshen (200)), where the content includes the information which can be transferred among groups and style is image-specific information. When group observation is not available, we define content includes the factors shared across the whole dataset, such as pose. Take the human face dataset CelebA (Liu et al.) (015) as an example, the content encodes pose, and style encodes identity, and multi-viws of the same identity have the same style embeddings, but different content embeddings, i.c., poses.
Based on the above definitions, we propose new problem formulation and network architecture by introducing an inductive bias: assigning different and independent roles to content and style when approximating the real data distributions. Specifically, embeddings of individual images to share a common distribution, and the style embeddings are used to scale and shift the common distribution match target image distribution via generator.
shown in Figure I we force the content

---

Under review as a conference paper at ICLR 2021
Style Latent Space
C-S Fusion Block
fu
fo
Content Latent Space
X
Generator
I Sampling
5K
Overview of our framework. Cz. Cj C labelled with different shapes are embeddings sampled from a shared distribution V. St, Sj, Sy labelled with different colors are embeddings from the style latent space. fo and fa are two fully-connected layers predicting the statistics parameters to scale and shift respectively to approximate the target image distributions via Generator. For each generated image from x 3 grid, the content and style embeddings are from the column and row respectively.
Figure| 1:
We follow [Bojanowski et al. (2018) and Gabbay & Hoshen (2020) to apply latent optimization to optimize the embeddings and the parameters of the generator. We also propose to use instance discrimination as a complementary constraint to assist the disentanglement. Please note that we only the image reconstruction loss as the supervision; no extra labeling is needed. As the content and style perform a different and independent role when modeling the data, they are disentangled to encode the shared and instance-specifie features respectively after the optimization.
The contributions our work are as follows: we achieve unsupervised C-S disentanglement by in- troducing an inductive bias in our formulation: assign different and independent roles content and style when modeling the real data distributions. Furthermore, we achieve better C-S disentanglement by leveraging instance discrimination. The experiments on several popular datasets demonstrate that our method achieves the state-of-the-an unsupervised C-S disentanglement and comparable or even better results than supervised methods. Besides, we propose to apply C-S disengagement new task: single view 3D reconstruction.
RELATED WORK
Unsupervised Disentanglement. disentangled representation can defined as one where indi- vidual latent units are sensitive to changes in individual generative factors. There have been lot of studies on unsupervised disentangled representation learning (Higgins et al. [017 Burgess al, 201 80
Kumar et al. 2017 Kim & Mnih 2018 Chen al., 2018). These models learn disentangled factors by factorizing aggregated posterior. They can also be used for C-S disentanglement, The learned factors be divided into two categories; one is content-related, the other is style-related. However, Locatello et al. (2019) proved that unsupervised disentanglement is impossible without introducing inductive bias both models and data. Therefore, these models currently unable to obtain a promising disentangled representation. Motivated by Locatello et al. (2019), we revisit and formulate the unsupervised C-S disentanglement problem introduce inductive bias.
C-S Disentanglement. Originated from style transfer, most of the prior works on C-S disentangle- ment divide latent variables into spaces relying on supervision. To achieve disentanglemment, Mathieu et al. (2016) and Szabó et al. (2018) combine the adversarial constraint and auto-encoders. Meanwhile, VAE (Kingma& & Welling 2014 is used with non-adversarial constraints, such cycle consistency (Jha et al. 2018) and evidence accumulation (Bouchacourt et al., 2018). Furthermore, latent optimization is shown to be superior to amortized inference (Gabbay & Hoshen 2020). Un- like the above works, WW et al. (2019c) propose a variational U-Net with structure learning for disentanglement an unsupervised manner. However, this method is limited by the learning of 2D landmarks. In our paper, we formulate C-S disentanglement and explore inductive bias for unsu- pervised disentanglement. Note that style transfer aims at modifying the domain style of an image while preserving its content, and its formulation focuses the relation between domains (Huang cet al. 2018a). Our formulation is defined in a single domain but can be extended to cross-domain, presented in Appendix G
2

---

Under review as a conference paper at ICLR 2021
3 EXPLORING INDUCTIVE BIAS FOR C-S DISENTANGLEMENT
In this section, We first formulate the C-S disentangle problem by exploring the inductive bias and propose C-S fusion block based on formulation. We then perform the ablation study to demonstrate how the C-S get disentangled. Finally, our loss functions are presented.
3.1 PROBLEM FORMULATION
We parametric the larget distribution (2 c) as Pg... (x/c), where 0 is the parameter of the generator Go that maps embeddings to images, 8; is the style embedding assigned to I,. In our formulation of P. we assign independent roles to content and style embeddings. { are sampled from the distribution of the shared conditional variable which is denoted as Vr { are the parameters to characterize P. Thus our inductive bias is introduced into formulation. V should be close to ground truth distribution of the dataset, e.g., Gaussian distribution, Uniform distribution, etc.
Our optimization target is to maximize the log-likelihood of P, and force c to follow the shared distribution v meanwhile:
Ba log Po.,, (xc T.|c ai)
8,04,84
8.L.
KL(p(c)|lV) 0,
where C = Gj indicates c; is the embedding assigned to 4;, and p(c) denotes the distribution of {ci) fim To solve this problem, introduce the Lagrange Multiplier as
min 0,01,88
Σ Bi,-, log Pa.,, (x= lilc= c.) + AKL(p(e)llV
3.2 PROPOSED NETWORK ARCHITECTURE
Here we propose a network architecture to address the formulated problem in Section. 3.1 In particular, design C-S fusion block assign different roles content and style in modeling real data distribution. As shown in Figure I we add this block before the generator to force the input follow the customized distribution.
Inspired by the observation that mean and variance of features carry the style information (Gatys) al., 2016 Li Wand 2016 et al. 2017; Huang & Belongie 2017), we use the style em- beddings to provide the statistics to scale and shift the shared distribution Y to match the target distribution as
24 (st) fs()..
(3)
where fo and fu are two fully connected layers predicting the mean and variance respectively. With this design, Eq. iis equivalent minimize
N Σ 1=1
min
14 Go(zi)l + XKL(p(c))
(4)
Please refer to Appendix G for the proof. To solve Eq, 4 for the reconstruction term, we adopt latent optimization to optimize 0, {cr (ea). { For the KL term, it can not be optimised directly. However, when V has some specific forms, we can adopt a normalization to force each of the content embeddings c follow it approximately.
We can choose different forms of to fit the ground truth distribution when solving the optimization problem, here we provide two examples:
Gaussian Distribution. Gaussian distribution is used to model the distribution of images in many works (Kingma& & Welling 2014 Kim & Mnih 2018 Higgins et al. 2017). When setting the shared distribution V as a zero mean, unit variance Gaussian distribution A (0, I), we can use instance nor- malization (IN) to force each of the content embeddings G, follow A 7) in the optimization
3

---

Under review as a conference paper at ICLR 2021
(a) Fuse
(b) Single w/o IN
(c) Single
(d) Multiple
Verification of our design for content and style. The topmost row and leftmost column pro- vide the content and style embeddings, respectively. A good disentanglement is that: horizontally, the style (identity) of the images maintain very well when the content (pose) varies, and vertically, the content (pose) of the images align very well when the style (identity) varies.
Figure 2:
process. By combining IN and Eq. 3] get the same formulation as AdaIN (Huang & Belongie,
2017], which is widely adopted in style transfer tasks (Huang et al. 20188 Kotovenko et al.) (0019).
Normalizing the feature map of the network to Gaussian helpful for network training (loffe &
Szegedy 20155 Wu & He, 2018), but our motivation for using normalization is to force the embed- dings share the same Gaussian distribution, which differs from these works.
Uniform Distribution. In many datasets, the distribution of content is close to uniform distribu- tion, e.g., in the Chairs (Aubry et al., (2014) dataset, the images are synthesized from dense views surrounding the objects. For these datasets, we set W to be uniform distribution, and normalize the content embeddings with Le normalization to force each of them to follow uniform distribution approximately (Muller 1959).
As shown Figure I we can use this C-S fusion block only before the generator, denoted as the Single C-S Fusion framework. We can also provide multiple paths to implement our design by inserting before every layer the generator, denoted as a Multiple C-S Fusion framework. For details of the network structures, please refer to Appendix
3.3 DEMYSTIFYING C-S DISENTANGLEMENT
In this subsection, we perform some experiments to verify that assigning different and independent roles to content and style for modeling real data distribution is the key to C-S disentanglement. The experimental setting can be found in Section
If we do not assign different roles, i.c.. concatenating content and style embedding as the input of the generator, the network can hardly disentangle any meaningful information for the CelebA dataset, shown in Figure 2 (a). Our Single C-S Fusion framework can disentangle the pose and identity of human faces, as shown in Figure (c). The content plays the role of modeling the shared distribution.
When the shared distribution constraint is removed, i.e., without normalization, the result is shown in Figure 2 (b), where the pose and identity can not be disentangled. For the Multiple C-S Fusion framework, multiple paths are provided, and the network has more flexibility to approximate the target distribution and outperforms the Single C-S Fusion framework, as shown in Figure (d).
Since the shared distribution is crucial, we experiment to demonstrate that better disentanglement can be achieved by choosing a better distribution to fit the dataset. For the real-world dataset CelebA, the distribution of pose better modeled as a Gaussian distribution. As Figure (a) and (b) show, IN achieves better disentanglement than L2. For the synthetic Chairs (Aubry et al.) (2044) dataset, distribution of pose close to uniform distribution rather than Gaussian distribution. Figure (c) and (d) show that the La normalization results in better identity and pose consistency.
Figure 3: Generated images at different training steps. The first and second rows share the same style embedding. The second and third rows share the same content embedding.
4

---

Under review as a conference paper at ICLR 2021
(a) IN
(b) L2 Normalization
(c) IN
(d) L2 Normalization
Figure 4: Comparison of the disentanglement with different normalizations. Instance Normalization (IN) achieves better result CelebA, e.g. the face identities are alike with the query. L2 normalization outperforms on Chairs, where the shapes of chairs are more consistent each row.
To better understand how our design helps to guide the disentanglement, we visualize the generated images during the training process in Figure As the generated images show, mean shape of faces first learned. Then the faces start to rotate, which indicates space. After that, the identity features emerge as the style starts to learn parameters for customizing the shared distribution to approximate the real faces distribution.
pose is disentangled to the content
3.4 LOss FUNCTION
Perceptual Loss, Perceptual Loss is widely used in weakly supervised and unsupervised meth-
ods (Wu al. 2020 Ren et al. 2020, Wu al.) 2019c). Gabbay & Hoshen (2020) claimed that perceptual loss is not extra supervision for disentanglement. We adopt a VGG (Simonyan & Zis-
serman 2015) perceptual loss Lp as a reconstruction loss Eq. implemented by Hoohen et al. (2019).
Instance Discrimination. Instance discrimination can automatically discover appearance similarity among semantic categories (Wu et al.) 2018). Inspired by this, we propose to use instance discrim- ination as a complementary constraint to enhance consistency among the images sharing the same style embeddings. We denote the instance discrimination loss LTD. The implementation detail can be found in Appendix C.3
Information Bottleneck. [Burgess al. (2018a) propose improving the disentanglement in 3-VAE by controlling the capacity increment, i.c., forcing the KL divergence to be a controllable value. This motivated us control the information bottleneck capacity of content and style help to avoid leakage. This loss is denoted as LIB. The details of this loss are provided in Appendix C.4
Our full objective is
wpLp winCin WinLin,
(5)
where hyperparameters wp, WIB, and WID represent the weights for each loss term respectively. The ablation study for the loss terms is presented in Appendix E
EXPERIMENTS
In this section, we perform quantitative and qualitative experiments to evaluate our method on seen data following common practice. We our method on several datasets: Car3D (Reed al., 2015).
Chairs (Aubry et al. 2014). CelebA (Liu et al. 2015). For details of the datasets, please refer to Appendix B
Baselines. Among all the prior works, we choose several state-of-the-art class-supervised C-S dis- entanglement benchmarks for comparisons: Cycle VAE (Tha et al. 2018), a variant of VAE using cycle-consistency: DrNet (Denton & Birodkar) 2017), adversarial approach; Lord (Gabbay & Hoshen] 2020), a latent optimization method. We also choose two unsupervised disentanglement methods: FactorVAE (Kim & Mnih| 2018), method that encourages the distribution of represen- tations to be factorial; Wu et al. (2019c) a two-branch VAE framework based on unsupervised structure learning. More details for baselines are presented in Appendix B
'There is open-sourced implementation for it. We modify https: thu.. thub. com/CompV's s vunet and provide pseudo ground truth landmarks the network. Thus it becomes semi-supervised.
5

---

Under review as a conference paper at ICLR 2021
Table 1: Performance comparison in content tranfer metric ( lower is better). For Wu et al. (019cc
provide pseudo facial landmarks and have suitable landmarks for cars and chairs.
Method
Supervision
Cars3l)
Chairs
CelebA
DrNet (Dntonn & Birodkar 2017
0. 0,146
0.294
0,221
Cycle-
VAE VAE (Jha al. 2018)
0. 148
0.240
0,202
Lord (Gabbay & Hoshen
2020
0.089
0.121
0,163
FactorVAE (Kim & Mnih
2018
0.190
0.287
0,369
Wu et al. (2019c)
0,185
Ours
0.082
0.190
0.161
(c)
(d)
(e)
(a)
(b)
Figure 5: Demonstrations of the latent spaces by interpolation (a & b) and retrieval (c-f).
4.1 QUANTITATIVE EXPERIMENTS
We compare method (Multiple C-S Fusion framework) with the baselines on Car3D, Chairs and CelebA.
Content Transfer Metric. To evaluate our method's disentanglement ability, we follow the protocol of Gabbay & Hoshen (2020) to measure the quality of content transfer by LPIPS Zhang et al. 2018). Details are presented in Appendix | The results are shown in Table [ We achieve the best performance among the unsupervised methods, even though pseudo label is provided for Wu al. 2019c). Furthermore, our method comparable to or even better than the supervised ones.
Classification Metric. Classification accuracy is used to evaluate disentanglement in the literature (Denton & Birodkar 2017 Tha et al.) 20188] Gabbay & Hoshen 2020). Following Tha et al. (2018), train two models of single fully-connected layer to classify content labels from style embed- dings and classify style labels from content embeddings. Low classification accuracy indicates that the leakage between content and style is small. Due to no content annotations of CelebA, we regress the position of the facial landmarks from the style embeddings. The results are summarized in Ta- method is comparable to several methods. We observe that
ble Though without supervision, the classification metric is also influenced by information capacity and dimensions of embeddings. For FactorVAE (Kim & Mnih, 2018). the poor reconstruction quality indicates that the latent em- beddings encode very small amount of information that can hardly be classified. The dimensions of the latent vectors of different methods vary from ten to hundreds. Actually, the higher dimension usually leads to easier classification. Based on the above observations, the classification metric may be appropriate for disentanglement, which also observed in Liu al. (220)).
4.2 QUALITATIVE EXPERIMENTS
Disentanglement & alignment. In Figure and (b), we conduct linear interpolation to show the variation in the two latent manifolds. Horizontally. the identity is changed smoothly with the inter- polated style latent space while maintaining the pose information. Vertically, the identity remains the same as the pose changes. These results illustrate the following points: 1) The learned content

---

Under review as a conference paper at ICLR 2021
Table 2: Performance comparison in classification metric (lower is better). For Wu al. (2019c) ) we provide pseudo ground truth landmarks. Note that the column (R((s) + c) presents the error of face landmark regression from the style codes (higher better).
Cars3D
Chairs
CelebA
Method
Supervision
8 -
8 4-
8 >
c
+ c
R(s) +
c
8 C
DrNet
0.27
0.03
0.06
0.01
4.99
0.00
Cycle- VAE Lord
/
0.81
0.77
0.60
0.01
2.80
0.12
0.03
0.09
0.02
0.01
4.42
0.01
Vac Wu et al. Ours
0.07
0.01
0.14
0.01
5.34
0.00
X
5,42
0.11
0.33
0.24
0.66
0.05
4.15
0.05
(a) MUNIT
(b) Park et al. (2020)
(c) Ours
(d) Our fine
Figure 6: Comparison with MUNIT (Huang et al.
2018b) and Park al. (020). MUNIT (Huang
ct al. 2018b) and Park et al. (2020) learn the texture information which is different from Ours (c).
Our fine (d) that we only exchange the fine styles.
and style spaces are continuous. Columns of the left and right figures share the same pose, sug- gesting that the learned content spaces are aligned. 3) Style-related information is maintained when changing the content embedding and vice versa, suggesting the good disentanglement.
We perform retrieval on the content and style latent spaces, respectively. As shown in Figure 5 (c) and (d), the nearest neighbors in the content space share the same pose but have different identities, which reveals the alignment on content space. To better identify the faces, we let the nearest neigh- bors the style space share the same pose, and the generated faces look very similar, revealing that the style is well maintained, As shown in Figure (f), one interesting observation is that zero content embeddings lead to a canonical view. As we assume that the pose distribution of faces is (O0, I), the canonical view is the most common pose in the dataset and sampled from the peak of this distribution. We also show the faces with zero style embeddings in Figure (e), and looks like the mean face of the dataset.
Visual Analogy & Comparison. Visual analogy (Reed et al. 2015) is to switch between style and content embeddings for each pair within testset. We show the visual analogy results of our method against Factor VAE (Kim & Mnih 2018) (typical unsupervised baseline) and Lord (strongest super- vised baseline) in Figure 8 on Chairs, Car3D, and CelebA. The results of FactorVAE on all datasets have poor reconstruction quality and bad content transfer. On Cars3D, Lord has artifacts (e.g.. third column) and could not capture the color style of the test images (e.g., fourth row). On CelebA, the transfer result of Lord has ambiguity, e.g., the content embedding controls facial expression in the fifth column, while other content embeddings do not control expression. Our method achieves com- parable pose transfer to Lord and maintains the identity of the images. For more results (including other datasets), please refer to Appendix D
Comparsion with Image Translation. Starting with our assumption that content embeddings share the same distribution, and leveraging C-S fusion block, we achieve unsupervised content and style disentanglement without needing "swapping" operation and GAN loss constraint to extract the shared content information as image translation works (MUNIT (Huang et al., (0185) and Park et al. (2200) do. As shown in Figure for MUNIT (Huang et al. 2018h) and Park al. (2020), the
7

---

Under review as a conference paper at ICLR 2021
6
Figure 7: Inference for unseen images. Our method performs well on images from different do- mains: painting and cartoon.
content is low-level structure information. While in our case, the content is high-level semantic attribute the object, e.g., the pose attribute. As shown in Figure 6 (d), we can also achieve the similar performance to exchange the tone of the images by exchanging the fine style. 'he fine styles our method are the style inputs of the last C-S fusion block in the multiple C-S fusion framework.
4.3 UNSEEN IMAGES INFERENCE
Though we learn to disentangle in an unsupervised manner, we may need to process unseen images. An intuitive solution is train encoders to encode images to the latent spaces. We train style encoder E, and content encoder Eg by minimizing
LE [E. s;|li [..)
(6)
We apply our model trained on the CelebA dataset to faces collected by Wu et al. (2020) including paintings and cartoon drawings. As shown in Figure method can be well generalized unseen images from different domains.
5 NEW APPLICATION
In this work, we explore a new application of C-S disentanglement. For 3D reconstruction, single- view settings lack reliable 3D constraints, which can cause unresolvable ambiguities (Wu et al, 2019a). Thanks to our disentangled representations, we can generate multi-view images from single view by extracting the style embedding of the single view and then combining with multiple content embeddings. On Chairs, we adopt Pix2Vox (Xie et al.) 2019), framework for single-view, and multi-view 3D reconstruction to verify the advantages of our method. As shown in Figure / the 3D objects reconstructed from multi-view inputs generated from our method much better than those reconstructed from single view, and even comparable to those reconstructed from ground- truth multi-view images. For results on Celeba, please refer to Appendix D.3
CONCLUSION
We present unsupervised C-S disentanglement method, based on an inductive bias: assigning dif- ferent and independent roles to content and style when approximating the real data distributions. Our method outperforms other unsupervised approaches and achieves comparable formance than the state-of-the-ar supervised methods. We also propose to use it to help single-view 3D reconstruction,
even better per-
application of C-S disentanglement. As for the limitation, we fail
8

---

Under review as a conference paper at ICLR 2021
Fac
Lord
Ours
Figure 8: Comparison of visual analogy results on Chairs, Car3D and CelebA (from top to bottom).
Input
Our generated multi-view
Single
Ours
GT
Figure 9: 3D reconstruction results on Chairs. We generate multi-view from Input. Single: the object reconstructed by only Input. Ours: the object reconstructed by multi-view inputs. GT: the object reconstructed by the ground truth of multi-view inputs.
datasets containing multiple categories with large appearance variation, e.g., CIFAR-10 (Krizhevsky al. 2009), which does not match our assumption. Our method could be adopted help down- stream tasks, e.g., domain translation, RelD, etc. An interesting direction to apply our method instance discrimination. With disentangled representations, contrastive learning is expected to perform more effectively.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mathieu Aubry, Daniel Maturana, Alexei A. Efros, Bryan C. Russell, and Josef Sivic. Seeing 3d chairs: Exemplar part-based 2d-3d alignment using a large dataset of CAD models. In CVPR, 2014.
Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. Optimizing the latent space of generative networks. In JCML, 2018.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In AAAI, 2018.
Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In 2017,
Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 8-vae. CoRR, 100.03994,,,9, 8a.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in beta-vae. arXiv preprint arXiv:1804.03599, 2018b.
Brian Chao. A collection of high-quality anime faces.
Anime Sace-Daaaset) 2019.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. In NeurPIS, 2018.
Emily L. Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In NeurIPS, 2017.
Aviv Gabbay and Yedid Hoshen. Demystifying inter-class disentanglement. ICLR, 2020.
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In CVPR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basie visual concepts with constrained variational framework. In ICLR, 2017.
Yedid Hoshen, Ke Li, and Jitendra Malik. Non-adversarial image synthesis with generative latent
nearest neighbors. In CVPR, 2019.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, 2017.
Xun Huang, Ming- Yu Liu, Serge J. Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In ECCV, 2018a.
Xun Huang, Ming-Yu Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In ECCV, 2018b.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In /CML, 2015.
Ananya Harsh Jha, Saket Anand, Maneesh Singh, and V. S. R. Veeravasarapu. Disentangling factors of variation with cycle-consistent variational auto-encoders. In ECCV, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
10

---

Under review as a conference paper at ICLR 2021
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In CVPR, 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In JCML, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Dmytro Kotovenko, Artsiom Sanakoyeu, Sabine Lang, and Bjorn Ommer. Content and style disen- tanglement for artistic style transfer. In ICCV, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference disentan- gled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.0084 2017.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: htlp://yann.eecue wttcuu.ccm//xd////iisss.. 2010.
Chuan Li and Michael Wand. Combining markov random fields and convolutional neural networks for image synthesis. In CVPR, 2016.
Yanghao Li, Naiyan Wang, Jiaying Liu, and Xiaodi Hou. Demystifying neural style transfer. In IJCAI, 2017.
Yu-Jhe Li, Ci-Siang Lin, Yan-Bo Lin, and Yu-Chiang Frank Wang. identification via unsupervised pose disentanglement and adaptation. In ICCV, 2019.
Cross-dataset person re-
Xiao Liu, Spyridon Thermos, Gabriele Valvano, Agisilaos Chartsias, Alison O'Neil, and Sotirios A. Tsaftaris. Metrics for exposing the biases of content-style disentanglement. CoRR, abs/2008. 10008.123777 12378, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learn- ing of disentangled representations. In ICML, 2019.
Michaël Mathieu, Junbo Jake Zhao, Pablo Sprechmann, Aditya Ramesh, and Yann LeCun. Disen- tangling factors of variation in deep representation using adversarial training. In NeurlPS, 2016.
Mervin E Muller. A note method for generating points uniformly n-dimensional spheres. Communications of the ACM, 19-2: 1959.
Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexci A. Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. NeurIPS, 2020.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. Jawahar. Cats and dogs. CVPR, 2012.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Scott E. Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-makinn NeurlPS, 2015.
Xuanchi Ren, Haoran Li, Zijian Huang, and Qifeng Chen. Self-supervised dance video synthesis conditioned on music. ACM MM, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
11

---

Under review as a conference paper at ICLR 2021
Attila Szabó, Qiyang Hu, Tiziano Portenier, Matthias Zwicker, and Paolo Favaro. Challenges in disentangling independent factors of variation. In JCLRW, 2018.
Fanzi Wu, Linchao Bao, Yajing Chen, Yonggen Ling, Yibing Song, Songnan Li. King Ngi Ngan, and Wei Liu. Mvf-net: Mulwi-view 3d face morphable model regression. In CVPR, 2019a.
Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. Unsupervised learning of probably sym- metric deformable 3d objects from images in the wild. In CVPR, 2020.
Wayne Wu, Kaidi Cao, Cheng Li, Chen Qian, and Chen Change Loy. Transgaga: Geometry-awan unsupervised image-to-image translation. CVPR, 2019b.
Wayne Wu, Kaidi Cao, Cheng Li, Chen Qian, and Chen Change Loy. Disentangling content and style via unsupervised geometry distillation. In ICLRW, 2019c.
Yuxin Wu and Kaiming He. Group normalization. In 2018.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, 2018.
Haozhe Xie, Hongxun Yao, Xiaoshuai Sun, Shangchen Zhou, and Shengping Zhang. Pix2vox: Context-aware 3d reconstruction from single and multi-view images. In ICCV, 2019.
Xianglei Xing, Tian Han, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Unsupervised disentan-
gling of appearance and geometry by deformable generator network. In CVPR, 2019.
Mang Ye, Xu Zhang, Pong C. Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In CVPR, 2019.
Aron Yu and Kristen Grauman. Fine-grained visual comparisons with local learning. In CVPR, 2014.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Weiwei Zhang, Jian Sun, and Xiaoou Tang. Cat head detection how to effectively exploit shape and texture features. In ECCV, 2008.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In ICCV, 2015.
Yang Zou, Xiaodong Yang, Zhiding Yu, BVK Kumar, and Jan Kautz. Joint disentangling and adaptation for cross-domain person re-identification. In ECCV, 2020.
A IMPLEMENTATION DETAILS
All the models are trained on an Nvidia Tesla V100 GPU. The model is implemented in Py- Torch (Paszke et al. (017). For Car3D, Chairs, and Celeba, we set the size of the style embedding d, to 256, the size of the content embedding de to 128, and the epoch to 200. All the images are resized to 64 64. For the hyperparameters, wp 5, wrB = 1, WTD = 1. We used Adam (Kingmal Ba 2015 with learning rate 0.0003 for the models and 0.003 for the latent spaces.
A. EXPERIMENTAL DETAILS
Content Transfer Metric. For Car3D and Chairs datasets, the content labels (ground truth) are available. For images 1; and Iú sampled from the testing set randomly, we compute LPIPS between Go(cs, 8,) and the corresponding ground truth from the same class of Iy. For CelebA, we randomly sample two images I, and I, with the same identity, and we retrieve an image that has the most similar pose from the test set, i.c., the nearest neighbor the 68 facial-land landmarks space. We measure the similarity between I3 and Go (c),
12

---

Under review as a conference paper at ICLR 2021
A.2 NETWORK STRUCTURE
Our Single C-S Fusion framework and Multiple C-S Fusion framework are shown in Figure T0 For the details of the reparametric module R. please refer to Appendix C.2
Repanastric
32
Generator
FC
Space
Convxxx
Upsrmpiig
Upsampling
Upsampling
Cone x
Con x
Togurezatic
C- Fusion Block
(a) Single C-S Fusion framework
Spece
Repircattis kcdack
Generator
Contant Lakast Space
Cone Convxx333
C. S Fusion
Uptampling
C. Fusion
Upsampling
C. Fusion
Upsammplin
Core
Fusion
Normalzzaioon
FC
FC
C-5 Fusion Block
(b) Multiple C-S Fusion framework
Figure 10: Details of network structure. For every upsampling layer, there is 3 3 convolutional layer following it. The embedding 81 and e₁ from style and content embedding space respectively first processed by the reparametric model then fed into the network in different ways. Then the image 1 is generated. The style and content latent space and network are jointly optimized under the supervision of the reconstruction loss between synthesized image f, and ground truth image I1 from the dataset.
B BASELINE DETAILS
For the datasets in the main paper, Car3D contains 183 car models, each rendered from 96 poses. Chairs consists of 1393 chair models, each rendered from 62 poses. CelebA contains 202,599 facial images of 10,177 celebrities.
For the baselines,
Dr- Net Denton & Birodkar 2017) Lord (Gabbay & Hoshen 2020 and Factor VAE Kim & Mnih 2018
open-sourced implementations for Cycle VAE (haa al.
2018
For Factor VAE, we traverse the latent space to select the dimensions related to pose as content em- bedding and treat the other dimensions style embedding. For Wu al. (2019c). there is no open- sourced implementation. We use the code from httpp::/tttttttttt com wwwom////////////ooooooo /vunet
13

---

Under review as a conference paper at ICLR 2021
which uses ground truth landmarks input instead of learning the landmarks unsupervisedly. To achieve the pseudo ground truth landmarks, we use the face detection library (Bulat & Tzimiropou- los 2017) for Celeba. We try to use the LI and perceptual loss for all the baselines and select the best.
We split the datasets into training and testing sets. For Celeba, randomly select 1000 among 10177 celebrities for testing. For Car3D, we randomly select 20 among 183 CAD models for testing. For Chairs, we randomly select 100 among 1393 models for testing. For baselines with group supervision, only the training sets are used for training. For unsupervised baselines and our method, all the datasets are used for training.
C TECHNICAL COMPONENTS
Here present three technical components that are helpful to the C-S disentanglement. The abla- tion study for these components is shown in Appendix
C.I LATENT OPTIMIZATION.
In the C-S disentanglement literature, common to use encoders to predict embeddings, while latent optimization (Bojanowski et al.] 2018 Gabbay & Hoshen (020) directly optimizes the em- beddings via back- proookgprppannngg without using encoders. Encoders have a large number of parameters and require lot more extra effort for training. Therefore, We adopt the latent optimization approach to update the latent spaces directly.
C.2 REPARAMETRIC MODULE
Inspired by VAE (Kingma & Welling 2014). we design reparametric module force the latent space to be continuous. Thus, the embeddings encoding similar information will get closer in the latent space. Assume we have mean embedding i with a standard deviation a, the reparametrized output is o. μ., where X ~ NO, I). To further simplify the problem, we set σ = 1 following Wu et al. (2019c) and Gabbay & Hoshen (2020). The mean embedding is the input style or content embedding. The reparametric module can make the latent space continuous, which is helpful for backpropagation. Though the training images have discrete identities, the optimized style embed- ding space is continuous. Different style embeddings of the people with similar appearances close to each other, as shown in Figure (c).
C.3 INSTANCE DISCRIMINATION LOSS
We first pretrain ResNet- 18 (He al. 2016 and define a collection of layers & }- Among several representative methods (Wu et al. 2018 Ye et al. 2019 He et al. 2020), we observe that the method in Wu al. (2018) achieves the best performance our task. Given two images I; and Ij. we mix the embeddings to generate u G(R(s;) R(c;)) and v G(R(s,), R(c;)). For samples sharing the same style embedding, we enforce the feature distance between them be close. This loss term can be written
CID = Σ Au (1) +1(x)]11 ( (1))).
where x = G(R(s)). R(c)) and y = G(R(s,). ((c))). The hyperparameters (Ar balance the contribution of each layer I to the loss. {Ni) are be [1, 1,
C.4 INFORMATION BOTTLENECK
Similar to Anneal VAE (Burgess et al. 2018a). we introduce a information bottleneck given by
LIB = 7 - Calli + rallc² -Calla
where C. and C. the information capacity controlling the amount of information of the content and style respectively. During training. C, and C, increase linearly. The rate increase controlled by the increase steps and the maximum value. By controlling the increase rate, the content is forced
14

---

Under review as a conference paper at ICLR 2021
to encode information first, that the learning process is more consistent with our assumptions about the data: the shared conditional variable is learned first.
For the information bottleneck, by taking the training process of the model without the information bottleneck as reference, we determine the increase steps and the maximum of the information capacity Ce and C.. We enhance the model inductive bias by tuning these parameters. For Chairs, we set the maximum 1.4 10 Ve and % to O. Note that our model achieves state-of-the-art performance Chairs even without information bottleneck.
C. to 5, the start value of C. to 2, the increase steps of C. to
D MORE RESULTS
In this section, we demonstrate more qualitative comparison and more qualitative results (including more datasets).
D.I MORE QUALITATIVE EXPERIMENTS
In the main paper, for unsupervised baselines, we only compare our method with FactorVAE (Kim & Mnih 2018) limited to space. As shown in Figure [T] we also outperform Wu et al. (2019c). For Wu et al. (2019c), the disentanglement poor, such that the content embeddings control almost all the factors while the style embeddings control the tone.
For datsets in the main paper, We provide more qualitative results in Fig. Moreover, we also apply our method on higher resolution images and achieve good performance, as shown in Figure 20
26 and 28
D.2 MORE DATASETS
Besides the datasets introduced in the main paper, we make additional experiments on other datasets:
such MNIST (LeCun al. 2010). Cat (Parkhi et al. 2012 Zhang al. 2008), Anime (Chao
2019) and Market-1501 (Zheng et al. 2015). MNIST has 70k examples for 10 handwritten digits.
Cat has 1.2k cat head images. Anime contains 63.632 anime faces. Market-1501 have 25,259 images. The results are shown in Figure 21 22 23 Furthermore, we show our results on the
Market- 1501 dataset in Figure 19. which demonstrates our method can disentangle the human pose and the appearance even though the skeletons have large variances.
60
60
Wu et al.
Our
Figure 11: Comparison between Wu et al. and our method. For Wu et al., the images are mainly determined by content embeddings, while style embeddings only change the tone.
15

---

Under review as a conference paper at ICLR 2021
(a) Amortized
(b) w/o R
(c) Ours
Figure 12: Ablation study. R indicates reparametric module.
D.3 MORE 3D RECONSTRUCTION
Our setting treats every image single identity (style) without ambiguity for augmenting single- view images. On Celeba, We use MVF-Net (Wu et al. 2019a) based on multi-view to reconstruct 3D) facial shapes. For given image, can get the corresponding style embedding content embedding. Then we can get the front, left, and right view of this image combining the extracted style embedding and prepared content embeddings As shown in Figure 13 our augmented multi-view images consistent, and the 3D meshes based on our method are more accurate than those based on Lord.
E
MORE ABLATION STUDY
Here perform more ablation study for the technical modules.
we use an amortized scheme instead of latent optimization scheme, there are leaks between style and content latent space, and the result is worse than latent optimization, as shown in Figure T2 (a) and (c). Furthermore, we do not use a reparametric module, we find the reconstruction perfor- mance is worse, shown in Figure 12 (b). For the instance discrimination loss, the comparison is shown in Table 4 The disentanglement better with an instance discrimination loss. For the in- formation bottleneck, shown in Table 3 the result with information bottleneck is much better than the one without it.
Table 3: Ablation study for infomation bottleneck on Chairs dataset. Lower is better.
Method
Content transfer
Ours (w/o Information Bottleneck) Ours
0.280
0.190
Table 4: Ablation study for instance discrimination on Celeba dataset. Lower is better.
Method
Content transfer
Ours (w/o Instance Discrimination) Ours
0.165
0.161
We retrieve the nearest neighborhood (facial landmarks space) of suggested inputs of MVF-Net and extract content emhedding.
16

---

Under review as a conference paper at ICLR 2021
Ours
Input
Lord
Figure 13: 3D face reconstruction. Given an image, we first generate multi-view images and then use them as augmented input.
(a) MUNIT
(b) Park et al. (2020)
(c) Ours
(d) Our fine
Figure 14:
Comparison with MUNIT (Huang et al. 20188) and Park et al. (2020) MUNIT (Huang
al., 2018b) and Park et al. (2020) learn the texture information which different from Ours (c).
Our fine (d) is that we only exchange the fine styles.
F COMPARISON WITH SELECTED RELATED WORK
Comparison with StyleGAN. our framework, the optimized content (conv) and style embeddings are disentangled representations of corresponding images. While StyleGAN (Karras et al.) (219)) keeps the input of the convolution branch as learnt constant for whole dataset and finds the feature space of the "style" branch has disentanglement ability. For StyleGAN2 (Karras et al.) 2020 D we select the subset of "style", which represents pose, as the content embedding and the rest subset the style embedding. As shown in Figure 15 StyleGAN2 entangled pose with other semantic attributes, such as hair and glasses. As shown in Figure 23 the content of our method on human faces pose attribute without entanglement.
Comparsion with MUNIT & Park et al. (2020). Starting with our assumption that content embed- dings share the same distribution, and leveraging AdalN-like operation, achieve unsupervised content and style disentanglement without needing "swapping" operation and GAN loss constraint extract the shared content information image translation works (MUNIT (Huang al. 2018h) and Park et al. (2020)) do. As shown in Figure 14 for MUNIT (Huang et al.) 2018b) and Park et al. (2020), the content low-level structure information. While our case, the content high-level semantic attribute of the object, e.g., the pose attribute. As shown in Figure 14 (d), we can also achieve the similar performance exchange the tone of the images by exchanging the fine style.
Wee the implementation from hftm://tcgrggggttttoccc////aaaaa htgp:ityyyyyyytttttttn
17

---

Under review as a conference paper at ICLR 2021
StyleGAN2
Figure 15: Performance of StyleGAN2 Karras et al. 2020 on human faces. For StyleGAN2, the content contains entangled semantic attribute, such as pose, hair and glasses. In our case, the content is pose, which is high-level semantic attribute of the object.
Figure 16: Examples of translating shoes to edge (left column) and translating edges to shoes (right column). Triplet order (left right) is: content, style, translation.
The fine styles in our method are the style inputs of the last C-S fusion block in the multiple C-S fusion framework.
G CROSS-DOMAIN APPLICATION
As shown in the main paper, the content and style are disentangled single domain. Based on our assumption, the cross-domain dataset also can be disentangled. In this section, we test our model on cross-domain dataset to further verify our assumption. some cases that we merge images from two domains, our method can still work and achieve performance, which is similar to domain translation. For example, Edges2Shoes YYu & Grauman] 2014) a dataset consisting of 50k paired shoe and edge map images. As shown in Figure 16, the content is edge structure, and the style is texture. Thanks to this, we can translate edge images into shoe images and vice versa without any additional operation.
18

---

Under review as a conference paper at ICLR 2021
R
3
8
2
Figure 17: Results of modified model on merged cross-domain dataset based on Celeba and Anime. The learned content embedding are well aligned both in the animation and reality domain.
Furthermore, once the domain labels are given, we can disentangle and align the cross-domant dataset. This experiment may be helpful for domain transfer and domain adaptation. We train model on the dataset that consists of Celeba and Anime. The model needs to be modified for learning cross-domain data: concatenate the domain embedding and the style embedding, take as the style embedding in the original model, and optimize the domain embedding during latent optimization. The learned poses are well aligned both in the animation and
The results are shown in Figure 17 reality domain.
19

---

Under review as a conference paper at ICLR 2021
(a)
(b)
(c)
Figure 18: Study on the influence of size of embeddings. For (a), we set the size of the style embedding d, to be 128, the size of the content embedding de to be 256. The content embeddings contain shape of face, facial expression and pose. For (b), dg = 256 and de = 256, and the content embeddings contain shape of face and facial expression. For (c), d, = 256 and de 128, which is the setting used our paper, the content embeddings contain pose.
20

---

Under review as a conference paper at ICLR 2021
Lord
FactorVAE
Ours
Figure 19: Comparison of visual analogy results on Market- 1501 dataset. Our method outperforms supervised method Lord Gabbay & Hoshen (2020) and unsupervised method FactorVAE. Kim & Mnih (0018) significantly.
21

---

Under review as a conference paper at ICLR 2021
Figure 20: Results on Celeba with 128 128 resolution. Zoom better view.
2
4
5
4
6
7
9
1141
22
5
6
8
7
9
8
2
8
4
8f
8
7
9
4
42844
8
1
9
5284
5
4
7
6
8
7
9
8 4 5 4
9 6
8
7
9
Figure 21: Results the MNIST dataset. Content indicates geometric attributes, and style indicates texture.
22

---

Under review as a conference paper at ICLR 2021
Figure 22: Results on the Cat dataset. Content indicates pose, and style indicates identity. The result
further qualitatively demonstrates the ability of our method to disentangle on real-world data.
3
5
Figure 23: Results on the Anime dataset. Content indicates pose, and style indicates identity.
23

---

Under review as a conference paper at ICLR 2021
Figure 24: More visual anology of our method on Car3D.
24

---

Under review as a conference paper at ICLR 2021
Figure 25: More visual anology of our method on Car3D.
25

---

Under review as a conference paper at ICLR 2021
Figure 26: More visual anology of our method on Chairs.
26

---

Under review as a conference paper at ICLR 2021
It E 
Figure 27: More visual anology of our method on Chairs.
27

---

Under review as a conference paper at ICLR 2021
2
Figure 28: More visual anology of our method on Celeba.
28

---

Under review as a conference paper at ICLR 2021
H PROOF
Our optimization target is to minimize the KL divergence between P and Q,
min
KI(R(x]c= c2 c)).
8,
Expanding the above KL term, we have,
p.(x(c=c.) ci) dr. Q0.a. ci)
min
Pa(ace= c)) log
(10)
The above integral equation cannot be directly calculated, but can be estimated by the sampled images {I,} P..
E P( 1 a) log P( R(==1)le=c)
Σ
(11)
0,01
Qa, ( 1=
Separating and Q from the above equation by logarithmic transformation, we have
N
Σ Pi(x = 1//c = c.) log Pi 1 = ci) L
mil
0,01,8
(12)
1=1
P1(x=11llc-- cj) ) log Qo.,, (x 10:==yyc=- c;)=
Since Pa = Tyle c)) is the dataset distribution, which is an unknown constant distribution, therefore, the first term is constant, the optimization target is equivalent to
Σ P((z I,lc l,g Q0.,, lylc= c.).
(13)
0.00
Rewriting it into mathematical expectation form, we have
N
max 0.G..
Ei log Q0.,, = lylc= c)),
(14)
where P; refers to Pa T c.). Our optimization target is equivalent to maximum likelihood estimation. Here we assume Q is Gaussian distribution,
1
Q8,a, (x|c ci)
V2TO
exp
202
|| Ce(ss,c)))3
(15)
Combining Eq. 14 and Eq. 15 have
1
max 0,CL,8
Σ
2 14.
Go(s..))))
(16)
Consequently, the final optimization target is
min
Σ 144 (2)2))13
(17)
0,04.8
Q.E.D.
29


Reviews:
[{"review_id": "sUKXcTf0aPE", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, authors introduce a new approach to content-style (C-S) disentanglement for multimodal unsupervised image-to-image translation. The main idea behind the proposed method is that the content information is encoded into the latent space common for both source and target domains, while the domain-specific style information is used to shift and scale the points from content distribution so they are mapped by the generator to the target domain distribution.\n\nThere are a few issues in this paper that I would like to be addressed.\n\n1. In the first page of the paper you write: \"When group observation is not available, we define content includes the factors shared across the whole dataset, such as pose.  Take the human face dataset CelebA (Liu et al., 2015) as an example, the content encodes pose, and style encodes identity, and multi-views of the same identity have the same style embeddings, but different content embeddings, i.e., poses\". In this paper, authors view only pose information of the faces in CelebA dataset as content; but according to the definition above, shouldn't facial expression also be included in the content as shared information across all examples?\n\n2. The proposed method suggests using style embedding to shift the content embedding before translation in Single case and within translation in Multiple case. How to address the fact that the same idea was used in a few other papers for multimodal cross-domain translation, such as MUNIT by Huang et al. ECCV'2018 or FUNIT by Liu et al. ICCV'2019?\n\n3. In addition to the main C-S fusion block, two more losses were introduced in this method: Instance Discrimination (ID) and Information Bottleneck (IB). To see the effect of each component, it would be helpful to see the ablation study results.\n\nOn the other hand, the paper is well-written and well-structured ans easy to follow; the translation results look promising. In addition, the quantitative metrics used in this paper, in particular the content transfer metric, are very reasonable for evaluation of disentanglement quality.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZIBydlj58CP", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the problem of unsupervised content style disentanglement. To this end, a content vector and a style vector is sampled from a given prior distribution. The style vector is decomposed to mean and std which is applied on the content code in the same manner as in AdaIN (C-S Fusion block). \n\nPros:\n\nThe paper is well written and clear, providing a clear formulation of the problem and the proposed architecture. \n\nThe method provided is well motivated and clear.\n\nThe experiments demonstrate some improvement on state of the art. Some ablation is performed on the use of normalization layer, showing its effect on disentanglement. A variety of datasets are considered, and both generation quality (LPIPS) and disentanglement (classification accuracy) are numerically evaluated. Overall the experimental evaluation is extensive and show some improvement over a number of baselines as well as a new application in 3D generation. \n\nCons:\n\nTo me the C-S fusion block is essentially the application of AdaIn from the style vector to the content vector (hence applying a change of style). Modeling the content vector as a shared parameter c and the style vector as an image-specific has been introduced in MUNIT before (and in other works). Other than architectural specific choices, the difference to MUNIT is the application of this framework in conjunction with GLO instead of using and style encoder and a content encoder. Even though MUNIT was designed to work with class level supervision, the method suggested seems very similar (other then the use of GLO instead of encoders) and it might be the case that MUNIT would therefore perform similarly. I therefore believe a comparison to MUNIT when both A and B are the same dataset (e.g celebA) would provide a better insight.\n\nImage2StyleGAN [1] and StyleGANv2 [2] are also able to disentangle content and style. It would be interesting to compare their method to the one suggested. The very latest work tackling this problem is that of Swapping Autoencoders [3] but this has only been recently published. \n\nSome concerns regarding the experiments: In table 2, the results of disentanglement are worse that that FactorVAE. Why is this the case? In addition, no comparison to baselines is provided for the Figure 7 - visual analogies on the Market-1501 dataset.\n\n[1] Image2stylegan: How to embed images into the stylegan latent space? In: IEEE International Conference on Computer Vision (ICCV) (2019)\n\n[2] Analyzing and Improving the Image Quality of StyleGAN. Karras et al.,\n\n[3] Swapping Autoencoder for Deep Image Manipulation. NeurIPS 2020. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "T8-T10YRgDT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper presents a novel combination of losses to learn representations that disentangle content from style. An ablation study is done with a few variants of the model, and extensive quantitative and qualitative experimental results on content/style disentangling demonstrate the model’s performance. \n\nOverall the experimental results look very nice and seem to compare well to other methods both  qualitatively and quantitatively. However, the scope of the contribution seems limited since the losses have already been introduced in the literature. The novelty seems to be in the embedding combination method, which is not explored as thoroughly as I’d expect. Also, the grounding of the proposed loss as a probabilistic model seems lacking to me.\n\nStrengths:\n* Impressive looking results on style transfer in a few distinct domains, including faces (celeba), chairs, and Car3D.\n* Simple method that could in principle be applied to a variety of domains.\n\nWeaknesses:\n* This paper would be stronger if it dispensed with the incorrect formalism introduced in Section 3, and introduced the training objective as a set of standard loss functions as in Eq. 4.\n* If this paper is about methods for combining embeddings for content/style recombination, I would hope to see a more thorough exploration of the different options. For example,  Eq. 2 is close to a bilinear model, as used in [4]. It would be interesting to see this combination method compared to explicitly, as it is a well known method for linear content/style recombination. It seems like only two methods (concatenation and the proposed method) were tried.\n* The proposed method seems to be an ad-hoc combination of losses already introduced in the literature. \n* “most previous C-S disentanglement works rely on supervision, which is hard to obtain for real data” --- full supervision of both content and style is difficult to obtain, but supervision of one variable alone is extremely common and represents the basis for most modern content/style decomposition methods. For example, see generic content/style decomposition models based on GANs [1, 3], and VAEs [2].\n* Denton & Birodkar 2017 does not use full supervision of content and style, rather they assume that the static components in the video sequence represent content, and everything else represents style.\n* Page 1 Paragraph 3: This definition of content --- as  factors shared across the whole dataset --- doesn’t make sense to me. Each face image has both an identity and a pose, so why is one content and not the other? It seems to me that the distinction is somewhat arbitrary, and probably determined by a target downstream task (e.g., face classification).\n\nClarity:\n* Section 3.1: This needs a more complete explanation. A few detailed questions follow:\n** Section 3.1: Is Q conditioned on s as well as c?\n** Equation 1: Since P(x|c) is NOT conditioned on s, wouldn’t this objective function encourage Q to be independent of s, i.e. Q(x|c,s) = Q(x | c) ?\n** Where is \\Psi in Eq. 1 ?\n* Eq. 3 - I don’t follow how this is optimizing KL( P(x | c) || Q_{s, \\theta}(x | c ) ). Please give a derivation.\n* Eq. 2+3 s_i, c_i are free parameters that the network is allowed to optimize, so how are  images not in the training set dealt with? Does it require training a new s_i, c_i for each new sample?\n* Which loss is Eq. 3 part of? I don’t see it listed in Eq. 4. \n\nRating:\nReferences:\n[1] Xun Huang,  Ming-Yu Liu,  Serge Belongie,  and Jan Kautz.   Multimodal unsupervised image-to-image translation. InECCV, 2018.\n[2] Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervisedlearning with deep generative models.  InAdvances in Neural Information Processing Systems,pp. 3581–3589, 2014\n[3] Michael F Mathieu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and377Yann LeCun. Disentangling factors of variation in deep representation using adversarial train-378ing. InAdvances in Neural Information Processing Systems, pp. 5040–5048, 2016\n[4] Tenenbaum, J. B., & Freeman, W. T. (2000). Separating style and content with bilinear models. Neural computation, 12(6), 1247-1283.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach to defining/tackling the question of separating "style" and "content" of images, and introduces a novel way to learn representation that disentangle these aspects of images. I think it offers some new ideas. The reviewers were split on the evaluation. Among the chief concerns with the initial submission were a problematic formulation of the objective, missing comparisons and analysis, and questions about novelty of the architecture (in particular w.r.t. AdaIn). I think the rebuttal/revision have addressed these fairly well. I do agree with R2 that some flaws remain, in particular the analysis could be more thorough/complete, and the paper could then be stronger. 
Decision: Reject