Title: UNSUPERVISED DISENTANGLING STRUCTURE AND APPEARANCE

Abstract: It is challenging to disentangle an object into two orthogonal spaces of structure and appearance since each can influence the visual observation in a different and unpredictable way. It is rare for one to have access to a large number of data to help separate the influences. In this paper, we present a novel framework to learn this disentangled representation in a completely unsupervised manner. We address this problem in a two-branch Variational Autoencoder framework. For the structure branch, we project the latent factor into a soft structured point tensor and constrain it with losses derived from prior knowledge. This encourages the branch to distill geometry information. Another branch learns the complementary appearance information. The two branches form an effective framework that can disentangle object's structure-appearance representation without any human annotation. We evaluate our approach on four image datasets, on which we demonstrate the superior disentanglement and visual analogy quality both in synthesized and real-world data. We are able to generate photo-realistic images with 256 × 256 resolution that are clearly disentangled in structure and appearance.

1. INTRODUCTION:

Structure and appearance are the two most inherent attributes that characterize an object visually. Computer vision researchers have devoted decades of efforts to understand object structure and extract features that are invariant to geometry change (Huang et al., 2007; Thewlis et al., 2017; Rocco et al., 2018) . Learning such disentangled deep representation for visual objects is an important topic in deep learning.

Figure 1 : Walking in the disentangled representation space: Here we show an example learned by our algorithm. Our approach effectively disentangles the structure and appearance space. Three cat faces in the bounding box are from real data while others are interpolated through our learned representations.

The main objective of our work is to disentangle object's appearance and structure in an unsupervised manner. Achieving this goal is non-trivial due to three reasons: 1) Without supervision, we can hardly guarantee the separation of different representations in the latent space. 2) Although some methods like InfoGAN (Chen et al., 2016) are capable of learning several groups of independent attributes from objects, attributes from these unsupervised frameworks are uninterpretable since we cannot pinpoint which portion of the disentangled representation is related to the structure and which to the appearance. 3) Learning structure from a set of natural real-world images is difficult.

To overcome the aforementioned challenges, we propose a novel two-branch Variational Autoencoder (VAE) framework, of which the structure branch aims to discover meaningful structural points to represent the object geometry, while the other appearance branch learns the complementary appearance representation. The settings of these two branches are asymmetric. For the structure branch, we add a layer-wise softmax operator to the last layer. This can be seen as a projection of a latent structure to a soft structured point tensor space. Specifically designed prior losses are used to constrain the structured point tensors so that the discovered points have high repeatability across images yet distributed uniformly to cover different parts of the object. To encourage the framework to learn a disentangled yet complementary representation of both appearance and structure, we introduce a Kullback-Leibler (KL) divergence loss and skip-connections design to the framework.

Extensive experiments demonstrate the effectiveness of the proposed method in manipulating the structure and appearance of natural images, e.g., cat faces in Figure 1 , outperform state-of-the-art algorithms (Chen et al., 2016; Higgins et al., 2017; Jakab et al., 2018) . We also conduct several experiments on MNIST-Color, 3D synthesized data and real photos.

2. METHODOLOGY:

In the absence of annotation on structure, we rely on prior knowledge on how object landmarks should distribute to constrain the learning and disentanglement of structural information. Our experiments show that this is possible given appropriate prior losses and learning architecture. We first formulate our loss function with a consideration on prior. Specifically, we follow the VAE framework and assume 1) the two latent variables z and y, which represent the appearance and structure, are generated from some prior distributions. 2) x follows the conditional distribution p(x|y, z). We start with a Bayesian formulation and maximize the log-likelihood over all observed samples x ∈ X. log p(x) = log p(y) + log p(x|y) -log p(y|x) ≥ log p(y) + log p(x, z|y) dz ≥ log p(y) + E q log p(x, z|y) q(z|x, y)

= log p(y) + E q log p(x|y, z)p(z|y) q(z|x, y) .

(1)

Equation 1 learns a deterministic mapping e(•; θ) from x to y, which we assume y is following a Gaussian distribution over N (e(x; ω), Σ). Term -log p(y|x) is non-negative. In the second line of the equation, we start to consider the factor z. Similar to VAE, we address the issue of intractable integral by introducing an approximate posterior q(y, z|x; φ) to estimate the integral using evidence lower bound (ELBO). By splitting the p(x|y, z) from the second term of the last expression, we obtain our final loss as,

EQUATION

The first term is the prior on y. The second term describes the conditional distribution of x given all representation. Ideally, if the decoder can perfectly reconstruct the x, the second term would be a delta function over x. The third term represents the Kullback-Leibler divergence between approximate. In the rest of this paper we name these three terms respectively as prior loss L prior , reconstruction loss L recon and KL loss L KL . the structure branch forces the representation into a Gaussian spatial probability distribution with an hourglass network eω. 2) the appearance branch E φ learns a complementary appearance representation to the structure.

2.1. PRIOR LOSS:

Inspired by Zhang et al. (2018) and Jakab et al. (2018) , we formulate our structure representation y as a soft latent structured point tensor. A re-projecting operator is applied here to force y to lie on a Gaussian spatial probability distribution space.

Following the notations from Newell et al. (2016) , we denote the direct outputs of the hourglass network e ω as landmark heatmaps h, and each channel of which represents the spatial location of a structural point. Instead of using max activations across each heatmap as landmark coordinates, we weighted average all activations across each heatmap. We then re-project landmark coordinates to spatial features with the same size as heatmaps by a fixed Gaussian-like function centered at predicted coordinates with a fixed standard deviation. As a result, we obtain a new tensor y with prior on structure representation.

Similar to the difficulty described in Zhang et al. (2018) , we find that training the structure branch with general random initialization tend to locate all structural points around the mean location at the center of the image. This could lead to a local minimum from which optimizer might not escape.

As such, we introduce a Separation Loss to encourage each heatmap to sufficiently cover the object of interest. This is achieved by the first part in Eq. 3, where we encourage each pair of i th and j th heatmaps to share different activations. σ can be regarded as a normalization factor here. Another prior constraint is that we wish the structural point to behave like landmarks to encode geometry structure information. To achieve this goal, we add a Concentration Loss to encourage the variance of activations h to be small so that it could concentrate at a single location. This corresponds to the second term in Eq. 3.

EQUATION

It is noteworthy that some recent works have considered the prior of latent factor. Dupont (2018) proposed a Joint-β-VAE by adding different prior distribution over several latent factors so as to disentangle continuous and discrete factors from data. Our work differs in that we investigates a different prior to disentangle visual structure and appearance.

2.2. RECONSTRUCTION LOSS:

For the second term we optimize the reconstruction loss of whole model, which will be denoted as generator G in the following context. We assume that the decoder D θ is able to reconstruct original input x from latent representation y and z, which is x = G(y, z). Consequently, we can design the reconstruction loss as L recon = x -x 1 .

However, minimizing L 1 / L 2 loss at pixel-level only does not model the perceptual quality well and makes the prediction look blurry and implausible. This phenomenon has been well-observed in the literature of super-resolution (Bruna et al., 2016; Sajjadi et al., 2017) . We consequently define the reconstruction loss as

L recon = x -x 1 + l λ l ψ l (x) -ψ l (x) 1

, where ψ l is the feature obtained from l-th layer of a VGG-19 model (Simonyan & Zisserman, 2014) pre-trained on ImageNet. It is also possible to add adversarial loss to further improve the perceptual reconstruction quality. Since the goal of this work is disentanglement rather than reconstruction, we only adopt the L recon described above.

2.3. KL LOSS:

We model q(z|x, y) as a parametric Gaussian distribution which can be estimated by the encoder network E φ . Therefore, the appearance code z can be sampled from q(z|x, y). Meanwhile, the prior p(z|y) can be estimated by the encoder network E θ . By using the reparametrization trick (Kingma & Welling, 2014), these networks can be trained end-to-end. In this work, only mean is estimated for the stability of learning. By modeling the two distributions as Gaussian with identity covariances, the KL Loss is simply equal to the Euclidean distance between their means. Thus, z is regularized by minimizing the KL divergence between q(z|x, y) and p(z|y).

Notice that with only prior and reconstruction loss. The framework only makes sure z is from x and the Decoder D θ will recover as much information of x as possible. There is no guarantee that z will learn a complementary of y. Towards this end, we design the network as concatenating the encoded structure representation by E θ with the inferred appearance code z. Then, the concatenated representation is decoded together by D θ . Moreover, skip-connections between E θ and D θ are also used to pass multi-level structure information to the decoder. Since enough structure information can be obtained from prior, any information about structure encoded in z incurs a penalty of the likelihood p(x|y, z) with no new information (i.e. appearance information) is captured. This design of network and the KL Loss result in a constraint to guide z to encode more information about appearance which is complementary to the structure prior.

2.4. IMPLEMENTATION DETAIL:

Each of the input images x is cropped and resized to 256 × 256 resolution. A one-stack hourglass network (Newell et al., 2016) is used as a geometry extractor e ω to project input image to the heatmap y ∈ R 256×256×30 , in which each channel represents one point-centered 2D-Gaussian map (with σ = 4). y is drawn in a single-channel map for visualization in Fig. 2 . Same network (with stride-2 convolution for downsample) is use for both E θ and E φ to obtain appearance representation z and the embedded structure representation as two 128-dimension vectors. A symmetrical deconvolution network with skip connection is used as the decoder D θ to get the reconstructed result x. All of the networks are jointly trained from scratch end-to-end. We detail the architectures and hyperparameters used for our experiments in appendix A.

3. RELATED WORK:

Unsupervised Feature Disentangle: Several pioneer works focus on unsupervised disentangled representation learning. Following the propose of GANs (Goodfellow et al., 2014) , Chen et al. (2016) purpose InfoGAN to learn a mapping from a group of latent variables to the data in an unsupervised manner. Many similar methods were purposed to achieve a more stable result (Higgins et al., 2017; Kumar et al., 2018) . However, these works suffer to interpret, and the meaning of each learned factor is uncontrollable. There are some following works focusing on dividing latent factors into different sets to enforce better disentangling. Mathieu et al. (2016) Unsupervised Structure Learning: Unsupervised learning structure from objects is one of the essential topics in computer vision. The rudimentary works focus on keypoints detection and learning a strong descriptor to match (Thewlis et al., 2017; Rocco et al., 2018) . Recent two concurrent works, from Jakab et al. (2018) and Zhang et al. (2018) , show the possibility of end-to-end learning of structure in Autoencoder formulations. Our work can be seen as extending their work to learn the complementary appearance representation as well (in other words, in the loss Eq. 1, they only consider the first two terms, and ignore the factor from z). Evaluation Metric: Less existed evaluation metric and benchmark can be utilized to evaluate the performance of disentanglement. Here we propose two forms of evaluation to study the behavior of the proposed framework: 1). Qualitative: we provide four kinds of qualitative results to show as many usages of the disentangled space as possible, i.e. conditional sampling, interpolation, retrieval, and visual analogy. 2). Quantitative: we apply several metrics that are widely employed in image generation (a) Structure consistency: content similarity metric (Li et al., 2017) and meanerror of landmarks (Bulat & Tzimiropoulos, 2017) . (b) Appearance consistency: style similarity metric (Johnson et al., 2016 ) (c). Disentangled ability: retrieval recall@K (Sangkloy et al., 2016) . (d) . Reconstruction and generation quality: SSIM (Wang et al., 2004) and Inception Score (Salimans et al., 2016) .

4. EXPERIMENTS:



4.2. RESULTS ON SYNTHESIZED DATASETS:

Diverse Generation. We first demonstrate the diversity of conditional generation results on MNIST-Color with the successfully disentangled structure and appearance in Fig. 3 . It can be observed that, given an image as a structure condition, same digit information with different appearance can be generated by sampling the appearance condition images randomly. While given an image as appearance condition, different digits with the same color can be generated by sampling different structural conditional images. Note that the model has no prior knowledge of the digit in the image as no label is provided, it effectively learns the disentanglement spontaneously.

Interpolation. In Fig. 3 , the linear interpolation results show reasonable coverage of the manifold.

From left to right, the color is changed smoothly from blue to red with interpolated appearance latent space while maintaining the digit information. Analogously, the color stays stable while one digit transforms into the other smoothly from top to down.

Retrieval. To demonstrate the disentangled ability of the representation learned by the model, we perform nearest neighbor retrieval experiments following Mathieu et al. (2016) on MNIST-Color.

With structure and appearance representation used, both semantic and visual retrieval can be performed respectively. The Qualitative results are shown in appendix A. Quantitatively, We use a commonly used retrieval metric Recall@K as in (Sangkloy et al., 2016; Pang et al., 2017) , where for a particular query digit, Recall@K is 1 if the corresponding digit is within the top-K retrieved results and 0 otherwise. We report the most challenging Recall@1 by averaging over all queries on the test set in Table 2 . It can be observed that the structure representation shows the best performance and clearly outperforms image pixel and appearance representation. In addition to the disentangled ability. This result shows that the structure representation learned by our model is useful for visual retrieval.

Visual Analogy. The task of visual analogy is that the particular attribute of a given reference image can be transformed to a query one (Reed et al., 2015) . We show the visual analogy results on MNIST-Color and 3D Chair in Fig. 4 . Note that even for the detail component (e.g. wheel and leg of 3D chair) the structure can be maintained successfully, which is a rather challenging task in previous unsupervised works (Chen et al., 2016; Higgins et al., 2017) . Taking the structure representation of a query image and the appearance representation of the reference one, our model can output an image which maintains the geometric shape of query image while capturing the appearance of the reference image.

4.3. RESULTS ON REAL-LIFE DATASETS:

We have so far only discussed results on the synthesized benchmarks. In this section, we will demonstrate the scalable performance of our model on several real-life datasets, i.e., Cat, Dog Face and CelebA. To the best knowledge of ours, there is no literature of unsupervised disentanglement before can successfully extend to photo-realistic generation with 256 × 256 resolution. Owing to the structural prior which accurately capture the structural information of images, our model can transform appearance information while faithfully maintain the geometry shapes.

Qualitative evaluation is performed by visually examining the perceptual quality of the generated images. In Fig. 7 , the swapping results along with the learned geometry heatmaps y are illustrated on Cat dataset. In can be seen that the geometry information, i.e., expression, head-pose, facial action, and appearance information i.e., hair texture, can be swapped between each other arbitrarily.

The learned geometry heatmaps can be shown as a map with several 2D Gaussian points, which successfully encode the geometry cues of a image by the location of its points and supply an effective prior for the VAE network. More results of visual analogy of real-life datasets on Standford Dog and CelebA dataset are illustrated in Fig. 5 . We observe that the model is able to generalize to various real-life images with large variations, such as mouth-opening, eye-closing, tongue-sticking and exclusive appearance.

For quantitative measurement, there is no standard evaluation metric of the quality of the visual analogy results for real-life datasets since ground-truth targets are absent. We propose to evaluate the structure and appearance consistency of the analogy predictions respectively instead. We use content similarity metric for the evaluation of structure consistency between a condition input x s and its guided generated images (e.g., for each column of images in Fig. 7 ). We use style similarity metric to evaluate the appearance consistency between a condition input x a and its guided generated images (e.g., each row of images in Fig. 7 ). These two metrics are used widely in image generation applications as an objective for training to maintain content and texture information (Li et al., 2017; Johnson et al., 2016) .

As content similarity metric is less sensitive to the small variation of images, we propose to use the mean-error of landmarks detected by a landmark detection network, which is pre-trained on manually annotated data, to evaluate the structure consistency. Since the public cat facial landmark annotations are too sparse to evaluate the structure consistency (e.g. 9-points (Zhang et al., 2008 )), we manually annotated 10k cat face with 18-points to train a landmark detection network for evaluation purpose. As for the evaluation of celebA, a state-of-the-art model (Bulat & Tzimiropoulos, 2017) with 68-landmarks is used. The results on the testing set of the two real-life datasets are reported in Table 1 . For each test image, 1k other images in the testing set are all used as the reference of structure or appearance for generating, in which mean value is calculated. In the baseline random setting, for one test image, the mean value is calculated by sampling randomly among the generated images guided by each image. The superior structure and appearance consistency of the images generated by our method can be obviously observed. identity, head pose and expression) of query image can be faithfully maintained while the appearance (e.g. the color of hair, beard and illumination) of reference image can be precisely transformed. As concrete examples, the output of the dog in the third column is still tongue-sticking while the hair color is changed, and in the last column of CelebA, even the fine-grain eye make-up is successfully transformed to the query image surprisingly.

4.4. COMPARISON TO OTHER METHODS:

Since there is hardly any literature before share exactly the same settings with us as discussed in the related work, we compare perceptual quality with the four most related unsupervised representation learning methods in Fig. 6 , including three disentangled factor learning methods, i.e., VAE (Kingma & Welling, 2014), β-VAE (Higgins et al., 2017) and InfoGAN (Chen et al., 2016) , and one unsupervised structure learning method (Jakab et al., 2018) . It can be observed that all of the three methods can automatically discover and learn to disentangle the factor of azimuth on 3D Chair dataset. However, it can be perceived that the geometry shape can be maintained much better in our approach than all the other methods, owing to the informative prior supplied by our structure branch. We randomly sample several query-reference pairs in testing set to compare with results reported in the paper of Jakab et al. (2018) . The results of unsupervised structure learning methods have severe artifacts and look more blurred compared with ours. Nevertheless, the identity of query face can be hardly kept in the results of Jakab et al. (2018) . 

5. CONCLUSION:

We extend VAE model to disentangle object's representation by structure and appearance. Our framework is able to mine structure from a kind of objects and learn structure-invariant appearance representation simultaneously, without any annotation. Our work may also reveal several potential topics for future research: 1) Instead of relying on supervision, using strong prior to restrict the latent variables seems to be a potential and effective tool for disentangling. 2) In this work we only experiment on near-rigid objects like chairs and faces, learning on deformable objects is still an opening problem.

3) The structure-invariant appearance representation may have some potentials on recognition tasks. 

:



Figures and Tables:

Figure 2: Architecture: Our framework follows an auto-encoder framework. It contains two branches: 1)

assign one code to the specified factors of variation associated with the labels, and left the remaining as unspecified variability. Similar toMathieu et al. (2016),Hu et al. (2018) then proposes to obtain disentanglement of feature chunks by leveraging Autoencoders, with the supervision of some same/different class pairs.Dupont (2018) divides latent variable into discrete and continuous one, and distribute them in different prior distribution. In our work, we give one branch of representation are more complicated prior, to force it to represent only the pose information for the object.Supervised Pose Synthesis: Recently the booming of GANs research improves the capacity of pose-guided image generation.Ma et al. (2017) firstly try to synthesize pose images with U-Netlike networks. Several works soon follow this appealing topic and obtain better results on human pose or face generation. A close work to us fromEsser et al. (2018) applied a conditional U-Net for shape-guided image generation. Nevertheless, existing works rely on massive annotated data, they need to treat pose of a object as input, or a strong pre-trained pose estimator.

We evaluate our method on four datasets that cover both synthesized and real world data: 1). MNIST-Color: we extend MNIST by either colorizing the digit (MNIST-CD) or the background (MNIST-CB) with a randomly chosen color followingGonzalez-Garcia et al. (2018). We use the standard split of training (50k) and testing (10k) set. 2). 3D Chair: Aubry et al. (2014) offers rendered images of 1393 CAD chair models. We take 1343 chairs for training and the left 50 chairs for testing. For each chair, 12 rendered images with different views are selected randomly. 3). Cat & Dog Face, we collect 6k (5k for training and 1k for testing) images of cat and dog from YFCC100M(Kalkowski et al., 2015) and Standford Dog(Khosla et al., 2011) datasets respectively. All images are center cropped around the face and scaled to the same size. 4). CelebA: it supplies plenty of celebrity faces with different attributes. The training and testing sizes are 160K and 20K respectively.

Figure 3: Conditional generation results:(a) Walking in the appearance space with fixed structure. (b)Walking in the structure space with fixed appearance. (c) A visualization of the disentangled space by linear interpolation. The Structure is smoothly changed in row-wise and the appearance is changed by each column.

Figure 4: Visual analogy results on synthesized datasets: (a) MNIST-CD. (b) MNIST-CB. (c) 3D Chair.

Figure 5: Visual analogy results on real-life datasets: (a) Standford Dog. (b) CelebA. The geometry (e.g.

Figure 6: Comparison to other methods. Qualitative results of disentangling performance of VAE, β-VAE, InfoGAN and Jakab et al. (2018). We demonstrate the disentanglement of the factor of azimuth for 3D chair dataset. Visual analogy results are demonstrated for face dataset.4.5 ABLATION STUDYIt is worth studying the effect of individual component of our method on the quality of generated images. Structural Similarities (SSIM) and Inception Scores (IS) are utilized to evaluate the reconstruction quality and the analogy quality. As reported in Table2, without KL-Loss, the network has no incentive to learn the shape invariant appearance of representation, almost all of the metrics degraded dramatically.

Figure 7: A grid of structure&appearance swapping visualization. The top row and left-most columns are random selected from the test set. In each column, the structure of the generated images are shown to be consistent with the top ones. In each row, the appearance of the generated images are shown to be consistent with the left-most ones.

Structure and appearance consistency evaluation on Cat and CelebA dataset (lower is better).

Table content: <table><tr><td>Method</td><td/><td>Cat</td><td/><td/><td>CelebA</td><td/></tr><tr><td/><td colspan="6">Style (×e -5 ) Content (×e -6 ) Landmark (%) Style (×e -5 ) Content (×e -6 ) Landmark (%)</td></tr><tr><td>Random</td><td>7.700</td><td>1.881</td><td>0.051</td><td>5.858</td><td>1.693</td><td>0.293</td></tr><tr><td>Ours</td><td>5.208</td><td>1.759</td><td>0.030</td><td>3.886</td><td>1.529</td><td>0.162</td></tr></table>

without KL-Loss, the network has no incentive to learn the shape invariant appearance of representation, almost all of the metrics degraded dramatically.

Table content: <table><tr><td>Method Pixel Appearance Structure</td><td>Color-Digit Color-Back 31.65 39.52 10.25 15.32 99.96 99.92</td><td>Method Real Data Without KL Loss Ours</td><td>Style (×e -5 ) Content (×e -6 ) Landmark (%) ---6.556 1.813 0.036 5.208 1.759 0.030</td><td>SSIM mean std 1.000 0.000 2.004 Inception Score mean std 0.157 0.406 0.103 1.72 0.189 0.449 0.121 1.968 0.181</td></tr><tr><td/><td>(a)</td><td/><td>(b)</td><td/></tr></table>

(a)  Retrieval results reported as recall@1 on MNIST-Color. (b) Ablation study.

Table content: <table/>

Peer review (iclr2019):
 This paper proposes an unsupervised method to disentangle the latent code of VAE. Overall, it is novel and well written. The experiment has shown good performance of the proposed method. I have some concerns as follows: 1. In Eq.(1), y is assumed to follow a Gaussian distribution. Is it possible that y follows a multinomial distribution? Then, this model can be used for clustering. 2. In section 2.3, the concatenation between z and y is used to learn a complementary of y. Why does the concatenation encourage to learn the complementary of y? More explanations are needed. Additionally, some experiments are needed to verify this claim.