Unsupervised Disentangling Structure and Appearance

1 INTRODUCTION . Structure and appearance are the two most inherent attributes that characterize an object visually . Computer vision researchers have devoted decades of efforts to understand object structure and extract features that are invariant to geometry change ( Huang et al. , 2007 ; Thewlis et al. , 2017 ; Rocco et al. , 2018 ) . Learning such disentangled deep representation for visual objects is an important topic in deep learning . The main objective of our work is to disentangle object ’ s appearance and structure in an unsupervised manner . Achieving this goal is non-trivial due to three reasons : 1 ) Without supervision , we can hardly guarantee the separation of different representations in the latent space . 2 ) Although some methods like InfoGAN ( Chen et al. , 2016 ) are capable of learning several groups of independent attributes from objects , attributes from these unsupervised frameworks are uninterpretable since we can not pinpoint which portion of the disentangled representation is related to the structure and which to the appearance . 3 ) Learning structure from a set of natural real-world images is difficult . To overcome the aforementioned challenges , we propose a novel two-branch Variational Autoencoder ( VAE ) framework , of which the structure branch aims to discover meaningful structural points to represent the object geometry , while the other appearance branch learns the complementary appearance representation . The settings of these two branches are asymmetric . For the structure branch , we add a layer-wise softmax operator to the last layer . This can be seen as a projection of a latent structure to a soft structured point tensor space . Specifically designed prior losses are used to constrain the structured point tensors so that the discovered points have high repeatability across images yet distributed uniformly to cover different parts of the object . To encourage the framework to learn a disentangled yet complementary representation of both appearance and structure , we introduce a Kullback-Leibler ( KL ) divergence loss and skip-connections design to the framework . Extensive experiments demonstrate the effectiveness of the proposed method in manipulating the structure and appearance of natural images , e.g. , cat faces in Figure 1 , outperform state-of-the-art algorithms ( Chen et al. , 2016 ; Higgins et al. , 2017 ; Jakab et al. , 2018 ) . We also conduct several experiments on MNIST-Color , 3D synthesized data and real photos . 2 METHODOLOGY . In the absence of annotation on structure , we rely on prior knowledge on how object landmarks should distribute to constrain the learning and disentanglement of structural information . Our experiments show that this is possible given appropriate prior losses and learning architecture . We first formulate our loss function with a consideration on prior . Specifically , we follow the VAE framework and assume 1 ) the two latent variables z and y , which represent the appearance and structure , are generated from some prior distributions . 2 ) x follows the conditional distribution p ( x|y , z ) . We start with a Bayesian formulation and maximize the log-likelihood over all observed samples x ∈ X . log p ( x ) = log p ( y ) + log p ( x|y ) − log p ( y|x ) ≥ log p ( y ) + log ∫ p ( x , z|y ) dz ≥ log p ( y ) + Eq log p ( x , z|y ) q ( z|x , y ) = log p ( y ) + Eq log p ( x|y , z ) p ( z|y ) q ( z|x , y ) . ( 1 ) Equation 1 learns a deterministic mapping e ( · ; θ ) from x to y , which we assume y is following a Gaussian distribution over N ( e ( x ; ω ) , Σ ) . Term − log p ( y|x ) is non-negative . In the second line of the equation , we start to consider the factor z . Similar to VAE , we address the issue of intractable integral by introducing an approximate posterior q ( y , z|x ; φ ) to estimate the integral using evidence lower bound ( ELBO ) . By splitting the p ( x|y , z ) from the second term of the last expression , we obtain our final loss as , L ( x , θ , φ , ω ) = − log pω ( y ) − Eqφ ( z|x , y ) log pθ ( x|y , z ) + KL ( qφ ( z|x , y ) ( z|x , y ) ||pθ ( z|y ) ) . ( 2 ) The first term is the prior on y . The second term describes the conditional distribution of x given all representation . Ideally , if the decoder can perfectly reconstruct the x , the second term would be a delta function over x . The third term represents the Kullback-Leibler divergence between approximate . In the rest of this paper we name these three terms respectively as prior loss Lprior , reconstruction loss Lrecon and KL loss LKL . 2.1 PRIOR LOSS . Inspired by Zhang et al . ( 2018 ) and Jakab et al . ( 2018 ) , we formulate our structure representation y as a soft latent structured point tensor . A re-projecting operator is applied here to force y to lie on a Gaussian spatial probability distribution space . Following the notations from Newell et al . ( 2016 ) , we denote the direct outputs of the hourglass network eω as landmark heatmaps h , and each channel of which represents the spatial location of a structural point . Instead of using max activations across each heatmap as landmark coordinates , we weighted average all activations across each heatmap . We then re-project landmark coordinates to spatial features with the same size as heatmaps by a fixed Gaussian-like function centered at predicted coordinates with a fixed standard deviation . As a result , we obtain a new tensor y with prior on structure representation . Similar to the difficulty described in Zhang et al . ( 2018 ) , we find that training the structure branch with general random initialization tend to locate all structural points around the mean location at the center of the image . This could lead to a local minimum from which optimizer might not escape . As such , we introduce a Separation Loss to encourage each heatmap to sufficiently cover the object of interest . This is achieved by the first part in Eq . 3 , where we encourage each pair of ith and jth heatmaps to share different activations . σ can be regarded as a normalization factor here . Another prior constraint is that we wish the structural point to behave like landmarks to encode geometry structure information . To achieve this goal , we add a Concentration Loss to encourage the variance of activations h to be small so that it could concentrate at a single location . This corresponds to the second term in Eq . 3 . Lprior = ∑ i6=j exp ( −||hi − hj || 2 2σ2 ) + Var ( h ) ( 3 ) It is noteworthy that some recent works have considered the prior of latent factor . Dupont ( 2018 ) proposed a Joint-β-VAE by adding different prior distribution over several latent factors so as to disentangle continuous and discrete factors from data . Our work differs in that we investigates a different prior to disentangle visual structure and appearance . 2.2 RECONSTRUCTION LOSS . For the second term we optimize the reconstruction loss of whole model , which will be denoted as generator G in the following context . We assume that the decoder Dθ is able to reconstruct original input x from latent representation y and z , which is x̂ = G ( y , z ) . Consequently , we can design the reconstruction loss as Lrecon = ‖x− x̂‖1 . However , minimizing L1 / L2 loss at pixel-level only does not model the perceptual quality well and makes the prediction look blurry and implausible . This phenomenon has been well-observed in the literature of super-resolution ( Bruna et al. , 2016 ; Sajjadi et al. , 2017 ) . We consequently define the reconstruction loss as Lrecon = ‖x − x̂‖1 + ∑ l λl‖ψl ( x ) − ψl ( x̂ ) ‖1 , where ψl is the feature obtained from l-th layer of a VGG-19 model ( Simonyan & Zisserman , 2014 ) pre-trained on ImageNet . It is also possible to add adversarial loss to further improve the perceptual reconstruction quality . Since the goal of this work is disentanglement rather than reconstruction , we only adopt the Lrecon described above . 2.3 KL LOSS . We model q ( z|x , y ) as a parametric Gaussian distribution which can be estimated by the encoder network Eφ . Therefore , the appearance code z can be sampled from q ( z|x , y ) . Meanwhile , the prior p ( z|y ) can be estimated by the encoder network Eθ . By using the reparametrization trick ( Kingma & Welling , 2014 ) , these networks can be trained end-to-end . In this work , only mean is estimated for the stability of learning . By modeling the two distributions as Gaussian with identity covariances , the KL Loss is simply equal to the Euclidean distance between their means . Thus , z is regularized by minimizing the KL divergence between q ( z|x , y ) and p ( z|y ) . Notice that with only prior and reconstruction loss . The framework only makes sure z is from x and the Decoder Dθ will recover as much information of x as possible . There is no guarantee that z will learn a complementary of y . Towards this end , we design the network as concatenating the encoded structure representation by Eθ with the inferred appearance code z . Then , the concatenated representation is decoded together by Dθ . Moreover , skip-connections between Eθ and Dθ are also used to pass multi-level structure information to the decoder . Since enough structure information can be obtained from prior , any information about structure encoded in z incurs a penalty of the likelihood p ( x|y , z ) with no new information ( i.e . appearance information ) is captured . This design of network and the KL Loss result in a constraint to guide z to encode more information about appearance which is complementary to the structure prior . 2.4 IMPLEMENTATION DETAIL . Each of the input images x is cropped and resized to 256 × 256 resolution . A one-stack hourglass network ( Newell et al. , 2016 ) is used as a geometry extractor eω to project input image to the heatmap y ∈ R256×256×30 , in which each channel represents one point-centered 2D-Gaussian map ( with σ = 4 ) . y is drawn in a single-channel map for visualization in Fig . 2 . Same network ( with stride-2 convolution for downsample ) is use for both Eθ and Eφ to obtain appearance representation z and the embedded structure representation as two 128-dimension vectors . A symmetrical deconvolution network with skip connection is used as the decoder Dθ to get the reconstructed result x̂ . All of the networks are jointly trained from scratch end-to-end . We detail the architectures and hyperparameters used for our experiments in appendix A . 3 RELATED WORK . Unsupervised Feature Disentangle : Several pioneer works focus on unsupervised disentangled representation learning . Following the propose of GANs ( Goodfellow et al. , 2014 ) , Chen et al . ( 2016 ) purpose InfoGAN to learn a mapping from a group of latent variables to the data in an unsupervised manner . Many similar methods were purposed to achieve a more stable result ( Higgins et al. , 2017 ; Kumar et al. , 2018 ) . However , these works suffer to interpret , and the meaning of each learned factor is uncontrollable . There are some following works focusing on dividing latent factors into different sets to enforce better disentangling . Mathieu et al . ( 2016 ) assign one code to the specified factors of variation associated with the labels , and left the remaining as unspecified variability . Similar to Mathieu et al . ( 2016 ) , Hu et al . ( 2018 ) then proposes to obtain disentanglement of feature chunks by leveraging Autoencoders , with the supervision of some same/different class pairs . Dupont ( 2018 ) divides latent variable into discrete and continuous one , and distribute them in different prior distribution . In our work , we give one branch of representation are more complicated prior , to force it to represent only the pose information for the object . Supervised Pose Synthesis : Recently the booming of GANs research improves the capacity of pose-guided image generation . Ma et al . ( 2017 ) firstly try to synthesize pose images with U-Netlike networks . Several works soon follow this appealing topic and obtain better results on human pose or face generation . A close work to us from Esser et al . ( 2018 ) applied a conditional U-Net for shape-guided image generation . Nevertheless , existing works rely on massive annotated data , they need to treat pose of a object as input , or a strong pre-trained pose estimator . Unsupervised Structure Learning : Unsupervised learning structure from objects is one of the essential topics in computer vision . The rudimentary works focus on keypoints detection and learning a strong descriptor to match ( Thewlis et al. , 2017 ; Rocco et al. , 2018 ) . Recent two concurrent works , from Jakab et al . ( 2018 ) and Zhang et al . ( 2018 ) , show the possibility of end-to-end learning of structure in Autoencoder formulations . Our work can be seen as extending their work to learn the complementary appearance representation as well ( in other words , in the loss Eq . 1 , they only consider the first two terms , and ignore the factor from z ) .

Summary: In this paper,  a conditional deep generative model is proposed for disentangling structure (more precisely shape)  and appearance.  The architecture of the proposed system is very similar to that in [A], however, in this paper different applications are considered. The paper is relatively well-written and a number of experiments are presented. However, they are that convincing.