Unsupervised Disentangling Structure and Appearance

Abstract: It is challenging to disentangle an object into two orthogonal spaces of structure and appearance since each can influence the visual observation in a different and unpredictable way. It is rare for one to have access to a large number of data to help separate the influences. In this paper, we present a novel framework to learn this disentangled representation in a completely unsupervised manner. We address this problem in a two-branch Variational Autoencoder framework. For the structure branch, we project the latent factor into a soft structured point tensor and constrain it with losses derived from prior knowledge. This encourages the branch to distill geometry information. Another branch learns the complementary appearance information. The two branches form an effective framework that can disentangle object's structure-appearance representation without any human annotation. We evaluate our approach on four image datasets, on which we demonstrate the superior disentanglement and visual analogy quality both in synthesis and real-world data. We are able to generate photo-realistic images with 256*256 resolution that are clearly disentangled in structure and appearance.

Reviews and author responses:

This paper proposes an unsupervised method to disentangle the latent code of VAE. Overall, it is novel and well written. The experiment has shown good performance of the proposed method.
 
I have some concerns as follows:
1.  In Eq.(1), y is assumed to follow a Gaussian distribution. Is it possible that y follows a multinomial distribution? Then, this model can be used for clustering.

2. In section 2.3, the concatenation between z and y is used to learn a complementary of y.  Why does the concatenation encourage  to learn the complementary of y? More explanations are needed.  Additionally, some experiments are needed to verify this claim.




In this paper,  a conditional deep generative model is proposed for disentangling structure (more precisely shape)  and appearance.  The architecture of the proposed system is very similar to that in [A], however, in this paper different applications are considered. The paper is relatively well-written and a number of experiments are presented. However, they are that convincing.

I have two main concerns regarding this paper.

1)	The authors have not taken into account recently proposed deep generative models for disentangling shape and appearance along other modes of visual variations. A non-exhaustive list is as follows:

*GAGAN: Geometry-Aware Generative Adversarial Networks
*Geometry-Contrastive GAN for Facial Expression Transfer
*Cross-View Image Synthesis using Conditional GANs
*Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance
*Neural Face Editing with Intrinsic Image Disentangling


The authors should discuss how the proposed method is different from the above-mentioned ones and compare the performance of the proposed model against that obtained by GAGAN and Deforming Autoencoders, which are very relevant to the proposed one models.

2)	Some of the experimental results are not convincing. For example, in Fig. 6 it seems to me that all the chairs produced by the proposed method are identical. In the same figure, Jakab’s method seems to produce more meaningful results than the proposed method which appears to implement texture style transfer, rather than shape transfer.

Considering all the above, I believe the paper needs substantial improvement prior to being considered for publication.


Reference

[Α] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Conditional image generation for
learning the structure of visual objects. NIPS, 2018. 


Summary: The authors present an autoencoding strategy for disentangling structure and appearance in image data. They achieve this using a learned, spatial prior in the VAE framework.

Writing: The paper contains grammatical errors and I found Section 2 (2.1 and 2.2, specifically) to be a bit confusing as many ideas were described with words when they could have been outlined more precisely mathematically. 

Major comments:
The paper takes ideas from Zhang et al and Jakab et al and put them in a VAE context. The paper, however, constructs the ELBO in such a way that distance it from many key ideas of the VAE. Particularly, the paper decomposes the ELBO into three terms and proceed to define these terms as they wish. In this way, the novel contributions of this paper are left unclear and many decisions left unjustified.

- Incorporating landmark/spatial information into autoencoders is not a new idea. Zhang et al and Jakab et al both train autoencoders with disentangled structure, and Finn, 2016 [1] uses a similar spatial landmark strategy when learning representations. Incorporating structural information as a prior distribution (as in this paper) is an interesting idea. However, it is not clear that the defined prior log p(y) is a proper density (integrates to 1). Given the importance of a prior distribution in VAEs, this choice should be precisely justified (maybe relate it to beta-VAE or just use a properly normalizable distribution)

- The paper chooses variational distributions in such a way that removes the entropy term from the KL divergences. Specifically, both q(z | x, y) and p(z | y) have fixed, identity covariance, resulting in the KL divergences equating to L_2 distance. An important part of the VAE is the explicit incorporation of uncertainty by means of learned variances. Although this can sometimes be problematic (see beta-VAE), neglecting to include these variances at all removes an important aspect of VAE.

- The paper includes a likelihood model which also throws away key ideas from VAE. Although some neural likelihood models as in the VAE have issues with blurriness, the reasons are still not completely understood. However, there are well explored alternatives to what the authors propose. For example, many image-based VAEs use Bernoulli likelihoods [2, 3] or autoregressive likelihoods [4]. Autoregressive models, especially, can produce sharp images. The authors introduce a unnormalizable likelihood which combines L1 loss with a function that incorporates L1 distance in VGG space. Using VGG in the likelihood model is unjustified and seems unnecessarily complicated, given the existence of powerful decoders that already exist in VAE literature.

- The authors incorporate various connections and concatenations between neural networks and distributions that further complicate the variational lower bound. For example, p(z | y) is concatenated to q(z | x, y) in addition to acting as a prior on q(z | x, y). This introduces a dependency between the likelihood model and the variational posterior which normally does not exist. Furthermore skip connections are introduced between E_\theta and D_\theta, which complicate the ELBO further. The authors should explicitly write out the loss function they are optimizing at this point or describe how they are modifying ELBO to justify these chocise.

Overall, I find it difficult to call this a variational autoencoder given the liberal modifications to the evidence lower-bound. However, even if I was to interpret this work as an autoencoder with a custom loss function, this model ends up very similar to that in Zhang et al with the main differences being the inclusion of an equivariance constraint in Zhang that is not present in this paper and that Zhang et al use a feature map that’s multiplied by landmarks to incorporate appearance information whereas this paper uses the z representation as appearance information. 

The qualitative results of this paper, although good looking, are very similar to qualitative results in Zhang et al and Jakab et al. A quick comment: in Figure 6, you should use the same celebrity faces when comparing Jakab against your own work. The quantitative results only compare to the same model trained without the KL loss term; this, in my mind, is more of a sanity check than a fair baseline. The authors should be comparing against alternate strategies that incorporate spatial information, such as Zhang et al and Jakab et al.

[1] Finn, Chelsea, et al. "Deep spatial autoencoders for visuomotor learning." 2016 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2016.
[2] Chen, Xi, et al. "Variational lossy autoencoder." arXiv preprint arXiv:1611.02731 (2016).
[3] http://ruishu.io/2018/03/19/bernoulli-vae/
[4] van den Oord, Aaron, et al. "Conditional image generation with pixelcnn decoders." Advances in Neural Information Processing Systems. 2016.

Meta-review: With an average review score of 4.67 and a short review for the one positive review it is just not possible to accept the paper.

Decision: rejected-papers