Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a method for generating images from attributes. The core idea is to learn a shared latent space for images and attributes with variational auto-encoder using paired samples, and additionally learn individual inference networks from images or attributes to the latent space using unpaired samples. During training the auto-encoder is trained on paired data (image, attribute) whereas during testing one uses the unpaired data to generate an image corresponding to an attribute or vice versa. The authors propose handling missing data using a product of experts where the product is taken over available attributes, and it sharpens the prior distribution. The authors evaluate their method using correctness i.e. if the generated images have the desired attributes, coverage i.e. if the generated images sample unspecified attributes well, and compositionality i.e. if  images can be generated from unseen attributes. Although the proposed method performs slightly poor compared to JMVAE in terms of concreteness when all attributes are provided, it outperforms when some of the attributes are missing (Figure 4a). It also outperforms existing methods in terms of coverage and compositionality.

Major comments:

The paper is well written, and summarizes its contribution succinctly.

I did not fully understand the 'retrofitting' idea. If I understood correctly, the authors first train \theta and \phi and then fix \theta to train \phi_x and \phi_y. If that is true, then is \calL(\theta, \phi, \phi_x, \phi_y) are right cost function since one does not maximize all three ELBO terms when optimizing \theta? Please clarify?

Minor comments:

- 'in order of increasing abstraction', does the order of gender-> smiling or not -> hair color matter? Or, is male, *, blackhair a valid option?

- what are the image sizes for the CelebA dataset

- page 5: double the

- Which multi-label classifier is used to classify images in attributes?
review_title: nice paper, needs some clarification

final_score: rating: 7: Good paper, accept
review_content: This paper presented a multi-modal extension of variational autoencoder (VAE) for the task "visually grounded imagination."  In this task,  the model learns a joint embedding of the images and the attributes. The proposed model is novel but incremental comparing to existing frameworks.  The author also introduced new evaluation metrics to evaluate the model performance concerning correctness, coverage, and compositionality. 

Pros:
1. The paper is well-written, and the contribution (both the model and the evaluation metric) potentially can to be very useful in the community.  
2. The discussion comparing the related work/baseline methods is insightful. 
3. The proposed model addresses many important problems, such as attribute learning, disentanged representation learning, learning with missing values, and proper evaluation methods. 

Cons/questions:
1. The motivation of the model choice of q is not clear.
Comparing to BiVCCA, apart from the differences that the author discussed, a big difference is the choice of q.  BiVCCA uses two inference networks q(z|x) and q(z|y), while the proposed method uses three. q(z|x), q(z|y), and q(z|x,y).  How does such model choice affect the final performance? 

2. Baselines are not necessarily sufficient. 
The paper compared the vanilla version of BiVCCA but not the one with factorized representation version. In the original VAECCA paper, the extension of using factorized representation (private and shared) improved the performance]. The author should also compare this extension of VAECCA.

3. Some details are not clear. 
a) How to set/learn the scaling parameter \lambda_y and \beta_y? If it is set as hyper-parameter, how does the performance change concerning them? 
b) Discussion of the experimental results is not sufficient. For example, why JMVAE performs much better than the proposed model when all attributes are given.  What is the conclusion from Figure 4(b)? The JMVAE seems to generate more diverse (better coverage) results which are not consistent with the claims in the related work.  The same applies to figure 5. 

review_title: Well written paper

final_score: rating: 7: Good paper, accept
review_content: The authors propose a generative method that can produce images along a hierarchy of specificity, i.e. both when all relevant attributes are specified, and when some are left undefined, creating a more abstract generation task. 

Pros:
+ The results demonstrating the method's ability to generate results for (1) abstract and (2) novel/unseen attribute descriptions, are generally convincing. Both quantitative and qualitative results are provided. 
+ The paper is fairly clear.

Cons:
- It is unclear how to judge diversity qualitatively, e.g. in Fig. 4(b).
- Fig. 5 could be more convincing; "bushy eyebrows" is a difficult attribute to judge, and in the abstract generation when that is the only attribute specified, it is not clear how good the results are.

review_title: Clear paper with convincing results

Meta-review: All three reviewers recommend acceptance. Good work, accept
Decision: Accept (Poster)