Title: Generative Models of Visually Grounded Imagination
Authors: Ramakrishna Vedantam
Ian Fischer
Jonathan Huang
Kevin Murphy
Venue: ICLR 2018
Keywords: variational autoencoders
generative models
language
vision
abstraction
compositionality
hierarchy

Abstract: It is easy for people to imagine what a man with pink hair looks like, even if they have never seen such a person before. We call the ability to create images of novel semantic concepts visually grounded imagination. In this paper, we show how we can modify variational auto-encoders to perform this task. Our method uses a novel training objective, and a novel product-of-experts inference network, which can handle partially specified (abstract) concepts in a principled and efficient way. We also propose a set of easy-to-compute evaluation metrics that capture our intuitive notions of what it means to have good visual imagination, namely correctness, coverage, and compositionality (the 3 C’s). Finally, we perform a detailed comparison of our method with two existing joint image-attribute VAE methods (the JMVAE method of Suzuki et al., 2017 and the BiVCCA method of Wang et al., 2016) by applying them to two datasets: the MNIST-with-attributes dataset (which we introduce here), and the CelebA dataset (Liu et al., 2015).

TL;DR: A VAE-variant which can create diverse images corresponding to novel concrete or abstract "concepts" described using attribute vectors.

Review discussion:
The authors propose a generative method that can produce images along a hierarchy of specificity, i.e. both when all relevant attributes are specified, and when some are left undefined, creating a more abstract generation task.
Pros:
+ The results demonstrating the method's ability to generate results for (1) abstract and (2) novel/unseen attribute descriptions, are generally convincing. Both quantitative and qualitative results are provided.
+ The paper is fairly clear.
Cons:
- It is unclear how to judge diversity qualitatively, e.g. in Fig. 4(b).
- Fig. 5 could be more convincing; "bushy eyebrows" is a difficult attribute to judge, and in the abstract generation when that is the only attribute specified, it is not clear how good the results are.
We are glad the reviewer thought that our paper was clear and has convincing results. Below we provide some clarifications to address the cons raised by the reviewer:
1. How to judge diversity in Fig. 4 b):
The new version provides a clearer caption to Fig. 4 b) explaining the qualitative results for MNIST-A. Hopefully this should make judging diversity easier.
2. Explanation for Fig. 5:
We have updated the text on Page. 9, to better explain the qualitative results for CelebA, in addition to further examples in Appendix A.9.
Further, we would like to point out a subtle distinction between attributes which are ‘absent’ and attributes which are ‘unspecified’. ‘Absence’ means that the value of an attribute is set to 0, while unspecified means that we don’t know whether the attribute should be set (1) or unset (0).
Thus, for Fig. 5, there are a total of 16 attributes specified (15 unset and 1 set) (for the most abstract generation example, male=*, smiling=*), as opposed to just ‘bushy eyebrows’ being specified (which is one of the premises of the reviewer’s concern).
This paper presented a multi-modal extension of variational autoencoder (VAE) for the task "visually grounded imagination."  In this task,  the model learns a joint embedding of the images and the attributes. The proposed model is novel but incremental comparing to existing frameworks.  The author also introduced new evaluation metrics to evaluate the model performance concerning correctness, coverage, and compositionality.
Pros:
1. The paper is well-written, and the contribution (both the model and the evaluation metric) potentially can to be very useful in the community.
2. The discussion comparing the related work/baseline methods is insightful.
3. The proposed model addresses many important problems, such as attribute learning, disentanged representation learning, learning with missing values, and proper evaluation methods.
Cons/questions:
1. The motivation of the model choice of q is not clear.
Comparing to BiVCCA, apart from the differences that the author discussed, a big difference is the choice of q.  BiVCCA uses two inference networks q(z|x) and q(z|y), while the proposed method uses three. q(z|x), q(z|y), and q(z|x,y).  How does such model choice affect the final performance?
2. Baselines are not necessarily sufficient.
The paper compared the vanilla version of BiVCCA but not the one with factorized representation version. In the original VAECCA paper, the extension of using factorized representation (private and shared) improved the performance]. The author should also compare this extension of VAECCA.
3. Some details are not clear.
a) How to set/learn the scaling parameter \lambda_y and \beta_y? If it is set as hyper-parameter, how does the performance change concerning them?
b) Discussion of the experimental results is not sufficient.
For example, why JMVAE performs much better than the proposed model when all attributes are given.  What is the conclusion from Figure 4(b)? The JMVAE seems to generate more diverse (better coverage) results which are not consistent with the claims in the related work.  The same applies to figure 5.
We are glad the reviewer thought that our contribution would be useful to the community and that our work addresses important problems. Below we address specific cons/ questions raised by the reviewer:
1. The motivation for the model choice of q:
In general, when there is information asymmetry in the two modalities (i.e. one instance in modality y corresponds to many instances in modality x), we believe it is beneficial to learn with a joint inference network and retrofit (like TELBO does) as opposed to trying to explain both modalities using the unimodal inference network (say for q(z| y)) where I(y) < I(x), since it is difficult for the latent variable in this case to explain 'all' the instances of the second (information rich) modality.
2. Comparison to private-VCCA:
A direct comparison to private-VCCA (from Wang.et.al. (VAECCA)) cannot be made since the private-VCCA model does not have a mechanism to do ‘imagination’. Concretely, the private-VCCA model cannot be applied for multimodal translation, i.e. going from x to y or vice versa, since in this model, p(x| h_x, z) needs access to h_x and z, while the inference networks that condition on 'y' are just q(h_y| y) and q(z| y), meaning there is no path to go from ‘y’ to ‘x’, which is needed for conditional generation.
We implemented a modified version of private-VCCA and /extended/ it to the BiVCCA setting (i.e. learning two inference networks) to obtain a version that /can/ do conditional generation. We call this model imagination-private-BiVCCA. Concretely, we learn three inference networks per modality: q(h_x| y), q(h_y|y) and q(z| y). Now, we see that given ‘y’, we can sample h_x and z, and thus, feed it through p(x| h_x, z) to generate images. Similar to Wang. et.al., we regularize the latent space in this model using dropout, searching for values in the range {0, 0.2, 0.4}. As with the BiVCCA model we present in this paper, we also search for \mu in range {0.3, 0.5, 0.7}. We find that this approach does about as well as the best BiVCCA method we already report in the paper and is still substantially worse than JMVAE or TELBO.
3. Details of the hyperparameters, \lambda_y and \beta_y:
Firstly, we clarified notation, replacing \beta_y with \gamma in the revised version. We have added details of how performance changes based on different hyperparameters for all the objective functions in appendix A.6. Concretely, the important parameters to tweak for TELBO are \lambda_y (with \gamma) being less important. For JMVAE, both \lambda_y and \alpha turn out to be fairly important, while for BiVCCA, \lambda_y is important while the choice of \mu is marginally important.
4. Why is JMVAE better at correctness when all attributes are specified?
Page. 6 explains the tradeoff between correctness and coverage implicit in choices of JMVAE v.s. TELBO objectives.
Briefly, our initial submission showed in appendix A.1. that JMVAE fits an inference network optimizing the KL divergence between the aggregate posterior (for the set of images given a label y_i) and the unimodal inference network q(z| y_i), to ‘cover’ the aggregate posterior. This ensures that the samples from q(z| y_i) are better tuned to work with p(x| z), since the elbo(x, y) term has a likelihood function p_{\theta_x}(x| z) which is fed samples from q(z| x, y), whose aggregate posterior q(z| y_i) tries to match.
In contrast TELBO regularizes q(z| y_i) to be ‘close’ to p(z) which leads to better coverage, but does not lead naturally to the expectation that we would achieve better correctness than a model like JMVAE, which in some sense has a tighter coupling between the likelihood and the q(z| y) terms. If the aggregate posterior matches the prior better, the gap in correctness between JMVAE and TELBO would reduce. Making this happen is an active research area in variational inference (see [A], [B]).
5. Clarifications about qualitative results:
The new version provides a clearer caption to Fig. 4 b) explaining the qualitative results for MNIST-A and adds new text on Page. 9 explaining the qualitative results on CelebA.
In addition, we show more qualitative results in Appendix A.9 (Page. 18, Figure. 13) discussing results comparing TELBO and JMVAE in terms of diversity.
References:
[A]: Tomczak, Jakub M., and Max Welling. 2017. “VAE with a VampPrior.” arXiv [cs.LG]. arXiv. http://arxiv.org/abs/1705.07120.
[B]: Hoffman, Matthew D., and Matthew J. Johnson. 2016. “Elbo Surgery: Yet Another Way to Carve up the Variational Evidence Lower Bound.” In Workshop in Advances in Approximate Bayesian Inference, NIPS. http://approximateinference.org/accepted/HoffmanJohnson2016.pdf.
The paper proposes a method for generating images from attributes. The core idea is to learn a shared latent space for images and attributes with variational auto-encoder using paired samples, and additionally learn individual inference networks from images or attributes to the latent space using unpaired samples. During training the auto-encoder is trained on paired data (image, attribute) whereas during testing one uses the unpaired data to generate an image corresponding to an attribute or vice versa. The authors propose handling missing data using a product of experts where the product is taken over available attributes, and it sharpens the prior distribution. The authors evaluate their method using correctness i.e. if the generated images have the desired attributes, coverage i.e. if the generated images sample unspecified attributes well, and compositionality i.e. if  images can be generated from unseen attributes.
Although the proposed method performs slightly poor compared to JMVAE in terms of concreteness when all attributes are provided, it outperforms when some of the attributes are missing (Figure 4a). It also outperforms existing methods in terms of coverage and compositionality.
Major comments:
The paper is well written, and summarizes its contribution succinctly.
I did not fully understand the 'retrofitting' idea. If I understood correctly, the authors first train \theta and \phi and then fix \theta to train \phi_x and \phi_y. If that is true, then is \calL(\theta, \phi, \phi_x, \phi_y) are right cost function since one does not maximize all three ELBO terms when optimizing \theta? Please clarify?
Minor comments:
- 'in order of increasing abstraction', does the order of gender-> smiling or not -> hair color matter? Or, is male, *, blackhair a valid option?
- what are the image sizes for the CelebA dataset
- page 5: double the
- Which multi-label classifier is used to classify images in attributes?
We thank the reviewer for the encouraging feedback on the paper. Below we provide clarifications to specific concerns.
1. Clarifications about “retrofitting”:
As explained on Page. 3, last paragraph, and by the reviewer, the retrofitting idea is that we first train \theta and \phi and then fix \theta to train \phi_x and \phi_y. As explained in the paper, we just optimize the first term in the cost function with respect to \theta and fix \theta_x and \theta_y when training the last two ELBO terms respectively. This is what we refer to as ‘retrofitting’. To clarify things further, we write the cost in terms of L(\theta_x, \theta_y, \phi_x, \phi_y, \phi) to clarify which parameters get used in each of the ELBO terms.
However, conceptually, we still retain the notation specifying the overall cost using \theta for succinctness, and for the possibility of performing semi-supervised learning with the objective (footnote, Page. 4), which is beyond the scope of the current paper.
2. Order of attributes:
Thanks a lot for raising this point! We have updated the paper with clarifications (Page. 1) explaining that the order of attributes does not matter.
3. Image sizes for CelebA:
The updated version of the paper contains the image sizes used for both MNIST-A and CelebA. The sizes used are the same across both the datasets, namely 64x64.
4. Details of the multi-label classifier:
Page. 15 provides more details of the multi-label classifier used for evaluating imagination on MNIST-A dataset. The multi-label classifier is a 2 layer convolutional neural network with four heads, where each head is a 1 hidden layer MLP trained end-to-end to optimize for the log-likelihood of the correct attribute label for an image.
We thank the reviewers for the thoughtful feedback and are encouraged the reviewers thought the paper was well written and clear (R1, R2, R3), addresses many important problems (R1), and makes useful contributions (R1).
We also thank the community for providing public comments on reproducibility of our work. We are glad that our conclusions were found to be reproducible and well substantiated, and will incorporate specific feedback for improving reproducibility.
We address the major points raised by the reviewers in an updated version of the paper. The key text to look at is highlighted in blue. Below we provide a summary of the major changes.
1. (R1, R2) Better explanation of qualitative results for both MNIST-A and CelebA:
The new version provides a clearer caption to Fig. 4 b) explaining the qualitative results for MNIST-A and adds new text on Page. 9 explaining the qualitative results on CelebA.
In addition, we show more qualitative results in Appendix A.9 (Page. 19, Figure. 13) comparing TELBO and JMVAE in terms of diversity.
2. (R1) Explanation of hyperparameter choices:
Thanks for raising this point. The new version adds a discussion of how the hyperparameter choices for different objectives change performance in Appendix A.6.
3. (R1) Clarification on why JMVAE generates more correct samples:
Page. 6 explains the tradeoff between correctness and coverage implicit in choices of JMVAE v.s. TELBO objectives.
In particular, while the earlier version explained why TELBO has better coverage, this version also adds the explanation for why JMVAE has better correctness for seen concepts.
4. (R3) Clarification on the order of attributes being important v.s. Not:
Page. 1 adds text to clarify that the order in which the attributes are dropped is not important and that the tree shown in Fig. 1 is just a subset of the overall DAG.
5. (R3) Details of the multi-label classifier used for evaluation:
Page. 15 provides more details of the multi-label classifier used for evaluating imagination on MNIST-A dataset.
6. We add more clarifications explaining how we pick hyperparameters (Page. 4, Page.7), typical training times for our models (Page. 7), and provide the size of the images we generate (Page. 7) (R3).
As mentioned earlier, all the changes above can be conveniently located by following text highlighted in blue across the paper. We elaborate further on these points and address specific reviewer concerns in individual replies.

Meta-review:
role: PC
data: comment: All three reviewers recommend acceptance. Good work, accept

Decision: Accept (Poster)