Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: I am not entirely sure whether the proof in Appendix A is correct. Especially, in the first equality in Equation (5), authors replace p(z|w) into q(z|w) (and similarly for p(z|x) and q(z|x)), but this doesn't seem correct since p(z|w) and p(z|x) are posteriors, not proposal distribution under a proposed generative model p(x|z)p(w|z)p(z).
review_title: Question on Equation (5)

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an alternative to Conditional Variational Auto-Encoders and Conditional MultiModal Auto-Encoders to perform inference of missing modalities in dataset with multiple modalities. The proposed approach is a Variational Auto-Encoder jointly on all the modalities  with additional KL divergence penalties between the approximate posterior given all the modalities and the approximate posterior given a subset of the modalities. The approach is named Joint Multimodal Variational Auto-Encoder.
The authors make a connection between this approach and the Variation of Information. It is unclear why the authors chose the JMVAE approach instead of a more elegant Variation of Information approach.
Another unaddressed issue is the scalability of the method. As far as I can tell (given that no code is provided and the specification of the encoder is missing), this approach requires a new encoder per subset of missing modalities. Right now this approach seems to scale since there are only two modalities.
The fact that the estimating the log-likelihood log(p(x)) using multiple modalities provide a lower value than with just one in Table 1 is a bit odd. Could you explain that ?
The comparison with between the representation learned by JMVAE and CVAE might be unfair given that the representation of CVAE is learned conditionally, on the label in the case of MNIST, and should therefore not consider the label in this representation. Intuitively, this representation could represent "style" as shown in (Kingma et al., 2014) in their conditional generation figure.
For CelebA, comparing log-likelihood on models that use GANs is probably not significant since GAN does not optimizes log-likelihood. 
Overall this is an interesting problem and there are also interesting ideas worth exploring further, but the execution of the paper requires more work.
review_title: null

final_score: rating: 3: Clear rejection
review_content: The proposed method of modeling multimodal datasets is a VAE with an inference network for every combination of missing and present modalities. The method is evaluated on modeling MNIST and CelebA datasets.

MNIST is hardly a multimodal dataset. The authors propose to use the labels as a separate modality that gets modeled with a variational autoencoder. The reviewer finds this choice perplexing.
Even then the modalities are never actually missing, so the applicability of the suggested method is questionable.
In addition the differences in log-likelihoods between different models are tiny, and likely to be due to noise.

The other experiment reports log-likelihood of models that were not trained to maximize log-likelihood. It is not clear what conclusions can be drawn from such comparison.
review_title: Some interesting ideas, but has major problems

final_score: rating: null
review_content: Hello, 

Modeling multimodal data is indeed an interesting problem. However, I fail to understand what motivated you to pick a JMVAE-kl model over, for example, training a VAE with random masking. You could use random binary mask b_x and b_w and then use an encoder q(z|b_x * x, b_w * w, b_x, b_w). This kind of modeling using binary masks as input is common practice in neural recommender systems and seems more principled. Did you try that approach ? How does it compare to the current path you are taking ?
review_title: Masking

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper introduces the joint multimodal variational autoencoder, a directed graphical model for modeling multimodal data with latent variable. the model is rather straightforward extension of standard VAE where two data modalities are generated from a shared latent representation independently. In order to deal with missing input modalities or bi-directional inference between two modalities the paper introduces modality-specific encoder that is trained to minimize the KL divergence of latent variable distributions between joint and modality-specific recognition networks. The paper demonstrates its effectiveness on MNIST and CelebA datasets, both in terms of test log-likelihoods and the conditional image generation and editing.

The proposed method is rather straightforward extension of VAE and therefore the model should inherent the probabilistic inference methods of VAE. For example, for missing data modalities, the model should be able to infer joint representation as well as filling in the missing modalities via iterative sampling as introduced by Rezende et al. (2014). Given marginal improvement, I am not convinced by the contribution of modality-specific encoders in Section 3.3. In addition, the inference methods introduced for generating Figure 5 looks somewhat unprincipled; I am wondering the conditional image generation results by following more principled approach (e.g., iterative sampling). Experimental results on joint image-attribute generation is also missing.
review_title: The JMVAE is rather straightforward extension of VAE.

Meta-review: This paper addresses the importance task of learning generative models of multiple modalities. There are two concerns about the paper: limited novelty, which will not have sufficient impact; ineffectiveness of evaluation. The paper extends VAEs in an interesting way, but this extension on its own does provide sufficient new insight understanding. And the log-likelihood evaluations and data sets are not enough to be convincing. As a result, the paper is not yet ready for acceptance at the conference.
Decision: Reject