Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a variant of multimodal VAE model. It also argues in favour of product-of-experts approaches vs. additive mixture-of-experts ones.
The paper clearly frames the contribution within the relevant literature, the introduction is well written and the paper is well structured. Variants of multimodal VAE are also introduced (although acronyms are not made explicit, which would make the exposition clearer), and the new derivation is presented and explained.
Claims are supported by experimental results, that use MNIST, SVHN, CUB-Captions. A final discussion summarises the claimed contribution and advantages of the proposed approach.

Comments:
- Please make acronyms explicit early on to make the exposition clearer
- Figure 1 is a nice visualisation, very helpful
- Does the proposed approach generalise easily to more than two modalities?
- How well do you expect your approach to work on modalities presenting considerably different dimensionality (also in comparison with other approaches)?
- Quantitative results show comparable scorse between SVAE and VAEVAE variants. Where/why would SVAE be most beneficial/advantageous? 
- Which applications would see MoE approaches outperform PoE? I think this point is important for completing the analysis and comparison of the two approaches. Is there a case where PoE and MoE perform comparably, and why would that happen?

review_title: Good experimental defence of PoE, marginal originality

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper discusses and evaluates different generative models for multimodal data. Specifically, the authors are interested in comparing product-of-experts (PoE) to mixture-of-experts (MoE) in VAEs as ways to handle multimodality.

They also propose a novel model (SVAE) built on the PoE approach that, compared to previous models, aims to better handle missing modalities. In particular, they introduce additional networks that estimate the marginal distributions of the latent representations of the missing modalities given the observed ones.

They then evaluate multiple models on three tasks that highlight the difference of behavior in PoE and MoE-based models, concluding that PoEs are useful to model "AND" relationships in a multimodal setting.


################################################

Strong points:

-The paper is easy to follow, clearly presents the context and the different approaches. The proposed model is also elegantly designed and presented, with differences with previous methods highlighted.

-The authors seem to have provided most of the information needed for reproducibility.


Weaknesses:

-While SVAE is useful in the context of the paper as it uses explicit PoEs, the experiments don't show conclusive differences in performances with VAEVAE and VAEVAE* variants, reducing the potential impact of the novel elements of the paper. Actually, VAEVAE* (that is very close to the prior existing VAEVAE) obtains better or comparable performances except on the MNIST-split task with 1% or less paired data.
Also, uncertainty bars in the plots would be very useful, especially since there seem to be very large variations in what should theoretically be mostly monotonous curves (Figure 4 and Figure 6 left).

-The MNIST-split results are used to support the idea that PoEs are better-suited to model "AND" relationships (both modalities carry complementary information and are needed for the task). However, it seems that a system (composed of VAEVAE and the oracle) that takes as input only the top part is able to predict the correct class with an accuracy of 0.887. That tends to indicate that the evaluated task is fundamentally an "OR" task, and therefore ill-suited to evaluate the adequacy of the method to capture "AND" relationships. Can the authors comment on this potential issue?

-The paper is articulated around proving that PoE models can obtain good performances. Since VAEVAE already performs well and that the opposite view is only supported by a single publication, the importance of this particular contribution is questionable.


################################################

Score motivation:

Mainly, I believe the shown results aren't conclusive enough. They can't support the proposed model, nor the other insights of the paper.


################################################

Other question:

Can the authors comment on why MMVAE is terrible at auto-encoding an MNIST image (0.539 accuracy in table 1)? It seems very low.



################################################

################################################

Post-Rebuttal Update:

I'd like to thank the authors for their updates, the additional experiments are especially welcome. After taking into consideration the responses and the new evidence, I believe the concerns I raised still stand. Therefore, I keep the previous rating.
review_title: Comparing product-of-experts against mixture-of-experts in VAEs. Results don't seem conclusive enough.

final_score: rating: 4: Ok but not good enough - rejection
review_content: # Update
I thank the authors for their comments and answers. While they agree on some of the concerns I raised, others are still left open.
I believe they could be addressed in new major revision of the paper and I encourage authors to do so.

# Summary
This paper qualifies as a discussion around architectural choices when using variational-autoencoders (VAEs) for multi-modal learning.
The main claim is that architectures supporting the mixture-of-experts (MoEs) paradigm favor benchmarks where modalities appear in an 'OR'-relationship, while those implementing products-of-experts (PoEs) favor 'AND'-relationships.
To show this point and in 'defense' of PoE-VAEs, the paper introduces a missing-value imputation experiment over MNIST images to evaluate the second aspect.
As another contribution, the authors introduce the SVAE as a variation of a sota PoE-VAE, the VAEVAE, for bi-modal learning. Specifically, they introduce a variation that employs auxiliary encoder networks for each modality and derive accordingly a new composite ELBO loss to train them.
%
The point raised by author is reasonable, but the execution of the experimental section and little relevance of the introduced SVAE limit the value of the paper in its current state. Detailed comments follow.


# Presentation
The paper is generally understandable and well-written.
The introduction might benefit a rewriting as to gently introduce the larger scope of multimodal learning and then focusing on the more recent advancements, such as PoE-VAEs and MoE-VAEs.
For example, the first lines of the first paragraph directly jump to PoEs without providing the reader enough background.

I suggest authors to clarify that the scenario they are tackling is that of generative modeling in presence of missing data on the _inputs_ (as a modality missing is a special case of missing not at random) more than referring to it as semi-supervised learning.
I recognize that the authors follow a recent trend in the VAE-literature, but classically, semi-supervised has been referred to as the case when missing values are on the _output_, in a clear discriminative setting. 

Some typos are present, e.g., "which corresponds to ”AND” combination of the modalities and favors the MoE architecture" (should be "OR"). Clearly, they can be easily fixed by an additional proofreading pass.

# Contributions
On the one hand, the discussion of the pros and cons of PoE-VAEs and MoE-VAEs boils down to one conjecture ('AND' versus 'OR' modality fusion) that is shallowly tested in the experimental section.
Which can be greatly strengthened, see comments below.

On the other hand, the introduction of the SVAE architecture, while potentially appealing, seems a minor contribution after seeing that VAEVAE and MMVAE generally outperform it in the graphs and tables in the experimental section (with the exception of the first experiment, joint modalities and one single percentage of low supervision).
I would advice authors to explicitly say what is the benefit (maybe didactic?) of introducing the SVAE as a new model.

Furthermore, a limitation of a certain regard concerns dealing with bi-modal data only in the text and in the experiments. 
This makes unclear what is the price to pay to scale these architectures to truly-multimodal data.
For examples, Eqs. 7-1 suggest that the new composite ELBO can be extended to include uni-modal ELBO terms for each modality. However, one ordering over modalities for conditioning (according to Appendix B) shall be chosen, and it is not clear how this can influence learning and inference (in a similar fashion variable ordering influences autoregressive models).
Architecture-wise, it is not self-evident how many additional encoder components are needed for more than two modalities. If more than one per modality, then the challenges should be discussed in depth.

# Experiments
As already stated, experiments are limited to bi-modal data only.
One additional downside of the experiments is that only coherence inter-modalities is measured as a metric. Sample/modality quality is not discussed, not even reported in a qualitative way (for the exception for some samples for the CUB dataset and some MNIST image reconstructions only for SVAE in Fig.3).
I suggest authors to report the FID scores (or any other suitable variant like KID or precision-recall curves) of the joint and single modalities for the generated images to assess their quality. This is a fundamental aspect as modalities can be coherent but very far from the true data distribution or still not exactly close to the reconstruction, which is just a mode of the whole distribution.
Along this direction, one shall evaluate conditional sampling and not only reconstructions to see if the VAE have collapsed to pointwise densities.

Lastly, the CUB experiment provides some empirical evidence that is hard to evaluate or pose in the context of generative modelling.
In order to follow the MMVAE paper, the authors are decoding images not in pixel space, but in the latent space of a  ResNet-101. Then the showed images are the nearest neighbours in the training data.
This is the opposite of the generative modelling paradigm, and misleading: equivalently accurate and good-looking final images could come from a model memorizing the training set.
I advice authors to evaluate the quality of generated samples and reconstructions in the pixel space (with the metrics discussed above), alternatively, to introduce a proper decoder for the ResNet-101 embedding or not to include the above experiment at all.
review_title: A discussion on PoE-based VAEs versus MoE-based ones that can be strenghtened by deeper experiments and discussions

final_score: rating: 5: Marginally below acceptance threshold
review_content: Pros:
This paper proposed SVAE as a more general form for the previous VAEVAE model.

Cons:
1). In the subsection "Image and text: CUB-caption", the definition of \phi(\tilt{k}) is a little bit confusing. Is there a typo (\tilt{k} == k), or what is k, a constant?
2). From all the experiments listed in the paper, we can see that VAEVAE performs best. And the content is lack of analysis why VAEVAE is better than SVAE in practice. From the subsection "SVAE vs VAEVAE" we know that SVAE is a general form of VAEVAE, but it becomes less important since the general form performs worse than a special case in practice. In other words, the importance of the works is not that convinicing. At least, as a reader, we expect to see in some cases, SVAE is better than VAEVAE.
3) In equation (10), will weighted sum be a better choice?
review_title: review

Meta-review: The reviewers found this to be an interesting and clearly-written paper, but broadly agreed that it is not yet ready for acceptance. In particular, multiple reviewers felt that the experiments don't show clear benefits of the proposed SVAE approach when compared to the VAEVAE and other baselines; nor do they sufficiently back up the central claim regarding relative benefit of PoE vs MoE for either "AND" or "OR" relations. Hopefully the comments and suggestions from the reviewers, particularly regarding framing and experimental validation, will help in revising the paper.
Decision: Reject