Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: It is challenging to infer a joint representation from arbitrary subsets of multimodalities, and the state-of-the-art approaches (mixture-based multimodal VAEs) attempt to accomplish this by training to generate all modalities from a joint representation inferred from missing modalities, but the quality of modality generation is lower than that of unimodal VAEs, and this limitation is theoretically unavoidable.  Therefore, the authors propose a coordination-based model that brings the representation inferred from each modality closer to that inferred from all modalities. Experiments with diverse and challenging datasets show the advances of the proposed method.
strength_and_weaknesses: 
Strength:

- The authors point out why these existing coordinate-based models perform poorly on cross-model generation, and propose a novel model to fix the issue.
- The proposed model can mitigate the limitations in multimodal VAEs and performs well in both cross-coherence and generation quality.

clarity,_quality,_novelty_and_reproducibility: The paper is well written, and the proposed model is novel.
summary_of_the_review: Unfortunately, this paper lies outside of my field of expertise, therefore, my comment may be biased. Area chairs are suggested to seek opinions from other reviewers.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper describes recent development in Multimodal VAE in extensive details. 
Specifically, versions of the ELBO where the latent models , based on subset of modalities, are forced to be similar to
the latent model that is estimated with all the modalities is explained as Product of Experts like model. The paper goes on 
to propose a weighted sum of reconstruction loss together with Product of Experts like posterior model.
  
strength_and_weaknesses: 
The paper is very well written and easy to follow. I have the following comments and suggestions.

Comments:
- How is the unimodal posterior $q_\theta(z|X)$ computed and sampled from? The main motivation of mixture models is to be scalable, i.e., they can be used for large number of modalities just as effectively. The equations described in (5) and (7) look like something that is not going to scale very well with the number of modalities. Is it possible to show how the approach scales as the modality increase, empirically? 
- The paper reads more like a survey. It would have been much better if the Sections from Section 2. up to Subsection 3.3 are condensed and the contributions, Section 3.3 and Section 3.4, are expanded upon.
- Related works is not specialized to this work-- there is a paragraph about transformer based multimodal models.

Suggestions:
- Please include what $H(\cdot)$ actually is, this is below Eq(4). It is much better to define it there than forcing the reader to scroll through the reference. 
- Please improve result reporting in Table-1. The abbreviated dataset description and their categorization is not clear.
clarity,_quality,_novelty_and_reproducibility: The paper is very clear. Novelty is not sufficiently argued for (I am referring to the weighted reconstruction loss).
summary_of_the_review: The paper is an informative piece of work on multimodal VAE. Although, it reads like a survey paper with added suggestions. My main concern is the scalability of the approach which mixture models are meant to be efficient on.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper deals with multimodal VAEs. In particular it tries to mitigate the current limitations of models in this class. After highlighting the shortcomings of mixture-based approaches, which have been uncovered in previous work, the authors investigate the relationship between two previously proposed product-based multimodal VAEs, namely MVTCAE and MMJSD, and categorize them as coordination-based approaches. Coordination-based multimodal VAEs assume a product-of-experts joint posterior to have high-quality generated samples, and introduce specific terms in the objective to bring unimodal and joint encoders closer together, to obtain coherence between modalities. The authors propose CRMVAE, by extending the MVTCAE to include unimodal reconstruction terms in the objective, and show empirical results in multiple experiments indicating an overall improvement over existing approaches in terms of coherence and quality of generation.
strength_and_weaknesses: **Strengths**: 
- The paper investigates a relevant problem, trying to overcome the limitations of existing multimodal VAEs.
- The paper is well-written and well-structured.
- The Experiments section shows the authors have made an effort to evaluate their proposed method on a wide range of datasets, and empirical results indicate a clear improvement over previous methods.

**Weaknesses**:

- The main contribution of the authors when proposing the CRMVAE is introducing unimodal reconstruction terms in the objective. However, the rationale behind this choice seems to be rather based on intuition and lacks theoretical grounding, leaving some open questions. It is e.g. unclear why optimizing for unimodal reconstruction of each modality, i.e. reconstruction of each modality given the latent code inferred from the corresponding unimodal encoder, would improve coherence for cross-generation.
- The experimental results are extensive in the sense of models being evaluated on a wide range of datasets, but there would be additional results that would give important insights on whether certain current limitations of multimodal VAEs are truly overcome. For example, how do the models compare on generation performance, when the joint latent code is sampled from a prior? This is an important aspect when trying to learn a joint distribution of all modalities [1-2], and I think coherence across generated modalities in this setting would show if bringing unimodal and joint posteriors closer together effectively leads to encoding shared content effectively in the latent space.
- From my understanding it is unclear if the equality in section 3.4 below equation 7, central to deriving the ELBO, is justified for the considered setting. Could the authors elaborate on that? Also, it seems like the assumption could be stated more clearly: Are the integrals actually necessary for the left-hand side, or is it equivalent to assuming that the joint posterior is equal to each unimodal posterior? Can the assumption be stated in terms of a (conditional) independence relation? 
- Another important point where this work can in my opinion be improved is reproducibility. Being quite familiar with the mentioned models and experimental settings, I would have liked to replicate some of the results shown in the paper, but the authors did not share their code, which makes it hard to do so. In addition, I tried to reproduce the results for MVTCAE in the CUB Image-Captions experiment, using the public repo for the original paper, but I was not able to obtain the same performance shown in the paper for cross-generation, despite using the same settings. I think it would be helpful to see the code to understand the implementation better and reproduce the results. 
- [Minor] I believe a logarithm is missing in the RHS of the equation in Proof 1. 

**References**
- [1] Yuge Shi, et al.: Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models. NeurIPS 2019.
- [2] Thomas M. Sutter, et al.: Multimodal Generative Learning Utilizing Jensen-Shannon-Divergence. NeurIPS 2020.
clarity,_quality,_novelty_and_reproducibility: **Clarity**: the flow and exposition of the paper are clear. However, it would be preferable to have more results (e.g. in the Appendix) for a clearer understanding of model performance in the experiments (e.g. generation when latent code is sampled from a prior, image-to-caption generation). 

**Novelty**: This work analyzes the relationship between MVTCAE and MMJSD, and categorizes them as coordination-based approaches. Then proposes CRMVAE, which extends the MVTCAE by adding unimodal reconstruction terms. The authors stress that their addition is beneficial for model performance and this is confirmed in the showcased experimental results, but the introduction of these terms is backed only with an intuitive argument, and more empirical results would be helpful to see if limitations of existing multimodal VAEs are fully addressed (see point 2 Weaknesses).

**Reproducibility**: The authors did not share their code and the paper alone does not provide sufficient details, which makes it hard to reproduce their results. 

**Quality**: There are some aspects of high-quality work, such as the inclusion of a variety of experimental settings, but also some aspects in which quality could be improved, such as reproducibility.

summary_of_the_review: The paper is well-written, deals with a relevant problem, and presents promising empirical results. The authors propose a model in the class of coordination-based multimodal VAEs that extends MVTCAE, by adding unimodal reconstruction terms in the objective. However, behind this addition, there only seems to be an intuitive argument. While empirical results are promising, reproducibility can be improved, and more empirical results could give a clearer picture on whether limitations of existing multimodal VAEs are fully addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper suggests modifying the variational objective in mixture-based multi-modal VAEs by including both a uni-modal as well as a multi-modal reconstruction term. The authors also aim to present a unified perspective on mixture-based multimodal VAEs by showing that methods motived by a Jensen-Shannon-Divergence or Total Correlation objective can be viewed as mixture-based VAEs under some assumptions. Empirical results indicate the proposed approach improves the tradeoff of generative quality versus cross-coherence for different multi-modal VAEs.


---
Post-rebuttal comments:
Having read the other reviews and the authors' response, I feel that the submission is stil borderline.
The authors response clarified the comments that I had previously.
The presentation in the updated version has been improved.

---
strength_and_weaknesses: Strengths:
- The idea to include the uni-modal reconstruction terms appears well motivated.
- The empirical performance suggests that the new approach yields better generative quality (e.g., FID scores) or cross-coherence (e. g., classification accuracy) than most previous approaches.
- The paper adds some interesting connection between MVTCAE and JMVAE and mixture-based multi-modal VAEs.

Weaknesses:
- While the suggested approach improves the cross-coherence compared to previous work, the generation quality does not improve necessarily compared to some methods.
- The algorithmic change to include the unimodal reconstruction terms is somewhat small. 

Comments:
- To clarify Figure 2, do the different number of modalities mean that a different generative model is trained for different number of modalities, or is this the same generative model that has a different number of missing modalities when encoding the latents?
- Does Lemma 1 assume a MoE approximate posterior aggregation as in Lemma 2?
- Does the approach give good accuracy to classify the labels for PolyMNIST/Translated-PolyMNIST in the latent space?

clarity,_quality,_novelty_and_reproducibility: The paper is generally well written. The presentation is largely clear. However, it would be clearer to summarize exactly what constitutes the method CRMVAE: Eq. (5) that replaces the reconstruction term with eq. (7), and a MoE aggregation? 
Relevant prior work is cited appropriately.

summary_of_the_review: The suggested approach of including both uni- and multi-modal reconstruction terms is novel as far as I am aware. It appears to be a rather small adjustment algorithmically compared to previous work yet it can yield improved performance, particularly with respect to the cross-modal coherency.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper tackles the problem of learning multi-modal VAEs that can jointly represent modalities. The paper first analyzes the limitations of current mixture-based methods and studies product-based methods which can be categorized as coordination-based approaches. The paper then proposes a uni-modal reconstruction loss to account for the lack of cross-generation terms which are known to produce a looser bound when performing ELBO optimization. Results are shown across a number of datasets, showing improvements in generation quality and coherence. 

  Overall, the reviewers appreciated the importance of the problem setting, clarity of the paper (though its focus was mostly on the connections between various methods rather than the proposed method), and the overall empirical performance. The main weaknesses that reviewers agreed on is the significance/impact of the contribution (venE, P3oY), main (non-theory-based) motivation of the approach (venE, P3oY), and in some cases empirical generation performance (Jrda). The authors rebutted some of these claims, but ultimately the reviewers were still very much borderline tending towards rejection. Indeed, some of the additional experiments (e.g. label classification results and generation when the joint latent code is sampled from a prior) showed that the additional reconstruction term presents some aspects (e.g. additional normalization) which could be problematic. 

Considering that all of the reviewers had some concerns about the level of contribution of the paper (discounting the strongest score which indicated lack of expertise in the area), I recommend that the paper and approach be significantly improved and submitted to a future venue, and not accepted as is. For example, the paper mixes some interesting insights (unified coordination based framework) and then proposes a relatively small contribution. If the latter is the focus, then this should be beefed up, adding more insights and motivation for the method in addition to more convincing and thorough experimental results (including improving performance on some of the experiments suggested by the reviewers). 
justification_for_why_not_higher_score: Overall, the reviewers all agreed on borderline rejection (the reviewer with score of 8 was not familiar with this field and lacked thorough comments) due to the split focus on a unified understanding combined with a small contribution, as well as several limitations in the performance (some that came out after rebuttal).
Decision: Reject