Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper discusses a surprising failure mode of multi-modal VAEs, in which their generation quality lags behind that of unimodal VAE. A theoretically grounded explanation based on the ELBO of the multi-modal VAE is proposed to explain this gap, and extensive ablation experiments validating implications of the proposed explanation are carried out. Finally, the theory and experiments proposed in the paper have implications for a large body of research into multi-modal VAEs.
main_review: **Strengths**
* The authors do a great job covering a wide body of research on multi-modal VAEs.
* The paper proposes a theoretically grounded explanation for the empirically observed lower generation quality of multi-modal VAEs. The explanation appears sound.
* Extensive experiments on simple datasets are carried out to further support the results.
* Limitations of multi-modal VAEs discussed in the paper would be of interest of a wide audience beyond machine learning research.

**Weaknesses**
* The paper would benefit from a more detailed introduction of the datasets, especially PolyMNIST, e.g. why does it have 5 modalities and what are they?
* I was not fully convinced by the comparison between VAEs that rely on sub-sampling and those that do not. In particular, VAEs that rely on sub-sampling see less data during training compared to those that do not. Would conclusions change if VAEs that rely on sub-sampling would be trained proportionally longer?
summary_of_the_review: Fundamental limitations of multi-modal VAEs discussed and experimentally validated in the paper would be of interest of a large audience and can potentially impact many applications of these methods (e.g. in natural sciences, bioinformatics, etc). Intuition and theoretical justification of the limitations appear sound, and is supported by extensive experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The manuscript attempts to study some of the existing multi-modal VAE models, MVAE, MMVAE, and MoPoEVAE and to identify their limitations in terms of generative quality and generative coherence. The authors study the impact of sub-sampling on the generative quality and show that if the shared information among modalities cannot be fully predicted across modalities, the model would lack generative coherence. They demonstrate their findings for two multi-modal datasets, i.e., PolyMNIST and Caltech Birds. To better show the limitations of multi-modal VAEs when applied to more complex data, the authors made an augmented version of PolyMNIST, called Translated-PolyMNIST including 5 modalities. 
main_review: Multi-modal data analysis is an important topic with several real-world applications. The authors' contribution is to characterize the limitations of existing VAEs as a step towards improving these methods, which is quite valuable. However, for this conference, I expect the authors to also suggest a solution for the discussed limitations. While it is quite useful to quantify the generative model performance in the presence of multi-modal data, it is not surprising to see that when all modalities are present, the joint distribution approximation for VAEs that use subsets of modalities (MMVAE) is not as accurate as VAEs that use the joint posterior as a product of unimodal posteriors (MVAE). These are discussed by Sutter et al., ICLR2021 as well.

The paper is very well written and well organized. Both theoretical and experimental results seem valuable.  However, it would be nice if the authors could show how we can overcome the lack of generative quality due to the discrepancy, $\Delta(x,s)$ in Theorem 1 and how we should balance the “sufficient diversity” and  “redundant information” with respect to the existing modalities. I think these are critical points that need to be somehow addressed.

summary_of_the_review: The authors' contribution on formulating and quantifying the limitation of the existing VAEs for multi-modal data analysis is interesting and valuable. However, I think the authors need to at least address a few of these limitations to show how the current methods can be improved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors show theoretically that in mixture-based multimodal VAEs, sub-sampling of modalities can make ELBO less tight. They also introduced a more complex dataset, Translated-PolyMNIST and Caltech Birds with real images, to reveal the limitations of multimodal VAEs in practice.
main_review: Strengths:
- The paper is very well organized and easy to read.
- It theoretically shows a fundamental problem in mixture-based multimodal VAEs, which has not been pointed out before. This problem is intuitively convincing.
- The experiments using more complex datasets clearly support the theoretical results of this study. These results are clearly novel and important for future research in multimodal VAEs.

Weaknesses:
- The authors define models with Equation 2 as the objective as mixture-based multimodal VAEs, and they show the limitations of Equation 2 due to sub-sampling. However, this objective is a lower bound of the "true" objectives of MMVAE and MoPoE-VAE, as pointed out by the authors. Therefore, it is unclear whether the problems pointed out in this paper occur only in Equation 2 or also in these true objectives (I understand that the actual implementation in [Sutter+ 2021] uses Equation 2, but my point is that the authors should clarify the scope of this problem.).
- The use of the term "sub-sampling" in this paper is ambiguous. In this paper, modality sub-sampling is a very important concept, and the authors show that it increases the irreducible discrepancy and makes the lower bound less tight. Sub-sampling here is supposed to mean extracting a subset from multiple subsets of modalities, but the authors do not define it clearly. Moreover, they also use the term to mean ELBO sub-sampling in MVAE, which may cause confusion. The authors should clarify the meaning of this term.

Minor comment:
- Is there any reason why the usual notation for encoders and decoders in VAEs ($p_{\theta}$ and $q_{\phi}$) are reversed in this paper? (Of course, there is nothing wrong with this, as it is consistent throughout the paper.)
summary_of_the_review: Although some concerns remain about the terminology and the scope of the issue the authors are advocating, the paper makes some novel points and contributions to the study of multimodal VAEs. Therefore, I judge this paper to be above the acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper looks at the ELBO loss function for multimodal VAE, and theoretically proves a nontrivial gap $\Delta(X,S)$ between the likelihood and ELBO, from an information theory point of view. The paper claims this gap leads to quality drop of multimodal VAEs. The paper then conducts experiments and empirically examines the tradeoff between unimodal and multimodal VAEs. 
main_review: Overall, the paper is clearly written and easy to follow. The topic of this paper is very interesting and could have some implications in the area of VAE, or more broadly, generative models. 

The strengths of this paper include the following.
- The writing in most parts of the paper is clear. 
- The theory presented in section 4 is concise but very informative. 
- The experiments are extensive.

The weakness of this paper includes the following. 
- As an addition to Thm 1, it is useful to give an example that compares the ELBO and the gap $\Delta$. This should tell us which term dominates. If the ELBO dominates then your theory may not be able to explain the quality drop.
- As for the claim that the gap $\Delta$ degrades quality of multimodal VAEs compared to unimodal VAEs, I think the authors should have a paragraph discussing the following. Are you assuming that a larger gap $\Delta$ leads to worse quality? However, optimizing the ELBO can sometimes be better than optimizing the likelihood (e.g. VAE sometimes outperforms likelihood-based models such as normalizing flows), especially when the model has limited capacity - as in this case, likelihood-based models suffer from memorizing a huge amount of modes. From this view, the gap between likelihood and ELBO (for unimodal VAE) may have good inductive bias. So, could the $\Delta$ in Thm 1 have any good inductive bias? 
- Following the above point, it is also valuable to conduct an experiment where you gradually increase $\Delta$ and see if the quality monotonically drops. 
- There is no theory for coherence. In section 5.2, the description about this concept is not very clear. What does it mean by saying "a model's ability to generate semantically related samples across modalities"? Also, a formal, mathematical definition for the leave-one-out coherence also helps understanding of this part. 

Some minors:
- Eq (1): $\sum_{A\in\mathcal{S}}$ is not precise because you are in fact summing over $(A,\omega_A)$.
- Figures should be made larger for better readability.

The evaluation of this paper is based on the above points. The score may be modified based on feedback and updates from authors.
summary_of_the_review: Theory is concise but informative, and experiments are extensive. However, there are a few key things missed from the paper, as discussed in the main review. 

--------------------
Update after rebuttal

I have read the response and I will keep my score. It is a very interesting paper. I encourage the authors to add some of the discussions to either the main text or the appendix.
review_title: null

Meta-review: This paper provides an investigation into the quality of generations made by multimodal VAEs. All reviewers were in favor of accepting the paper, and there was quite a bit of detailed discussion and clarifications in the revised version of the paper which led two reviewers to raise their ratings. Overall this is an interesting contribution to the area and is an excellent fit for ICLR.
Decision: Accept (Poster)