Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper addresses the problem of formulating a well-justified ELBO for multi-modal VAEs.  The paper gives a thorough recounting of previous attempts at combining and training the per-mode posterior (approximations).  Previous workarounds involve, for instance, subsampling subsets.  This paper proposes approximating the joint posterior with a variational model and then adding the KLD to this variational posterior as an extra term on the ELBO, analogous with but an approximation to the KLD between the approximate and true posteriors that controls the gap in the ELBO.  The experiments report classification performance on MNIST + SVHN + Text and bi-modal CelebA.
main_review: Perhaps it is because I am not an expert in multi-modal VAEs, but I found this paper quite hard to read.  It presents to the reader a series of models, variational formulations, and ELBOs, and I had a hard time tracking the relationships between each.  My primary critique then is on the paper's presentation of its innovation, which takes form in Equations 10 and 11.  I really wanted to see a derivation that connects equations 10 and 11.  Since there are variational models on both sides of Equation 10, it's not clear to me how the (generative) model joint $P(X, Z)$ arises in the ELBO presented in Equation 11.  Relatedly, I found the paper's claims about the proposed framework, such as "it is unlikely that the performance of inference and generation will be degraded by the difficulty of cross-generation" (p 6) to be hard to agree with or verify as these concepts are not defined with any rigor.
summary_of_the_review: The paper's presentation---specifically, the lack of a derivation that connects Eqs 10 and 11---makes it very hard for me to assess the paper's contributions.  Thus, I think the paper needs a major revision to improve clarity.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: SMVAE aims to solve the challenge of combining multiple modalities into a coherent latent space using VAE framework. SMVAE is based on MoPoE-VAE model, the main architectural change is replacing the MoE rule with minimising the KL-divergence between the latent space distribution generated by the PoE combined experts and the joint encoder. 


(The reviewer is sorry for the slight delay for submission as he is still on vacation and marked november third as deadline for submission. In the interest of time,the following is just a short summary of my notes)
main_review: Good:
+ Linear increase in the number of parameters with growing number of modalities is a strong feature of the presented model
+ The computational complexity of evaluating the loss function is substantially reduced, which plays an important role with a large number of modalities
+ The motivation and connection to the previous work is clearly indicated, and the paper is easy to follow 

Concerns:
1. Only two datasets are considered, which have 2 and 3 modalities. Since the scalability of the computational cost of SMVAE with the increasing number of modalities is one of the main motivations for the suggested method, benchmarking it on more modalities seems important. The paper follows the experiments conducted in MoPoE-VAE study, with the exception of PolyMNIST of 5 modalities. Conducting this experiment would be crucial for supporting the main claims of the paper and completing the benchmark against MoPoE-VAE.

2. The evaluations for several experiments do not have the same results as reported in the MoPoE-VAE paper. this affects especially Table 2, where the results of MoPoE-VAE are often a lot worse than in the original publication(also, Table 2). As far as I can see, the results of SMVGON in table 3 all lie within the reported confidence intervalls

3. The cross-generation improvement is one of the main points of the paper. However it is not really supported by the presented MNIST-SVHN-Text experiments. Out of 9 “cross-generation” experiments in Table 3, where the modality of interest is not part of the input (e.g. generating text from MNIST and SVNH), only 4 result in SMVAE outperforming MoPoE-VAE and MMVAE (as indicated by bold coloring, see minor concern below)

4. Very few qualitative examples of MNIST-SVHN-Text, and none of Bimodal Celeba 


Small concerns:
-almost all tables miss proper description and caption. e.g., in Table 3 it is guesswork to figure out which of the header rows mean what.
- The result tables mark several results bold as the best, even though they only differ in the third digit. It is unclear whether those differences are significant
- Some previous multi-modal VAE models are relevant but are not part of the literature review: PoE-based SVAE model arXiv:2101.07240, which is a generalisation of MVAE and has a quadratic increase in parameters with increasing number of modalities, and VAEVAE arXiv:1912.05075 that has architectural resemblance also using the surrogate joint posterior (or a joint encoder, as introduced in JMVAE study).

Concern 1 is for me the most important one and it is unclear to me why this experiment has not been conducted. Otherwise the authors should clarify the result differences in Concern 2.
summary_of_the_review: The paper includes a novel formulation of  multi-modal VAE learning, but the experimental section does not support the main claim of scalability of training and the results of the strongest reported baseline have large differences from the ones reported in the original publication. If these problems are addressed, I can see raising my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a model for learning joint representations for multimodal data that allows inference when a few of the modalities are missing. Previous approaches addressed the missing modality problem by treating the inference network for each modality as an expert and combining the information across these experts. The proposed model claims to be useful in the following scenarios:
1) When cross-generation is difficult (and hence, MoPoE-VAE fails).
2) When the number of modalities is large.

The authors achieve this by learning a surrogate posterior conditioned on all the modalities while simultaneously optimizing the KL-Divergence between the surrogate posterior and a subset-conditioned posterior. The surrogate posterior is learnt by optimizing a variational lower bound to the log-likelihood objective.
The authors evaluate the representations learnt by the model on secondary tasks and report improved performance. The generated modalities also perform well on secondary tasks.

main_review: Strengths
-------------
- The authors do a good job of explaining the works that their paper builds upon.
- Through the experiments, the authors are able to confirm that their objective outperforms cross-generation-based objectives.

Weaknesses
------------------
- It is not clear why the ELBO and the KL-divergence (between the all-modality posterior and subset-modality posterior) need to be optimized simultaneously. A more intuitive approach would be to learn the all-modality posterior first by optimizing the ELBO and then optimize the KL-divergence term.
- The objective optimized in this paper appears to be a much weaker lower bound to the log-likelihood (due to the presence of the negative KL-divergence). 
- The authors do not do a very good job of motivating the problem of learning representations when cross-modal generation is difficult. They should have conducted a few experiments where cross modal generation is difficult to ascertain that the representations learnt on these datasets by the baselines are bad.
summary_of_the_review: Overall, the paper does a reasonable job of explaining the background for multimodal retrieval as well as conducting experiments to demonstrate their superior performance. However, the motivation for their work could be improved. The motivation for choosing the specific objective should also have been discussed in detail.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new approach to the class of scalable, multimodal VAEs. They propose to use a surrogate joint posterior which, using an additional sum of KL-divergence terms, helps keeping the posterior approximation of subsets of modalities similar to the joint posterior approximation.
The proposed method is evaluated on two different datasets, MNIST-SVHN-Text and bimodal CelebA where the proposed method shows good results.
main_review: ### Strengths
- Interesting approach to scalable, multimodal VAEs. It seems to be a reasonable trade-off between computational complexity and memory-constraints.
- Disregarding some typos and missing information (see minor comments), clear description of the proposed method

### Weaknesses
- For empirical evaluation, quantitative results on the quality of generated samples are missing as well as classification results of randomly generated samples (see Shi et al, 2019). Also, there are no results reported with respect to test set log-likelihoods. Previous work on multimodal VAEs showed that there is a trade-off between quality of generated samples and their coherence (see Sutter et al, 2021). Hence, without any quantitative results regarding the quality of samples, the assessment of a model’s performance is somehow not complete in my opinion.
- There are multiple typos in this work (e.g. eq. 10)

### Questions
- In Sutter et al, 2020, the authors propose a similar concept as yours. Using the Jensen-Shannon-divergence, the unimodal posterior approximation are forced to be similar to the joint posterior approximation. Could you share your thoughts on the similarity between the two approaches?
- Shouldn’t the results in Table 1 for the latent representation and the results in Table 2 (last row, 2 rows at the bottom) be the same numbers?
- In section 3.2, what do you mean by ‘it is unlikely that the performance of inference and generation will be degraded..’? Unlikely is not a very technical term. Therefore, it is difficult to assess this statement in my opinion.

### Minor Comments
- I had difficulties finding the definition of the surrogate joint posterior. I think it is defined as Gaussian distribution as well. I could only find it for the VGON case, but not for the VAE case of the proposed method. Is this correct?
- I would appreciate to see some qualitative results for the CelebA experiment as well.
summary_of_the_review: This is an interesting approach to scalable, multimodal learning with promising results regarding the learned latent representations and coherence of generated samples. Nevertheless, quantitative results on the quality of generated samples (including random samples) are absolutely necessary for multimodal VAEs in my opinion. Therefore, I can only give a weak reject at this point.
review_title: null

Meta-review: PAPER: This paper proposes a method to learn joint representations from potentially missing data when (1) cross-generation may be difficult, and/or (2) with large number of modalities. This is achieved by minimizing the divergence between a surrogate joint posterior and inferences from arbitrary subsets.
DISCUSSION: The reviews and discussion brought many relevant issues and concerns. The authors submitted a revised version that improved the clarity of the paper and added an important experiment with PolyMNIST. In their responses, authors also addressed some misunderstanding about JMVAE-KL. The comparison with a relatively similar work, from Sutter et al., 2020, was only mentioned in the related work, with no direct comparisons. Also, the authors did not directly address the issue of studying tradeoffs between quality of generated samples and their coherences. It should also be noted that the advantage of the proposed SMVAE is marginal when the number of modalities increases, for the latent representation experiments on PolyMNIST.
SUMMARY: Enthusiasm for this paper was not unanimous. The reviewers brought some concerns about its differentiation with priori work, such as Sutton et al., 2020, and about a more detailed analysis of the tradeoffs. While the clarity of the paper improved during the revision, a good number of issues remained. I am leaning towards rejection.
Decision: Reject