Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides a modified version of the existing MMVAE model called MMVAE+ for weakly-supervised generative learning with multiple modalities. The paper aims to overcome the trade-off between generative quality and generative coherence by suggesting having separate latent encoding for both shared and private features within each modality and a new ELBO. The authors also introduce auxiliary distributions for private features to achieve the cross-modal reconstruction. Experiments show their model achieves both good generative coherence and high generative quality in challenging experiments.

strength_and_weaknesses: Strengths：
The paper is well-written, clear and well-organized.
Experiments show their model achieves both good generative coherence and high generative quality and is robust to the size of the private space.

Weaknesses:
(Lee, Pavlovic 2021) should also be included as a baseline. 
It would be good to have an ablation study for the auxiliary distributions.
The figures can be improved:
It would be better to have an explanation on the boxes, lines and dashed lines for Figure 2.
It would be better to have an explanation for the other rows in Figure 3, why do they come in blocks?
The axis should be consistent in Figure 5. In (b),  lower is better but 0 is at the top, which confuses the readers.

clarity,_quality,_novelty_and_reproducibility: The paper is written with good clarity in terms of how the ideas and model formulations are conveyed to the readers. 

Extensive experiments justify the problems the authors raise and the effectiveness of the solution they provided.

Although with good experiments and results, the paper lacks contribution and novelty to the problem and the solution, as factorizing subspaces for private and shared features of different modalities and introducing cross-modal reconstruction loss have been implemented in (Lee, Pavlovic 2021) paper, which should also be included as a baseline.  Comparing Eq (4) with (5),  (5) is adding the sum(n!=m) term in (4) again, with the auxiliary distributions. It would be good to have an ablation study for the auxiliary distributions to show the novelty.

summary_of_the_review: Although the paper is well written and the arguments are well justified, the paper lacks technical and empirical novelty to contribute as the problem and solution has been raised by an existing publication. 

-----------------------------------------------------------------------
Post rebuttal comments:

After reading the rebuttal and the other reviews, we raise the score to 6.

As one of the contributions is the auxiliary distribution, it would be interesting to see how different choices of auxiliary distributions would affect the experiment results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a new multi-modal VAE model which includes modality-specific latent variables, as well as shared latent variables with the latter encoded through a mixture-of-experts model.  The new variational bound allows to utilize private latent variables that do not hinder cross-modal coherence and generation, in contrast to previous work where this does not hold in a robust way. Empirical results against different multi-modal VAEs for different benchmark datasets illustrate that the approach improves generative coherence and generative quality.

----
Post-rebuttal comments:

Following the response from the authors and the other reviews, I keep my accept recommendation. 
The authors have addressed my question as to how the (conditional) generative performance changes when additional modalities are available. The new experiments indicate that generative performance does not decrease, in contrast to some previous work.
The additional experiments to address comments from the other reviewers also improve the experimental validation of the suggested approach.

----
strength_and_weaknesses: Strengths:
- Previous work on multi-modal generative models have struggled to yield both generative coherence and generative quality. The suggested approach illustrates an empirical improvement for both criteria on challenging multimodal datasets. 
- The suggested variational objective is interesting and new, as far as I am aware. It offers a different approach to include modality specific latent variables compared to previous work that appears more robust.

Weaknesses:
- It is not clear to me if the gap between the variational bound and the log-likelihood does increase with increasing modalities (as in Daunhawer et al., 2022)? 
- Do additional modalities compromise the generative performance? It would be helpful to see how the generative/cross-coherence performance varies empirically, for instance in the PolyMNIST experiments with different numbers of modalities.

Actionable feedback:
- The proportionality following eq. (4) is not clear to me.


Comments:
- It appears that the auxiliary prior distributions could be quite general. Is there a specific reason why they have been chosen to be zero centered Gaussians? Does the tightness of the bound relative to the log-likelihood depend on the KL divergence between $p$ and $r$ or $q(w_m|x_m)$ and $r(w_m)$, and could I improve generative quality, for example, by using implicit densities for r(as I do not need to evaluate their density)?


clarity,_quality,_novelty_and_reproducibility: The paper is well written. The presentation is clear with literature cited appropriately.
summary_of_the_review: The paper suggests a novel variational bound for multi-modal VAEs that seems well motivated and appears to yield good empirical results.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The work presented in this paper addresses the important problem of multimodal representation learning. Following a large body of work in this domain, the proposed approach builds on an extension of a Variational Auto Encoder (VAE) such that it can encode and decode multiple modalities. The crux of the idea presented in this paper is to partition the latent space into modality-specific and shared latent variables, making sure that the modality-specific latent are not “polluted” by the share information. This is achieved through a modification of the ELBO that originates from the work of Shi et al. ‘19, that uses a mixture of experts approach to obtain the joint approximate posterior. By decomposing the ELBO following Shi et al. ‘19, they observe that for a given modality, the portion that pertains to all other modalities should be trained by assuming a learned “pseudo” prior for their latent variables.
Experimental results use mainly two datasets: PolyMNIST, which is a variant of MNIST including background images to simulate multiple modalities, and the CUB dataset, which contains two modalities: images and captions. In their experiments, the authors rely on standard metrics from the literature (or their reinterpretation) including generative coherence (both conditional and unconditional) and generative quality (measured through the lenses of the FID score). Experiments serve the purpose of illustrating that the proposed method strikes a good balance between the two objectives (coherence and quality), which is a pain point of previous work.
strength_and_weaknesses: Strength:

* This work follows a line of research that tackle an important problem, that of multimodal representation learning

* The paper is well written, clear, and easy to follow, and builds on many previous work, with the added value of clarifying some aspects of prior work that where not completely clear in the original papers

* Experiments are well conceived 

Weaknesses:

* The idea presented in this work (to the best of my understanding) can be summarized as follows: shared information should not pollute modality-specific latent information. To ensure this, a variant of the mixture of expert ELBO is derived, by introducing learnable pseudo-priors that induce the reconstruction of a given modality to rely solely on its modality-specific latent space. A natural question is as follows: since you assume independence between shared and modality-specific latent spaces, and further you constrain modality-specific information to be solely responsible for reconstruction, then why not training individual VAEs, one per modality, then a multimodal VAE to learn a shared representation, and then “stitch” together all latent spaces?
In other words, what I miss is a naive baseline to double check that the idea presented in this paper is a valid approach when compared to a simpler method.

* I find Expression (5) in the main paper not very clear. In a previous workshop paper that presents the very same idea and the very same method I find the use of a tilde on the “other-modalities” private latent variables easier to understand and more clear. On the good side, I think the textual explanations of the ELBO are much more useful in this version of the paper.

* In the experiments, unconditional coherence seem to be a more difficult objective to attain, as was also shown in prior works. In this case, we usually sample from the prior on the latent space, and only rely on decoders to generate each modality. Then we check the coherence of each generated modality. 1) It would be useful to clarify how does this work when you have a latent space that is segregated. I assume you take the prior of each modality-specific latent, and the shared prior, but it would be good to state this clearly (in the appendix). Why do you only have results on PolyMNIST for this challenging metric, and not for CUB? 
Moreover, when we look at table 3 in the appendix (for PolyMNIST), we can see that the proposed method really struggles to achieve high generation quality (there, the MVAE approach based on PoE works better). It is also interesting to note that additional “tricks” to improve base models (importance sampling, double reparametrization), seem to hurt generation quality, which is not expected.

* In the experiments, when we look at “competitors”, we learn that if the capacity is sufficient, then the modality-specific encoder can learn all the information (both shared and private) such that an adverse effect of a shortcut appears. But if all information is stored in the modality-specific latent space, and this is true for each modality, then wouldn’t this be enough to achieve coherence? Aren’t we simply looking at the “fallacy” of the MoE approach that, by construction, samples only one of the components for the generation?

* Fig 5 raise several questions. 
** Fig 5b indicates latent classification accuracy, obtained by training a simple linear model on “freezed” encoders that produce latent features. It seems counter intuitive to state that lower accuracy is better. This is because the goal is to show that if we only use modality-specific encodings we should not be able to classify correctly, if the hypothesis of no shared information being present in those latent spaces. Beside the fact that I am not convinced that this metric (first introduced in Shi et al ‘19) is useful to state anything about the latent space natural segregation, I think that comments to this figure should be more informative. 
** Fig 5 c: I assume this figure is for the conditional generation right? Since there are works that criticize the use of FID score to assess the generative quality [1], I wonder why we do not have a table with likelihood values, or Bit per Dimension, which can also be useful to assess the quality of the model.
** Overall, from the three sub-figures in fig 5, we can remark that mmJSD is a close competitor to MMVAE+. I think this should be acknowledged and commented.

* The last paragraph before section 4.3 claims for an optimal balance between coherence and generation quality. How can you claim optimality?

* In section 4.3, qualitative results for the CUB dataset, in terms of coherence, use an original metric discussed in the appendix. This is an important detail that I think should be discussed at least a minimum in the main paper. Indeed, the proposed metric is somehow questionable and should be put upfront for the reader to properly understand. In my opinion, it is not clear why you could have not used the same technique used in Shi et al ‘19, by taking the learned representation of your ResNet, in a similar way as if you used a pre-trained network. The introduction of artificial captions and the count of correctly colored pixels suffers from the problems stated by the authors, e.g. counting only the background color. Using the top-2 most frequent pixel colors might work in some cases an not others, so I am not sure how much can we rely on this coherence metric. Also, as commented above, this is conditional coherence, which seems an easier task to achieve than unconditional coherence. Do you have any comments why, and why didn’t you show unconditional coherence metric fo CUB?


[1] @misc{FID_critics,
  url = {https://arxiv.org/abs/2203.06026},
  author = {Kynkäänniemi, Tuomas and Karras, Tero and Aittala, Miika and Aila, Timo and Lehtinen, Jaakko},
  title = {The Role of ImageNet Classes in Fréchet Inception Distance},
  publisher = {arXiv},
  year = {2022},
}

clarity,_quality,_novelty_and_reproducibility: The paper reads very well, it is clear that it presents high quality work and a thorough experimental campaign.

On the novelty, one could comment that the presented method is a variant (eq 5) of known methods, and that the key observation of trying to avoid “pollution” of private latent spaces is not new. However, I think the derivation to be correct, and — contrary to some previous work — the authors spent effort and energy in proving that the proposed ELBO is indeed a valid ELBO. In a previous version of this work, appeared as a workshop paper in ICLR 2022, the appendix show the proofs for Lemma 2 which I found interesting, because it attempts at showing that the parameters of the private encoders do not depend on cross-modal reconstruction. I wonder why in this version that lemma has been omitted.

Concerning the reproducibility of the work, I think this is somehow easy, as the key idea consists in a somehow “innocent” modification of the ELBO of the MMVAE approach. As such, this is a good point for this work.
summary_of_the_review: In summary, this work is well presented, well written and the motivation is clear. I think this line of work presents the important trade-off between coherence and generation quality that affects VAE-based method for multimodal representation learning, and this is commendable.
I have spotted some problems that can be summarized as: 1) given the key idea in this work, it would be possible to conceive a naive baseline to make sure that the claims presented in the paper are indeed correct, and this is missing, 2) Experiments seem not to be conclusive about the superiority of the proposed method with respect to state of the art. There cases in which this seems true, and others in which the “optimality” of the proposed solution cannot be verified.

Lastly, since this work is not completely novel, and the strength are not on the methodological contributions, I would advice the authors to improve the experimental section, by considering at least one additional dataset [2], or [3].

[2] Vasco M, Yin H, Melo FS, Paiva A. Leveraging hierarchy in multimodal generative models for effective cross-modality inference. Neural Netw. 2022 Feb;146:238-255. doi: 10.1016/j.neunet.2021.11.019. Epub 2021 Nov 24. PMID: 34906760.


[3] @misc{https://doi.org/10.48550/arxiv.2107.07502,
  url = {https://arxiv.org/abs/2107.07502},
  author = {Liang, Paul Pu and Lyu, Yiwei and Fan, Xiang and Wu, Zetian and Cheng, Yun and Wu, Jason and Chen, Leslie and Wu, Peter and Lee, Michelle A. and Zhu, Yuke and Salakhutdinov, Ruslan and Morency, Louis-Philippe},
  title = {MultiBench: Multiscale Benchmarks for Multimodal Representation Learning},
  publisher = {arXiv},
  year = {2021},
}


============================

Post rebuttal comments

============================

I've read the various discussions in the rebuttal, and find the new results compelling in verifying the merits of the proposed method. I will update my score from 5 to 8.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a new multi-modal VAE model which includes modality-specific latent variables, as well as shared latent variables with the latter encoded through a mixture-of-experts model. The new variational bound allows to utilize private latent variables that do not hinder cross-modal coherence and generation, and at the same time avoid shortcuts for generation. Empirical results against different multi-modal VAEs for different benchmark datasets illustrate that the approach improves generative coherence and generative quality.

Strength:
The method is well-motivated and the derivation/results are convincing. The authors also addressed the review comments well.

Weakness:
There are some references missing by the paper (a common issue for this specific field), and more detailed comparisons with missing references are desired.
justification_for_why_not_higher_score: While the solution for avoiding the shortcut is smart, after all it is a relatively small modification of the baseline approach.
justification_for_why_not_lower_score: The paper proposes a simple yet effective solution to the challenging issue of learning private and shared variables for multi-view data. It makes a contribution to the literature.
Decision: Accept: poster