Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a Hierarchical framework for multimodal learning HMVAE. They define modality specific latent factor as well as the shared latent factor across multiple modalities. They represent modality-specific variations using latent variables dependent on the shared top-level variable. They parameterize the posterior distribution over the shared latent variable using a mixture of experts. The modality specific latent factors are adaptive inferred with both bottom-up and top-down information. They evaluate the proposed method on the Oxford Flower and the CUB datasets with various cross-modal experiments.

main_review: The main contribution of the paper compared to the most similar baseline model MHVAE is in that they utilize the mixture of experts and can, potentially, build a deeper hierarchy.

However, there are a few concerns about the paper, the approach proposed, and the evaluation:

* On page 3, authors clarify the differences between MHAVE and HMAVE(the proposed model). However, MHVAE is also a hierarchical model, and thus, Fig.17a seems not a proper graph for MHVAE. It is not clear why the authors argue that MHVAE is “non” hierarchical model.

* In section 4, the baseline models are not consistent throughout the experiments. Though section 4.4 shows comparison to SOTA methods that have modality specific latent factors, the authors use MVAE and MMVAE in section 4.2 and 4.3, both of which have only the shared latent space without the modality specific space. I find this unfair.

* In section 4.4, the authors implement the MDVAE model by themselves rather than use one of the mentioned MDVAE model implementations. For fair comparison, it would be better to use the baseline network models or justify why that was not done.

* The top row of page 8, they claim “when using deeper hierarchies because a hierarchical inference network can explicitly capture dependencies between different levels of abstraction”. This statement is not substantiated and needs to be grounded in experiments such as the ablation study wrt the hierarchy depth.

* In Figure 6, unconditioned generation results in the first row are provided with the captions above them, inducing confusion (ie, imply conditioning, not unconditional generation). In the unconditioned case, shouldn’t g follow the prior distribution, and not be conditioned on the input, as stated in the text?  This needs clarification.

* In Figure 8 caption, all descriptions indicate (a).

summary_of_the_review: Two primary element should be resolved to provide convincing arguments about the utility of the proposed approach. (1) Consistent baseline model comparisons throughout section 4 experiments.  (2) Provide clear comparative analysis with MHVAE and show what is specifically novel (or identical to) this close competitor and substantiate improved performance in clear experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new type of model called a hierarchical multimodal VAE (HMVAE) that captures modality-specific variations using latent variables dependent on a shared top-level variable, in a manner similar to a multi-layer hierarchy. Their assumption is that modality-specific variations can sometimes depend on the structure shared across modalities which motivates their design decision to have modality-specific variables dependent on a shared top-level multimodal variable, which is in contrast to existing works on multimodal generative models that factorize into marginally independent latent variables to capture modality-specific variations (in other words not depending on a shared top-level multimodal variable).

Experimental results show promising performance on the CUB and the Oxford Flower datasets and outperform existing methods in sample generation quality and quantitative measures as the held-out log-likelihood.
main_review: Strengths:
1. The proposed method is interesting and well-motivated.
2. Experimental results are quite comprehensive, and results are promising.
3. The paper is generally well-written and clear.

Weaknesses:
1. The model hinges on the motivation that 'modality-specific variations can sometimes depend on the structure shared across modalities'. Intuitively this seems contradictory since 'modality-specific variations' in one modality are by definition independent of those in another modality, so I am not sure what it means to 'depend on the structure shared across modalities'. It would be good to formally state this assumption and test it on real-world datasets, perhaps using some information-theoretic/dependency-measure metric?
2. The results are better but the improvement could be confounded by the increased number of parameters (I believe the hierarchical model does have more params + potentially more flexibility in modeling the multimodal data). It would be important to control this more carefully when reporting results.
3. Could have some qualitative results + human evaluation results for evaluating caption generation (figure 5).
4. Would also be good to have some human evaluation results for text to image generation (figure 6).
5. The paper tries very hard to distinguish itself from other factorized or hierarchical multimodal generative models (figure 7). While I believe that there is merit to their approach, there are still many possible confounding factors in my opinion, especially regarding the issue of the number of parameters. Also, since each directional arrow in the graphical model is parametrized by multi-layer neural nets, it is not clear the exact difference between explicitly defining a hierarchy of latent variables versus defining a shallow LVM (figure 7c) but having deeper layers in the model. 
6. I still have concerns over the novelty of the approach since the main contribution is to define a hierarchy of latent variables. It is not clear when and why this works (related to weakness point 1) so the paper can be improved if there were deeper insights in this part.
7. Quality of figure 8 text can be improved.
summary_of_the_review: The paper is well-written and the results are promising, but there are concerns over the motivation and novelty as compared to existing work, as well as possible confounding factors in the experimental setup.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a hierarchical multimodal VAE to capture the heterogeneity through latent variables dependent on a shared top-level variable. CUB and Oxford flower datasets were used for performance evaluation. 

main_review: It is a very interesting paper. The hierarchical multimodal VAE can capture the realistic variations and help the decoder to share features between different modalities. 

Questions:

1. How does the number of hierarchical levels pre-defined in real scenarios? The two experiments used L(m) = 2, will that be better to set it as a tuned parameter? 

2. The evaluation metrics used in the paper (Fig 4, 5, 6) seem difficult to conclude which method performs better.  Are there any other quantitative metrics other than likelihood can be used?

3. This method is limited to images and text data. Might be an application limitation for other multimodal data. 

Minor: In figure 8, all the captions for the subfigure are the same.

summary_of_the_review: Overall, I think the idea to consider the hierarchy for the latent variables between shared and modal-specific variations to learn the representation is novel and interesting. But, I do have concerns about this method’s application in real situations of multimodal learning.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a hierarchical multimodal VAE (HMVAE) that represents modality-specific variations using latent variables dependent on a shared top-level variable. They demonstrate that the proposed approach can represent multimodal heterogeneity and outperform existing methods in sample generation quality and quantitative measures.
main_review: Strengths
* a new approach incorporates a hierarchy between shared and modality-specific latent variables. 
* Paper is generally well-written.

Weaknesses
* evaluation is restricted to dense modalities such as images and text. 
* The paper can be improved if the author can provide further insight into why defining a hierarchy of latent variables provides a better result. 
* The authors claim that a hierarchical latent representation is an inductive bias that captures realistic data variations and guides learning. It would be helpful if they demonstrate this by showing the representation of modality-specific variations with quantification.
* The authors mentioned that they found that increasing the unimodal latent sizes tends to decrease crossmodal coherence. It would be interesting to show this result.
* It would be helpful if the author can add a heterogeneity metric as their goal was identifying the representation of heterogeneity within modalities. The current evaluation metrics used in this paper are somewhat subjective and difficult to claim which method performs better. 


summary_of_the_review: The authors proposed an HMVAE where unimodal latent variables depend on a shared latent variable. 
They demonstrated that the model improves generative modeling performance on multimodal data but is restricted to dense modalities such as images and text. 
The proposed method is well-motivated but the paper can be improved if the author clearly demonstrated how a hierarchy of latent variables improves identifing modality-specific latent variables.
review_title: null

Meta-review: PAPER: This paper presents a multimodal auto-encoder architecture built on the premise that unimodal variations can be best generated when taking advantage of a shared latent space. This is operationalized by defining a hierarchical model with two primary levels: a shared structure space and unimodal variations (which could be multi-layer). 
DISCUSSION: The reviewers and follow-up discussion brought many questions and issues. The authors submitted a significantly revised version of their paper which clarified many issues and added a few extra results. While many of the reviewers’ questions were addressed by the authors, it seems that reviewers ended up not changing significantly their review scores. One fundamental concern is if the basic assumption about the shared structure is effectively the proper way to approach such generative modeling task. The experimental for image generation did not seem to support this hypothesis.
SUMMARY: While the revised version was an improvement over the original submission, improving clarity and adding some experimental measures, the experimental results did not seem to always support the main hypothesis. Human evaluation results may help in this direction.
Decision: Reject