Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed to use neural processes for supervised multi-modal learning, which have the abilities to estimate the uncertainty of prediction and to handle missing modalities. Specifically, a directed acyclic graph is learned for each modality in the neural process, which are then used to construct a mixture-of-graphs (MoG) to sidestep the modalities missing problem. The introduction of inducing points makes the predictive distribution tractable. Experimentally, on the label prediction and uncertainty estimation tasks, the proposed model performs well compared to the recent multimodal learning methods.
main_review: Strength:

- The problem raised in this paper is interesting. Due to the presence of missing modalities at test time, estimating the uncertainty is important in practice.

- While the neural process has been applied in a wide range of ML tasks, it is the first time to be applied to multimodal learning.


Weakness:

- The writing of this paper is terrible. It is difficult to understand to the main content of this paper by simply going through the paper from top to bottom. I spend lots of time to figure out what’s the main idea of this paper, but failed. Then, I came to one of the reference of this paper, Functional Neural Process, then everything becomes clear. The idea in Functional Neural Process is introduced much better and clearer than this submission. 

- This submission looks strikingly similar to the paper of Functional Neural Process. I can say that the underlying ideas of the two paper are almost the same, including the introduction of directed acyclic graph in the model. A slight improvement is to mix multiple graph into one (MoG) to handle the modality missing problem. However, the content relevant to this part only account for a very small proportion. Most of the techniques used in the paper have been discussed in the previous FNP paper. It is better to pay more attention to how to adapt FNP on multimodal learning. Besides, how does the performance look like with FNP in figure 4 and s1?


- To address modality missing problems, the authors propose a MOG method to model the relationship among different modalities. However, the construction of graphs A and G exactly confuses me. You first construct A_v and G_v for each modality separately, and then the whole graph A and G is the sum of all subgraphs (A_v, G_v) on different modalities. That means, graphs A and G are shaping in a block-like, and there are no edges between different views. From this perspective, there are no interactions among different modalities. The whole training process is equal to the way that trains multi models on different modalities separately, but they share the same parameters. This is a quite different training scheme compared with previous multimodal learning models, and also confusing.

- Uncertainty estimation is an important issue in multimodal learning. Ideally, the more modalities missed at test time, the higher uncertainty it is. However, there are no corresponding experiments to evaluate this hypothesis. I think it is necessary to explore how does the missing rate at test time influence the uncertainty degrees.

- Another question is how do you evaluate the model. If I understand correctly, given a training data {\bar{x}, y}, you assign each data point x_v \in \bar{x} the same label, i.e., y, and use it to maximize the ELBO. At test time, according to the predictive distribution, you predict a label y_v for each test data x_v \in \bar{x}. Since there is a unique label for \bar{x}, my question is, how do you predict the label for \bar{x}, according to the predictions y_v on different views? Maybe I miss something, but I think it needs to be clarified more clear.

summary_of_the_review: The problem investigated in this paper is. But I think the contribution of this paper is not clear and the writing of this paper should be improved. I recommend the authors to take a thorough review of the field of neural process and think about how effectively adapt the it to address the modalities missing problems.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tries to use Bayesian relational generative model for scalable multi-modal learning. They propose a class of stochastic processes that learns a graph of dependencies between samples across multi-modal data types through adopting priors over the relational structure of the given data modalities. The so-called mRNP method can address the limitations in joint posterior approximation. 
main_review: The method is interesting, and the introduction is attractive to read the following. But the part for methods are a little confusing. Moreover, it seems the authors are not clear about variational inference. The paper in page 5 shows Evidence Lower Bound (ELBO) is minimized. In fact, it should be maximized.
summary_of_the_review: It may not be so fit for iclr.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a multi-modal Relational Neural Process, which learns a dependency structure among the samples.  To integrate information from multiple modalities, mixture-of-graphs (MoG) is introduced. 
main_review: This work proposed a multi-modal relational neural process to provide a predictive distribution. Meanwhile, MoG is introduced to integrate information from different modalities. Generally, this paper is easy to read.

I have the following concerns about the paper:


   1. A literature review is lacking, especially for Gaussian processes-based multi-modal / multi-view classification methods. The discussion of the multi-view Gaussian process method should be conducted. The following papers are very relevant to this paper.

Multiview learning with variational mixtures of Gaussian processes. Knowl. Based Syst. 2020

Multi-view representation learning with deep gaussian processes.

Multimodal similarity gaussian process latent variable model.


2. Some statements in the article are biased. For example, the introduction states that “existing generative models for multi-modal learning focus on latent representation, but do not fully incorporate the label information.”  Generative models are only one of the branches of multimodal learning, and many multimodal supervised models have been developed, e.g., DeepIMV in the paper.
The proposed MoG is not convincing and novel, which just averages the graphs from different modalities. In my opinion, this contribution is weak. 

3. The experiment of the paper is not sufficient. The paper only compared the two baseline methods on two real-world datasets. At the same time, the experiment did not use large-scale datasets to illustrate the author’s claim "makes mRNP scalable to large datasets through mini-batch optimization". 

In addition, I found some problems in the writing. 
Incorrect use of semicolon “;” and colon “:” in the introduction.
The meaning of the symbol is not introduced when it first appears (e.g., φ$_{sim}$ in Eq 2).
The title of the paper is multi-modal, but multi-view is used many times in the paper.
summary_of_the_review: The article should conduct more thorough literature research, method design, and improvement of writing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a Bayesian model that learns a predictive distribution given
multimodal data with labels. The novelty of the proposed method is that it
learns a dependency graph between samples from different modalities and
introduces a new mixture-of-graphs (MoG) method to aggregate different
dependency graphs.  Experimental results on several datasets demonstrate that
the proposed method, which is fully supervised, improves the predictive
performance compared to two unsupervised approaches and one supervised
baseline. Qualitative results corroborate the quality of the learned
representations, which can be visualized as relational graphs, whereas the
baselines require additional dimensionality reduction techniques for
visualization.
main_review: **Strengths:** The integration of a dependency graph is a novel contribution to
multimodal learning and overall the empirical results look promising. The paper
is transparent about the limitations of the proposed approach, for instance,
that it is fully supervised and that the results can dependent on the
choice of a good reference set.

**Weaknesses:** First, the paper makes strong claims about previous work that
are not sufficiently well supported. In particular, claims about the
computational efficiency and the improved approximation of the joint posterior
require a better formalization and/or empirical support. Second, the
experiments include mostly unsupervised baselines, but most of the reported
metrics are based on labels, which puts the unsupervised models at a clear
disadvantage. Hence, the experiments would benefit from having more supervised
baselines.

**Computational complexity:**
The paper makes strong claims about the computational inefficiency of previous
approaches (mVAE and DeepIMV), but it is not clear how the proposed method
resolves these issues. Since these arguments are stated very prominently in the
abstract, conclusion, and throughout the paper, a formal comparison of the
computational complexities would be helpful.

**Approximation of the joint posterior:**
The paper claims that the proposed method effectively approximates the joint
posterior of multi-modal data types, in contrast to previous methods.  However,
the unsupervised baselines do not model the same posterior and therefore the
comparison is arguably not very fair. Instead, one could compare to other
supervised or semi-supervised approaches (see separate paragraph). 

**Miscalibration of the precision of experts:**
The claim about the miscalibration of experts is positioned very prominently
(e.g., in the abstract and conclusion), but there might not be sufficient
evidence to justify using this explanation to argue in favor of the proposed
approach. I am aware that the claim originates from previous work, but there,
the statement was less central and the results did not depend on this
explanation. However, the present paper seems to convert a hypothetical
explanation to a fact that is heavily relied upon throughout the paper.

**Insufficient supervised and semi-supervised baselines:** 
Almost all of the baselines are unsupervised methods, whereas the proposed
approach is fully supervised (a semi-supervised extension is discussed as an
opportunity for future work). It is questionable to frame the requirement of
labeled data as an advantage of the proposed approach compared to existing
multimodal generative models, which are unsupervised and could potentially be
extended to handle semi-supervised data (Joy et al. 2021). Hence, the
experiments seem to be lacking supervised and semi-supervised baselines and the
paper has to explain why it would suffice to consider only DeepIMV as a
supervised baseline.


**Questions and comments:**

- Why do you refer to the proposed approach as a generative model?  Is the
  proposed approach able to *generate* samples from the respective modalities
  modalities? If not, the naming makes the comparison to multimodal generative
  models (e.g., the mVAE) confusing.
- Since you mention CCA-based methods, for which there are many semi-supervised
  and multi-view or multi-modal extensions, did you consider comparing to these
  approaches?
- In the introduction it is written that "training a PoE" is difficult, which
  is confusing. It requires an algorithm to train a model that integrates a
  PoE.
- The authors frequently write that the proposed method "learns individual views
  faithfully". This statement should be made more formal, especially since the
  quality of an encoding is relative to a downstream task.
- In the introduction, it is not clear why stochasticity is a problem for
  multimodal generative models. The statement is not backed up by an
  explanation or reference.
- The relation to attention mechanisms needs to be formalized. In its current
  form, the explanation of the relationship is too superficial.
- Table 2: mRNP accuracy seems to be on a different scale.
- Figure 2: consider using t-SNE or UMAP instead of PCA for the baselines.
- Figure 3: please add values for 0% on the x-axis and standard deviations for
  all methods. Table 4 suggests a significant overlap, if we consider standard
  deviations.


**References**
- Joy, T., Shi, Y., Torr, P. H., Rainforth, T., Schmon, S. M., and Siddharth,
  N.  (2021). Learning Multimodal VAEs through Mutual Supervision. arXiv
  preprint arXiv:2106.12570.

summary_of_the_review: The paper introduces an interesting approach to a relevant problem and reports
promising empirical results. However, the paper makes strong claims in its
comparison to previous work, some of which are not sufficiently well supported.
Further, for a fair comparison, the experiments would benefit from more
supervised and semi-supervised baselines. Therefore, I tend towards rejecting
the paper in its current form.
review_title: null

Meta-review: The paper extends the FNP model to multimodal settings using the mixture of graphs. However, there are legitimate concerns about the quality of experiments, such as baselines, as the reviewers mention. For example, mRNP is supervised, and comparison to DeepIMV is not fair. I encourage the authors to address them appropriately in the next version of the paper. 

The authors can significantly improve the presentation of ideas. Please avoid making hyperbole and excessively bold statements, as the reviewers have pointed out. This way, there will be room for a better demonstration of the novel parts of the paper. For example, the authors misuse the term "generative" for the proposed mRNP. There are multiple hand-waving statements about the role of uncertainty that are not well-supported in the current draft. I believe this paper can be a good paper by addressing the reviewers' comments.
Decision: Reject