Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The problem of multi-modal regression is considered in this paper and a method called Mixture of Normal-Inverse Gamma (MoNIG) is proposed. In the proposed method, an approximation is adopted to combine predictive normal-inverse gamma distributions from different modalities into one normal-inverse gamma for tractable uncertainty calculation.
main_review: The paper aims to develop a new method for trustworthy multi-modal regression which is an important problem. The performance is compared with Gaussian and Evidential Regression with concatenated features/hidden layers on two datasets and on a multimodal sentiment analysis with several baselines. The results show improvements over the baselines.

Concerns:

The methodology of the paper is a straightforward combination of existing techniques, Deep Evidential Regression, and the summation operation for NIG distributions. Thus, the methodological contribution of the paper is not very significant.

The NIG summation operation is originally introduced for Bayesian linear regression on large datasets, which is exact there, but it is not clear in the paper how valid the approximation is for the paper’s setup. 

For the experiments, simple baselines which are fitting GS or EVD to each modality and then averaging them, simple or weighted, are not considered.

Results in Table 7 suggest that the performance of the method still deteriorates even though noise is only added to one modality. So, the discussion in the paper regarding robustness to modality corruption can be adjusted accordingly.

None of the experiments contain error bars, so it is unclear how significant are some of the results, especially those with closer numbers.

The paper has some issues in terms of clarity. The provided details are not enough to reproduce the results. In Tables 5 and 6 the baselines do not have references. The details are not provided for the ablation studies.

Additional analysis like Figure 4 can also be done for the baselines for comparison.


--Update--

Thanks to the authors for their responses. The authors have clarified what Table 8 shows (I assume the weights are per sample) and provided some error bars which address some of my concerns. Hence, I have increased the score by 1 point. 

My major concern about the work remains up to some extent. While the insight of combining existing techniques for multimodal regression is new, there does not appear to be any significant technical challenges involved to make them work here. Additionally, in my view, the paper has room for improvement and in its current form does not properly answer why the uncertainties from applying the NIG summation in the paper’s setup are valid and reliable.

A few other suggestions to strengthen the experiments:

-Calibration curves of the estimated uncertainties can be included

-Modality and “pseudo-modality” specific uncertainties can be compared systematically with and without the NIG summation. Currently, there is no comparison in this regard.


limitations_and_societal_impact: The authors have not discussed the limitations and societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper tackles a shortcoming of existing multimodal learning methods that mainly focus on the integration of the information from different sources for prediction but ignore the confidence of prediction. The authors propose a Mixture of NormalInverse Gamma distributions (MoNIG) algorithm which estimates uncertainty for multimodal learning. Their model estimates uncertainty for each modality and also in corrupted modality settings.
main_review: Strengths:
1. The problem tackled is an important one and an important shortcoming in current work.
2. The proposed method is a nice extension of uncertainty estimation for single modality regression problems using Normal-inverse Gamma Distributions into one for multimodal problems using a mixture of distributions.
3. Experimental results are quite comprehensive on 3 sets of tasks spanning a few multimodal datasets.
4. Experimental results on prediction tasks are strong and show some improvement over current methods.
5. Experimental results also show that the model can work well in the presence of noisy modalities.

Weaknesses:
1. There can be more analysis of when the proposed method fails and other potential shortcomings of their approach.
2. There should be more analysis on what types of multimodal inputs and modalities the model estimates uncertainty for, including visualizations and qualitative examples.
3. The paper can benefit from some additional analysis and discussion of the additional training time and space complexity resulting from their model.
4. There should be some reference to recent work in learning robust multimodal representations [1,2].
5. The model currently handles multimodal interactions using a very simple late-fusion style mixture of distributions - how can the method be extended to more fine-grained multimodal representation learning methods such as those in Table 5 (e.g. RMFN, MFM etc.)
6. There is not enough evaluation to show that the model has accurately estimated uncertainty apart from some visualizations. How do you evaluate whether your model has estimated uncertainty correctly? Are there any metrics for judging this? I think this might be a good reference: https://github.com/uncertainty-toolbox/uncertainty-toolbox

[1] Liang et al., Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization. ACL 2019

[2] Lee et al., Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors. 2020

Not a weakness, but a general suggestion: The authors can take a look at https://arxiv.org/abs/2107.07502 for a wide suite of multimodal methods and datasets that they can test their method on.
limitations_and_societal_impact: I think the authors can say a bit more about the societal impact of multimodal models, including their applications in robotics, healthcare, education etc.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper focuses on an important problem in multimodal learning - current algorithms ignore the confidence of integration and prediction. 
main_review: This paper focuses on an important problem in multimodal learning - current algorithms ignore the confidence of integration and prediction. The authors smartly use Mixture of Normal Inverse Gamma distributions (MoNIG) to estimate uncertainty for multimodal learning. The model estimates uncertainty for each single modality and the final prediction as well. Experiments on both synthetic and real datasets well support the assumptions.

Strengths:
- Trustworthy multimodal learning is important and interesting, and the motivation of the paper is very clear and strong. 
-The introduced pseudo modality is interesting and necessary, which enhances the sufficient multimodal interactions beyond explainability.
-Both the detailed theoretical analysis and experimental results strongly support the motivation. Especially, the uncertainty estimation and ablation study provide strong evidence to show which parts contribute to the improvements.
-The writing and organization are good making the paper quite easy to follow.

Minors：
-It is an elegant model for trustworthy multimodal regression. Can the model be applied or extended for classification task? It will be better if the authors could provide related discussion.
- The proposed method introduces uncertainty estimation, so what about the computational complexity of the proposed model compared with existing multimodal algorithms?

limitations_and_societal_impact: Minors：
-It is an elegant model for trustworthy multimodal regression. Can the model be applied or extended for classification task? It will be better if the authors could provide related discussion.
- The proposed method introduces uncertainty estimation, so what about the computational complexity of the proposed model compared with existing multimodal algorithms?
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a Mixture of Normal-Inverse Gamma distributions algorithm to estimate uncertainty in principle for adaptive integration of different modalities.

This research problem proposed in this paper is very important, and the author proposed some techniques and conducted some experiments.
main_review: This paper is innovative to some extent. In fact, it is difficult to measure the uncertainty of combining the information from different modalities.  

The writing of the paper needs further polishing. Some details of the paper are not detailed. For example, Figure 1 and Figure 2 are both confusing.
limitations_and_societal_impact: This paper does not clarify the limitations of the proposed method. 

In fact, the method proposed by the author has some assumptions about the distribution of the data. The rationality of the assumptions needs to be explained.
review_title: null

Meta-review: In this paper, the authors propose a new regression method for multi-modal (e.g., image and text) data. More specifically, the authors extend the Deep Evidential Regression method for a multi-modal regression problem. The method depends heavily on the existing methods. However, the formulation is new and the fusion strategy has some merit. Thus, I also vote for an acceptance. For the camera-ready version, I expect authors to revise the paper based on the reviewer's comments. 
Decision: Accept (Poster)