Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions
Authors: Huan Ma
Zongbo Han
Changqing Zhang
Huazhu Fu
Joey Tianyi Zhou
Qinghua Hu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Multimodal Regression
Trustworthy
Uncertainty
Evidential

Abstract: Multimodal regression is a fundamental task, which integrates the information from different sources to improve the performance of follow-up applications. However, existing methods mainly focus on improving the performance and often ignore the confidence of prediction for diverse situations. In this study, we are devoted to trustworthy multimodal regression which is critical in cost-sensitive domains. To this end, we introduce a novel Mixture of Normal-Inverse Gamma distributions (MoNIG) algorithm, which efficiently estimates uncertainty in principle for adaptive integration of different modalities and produces a trustworthy regression result. Our model can be dynamically aware of uncertainty for each modality, and also robust for corrupted modalities. Furthermore, the proposed MoNIG ensures explicitly representation of (modality-specific/global) epistemic and aleatoric uncertainties, respectively. Experimental results on both synthetic and different real-world data demonstrate the effectiveness and trustworthiness of our method on various multimodal regression tasks (e.g., temperature prediction for superconductivity, relative location prediction for CT slices, and multimodal sentiment analysis).

Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions
Huan Ma
Zongbo Han"
College of Intelligence and Computing Tianjin University Tianjin, China
College of Intelligence and Computing Tianjin University
Tianjin, China
Changqing Zhang!
Huazhu Fu
College of Intelligence and Computing Tianjin University Tianji,, China
Institute of High Performance Computing A*STAR
Singapore
Joey Tianyi Zhou
Qinghua Hu College of Intelligence and Computing Tianjin University Tianjin, China
Institute High Performance Computing A*STAR Singapore
Abstract
Multimodal regression is fundamental task, which integrates the information from different sources to improve the performance of follow-up applications. However, existing methods mainly focus on improving the performance and often ignore the confidence of prediction for diverse situations. In this study, we are devoted to trustworthy multimodal regression which is critical in cost-sensitive domains. To this end, we introduce a novel Mixture of Normal-Inverse Gamma distributions (MoNIG) algorithm, which efficiently estimates uncertainty in principle for adap- tive integration of different modalities and produces a trustworthy regression result. Our model can be dynamically aware of uncertainty for each modality, and also ro- bust for corrupted modalities. Furthermore, the proposed MoNIG ensures explicitly representation
epistemic and aleatoric uncertainties, respectively. Experimental results on both synthetic and different real-world data demonstrate the effectiveness and trustworthiness of our method on various multi- modal regression tasks (e.g.. temperature prediction for superconductivity, relative location prediction for CT slices, and multimodal sentiment analysis
Introduction
There plenty of multimodal data involved in the real world, and experience the world from different modalities [] For example, the autonomous driving systems are usually equipped with multiple sensors collect information from different perspectives [2]. In medical diagnosis (3). multimodal data usually come from different types of examinations, typically including a variety of clinical data. Effectively exploiting the information of different sources improve learning performance is standing and challenging goal in machine learning.
"Equal contribution.
1 Corresponding author. Please correspond to: zhangchanging@@jjuuddu..
Code: http:///gthhhhommmMHHHaanAAAAa
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Most multimodal regression methods 5 6 usually focus on improving the regression perfor- mance by exploiting the complementary information among multiple modalities. Despite effective- ness, it is quite risky for these methods to be deployed in cost-sensitive applications due to the lack of reliability and interpretabilityy One underlying deficiency is that traditional models usually assume the quality of each modality is basically stable, which limits them to produce reliable prediction, especially when some modalities are noisy or even corrupted [8] 9. Moreover, existing models only output lover-confident predictions [10,11, [ل]. which can not well support safe decision and might be disastrous for safety-critical applications.
Uncertainty estimation provides a way for trustworthy prediction [22 13 The decisions made by models without uncertainty estimation are untrustworthy because they are prone to be affected by noises or limited training data. Therefore, it is highly desirable to characterize uncertainty in the learning for Al-based systems. More specifically, when a model given an input that has never been seen or is severely contaminated, it should be able to express "I don't 't know". The untrustworthy models are vulnerable to be attacked and it may also lead to wrong decisions, where the cost is often unbearable in critical domains [14].
Basically, it can endow the model with trustworthiness by dynamically modeling uncertainty. There- fore, we propose a novel algorithm that conducts multimodal regression in a trustworthy manner. Specifically, our proposed algorithm is unified framework modeling uncertainty under fully probabilistic framework. Our model integrates multiple modalities by introducing a Mixture of Normal-inverse Gamma distributions (MoNIG), which hierarchically characterizes the uncertainties and accordingly promotes both regression accuracy and trustworthiness. In summary. the contribu- tions of this work include: (1) We propose a novel trustworthy multimodal regression algorithm. Our method effectively fuses multiple modalities under the evidential regression framework equipped with mopelity-specific uncertainty and global uncertainty. (2) To integrate different modalities, a novel MoNIG is designed to be dynamically aware of mopality-specific noise/corruption with the estimated uncertainty, which promisingly supports trustworthy decision making and also significantly promotes robustness well. (3) We conduct extensive experiments on both synthetic and real-application data, which validate the effectiveness, robustness, and reliability of the proposed model on different multimodal regression tasks (e.g.. critical temperature prediction for superconductivity [15], relative location of CT slices, and human multimodal sentiment analysis).
2
Related Work
2.1 Uncertainty Estimation
Quantifying the uncertainty of machine learning models has received extensive attention [66 17).
especially when the systems are deployed in safety-critical domains, such as autonomous vehicle control [18] and medical diagnosis 33. Bayesian neural networks 19 20 model the uncertainty by placing a distribution over model parameters and marginalizing these parameters to form a predictive distribution. Due to the huge parameter space of modern neural networks, Bayesian neural networks are highly non-convex and difficult in inference. To tackle this problem, [21] extends Variational Dropout [22] to the case when dropout rates unbounded, and proposes way to reduce the variance of the gradient estimator. A more scalable alternative way is MC Dropout
23 which
is simple to implement and has been successfully applied to downstream tasks [14 24 Deep
(25) have shown strong power in both classification accuracy and uncertainty estimation. It is observed that deep ensembles consistently outperform Bayesian neural networks that are trained using variational inference [10]. However, the memory and computational cost are quite high. For this issue, different deep sub-networks with shared parameters are trained for integration [26]. Deterministic uncertainty methods are designed to directly output the uncertainty and alleviate the overconfidence. Built upon RBF networks, [27] is able identify the out-of taat-o-dditributiin samples. (88 introduces new target criterion for model confidence, known as True Class Probability (TCP), to ensure the low confidence for the failure predictions. The recent approach employs the focal loss to calibrate the deep neural networks 29 (30 places Dirichlet priors over discrete classification predictions and regularizes divergence to a well-defined prior. Our model inspired by deep evidential regression [31] which is designed for single-mods modal data.
ensembles

---

2.2
Multimodal Learning
Multimodal machine learning aims to build models that can jointly exploit information from multiple Existing multimodal learning methods have achieved significant progress by
modalities [ 32).
integrating different modalities at different stages, namely early, intermediate and late fusion [T] The early fusion methods usually integrate the original data preprocessed features by simply concatenation 34 35]. The intermediate fusion provides more flexible strategy to exploit multimodal data for diverse practical applications 4 36
33
37
38
40 41 In late fusion, each 39
modality is utilized to train a separate model and the final output obtained by combining the predictions of these multiple models [5] 37
Although these multimodal learning approaches exploit the complementary information of multiple modalities from different perspectives, basically they are weak in modeling uncertainty which is important for trustworthy prediction especially when they are deployed in safety-critics domains.
3 Background
Regression has been widely used across a wide spectrum of applications. Given a dataset D {x,, 3/ } one principled way for regression is from a maximum likelihood perspective, where the likelihood for parameters w given the observed data is as follows:
ply Day w) II plus w,x:).
In practice, widely used maximum likelihood estimation is based on Gaussian distribution, which assumes that the target drawn from Gaussian distribution. Then we have the following expression of likelihood function:
N II N (ys (x,,,),
ply D2. D2. W.
(2)
The mean and variance indicate the prediction and corresponding predictive uncertainty, while the predictive uncertainty consists of two parts [42]: epistemic uncertainty (EU) and alcatoric uncertainty (AU). To learn both the alcatoric and epistemic uncertainties explicitly, the mean and variance assumed to be drawn from Gaussian and Inverse-Gamma distributions BI], respectively. Then the Normal Inverse- Inverse-Gamma. Gamma (NIG) distribution with parameters (6, 7, a, 3) can be considered higher-order conjugate prior of the Gaussian distribution parameterized with 0 = (um; o²):
y/ N (N,
N' (ò, 00,)""=)), o² 1- (a,3). 3),
(3)
where r(-) is the gamma function. In this case, the distribution of takes the form of a NIG distribution a,,
3
23 + 7 14)
p(p, 22 8, 8.
exp
(4)
I(a)
2T
20
where 8 R, > 0, 1, and 0.
During the training stage, the following loss is induced to minimize the negative log likelihood loss:
1
LL (w) log
log (92) a 2) log ((v 8) + 22) + log V
(5)
where Ω 23311 + 7) and
T(a) r(a+1)
The total loss, L(w), consists of two terms for maximizing the likelihood function and regularizing evidence:
((w) LLL + (w")
6
where (Rw the penalty for incorrect evidence (more details are shown in the supplement), and the coefficient À 0 balances these two loss terms.

---

Vidio
Audio
Villa
Audio
Text
Andiu
NIG
Pseudo mocality
Prediction
Output
NIG
NIO1
NIG
ninstoric
Outpur
Ontpal ((0yge1)
MoNIG
High anight
eptatemic
(a)
(b)
Figure 1: Strategies for multimodal regression. Feature fusion (a), decision fusion (b), the proposed MoNIG (c), and an instance of MoNIG on the human multimodal sentiment analysis task (d).
For multimodal regression 55 some approaches have achieved impressive performance. How- ever, existing multimodal regression algorithms mainly focus on improving the accuracy and provide deterministic prediction without the information of uncertainty, making these models limited for trustworthy decision making. For this issue, we develop a novel algorithm which can capture the modality-specifi uncertainty and accordingly induces a reliable overall uncertainty for the final output. Beyond trustworthy decision, our approach automatically alleviates the impact from heavily noisy or corrupted modality. Compared with the intermediate fusion effectively distinguish which modality is noisy corrupted for different samples, and accordingly can take the modality-specifte uncertainty into account for robust integration.
4
4 44, our method can
Trustworthy Multimodal Regression
In this section, we introduce the proposed algorithm fusing multiple modalities at both feature and predictive distribution levels. Specifically, we train deep neural networks to model the hyperparame- ters of the higher-order evidential distributions of multiple modalities (and the pseudo modality) and then merge these predicted distributions into one by the mixture of NIGs (MoNIG), In the following subsections, we first provide discussion about existing fusion strategies and ours in multimodal regression. Then principle. Finally, we elaborate the overall pipeline of our model in training and define the epistemic uncertainty and alcatoric uncertainty respectively.
introduce MoNIG to integrate NIG distributions of different modalities in
4.1 Overview of Fusion Strategy
Im=1 Vi J. where x the input feature vector of the m-th modality of the i-th sample, y4 is the corresponding target, and M is the total number of modalities, the intuitive goal is to learn function fm for each modality reaching the following target: min Σm=1 (Xm, w..) - While to model the uncertainty, assume that the observed target y drawn from Gaussian distribution, i.c., y M(Hm oZ.). Meanwhile, inspired by [3I, as shown in Eq. we assume that the mean and variance are drawn from Gaussian and Inverse- Garmma distributions, respectively.
Consider the following multimodal regression problem: given dataset D = {xx""
Considering multimodal fusion, there are typically two representative strategies, i.e., feature fusion and decision fusion. Feature fusion (Fig. [a)) (Fig-[a)) jointly learns unified representations and based on which conducts regression. Although simple, the limitation is also obvious. some modalities of some samples are noisy or corrupted, the algorithms based on this strategy may fail. In other words, these methods do not take the modallty-sspeciffe uncertainty into account and can not be adaptively aware of quality variation of different modalities for different samples. The other representative strategy is known as decision fusion (Fig. [(b) which trains one regression model for each modality, and the final prediction is regarded as function of the predictions of all modalities, Although flexible, is also risky for the cases that each modality is insufficient to make trustworthy decision due to partial observation.
We propose MoNIG, which can elegantly address the above issues in a unified framework. Firstly, we explicitly represent both modality and global uncertainties to endow our model with the ability of adaption for dynamical modality variation. The modality-specific uncertainty can dynamically guide the integration for different samples and the global uncertainty represents the

---

uncertainty of the final prediction. Secondly, we introduce the feature level fusion producing a pseudo modality to make full use of the complementary information and, accordingly generate an enhanced branch to supplement the decision fusion (Fig ))).
4.2 Mixture of Normal-inverse Gamma Distributions
In this section, we focus on fusing multiple NIG distributions from original modalities and the gener- ated pseudo modality. The key challenge is how to reasonably integrate multiple NIG distributions into a uniform NIG. The widely used product of experts (PoE) [45] not suitable for our case since there is no guarantee that the product of multiple NIGs is still NIG. Moreover, integration with PoE tends to be affected by noisy modalities [46]. For these issues, another natural way is integrating a set of NIGs with the following additive strategy:
Σ
NIG(Ems Tm Oim 9m))
y
M
Although simple in form, there are two main limitations for Eq. 7 First, simply averaging over all NIGs does not take the uncertainties of different NIGs into account, and thus it may be seriously affected by noisy modalities. Second, it is intractable infer the parameters for the fused NIG distribution in practice since there is no closed-fome solution. Therefore, we introduce the NIG summation operation 47 to approximately solve this problem. Specifically. the NIG summation operator defines novel operation to ensure new NIG distribution after the fusion of two NIG distributions.
Definition 4.1. (Summation of NIG distributions) Given two NIG distributions, i.e. NIG( n, a1, B1) and NIG( 72, oz B2), the definition of the summation of these two NIG distributions
a,,,3, & NIG((11, n. 01) @ NIG(Ee, 72,02,000 02 82),
(8)
where
8 = (31 + 72) = + 7282),
a= 01 02 + 2
(9)
1
7 = 71 +
B= + B2 + 2 n (61 - 8)2 +
2 22(622 - 5)2.
Therefore, fusing NIG distributions in this way endows our model with several promising properties (shown in Pro. 1 which allow us use it for trustworthy multimodal regression based on uncertainty. Then we substitute Eq. 7 with the following operation:
NIG a, = (TG mo00i,nn., an B1) @ NIG(62, 72, .2, 32) $ ... ④ NIG(ŌM,TM,,M,
TM, Bas) (10)
where C represents the summation operation of two NIG distributions.
The NIG summation can reasonably make use of modalities with different qualities. Specifically, the parameter 7m indicates the confidence of a NIG distribution for the mean 5m [88] As shown in
Def. 4.1 if one modality is more confident with its prediction then it will contribute more to the final prediction. Moreover, 3 directly reflects both aleatoric uncertainty and epistemic uncertainty (Eq, [2] which consists of two parts, i.e., the sum of 31 and 32 from multiple modalities and the variance between the final prediction and that of every single modality. Intuitively, the final uncertainty is determined jointly by the modality-specinn uncertainty and the prediction deviation among different modalities.
Proposition 4.1. The summation operation in Definition of NIG distributions has the following properties:
Commutativity:
(2,,12, NIG (61, n, 01, NIG 12, 02, B2) NIG 72, 02.00, (2) @ NIG (82, n, 01 B1).
2. Associativity:
(IG a1 ( N (12, 72, 02, B2) o NIG (8s, 73, az, Bz) | (62, 72, 02 32) @ NIG (63, 73, 03, {3)].
= (8 B1) @ [NIG
5

---

ProMoMe:
FestMotadi
56
30
33
Ephienic
Alabung
-30
-30
110
-10
100
-13
-50
-25
80
24
50
-75
-5.
-25
0.
25
5.
-75
-3.
-25
06
25
55
75
(a) Prediction
(b) Uncertainty estimation
Figure 2: (a) Regressed curves for each modality and the fused ones (with MoNIG), where gray shadow areas indicate there is no training data available: (b) the left and right subfigures demonstrate the estimated aleatoric and epistemic uncertainties, respectively.
The above two properties can be easily proved (refer to the supplement). Based on the summation operation, our multimodal regression algorithm has the following advantages. (1) Flexibility: accord- ing to Def. 4.1 and Pro. 4.1 we can fuse an arbitrary number of NIG distributions conveniently. (2)
Explainability:
explain the detailed fusion process for multiple NIG distributions and observe the belief degree of each NIG distribution according to Eq. 9
(3) Trustworthiness; using Def. [.]
allows fuse multiple NIG distributions into new NIG distribution, which is critical explicitly provide both epistemic uncertainty and aleatoric uncertainty for trustworthy decision making, (4)
Optimizability: compared with struggling to seek closed-form solution and conducting complex optimization, it is much more efficient to optimize using Def. 4.1 for multiple NIGs fusion.
4.3 Overall Learning Framework
Inspired by multi-task learning, we define the final loss function the sum of losses of multiple modalities (including the pseudo modality) and the predictive-level fused distribution:
M
((w)
Σ m==
Lm (m) Lr(w) - LMONII(WW.
(11)
where Lm (w) is obtained according to Eq.
6 Specifically, [p(w) is the pseudo modality loss (w) and LMONIG(W) is the fused distribution loss defined MoNIG((W + XC MI w). More detailed derivation for the overall loss in the supplement. The proposed method is general uncertainty-awar fusion module so that the pscudo modality may be obtained in different ways (c.g. features concatenation or concatenation
defined as (w()) = L (w) + XC
as CMONIG(WW (w) CNLL
after representation learning).
Given the MoNIG distribution, the aleatoric and epistemic uncertainties are defined as:
3
B
E
0
Vorlő
(12)
7(
aleatoric
epistemic
For clarification, we provide the following remarks: (1) The objective is unified learnable framework, and thus the branches of original modalities, pseudo modality, and the fused global branch can be improved with each other with the multi-task learning strategy; (2) The final output is explainable according to the local smpeaiitysspecffic and global (fused) uncertainties, and the alcatoric and epistemic uncertainties well.
5 Experiments
To demonstrate the effectiveness of our model, we conduct experiments on synthetic and real- world data including physical (Superconductivityy medical (CT slice and multimodal sentiment

---

1.3
Sup
1,9
0.8
EUitest
Uncortalets
0.4
0.4
0.2
0.00
0.0
Samples
(a) Epistemic uncertainty
(b) Superconductivity
(c) CT Slides
Figure 3:
Epistemic uncertainty with different number of training samples (a), and relationship between the uncertainty and noise degree (b)-(c).
analysis t Furthermore, we also conduct experiments to validate the effectiveness of uncertainty estimation in a variety of conditions. Finally, the ablation study is conducted further assessing the effectiveness of the approach in terms of uncertainty quantification and investigating whether the promising performance due to the novel strategy.
5.1 Experiment Synthetic Data
To intuitively illustrate the properties of our model, firstly we conduct experiments on synthetic data. Following [31 49], we conduct sampling with y x + E, where NO 3) for x
[-4, 22] U|2. 4j. While for the sampling data points x E (2, -2) we add stronger noise on input, where e (0, o) with
3 0.8 (2 - |z[), and the m-th modality input Im &x ~ NO,, 0.01). The visualization and corresponding analysis could be found in Fig. 2 where the left and right subfigures demonstrate the prediction ability and estimated aleatoric/epistemii uncertainties, respectively. According to Fig. 2(a), it is observed that all curves fit well to the ground- truth curve given enough training data. From the left subfigure of Fig. 2b). we find that the estimated aleatoric uncertainty (AU) (z E 2- 2))x is much higher compared with that of x E [-4, 2| Ui2, 4), which consistent with the basic assumption. For the right subfigure, it demonstrates that the epistemic uncertainty (EU) is much higher when there is no training data, validating that our model can well characterize the epistemic uncertainty.
5.2 Temperature Prediction for Superconductivity
Superconductivity contains 21263 samples, and all samples are described by two modal- ities. The first one contains 81 features of its experimental properties, and the second con- tains the chemical formula extracted into 86 features. The goal here to predict the crit- ical temperature (values are in the range [O, 185]) based on these two modalities. In our experiment, use 10633, 4000, and 6600 samples as training, validation, and test data, respectively. We compare our model with existing approaches including Gaussian (GS) in Eq. 2 Evidential (EVD) [1I, where "IF" and "DF" represent different fusion strate- gies, i.e, "IF" and "DF" denoting interme- diate level (concatenating the features from hidden layer) and data level (concatenating the original features) fusion, respectively. "Pseudo" indicates that the pseudo modality (concatenating the features from hidden layer) is involved. The performance from our approach (with the Adam optimizer: learning-rate ations) is superior to the compared methods, which is with the lowest Root Mean Squared Error (RMSE). We also impose different degrees of noise in different conditions to half of the test samples as cattof-distribution (OOD) samples, and try to distinguish them with uncertainty. "AU" and "EU"
Table I: Multimodal regression (RMSE L).
Mod1
Mod2
GS (I/DF)
EVD (IF/F)
Ours (Pseudo)
12.93
12.2
13.93/13.78
14.19/12.79
12.19 (11.70)
Table 2: OOD -iitt-riiittiitu detection (AU- ROC 1) using the estimated uncertainty.
Modality
Noise
GS
EVD
Ours
EU
AU
EU
0.1
0,503
0.503
0.494
0.586
0.536
Modl
0.511
0.782
0.551
Mod2
0.1
0,503
0.507
0.486
0.839
0.577
0.539
RandMod
0.1
0.502
0.502
0.488
0.699
0.549
0.508
0.492
0.426
0.812
0.539
AllMod
0.507
0.509
0.484
0.838
0.584
0.5
0.518
0.494
0.424
0.887
0.546
1e for 400 iter-
http://immoortallmutiioom

---

indicate the uncertainties used to distinguish OOD samples. According to Table 2 our model achieves clearly better performance in terms of AUROC on different noise conditions (a higher value indicates a better uncertainty estimation).
5.3 Relative Location Prediction for CT Slices
CT Slices are retrieved from a set of 53500 CT images from 74 different patients. Each CT slice is described by two modalities. The first modality describes the location of bone structures in the image, containing 240 at-
Table 3: Multimodal regression (RMSE J).
Mod1
Mod2
GS (IF/DF)
EVD (IF/DF)
Ours (Pscudo)
4.49
1.64/1.70
2.97/3.27
0.91 (0.79)
Table 4: OOD (out-of-distributin detection (AU-
tributes. The second modality describes the location of air inclusions inside of the body, containing 144 attributes. The goal is pre- dict the relative location of the image on the axial axis. The target values are in the range [0, 180], where 0 denotes the top of the head and 180 the soles of the feet. We divide the dataset into (6750/10000/11777 as training, validation, and test data, respec- tively. The other settings are the same as those on the Superconductivity dataset. We also test the ability of OOD detection and the results are reported in Table 4 It clear that our model achieves much better performance than comparisons.
ROC f) using the estimated uncertainty.
Modality
Nolse
GS
AU
EU
AU
EU
0.504
0.504
0.502
0.532
0.543
Mod1
0.5
0,582
0.638
0.610
0.771
0.806
0,503
0.503
0.500
0.519
0.511
Mod2
0.5
0.529
0.600
0.584
0.870
0.866
samples
RandMod
0.5
0,570
0,503
0.504
0.619
0.505
0.601
0.569
0.835
0.852
0,582
AllMod
0.507
0.509
0.508
0.600
0.615
0.5
0.618
0.672
0.631
0.808
0.839
5.4 Human Multimodal Sentiment Analysis
CMU-MOSI & MOSEL CMU-MOSI (50] is multimodal sentiment analysis dataset consisting of 2,199 short monologue video clips, while MOSEI (5T] is a sentiment and emotion analysis dataset consisting of 23,454 movie review video clips taken from YouTube. Each task consists of word- aligned and an unaligned version. For both versions, the multimodal features are extracted from the textual 122] visual and acoustic (53 modalities.
Table 5: Results on CMU-MOSI.
Table 6: Results CMU-MOSEL
Metric
Acc!
Ace.
EL
MAE
Cor
Mettic
Acc
Acc.
Fl
MAE I Corr
(Ward Aligned)
EMMU-MOS
Sentiment
(Word Aligned)
CMU
MOSEI
Sentiment
EF-LSTM
33.7
75.3
15.2
1.023
0.608
EFLSTM
47.4
18.2
17.9
0.642
0.616
LF-LSTM
35.32
76.8
76.7
1.015
0.625
LF-LSTM
48.8
80.6
80.6
0.619
0.659
RAVEN [54]
78.0
76.6
0.915
0.691
RAVEN 3E
50.02
79.1
79.5
0.614
0.662
MCTN 55
35.6
79.3
79.1
0.909
0.676
MCTN 55
49.6
79.8
80.6
0.609
0.670
Gaussian
32.9
78.4
78.4
0.982
0.657
Gaussian
81.62
81.92
0.613
0.677
Evidenial
33.4
77.6
77.6
0.974
0.655
Evidential
48.9
81.0
81.2
0.612
0.671
MoNIG
35.0
80.2"
801.4
0.959
0.664
MoNIG
50.21
$1.8
82.0'
0.602
0.682
MoNIG (pscudo)
34.1
80.6'
80.6
0.951
0.680
MoNIG (pscudo)
50.02
81.0
81.5
0.600'
0.688
(Unaligned)
MOMUMMOSTSS Sentizneni
(Unaligned]
CMU-MOSEI Sentiment
CTC + BELLSTM
14.5
1.078
0.542
CIC+EFLSIR 56
46.3
76.1
75.9
0.680
0.385
LF-LSTM
33.7
77.6
77.8
0.988
0.624
LFF.STM
77.5
78.2
0.624
0.656
CTC+ MCTN
55
32.7
75.9
76.4
0.991
0.613
CTC RAVEN 54
45.5
75.4
75.7
0.664
0.599
CTC RAVEN 54
31.7
72.7
73.1
1.076
0.544
CTC MCTN 53
48.2
79.3
79.7
0.631
0.645
Gaussian
32.4
77.6
77.5
1.005
0.634
Gaussian
81.0
81.3
0.618
0.676
Evidential
32.9
78.7
78.7
0,988
0.651
Evidential
49.2
81.3
81.7
0.608
0.676
MuNIG
35.8
79.3
79.3-
0.972
0.664
MoNIG
50.7
81.7
81.9
0.598'
0.693
MoNIG (psesudo)
34.7
79.1"
79.1
0.958
0.669
MoNIG (pscude)
49.5
81.7
82.0
0.612
0.673
For CMU-MOSEI dataset, similarly to existing work, there are 16326, 1871, and 4659 samples used as the training, validation, and test data, respectively. For CMU-MOSI dataset, we use 1284, 229.
and 686 samples as training, validation, and test data, respectively. Similar to previous work [5T 55 we employ diverse metrics for evaluation: 7-class accuracy (Acc7), binary accuracy (Ace2), Fl score, mean absolute error (MAE), and the correlation (Corr) of the model's prediction with human. We directly concatenate the features extracted from the temporal convolutional layers of different networks corresponding to different modalities as a pseudo modality. Our method achieves competitive performance even compared with the sfate-of-the-art multimodal sentiment classification methods.
8

---

5.5
Uncertainty Estimation
Unceettee Moet
2.0
relse
Uecoeetiin Wood
2.0
Mos1
Uncretterty or Mool
Mos1 Nos2
Moz2
Mod2
rotse Nod2
% +
Uncertairty of Modi
Uncerbality of Modl
Uncerbairty of Mool
Uncertairty Modt
(a) 6= 0.1
(b) c= 0.3
(c) c 0.5
(d) 6 1.0
Figure 4: Sensitivity identifying noisy modality. We randomly select one from two modalities and add different degree of noise on it.
To validate the ability of epistemic estimation of our model in real data, we gradually increase the ratio of training samples from 20% to 100% of all training data. According to Fig, 3 a). we can find that the overall epistemic uncertainty declines steadily as the number of training samples increases.
We illustrate the relationship between the uncertainty and different degree of noise (Fig. (c))).) It observed that as the noise goes stronger. both aleatoric and epistemic uncertainties become larger, which implies that our model can be adaptively aware of possible noise in practical tasks. It should be pointed out that data noise (AU) will also significantly affect the EU under limited training data.
To investigate the sensitivity for noisy modality, we add different degrees of Gaussian noise (i.e., of two modalities which is randomly selected. There 500 samples that are associated with noisy modality and noisy modality 2, respectively. As shown in Fig. 4 our algorithm can effectively distinguish which modality noisy for different samples. Overall, the proposed method can capture global uncertainty (Fig. 3 and also has a very effective perception of modality-specific noise. Accordingly, the proposed algorithm can provide potential explanation for erroneous prediction.
zero and varying variance c)
5.6 Ablation Study
Robustness noise. We add different degrees of Gaussian noise (i.e., zero mean and varying vari- ance E) to one of two modalities which is randomly selected, then compare our method with EVD [31] (concatenating the original features). The results in Table validate that our method can dexterously esti- mate the uncertainty for promising performance due to taking the modality-specit count for the integration, while it is difficult for the counterpart, i.c., EVD with concatenated features.
Table 7: Comparison between EVD 31 and
ours (RMSE J).
Dataset
Method
0.01
0.05
0.1
Sup
EVD
15.75
28.91
44.80
13.95
17.38
21.82
EVD
2.45
3.40
5.35
CT
Ours
0.97
1.34
2.23
uncertainty into ac-
Comparison with different decision fusion strate- gies. To clarify which part contributes to the im- provements, compare different decision fusion strategies: average, weighted average with AU, and weighted average with EU. Since our method em- ploys the modality-specifie uncertainty to dynami- cally guide the integration for different samples, ours performs as the best as shown in Table 8
Table Comparison between different fusion strategies (RMSE 4).
Dataset
Average Weighted (AU/EU) Ours
Sup CT
13.01
13.17/13.03
12.19
2.87
5.71/1.11
0.91
Effectiveness of uncertainty estimation. To quanti- tatively investigate the uncertainty estimation, we de- fine rank-based criterion to compare ours with other methods, which directly measures the inconsistency between the estimated uncertainty and predictive er- ror. The uncertainty-error inconsistency rate (UEIR)
Table 9: Quantitative evaluation for uncer- tainty (UEIR J).
Method AU (EVD/OURS) EU (EVD/Ours)
UEIR (%) 13.73/12.65
12.37/11.70
is defined UEIR Numjuwwast x 100%, where Numincoone is the number of sample pairs that RMSE
Num,
and uncertainty are in the opposite order (e.g., RMSE, > RMSE, & uncertainty, < uncertainty,).

---

and Numau is the total number of pairs for test samples. A smaller value implies a better uncertainty estimation, More detailed description of Table [ and Table iable/is shown in the supplementary material.
Conclusion
In this paper, we propose novel trustworthy multimodal regression model, which elegantly character- izes both modality-speciffe and fused uncertainties and thus can provide trustworthiness for the final output. This is quite important for real-world (especially cost-sensitive) applications. The introduced pseudo modality further enhances the exploration of complementarity among multiple modalities producing more stable performance. The proposed model is unified learnable framework, and thus can be efficiently optimized. Extensive experimental results on diverse applications and conditions validate that our model achieves impressive performance in terms of regression accuracy and also obtains reasonable uncertainty estimation to ensure trustworthy decision making. It interesting to apply the proposed algorithm for more real-world applications in the future. Moreover, it is also important to extend the regression algorithm for trustworthy multimodal classification.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 61976151, 6173201 1, and the Natural Science Foundation of Tianjin of China under Grant 19JCY- BJC15200. We thank Alexander Amini (Massachusetts Institute of Technology), the author of the paper 331, for his generous help in explaining his work and providing useful reference materials.
References
[1] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learn- ing: A survey and taxonomy. IEEE transactions on patlern analysis and machine intelligence, 2018.
[2] Hyunggi Cho, Young-Wor Woo Seo, BVK Vijaya Kumar, and Ragunathan Raj Rajkumar. A multi- sensor fusion system for moving object detection and tracking in urban driving environments. In ICRA, 2014.
[3] Richard Perrin, Anne M Fagan, and David M Holtzman. Multimodal techniques for diagnosis and prognosis of alzheimer's 's disease. Nature, 461(7666:916-922,, 2009.
[4] Quan Gan, Shangfei Wang, Longfei Hao, and Qiang Ji. A multimodal deep regression bayesian network for affective video content analyses. In ICCV, 2017.
[5] Guangnan Ye, Dong Liu, I-Hong Jhuo, and Shih-Fu Chang. Robust late fusion with rank minimization. In CVPR, 2012.
[6] Fabon Dzogang, Marie-Jeanne Lesot, Maria Rifqi, and Bernadette Bouchon-Meunier. Early fusion of low level features for emotion mining. Biomedical Informatics Insights, 2012.
[7] H. Gunes and M. Piccardi. Affect recognition from face and body: early fusion vs. late fusion. In IEEE International Conference Systems, 2006.
[8] Paul Pu Liang, Zhun Liu, Yao-Hung Hubert Tsai, Qibin Zhao, Ruslan Salakhutdinov, and Louis-Philiipe Morency. Learning representations from imperfect time series data via tensor rank regularization. arXiv: arXiv:1907.0000 1907.01011, 2019.
[9] Michelle A. Lee, Matthew Tan, Yuke Zhu, and Jeannette Bohg. Detect, reject, correct: Cross- modal compensation of corrupted sensors. arXiv:2012.00201, 2020.
[10] Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D. Sculley, Joshua V. Dillon, Jie Ren, and Zachary Nado. Can you trust your model's uncer- tainty? evaluating predictive uncertainty under dataset shift. In NeurIPS, 2019.
[11] Dennis Ulmer and Giovanni Cinà. Know your limits: Uncertainty estimation with relu classifiers fails at reliable ood detection. arXiv:2012.05329, 2021.
10

---

[12] Moloud Abdar. Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges. XXXi:2011.06225, 2020.
[13] Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. classification. In ICLR, 2021.
Trusted multi-view
[14] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In NeurIPS, 2017.
[15] K. Hamidieh. A data-driven statistical model for predicting the critical temperature of superconductor. Computational Materials Science, 154:346-354, 2018.
[16] Danijar Hafner, Dustin Tran, Timothy P. Lillicrap, Alex Irpan, and James Davidson. Noise contrastive priors for functional uncertainty. UAI, 2019.
[17] Kefaya Qaddoum and E. L. Hines. Reliable yield prediction with regression neural networks. In WSEAS international conference on systems theory and scientific computation, 2012.
[18] Alireza Khodayari, Ali Ghaffari, Sina Ameli, and Jamal Flahatgar. A historical review on lateral and longitudinal control of autonomous vehicle motions. In International Conference on Mechanical Electrical Technology, 2010.
[19] Radford M Neal. Bayesian learning for neural networks. Springer Science & Business Media, 2012.
[20] David JC MacKay. Bayesian interpolation. Neural computation, 4(3):415-44) 1992.
[21] Dmitry Molchanov, Arsenii Ashukha, and Dmitry P. Vetrov. Variational dropout sparsifies deep neural networks. ICML, 2017.
[22] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. Computer Science, 2012.
[23] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
[24] Jishnu Mukhoti and Yarin Gal. Evaluating bayesian deep learning methods for semantic segmentation. CoRR, 2018
[25] Balaji Lakshminarayanan Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, 2017.
[26] Javier Antorán, James Urquhart Allingham, and José Miguel Hermández-Lobato. Depth uncer- tainty in neural networks. In NeurIPS, 2020.
[27] Joost van Amersfoort, Lewis Smith, Yee whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. In ICML, 2020.
[28] Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Address- ing failure prediction by learning model confidence. In NeurIPS, 2019.
[29] Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. In NeurlPS, 2020.
[30] Murat Sensoy, Lance M. Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. NeurIPS, 2018
[31] Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. In NeurlPS, 2020.
[32] Yanghangyan Jiang, Qianqian Xu, Zhiyong Yang, Xiaochun Cao, and Qingming Huang. Dm2c: Deep mixed-modal clustering. In NeurIPS, 2019.

---

[33] Dhanesh Ramachandram and Graham W Taylor. Deep multimodal learning: A survey on recent advances and trends. IEEE Signal Processing Magazine, 2017.
[34] Douwe Kiela, Edouard Grave, Armand Joulin, and Tomás Mikolov. Efficient large-scale multi-modal classification. In AAAI, 2018.
[35] Soujanya Poria, Erik Cambria, and Alexander Gelbukh. Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. In Proceedings of the 2015 conference on empirical methods in natural language processing. 2015.
[36] S. Antol, A. Agrawal, J. Lu, M. Mitchell, and D. Parikh. Vqa: Visual question answering. International Journal of Computer Vision, 123(1):4-31, 2015.
[37] Y. Xia, F. Liu, Y. Dong, J. Cai, and H. Roth. 3d semi-supervised learning with uncertainty-aware view co-training. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), 2020.
[38] Dong Yi, Zhen Lei, and Stan Li. Shared representation learning for heterogenous face recognition. In 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), 2015.
[39] W. Ouyang, C. Xiao, and X. Wang. Multi-source deep learning for human pose estimation. In CVPR, 2014.
[40] Changqing Zhang, Yajie Cui, Zongbo Han, Joey Tianyi Zhou Huazhu Fu, and Qinghua Hu. Deep partial multi-view learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[41] Changqing Zhang, Huazhu Fu, Qinghua Hu, Xiaochun Cao, Yuan Xie, Dacheng Tao, and Dong Xu. Generalized latent multi-view subspace clustering, IEFE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[42] A. D. Kiureghian and O. Ditlevsen. Aleatory or epistemic? does it matter? Structural Safety, 3 31(2):105-112, 2009.
[43] X. Li, C. Shen, Q. Shi, A. Dick, and Avd Hengel. Non-sparse linear representations for visual tracking with online reservoir metric learning. In CVPR, 2012.
[44] S. Bahrami, F. Dornaika, and A. Bosaghzadeh. Joint auto- weighted graph fusion and scalable semi-supervised learning. Information Fusion, 66(1):213- 222222222222222 2021.
[45] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-180) 2002.
[46] Yuge Shi, Siddharth Narayanaswamy, Brooks Paige, and Philip H. S. Torr. Variational mixture- of-experts autoencoders for multi-modal deep generative models. In NeurlPS, 2019.
[47] Hang Qian al. Big data bayesian linear regression and variable selection by gamma summation. Bayesian Analysis, 13(4):1111-1035 2018
[48] Michael Jordan. The exponential family: Conjugate priors. 2009.
[49] José Miguel Hernández-Lobato and Ryan P. Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In ICML, 2015.
[50] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philipen Morency. MOSI: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. CoRR, 2016.
[51] P. P. Liang, Z. Liu, A. Zadeh, and I.. P. Morency. Multimodal language analysis with recurrent multistage fusion. In Meeting of the Association for Computational Linguistics, 2018
[52] Pennington, R. Socher, and C. Manning. Glove: Global vectors for word representation. In Conference Empirical Methods in Natural Language Processing, 2014.
12

---

[53] Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. Covarep: A collaborative voice analysis repository for speech technologies. In Acoustics Speech and Signal Processing (ICASSP), 2014.
[54] Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis- Chiis-Philippp Morency. Words can shift: Dynamically adjusting word representations using nonverbal behaviors. In AAAI, 2019.
[55] Hai Pham, Paul Pu Liang, Thomas Manzini, Louis- Phiis-PPil Morency, and Barnabás Póczos. Found in translation: Learning robust joint representations by cyclic translations between modalities. In AAAI, 2019.
[56] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In ICML, 2006.
[57] Christopher M Bishop. Puttern Recognition and Machine Learning (Information Science and Statistics). Springer- Verlag New York, Inc., 2006.
13


Reviews:
[{"review_id": "_xM62hvxgJF", "reviewer": "Reviewer_vgoM", "summary": "The problem of multi-modal regression is considered in this paper and a method called Mixture of Normal-Inverse Gamma (MoNIG) is proposed. In the proposed method, an approximation is adopted to combine predictive normal-inverse gamma distributions from different modalities into one normal-inverse gamma for tractable uncertainty calculation.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper aims to develop a new method for trustworthy multi-modal regression which is an important problem. The performance is compared with Gaussian and Evidential Regression with concatenated features/hidden layers on two datasets and on a multimodal sentiment analysis with several baselines. The results show improvements over the baselines.\n\nConcerns:\n\nThe methodology of the paper is a straightforward combination of existing techniques, Deep Evidential Regression, and the summation operation for NIG distributions. Thus, the methodological contribution of the paper is not very significant.\n\nThe NIG summation operation is originally introduced for Bayesian linear regression on large datasets, which is exact there, but it is not clear in the paper how valid the approximation is for the paper’s setup. \n\nFor the experiments, simple baselines which are fitting GS or EVD to each modality and then averaging them, simple or weighted, are not considered.\n\nResults in Table 7 suggest that the performance of the method still deteriorates even though noise is only added to one modality. So, the discussion in the paper regarding robustness to modality corruption can be adjusted accordingly.\n\nNone of the experiments contain error bars, so it is unclear how significant are some of the results, especially those with closer numbers.\n\nThe paper has some issues in terms of clarity. The provided details are not enough to reproduce the results. In Tables 5 and 6 the baselines do not have references. The details are not provided for the ablation studies.\n\nAdditional analysis like Figure 4 can also be done for the baselines for comparison.\n\n\n--Update--\n\nThanks to the authors for their responses. The authors have clarified what Table 8 shows (I assume the weights are per sample) and provided some error bars which address some of my concerns. Hence, I have increased the score by 1 point. \n\nMy major concern about the work remains up to some extent. While the insight of combining existing techniques for multimodal regression is new, there does not appear to be any significant technical challenges involved to make them work here. Additionally, in my view, the paper has room for improvement and in its current form does not properly answer why the uncertainties from applying the NIG summation in the paper’s setup are valid and reliable.\n\nA few other suggestions to strengthen the experiments:\n\n-Calibration curves of the estimated uncertainties can be included\n\n-Modality and “pseudo-modality” specific uncertainties can be compared systematically with and without the NIG summation. Currently, there is no comparison in this regard.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not discussed the limitations and societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2-3"}, "extra_text": {}}, {"review_id": "P4M_rt0t8_Z", "reviewer": "Reviewer_4yAM", "summary": "This paper tackles a shortcoming of existing multimodal learning methods that mainly focus on the integration of the information from different sources for prediction but ignore the confidence of prediction. The authors propose a Mixture of NormalInverse Gamma distributions (MoNIG) algorithm which estimates uncertainty for multimodal learning. Their model estimates uncertainty for each modality and also in corrupted modality settings.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The problem tackled is an important one and an important shortcoming in current work.\n2. The proposed method is a nice extension of uncertainty estimation for single modality regression problems using Normal-inverse Gamma Distributions into one for multimodal problems using a mixture of distributions.\n3. Experimental results are quite comprehensive on 3 sets of tasks spanning a few multimodal datasets.\n4. Experimental results on prediction tasks are strong and show some improvement over current methods.\n5. Experimental results also show that the model can work well in the presence of noisy modalities.\n\nWeaknesses:\n1. There can be more analysis of when the proposed method fails and other potential shortcomings of their approach.\n2. There should be more analysis on what types of multimodal inputs and modalities the model estimates uncertainty for, including visualizations and qualitative examples.\n3. The paper can benefit from some additional analysis and discussion of the additional training time and space complexity resulting from their model.\n4. There should be some reference to recent work in learning robust multimodal representations [1,2].\n5. The model currently handles multimodal interactions using a very simple late-fusion style mixture of distributions - how can the method be extended to more fine-grained multimodal representation learning methods such as those in Table 5 (e.g. RMFN, MFM etc.)\n6. There is not enough evaluation to show that the model has accurately estimated uncertainty apart from some visualizations. How do you evaluate whether your model has estimated uncertainty correctly? Are there any metrics for judging this? I think this might be a good reference: https://github.com/uncertainty-toolbox/uncertainty-toolbox\n\n[1] Liang et al., Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization. ACL 2019\n\n[2] Lee et al., Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors. 2020\n\nNot a weakness, but a general suggestion: The authors can take a look at https://arxiv.org/abs/2107.07502 for a wide suite of multimodal methods and datasets that they can test their method on.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors can say a bit more about the societal impact of multimodal models, including their applications in robotics, healthcare, education etc.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns as far as I can tell.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "Hqo35gTHcC", "reviewer": "Reviewer_BXmu", "summary": "This paper proposes a Mixture of Normal-Inverse Gamma distributions algorithm to estimate uncertainty in principle for adaptive integration of different modalities.\n\nThis research problem proposed in this paper is very important, and the author proposed some techniques and conducted some experiments.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is innovative to some extent. In fact, it is difficult to measure the uncertainty of combining the information from different modalities.  \n\nThe writing of the paper needs further polishing. Some details of the paper are not detailed. For example, Figure 1 and Figure 2 are both confusing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper does not clarify the limitations of the proposed method. \n\nIn fact, the method proposed by the author has some assumptions about the distribution of the data. The rationality of the assumptions needs to be explained.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "EAXMRNdXGna", "reviewer": "Reviewer_8nwK", "summary": "This paper focuses on an important problem in multimodal learning - current algorithms ignore the confidence of integration and prediction. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper focuses on an important problem in multimodal learning - current algorithms ignore the confidence of integration and prediction. The authors smartly use Mixture of Normal Inverse Gamma distributions (MoNIG) to estimate uncertainty for multimodal learning. The model estimates uncertainty for each single modality and the final prediction as well. Experiments on both synthetic and real datasets well support the assumptions.\n\nStrengths:\n- Trustworthy multimodal learning is important and interesting, and the motivation of the paper is very clear and strong. \n-The introduced pseudo modality is interesting and necessary, which enhances the sufficient multimodal interactions beyond explainability.\n-Both the detailed theoretical analysis and experimental results strongly support the motivation. Especially, the uncertainty estimation and ablation study provide strong evidence to show which parts contribute to the improvements.\n-The writing and organization are good making the paper quite easy to follow.\n\nMinors：\n-It is an elegant model for trustworthy multimodal regression. Can the model be applied or extended for classification task? It will be better if the authors could provide related discussion.\n- The proposed method introduces uncertainty estimation, so what about the computational complexity of the proposed model compared with existing multimodal algorithms?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Minors：\n-It is an elegant model for trustworthy multimodal regression. Can the model be applied or extended for classification task? It will be better if the authors could provide related discussion.\n- The proposed method introduces uncertainty estimation, so what about the computational complexity of the proposed model compared with existing multimodal algorithms?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.5 hour"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose a new regression method for multi-modal (e.g., image and text) data. More specifically, the authors extend the Deep Evidential Regression method for a multi-modal regression problem. The method depends heavily on the existing methods. However, the formulation is new and the fusion strategy has some merit. Thus, I also vote for an acceptance. For the camera-ready version, I expect authors to revise the paper based on the reviewer's comments. 
Decision: Accept (Poster)