Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions

1 Introduction . There are plenty of multimodal data involved in the real world , and we experience the world from different modalities [ 1 ] . For example , the autonomous driving systems are usually equipped with multiple sensors to collect information from different perspectives [ 2 ] . In medical diagnosis [ 3 ] , multimodal data usually come from different types of examinations , typically including a variety of clinical data . Effectively exploiting the information of different sources to improve learning performance is a long-standing and challenging goal in machine learning . ∗Equal contribution . †Corresponding author . Please correspond to : zhangchangqing @ tju.edu.cn . 3Code : https : //github.com/MaHuanAAA/MoNIG . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Most multimodal regression methods [ 4 , 5 , 6 , 7 ] usually focus on improving the regression performance by exploiting the complementary information among multiple modalities . Despite effectiveness , it is quite risky for these methods to be deployed in cost-sensitive applications due to the lack of reliability and interpretability . One underlying deficiency is that traditional models usually assume the quality of each modality is basically stable , which limits them to produce reliable prediction , especially when some modalities are noisy or even corrupted [ 8 , 9 ] . Moreover , existing models only output ( over-confident ) predictions [ 10 , 11 ] , which can not well support safe decision and might be disastrous for safety-critical applications . Uncertainty estimation provides a way for trustworthy prediction [ 12 , 13 ] . The decisions made by models without uncertainty estimation are untrustworthy because they are prone to be affected by noises or limited training data . Therefore , it is highly desirable to characterize uncertainty in the learning for AI-based systems . More specifically , when a model is given an input that has never been seen or is severely contaminated , it should be able to express “ I don ’ t know ” . The untrustworthy models are vulnerable to be attacked and it may also lead to wrong decisions , where the cost is often unbearable in critical domains [ 14 ] . Basically , it can endow the model with trustworthiness by dynamically modeling uncertainty . Therefore , we propose a novel algorithm that conducts multimodal regression in a trustworthy manner . Specifically , our proposed algorithm is a unified framework modeling uncertainty under a fully probabilistic framework . Our model integrates multiple modalities by introducing a Mixture of Normal-inverse Gamma distributions ( MoNIG ) , which hierarchically characterizes the uncertainties and accordingly promotes both regression accuracy and trustworthiness . In summary , the contributions of this work include : ( 1 ) We propose a novel trustworthy multimodal regression algorithm . Our method effectively fuses multiple modalities under the evidential regression framework equipped with modality-specific uncertainty and global uncertainty . ( 2 ) To integrate different modalities , a novel MoNIG is designed to be dynamically aware of modality-specific noise/corruption with the estimated uncertainty , which promisingly supports trustworthy decision making and also significantly promotes robustness as well . ( 3 ) We conduct extensive experiments on both synthetic and real-application data , which validate the effectiveness , robustness , and reliability of the proposed model on different multimodal regression tasks ( e.g. , critical temperature prediction for superconductivity [ 15 ] , relative location of CT slices , and human multimodal sentiment analysis ) . 2 Related Work . 2.1 Uncertainty Estimation . Quantifying the uncertainty of machine learning models has received extensive attention [ 16 , 17 ] , especially when the systems are deployed in safety-critical domains , such as autonomous vehicle control [ 18 ] and medical diagnosis [ 3 ] . Bayesian neural networks [ 19 , 20 ] model the uncertainty by placing a distribution over model parameters and marginalizing these parameters to form a predictive distribution . Due to the huge parameter space of modern neural networks , Bayesian neural networks are highly non-convex and difficult in inference . To tackle this problem , [ 21 ] extends Variational Dropout [ 22 ] to the case when dropout rates are unbounded , and proposes a way to reduce the variance of the gradient estimator . A more scalable alternative way is MC Dropout [ 23 ] , which is simple to implement and has been successfully applied to downstream tasks [ 14 , 24 ] . Deep ensembles [ 25 ] have shown strong power in both classification accuracy and uncertainty estimation . It is observed that deep ensembles consistently outperform Bayesian neural networks that are trained using variational inference [ 10 ] . However , the memory and computational cost are quite high . For this issue , different deep sub-networks with shared parameters are trained for integration [ 26 ] . Deterministic uncertainty methods are designed to directly output the uncertainty and alleviate the overconfidence . Built upon RBF networks , [ 27 ] is able to identify the out-of-distribution samples . [ 28 ] introduces a new target criterion for model confidence , known as True Class Probability ( TCP ) , to ensure the low confidence for the failure predictions . The recent approach employs the focal loss to calibrate the deep neural networks [ 29 ] . [ 30 ] places Dirichlet priors over discrete classification predictions and regularizes divergence to a well-defined prior . Our model is inspired by deep evidential regression [ 31 ] which is designed for single-modal data . 2.2 Multimodal Learning . Multimodal machine learning aims to build models that can jointly exploit information from multiple modalities [ 1 , 32 ] . Existing multimodal learning methods have achieved significant progress by integrating different modalities at different stages , namely early , intermediate and late fusion [ 1 , 33 ] . The early fusion methods usually integrate the original data or preprocessed features by simply concatenation [ 34 , 35 ] . The intermediate fusion provides a more flexible strategy to exploit multimodal data for diverse practical applications [ 4 , 36 , 37 , 38 , 39 , 40 , 41 ] . In late fusion , each modality is utilized to train a separate model and the final output is obtained by combining the predictions of these multiple models [ 5 , 7 , 37 ] . Although these multimodal learning approaches exploit the complementary information of multiple modalities from different perspectives , basically they are weak in modeling uncertainty which is important for trustworthy prediction especially when they are deployed in safety-critical domains . 3 Background . Regression has been widely used across a wide spectrum of applications . Given a dataset D = { xi , yi } Ni=1 , one principled way for regression is from a maximum likelihood perspective , where the likelihood for parameters w given the observed data D is as follows : p ( y | Dx , w ) = N∏ i=1 p ( yi | w , xi ) . ( 1 ) In practice , a widely used maximum likelihood estimation is based on Gaussian distribution , which assumes that the target y is drawn from a Gaussian distribution . Then we have the following expression of likelihood function : p ( y | Dx , w , σ2 ) = N∏ i=1 N ( yi | f ( xi , w ) , σ2 ) . ( 2 ) The mean and variance indicate the prediction and corresponding predictive uncertainty , while the predictive uncertainty consists of two parts [ 42 ] : epistemic uncertainty ( EU ) and aleatoric uncertainty ( AU ) . To learn both the aleatoric and epistemic uncertainties explicitly , the mean and variance are assumed to be drawn from Gaussian and Inverse-Gamma distributions [ 31 ] , respectively . Then the Normal Inverse-Gamma ( NIG ) distribution with parameters τ = ( δ , γ , α , β ) can be considered as a higher-order conjugate prior of the Gaussian distribution parameterized with θ = ( µm , σ 2 ) : yi ∼ N ( µ , σ2 ) , µ ∼ N ( δ , σ2γ−1 ) , σ2 ∼ Γ−1 ( α , β ) , ( 3 ) where Γ ( · ) is the gamma function . In this case , the distribution of y takes the form of a NIG distribution NIG ( δ , γ , α , β ) : p ( µ , σ2︸ ︷︷ ︸ θ | δ , γ , α , β︸ ︷︷ ︸ τ ) = βα Γ ( α ) √ γ σ √ 2π ( 1 σ2 ) α+1 exp { −2β + γ ( δ − µ ) 2 2σ2 } , ( 4 ) where δ ∈ R , γ > 0 , α > 1 , and β > 0 . During the training stage , the following loss is induced to minimize the negative log likelihood loss : LNLL ( w ) = 1 2 log ( π γ ) − α log ( Ω ) + ( α+ 1 2 ) log ( ( y − δ ) 2γ + Ω ) + log Ψ , ( 5 ) where Ω = 2β ( 1 + γ ) and Ψ = ( Γ ( α ) Γ ( α+ 12 ) ) . The total loss , L ( w ) , consists of two terms for maximizing the likelihood function and regularizing evidence : L ( w ) = LNLL ( w ) + λLR ( w ) , ( 6 ) where LR ( w ) is the penalty for incorrect evidence ( more details are shown in the supplement ) , and the coefficient λ > 0 balances these two loss terms . For multimodal regression [ 5 , 7 , 6 ] , some approaches have achieved impressive performance . However , existing multimodal regression algorithms mainly focus on improving the accuracy and provide a deterministic prediction without the information of uncertainty , making these models limited for trustworthy decision making . For this issue , we develop a novel algorithm which can capture the modality-specific uncertainty and accordingly induces a reliable overall uncertainty for the final output . Beyond trustworthy decision , our approach automatically alleviates the impact from heavily noisy or corrupted modality . Compared with the intermediate fusion [ 4 , 43 , 44 ] , our method can effectively distinguish which modality is noisy or corrupted for different samples , and accordingly can take the modality-specific uncertainty into account for robust integration . 4 Trustworthy Multimodal Regression . In this section , we introduce the proposed algorithm fusing multiple modalities at both feature and predictive distribution levels . Specifically , we train deep neural networks to model the hyperparameters of the higher-order evidential distributions of multiple modalities ( and the pseudo modality ) and then merge these predicted distributions into one by the mixture of NIGs ( MoNIG ) . In the following subsections , we first provide discussion about existing fusion strategies and ours in multimodal regression . Then we introduce MoNIG to integrate NIG distributions of different modalities in principle . Finally , we elaborate the overall pipeline of our model in training and define the epistemic uncertainty and aleatoric uncertainty respectively . 4.1 Overview of Fusion Strategy . Consider the following multimodal regression problem : given a datasetD = { { xmi } M m=1 , yi } , where xmi is the input feature vector of the m-th modality of the i-th sample , yi is the corresponding target , and M is the total number of modalities , the intuitive goal is to learn a function fm for each modality reaching the following target : min ∑M m=1 ( fm ( xm , wm ) − y ) 2 . While to model the uncertainty , we assume that the observed target y is drawn from a Gaussian distribution , i.e. , y ∼ N ( µm , σ2m ) . Meanwhile , inspired by [ 31 ] , as shown in Eq . 3 , we assume that the mean and variance are drawn from Gaussian and Inverse-Gamma distributions , respectively . Considering multimodal fusion , there are typically two representative strategies , i.e. , feature fusion and decision fusion . Feature fusion ( Fig . 1 ( a ) ) jointly learns unified representations and based on which conducts regression . Although simple , the limitation is also obvious . If some modalities of some samples are noisy or corrupted , the algorithms based on this strategy may fail . In other words , these methods do not take the modality-specific uncertainty into account and can not be adaptively aware of quality variation of different modalities for different samples . The other representative strategy is known as decision fusion ( Fig . 1 ( b ) ) which trains one regression model for each modality , and the final prediction is regarded as a function of the predictions of all modalities . Although flexible , it is also risky for the cases that each modality is insufficient to make trustworthy decision due to partial observation . We propose MoNIG , which can elegantly address the above issues in a unified framework . Firstly , we explicitly represent both modality-specific and global uncertainties to endow our model with the ability of adaption for dynamical modality variation . The modality-specific uncertainty can dynamically guide the integration for different samples and the global uncertainty represents the uncertainty of the final prediction . Secondly , we introduce the feature level fusion producing a pseudo modality to make full use of the complementary information and , accordingly generate an enhanced branch to supplement the decision fusion ( Fig 1 ( c ) ) .

Summary: This paper tackles a shortcoming of existing multimodal learning methods that mainly focus on the integration of the information from different sources for prediction but ignore the confidence of prediction. The authors propose a Mixture of NormalInverse Gamma distributions (MoNIG) algorithm which estimates uncertainty for multimodal learning. Their model estimates uncertainty for each modality and also in corrupted modality settings.