Title: Beyond Unimodal: Generalising Neural Processes for Multimodal Uncertainty Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Uncertainty estimation

multimodality

neural processes
Abstract: Uncertainty estimation is an important research area to make deep neural networks (DNNs) more trustworthy. While extensive research on uncertainty estimation has been conducted with unimodal data, uncertainty estimation for multimodal data remains a challenge. Neural processes (NPs) have been demonstrated to be an effective uncertainty estimation method for unimodal data by providing the reliability of Gaussian processes with efficient and powerful DNNs. While NPs hold significant potential for multimodal uncertainty estimation, the adaptation of NPs for multimodal data has not been carefully studied. To bridge this gap, we propose Multimodal Neural Processes (MNPs) by generalising NPs for multimodal uncertainty estimation. Based on the framework of NPs, MNPs consist of several novel and principled mechanisms tailored to the characteristics of multimodal data. In extensive empirical evaluation, our method achieves state-of-the-art multimodal uncertainty estimation performance, showing its appealing robustness against noisy samples and reliability in out-of-distribution detection with faster computation time compared to the current state-of-the-art multimodal uncertainty estimation method.

Beyond Unimodal: Generalising Neural Processes for Multimodal Uncertainty Estimation

Abstract

Uncertainty estimation is an important research area to make deep neural networks (DNNs) more trustworthy. While extensive research on uncertainty estimation has been conducted with unimodal data, uncertainty estimation for multimodal data remains a challenge. Neural processes (NPs) have been demonstrated to be an effective uncertainty estimation method for unimodal data by providing the reliability of Gaussian processes with efficient and powerful DNNs. While NPs hold significant potential for multimodal uncertainty estimation, the adaptation of NPs for multimodal data has not been carefully studied. To bridge this gap, we propose Multimodal Neural Processes (MNPs) by generalising NPs for multimodal uncertainty estimation. Based on the framework of NPs, MNPs consist of several novel and principled mechanisms tailored to the characteristics of multimodal data. In extensive empirical evaluation, our method achieves state-of-the-art multimodal uncertainty estimation performance, showing its appealing robustness against noisy samples and reliability in out-of-distribution detection with faster computation time compared to the current state-of-the-art multimodal uncertainty estimation method.

1 Introduction

Uncertainty estimation of deep neural networks (DNNs) is an essential research area in the development of reliable and well-calibrated models for safety-critical domains [49, 45]. Despite the remarkable success achieved by DNNs, a common issue with these model is their tendency to make overconfident predictions for both in-distribution (ID) and out-of-distribution (OOD) samples [16, 43, 35]. Extensive research has been conducted to mitigate this problem, but most of these efforts are limited to unimodal data, neglecting the consideration of multimodal data [57, 11, 50, 63, 38, 34].

In many practical scenarios, safety-critical domains typically involve the processing of multimodal data. For example, medical diagnosis classification that uses X-ray, radiology text reports and the patient's medical record history data can be considered as a form of multimodal learning that requires trustworthy predictions [8]. Despite the prevalence of such multimodal data, the problem of uncertainty estimation for multimodal data has not been comprehensively studied. Furthermore, it has been demonstrated that using existing unimodal uncertainty estimation techniques directly for multimodal data is ineffective, emphasising the need for a meticulous investigation of multimodal uncertainty estimation techniques [24, 17].

Several studies investigated the problem of multimodal uncertainty estimation with common tasks including 1) assessing calibration performance, 2) evaluating robustness to noisy inputs, and 3) detecting OOD samples. Trusted Multi-view Classification (TMC) [17] is a single-forward deterministic classifier that uses the Demptser's combination rule [7] to combine predictions obtained from different modalities. The current state-of-the-art (SOTA) in this field is Multi-view Gaussian Process (MGP) [24], which is a non-parametric Gaussian process (GP) classifier that utilises the product-of-experts to combine predictive distributions derived from multiple modalities. Although MGP has shown reliability of GPs in this context, it should be noted that the computational cost of a GP increases cubically with the number of samples [19, 65].

Neural processes (NPs) offer an alternative approach that utilises the representation power of DNNs to imitate the non-parametric behaviour of GPs while maintaining a lower computational cost [13, 14]. It has been shown that NPs can provide promising uncertainty estimation for both regression [28, 33, 29] and classification tasks [62, 25] involving unimodal data. Despite the promising potential of NPs for multimodal data, to the best of our knowledge, no research has yet investigated the feasibility of using NPs for multimodal uncertainty estimation.

In this work, we propose a new multimodal uncertainty estimation framework called Multimodal Neural Processes (MNPs) by generalising NPs for multimodal uncertainty estimation. MNPs have three key components: the dynamic context memory (DCM) that efficiently stores and updates informative training samples, the multimodal Bayesian aggregation (MBA) method which enables a principled combination of multimodal latent representations, and the adaptive radial basis function (RBF) attention mechanism that facilitates well-calibrated predictions. Our contributions are:

We introduce a novel multimodal uncertainty estimation method by generalising NPs that comprise the DCM, the MBA, and the adaptive RBF attention.

We conduct rigorous experiments on seven real-world datasets and achieve the new SOTA performance in classification accuracy, calibration, robustness to noise, and OOD detection.

We show that MNPs achieve faster computation time (up to 5 folds) compared to the current SOTA multimodal uncertainty estimation method.

2 Background

Multimodal Classification The aim of this research is to investigate uncertainty estimation for multimodal classification. Specifically, we consider a multimodal training dataset D train = {{ x i m } M m =1 , y i } N train i =1 , where M is the number of input modalities and N train is the number of training samples. We assume that each i th sample has M modalities of input x i m ∈ ❘ d m with the input dimension of d m and a single one-hot encoded label y i with the number of classes K . In this study, we consider the input space to be a feature space. The objective is to estimate the labels of test samples { y i } N test i =1 given D test = {{ x i m } M m =1 } N test i =1 , where N test is the number of test samples.

Neural Processes NPs are stochastic processes using DNNs to capture the ground truth stochastic processes that generate the given data [13]. NPs learn the distribution of functions and provide uncertainty of target samples, preserving the property of GPs. At the same time, NPs exploit function approximation of DNNs in a more efficient manner than GPs. For NPs, both training and test datasets have a context set C = { C X , C Y } = { x C i , y C i } N C i =1 and a target set T = ( T X , T Y ) = { x T i , y T i } N T i =1 with N C being the number of context samples and N T the number of target samples 2 , and the learning objective of NPs is to maximise the likelihood of p ( T Y | C, T X ) .

Conditional Neural Processes (CNPs), the original work of NPs [13], maximise p ( T Y | r ( C ) , T X ) with r ( C ) = 1 N C ∑ N C i =1 enc ρ ( cat [ x C i ; y C i ] ) ∈ ❘ d e where enc ρ is an encoder parameterised by ρ , cat [ · ; · ] is the concatenation of two vectors along the feature dimension, and d e is the feature dimension. The mean vector r ( C ) is the permutation invariant representation that summarises the context set which is passed to a decoder with T X to estimate T Y .

The original CNPs use the unweighted mean operation to obtain r ( C ) by treating all the context points equally, which has been shown to be underfitting [28]. To improve over this, Attentive Neural

𝑪

𝑿,𝟏

𝒎

𝑪

𝒀,𝟏

𝒎

𝑪

𝑿,𝟐

𝒎

𝑪

𝒀,𝟐

𝒎

DCM

𝒓

𝟏

𝒎

𝒓

𝟐

𝒎

𝒔

𝟏

𝒎

𝒔

𝟐

𝒎

𝒖

𝒎

𝑪

𝑿,𝟏

𝒎

𝑪

𝑿,𝟐

𝒎

𝑻

𝑿

𝒎

Values

Query

𝒓

∗

𝒎

𝒔

∗ 𝒎

𝒖

𝟏

𝒎

𝒖

𝟐

𝒎

𝒒

𝟏

𝒎

𝒒

𝟐

𝒎

𝒒

𝒎

𝐞𝐧𝐜

𝝓

𝒎

𝐞𝐧𝐜

𝝍

𝒎

𝐞𝐧𝐜

𝜽

𝒎

𝐞𝐧𝐜

𝝎

𝒎

Adaptive

RBF

Attention

m

m

Modality m

Module

𝑻

(

𝒀

MBA

Decoder

Modality M

Module

Modality 1

Module

m

Mean operation

~

Sampling

~

𝒁

Processes (ANPs) [28] leveraged the scaled dot-product cross-attention [60] to create target-specific context representations that allocate higher weight on the closer context points:

where A ( T X , C X ) is the attention weight with T X as the query, C X is the key, the encoded context set is the value, and d is the input dimension of T X and C X . The resulted target-specific context representation has been shown to enhance the expressiveness of the task representation [28, 53, 29, 9, 47, 67].

The training procedure of NPs often randomly splits a training dataset to the context and the target sets (i.e., N C + N T = N train ). In the inference stage, a context set is provided for the test/target samples (e.g., a support set in few-shot learning or unmasked patches in image completion). While the context set for the test dataset is given for these tasks, NPs also have been applied to other tasks where the context set is not available (e.g., semi-supervised learning [62] and uncertainty estimation for image classification [25]). In those tasks, existing studies have used a context memory during inference, which is composed of training samples that are updated while training [25, 62]. It is assumed that the context memory effectively represents the training dataset with a smaller number of samples. We build upon these previous studies by leveraging the concept of context memory in a more effective manner, which is suitable for our task.

Note that the existing NPs such as ETP [25] can be applied to multimodal classification by concatenating multimodal features into a single unimodal feature. However, in Section 5.1, we show that this approach results in limited robustness to noisy samples.

3 Multimodal Neural Processes

The aim of this study is to generalise NPs to enable multimodal uncertainty estimation. In order to achieve this goal, there are significant challenges that need to be addressed first. Firstly, it is essential to have an efficient and effective context memory for a classification task as described in Section 2. Secondly, a systematic approach of aggregating multimodal information is required to provide unified predictions. Finally, the model should provide well-calibrated predictions without producing overconfident estimates as described in Section 1.

Our MNPs, shown in Figure 1, address these challenges with three key components respectively: 1) the dynamic context memory (DCM) (Section 3.1), 2) the multimodal Bayesian aggregation (MBA) (Section 3.2), and 3) the adaptive radial basis function (RBF) attention (Section 3.3). From Section 3.1 to 3.3, we elaborate detailed motivations and proposed solutions of each challenge, and

Section 3.4 outlines the procedures for making predictions. Throughout this section, we refer the multimodal target set T M = ( { T m X } M m =1 , T Y ) = {{ x i m } M m =1 , y i } N T i =1 to the samples from training and test datasets and the multimodal context memory C M = { C m X , C m Y } M m =1 to the context set.

3.1 Dynamic Context Memory

Motivation In NPs, a context set must be provided for a target set, which however is not always possible for non-meta-learning tasks during inference. One simple approach to adapt NPs to these tasks is to randomly select context points from the training dataset, but its performance is suboptimal as randomly sampling a few samples may not adequately represent the entire training distribution. Wang et al. [62] proposed an alternative solution by introducing a first-in-first-out (FIFO) memory that stores the context points with the predefined memory size. Although the FIFO performs slightly better than the random sampling in practice, the performance is still limited because updating the context memory is independent of the model's predictions. Refer to Appendix C.1 for comparisons.

Proposed Solution To overcome this limitation of the existing context memory, we propose a simple and effective updating mechanism for the context memory in the training process, which we call Dynamic Context Memory (DCM). We partition context memory { C m X , C m Y } of m th modality into K subsets ( K as the number of classes) as { C m X , C m Y } = { C m X,k , C m Y,k } K k =1 to introduce a class-balance context memory, with each subset devoted to one class. Accordingly, N m = N m K × K where N m is the number of context elements per modality (i.e., the size of { C m X , C m Y } ) and N m K is the number of class-specific context samples per modality. This setting resembles the classification setting in [13], where class-specific context representations are obtained for each class.

DCMis initialised by taking N m K random training samples from each class of each modality and is updated every mini-batch during training by replacing the least 'informative' element in the memory with a possibly more 'informative' sample. We regard the element in DCM that receives the smallest attention weight (i.e., A ( T X , C X ) in Equation (1)) during training as the least informative one and the target point that is most difficult to classify (i.e., high classification error) as a more informative sample that should be added to DCM to help the model learn the decision boundary. Formally, this can be written as:

where [ i, :] indicates the i th row vector of a matrix, [ i, j ] indicates the i th row and the j th column element of a matrix, and ̂ T m Y is the predicted probability of the m th modality input. i ∗ selects the context memory element that receives the least average attention weight in a mini-batch during training, and j ∗ selects the target sample with the highest classification error. To measure the error between the predictive probability and the ground truth of a target sample, one can use mean squared error (MSE) (Equation (3)) or cross-entropy loss (CE). We empirically found that the former gives better performance in our experiments. Refer to Appendix C.1 for ablation studies comparing the two and the impact of N m on its performance.

The proposed updating mechanism is very efficient as the predictive probability and the attention weights in Equation (3) are available with no additional computational cost. Also, in comparison to random sampling which requires iterative sampling during inference, the proposed approach is faster in terms of inference wall-clock time since the updated DCM can be used without any additional computation (refer to Appendix C.1 for the comparison).

3.2 Multimodal Bayesian Aggregation

Motivation With DCM, we obtain the encoded context representations for the m th modality input as follows:

where ϕ and ψ are the encoders' parameters for the m th modality. Next, given the target set T m X , we compute the target-specific context representations with the attention mechanism:

where the attention weight A ( T m X , C m X ) ∈ ❘ N T × N m can be computed by any scoring function without loss of generality. Note that a single i th target sample consists of multiple representations from M modalities { r m ∗ ,i = r m ∗ [ i, :] ∈ ❘ d e } M m =1 . It is important to aggregate these multiple modalities into one latent variable/representation z i for making a unified prediction of the label.

Proposed Solution Instead of using a deterministic aggregation scheme, we propose Multimodal Bayesian Aggregation (MBA), inspired by [61]. Specifically, we view r m ∗ ,i as a sample from a Gaussian distribution with mean z i : p ( r m ∗ ,i | z i ) = N ( r m ∗ ,i | z i , diag ( s m ∗ ,i ) ) where s m ∗ ,i = s m ∗ [ i, :] ∈ ❘ d e is the i th sample in s m ∗ . Additionally, we impose an informative prior on z i : p ( z i ) = ∏ M m =1 N ( u m , diag ( q m )) with the mean context representations of u m ∈ ❘ d e and q m ∈ ❘ d e , which assign uniform weight across the context set as follows:

where θ and ω are the encoders' parameters for the m th modality. Note that the encoders in Equation (4) and (6) are different because they approximate different distribution parameters.

Lemma 3.1 (Gaussian posterior distribution with factorised prior distribution) . If we have p ( x i | µ ) = N ( x i | µ, Σ i ) and p ( µ ) = ∏ n i =1 N ( µ 0 ,i , Σ 0 ,i ) for n i.i.d. observations of D dimensional vectors, then the mean and covariance of posterior distribution p ( µ | x ) = N ( µ | µ n , Σ n ) are:

As both p ( z i ) and p ( r m ∗ ,i | z i ) are Gaussian distributions, the posterior of z i is also a Gaussian distribution: p ( z i | r ∗ ,i ) = N ( z i | µ z i , diag ( σ 2 z i ) ) , whose mean and variance are obtained by using Lemma 3.1 as follows:

where ⊘ and ⊗ are element-wise inverse and element-wise product respectively (see Appendix A for proof).

We highlight that if the variance s m ∗ ,i of a modality formed by the target-specific context representation is high in σ 2 z i , q m formed by the mean context representations dominates the summation of the two terms. Also, if both s m ∗ ,i and q m are high for a modality, the modality's contribution to the summation over modalities is low. By doing so, we minimise performance degradation caused by uncertain modalities (see Section 5.1 for its robustness to noisy samples). Refer to Appendix C.2 for ablation studies on different multimodal aggregation methods.

3.3 Adaptive RBF Attention

Motivation The attention weight A ( T m X , C m X ) in Equation (3) and (5) can be obtained by any attention mechanism. The dot-product attention is one of the simple and efficient attention mechanisms, which has been used in various NPs such as [28, 53, 9, 47]. However, we have observed that it is not suitable for our multimodal uncertainty estimation problem, as the dot-product attention assigns excessive attention to context points even when the target distribution is far from the context distribution. The right figure in Figure 2a shows the attention weight in Equation (1) with A ( x m OOD , C m X ) where x m OOD is a single OOD target sample (grey sample) far from the context set (red and blue samples). This excessive attention weight results in overconfident predictive probability for OOD samples (see the left figure of Figure 2a), which makes the predictive uncertainty of a classifier hard to distinguish the ID and OOD samples.

0.1

0.3

0.5

0.7

0.9

Probability

Context Memory Elements

0.0

0.2

0.4

0.6

0.8

Attention Weight (OOD)

(a) Dot-product

0.1

0.3

0.5

0.7

0.9

Probability

Context Memory Elements

0.000

0.005

0.010

0.015

0.020

Attention Weight (OOD)

(b) RBF without

L

RBF

0.1

0.3

0.5

0.7

0.9

Probability

Context Memory Elements

0.000

0.005

0.010

0.015

0.020

Attention Weight (OOD)

(c) RBF with L RBF

Input: Multimodal data T M as the target set, the number of input modalities M , and the number of classes K .

Initialise: Multimodal context memory C M by randomly sampling T M .

for Mini-batch do

for Modality m = 1 to M do

Context representations r m , s m ← Equation (4)

Attention weight A ( T m X , C m X ) ← Equation (9)

Target-specific context representations r m ∗ , s m ∗ ← Equa- tion (5)

Mean context representations u m , q m ← Equation (6)

Unimodal predictive probability ̂ T m Y ← Equation (13)

Unimodel prediction loss L m T Y ← Equation (14)

Unimodal supervised contrastive loss L m CL ← Equation (10)

end for

Gaussian posterior distribution with aggregated multi- modal representations p ( Z | R ∗ ( C M , T M X )) ← Equation (8)

Unified predictive probability ̂ T Y ← Equation (12) Optimise with L

for Modality m = 1 to M do

for Label class k = 1 to K do

Update context memory C m X,k ← Equation (2)-(3)

end for

end for

end for

(d) Pseudocode of MNPs

Proposed Solution To address the overconfident issue of the dot-product attention, we propose an attention mechanism based on RBF. RBF is a stationary kernel function that depends on the relative distance between two points (i.e., x -x ′ ) rather than the absolute locations [65], which we define as κ m ( x, x ′ ) = exp ( -1 2 || x -x ′ ( l m ) 2 || 2 ) where || · || 2 is the squared Euclidean norm, and l m is th the lengthscale parameter that controls the smoothness of the distance in the m modality input space. The RBF kernel is one of the widely used kernels in GPs [64, 65], and its adaptation with DNNs has shown well-calibrated and promising OOD detection [40, 41, 58, 24]. Formally, we define the attention weight using the RBF kernel as:

where the elements of G ( T m X , C m X ) ∈ ❘ N T × N m are [ G ] ij = κ m ( T m X [ i, :] , C m X [ j, :]) , and Sparsemax [39] is an alternative activation function to Softmax. It is defined as Sparsemax ( h ) := argmax p ∈ ∆ K -1 || p - h || 2 where ∆ K -1 is the K -1 dimensional simplex ∆ K -1 := { p ∈ ❘ K | 1 T p = 1 , p ≥ 0 } . Here we use Sparsemax instead of the standard Softmax because Sparsemax allows zero-probability outputs. This property is desirable because the Softmax's output is always positive even when κ m ( x, x ′ ) = 0 (i.e., || x -x ′ || 2 →∞ ) leading to higher classification and calibration errors (see Appendix C.3 for ablation studies).

The lengthscale l m is an important parameter that determines whether two points are far away from each other (i.e., κ m ( x, x ′ ) → 0 ) or close to each other (i.e., 0 < κ m ( x, x ′ ) ≤ 1 ). However, in practice, l m has been either considered as a non-optimisable hyperparameter or an optimisable parameter that requires a complex initialisation [24, 59, 58, 42, 65, 56]. To address this issue, we propose an adaptive learning approach of l m to form a tight bound to the context distribution by leveraging the supervised contrastive learning [27]. Specifically, we let anchor index i ∈ T ind ≡ { 1 , . . . , N T } with

negative indices N ( i ) = T ind \{ i } and positive indices P ( i ) = { p ∈ N ( i ) : T Y [ p, :] = T Y [ i, :] } . Given an anchor sample of the target set, the negative samples refer to all samples except for the anchor sample, while the positive samples refer to other samples that share the same label as the anchor sample. We define the multimodal supervised contrastive loss as:

where | P ( i ) | is the cardinality of P ( i ) , and τ is the temperature scale. This loss encourages higher RBF output of two target samples from the same class and lower RBF output of two target samples from different classes by adjusting the lengthscale. In addition to L M CL , a l 2 -loss is added to form the tighter bound by penalising large lengthscale. Overall, the loss term for our adaptive RBF attention is L RBF = L M CL + α ∗ 1 M ∑ M m =1 || l m || with the balancing coefficient α .

We show the difference between the RBF attention without L RBF (see Figure 2b) and the adaptive RBF attention with L RBF (see Figure 2c). It can be seen that the decision boundary modelled by the predictive probability of the adaptive RBF attention is aligned with the data distribution significantly better than the non-adaptive one. For ablation studies with real-world datasets, see Appendix C.4.

3.4 Conditional Predictions

We follow the standard procedures of Gaussian process classification to obtain predictions [65, 24, 42, 19] where we first compute the predictive latent distribution p ( f ( T M X ) | C M , T M X ) as a Gaussian distribution by marginalising Z = { z i } N T i =1 :

where T M X = { T m X } M m =1 , R ∗ ( C M , T M X ) = { r ∗ ,i } N T i =1 , and p ( f ( T M X ) | Z ) is parameterised by a decoder. Then, we obtain the predictive probability ̂ T Y by:

Similarly, we can obtain the unimodal predictive latent distribution p ( f ( T m X ) |{ C m X , C m Y } , T m X ) and the unimodal predictive probability ̂ T m Y for the m th modality as:

We minimise the negative log likelihood of the aggregated prediction and the unimodal predictions by:

Since Equations (11)-(13) are analytically intractable, we approximate the integrals by the Monte Carlo method [44]. The overall loss for MNPs is L = L T Y + β ∗ L RBF where β is the balancing term. Refer to Figure 2d for the summarised steps of training MNPs.

4 Related Work

Neural Processes CNP and the latent variant of CNP that incorporates a latent variable capturing global uncertainty were the first NPs introduced in the literature [13, 14]. To address the under-fitting issue caused by the mean context representation in these NPs, Kim et al. [28] proposed to leverage the attention mechanism for target-specific context representations. This approach has been shown to be effective in subsequent works [53, 29, 9, 47, 67, 22]. Many other variants, such as SNP [51, 67], CNAP [48], and MPNPs [5] were proposed for common downstream tasks like 1D regression, 2D image completion [28, 14, 15, 10, 61, 26, 20], and image classification [13, 62, 25]. However, none of these studies have investigated the generalisation of NPs to multimodal data. This study is the first to consider NPs for multimodal classification and its uncertainty estimation.

Multimodal Learning The history of multimodal learning that aims to leverage multiple sources of input can be traced back to the early work of Canonical Correlation Analysis (CCA) [21]. CCA learns the correlation between two variables which was further improved by using feed-forward networks by Deep CCA (DCCA) [2]. With advances in various architectures of DNNs, many studies on multimodal fusion and alignment were proposed [4, 12, 55, 6]. In particular, transformer-based models for vision-language tasks [36, 52, 1] have obtained great attention. Nonetheless, most of these methods were not originally intended for uncertainty estimation, and it has been demonstrated that many of them exhibit inadequate calibration [16, 43].

Multimodal Uncertainty Estimation Multimodal uncertainty estimation is an emerging research area. Its objective is to design robust and calibrated multimodal models. Ma et al. [37] proposed the Mixture of Normal-Inverse Gamma (MoNIG) algorithm that quantifies predictive uncertainty for multimodal regression. However, this work is limited to regression, whereas our work applies to multimodal classification. Han et al. [17] developed TMC based on the Dempster's combination rule to combine multi-view logits. In spite of its simplicity, empirical experiments showed its limited calibration and capability in OOD detection [24]. Jung et al. [24] proposed MGP that combines predictive posterior distributions of multiple GPs by the product of experts. While MGP achieved the current SOTA performance, its non-parametric framework makes it computationally expensive. Our proposed method overcomes this limitation by generalising efficient NPs to imitate GPs.

5 Experiments

Apart from measuring the test accuracy, we assessed our method's performance in uncertainty estimation by evaluating its calibration error, robustness to noisy samples, and capability to detect OOD samples. These are crucial aspects that a classifier not equipped to estimate uncertainty may struggle with. We evaluated MNPs on seven real-world datasets to compare the performance of MNPs against four unimodal baselines and three multimodal baselines.

To compare our method against unimodal baselines, we leveraged the early fusion (EF) method [3] that concatenates multimodal input features to single one. The unimodal baselines are (1) MC Dropout (MCD) [11] with dropout rate of 0.2, (2) Deep Ensemble (DE) [32] with five ensemble models, (3) SNGP 's GP layer [35], and (4) ETP [25] with memory size of 200 and a linear projector. ETP was chosen as a NP baseline because of its original context memory which requires minimal change of the model to be used for multimodal classification.

The multimodal baselines consist of (1) Deep Ensemble (DE) with the late fusion (LF) [3] where a classifier is trained for each modality input, and the final prediction is obtained by averaging predictions from all the modalities, (2) TMC [17], and (3) MGP [24]. For TMC and MGP, we followed the settings proposed by the original authors. In each experiment, the same feature extractors were used for all baselines. We report mean and standard deviation of results from five random seeds. The bold values are the best results for each dataset, and the underlined values are the second-best ones. See Appendix B for more detailed settings.

5.1 Robustness to Noisy Samples

Experimental Settings In this experiment, we evaluated the classification performance of MNPs and the robustness to noisy samples with six multimodal datasets [24, 17]. Following [24] and [17], we normalised the datasets and used a train-test split of 0.8:0.2. To test the robustness to noise, we added zero-mean Gaussian noise with different magnitudes of standard deviation during inference (10 evenly spaced values on a log-scale from 10 -2 to 10 1 ) to half of the modalities. For each noise level, all possible combinations of selecting half of the modalities (i.e., ( M M/ 2 ) ) were evaluated and averaged. We report accuracy and expected calibration error (ECE) for each experiment. Please refer to Appendix B for details of the datasets and metrics.

Results We provide the test results without noisy samples in Table 1, 2, and 5 and the results with noisy samples in Table 3. In terms of accuracy, MNPs outperform all the baselines in 5 out of 6 datasets. At the same time, MNPs provide the most calibrated predictions in 4 out of 6 datasets, preserving the non-parametric GPs' reliability that a parametric model struggles to achieve. This shows that MNPs bring together the best of non-parametric models and parametric models. Also,

MNPs provide the most robust predictions to noisy samples in 5 out of 6 datasets achieved by the MBA mechanism. Both unimodal and multimodal baselines except MGP show limited robustness with a large performance degradation.

In addition to its superior performance, MNPs, as shown in Table 5, are also faster than the SOTA multimodal uncertainty estimator MGP in terms of wall-clock time per epoch (up to × 5 faster) measured in the identical environment including batch size, GPU, and code libraries (see Appendix B for computational complexity of the two models). This highly efficient framework was made possible

by DCM that stores a small number of informative context points. It also highlights the advantage of using efficient DNNs to imitate GPs, which a non-parametric model like MGP struggles to achieve.

5.2 OODDetection

Experimental Settings Following the experimental settings of [24], we trained the models with three different corruption types of CIFAR10-C [18] as a multimodal dataset and evaluated the OOD detection performance using two different test datasets. The first test dataset comprised half CIFAR10C and half SVHN [46] samples, while the second test dataset comprised half CIFAR10-C and half CIFAR100 [31] samples. We used the Inception v3 [54] pretrained with ImageNet as the backbone of all the baselines. The area under the receiver operating characteristic (AUC) is used as a metric to classify the predictive uncertainty into ID (class 0) and OOD (class 1).

Results Table 4 shows test accuracy and ECE with CIFAR10-C and OOD AUC against SVHN and CIFAR100. MNPs outperform all the baselines in terms of ECE and OOD AUC. A large difference in OOD AUC is observed which shows that the proposed adaptive RBF attention identifies OOD samples well. Also, we highlight MNPs outperform the current SOTA MGP in every metric. A marginal difference in test accuracy between DE (LF) and MNPs is observed, but MNPs achieve much lower ECE (approximately 8 . 6 folds) with higher OOD AUC than DE (LF).

6 Conclusion

In this study, we introduced a new multimodal uncertainty estimation method by generalising NPs for multimodal uncertainty estimation, namely Multimodal Neural Processes. Our approach leverages a simple and effective dynamic context memory, a Bayesian method of aggregating multimodal representations, and an adaptive RBF attention mechanism in a holistic and principled manner. We evaluated the proposed method on the seven real-world datasets and compared its performance against seven unimodal and multimodal baselines. The results demonstrate that our method outperforms all the baselines and achieves the SOTA performance in multimodal uncertainty estimation. A limitation of this work is that despite the effectiveness of the updating mechanism of DCM in practive, it is not theoretically guaranteed to obtain the optimal context memory. Nonetheless, our method effectively achieves both accuracy and reliability in an efficient manner. We leave developing a better updating mechanism for our future work. The broader impacts of this work are discussed in Appendix D.

References

H. Akbari, L. Yuan, R. Qian, W.-H. Chuang, S.-F. Chang, Y. Cui, and B. Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 24206-24221. Curran Associates, Inc., 2021.

G. Andrew, R. Arora, J. Bilmes, and K. Livescu. Deep canonical correlation analysis. In S. Dasgupta and D. McAllester, editors, Proceedings of the 30th International Conference on Machine Learning , volume 28 of Proceedings of Machine Learning Research , pages 1247-1255, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.

T. Baltrušaitis, C. Ahuja, and L.-P. Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence , 41(2):423-443, 2019. doi: 10.1109/TPAMI.2018.2798607.

G. Bhatt, P. Jha, and B. Raman. Representation learning using step-based deep multi-modal autoencoders. Pattern Recognition , 95:12-23, 2019. ISSN 0031-3203. doi: https://doi.org/10. 1016/j.patcog.2019.05.032.

C. Cangea, B. Day, A. R. Jamasb, and P. Lio. Message passing neural processes. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning , 2022.

S. D. Dao, D. Huynh, H. Zhao, D. Phung, and J. Cai. Open-vocabulary multi-label image classification with pretrained vision-language model. In 2023 IEEE International Conference on Multimedia and Expo (ICME) , pages 2135-2140. IEEE, 2023.

A. Dempster. Upper and lower probabilities induced by a multi- valued mapping. Annals of Mathematical Statistics , 38:325-339, 1967.

J. F. Dipnall, R. Page, L. Du, M. Costa, R. A. Lyons, P. Cameron, R. de Steiger, R. Hau, A. Bucknill, A. Oppy, E. Edwards, D. Varma, M. C. Jung, and B. J. Gabbe. Predicting fracture outcomes from clinical registry data using artificial intelligence supplemented models for evidence-informed treatment (praise) study protocol. PLOS ONE , 16(9):1-12, 09 2021. doi: 10.1371/journal.pone.0257361.

L. Feng, H. Hajimirsadeghi, Y. Bengio, and M. O. Ahmed. Latent bottlenecked attentive neural processes. In The Eleventh International Conference on Learning Representations , 2023.

A. Foong, W. Bruinsma, J. Gordon, Y. Dubois, J. Requeima, and R. Turner. Meta-learning stationary stochastic process prediction with convolutional neural processes. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 8284-8295. Curran Associates, Inc., 2020.

Y. Gal and Z. Ghahramani. Bayesian convolutional neural networks with bernoulli approximate variational inference. arXiv preprint arXiv:1506.02158 , 2015.

M. Gao, J. Jiang, G. Zou, V. John, and Z. Liu. Rgb-d-based object recognition using multimodal convolutional neural networks: A survey. IEEE Access , 7:43110-43136, 2019. doi: 10.1109/ ACCESS.2019.2907071.

M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. Rezende, and S. M. A. Eslami. Conditional neural processes. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1704-1713. PMLR, 10-15 Jul 2018.

M. Garnelo, J. Schwarz, D. Rosenbaum, F. Viola, D. J. Rezende, S. Eslami, and Y. W. Teh. Neural processes. arXiv preprint arXiv:1807.01622 , 2018.

J. Gordon, W. P. Bruinsma, A. Y. K. Foong, J. Requeima, Y. Dubois, and R. E. Turner. Convolutional conditional neural processes. In International Conference on Learning Representations , 2020.

C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1321-1330. PMLR, 06-11 Aug 2017.

Z. Han, C. Zhang, H. Fu, and J. T. Zhou. Trusted multi-view classification. In International Conference on Learning Representations , 2021.

D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations , 2019.

J. Hensman, A. Matthews, and Z. Ghahramani. Scalable Variational Gaussian Process Classification. In G. Lebanon and S. V. N. Vishwanathan, editors, Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics , volume 38 of Proceedings of Machine Learning Research , pages 351-360, San Diego, California, USA, 09-12 May 2015. PMLR.

P. Holderrieth, M. J. Hutchinson, and Y. W. Teh. Equivariant learning of stochastic fields: Gaussian processes and steerable conditional neural processes. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 4297-4307. PMLR, 18-24 Jul 2021.

H. Hotelling. Relations between two sets of variates. Biometrika , 28(3/4):321-377, 1936. ISSN 00063444.

S. Jha, D. Gong, H. Zhao, and Y. Lina. NPCL: Neural processes for uncertainty-aware continual learning. In Advances in Neural Information Processing Systems , 2023.

A. Jøsang. Belief fusion. In Subjective Logic , volume 3, chapter 12, pages 207-236. Springer, 2016.

M. C. Jung, H. Zhao, J. Dipnall, B. Gabbe, and L. Du. Uncertainty estimation for multi-view data: The power of seeing the whole picture. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022.

M. Kandemir, A. Akgül, M. Haussmann, and G. Unal. Evidential turing processes. In International Conference on Learning Representations , 2022.

M. Kawano, W. Kumagai, A. Sannai, Y. Iwasawa, and Y. Matsuo. Group equivariant conditional neural processes. In International Conference on Learning Representations , 2021.

P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 18661-18673. Curran Associates, Inc., 2020.

H. Kim, A. Mnih, J. Schwarz, M. Garnelo, A. Eslami, D. Rosenbaum, O. Vinyals, and Y. W. Teh. Attentive neural processes. In International Conference on Learning Representations , 2019.

M. Kim, K. R. Go, and S.-Y. Yun. Neural processes with stochastic attention: Paying more attention to the context dataset. In International Conference on Learning Representations , 2022.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto , 2009.

B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 6405-6416, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.

B.-J. Lee, S. Hong, and K.-E. Kim. Residual neural processes. Proceedings of the AAAI Conference on Artificial Intelligence , 34(04):4545-4552, Apr. 2020. doi: 10.1609/aaai.v34i04. 5883.

J. Lee, M. Humt, J. Feng, and R. Triebel. Estimating model uncertainty of neural networks in sparse information form. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 5702-5713. PMLR, 13-18 Jul 2020.

J. Liu, Z. Lin, S. Padhy, D. Tran, T. Bedrax Weiss, and B. Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 7498-7512. Curran Associates, Inc., 2020.

J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

H. Ma, Z. Han, C. Zhang, H. Fu, J. T. Zhou, and Q. Hu. Trustworthy multimodal regression with mixture of normal-inverse gamma distributions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 6881-6893. Curran Associates, Inc., 2021.

A. Malinin and M. Gales. Predictive uncertainty estimation via prior networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , NIPS'18, page 7047-7058, Red Hook, NY, USA, 2018. Curran Associates Inc.

A. Martins and R. Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In M. F. Balcan and K. Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 1614-1623, New York, New York, USA, 20-22 Jun 2016. PMLR.

L. Meronen, C. Irwanto, and A. Solin. Stationary activations for uncertainty calibration in deep learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 2338-2350. Curran Associates, Inc., 2020.

L. Meronen, M. Trapp, and A. Solin. Periodic activation functions induce stationarity. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 1673-1685. Curran Associates, Inc., 2021.

D. Milios, R. Camoriano, P. Michiardi, L. Rosasco, and M. Filippone. Dirichlet-based gaussian processes for large-scale calibrated classification. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

M. Minderer, J. Djolonga, R. Romijnders, F. A. Hubis, X. Zhai, N. Houlsby, D. Tran, and M. Lucic. Revisiting the calibration of modern neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

K. P. Murphy. Probabilistic machine learning: an introduction . MIT press, 2022.

T. Nair, D. Precup, D. L. Arnold, and T. Arbel. Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. Medical Image Analysis , 59:101557, 2020. ISSN 1361-8415. doi: https://doi.org/10.1016/j.media.2019.101557.

Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning , 2011.

T. Nguyen and A. Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 16569-16594. PMLR, 17-23 Jul 2022.

J. Requeima, J. Gordon, J. Bronskill, S. Nowozin, and R. E. Turner. Fast and flexible multitask classification using conditional neural adaptive processes. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

A. G. Roy, S. Conjeti, N. Navab, and C. Wachinger. Bayesian quicknat: Model uncertainty in deep whole-brain segmentation for structure-wise quality control. NeuroImage , 195:11-22, 2019. ISSN 1053-8119. doi: https://doi.org/10.1016/j.neuroimage.2019.03.042.

M. Sensoy, L. Kaplan, and M. Kandemir. Evidential deep learning to quantify classification uncertainty. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

G. Singh, J. Yoon, Y. Son, and S. Ahn. Sequential neural processes. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019.

A. Suresh and S. Srinivasan. Improved attentive neural processes. In Fourth Workshop on Bayesian Deep Learning at the Conference on Neural Information Processing Systems , volume 14, page 17, 2019.

C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2016.

H. Toriya, A. Dewan, and I. Kitahara. Sar2opt: Image alignment between multi-modal images using generative adversarial networks. In IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium , pages 923-926, 2019. doi: 10.1109/IGARSS.2019.8898605.

N. Ulapane, K. Thiyagarajan, and S. Kodagoda. Hyper-parameter initialization for squared exponential kernel-based gaussian process regression. In 2020 15th IEEE Conference on Industrial Electronics and Applications (ICIEA) , pages 1154-1159, 2020. doi: 10.1109/ ICIEA48937.2020.9248120.

M. Valdenegro-Toro. Deep sub-ensembles for fast uncertainty estimation in image classification. arXiv preprint arXiv:1910.08168 , 2019.

J. Van Amersfoort, L. Smith, Y. W. Teh, and Y. Gal. Uncertainty estimation using a single deep deterministic neural network. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 9690-9700. PMLR, 13-18 Jul 2020.

J. van Amersfoort, L. Smith, A. Jesson, O. Key, and Y. Gal. On feature collapse and deep kernel learning for single forward pass uncertainty. arXiv preprint arXiv:2102.11409 , 2021.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

M. Volpp, F. Flürenbrock, L. Grossberger, C. Daniel, and G. Neumann. Bayesian context aggregation for neural processes. In International Conference on Learning Representations , 2021.

J. Wang, T. Lukasiewicz, D. Massiceti, X. Hu, V. Pavlovic, and A. Neophytou. NP-match: When neural processes meet semi-supervised learning. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 22919-22934. PMLR, 17-23 Jul 2022.

Y. Wen, D. Tran, and J. Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations , 2020.

C. Williams. Computing with infinite networks. In M. Mozer, M. Jordan, and T. Petsche, editors, Advances in Neural Information Processing Systems , volume 9. MIT Press, 1996.

C. K. Williams and C. E. Rasmussen. Gaussian Processes for Machine Learning . The MIT Press, 2006.

B. Xu, N. Wang, T. Chen, and M. Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853 , 2015.

J. Yoon, G. Singh, and S. Ahn. Robustifying sequential neural processes. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 10861-10870. PMLR, 13-18 Jul 2020.

A Lemma and Proof

For the comprehensiveness of proof, we duplicate Lemma 3.1 here.

Lemma A.1 (Gaussian posterior distribution with factorised prior distribution) . If we have p ( x i | µ ) = N ( x i | µ, Σ i ) and p ( µ ) = ∏ n i =1 N ( µ 0 ,i , Σ 0 ,i ) for n i.i.d. observations of D dimensional vectors, then the mean and covariance of posterior distribution p ( µ | x ) = N ( µ | µ n , Σ n ) are:

Proof.

where we dropped constant terms for clarity. From Equation (18) and (19), we can see that:

If we use Lemma A.1 with diagonal covariance matrices for p ( r m ∗ ,i | z i ) = N ( r m ∗ ,i | z i , diag ( s m ∗ ,i ) ) and p ( z i ) = ∏ M m =1 N ( u m , diag ( q m )) , we can obtain the posterior distribution of N ( z i | µ z i , diag ( σ 2 z i ) ) as follows:

where ⊘ is the element-wise inversion, and ⊗ is the element-wise product.

B Experimental Details

In this section, we outline additional details of the experimental settings including the datasets (Appendix B.1), hyperparameters of the models used (Appendix B.2), metrics (Appendix B.3), and a brief analysis of computational complexity of MGP and MNPs (Appendix B.4). For all the experiments, we used the Adam optimiser [30] with batch size of 200 and the Tensorflow framework. All the experiments were conducted on a single NVIDIA GeForce RTX 3090 GPU.

B.1 Details of Datasets

Synthetic Dataset Figure 2 shows the predictive probability and the attention weight of different attention mechanisms. Here, we describe the dataset and the settings used for the demonstrations.

We generated 1,000 synthetic training samples (i.e., N train = 1 , 000 ) for binary classification by using the Scikit-learn's moon dataset 3 with zero-mean Gaussian noise ( std = 0 . 15 ) added. The test samples were generated as a mesh-grid of 10,000 points (i.e., 100 × 100 grid with N test = 10 , 000 ). The number of points in the context memory N m was set to 100. In this demonstration, we simplified the problem by setting M = 1 which is equivalent to the unimodal setting and illustrated the difference in attention mechanisms.

Robustness to Noisy Samples Dataset In Section 5.1, we evaluated the models' robustness to noisy samples with the six multimodal datasets. The details of each dataset are outlined in Table 6. These datasets lie within a feature space where each feature extraction method can be found in [17].

OODDetection Dataset Weused CIFAR10-C [18] which consists of corrupted images of CIFAR10 [31]. 15 types of corruptions and five levels of corruption for each type are available for the dataset. Following [24], we used the first three types as multimodal inputs with different levels of corruption (1, 3, and 5).

B.2 Details of Models

In our main experiments, four unimodal baselines with the early fusion (EF) method [3] (MC Dropout, Deep Ensemble (EF), SNGP, and ETP) and three multimodal baselines with the late fusion (LF) method [3] (Deep Ensemble (LF), TMC, and MGP) were used. In this section, we describe the details of the feature extractors and each baseline.

Feature Extractors We used the same feature extractor for all the methods to ensure fair comparisons of the models. For the synthetic dataset, the 2D input points were projected to a high-dimensional space ( d m = 128 ) with a feature extractor that has 6 residual fully connected (FC) layers with the ReLU activation. For the OOD detection experiment, the Inception v3 [54] pretrained with ImageNet was used as the feature extractor. Note that the robustness to noisy samples experiment does not require a separate feature extractor as the dataset is already in a feature space.

MCDropout Monte Carlo (MC) Dropout [11] is a well-known uncertainty estimation method that leverages existing dropout layers of DNNs to approximate Bayesian inference. In our experiments, the dropout rate was set to 0.2 with 100 dropout samples used to make predictions in the inference stage. The predictive uncertainty was quantified based on the original paper [11].

Deep Ensemble Deep ensemble [32] is a powerful uncertainty estimation method that trains multiple independent ensemble members. In the case of the unimodal baseline, we employed five ensemble members, whereas for the multimodal baseline, a single classifier was trained independently for each modality input. In both scenarios, the unified predictions were obtained by averaging the predictions from the ensemble members, while the predictive uncertainty was determined by calculating the variance of those predictions.

SNGP Spectral-normalized Neural Gaussian Process (SNGP) [35] is an effective and scalable uncertainty estimation method that utilises Gaussian process (GP). It consists of a feature extractor with spectral normalisation and a GP output layer. Since we used the identical feature extractor for all the baselines, we only used the GP layer in this work. Following [24], the model's covariance matrix was updated without momentum with λ = π/ 8 for the mean-field approximation. As the original authors proposed, we quantified the predictive uncertainty based on the Dempster-Shafer theory [7] defined as u ( x ) = K/ ( K + ∑ K k =1 exp( logit k ( x ))) where logit k ( · ) is the k th class of output logit with the number of classes K .

ETP Evidential Turing Processes (ETP) [25] is a recent variant of NPs for uncertainty estimation of image classification. Since none of the existing NPs can be directly applied to multimodal data, there are several requirements to utilise them for multimodal classification: 1) a context set in the inference stage (e.g., context memory) and 2) a method of processing multimodal data. ETP was selected due to its inclusion of the original context memory, requiring minimal modifications to be applicable to our task. We used the memory size of 200 and quantified the predictive uncertainty with entropy as proposed by the original paper [25].

TMC Trusted Multi-view Classification (TMC) [17] is a simple multimodal uncertainty estimation based on the Subjective logic [23]. We used the original settings of the paper with the annealing epochs of ten for the balancing term. TMC explicitly quantifies its predictive uncertainty based on the Dempster-Shafer theory [7].

MGP Multi-view Gaussian Process (MGP) [24] is the current SOTA multimodal uncertainty estimation method that combines predictive posterior distributions of multiple GPs. We used the identical settings of the original paper with the number of inducing points set to 200 and ten warm-up epochs. Its predictive uncertainty was quantified by the predictive variance as proposed by the original paper [24].

MNPs(Ours) The encoders and decoder in Multimodal Neural Processes (MNPs) consist of two FC layers with the Leaky ReLU activation [66] after the first FC layer. A normalisation layer is stacked on top of the second FC layer for the encoders. For enc m ψ ( · ) and enc m ω ( · ) that approximate the variance of distributions, we ensure positivity by transforming the outputs as h + = 0 . 01 + 0 . 99 ∗ Softplus ( h ) where h is the raw output from the encoders. l m of the adaptive RBF attention was initialised as 10 ∗ ✶ ∈ R d m , and DCM was initialised by randomly selecting training samples. We used five samples for the Monte Carlo method to approximate the integrals in Equations (11)-(13), which we found enough in practice. Refer to Table 7 for the hyperparameters of MNPs. We provide the impact of N m on the model performance in Appendix C.1.

B.3 Details of Metrics

Apart from test accuracy, we report the expected calibration error (ECE) [16] and the area under the receiver operating characteristic curve (AUC). ECE is defined as:

where n is the number of testing samples, B i is a bin with partitioned predictions with the number of bins b , | B i | is the number of elements in B i , acc ( B i ) is the accuracy of predictions in B i , and

conf ( B i ) is the average predictive confidence in B i . Following [35] and [24], we set b = 15 . AUC was used for the OOD detection experiment with ground truth labels of class 0 being the ID samples and class 1 being the OOD samples. Each model's predictive uncertainty was used as confidence score to predict whether a test sample is a ID or OD sample.

B.4 Computational Complexity of MGP and MNPs

In addition to the empirical difference of wall-clock time per epoch in Table 5, we provide computational complexity of the two models in Table 8. We assume that the number of inducing points in MGP equals to the number of context points in MNPs. During training of MNPs, each modality requires a cross-attention ( O ( N m N T ) ) and a contrastive learning ( O (( N T ) 2 ) ) that sum to O ( M ( N m N T +( N T ) 2 )) with M being the number of modalities, whereas during inference, each modality only requires the cross-attention which results in O ( MN m N T ) .

C Ablation Studies

In this section, we analyse MNPs' performance with different settings and show the effectiveness of the proposed framework.

C.1 Context Memory Updating Mechanisms

We compare the updating mechanism of DCM based on MSE in Equation (2)-(3) with three other baselines: random sampling, first-in-first-out (FIFO) [62], and cross-entropy based (CE). Random sampling bypasses DCM and randomly selects training samples during inference. For FIFO, we follow the original procedure proposed by [62] that updates the context memory during training and only uses it during inference. CE-based mechanism replaces j ∗ in Equation (3) with j ∗ =

We provide experimental results for all the experiments outlined in Section 5. We highlight that random sampling and FIFO achieve high accuracy both without noise and with noise as shown in Table 9 and 11. However, MSE and CE outperform the others in terms of ECE in Table 10 and OOD AUC in Table 12. As MSE and CE select the new context points based on classification error, the selected context points tend to be close to decision boundary, which is the most difficult region to classify. We believe this may contribute to the lower calibration error, suppressing overconfident predictions. The MSE and CE mechanisms show comparable overall results, but we selected MSE for its lower ECE. In terms of time efficiency, Table 13 shows that random sampling is slower than the other three methods.

For DCM updated by MSE, we also provide difference in performance for a range of number of context points N m in Figure 3-9. For every figure, the bold line indicates the mean value, and the shaded area indicates 95% confidence interval. Unsurprisingly, the training time and the testing time increase with respect to N m . The general trend in test accuracy across the datasets shows the benefit of increasing the number of context points. However, the performance gain in ECE and OOD AUC is ambivalent as different patterns are observed for different datasets. We leave an in-depth analysis of this behaviour for our future study.

50

100

150

200

250

N m

0.993

0.994

0.995

Test accuracy

50

100

150

200

250

N m

0.005

0.006

0.007

Test ECE

50

100

150

200

250

300

320

340

360

380

Training time (ms/epoch)

50

100

150

200

250

20

25

30

Testing time (ms/epoch)

N m

N m

50

100

150

200

250

N m

0.7

0.8

0.9

Test accuracy

50

100

150

200

250

N m

0.04

0.06

0.08

0.10

Test ECE

50

100

150

200

250

N m

32.5

35.0

37.5

40.0

Training time (ms/epoch)

50

100

150

200

250

N m

5

10

Testing time (ms/epoch)

100

200

300

400

500

N m

0.940

0.945

0.950

0.955

Test accuracy

100

200

300

400

500

N m

0.035

0.040

0.045

0.050

0.055

Test ECE

100

200

300

400

500

N m

60

62

64

Training time (ms/epoch)

100

200

300

400

500

N m

6.5

7.0

7.5

Testing time (ms/epoch)

200

400

600

800

N m

0.925

0.930

0.935

Test accuracy

200

400

600

800

N m

0.015

0.020

0.025

Test ECE

200

400

600

800

N m

1600

1800

2000

2200

Training time (ms/epoch)

200

400

600

800

N m

60

80

100

Testing time (ms/epoch)

100

200

300

400

500

N m

0.76

0.77

0.78

Test accuracy

100

200

300

400

500

N m

0.030

0.035

0.040

0.045

Test ECE

100

200

300

400

500

N m

340

350

360

370

Training time (ms/epoch)

100

200

300

400

500

N m

24

26

28

30

Testing time (ms/epoch)

100

200

300

400

500

N m

0.2

0.4

0.6

Test accuracy

100

200

300

400

500

N m

0.025

0.050

0.075

0.100

Test ECE

100

200

300

400

500

N m

550

600

650

700

Training time (ms/epoch)

100

200

300

400

500

N m

35

40

45

Testing time (ms/epoch)

100

200

300

400

500

N m

0.744

0.746

0.748

0.750

Test accuracy

100

200

300

400

500

N m

0.010

0.015

0.020

0.025

0.030

Test ECE

100

200

300

400

500

N m

0.85

0.86

0.87

OOD AUC (SVHN)

100

200

300

400

500

N m

0.77

0.78

0.79

OOD AUC (CIFAR100)

100

200

300

400

500

N m

10000

12000

14000

Training time (ms/epoch)

100

200

300

400

500

N m

500

600

700

Testing time (ms/epoch)

C.2 Multimodal Aggregation Methods

We demonstrate the performance of MBA compared with two other methods namely 'Concat' and 'Mean'. 'Concat' bypasses MBA and directly provides r m ∗ of multiple modalities to the decoder (see Figure 1) by simple concatenation followed by passing to a MLP which lets p ( f ( T M X ) | C M , T M X ) in Equation (12) be parameterised by a decoder where { C M , T M X } = MLP ( Concat ( { r m ∗ } M m =1 )) . Concat ( · ) represents concatenating multiple vectors along their feature dimension. Similarly,'Mean' also bypasses MBA and simply averages the multiple modalities into single representation. Formally, p ( f ( T M X ) | C M , T M X ) parameterised by a decoder where { C M , T M X } = 1 M ∑ M m =1 r m ∗ .

The results are shown in Table 14-17. In every case, MBA outperforms both baselines. While similar performance can be observed for Handwritten, Scene15, and Caltech101, large differences are observed in CUB, PIE, and HMDB across different metrics. The test accuracy of CIFAR10 is almost consistent across all methods, but large gaps in ECE and OOD performance are observed. This highlights the importance of MBA, especially in robustness and calibration performance.

C.3 Attention Types

We decompose the attention weight A ( T m X , C m X ) in Equation (9) as follows:

where Norm ( · ) is the normalisation function such as Softmax and Sparsemax, and Sim ( · , · ) as the similarity function such as the dot-product and the RBF kernel. We provide experimental results of four different combinations of normalisation functions and similarity functions in Table 18-21.

Among the four combinations, the RBF function with Sparsemax outperforms the others in most cases. More importantly, Table 20 shows a large difference in robustness to noisy samples between the RBF function with Sparsemax and the dot-product with Sparsemax, even when a marginal difference in accuracy is shown in Table 18. For instance, for the PIE dataset, the difference in accuracy without noisy samples is 0.3, but the difference increases to 6.0 in the presence of noisy samples. The same pattern is observed with OOD AUC in Table 21. This illustrates the strength of RBF attention that is more sensitive to distribution-shift as shown in Figure 2. Lastly, for both similarity functions, Sparsemax results in superior overall performance.

C.4 Adaptive Learning of RBF Attention

We have shown that the effectiveness of learning the RBF attention's parameters with the synthetic dataset in Figure 2. We further provide the ablation studies with the real-world datasets in Table 22-25.

D Broader Impacts

As a long-term goal of this work is to make multimodal classification of DNNs more trustworthy by using NPs, it has many potential positive impacts to our society. Firstly, with transparent and calibrated predictions, more DNNs can be deployed to safety-critical domains such as medical diagnosis. Secondly, this work raises awareness to the machine learning society to evaluate and review reliability of a DNN model. Lastly, our study shows the potential capability of NPs in more diverse applications. Nevertheless, a potential negative impact may exist if the causes of uncertain predictions are not fully understood. To take a step further to reliable DNNs, the source of uncertainty should be transparent to non-experts.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the problem of uncertainty estimation specifically for multi-modal data, i.e., inputs consisting of different sources. Specifically, it improves the popular Neural Process (NP) in three aspects: dynamic context update, multi-modal Bayesian aggregation, and a novel attention mechanism based on the RBF kernel. The paper demonstrates superior test accuracy, uncertainty estimation (calibration and OOD detection), and robustness on multiple benchmarks compared to prior works.\n\nSTRENGTHS:\n* The paper studies the less well-studied area of uncertainty estimation for multi-modal data. This has practical implications as many real-world applications can be multi-modal.  \n* The paper presents multiple metrics including accuracy, calibration, OOD detection, and robustness, which provide a holistic evaluation of the proposed method. \n* The idea of dynamic context update is most interesting to me. Specifically, the method replaces uninformative context samples with informative ones where \"uninformative-ness\" is measured by attention weights and \"informative-ness\" is measured by difficulty of classification.\n\nWEAKNESSES:\n* While the paper presents three innovations and claims that they are all tailored to multi-modal data, only the Bayesian aggregation is inherently related to multi-modal inputs. The dynamic context update and the RBF context mechanism do not utilize the multi-modal characteristic. This makes the contributions and claims less coherent. \n\n* It seems the Bayesian aggregation mechanism is not ablated. It's not clear how much improvement it brings to the overall performance as it is the main mechanism responsible for tackling multi-modal inputs. Does the improved robustness come from the aggregation algorithm or the other two components?\n\nQUESTIONS:\n* Can the authors comment on the importance of the Bayesian aggregation component? \n* It would be good to consolidate a single table (for one dataset) to directly show the effectiveness of each component in terms of accuracy, uncertainty, and robustness.\n\nLIMITATIONS:\nThe work does not have an apparent negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a confidence calibration algorithm for multimodal classification problems. The algorithm includes three key components: 1) dynamic context memory, 2) multimodal Bayesian aggregation, and 3) adaptive radial basis function. The algorithm is evaluated on multiple benchmarks using classification accuracy, calibration, and OOD performance. On multiple benchmarks, the proposed algorithm is able to obtain SOTA results.\n\nSTRENGTHS:\n1, The targeted problem and challenges are explicitly stated. For each challenge, a technically reasonable solution is proposed.\n2, The proposed algorithm is able to obtain strong empirical results, obtaining SOTA results on multiple benchmarks and different metrics.\n3, Comprehensive ablation studies have been conducted to demonstrate each components.\n4, As illustrated in Figure 2, the proposed adaptive RBF algorithm outperformed other attention mechanisms.\n\nWEAKNESSES:\n1, Some notation definitions are confusing. Please refer to section \"Questions\" for details.\n2, Because of the above notation section 3.2 is a bit hard to follow.\n3, Ablation study is conducted for individual components. It would be interesting to design ablation studies and see how each component affects the end-to-end performance.\n4, The component directly related to multimodality learning, i.e., multimodal Bayesian aggregation, is based on previously work [1]. It would be good if the authors can explain how the proposed MBA component is different from previous work, either in the revision or rebuttal. \n\n[1] M. Volpp, F. Flürenbrock, L. Grossberger, C. Daniel, and G. Neumann. Bayesian context aggregation for neural processes. In International Conference on Learning Representations, 2021.\n\nQUESTIONS:\n1, **Line 71**, from my understanding, the context set C and target set T contains both training and test datasets. But $N_C +N_T=N_{\\text{train}}$. Why is the summation of $N_C$ and $N_T$ only equal to the number of training points?\n\n2, **Line 148 and 150**, what are $r^m$  and $s^m$? How are they connected/different from $r^m_{\\*}$ and $s^m_{\\*}$?\n\n3, **Line 160**, how are encoders parameterized by $\\theta$ and $\\omega$ connected/different from those by $\\phi$ and $\\psi$?\n\nLIMITATIONS:\nThe authors explicitly discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a multimodal neural processes (neural network generaliation of Gaussian processes) model.\n\nThe overall approach has several novel elements:\n* A way to maintain a dynamic context set throughout training (e.g a support set for few-shot learning, these context sets are needed for neural processes) \n* A Bayesian aggregation scheme for combining multiple modalities\n* An adaptive RBF attention mechanism as an alternative to (vanilla) dot product attention which the authors argue is overconfident on OOD samples.\n\nExperiments on a suite of datasets show that the proposed method is faster, more accurate and better calibrated, and better at detecting OOD samples than prior work.\n\nSTRENGTHS:\nThis reads like a polished piece of work with clear writing, extensive ablations (in the supplementary) and novel technical contributions.  Experimental results seem fairly convincing (that the proposed method is indeed better than prior similar works).  My caveat is that my familiarity with this part of the literature on Neural Processes is passing at best.\n\nWEAKNESSES:\nI have no major complaints about this paper, but some things to point out are:\nIt is pretty unclear what are the actual multiple modalities in the datasets that are used for experiments (which makes these experiments not particularly compelling perhaps unless you are already familiar with this line of work).  I had to trace through a series of cited works to figure out that (likely) these different modalities are different features extracted e.g using different networks.\nSection 3.2 could be written more clearly — I couldn’t figure out why there were multiple encoders until encountering Lemma 3.1 — it’d be much better to explain this up front.\nA nit: it’s also not clear when reading section 3.1 how often the dynamic context memory is meant to be updated until I read the pseudocode much later (I’d recommend saying that it’s a per-minibatch update somewhere).\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}

comments: SUMMARY:
This paper proposes a new method for multimodal uncertainty estimation by extending neural processes. The authors summarize three challenges to do that and give solutions correspondingly. Experimental results show that the proposed method is more robust and outperforms existing baselines.

---

Thanks for the clarification! My main concerns are addressed.

STRENGTHS:
The proposed method (MNPs) can achieve good performance empirically.

WEAKNESSES:
1. The motivation is not clear. Indeed, there is little discussion about neural processes with multimodal data, and extending unimodal neural processes to multimodal scenarios could be challenging. However, existing works discussed the multi-view data [16,21]. The reason why extending the neural processes for multimodal data (instead of extending/improving [16,21]) remains unclear.


2. Lack of understanding. I would like to suggest that the authors add some ablation studies. From current experiments, we can only see that proposed MNPs outperform baselines. However, why it can do that is not clear, e.g., why the unimodal method (DE) can achieve the best test accuracy.

QUESTIONS:
(Line 185): Maybe I missed something, but why using RBF can address the overconfident issue?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends one of the promising uncertainty estimation method: Neural Process from unimodal to multimodal. This is motivated by the fact that: current techniques are predominantly designed for unimodal data, and directly applying them to multimodal information is ineffective. However, this extension poses several challenges: 1) how fuse information from different modalities effectively and efficiently; 2) the context memory of the original design grows in proportion to M - the number of modality, which is memory-consuming, and sets the question of how to maintain it small yet informative; 3) how to find an appropriate length-scale of the Radial Basis Function (RBF) that ensures a tight boundary between in-distribution and out-of-distribution data.\nTo tackle these challenges, the paper introduces three solutions: 1) Dynamic Context Memory to choose the most informative samples in the context memory, 2) Multimodal Bayesian Aggregation, and 3) Adaptive RBF Attention. Empirical experiments show that the method can outperform other unimodal and multimodal baselines in terms of maintaining good accuracy and calibration at the presence of noise and in terms of out-of-distribution detection.\n\nSTRENGTHS:\n1. originality: the first to extend Neural Process from unimodal to multimodal and effectively combine existing techniques to solve some of the challenges incurred.\n2. clarity: in general, the paper articulates its ideas clearly, even though some parts remain difficult to understand or unclear in terms of details (see the weakness part)\n3. significance: the proposed method shows promising results in terms of outperform most unimodal and multimodal baselines.\n\nWEAKNESSES:\n**Clarity**: the clarity of the technical introduction (Sec. 3) and the experiment (Sec. 4) parts can be largely improved. \n1. The method introduction part could be **challenging to comprehend for readers unfamiliar with the Neural Process and its technical details**. It lacks adequate contextual knowledge, such as an explanation for why the context memory is required, why it is used to store training samples rather than using a context feature. I think the Sec. 2 should be improved to be more comprehensive and general in introducing the Neural Process, to avoid the readers to revisit several referenced papers repeatedly.\n1. The **math details can be overwhelming and at times confusin**g. For instance, when reviewing Sec 3.3, it necessitates frequent referral back to Section 3.2. It would be helpful if the semantic meaning of symbols is reintroduced when they appear, preventing the need for readers to revisit previous sections for notation understanding.\n\t1. **Derivation details**: The logic in some derivations is not easy to understand, e.g. Line 221 introduces the function $f(T_X^M)$ without any prior explanation. Also, there is no explanation on how equation (12) is derived, and why softmax is applied on a density function. These issues impede reading and understanding. The formula (4) and formula (6) are not consistent as their parameter notation is not the same on $\\omega$ and $\\psi$.\n\t2. The MNPs **pseudocode** lacks a high-level introduction and has heavy reference links, failing to provide clarifying information.\n\t3. **Captions**: Some table captions lack sufficient information. For instance, Table 4/5 does not clearly describe the dataset and settings used.\n\t4. **Experimental Setup**: The presentation of the experimental settings, such as how datasets e.g. Caltech101 were modified into multi-view datasets, is not clear.\n\n**Soundness**: Despite the multimodal setting, the experiments use only multi-view image data. The lack of experiments with multimodal combinations such as image + text impedes the proof of the model's effectiveness in various settings.\n\nQUESTIONS:\nsee questions listed on the weakness part\n\nLIMITATIONS:\nThe author have listed one limitation that the updating mechanism is not theoretically guaranteed to obtain the optimal context memory.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-23"}