Beyond Unimodal: Generalising Neural Processes for Multimodal Uncertainty Estimation
Authors: Myong Chol Jung
He Zhao
Joanna Dipnall
Lan Du
Venue: NeurIPS 2023 poster (2023)
Keywords: Uncertainty estimation
multimodality
neural processes

Abstract: Uncertainty estimation is an important research area to make deep neural networks (DNNs) more trustworthy. While extensive research on uncertainty estimation has been conducted with unimodal data, uncertainty estimation for multimodal data remains a challenge. Neural processes (NPs) have been demonstrated to be an effective uncertainty estimation method for unimodal data by providing the reliability of Gaussian processes with efficient and powerful DNNs. While NPs hold significant potential for multimodal uncertainty estimation, the adaptation of NPs for multimodal data has not been carefully studied. To bridge this gap, we propose Multimodal Neural Processes (MNPs) by generalising NPs for multimodal uncertainty estimation. Based on the framework of NPs, MNPs consist of several novel and principled mechanisms tailored to the characteristics of multimodal data. In extensive empirical evaluation, our method achieves state-of-the-art multimodal uncertainty estimation performance, showing its appealing robustness against noisy samples and reliability in out-of-distribution detection with faster computation time compared to the current state-of-the-art multimodal uncertainty estimation method.

Despite the Beyond Unimodal: Generalising Neural Processes for Multimodal Uncertainty Estimation
Myong Chol Jung
He Zhao CSIRO's Data61 he. zhao@ieee.or org
Monash University
david. jung@monash, edu
Joanna Dipnall
Lan Du
Monash University
Monash University
jo di dippalllee 10mnaas0mmnsh...mmmm edu
1 aan du@monash. edu
Abstract
Uncertainty estimation is an important research area to make deep neural networks (DNNs) more trustworthy. While extensive research uncertainty estimation has been conducted with unimodal data, uncertainty estimation for multimodal data remains challenge. Neural processes (NPs) have been demonstrated to be an effective uncertainty estimation method for unimodal data by providing the reliability of Gaussian processes with efficient and powerful DNNs. While NPs hold significant potential for multimodal uncertainty estimation, the adaptation NPs for multimodal data has not been carefully studied. To bridge this gap, propose Multimodal Neural Processes (MNPs) by generalising NPs for multimodal uncertainty estimation. Based the framework of NPs. MNPs consist of several novel and principled mechanisms tailored to the characteristics of multimodal data, In extensive empirical evaluation, method achieves state-of-the-a multimodal uncertainty estimation performance, showing its appealing robustness against noisy samples and reliability out-o--distribution detection with faster computation time compared to the current state-of-the-as multimodal uncertainty estimation method.
Introduction
Uncertainty estimation of deep neural networks (DNNs) is an essential research area in the devel- opment of reliable and well-calibrated models for safety-critical domains 44 45
remarkable success achieved by DNNs, common issue with these model is their tendency to make overconfident predictions for both in-distribution (ID) and out-tf-distribuution (OOD) samples
16 43 35) Extensive research has been conducted to mitigate this problem, but most of these efforts are limited to unimodal data, neglecting the consideration of multimodal data
50
In many practical scenarios, safety-critical domains typically involve the processing of multimodal data. For example, medical diagnosis classification that uses X-ray, radiology text reports and the patient's 's medical record history data can be considered as a form of multimodal learning that requires trustworthy predictions [ Despite the prevalence of such multimodal data, the problem of uncertainty estimation for multimodal data has not been comprehensively studied. Furthermore, it has been demonstrated that using existing unimodal uncertainty estimation techniques directly for multimodal data is ineffective, emphasising the need for meticulous investigation of multimodal uncertainty estimation techniques 24 17
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Several studies investigated the problem of multimodal uncertainty estimation with common tasks including 1) assessing calibration performance, 2) evaluating robustness noisy inputs, and 3) detecting OOD samples. Trusted Multi-view Classification (TMC) 17 is single-forward deter- ministic classifier that uses the Demptser's combination rule 7 to combine predictions obtained from different modalities. The current state-of-the-art (SOTA) in this field is Multi-view Gaussian Process (MGP) [24], which is a non-pat con-parametric Gaussian process (GP) classifier that utilises the product-of-experts to combine predictive distributions derived from multiple modalities. Although MGP has shown reliability of GPs in this context, should be noted that the computational cost of a
GP increases cubically with the number of samples [99 65
Neural processes (NPs) offer an alternative approach that utilises the representation power of DNNs to imitate the non-pan non-parametric behaviour of GPs while maintaining a lower computational cost L3 14] It
has been shown that NPs can provide promising uncertainty estimation for both regression 28 33 29
and classification tasks [62 25 involving unimodal data. Despite the promising potential of NPs for multimodal data, to the best of our knowledge, no research has yet investigated the feasibility of using NPs for multimodal uncertainty estimation.
In this work, we propose a new multimodal uncertainty estimation framework called Multimodal Neural Processes (MNPs) by generalising NPs for multimodal uncertainty estimation. MNPs have three key components: the dynamic context memory (DCM) that efficiently stores and updates informative training samples, the multimodal Bayesian aggregation (MBA) method which enables a principled combination of multimodal latent representations, and the adaptive radial basis function (RBF) attention mechanism that facilitates well-calibrated predictions. Our contributions are:
1. We introduce a novel multimodal uncertainty estimation method by generalising NPs that comprise the DCM, the MBA, and the adaptive RBF attention.
2. We conduct rigorous experiments on seven real-world datasets and achieve the new SOTA perfor- mance in classification accuracy, calibration, robustness to noise, and OOD detection.
We show that MNPs achieve faster computation time (up to 5 folds) compared to the current SOTA multimodal uncertainty estimation method.
2 Background
Multimodal Classification for multimodal classification. Specifically, we consider a multimodal training dataset Dtrain = {a;") MM mm=1 9t ) Me where M is the number of input modalities and Norain is the number of training samples. We assume that each 7th sample has M modalities of input 2: E Rd"" with the input dimension of d and a single one-hot encoded label ys with the number of classes K. In this study, we consider the input space to be a feature space. The objective is to estimate the labels of test samples {y: {
The aim of this research is investigate uncertainty estimation
given Drest = ({=?) m== M
M Mreat where Ntest is the number of test samples.
Neural Processes NPs are stochastic processes using DNNs to capture the ground truth stochastic processes that generate the given data [3. NPs learn the distribution of functions and provide uncertainty of target samples. preserving the property of GPs. At the same time, NPs exploit function approximation of DNNs in a more efficient manner than GPs. For NPs, both training and test datasets have a context set C = [Cx, Cr = {-.f of J. and a target set T = (Txx Ty) = { y J with Ne being the number of context samples and Ny the number of target sample and the learning objective of NPs is to maximise the likelihood of C(Tx)
Conditional Neural Processes (CNPs), the original work of NPs [3], maximise p(Ty Tx) with Ne r(C) No EN enc, (cat 22 :f: E Rd. where enc, is an encoder parameterised by p. catl;; is the concatenation of two vectors along the feature dimension, and de is the feature dimension. The mean vector r(C') the permutation invariant representation that summarises the context set which is passed to a decoder with Tx to estimate Ty.
The original CNPs the unweighted mean operation obtain r(C) by treating all the context points equally, which has been shown to be underfitting (28) To improve over this, Attentive Neural
"The modality notation is intentionally omitted the original NPs are designed for unimodal inputs [3 14

---

Keys
Query
Modality
Values
F1
ence
Adaptive RBF Attention
Modality Module
DCM
ence
MBA
Decoder
Modality M Module
Mean operation
4
Sampling
Figure Model diagram of MNPs: DCM refers to Dynamic Context Memory, MBA refers to Multimodal Bayesian Aggregation, and RBF refers to radial basis function.
Processes (ANPs) 28 leveraged the scaled dot-product cross-attention 660 to create target-specific context representations that allocate higher weight the closer context points:
r- (C,Tx) = Softmax(TxxC3/ ene (cn
(I)
A(Tx.Cx)EI(MxXX
IR Rexxe
where A(Tx, Cx) is the attention weight with Tx as the query, Cx is the key, the encoded context set is the value, and is the input dimension of Tx and Cx. The resulted target-specific context representation has been shown to enhance the expressiveness of the task representation 28 53 47.67 67].
The training procedure NPs often randomly splits a training dataset to the context and the target sets (i.e., No + Nr Ntrain). In the inference stage, context set is provided for the test/target samples (e.g., a support set few-shot learning or unmasked patches in image completion). While the context set for the test dataset is given for these tasks, NPs also have been applied to other tasks where the context set is not available (e.g., semi cemi-supervised learning [62 and uncertainty estimation for image classification )- In those tasks, existing studies have used context memory during inference, which is composed of training samples that are updated while training 25 62]. It is
assumed that the context memory effectively represents the training dataset with smaller number of samples. We build upon these previous studies by leveraging the concept of context memory in a more effective manner, which suitable for our task.
Note that the existing NPs such as ETP 25 can be applied to multimodal classification by concate- nating multimodal features into single unimodal feature. However, in Section 5.11 show that this approach results limited robustness to noisy samples.
Multimodal Neural Processes
The aim of this study is to generalise NPs to enable multimodal uncertainty estimation. In order to achieve this goal, there are significant challenges that need to be addressed first. Firstly, it is essential to have an efficient and effective context memory for a classification task as described in Section Secondly, systematic approach of aggregating multimodal information is required to provide unified predictions. Finally, the model should provide well-calibrated predictions without producing overconfident estimates as described in Section[]]
Our MNPs, shown in Figure [ address these challenges with three key components respectively: 1) the dynamic context memory (DCM) (Section 3.1 1). 2) the multimodal Bayesian aggregation (MBA) (Section 3.2). and 3) the adaptive radial basis function (RBF) attention (Section 3.3). From Section 3.1 to 3 claborate detailed motivations and proposed solutions of each challenge, and

---

Section 3.4 outlines the procedures for making predictions. Throughout this section, we refer the multimodal target set T' = (fTx) m = { mm=1 Vi J Jizi to the samples from training and test datasets and the multimodal context memory C = {Ox) Cy Jm=1 to the context set.
L
3.1 Dynamic Context Memory
Motivation In NPs, context must be provided for target set, which however always possible for non-meta-learnin tasks during inference. One simple approach to adapt NPs to these tasks is to randomly select context points from the training dataset, but its performance is suboptimal as randomly sampling a few samples may not adequately represent the entire training distribution. Wang et al. [62] proposed an alternative solution by introducing a first-in-firttout (FIFO) memory that stores the context points with the predefined memory size. Although the FIFO performs slightly better than the random sampling in practice, the performance is still limited because updating the context memory is independent of the model's predictions. Refer to Appendix C.1 for comparisons.
Proposed Solution To overcome this limitation of the existing context memory, propose simple and effective updating mechanism for the context memory in the training process, which we call Dynamic Context Memory (DCM). We partition context memory {C C.r) of m²h modality into K subsets (K as the number of classes) {CT : C {C*,, : C 1K 1k=1 to introduce class-balance context memory, with each subset devoted to one class. Accordingly, N = No K where No is the number of context elements per modality (i.e., the size of {C CT and Nm is the number of class-specific context samples per modality. This setting resembles the classification setting in [33, where class-specific context representations are obtained for each class.
DCM is initialised by taking Nº random training samples from each class of each modality and is updated every mini-batch during training by replacing the least "informative" element in the memory with possibly more "informative" sample. We regard the element in DCM that receives the smallest attention weight (i.e.. A(Tx, Cx) Equation (I)) during training as the least informative one and the target point that is most difficult to classify (i.e., high classification error) more informative sample that should be added to DCM to help the model learn the decision boundary. Formally, this can be written as:
CR - T.L"'. EE (L, m (L....MM}
(2)
NT
K
i' = argmin iE......, Non NT
Σ ACTR, CRA
argmax jE(I.....NTT
-
(3)
K
TR NTXT
where :] indicates the 7th row vector of matrix, [i.j] indicates the 7th row and the jth column element of a matrix, and T't is the predicted probability of the mh modality input. i selects the context memory element that receives the least average attention weight in mini-batch during training, and j" selects the target sample with the highest classification error. To measure the error between the predictive probability and the ground truth of target sample, can use squared error (MSE) (Equation (3) or cross-entropy loss (CE). We empirically found that the former gives better performance in our experiments. Refer to Appendix C. for ablation studies comparing the two and the impact of N on its performance.
The proposed updating mechanism is very efficient as the predictive probability and the attention weights in Equation (3) are available with no additional computational cost. Also, in comparison to random sampling which requires iterative sampling during inference, the proposed approach is faster in terms of inference wall-clock time since the updated DCM can be used without any additional computation (refer to Appendix C.I for the comparison).
3.2 Multimodal Bayesian Aggregation
Motivation as follows:
With DCM, obtain the encoded context representations for the mh modality input
-= ency (cat CPI) R N" xd.
TT enc" (cat (cat(CCX; Cy1) CTT) R N xd.
(4)

---

where & and j are the encoders' parameters for the m modality. Next, given the target set TX, we compute the target-specific context representations with the attention mechanism:
A(TX) ) E IR NTxd,
ST A(TX: C th E R Nrxx.
(5)
where the attention weight A(Tx: Cx) E I RN can be computed by any scoring function without loss of generality. Note that a single ith target sample consists of multiple representations from M modalities {r = r [ E RT I=-1- It is important to aggregate these multiple for making a unified prediction of the label.
modalities into one latent variableereprresetttiio
Proposed Solution Instead of using deterministic aggregation scheme, propose Multi- modal Bayesian Aggregation (MBA), inspired by [6]]. Specifically, we view + as a sample from Gaussian distribution with mean z;: (zz) N (173 (r" where s s [t, :] R& is the 7th sample in s.. Additionally, we impose an informative prior on 2j: p(z;) = I NN (w"", ddiag("")) with the mean context representations um E Rd and
9tth E Rd,, which assign uniform weight across the context set as follows:
m
Nom Σ ency (eat (C (1, :]: (T:))))) 8th
Σ
ene (cat (C C7,, (6)
N
where and are the encoders' parameters for the th modality. Note that the encoders in Equation (4) and (6) are different because they approximate different distribution parameters.
Lemma 3.1 (Gaussian posterior distribution with factorised prior distribution). we have p(r;lu) = (zalu, Ej) and p(u) II. (No, Σo.) for i.i.d. observations of D dimensional vectors, then the mean and covariance of posterior distribution p(u(ær = - (ullun En) are:
Σn
E + mo!1)
An Σn
Σ
(IT x; + IT.
As both p(zi) and p(r (z:) are Gaussian distributions, the posterior of is also Gaussian distribu- tion: (x,)) N (2il diag(o (o.))) whose mean and variance are obtained by using Lemma 3.1 as follows:
M Σ ((mH)) + (a)))0 mm=1
M Σ -
(s u s (m))
2
M/,, = 02, s
(r.)h @
(8)
where @ and @ are element-wise inverse and element-wise product respectively (see Appendix for proof).
We highlight that if the variance sh of modality formed by the target-specific context representation is high in o². 9thh formed by the mean context representations dominates the summation of the two terms. Also, if both sw, and q"" are high for modality, the modality's contribution to the summation over modalities is low. By doing so, we minimise performance degradation caused by uncertain modalities (see Section 1 for its robustness to noisy samples). Refer Appendix C for ablation studies on different multimodal aggregation methods.
3.3 Adaptive RBF Attention
The attention weight A(T C. in Equation 3 and (5) can be obtained by any atten- tion mechanism. The dot-prt attention is one of the simple and efficient attention mechanisms, which has been used in various NPs such as (28 53 9 47]. However, we have observed that is not suitable for our multimodal uncertainty estimation problem, as the dot-product attention assigns excessive attention to context points even when the target distribution far from the context distribu- tion. The right figure in Figure 2a shows the attention weight in Equation ( with A(x C') where FOoD single OOD target sample (grey sample) far from the context set (red and blue samples). This excessive attention weight results in overconfident predictive probability for OOD samples (see the left figure of Figure 2a), which makes the predictive uncertainty of a classifier hard to distinguish the ID and OOD samples.
Motivation
5

---

09
Atnestion Weighn (DOD)
Input: Multimodal data T the target set, the number of input modalities M, and the number of classes K.
0.6
07
Frobaanity
Initialise: Multimodal context memory CM by randomly sampling T
R.5
0.
for Mini-butch do for Modality
9.1
0.0
Coatext Memory Elaments
M do Context representations t Equation f
Attention weight A(TX' CX)) Equation
(a) Dot-product
Target-specific context representations r."h 8 Equa-
Attes Atteziion Weigh (OOD)
0.020
tion (5
Mean context representations w"", """""" <- Equation 6
23
"podadiiltt
5.015
Unimodal predictive probability Py E-Euuaionn
0.010
Unimodel prediction loss Ciyy < Equation [4]
Unimodal supervised contrustive loss LZ1 < Equation
0.005
100
end for
0.000
Menory Elanents
Gaussian posterior distribution with aggregated multi- modal representations p( R2 (C ,Tx + Equation
(b) RBF without CRBF
0.020
Unified predictive probability Ty + Equation 12
Attenton Wagh (0OD)
Optimise with L for Modality m = do
Label class
0.010
Update context memory Cx Equation
0.005
end for
end for
Memoory Elements
(c) RBF with CRBF
(d) Pseudocode of MNPs
Figure 2: Predictive probability of upper circle samples (blue) is shown in the left column with (a) the dot-product attention, (b) the RBF attention without CRAF, (c) and the RBF attention with LRAF. Upper circle samples (blue) are class 1, lower circle samples (red) are class 2, and grey samples are OOD samples. The attention weight A A(="o C of OOD sample across 100 context points shown in the right column. The summarised steps of training MNPs are shown in (d). Refer to Appendix B for the experimental settings.
Proposed Solution To address the overconfident issue of the dot-product attention, we propose an attention mechanism based RBF. RBF is stationary kernel function that depends the relative distance between two points (i.e., x - x²) rather than the absolute locations [65] which we
define as nMTh (x, = exp
{-1)) where || ||² is the squared Euclidean norm, and
the lengthscale parameter that controls the smoothness of the distance in the m² modality input space. The RBF kernel is of the widely used kernels in GPs 664
65 and its adaptation with Formally, we define
DNNs has shown well-calibrated and promising OOD detection
40
58
the attention weight using the RBF kernel
A(Tx,
C = Sparsemax(G(1y) C*)
where the elements of G(TX) CX) E Ny X are [G] = TM (Tx ji :] CX :)) and Sparsemax argmax
39 is alternative activation function to Softmax. It defined as Sparsemax
PEAK
0}. Here we use Sparsemax instead of the standard Softmax because Sparsemax allows zero-probabilitg outputs. This property is desirable because the Softmax's 's output is always positive even when n²' (x, x') 0
2 where --1 is the K dimensional simplex K- = (p I I
(i.e., 1|x z' 22 20) leading to higher classification and calibration emoor (sec Appendix for ablation studies).
The lengthscale 1r is an important parameter that determines whether two points are far away from each other (i.e., (x, x²) 0) or close to each other (i.e., 0 < 1z" (xx, x²) < 1). However, in practice, I has been either considered as non-optimisable hyperparameter or an optimisable parameter that requires complex initialisation (24
56 To address this issue, we propose an adaptive learning approach pm to form a tight bound to context distribution by leveraging the supervised contrastive learning
59
58
42
65
[27] Specifically, we let anchor index e Tind {1...., NN. with

---

negative indices (1) = T {i) and positive indices P(i) {p N() - Given an anchor sample of the target set, the negative samples refer all samples except for the anchor sample, while the positive samples refer to other samples that share the same label as the anchor sample. We define the multimodal supervised contrastive loss as:
1 feL
Σ M ==
Σ
M
Nr
log
exp (NIT (TX 11, (p, :1/T) D/T))
CL
M
P(
EMEN(D exp (RT T Tx (,,
PEP())
(10)
where [P()] is the cardinality of P(i), and J is the temperature scale. This loss encourages higher RBF output of two target samples from the same class and lower RBF output of two target samples from different classes by adjusting the lengthscale. In addition to LCL. l2-loss is added to form the tighter bound by penalising large lengthscale. Overall, the loss term for our adaptive RBF attention A
CRBF = Le.. + a* M
|| with the balancing coefficient a.
We show the difference between the RBF attention without LRBF (see Figure 2b and the adaptive RBF attention with LRAF (see Figure (c). can be seen that the decision boundary modelled by the predictive probability of the adaptive RBF attention is aligned with the data distribution significantly better than the non-adaptive one. For ablation studies with real-world datasets, see Appendix C.4
3.4
Conditional Predictions
We follow the standard procedures of Gaussian process classification to obtain predictions [65 24 42 19
where we first compute the predictive latent distribution p( (T JCM ,TX) as a Gaussian distribution by marginalising
( TMM = f 100=-p((00 D(F(TM)
TX) dz
(11)
where T (TT) M== RM TX) (ra,) NT and p(S(T)) ()) is parameterised by
decoder. Then, we obtain the predictive probability Ty by:
Tr ( , d)X)))
(12)
Similarly, we can obtain the unimodal predictive latent distribution p (f (C.. CV and the unimodal predictive probability Ty for the m modality as:
f
(13)
We minimise the negative log likelihood of the aggregated prediction and the unimodal predictions by:
M Σ
M
[ogg Tv
= M
[oog Tr
Σ
Ery
log Ty
(14)
M
m=1
1==1
Since Equations (1)--13) are analytically intractable, we approximate the integrals by the Monte Carlo method [44]. The overall loss for MNPs is = Cty + CRBF where 3 is the balancing
term. Refer to Figure 2d for the summarised steps of training MNPs.
4 Related Work
Neural Processes CNP and the latent variant of CNP that incorporates latent variable capturing
global uncertainty were the first NPs introduced in the literature [13] 14] To address the under-fitting issue caused by the mean context representation in these NPs, Kim et al. (28) proposed to leverage the attention mechanism for target-specific context representations. This approach has been shown to be effective in subsequent works [53
67 (2). Many other variants, such as SNP (71, CNAP 48 and MPNPs 5 were proposed for common downstream tasks like 1D regression,
29
9
47
image completion 28
14 15
10
61
26
20 and image classification [13 62 25 However, none
of these studies have investigated the generalisation of NPs to multimodal data. This study is the first
to consider NPs for multimodal classification and its uncertainty estimation.
7

---

Multimodal Learning The history of multimodal learning that aims to leverage multiple sources of input can be traced back to the early work of Canonical Correlation Analysis (CCA) [2]] CCA learns the correlation between two variables which was further improved by using feed-forward networks by Deep CCA (DCCA) (2 With advances various architectures of DNNs, many studies on multimodal fusion and alignment were proposed A 12 55 6 In particular, transformer- based models for vision- asioonlanguage tasks [36 52 | have obtained great attention. Nonetheless, most of these methods were not originally intended for uncertainty estimation, and it has been demonstrated that many of them exhibit inadequate calibration [6
Multimodal Uncertainty Estimation Multimodal uncertainty estimation is emerging research area. Its objective is to design robust and calibrated multimodal models. Ma et al. [37] proposed the Mixture of Normal-Inverss Gamma (MoNIG) algorithm that quantifies predictive uncertainty for multimodal regression. However, this work is limited to regression, whereas our work applies to multimodal classification. Han et al. [77 developed TMC based the Dempster's combination rule to combine multi-view logits. In spite of its simplicity, empirical experiments showed its limited calibration and capability in OOD detection
[24]. Jung et al. [24] proposed MGP that combines predictive posterior distributions of multiple GPs by the product of experts. While MGP achieved the
current SOTA performance, its non-parametrie framework makes it computationally expensive. Our proposed method overcomes this limitation by generalising efficient NPs to imitate GPs.
5
Experiments
Apart from measuring the test accuracy, we assessed our method's performance in uncertainty estimation by evaluating its calibration error, robustness to noisy samples, and capability detect OOD samples. These are crucial aspects that classifier not equipped to estimate uncertainty may struggle with. We evaluated MNPs on seven real-world datasets compare the performance of MNPs against four unimodal baselines and three multimodal baselines.
To compare our method against unimodal baselines, we leveraged the early fusion (EF) method 3 that concatenates multimodal input features to single one. The unimodal baselines are (1) MC Dropout (MCD) 11 with dropout rate of 0.2, (2) Deep Ensemble (DE) [32 with five ensemble models, (3) SNGP's GP layer (35). and (4) ETP 25) with memory size of 200 and a linear projector.
ETP was chosen as a NP baseline because of its original context memory which requires minimal change of the model to be used for multimodal classification.
The multimodal baselines consist of (1) Deep Ensemble (DE) with the late fusion (LF) [ where a classifier is trained for each modality input, and the final prediction is obtained by averaging predictions from all the modalities, (2) TMC (17), and (3) MGP (24. For TMC and MGP, followed the settings proposed by the original authors. In each experiment, the same feature extractors were used for all baselines. We report mean and standard deviation of results from five random seeds. The bold values are the best results for each dataset, and the underlined values are the second-best ones. See Appendix B for more detailed settings.
5.1 Robustness to Noisy Samples
Experimental Settings In this experiment, we evaluated the classification performance of MNPs and the robustness noisy samples with six multimodal datasets [24 17]. Following 24 and [17]. we normalised the datasets and used a train-test split of 0.8:0.2. To test the robustness to noise, we added zero-mean Gaussian noise with different magnitudes of standard deviation during inference (10 evenly spaced values on a log-scale from 10-2 to 10²) to half of the modalities. For each noise level, all possible combinations of selecting half of the modalities (i.e., (M/2) were evaluated and averaged. We report accuracy and expected calibration error (ECH) for each experiment. Please refer to Appendix B for details of the datasets and metrics.
We provide the test results without noisy samples in Table I and and the results with noisy samples in Table In terms of aceuracy, MNPs outperform all the baselines in 5 out of datasets. At the same time, MNPs provide the most calibrated predictions in 4 out of 6 datasets, preserving the paramarai GPs reliability that a parametric model struggles to achieve. This shows that MNPs bring together the best of parammreetticc models and parametric models. Also,
Results
8

---

Table 1:
Test accuracy (1).
Dataset
Method
Handwritten
CUB
PIE
Caltech101
Scene
HMDB
MCD
99.25+0.00
92.33++1.09
91.32+00.2
92.95+0.22
71.75++0.25
71.68+00.36
DE (EF)
99.20=10.11
93.16:0.70
91.76110.3
92.99:0.009
72.70=0.39
71,67:0.23
SNGP
98.85=0.22
89.50=0.75
87.06=1.23
91.24:10.6
64.68=4.03
67.65=1.03
ETP
98.75-0.255
92.33:11.9
91.76:10.62
92.08:10.33
72.58=1.35
67.43
DE (LF)
99.25=10.00
92.33:10.70
87.2110.66
92.97:1.13
67.05=0.38
69,98110.6
TMC
98.10=10.14
91.17=1.46
91.1811.72
91.63:1028
67.68-00227
65.1710.87
MGP
98.60+0.14
92.33++0.70
92.06+00.96
93.00++0.33
70.00+0.53
72.30+-0.19
MNPs (Ours)
99.50-00.0
93.50+11.71
95.00+00662
93.46+0..22
77.90-0.71
71.97+0.43
Table 2: Test ECE (4).
Dataset
Method
Handwritten
CUB
PIE
Caltech101
Scenel5
HMDB
MCD
0.009-100000
0.069:10.017
0.299110.005
0.017100003
0.1811:0.003
0.388=0.004
DE (EF)
0.007110000
0.05410.010
0.269:10.004
0.036:0.001
0.089-10.003
0.095=0.003
SNGP
0.023=+0.004
0.200=000010
0.852++0.012
0.442=000004
0.111=0.063
0.227=0.010
ETP
0.020::0.002
0.051=0.009
0.287+0.007
0.096:t0.002
0.045=:0008
0.100=0.010
DE (LF)
0.292++00001
0.270=0.009
0.567++0.006
0.023++0.02
0.319=0.0055
0.270=0.003
TMC
0.013+-0.002
0.141+0.0012
0.072+000011
0.068+0.002
0.180++0.004
0.594-0.008
MGP
0.006:10.00 0.004
0.03810.007
0.009-10.007
0.00910.003
0.062:10.006
0.036
MNPs (Ours)
0.005:10.001
0.049110008
0.040-10.005
0.017+00003
0.039
0.026
Table 3: Average test accuracy across 10 noise levels (t).
Datasct
Method
Handwritten
CUB
PIE
Caltech101
Scene15
HMDB
MCD
02.15
76.08-+0.61
64.65
73.45+-0.11
48.97-00.3
42.63
DE (EF)
82.16-00.18
76.94++0.82
65.53++0.20
03.994-0.19
49.45--0.35
41.92+-0.06
SNGP
72.46+00.41
61.27++1.24
56.52+0.69
56.57+0.11
1..1
37.49+-0.42
ETP
75.85=0.31
73.99=1.12
62.6410.35
66.6240008
46.17=0.58
38.2810.23
DE (LF)
95.63=0.08
76.16-0.28
67.69+0.35
81.85+0.144
50.13=0.27
43.011:0.19
TMC
82.4410.15
74.1910.69
62.18:0.80
71.77:10.22
42.52-10.29
36.6110.30
MGP
97.66-0.112
85.48.10.25
90.9740.19
92.68110.23
63.7410.56
67.02-10.2
MNPs (Ours)
98.5810.10
88.96
93.80-10.49
92.8310.18
7..14
64.1110.15
Table 5: Speed-up (xfolds) measured by the ratio of wall- clock time per epoch of MNPs against MGP
Table 4: Test accuracy (f), ECE (4), and OOD detection AUC (t).
OOD AUC
Method
Test accuracy
BCB
SVHN
CIFAR100
Dataset
Training
Testing
MCD
74.76:10.27
0.01310.002
0.788-100022
0.725
DE (EF)
72.95:10.13
0.154_0.048
0.76910.008
0.721:0.014
Handwritten
1.62
2.66
SNGP
61.51:10.30
0.020=0.003
0.753+0.026
0.705=0.024
CUB
1.41
2.12
ETP
74.42-10.20
0.032-0.003
0.78010.011
0.69510.007
PIE
2.88
5.37
DE (LF)
75.40:100006
0.095-0.001
0.722-100016
0.693=0.006
Caltech101
1.56
3.30
TMC
72.42+00.05
0.108=0.001
0.681-10.04
0.675=0.0006
Scene15
MGP
73.30++0.05
0.018=0.001
0.803++0.007
0.748+0.007
HMDB
2.03
3.27
MNPs (Ours)
74.92+0.07
0.011=0.001
0.872-10.002
0.786=2.005
CIFARIO-C
1.17
2.16
MNPs provide the most robust predictions to noisy samples in 5 out of 6 datasets achieved by the MBA mechanism. Both unimodal and multimodal baselines except MGP show limited robustness with large performance degradation.
In addition to superior performance, MNPs, as shown Table are also faster than the SOTA multimodal uncertainty estimator MGP in terms of wall-clock time per epoch (up to x5 faster) measured the identical environment including batch size, GPU, and code libraries (see Appendix for computational complexity of the two models). This highly efficient framework made possible

---

by DCM that stores a small number of informative context points. It also highlights the advantage of using efficient DNNs to imitate GPs, which a non con-paraettric model like MGP struggles to achieve.
5.2 OOD Detection
Experimental Settings Following the experimental settings of [24], we trained the models with three different corruption types of CIFAR I0-C [88] multimodal dataset and evaluated the OOD detection performance using two different test datasets. The first test dataset comprised half CIFAR 10- C and half SVHN [46] samples, while the second test dataset comprised half CIFARIO-C and half CIFARI00 BI samples. We used the Inception v3 [54] pretrained with ImageNet as the backbone of all the baselines. The area under the receiver operating characteristic (AUC) is used metric to classify the predictive uncertainty into ID (class 0) and OOD (class 1).
Results Table Sshows test accuracy and ECE with CIFARIO-C and OOD AUC against SVHN and CIFARIO0 100. MNPs outperform all the baselines in terms of ECE and OOD AUC. A large difference in OOD AUC is observed which shows that the proposed adaptive RBF attention identifies OOD samples well. Also, we highlight MNPs outperform the current SOTA MGP in every metric. A marginal difference in test accuracy between DE (LF) and MNPs is observed, but MNPs achieve much lower ECE (approximately 8.6 folds) with higher OOD AUC than DE (LF).
Conclusion
In this study, we introduced new multimodal uncertainty estimation method by generalising NPs for multimodal uncertainty estimation, namely Multimodal Neural Processes. Our approach leverages simple and effective dynamic context memory, Bayesian method of aggregating multimodal representations, and an adaptive RBF attention mechanism in holistic and principled manner. We evaluated the proposed method on the seven real-world datasets and compared its performance against seven unimodal and multimodal baselines. The results demonstrate that our method outperforms all the baselines and achieves the SOTA performance in multimodal uncertainty estimation. A limitation of this work is that despite the effectiveness of the updating mechanism of DCM in practive, it is not theoretically guaranteed obtain the optimal context memory. Nonetheless, our method effectively achieves both accuracy and reliability in an efficient manner. We leave developing better updating mechanism for our future work. The broader impacts of this work are discussed in Appendix D
References
[1] H. Akbari, L. Yuan, R. Qian, W.-H. Chuang, S.-F Chang, Y. Cui, and B. Gong. Vatt: Trans- formers for multimodal self-supervised learning from raw video, audio and text. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Informa- tion Processing Systems, volume 34, pages 24206 242211. Curran Associates, Inc., 2021.
[2] G. Andrew, R. Arora, J. Bilmes, and K. Livescu. Deep canonical correlation analysis. In S. Dasgupta and D. McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 1247-1255, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
[3] T. Baltrušaitis, C. Ahuja, and L.-P. Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):423 443, 2019. doi: 10. (099/TAAM..018.22799888
[4] G. Bhatt, P. Jha, and B. Raman. Representation learning using based deep modal autoencoders. Pattern Recognition, 95:12-23, 2019. ISSN 0031-3203. doi: https://doi.or/110. /016//.patcoo... 200..00..03....
[5] C. Cangea, B. Day. A. R. Jamasb, and P. Lio. Message passing neural processes. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning, 2022.
[6] S. D. Dao, D. Huynh, H. Zhao, D. Phung, and J. Cai. Open-vocabulary multi-label image classification with pretrained vision-language model. 2023 IEEE International Conference on Multimedia and Expo (ICME), pages 2135-2140. IEEE, 2023.
10

---

[7] A. Dempster. Upper and lower probabilities induced by multi- valued mapping. Annals of Mathematical Statistics, 38:325-339, 1967.
[8] F. Dipnall, Page, L. Du, M. Costa, R. Lyons, Cameron, R. de Steiger, R. Hau, A. Bucknill, A. Oppy, E. Edwards, D. Varma, M. C. Jung, and B. J. Gabbe. Predicting fracture outcomes from clinical registry data using artificial intelligence supplemented models for evidence-informed treatment (praise) study protocol. PLOS ONE, 16(9):1-12, 09 2021. doi: 10. ptw.:maaal..ooe0223366.6.
[9] L. Feng, H. Hajimirsadeghi, Y. Bengio, and M. O. Ahmed. Latent bottlenecked attentive neural processes, In The Eleventh International Conference on Learning Representations, 2023.
[10] A. Foong, W. Bruinsma, J. Gordon, Y. Dubois, J. Requeima, and R. Turner. Meta-learning stationary stochastic process prediction with convolutional neural processes. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 8284-8295. Curran Associates, Inc., 2020.
[11] Y. Gal and Z. Ghahramani, Bayesian convolutional neural networks with bernoulli approximate variational inference. arXiv preprint #xXiv:1566002188 2015.
[12] M. Gao, J. Jiang, G. Zou, V. John, and Z. Liu. Rgb-d-based object recognition using multimodal convolutional neural networks: A survey. IEEE Access, 7:43 10-43136, 2019. doi: 10.1109/ SCCESS.2019929000
[13] M. Gareelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. Rezende, and S. M. A. Eslami. Conditional neural processes. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1704-1713. PMLR, 10-15 Jul 2018.
[14] M. Garnelo, Schwarz, D. Rosenbaum, F. Viola, D. J. Rezende, S. Eslami, and Y. W. Teh. Neural processes. arXiv preprint / 2018
[15] J. Gordon, W. P. Bruinsma, A. Y. K. Foong, J. Requeima, Y. Dubois, and R. E. Tumner Convolu- tional conditional neural processes. In International Conference on Learning Representations, 2020.
[16] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321-1330. PMLR, 06-11 Aug 2017.
[17] Z. Han, C. Zhang, H. Fu, and J. T. Zhou. Trusted multi-view classification. In International Conference on Learning Representations, 2021.
[18] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representa- tions, 2019.
[19] Hensman, A. Matthews, and Z. Ghahramani. Scalable Variational Gaussian Process Clas- sification. In G. Lebanon and S. V. N. Vishwanathan, editors. Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statisties, volume 38 of Proceedings of Machine Learning Research, pages 351-360, San Diego, California, USA, 09 12 May 2015. PMLR.
[20] Holderrieth, M. J. Hutchinson, and Y. W. Teh. Equivariant learning of stochastic fields: Gaussian processes and steerable conditional neural processes. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4297-4307. PMLR, 18-24 Jul 2021.
[21] H. Hotelling. Relations between two sets of variates. 23411)3327 ISSN 00063444.
[22] S. Jha, D. Gong, H. Zhao, and Y. Lina. NPCL: Neural processes for uncertainty-aware continual learning, In Advances in Neural Information Processing Systems, 2023.
11

---

[23] A. Jøsang. Belief fusion. In Subjective Logic, volume 3, chapter 12, pages 207-236. Springer, 2016.
[24] M. C. Jung, H. Zhao, Dipnall, Gabbe, and L. Du. Uncertainty estimation for multi-view data: The power of seeing the whole picture. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
[25] M. Kandemir, A. Akgül, M. Haussmann, and G. Unal. Evidential turing processes. In Interna- tional Conference on Learning Representations, 2022.
[26] M. Kawano, W. Kumagai, A. Sannai, Y. Iwasawa, and Y. Matsuo. Group equivariant conditional neural processes. In International Conference on Learning Representations, 2021.
[27] P. Khosla, Teterwak, C. Wang, A. Sarna, Tian, Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18661-18673 Curran Associates, Inc., 2020.
[28] H. Kim, A. Mnih, J. Schwarz, M. Garnelo, A. Eslami, D. Rosenbaum, O. Vinyals, and Y. W. Teh. Attentive neural processes, In International Conference on Learning Representations, 2019.
[29] M. Kim, K. R. Go, and S.-Y. Yun. Neural processes with stochastic attention: Paying more attention to the context dataset. In International Conference on Learning Representations, 2022.
[30] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[31] A. Krizhevsky and G. Hinton. Learning multiple layers features from tiny images. Master's s thesis, Department of Computer Science, University of Toronto, 2009.
[32] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 6405-6416, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964
[33] B.-J. Lee, S. Hong, and K.-E. Kim. Residual neural processes. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):4545-4552 Apr. 2020. doi: (0.00///aa..334000 5883.
[34] J. Lee, M. Humt, J. Feng, and R. Triebel. Estimating model uncertainty of neural networks in sparse information form. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5702-5713. PMLR, 13-18 Jul 2020.
[35] Liu, Z. Lin. S. Padhy, D. Tran, T. Bedrax Weiss, and B. Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages
Curran Associates, Inc., 2020.
[36] J. Lu, D. Batra, D. Parikh, and S. Lec. Vilbert: Pretraining task-agnostic visiolinguistic tasks. In H. Wallach, H. Larochelle. A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
representations for vision-and-languaa
[37] H. Ma, Z. Han, C. Zhang, H. Fu, J. T. Zhou, and Q. Hu. Trustworthy multimodal regression with mixture of normal-inverse gamma distributions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 6881-6893. Curran Associates, Inc., 2021.
[38] A. Malinin and M. Gales. Predictive uncertainty estimation via prior networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 7047-7058, Red Hook. NY, USA, 2018. Curran Associates Inc
12

---

[39] A. Martins and R. Astudillo. From softmax to sparsemax: A sparse model of attention and mulli-llbee classification. In M. F. Balcan and K. Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1614-1623, New York, New York, USA, 20-22 Jun 2016. PMLR.
[40] L. Meronen, C. Irwanto, and A. Solin. Stationary activations for uncertainty calibration in deep learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2338-2333 2350. Curran Associates, Inc., 2020.
[41] L. Meronen, M. Trapp, and A. Solin. Periodic activation functions induce stationarity. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 1673-1685. Curran Associates, Inc., 2021.
[42] D. Milios, Camoriano, Michiardi, L. Rosasco, and M. Filippone. Dirichlet-based gaussian processes for scale calibrated classification. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Desa-BBanchh, and R. Garnett, editors, Advances Neural Information Pro- cessing Systems, volume 31. Curran Associates, Inc., 2018.
[43] M. Minderer, J. Djolonga, R. Romijnders, F. A. Hubis, X. Zhai, N. Houlsby, D. Tran, and
M. Lucic. Revisiting the calibration of modern neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[44] K. P. Murphy. Probabilistic machine learning: an introduction. MIT press, 2022.
[45] T. Nair, D. Precup, D. L. Arnold, and T. Arbel. Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. Medical Image Analysis, 59:101557,
2020. ISSN 1361-8415. doi: https://doi.org orgg 1..10.000000 1016/)..mdia..220 2019..
[46] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in nat- ural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[47] T. Nguyen and A. Grover. Transformer neural processes: Uncertainty- aware meta learning via sequence modeling. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages (6569-16599) PMLR, 17-23 Jul 2022.
[48] J. Requeima. J. Gordon, J. Bronskill, S. Nowozin, and R. E. Turner. Fast and flexible multi- task classification using conditional neural adaptive processes. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[49] A. G. Roy, S. Conjeti, N. Navab, and C. Wachinger. Bayesian quicknat: Model uncertainty in deep whole-brain segmentation for structure- wise quality control. Neurolmage, 195:11-22, 2019. ISSN 1053-8119. doi: https://doi.org/00. 10t///0.....111. neuroimage.2019..0 201.00.........
[50] M. Sensoy, L. Kaplan, and M. Kandemir. Evidential deep learning to quantify classification uncertainty. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Biance and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[51] G. Singh, J. Yoon, Y. Son, and S. Ahn. Sequential neural processes. In H. Wallach, H. Larochelle, A. Beygelzimer, A'Alché-Bucc E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[52] C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
[53] A. Suresh and S. Srinivasan. Improved attentive neural processes. In Fourth Workshop on Rayesian Deep Learning at the Conference on Neural Information Processing Systems, volume 14, page 17, 2019.
13

---

[54] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception archi- tecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2016.
[55] H. Toriya, A. Dewan, and I. Kitahara. Sar2opt: Image alignment between multi-modal images using generative adversarial networks. In IGARSS 2019 2019 JEEE International Geoscience and Remote Sensing Symposium, pages 923 926, 2019. doi: 10.1 10.1109/IAASSS.000 21..SS....1.....0000
[56] N. Ulapane, K. Thiyagarajan, and S. Kodagoda. Hyper-parameter initialization for squared exponential kernel-based gaussian process regression. In 2020 15th IEEE Conference on Industrial Electronics and Applications (ICIEA), pages 1 1154-1159, 2020. doi: 10.1109/ 2000449997.220.92455
[57] M. Valdenegro-Tore Toro. Deep sub- eobeeblsebles for fast uncertainty estimation in image classification. Xrxiv preprint arXiv: 1910008100..... 2019.
[58] Van Amersfoort, L.. Smith, W. Teh, and Y. Gal. Uncertainty estimation using a single deep deterministic neural network. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9690-9700. PMLR, 13-18 Jul 2020.
[59] J. Amersfoort, L. Smith, A. Jesson, O. Key, and Y. Gal. On feature collapse and deep kernel learning for single forward pass uncertainty. arXiv preprint arXiv:2 1409, 2021.
[60] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, I. u. Kaiser, and
I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[61] M. Volpp, F. Flürenbrock, I.. Grossherger, C. Daniel, and G. Neumann. Bayesian context aggregation for neural processes. In International Conference Learning Represeniations, 2021.
[62] J. Wang, T. Lukasiewicz, D. Massiceti, X. Hu, V. Pavlovic, and A. Neophytou. NP-match: When neural processes meet semi- supervised learning. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 22919-22934. PMLR, 17-23 Jul 2022.
[63] Y. Wen, D. Tran, and J. Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020.
[64] C. Williams. Computing with infinite networks. M. Mozer, M. Jordan, and T. Petsche, editors, Advances in Neural Information Processing Systems, volume 9. MIT Press, 1996.
[65] C. K. Williams and C. E. Rasmussen. Gaussian Processes for Machine Learning. The MIT Press, 2006.
[66] B. Xu, N. Wang, T. Chen, and M. Li. Empirical evaluation of rectified activations in convolu- tional network. arXiv preprint a 2015.
[67] J. Yoon, G. Singh, and S. Ahn. Robustifying sequential neural processes. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages 10861-10870 PMLR, 13-18 Jul 2020.
14

---

A Lemma and Proof
For the comprehensiveness of proof, we duplicate Lemma 3.1 here.
Lemma A.I (Gaussian posterior distribution with factorised prior distribution). we have p(r:lp) (z.l,, p, Ed) and p(u) = II== (po,, Eo.a) for n i.i.d. observations of D dimensional vectors, then the mean and covariance of posterior distribution p(u]z) N (pl 2n,) are:
En
Σ [2
+ E5.t)
Un En
Σ
[2 tu Eo.
(15)
Proof.
p(((x)
((xx) P|E exp
1
(-1
(-2 (xt - =, (2; 11)
1
1
1
exp - -2(0 mo,i) T Eo (p moos)
(16)
(2x) Ev,il
X exp
(u xz1)² - (+= x;;) (ti)+ + (p- - mo 2 (+ - moos))
(17)
2
1=1
X exp
1²
[ + E 2n²²
Σ
(IT²z; xs + Eo.
(18)
2
exp
(u nn))
T (1c mm)
(19)
(2
P|
where we dropped constant terms for clarity. From Equation
18 and 19
that:
D
Eg!),
En pn
E 2r Eo.
(20)
Σn =
(Σ- + E Mn En
[2 bu + Eo.
(21)
i=1
If we use Lemma A.1 with diagonal covariance matrices for p(r"),2;)) = = (rT) diag(""))) and
p(2;)= M7 N mm diag(g"))), we can obtain the posterior distribution of / (zilMz, diag(o ))
as follows:
2
Σ +
1/ 0z
Σ (r (s nm 2 ((m)
(22)
where 2 is the element-wise inversion, and @ is the element-wise product.
B Experimental Details
In this section, we outline additional details of the experimental settings including the datasets (Appendix B. hyperparameters of the models used (Appendix [..2). metrics (Appendix B.3). and a brief analysis of computational complexity of MGP and MNPs (Appendix [.4]. For all the experiments, we used the Adam optimiser (30) with batch size of 200 and the Tensorflow framework. All the experiments were conducted on single NVIDIA GeForce RTX 3090 GPU.
15

---

B.1 Details of Datasets
Synthetic Dataset Figure 2 shows the predictive probability and the attention weight of different attention mechanisms. Here, we describe the dataset and the settings used for the demonstrations.
We generated 1,000 synthetic training samples (i.e., Ntrain using the Scikit-learn's 's moon dataset with zero-mean Gaussian noise (std = 0.15) added. The test samples were generated as a mesh-grid of 10,000 points (i.e., 100 x 100 grid with Niest 10, 000). The number of points in the context memory N was set to 100. In this demonstration, we simplified the problem by setting M difference in attention mechanisms.
000) for binary classification by
which equivalent to the unimodal setting and illustrated the
Robustness to Noisy Samples Dataset In Section 5.1 noisy samples with the six multimodal datasets. The details of each dataset are outlined in Table 6 These datasets lie within feature space where each feature extraction method can be found in [17
evaluated the models' robustness to
Table 6: Multimodal datasets used for evaluating robustness to noisy samples.
Dataset
Handwritten
CUB
PIE
Caltech101
Scene15
HMDB
# of modalities
2
2
Types of modalities
Images 2,000
Image&Tert 11,788
Images 680
Images 8.677
Images 4,485 15
Images 6,718
of samples # of classes
10
10
68
101
51
OOD Detection Dataset We used CIFARIO-C [8] which consists of corrupted images of CIFARIO 31. 15 types of corruptions and five levels of corruption for each type are available for the dataset. Following (24), we used the first three types as multimodal inputs with different levels of corruption (1, 3, and 5).
B.2
Details of Models
In our main experiments, four unimodal baselines with the early fusion (EF) method [3 (MC Dropout, Deep Ensemble (EF), SNGP, and ETP) and three multimodal baselines with the late fusion (LF) method 3 (Deep Ensemble (LF), TMC, and MGP) were used. In this section, we describe the details of the feature extractors and each baseline.
Feature Extractors We used the same feature extractor for all the methods to ensure fair compar- isons of the models. For the synthetic dataset, the 2D input points were projected a high-dimensional space (dm = 128) with feature extractor that has 6 residual fully connected (FC) layers with the ReLU activation. For the OOD detection experiment, the Inception was used as the feature extractor. Note that the robustness to noisy samples experiment does not require a separate feature extractor as the dataset is already in a feature space.
(54) pretrained with ImageNet
MC Dropout Monte Carlo (MC) Dropout 11 is a well-known uncertainty estimation method that leverages existing dropout layers of DNNs to approximate Bayesian inference. In our experiments, the dropout rate set to 0.2 with 100 dropout samples used to make predictions in the inference stage. The predictive uncertainty was quantified based on the original paper [
Deep Ensemble Deep ensemble 32 is a powerful uncertainty estimation method that trains multiple independent ensemble members. In the case the unimodal bascline, we employed five ensemble members, whereas for the multimodal bascline, a single classifier was trained independently for each modality input. In both scenarios, the unified predictions were obtained by averaging the predictions from the ensemble members, while the predictive uncertainty was determined by calculating the variance of those predictions.
learn http:///ww....r....o.//////////a datasets www.smaaaeeaaaaaeeeeeeeeeeee
htm
16

---

SNGP opectral-normalicn Neural Gaussian Process (SNGP) [35 is an effective and scalable uncertainty estimation method that utilises Gaussian process (GP). consists of feature extractor with spectral normalisation and GP output layer. Since we used the identical feature extractor for all the baselines, we only used the GP layer in this work. Following (24), the model's covariance matrix was updated without momentum with À = x //8 8 for the mean-field approximation. As the original authors proposed, we quantified the predictive uncertainty based on the Dempster-Shafes theory 7 defined as u(x) = (K + > exp (logit (I))) where logit (-) is the h class of output logit with the number of classes K.
ETP Evidential Turing Processes (ETP) 25 is recent variant of NPs for uncertainty estimation of image classification. Since none of the existing NPs can be directly applied to multimodal data, there are several requirements to utilise them for multimodal classification: 1) context set in the inference stage (e.g., context memory) and 2) method of processing multimodal data. ETP was selected due to its inclusion of the original context memory, requiring minimal modifications to be applicable
task. We used the memory size of 200 and quantified the predictive uncertainty with entropy as proposed by the original paper 25
TMC Trusted Multi-view Classification (TMC) 17 is a simple multimodal uncertainty estimation based on the Subjective logic [23). We used the original settings the paper with the annealing epochs of ten for the balancing term. TMC explicitly quantifies its predictive uncertainty based on Dempster-Shafer theory
MGP Multi-view Gaussian Process (MGP) 24 is the current SOTA multimodal uncertainty
estimation method that combines predictive posterior distributions multiple GPs. We used the identical settings of the original paper with the number of inducing points set to 200 and ten warm-up epochs. predictive uncertainty was quantified by the predictive variance as proposed by the original paper (24].
MNPs (Ours) The encoders and decoder in Multimodal Neural Processes (MNPs) consist of two FC layers with the Leaky ReLU activation [66) after the first FC layer. normalisation layer is stacked on top of the second FC layer for the encoders. For enc (-) and enc (-) that approximate the variance of distributions, we ensure positivity by transforming the outputs as h4 0.01 + 0.99 * Softplus (h) where h is the raw output from the encoders. 1" of the adaptive RBF attention was initialised as 10 + 1 E Rd' and DCM was initialised by randomly selecting training samples. We used five samples for the Monte Carlo method to approximate the integrals in Equations (1T)- which
found enough in practice. Refer to Table 7 for the hyperparameters of MNPs. We provide the impact of nM on the model performance in Appendix C.I
Table 7: Hyperparameters of MNPs.
Dataset
Parameter
Handwritten
CUB
PIE
Caltech101
Scene15
HMDB
CIFARIO-C
100
200
300
700
300
400
200
1
0.03
1
0.0001
1
8
1
1
1
0.25
0.01
0.1
0.01
0.5
0.01
0.01
B.3 Details of Metries
Apart from test accuracy, we report the expected calibration error (ECE) [16 and the area under the receiver operating characteristic curve (AUC). ECE defined as:
1 ECE = 71 Σ B.Ilacc((B.) - conf(B.)
where n is the number of testing samples, B, is bin with partitioned predictions with the number of bins 6, [Ba] is the number of elements in B,, (cc(B.) is the accuracy of predictions in B,, and
17

---

(B) is the average predictive confidence in B, Following 35 and [24] we set b 15. AUC was used for the OOD detection experiment with ground truth labels of class 0 being the ID samples
and class 1 being the OOD samples. Fach model's predictive uncertainty was used as confidence score to predict whether a test sample is OD sample.
B.4 Computational Complexity of MGP and MNPs
In addition to the empirical difference of wall-clock time per epoch in Table 5 we provide com- putational complexity of the two models in Table g We assume that the number of inducing points in MGP equals to the number of context points in MNPs. During training of MNPs, each modality requires cross-attention (P(N"NN)) and a contrastive learning (O((N))))) that sum to (MM(NN" Ny + )N) with M being the number of modalities, whereas during inference, each modality only requires the cross-attention which results in O(MN"
Table 8: Computational complexity of MGP and MNPs.
Training
Inference
MGP
(MM(N)) O(M(N NT (Nr)
((M(N O(MN"N, NT)
MNPs (Ours)
C
Ablation Studies
In this section, we analyse MNPs' performance with different settings and show the effectiveness of the proposed framework.
C.1 Context Memory Updating Mechanisms
We compare the updating mechanism of DCM based on MSE Equation with three other baselines: random sampling, sampling bypasses DCM and randomly selects training samples during inference. For FIFO, we follow the original procedure proposed by (62] that updates the context memory during training and only uses it during inference. ClE-based mechanism replaces j in Equation with j argmax K _ -Ty Li, log (TT k)) JE(
(FIFO) 62), and cross-entropy based (CE). Random
We provide experimental results for all the experiments outlined in Section 5 random sampling and FIFO achieve high accuracy both without noise and with noise as shown in Table and I However, MSE and CE outperform the others in terms of ECE in Table IO] and OOD AUC in Table 12 As MSE and CE select the new context points based on classification error, the selected context points tend to be close to decision boundary, which is the most difficult region to classify. We believe this may contribute to the lower calibration error, suppressing overconfident predictions. The MSE and CE mechanisms show comparable overall results, but selected MSE for its lower ECE. In terms of time efficiency, Table 13 shows that random sampling is slower than the other three methods.
We highlight that
and
For DCM updated by MSE, we also provide difference in performance for a range number of context points N in Figure 3 shaded area indicates 95% confidence interval. Unsurprisingly, the training time and the testing time increase with respect to n The general trend in test accuracy across the datasets shows the benefit of increasing the number of context points. However, the performance gain in ECE and OOD AUC ambivalent as different patterns are observed for different datasets. We leave an in-depth analysis of this behaviour for our future study.
For every figure, the bold line indicates the mean value, and the
18

---

Table 9: Test accuracy with different context memory updating mechanisms (f).
Updating Mechanism
Dataset
Handwritten
CUB
PIE
Caltech101
Scene
HMDB
Random FIFO
99.40=0.14
88.50=5.12
94.85-10.00
90.3811.38
76.03112.96
68.42:10.53
99.30=0.11
90.33:1.26
95.29-11885
91.0910.97
76.08:11.92
69.65:10666
CE
99.40=0.14
93.67-2.25
93.00=1.43
93.5910.27
77.40110.73
70.77
MSE
9.00
93.50=1.71
93.0010.62
93.46110.32
77.9010.71
71.97-1.43
Table 10: Test ECE with different context memory updating mechanisms (4).
Updating Mechanism
Datasel
Handwritten
CUB
PIE
Caltech101
Scene 15
HMDB
Random FIFO
0.007=0.001
0.069:10.029
0.050-10.000
0.043:100005
0.059=0.061
0.052-10.006
0.001
0.06710.021
0.057:10.016
0.027:10004
0.056=0.048
0.032-40.007
CE MSE
0.006=0.001 0.050=0.016
0.041=00009
0.017:1.003
0.038=0.010
0.034
0.005=0.001
0.04911.008
0.040:1.005
0.017110.00
0.038
0.028:1.0066
Table 11: Average test accuracy across 10 noise levels with different context memory updating mechanisms (t).
Updating Mechanism
Dataset
Handwritten
CUB
PIE
Caltech101
Scene 15
HMDB
Random
98.39:10.21
83.1124.08
92.55:10.55
89.3611.118
72.85+22.30
62. 10:0.44
FIFO
98.5110.11
85,86=2.877
93.8111..77
89.5911.02
72.59:1.82
63.00:10.80
CE
98.4911.113
93.75110.2 18.8
92.8710.21
73.9810.41
63.9710.71
MSE
98.58110.10
88.9
93.80.10.49
92.8310.11
74.1410.35
64.1110.115
Table 12: Test accuracy (f), ECE (4), and OOD detection AUC (f) with different context memory updating mechanisms.
Updating Mechanism
OOD AUC 1
Test accuracy +
ECE 1
SVHN
CIFARI00
Random FIFO
74.61+:0.22
0.073+0.005
0.860-10.003
0.777=0.002
74.82+00.11
0.073+00.006
0.862:t0.077
0.778=0.005
CE
74.70:10.19
0.013:10.022
0.8711100004
0.789
MSE
74.92+10.07
0.011=0.001
0.872
0.786=0.005
Table 13: Wall-clock inference time (ms/epoch) with different context memory updating mechanisms.
Updating Mechanism
Dataset
Handwritten
CUB
PIB
Caltech101
Scene15
HMDB
CIFAR10-C
Random
31.80:13.68
8.15::2.80
12,14::3.09
255.37=:1325
33.7343.55
79.1925.95
710.48:18.58
FIFO
24.91.10.68
5.87.13.06
7.20:12.77
101 101.02-12.90
25.04:13.35
41.50-12.4
496.23 10.85
25.00:10.28
5.61:1..5
6.85:11.04
101.10::2.59
25.47:1..77
43.45.:..85
500.7941.00
MSE
22.53:11.88
5.57:11558
6.70.10.92
101.01:2.38
26.60=10.37
41.87=2.10
493.1819.91
19

---

0.995
0.007
accuracy
Test ECE
0.994
0.006
0.993
0.005
100
200
250
150
250
Nm
Nm
Training time (ms/cpooh)
380
Testing time (1tss (tsseppoch))
30
360
340
25
320
300
20
50
100
200
250
100
200
250
N
Nm
Figure 3: Test accuracy, ECE, average training time, and average testing time with different N for the Handwritten dataset.
0.10
Test accuracy
Test BCG
0.08
0.06
0.04
50
100
150
200
250
50
100
150
200
250
Nm
Nm
Training timte (ms/epoch)
Testing time (mns) (ppeh)
40.0
10
37.5
35.0
32.5
50
100
150
200
250
100
150
200
250
Nm
Figure 4: Test accuracy, ECE, average training time, and average testing time with different N for the CUB dataset.
20

---

0.055
0.955
Test accuracy
Test LCB
0.050
0.950
0.045
0.945
0.040
0.940
0.035
200
400
500
300
Nm
N
Training time (msiepoch)
Testing time (11s) (its/epoeh)
7.5
64
7.0
60
6.5
100
200
300
400
100
200
300
400
500
Nm
Nm
Figure 5: Test accuracy, ECE, average training time, and average testing time with different N for the PIE dataset.
0.025
0.935
Test accuracy
Test ECG
0.930
0.020
0.925
0.015
200
400
600
800
200
400
600
800
Nm
Nm
Training titte (ms/cpoch)
Testing time (msl/pppc) (epoch)
2200
100
1800
1600
60
200
400
600
800
200
400
600
800
Nm
Nm
Figure 6: Test accuracy, ECE, average training time, and average testing time with different N for the Caltech101 dataset.
21

---

0.045
Test accuracy
0.78
Test LCB
0.040
0.77
0.035
0.76
0.030
100
200
300
400
500
300
500
N
N
370
Training (nsleppoc))
Testing time (11s/ (ms/epoch)
360
28
3.50
340
24
100
200
500
200
300
500
Nm
Nm
Figure 7: Test accuracy, ECE, average training time, and average testing time with different N for the Scene15 dataset.
0.100
Test accuracy
Test ECB
0.075
0.4
0.050
0.025
100
200
300
400
500
100
200
300
400
500
Nm
Nm
Training timte (ms/eppcc)
700
Testing time (msi/ (mss/poohh)
45
650
600
35
550
100
200
300
400
500
100
200
300
400
500
Nm
Figure 8: Test accuracy, ECE, average training time, and average testing time with different N for the HMDB dataset.
22

---

0.030
0.750
Test accuracy
0.025
0.748
Test LCD
0.020
0.746
0.015
0.744
0.010
100
400
200
Nm
N
OOD AUC AUC (CIFAR100)
0.79
OOD (SVHN)
0.87
0.78
0.86
0.77
0.85
100
200
300
400
500
100
200
300
400
500
Nm
N
Training time (m1s hassepoch)
14000
Testing time (ww (mnwepoch)
700
12000
100
200
300 N
400
500
100
200
300
400
500
Nm
Figure 9: Test accuracy, ECE, OOD AUC (SVHN), OOD AUC (CIFARI00), average training time, and average testing time with different N for the CIFARIO-C dataset.
C.2 Multimodal Aggregation Methods
We demonstrate the performance of MBA compared with two other methods namely "Concat" and
"Mean" "Concat" bypasses MBA and directly provides r multiple modalities to the decoder (see
Figure 1b by simple concatenation followed by passing to MIP which lets p( (T ) T
in Equation 12) be parameterised by decoder where {CM, T MLPP(Coneat fr Jm (r 1)). Concat(.) represents concatenating multiple vectors along their feature dimension. Similarly," "Mean" also bypasses MBA and simply averages the multiple modalities into single representation. Formally, Em=r"."
P(f(TA))C" T parameterised by decoder where {C'M T
every case, MBA outperforms both baselines. While similar performance can be observed for Handwritten, Scene15, and Caltech101, large differences are observed in CUB, PIE, and HMDB across different metrics. The test accuracy CIFAR10 almost consistent across all methods, but large gaps in ECE and OOD performance observed. This highlights the importance of MBA, especially in robustness and calibration performance.
The results are shown in Table 14 17
Table 14: Test accuracy with different multimodal aggregation methods (t).
Aggregation Methods
Dataset
Handwritten
CUB
PIE
Caltech101
Scene15
HMDB
Concat
99.35++0.2
89.00
89.71+2.49
92.63++0.18
77.18++064
56.06+t2.11
Mean
99.45+0.11
92.50-2.43
90.88-+2.24
93.1440.25
77.60+00.6
57.80+1.97
MBA
99.50+0.00
93.50=1.71
95.00=0..2
93.46+0.32
77.90+00.1
71.97=00.4
23

---

Table 15: Test ECE with different multimodal aggregation methods (4).
Aggregation Methods
Dataset
Handwritten
CUB
PIE
Caltech101
Scene
HMDB
Concat
0.007+0.001
0. 109=0.008
0.092=0.020
0.038+0.009
0.061=0.005
0.060#0.017
Mean
0.006+0.001
0.057=0.012
0.059=0.008
0.030#0.004
0.038=0.005
0.117=0.014
MBA
0.005+0.001
0.049=0.008
0.040=0.005
0.017#0.003
0.038=0.009
0.028=0.006
Table 16: Average test accuracy across noise levels with different multimodal aggregation methods
(1).
Aggregation Methods
Dataset
Handwritten
CUB
PIE
Caltech101
Scene15
HMDB
Concat
97.71=0.46
85.51#1.42
85.94+2.48
89.84+0.17
72.23=0.52
45. 45.22+2.86
Mean
98.42+0.09
88.27=1.83
88.7442.33
92.0740.16
49.5842.24 74.14#0.35
74.06+0.28
MBA
98.58+0.10
88.96:1.98
93.80=0.49
92.83+0.18
64.11=0.15
Table 17: Test accuracy (t). ECE (4), and OOD detection AUC (f) with different multimodal aggregation methods.
Aggregation Methods
OOD AUC
Test accuracy 1
ECE
SVHN
CIFARI00
Concat
74.24+0.27
0.12540.000
0.781
0.72840.004
Mean MBA
74.72+0.24
0.10940.000
0.803+0.007
0.742+0.003
74.92+0.07
0.001:0.001
0.872:0.002
0.786+0.005
C.3 Attention Types
We decompose the attention weight A(T CT) in Equation (9) follows:
A(TT. C = Norm Norm(Sim(77' C?))
(23)
where Norm(-) is the normalisation function such as Softmax and Sparsemax, and Sim(,, as the similarity function such the not-product and the RBF kernel. We provide experimental results of four different combinations of normalisation functions and similarity functions in Table 18
Among the four combinations, the RBF function with Sparsemax outperforms the others in most cases. More importantly, Table shows a large difference in robustness to noisy samples between the RBF function with Sparsemax and the dot-product with Sparsemax, even when a marginal difference in accuracy is shown Table 18 For instance, for the PIE dataset, the difference in accuracy without noisy samples is 0.3, but the difference increases to 6.0 in the presence of noisy samples. The same pattern observed with OOD AUC in Table 21] This illustrates the strength of RBF attention that more sensitive to distribution-shiff Sparsemax results in superior overall performance.
shown in Figure Lastly, for both similarity functions,
Table 18: Test accuracy with different attention mechanisms (f).
Similarity Function
Normalisation Function
Dataset
Handwritten
CUB
PIE
Caltech101
Scene15
HMDB
Softmax
98.80=0.45
87.0016.42
75.15:13.00
82.95=:0.47
69.83=:1.41
16.28:t1.18
RBF
Sparsemax
99.50-1000
93.50-11.71
95.00-10.62
93.46:10.22
77. 77.90-10.71
71.97-0.43
Softmax
99.00:10.18
79.6743.94
86.3244..88
88.90-10.36
74.95.10.33
64.68.10.78
Dot
Sparsemax
98.95-0.11
82.1742.67
94.26:11.90
92.46-10.26
78.30-11.06
63.23++1.89

---

Table 19: Test ECE with different attention mechanisms (4).
Similarity Function
Normalisation Function
Handwritten
CUB
PIE
Caltech101
Scene15
HMDB
Softmux
0.019=000055
0.084400.200
0. 0.100=000017
0.025=00004
0,202=000119 0.152++0.007
Sparsemax
0.005=0.001
0.04910.008
0.040=0.005
0.017=0.003
0.038:10.009
0.028=00006
Softmax
0.008=00.003
0.166:+0.015
0.033-00.007
0.061+:0.010
Dot
Sparsemax
0.010=00001
0.1311:0.028
0.053:0.010
0.025:0.002
0.032-100008
0.084=00015
Table 20: Average test accuracy with different attention mechanisms (t).
Similarity Function
Normalisation Function
Dutaset
Handwritten
CUB
PIE
Caltech101
Sccne
HMDB
Sofimax
94.56+00.66
82.58+5.98
65.88+22.98
81.23+0.29
67.77-1 1.05
38.63++-0.3
RBF
Sparsemax
98.58-10.10
88,9621.98
93.80=1.499
92.83:10.18
74.14-0.35
64.11-10.15
Softmax
77.99++0.32
73.89+1.77
70.80+1.71
63.80=00112
58.74-0.24
34.28+0045
Dot
Sparsemax
96.00++0.24
70.30+2.61
87.44+1444
81.95+1992
67.84-1.00
40.26-+0.56
Table 21: Test accuracy (t), ECE (4), and OOD detection AUC (1) with different attention mecha- nisms.
Similarity Function
Normalisation
OOD AUC
Function
Test accuracy
ECE 4
SVHN
CIFARTO0
Softmax
67 67.65++0116
0.080=000001
0.864+0.006
0.771=:0.006
Sparsemax
74.92-10.07
0.011-:.001
0.872:10.002
0.786=10.005
Softmax
68.81=:0.62
0.130::00019
0.849:00009
0.775=00.05
Dot
Sparsemax
75.07++0.09
0.055++00001
0.837+-0.004
0.765++00004
C.4 Adaptive Learning of RBF Attention
We have shown that the effectiveness of learning the RBF attention's parameters with the synthetic dataset in Figure
We further provide the ablation studies with the real-world datasets in Table
22 25
Table 22: Test accuracy with and without CRBF (t).
Dataset
Method
Handwritten
CUB
PIE
Caltech101
Scene15
HMDB
Without CRAF With CRDF
96.85=0.229 99.50-00.00
91.17=2.40 93.50 1.71
93.38+1.27
92.64 95.00
48.95=1.70 04.45+0045 77.90
0..32
71.97-10.43
Table 23: Test ECE with and without CABF (L).
Dataset
Method
Handwritten
CUB
Caltech101
Scene15
HMDB
Without LRBF
0.007=0.001
0.008 0.049+00008
0,043+-0.007
0.036+0.004
0.054++0.011
0.043++0.008
With LABF
0.005-0.00
0.040+0.000
0.017+0.003
0.038++0.00 0.010
0.028+00.06
Table 24: Average test accuracy across 10 noise levels with and without CABF (t)-
Dataset
Method
Handwritten
CUB
PIE
Caltech101
Scene15
HMMDB
Without CABF 89.44 98.58=0.10
86.69.1..6 88.96=1.998
91.50.L0.94
92.32.10.27 93.80-10.49
37.93 11.18.0038
With LUBE
92.83-10.18
74.14110.35
64.11=:0.15
25

---

Table 25: Test aceuracy (1). ECE (L), and OOD detection AUC (1) with and without CABF.
OOD A
Method
Test accuracy 1
ECE +
SVHN
CIFAR100
Without LaBF
74.961:.16 74.9210.07
0.019:0.002 0.01110.001
0.822::0.004
0.746:10.00
With CRBF
0.872:10.02
0.786110.005
D Broader Impacts
As long-term goal of this work is to make multimodal classification of DNNs more trustworthy by using NPs, it has many potential positive impacts to our society. Firstly, with transparent and calibrated predictions, more DNNs can be deployed to safety-critic diagnosis. Secondly, this work raises awareness to the machine learning society to evaluate and review reliability of DNN model. Lastly, diverse applications. Nevertheless, a potential negative impact may exist if the causes of uncertain predictions are not fully understood. To take step further reliable DNNs, the source of uncertainty should be transparent to non-experts.
domains such medical
study shows the potential capability of NPs in more
26


Reviews:
[{"review_id": "IELGsmTZ9I", "reviewer": "Reviewer_fi6w", "summary": "This paper extends one of the promising uncertainty estimation method: Neural Process from unimodal to multimodal. This is motivated by the fact that: current techniques are predominantly designed for unimodal data, and directly applying them to multimodal information is ineffective. However, this extension poses several challenges: 1) how fuse information from different modalities effectively and efficiently; 2) the context memory of the original design grows in proportion to M - the number of modality, which is memory-consuming, and sets the question of how to maintain it small yet informative; 3) how to find an appropriate length-scale of the Radial Basis Function (RBF) that ensures a tight boundary between in-distribution and out-of-distribution data.\nTo tackle these challenges, the paper introduces three solutions: 1) Dynamic Context Memory to choose the most informative samples in the context memory, 2) Multimodal Bayesian Aggregation, and 3) Adaptive RBF Attention. Empirical experiments show that the method can outperform other unimodal and multimodal baselines in terms of maintaining good accuracy and calibration at the presence of noise and in terms of out-of-distribution detection.", "questions": "see questions listed on the weakness part", "limitations": "The author have listed one limitation that the updating mechanism is not theoretically guaranteed to obtain the optimal context memory.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. originality: the first to extend Neural Process from unimodal to multimodal and effectively combine existing techniques to solve some of the challenges incurred.\n2. clarity: in general, the paper articulates its ideas clearly, even though some parts remain difficult to understand or unclear in terms of details (see the weakness part)\n3. significance: the proposed method shows promising results in terms of outperform most unimodal and multimodal baselines. ", "weaknesses": "**Clarity**: the clarity of the technical introduction (Sec. 3) and the experiment (Sec. 4) parts can be largely improved. \n1. The method introduction part could be **challenging to comprehend for readers unfamiliar with the Neural Process and its technical details**. It lacks adequate contextual knowledge, such as an explanation for why the context memory is required, why it is used to store training samples rather than using a context feature. I think the Sec. 2 should be improved to be more comprehensive and general in introducing the Neural Process, to avoid the readers to revisit several referenced papers repeatedly.\n1. The **math details can be overwhelming and at times confusin**g. For instance, when reviewing Sec 3.3, it necessitates frequent referral back to Section 3.2. It would be helpful if the semantic meaning of symbols is reintroduced when they appear, preventing the need for readers to revisit previous sections for notation understanding.\n\t1. **Derivation details**: The logic in some derivations is not easy to understand, e.g. Line 221 introduces the function $f(T_X^M)$ without any prior explanation. Also, there is no explanation on how equation (12) is derived, and why softmax is applied on a density function. These issues impede reading and understanding. The formula (4) and formula (6) are not consistent as their parameter notation is not the same on $\\omega$ and $\\psi$.\n\t2. The MNPs **pseudocode** lacks a high-level introduction and has heavy reference links, failing to provide clarifying information.\n\t3. **Captions**: Some table captions lack sufficient information. For instance, Table 4/5 does not clearly describe the dataset and settings used.\n\t4. **Experimental Setup**: The presentation of the experimental settings, such as how datasets e.g. Caltech101 were modified into multi-view datasets, is not clear.\n\n**Soundness**: Despite the multimodal setting, the experiments use only multi-view image data. The lack of experiments with multimodal combinations such as image + text impedes the proof of the model's effectiveness in various settings.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and comments. We would like to address the following statements.\n\n> \"It lacks adequate contextual knowledge, such as an explanation for why the context memory is required, why it is used to store training samples rather than using a context feature.\"\n\nWe will provide more explanations in the revision as suggested. Here we would like to clarify why context memory is required. NPs require a labelled context set during both training and inference in order to make predictions for a target set. During training, training samples, which are labelled, can be used as the context set. However, during the inference stage, the labelled context set is not given for our task. Thus, a context memory which stores labelled training samples is crucial to enable an efficient and effective inference. \n\nThe context memory $\\\\{ C_X,C_Y \\\\}$ stores training samples in input space so that when a target sample $T_X$ is given, the distance between $C_X$ and $T_X$ is measured to weight the context elements for generating target-specific context representations in a non-parametric way (i.e., weighted sum of encoded context samples for a specific target sample). It is worth noting that, as we stated in L65, we consider the input space to be a feature space.\n\n> “I think the Sec. 2 should be improved to be more comprehensive and general in introducing the Neural Process, to avoid the readers to revisit several referenced papers repeatedly…It would be helpful if the semantic meaning of symbols is reintroduced when they appear, preventing the need for readers to revisit previous sections for notation understanding.”\n\nWe appreciate your suggestion. We will include a general introduction of NPs at Section 2 in the revised manuscript.\n\n> “The logic in some derivations is not easy to understand, e.g. Line 221 introduces the function $f(T^M_X)$ without any prior explanation. Also, there is no explanation on how equation (12) is derived, and why softmax is applied on a density function.”\n\nThank you for pointing this out. Please let us provide a detailed explanation here. In Gaussian process classification such as [60, 21, 38, 18] where the likelihood  is categorical, class probability is obtained in two stages. First, the predictive distribution is acquired in the form of a Gaussian distribution, which is then sampled to squash through the softmax function. Similarly in NPs, the predictive distribution is acquired as a Gaussian distribution $p(T_Y\\\\vert T_X,C_X,C_Y)$ which is suitable for regression but not for classification. Thus, we introduced a latent function $f(\\\\cdot)$ to estimate the latent distribution $p(f(T^M_X)|C^M,T^M_X)$ which is then sampled to squash through the softmax function to estimate the class probability.\n\n> “The formula (4) and formula (6) are not consistent as their parameter notation is not the same on $\\\\omega$ and $\\\\psi$.”\n\nWe would like to clarify that the parameters $\\\\phi$, $\\\\psi$, $\\\\theta$, and $\\\\omega$ are meant to be different, indicating four distinct encoders. The reasons for using different encoders are that $ r_m^* $ for $\\\\phi$, $s_m^*$ for $\\\\psi$, $u^m$ for $\\\\theta$, and $q^m$ for $\\\\omega$ are used to approximate different distribution parameters: $ r_m^* $  as samples from $p({r^m_*}\\_i \\vert z_i)$, $s_m^*$ as diagonal covariance of $p({r^m_*}\\_i \\vert z_i)$, $u^m$ as mean of prior, and $q^m$ as diagonal covariance of prior. \n\n> “The MNPs pseudocode lacks a high-level introduction and has heavy reference links, failing to provide clarifying information.”\n\nWe agree with the reviewer. We will add short comments for each line to make it more interpretable.\n\n> “Captions: Some table captions lack sufficient information. For instance, Table 4/5 does not clearly describe the dataset and settings used.”\n\nWe kept captions as concise as possible due to the space constraints. Please refer to Section 5.1, 5.2, and Appendix B for experimental details. \n\n> “Experimental Setup: The presentation of the experimental settings, such as how datasets e.g. Caltech101 were modified into multi-view datasets, is not clear.” “Despite the multimodal setting, the experiments use only multi-view image data. The lack of experiments with multimodal combinations such as image + text impedes the proof of the model's effectiveness in various settings.”\n\nThank you for bringing this to our attention. In Appendix B.1, we provided an outline of the dataset details but missed information about the types of input modalities and the preprocessing. We will add these to our revised manuscript.\n\nIn summary, the datasets consist of a range of input modalities. Handwritten, PIE, HMDB, Scene15, Caltech101, CIFAR10, SVHN, and CIFAR100 are image datasets with diverse features associated with each image. This diversity comes from either different feature extraction methods or augmentation to generate multiple modalities from a single image. Additionally, CUB represents an image+text dataset.\n\nIt's important to clarify that we consider multimodality to involve multiple inputs regardless of whether these inputs share the same type (e.g., image+image) or differ in their types (e.g., image+text)."}, {"review_id": "agRl2jABkG", "reviewer": "Reviewer_2g2g", "summary": "This paper proposes a new method for multimodal uncertainty estimation by extending neural processes. The authors summarize three challenges to do that and give solutions correspondingly. Experimental results show that the proposed method is more robust and outperforms existing baselines.\n\n---\n\nThanks for the clarification! My main concerns are addressed.", "questions": "(Line 185): Maybe I missed something, but why using RBF can address the overconfident issue?  ", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed method (MNPs) can achieve good performance empirically.", "weaknesses": "1. The motivation is not clear. Indeed, there is little discussion about neural processes with multimodal data, and extending unimodal neural processes to multimodal scenarios could be challenging. However, existing works discussed the multi-view data [16,21]. The reason why extending the neural processes for multimodal data (instead of extending/improving [16,21]) remains unclear.\n\n\n2. Lack of understanding. I would like to suggest that the authors add some ablation studies. From current experiments, we can only see that proposed MNPs outperform baselines. However, why it can do that is not clear, e.g., why the unimodal method (DE) can achieve the best test accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and comments. We would like to address the following points.\n\n> “However, existing works discussed the multi-view data [16,21]. The reason why extending the neural processes for multimodal data (instead of extending/improving [16,21]) remains unclear.”\n\nAs we stated our motivation in L38-46, NPs bring the best of both worlds of GP, which has been shown to be well-calibrated and robust to domain-shift [21, 31, R1], and DNNs, which have representation power and efficiency. The current SOTA model [21] is a GP, which is robust but computationally expensive at the same time. We show in our experiments that we can achieve better or comparable performance with faster computational time (up to five folds). Also, in [21, R2] and our experiments, TMC [16] has been shown to have limited capability in calibration performance and OOD detection, which are essential uncertainty estimation downstream tasks [21, 31]. Therefore, existing methods are either less effective or less efficient, which motivate the development of our approach.\n\n> “I would like to suggest that the authors add some ablation studies. From current experiments, we can only see that proposed MNPs outperform baselines. However, why it can do that is not clear, e.g., why the unimodal method (DE) can achieve the best test accuracy.”\n\nPlease refer to Appendix C for ablation studies comparing different context memory updating mechanisms, attention types, and adaptive learning of RBF attention. We would like to highlight that DE has achieved the best test accuracy for only one out of the seven datasets. In the other six datasets, our method outperformed DE. For uncertainty estimation, metrics like ECE, robustness to noise, and OOD detection performance are more important as they directly or indirectly quantify uncertainty estimation performance. For those metrics, our method outperformed DE in all seven datasets, which demonstrates the superior uncertainty estimation performance of our method.\n\n> “Why using RBF can address the overconfident issue?”\n\nWe explained the reason in L186-191 of the paper. But please let us elaborate more here. The core component of RBF is the lengthscale controlling  the degree of smoothness in distance calculations. To illustrated, considering two points  $x$  and $x'$ where ${\\\\vert\\\\vert x-x' \\\\vert\\\\vert}^2=1$, the RBF value (indicative of closeness) can vary significantly based on the lengthscale: 0.98 if the lengthscale is 5 and 1.93e-22  if the lengthscale is 0.1. This underlines that an appropriately determined lengthscale can distinguish whether or not two samples come from different distributions. As highlighted in L203, this lengthscale parameter is often predetermined as a hyperparameter or an optimisable parameter that requires a complex initialisation. To overcome this limitation, we have proposed $\\\\mathcal{L}\\_{RBF}$ to optimise the lengthscale without any complex initialisation. The effectiveness of this approach is demonstrated in Figure 2, Appendix C.2, and Appendix C.3.\n\nAdditional references:\n\n[R1] S. G. Popescu, D. J. Sharp, J. H. Cole, K. Kamnitsas, and B. Glocker. Distributional gaussian process layers for outlier detection in image segmentation. In A. Feragen, S. Sommer, J. Schnabel, and M. Nielsen, editors, Information Processing in Medical Imaging, pages 415–427, Cham, 2021. Springer International Publishing. ISBN 978-3-030-78191-0.\n\n[R2] K. Zou, T. Lin, X. Yuan, H. Chen, X. Shen, M. Wang, and H. Fu. Reliable multimodality eye disease screening via mixture of student’s t distributions. arXiv preprint arXiv:2303.09790, 2023.\n\n[R3] Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and J. Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.\n\n[R4] J. Mukhoti, A. Kirsch, J. van Amersfoort, P. H. Torr, and Y. Gal. Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty. arXiv preprint arXiv:2102.11582, 2021."}, {"review_id": "CQiVKriIoq", "reviewer": "Reviewer_U6Gp", "summary": "This paper proposes a multimodal neural processes (neural network generaliation of Gaussian processes) model.\n\nThe overall approach has several novel elements:\n* A way to maintain a dynamic context set throughout training (e.g a support set for few-shot learning, these context sets are needed for neural processes) \n* A Bayesian aggregation scheme for combining multiple modalities\n* An adaptive RBF attention mechanism as an alternative to (vanilla) dot product attention which the authors argue is overconfident on OOD samples.\n\nExperiments on a suite of datasets show that the proposed method is faster, more accurate and better calibrated, and better at detecting OOD samples than prior work.\n", "questions": "See above.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This reads like a polished piece of work with clear writing, extensive ablations (in the supplementary) and novel technical contributions.  Experimental results seem fairly convincing (that the proposed method is indeed better than prior similar works).  My caveat is that my familiarity with this part of the literature on Neural Processes is passing at best.", "weaknesses": "\nI have no major complaints about this paper, but some things to point out are:\nIt is pretty unclear what are the actual multiple modalities in the datasets that are used for experiments (which makes these experiments not particularly compelling perhaps unless you are already familiar with this line of work).  I had to trace through a series of cited works to figure out that (likely) these different modalities are different features extracted e.g using different networks.\nSection 3.2 could be written more clearly — I couldn’t figure out why there were multiple encoders until encountering Lemma 3.1 — it’d be much better to explain this up front.\nA nit: it’s also not clear when reading section 3.1 how often the dynamic context memory is meant to be updated until I read the pseudocode much later (I’d recommend saying that it’s a per-minibatch update somewhere).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback. We would like to address the reviewer’s suggestions.\n\n> “It is pretty unclear what are the actual multiple modalities in the datasets that are used for experiments”\n\nWe acknowledge that we have missed the details of the input modality in Appendix B.1. We will add the types of input modalities and the preprocessing steps for each dataset in our revised manuscript. \n\n> “Section 3.2 could be written more clearly — I couldn’t figure out why there were multiple encoders until encountering Lemma 3.1 — it’d be much better to explain this up front. A nit: it’s also not clear when reading section 3.1 how often the dynamic context memory is meant to be updated until I read the pseudocode much later (I’d recommend saying that it’s a per-minibatch update somewhere).”\n\nThank you for the suggestion. We agree with the reviewer. We will elaborate why multiple encoders are necessary in the beginning of Section 3.2, and explain the per-minibatch update in the beginning of Section 3.1.\n"}, {"review_id": "PQ4B0uHyW2", "reviewer": "Reviewer_RNhe", "summary": "This work proposes a confidence calibration algorithm for multimodal classification problems. The algorithm includes three key components: 1) dynamic context memory, 2) multimodal Bayesian aggregation, and 3) adaptive radial basis function. The algorithm is evaluated on multiple benchmarks using classification accuracy, calibration, and OOD performance. On multiple benchmarks, the proposed algorithm is able to obtain SOTA results.", "questions": "1, **Line 71**, from my understanding, the context set C and target set T contains both training and test datasets. But $N_C +N_T=N_{\\text{train}}$. Why is the summation of $N_C$ and $N_T$ only equal to the number of training points?\n\n2, **Line 148 and 150**, what are $r^m$  and $s^m$? How are they connected/different from $r^m_{\\*}$ and $s^m_{\\*}$?\n\n3, **Line 160**, how are encoders parameterized by $\\theta$ and $\\omega$ connected/different from those by $\\phi$ and $\\psi$?", "limitations": "The authors explicitly discussed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1, The targeted problem and challenges are explicitly stated. For each challenge, a technically reasonable solution is proposed.\n2, The proposed algorithm is able to obtain strong empirical results, obtaining SOTA results on multiple benchmarks and different metrics.\n3, Comprehensive ablation studies have been conducted to demonstrate each components.\n4, As illustrated in Figure 2, the proposed adaptive RBF algorithm outperformed other attention mechanisms.", "weaknesses": "1, Some notation definitions are confusing. Please refer to section \"Questions\" for details.\n2, Because of the above notation section 3.2 is a bit hard to follow.\n3, Ablation study is conducted for individual components. It would be interesting to design ablation studies and see how each component affects the end-to-end performance.\n4, The component directly related to multimodality learning, i.e., multimodal Bayesian aggregation, is based on previously work [1]. It would be good if the authors can explain how the proposed MBA component is different from previous work, either in the revision or rebuttal. \n\n[1] M. Volpp, F. Flürenbrock, L. Grossberger, C. Daniel, and G. Neumann. Bayesian context aggregation for neural processes. In International Conference on Learning Representations, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and questions. We would like to address the following questions.\n\n> \"Line 71, from my understanding, the context set C and target set T contains both training and test datasets. But $N_C +N_T=N_{\\\\text{train}}$. Why is the summation of $N_C$ and $N_T$ only equal to the number of training points?\"\n\nWe would like to clarify that a context set and a target set are required for training and test datasets separately, This means that during training, the training dataset can be split to a context set and a target set (i.e., $N_C +N_T=N_{\\\\text{train}}$). Similarly, during testing, a context set and a target set are required. However, since we have no label information of the test dataset, the context set has to be given from somewhere else. A simple solution can be using the entire training set as the labelled context set, but it is highly inefficient especially for large datasets. Therefore, we have proposed dynamic context memory to store a smaller number of informative training samples during training that can be used in the inference stage.\n\n> “Line 148 and 150, what are $r^m$ and $s^m$? How are they connected/different from $r^m_*$ and $s^m_*$?”\n\n$r^m$ and $s^m$ are the encoded context representations which have no information about target samples. The attention mechanism measures the distance between the target samples and produces the weighted sum of $r^m$ and $s^m$ for specific target samples. These target-specific context representations are $r^m_*$ and $s^m_*$.\n\n> “Line 160, how are encoders parameterized by $\\\\theta$ and $\\\\omega$ connected/different from those by $\\\\phi$ and $\\\\psi$?”\n\nThe encoders parameterised by $\\\\theta$ and $\\\\omega$ and those by $\\\\phi$ and $\\\\psi$ are used to approximate different distribution parameters. To be precise, $u^m$ encoded with $\\\\theta$ represents the mean of prior, $q^m$ encoded by $\\\\omega$ represents the diagonal covariance of prior, $r^m_*$ encoded by $\\\\phi$ represents the samples from $p({r^m_*}\\_i \\vert z_i)$, and  $s^m_*$ encoded by $\\\\psi$ represents the diagonal covariance of $p({r^m_*}\\_i \\vert z_i)$. Each of the encoders is implemented as an independent neural network.\n\n> “Ablation study is conducted for individual components. It would be interesting to design ablation studies and see how each component affects the end-to-end performance.”\n\nDue to the page limitation, the ablation studies were put in the appendix. We would like to highlight that we conducted all ablation studies which are outlined in Appendix C using the same experimental procedures including the datasets as our main experiments in order to show the effectiveness of each component. \n\n> “The component directly related to multimodality learning, i.e., multimodal Bayesian aggregation, is based on previously work [1]. It would be good if the authors can explain how the proposed MBA component is different from previous work, either in the revision or rebuttal.”\n\nOur proposed MBA has technical connections with the previous work [1]. Both methods are Bayesian approaches that aggregate multiple representations into a single one (latent variable $z$) by modelling the prior and likelihood of $z$ to obtain the posterior. However, there are significant differences between the previous work [1] and ours, regarding their intended applications and model specifications. \n\nThe previous study [1] aimed to aggregate representations of different instances (or observations) into a single representation of latent variable $z$ for the context samples, using an uninformative prior on $z$. Since its primary purpose is to summarise context samples, it has no interactions with target samples.\n\nIn contrast, our approach aggregates multiple modalities of the same instance (or an observation) into a single representation of latent variable $z$. We incorporate an informative prior $\\\\mathcal{N}(u^m,\\\\text{diag}(q^m)$ (Equation (6), L159, and Figure 1), derived from the context memory. Furthermore, by considering a target-specific representation ${r^m_*}\\_i $ (the output of the proposed adaptive RBF attention) as a sample of $p({r^m_*}\\_i \\vert z_i)$, we model $p({r^m_*}\\_i \\vert z_i)$ as a distribution after observing a target sample, and $p(z_i)$ as a distribution prior to observing a target sample."}, {"review_id": "VWrfDrDE79", "reviewer": "Reviewer_hoCV", "summary": "The paper tackles the problem of uncertainty estimation specifically for multi-modal data, i.e., inputs consisting of different sources. Specifically, it improves the popular Neural Process (NP) in three aspects: dynamic context update, multi-modal Bayesian aggregation, and a novel attention mechanism based on the RBF kernel. The paper demonstrates superior test accuracy, uncertainty estimation (calibration and OOD detection), and robustness on multiple benchmarks compared to prior works. ", "questions": "* Can the authors comment on the importance of the Bayesian aggregation component? \n* It would be good to consolidate a single table (for one dataset) to directly show the effectiveness of each component in terms of accuracy, uncertainty, and robustness.", "limitations": "The work does not have an apparent negative societal impact. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper studies the less well-studied area of uncertainty estimation for multi-modal data. This has practical implications as many real-world applications can be multi-modal.  \n* The paper presents multiple metrics including accuracy, calibration, OOD detection, and robustness, which provide a holistic evaluation of the proposed method. \n* The idea of dynamic context update is most interesting to me. Specifically, the method replaces uninformative context samples with informative ones where \"uninformative-ness\" is measured by attention weights and \"informative-ness\" is measured by difficulty of classification.  ", "weaknesses": "* While the paper presents three innovations and claims that they are all tailored to multi-modal data, only the Bayesian aggregation is inherently related to multi-modal inputs. The dynamic context update and the RBF context mechanism do not utilize the multi-modal characteristic. This makes the contributions and claims less coherent. \n\n* It seems the Bayesian aggregation mechanism is not ablated. It's not clear how much improvement it brings to the overall performance as it is the main mechanism responsible for tackling multi-modal inputs. Does the improved robustness come from the aggregation algorithm or the other two components? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We would like to address the following statements. \n\n> “While the paper presents three innovations and claims that they are all tailored to multi-modal data, only the Bayesian aggregation is inherently related to multi-modal inputs. The dynamic context update and the RBF context mechanism do not utilize the multi-modal characteristic. This makes the contributions and claims less coherent.”\n\nPlease note that our method is a joint neural process framework for uncertainty estimation of multimodal data. Although the framework consists of several components, all of them work coherently in an integrated way. Our proposal of dynamic context update and RBF enables neural processes to more accurately capture uncertainties, while the Bayesian aggregation makes neural processes able to deal with multimodal data. Our ablation study shows that the proposed framework is unable to work as effectively or efficiently as it does without any of the modules. From the perspective of multimodal uncertainty estimation via neural processes, we believe the innovations are coherent.\n\n> “It seems the Bayesian aggregation mechanism is not ablated. It's not clear how much improvement it brings to the overall performance as it is the main mechanism responsible for tackling multi-modal inputs. Does the improved robustness come from the aggregation algorithm or the other two components?” “Can the authors comment on the importance of the Bayesian aggregation component?”\n\nWe thank the reviewer for the constructive suggestion. We conducted additional ablation studies on MBA with two baselines namely “Concat” and “Mean”. “Concat” bypasses MBA and directly provides $r^m_*$ of multiple modalities to the decoder (see Figure 1) by simple concatenation followed by passing to a MLP which lets $p(f(T^M_X)|C^M,T^M_X)$ in Equation (12) be parameterised by a decoder where $\\\\{C^M,T^M_X\\\\}=MLP(Concat(\\\\{r^m_*\\\\}^M\\_{m=1}))$. $Concat(\\\\cdot)$ represents concatenating multiple vectors along their feature dimension. Similarly, “Mean” also bypasses MBA and simply averages the multiple modalities into single representation. Formally, $p(f(T^M_X)|C^M,T^M_X)$ parameterised by a decoder where $\\\\{C^M,T^M_X\\\\}=\\\\frac{1}{M} \\\\sum^M\\_{m=1}r^m_*$. We conducted the same main experiments, and the results are shown below:\n\nTest accuracy ($\\\\uparrow$). \n\\\\begin{array}{ccccccc} \\\\hline {} & \\\\text{Handwritten} & \\\\text{CUB} & \\\\text{PIE} & \\\\text{Caltech101} & \\\\text{Scene15} & \\\\text{HMDB} \\\\\\ \\\\hline \\\\text{Concat} & 99.35\\\\pm0.22 & 89.00\\\\pm1.24 & 89.71\\\\pm2.49 & 92.63\\\\pm0.18 & 77.18\\\\pm0.64 & 56.06\\\\pm2.13 \\\\\\  \\\\text{Mean} & 99.45\\\\pm0.11 & 92.50\\\\pm2.43 & 90.88\\\\pm2.24 & 93.14\\\\pm0.25 & 77.60\\\\pm0.56 & 57.80\\\\pm1.97 \\\\\\ \\\\text{MBA} &  \\\\mathbf{99.50\\\\pm0.00} & \\\\mathbf{93.50\\\\pm1.71} & \\\\mathbf{95.00\\\\pm0.62} & \\\\mathbf{93.46\\\\pm0.32} & \\\\mathbf{77.90\\\\pm0.71} & \\\\mathbf{71.97\\\\pm0.43} \\\\\\ \\\\hline \\\\end{array}\n\nTest ECE ($\\\\downarrow$). \n\\\\begin{array}{ccccccc} \\\\hline {} & \\\\text{Handwritten} & \\\\text{CUB} & \\\\text{PIE} & \\\\text{Caltech101} & \\\\text{Scene15} & \\\\text{HMDB} \\\\\\ \\\\hline \\\\text{Concat} & 0.007\\\\pm0.001 & 0.109±0.008 & 0.092±0.020 & 0.038±0.005 & 0.061±0.005 & 0.060±0.017 \\\\\\ \\\\text{Mean} & 0.006±0.001 & 0.057±0.012 & 0.059±0.008 & 0.030±0.004 & 0.038±0.005 & 0.117±0.014 \\\\\\ \\\\text{MBA} &  \\\\mathbf{0.005±0.001} & \\\\mathbf{0.049±0.008} & \\\\mathbf{0.040±0.005} & \\\\mathbf{0.017±0.003} & \\\\mathbf{0.038±0.009} & \\\\mathbf{0.028±0.006\n} \\\\\\ \\\\hline \\\\end{array}\n\nAverage test accuracy across 10 noise levels ($\\\\uparrow$). \n\\\\begin{array}{ccccccc} \\\\hline {} & \\\\text{Handwritten} & \\\\text{CUB} & \\\\text{PIE} & \\\\text{Caltech101} & \\\\text{Scene15} & \\\\text{HMDB} \\\\\\ \\\\hline \\\\text{Concat} & 97.71±0.46 & 85.51±1.42 & 85.94±2.48 & 89.84±0.17 & 72.23±0.52 & 45.22±2.86 \\\\\\ \\\\text{Mean} & 98.42±0.09 & 88.27±1.83 & 88.74±2.33 & 92.07±0.16 & 74.06±0.28 & 49.58±2.24 \\\\\\ \\\\text{MBA} &  \\\\mathbf{98.58±0.10} & \\\\mathbf{88.96±1.98} & \\\\mathbf{93.80±0.49} & \\\\mathbf{92.83±0.18} & \\\\mathbf{74.14±0.35} & \\\\mathbf{64.11±0.15} \\\\\\ \\\\hline \\\\end{array} \n\nTest accuracy ($\\\\uparrow$), ECE ($\\\\downarrow$), and OOD detection AUC ($\\\\uparrow$). \n\\\\begin{array}{ccccc} \\\\hline & \\\\text{Test accuracy} \\\\uparrow & \\\\text{ECE} \\\\downarrow & \\\\text{OOD AUC (SVHN)} \\\\uparrow & \\\\text{OOD AUC (CIFAR100)} \\\\uparrow \\\\\\ \\\\hline \\\\text{Concat} & 74.24±0.27 & 0.125±0.005 & 0.781±0.016 & 0.728±0.004 \\\\\\ \\\\text{Mean} & 74.72±0.24 & 0.109±0.003 & 0.803±0.007 & 0.742±0.003 \\\\\\ \\\\text{MBA} & \\\\mathbf{74.92±0.07} & \\\\mathbf{0.011±0.001} & \\\\mathbf{0.872±0.002} & \\\\mathbf{0.786±0.005} \\\\\\ \\\\hline\\\\end{array}\n\nIn every case, MBA outperformed both baselines. While similar performance was observed for Handwritten, Scene15, and Caltech101, large differences were observed in CUB, PIE, and HMDB across different metrics. The test accuracy of CIFAR10 is almost consistent across all methods, but large gaps in ECE and OOD performance were observed. This highlights the importance of MBA, especially in robustness and calibration performance."}]

Author rebuttal:
**Continued Rebuttal for Reviewer hoCV**

> “It would be good to consolidate a single table (for one dataset) to directly show the effectiveness of each component in terms of accuracy, uncertainty, and robustness.”

Thank you for the suggestion. We would like to provide the first and the last datasets which are Handwritten and CIFAR10 for the comparison.

Handwritten.
\\begin{array}{c|ccc|ccc|ccc} \\hline & {} & \\text{Context memory} & {} & {} & \\text{Attention} &  {} & {} & \\text{Multimodal aggregation}  & {} \\\ {} & \\text{Random} & \\text{FIFO} & \\text{MSE (Ours)} & \\text{Dot Softmax} & \\text{RBF Softmax} & \\text{RBF Sparsemax (Ours)} & \\text{Concat} & \\text{Mean} & \\text{MBA (Ours)} \\\ \\hline \text{Test accuracy} \\uparrow & 99.40±0.14 & 99.30±0.11 & \\mathbf{99.50±0.00} & 99.00±0.18 & 98.80±0.45 & \\mathbf{99.50±0.00} & 99.35±0.22 & 99.45±0.11 & \\mathbf{99.50±0.00} \\\ \text{ECE} \\downarrow& 0.007±0.001 & 0.007±0.001 & \\textbf{0.005±0.001} & 0.008±0.003 & 0.019±0.005 & \\textbf{0.005±0.001} & 0.007±0.001 & 0.006±0.001 & \\textbf{0.005±0.001} \\\ \\text{Accuracy with noisy inputs} \\uparrow & 98.39±0.21 & 98.51±0.11 & \\mathbf{98.58±0.10} & 77.99±0.32 & 94.56±0.66 & \\mathbf{98.58±0.10} & 97.71±0.46 & 98.42±0.09 & \\mathbf{98.58±0.10} \\\ \\hline\\end{array}

CIFAR10.
\\begin{array}{c|ccc|ccc|ccc} \\hline & {} & \\text{Context memory} & {} & {} & \\text{Attention} &  {} & {} & \\text{Multimodal aggregation}  & {} \\\ {} & \\text{Random} & \\text{FIFO} & \\text{MSE (Ours)} & \\text{Dot Softmax} & \\text{RBF Softmax} & \\text{RBF Sparsemax (Ours)} & \\text{Concat} & \\text{Mean} & \\text{MBA (Ours)} \\\ \\hline \text{Test accuracy} \\uparrow & 74.61±0.22 & 74.82±0.11 & \\mathbf{74.92±0.07} & 68.81±0.62 & 67.65±0.16 & \\mathbf{74.92±0.07} & 74.24±0.27 & 74.72±0.24 & \\mathbf{74.92±0.07} \\\ \text{ECE} \\downarrow& 0.073±0.005 & 0.073±0.006 & \\mathbf{0.011±0.001} & 0.130±0.019 & 0.080±0.001 & \\mathbf{0.011±0.001} & 0.125±0.005 & 0.109±0.003 & \\mathbf{0.011±0.001} \\\ \\text{OOD SVHN} \\uparrow & 0.860±0.003 & 0.862±0.007 & \\mathbf{0.872±0.002} & 0.849±0.009 & 0.864±0.006 & \\mathbf{0.872±0.002} & 0.781±0.016 & 0.803±0.007 & \\mathbf{0.872±0.002} \\\ \\text{OOD CIFAR100} \\uparrow & 0.777±0.002 & 0.778±0.005 & \\mathbf{0.786±0.005} & 0.775±0.005 & 0.771±0.006 & \\mathbf{0.786±0.005} & 0.728±0.004 & 0.742±0.003 & \\mathbf{0.786±0.005} \\\ \\hline\\end{array}

By examining the two tables, it becomes clear that every component has an impact on all metrics. Nevertheless, noticeable differences, particularly in test accuracy, ECE, and robustness to noise, can be observed across different attention types. Notably, when it comes to OOD performance, the multimodal aggregation methods show more substantial differences.

Meta-review: This paper looks at uncertainty estimation and out-of-distribution detection (OOD) in the context of multi-modal models. Specifically, the works looks at it from the perspective of neural processes (NPs) that maintain a dynamic context memory, and proposes a multimodal Bayesian aggregation method as well as adaptive radial basis functions. Results are demonstrated with respect to calibration performance, robustness under shift, and OOD detection for a set of multi-modal datasets including those with heterogeneous features or images+text as different modalities.

  The reviewers appreciated the novelty and significance of the proposed method, and mentioned a number of weaknesses including: 1) Lack of motivation, context, and detail for those not familiar with NPs (fi6w, 2g2g), 2) Overall lack of details, precision, and consistent notation for the derivation portions (fi6w, RNhe), 3) Description of what modalities are covered by the datasets (fi6w, U6Gp) and motivation of the method w.r.t to setting (hoCV), and 4) Lack of understanding and more thorough analysis about why the method performs well (2g2g, hoCV). 

  In responses, the authors provided a rebuttal, including new experiments (Oxford-102 with images and captions, as well as new ablations) or pointers to those in the appendix. In the end, several reviewers mentioned that they were satisfied with the response, at least partially, and kept their overall positive score or increased it. After considering all of the materials (paper, reviews, and rebuttal) I believe the paper does make a contribution to an interesting underexplored area of uncertainty estimation for multi-modal datasets. The method is well-motivated and the results improve upon the state of the art. As a result, I recommend acceptance.
Decision: Accept (poster)