SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Myong Chol Jung
david.jung@monash.edu
He Zhao
CSIRO's Data61
he.zhao@ieee.org
Joanna Dipnall
jo.dipnall@monash.edu
Lan Du*
lan.du@monash.edu
Uncertainty estimation is an important research area to make deep neural networks (DNNs) more trustworthy. While extensive research on uncertainty estimation has been conducted with unimodal data, uncertainty estimation for multimodal data remains a challenge. Neural processes (NPs) have been demonstrated to be an effective uncertainty estimation method for unimodal data by providing the reliability of Gaussian processes with efficient and powerful DNNs. While NPs hold significant potential for multimodal uncertainty estimation, the adaptation of NPs for multimodal data has not been carefully studied. To bridge this gap, we propose Multimodal Neural Processes (MNPs) by generalising NPs for multimodal uncertainty estimation. Based on the framework of NPs, MNPs consist of several novel and principled mechanisms tailored to the characteristics of multimodal data. In extensive empirical evaluation, our method achieves state-of-the-art multimodal uncertainty estimation performance, showing its appealing robustness against noisy samples and reliability in out-of-distribution detection with faster computation time compared to the current state-of-the-art multimodal uncertainty estimation method.
Uncertainty estimation of deep neural networks (DNNs) is an essential research area in the development of reliable and well-calibrated models for safety-critical domains [49, 45]. Despite the remarkable success achieved by DNNs, a common issue with these model is their tendency to make overconfident predictions for both in-distribution (ID) and out-of-distribution (OOD) samples [16, 43, 35]. Extensive research has been conducted to mitigate this problem, but most of these efforts are limited to unimodal data, neglecting the consideration of multimodal data [57, 11, 50, 63, 38, 34].
In many practical scenarios, safety-critical domains typically involve the processing of multimodal data. For example, medical diagnosis classification that uses X-ray, radiology text reports and the patient's medical record history data can be considered as a form of multimodal learning that requires trustworthy predictions [8]. Despite the prevalence of such multimodal data, the problem of uncertainty estimation for multimodal data has not been comprehensively studied. Furthermore, it has been demonstrated that using existing unimodal uncertainty estimation techniques directly for multimodal data is ineffective, emphasising the need for a meticulous investigation of multimodal uncertainty estimation techniques [24, 17].
Several studies investigated the problem of multimodal uncertainty estimation with common tasks including 1) assessing calibration performance, 2) evaluating robustness to noisy inputs, and 3) detecting OOD samples. Trusted Multi-view Classification (TMC) [17] is a single-forward deterministic classifier that uses the Dempster's combination rule [7] to combine predictions obtained from different modalities. The current state-of-the-art (SOTA) in this field is Multi-view Gaussian Process (MGP) [24], which is a non-parametric Gaussian process (GP) classifier that utilises the product-of-experts to combine predictive distributions derived from multiple modalities. Although MGP has shown reliability of GPs in this context, it should be noted that the computational cost of a GP increases cubically with the number of samples [19, 65].
Neural processes (NPs) offer an alternative approach that utilises the representation power of DNNs to imitate the non-parametric behaviour of GPs while maintaining a lower computational cost [13, 14]. It has been shown that NPs can provide promising uncertainty estimation for both regression [28, 33, 29] and classification tasks [62, 25] involving unimodal data. Despite the promising potential of NPs for multimodal data, to the best of our knowledge, no research has yet investigated the feasibility of using NPs for multimodal uncertainty estimation.
In this work, we propose a new multimodal uncertainty estimation framework called Multimodal Neural Processes (MNPs) by generalising NPs for multimodal uncertainty estimation. MNPs have three key components: the dynamic context memory (DCM) that efficiently stores and updates informative training samples, the multimodal Bayesian aggregation (MBA) method which enables a principled combination of multimodal latent representations, and the adaptive radial basis function (RBF) attention mechanism that facilitates well-calibrated predictions. Our contributions are:
1. We introduce a novel multimodal uncertainty estimation method by generalising NPs that comprise the DCM, the MBA, and the adaptive RBF attention.
2. We conduct rigorous experiments on seven real-world datasets and achieve the new SOTA performance in classification accuracy, calibration, robustness to noise, and OOD detection.
3. We show that MNPs achieve faster computation time (up to 5 folds) compared to the current SOTA multimodal uncertainty estimation method.
Multimodal Classification The aim of this research is to investigate uncertainty estimation for multimodal classification. Specifically, we consider a multimodal training dataset  $D_{train} = \{\{x_i^m\}_{m=1}^M, y_i\}_{i=1}^{N_{train}}$ , where  $M$  is the number of input modalities and  $N_{train}$  is the number of training samples. We assume that each  $i^{th}$  sample has  $M$  modalities of input  $x_i^m \in \mathbb{R}^{d^m}$  with the input dimension of  $d^m$  and a single one-hot encoded label  $y_i$  with the number of classes  $K$ . In this study, we consider the input space to be a feature space. The objective is to estimate the labels of test samples  $\{y_i\}_{i=1}^{N_{test}}$  given  $D_{test} = \{\{x_i^m\}_{m=1}^M\}_{i=1}^{N_{test}}$ , where  $N_{test}$  is the number of test samples.
Neural Processes NPs are stochastic processes using DNNs to capture the ground truth stochastic processes that generate the given data [13]. NPs learn the distribution of functions and provide uncertainty of target samples, preserving the property of GPs. At the same time, NPs exploit function approximation of DNNs in a more efficient manner than GPs. For NPs, both training and test datasets have a context set  $C = \{C_X, C_Y\} = \{x_i^C, y_i^C\}_{i=1}^{N_C}$  and a target set  $T = (T_X, T_Y) = \{x_i^T, y_i^T\}_{i=1}^{N_T}$  with  $N_C$  being the number of context samples and  $N_T$  the number of target samples², and the learning objective of NPs is to maximise the likelihood of  $p(T_Y|C, T_X)$ .
Conditional Neural Processes (CNPs), the original work of NPs [13], maximise  $p(T_Y|r(C),T_X)$  with  $r(C) = \frac{1}{N_C}\sum_{i = 1}^{N_C}\mathrm{enc}_{\rho}(cat[x_i^C;y_i^C])\in \mathbb{R}^{d_e}$  where  $\mathrm{enc}_{\rho}$  is an encoder parameterised by  $\rho$ ,  $cat[\cdot ;\cdot ]$  is the concatenation of two vectors along the feature dimension, and  $d_{e}$  is the feature dimension. The mean vector  $r(C)$  is the permutation invariant representation that summarises the context set which is passed to a decoder with  $T_{X}$  to estimate  $T_{Y}$ .
The original CNPs use the unweighted mean operation to obtain  $r(C)$  by treating all the context points equally, which has been shown to be underfitting [28]. To improve over this, Attentive Neural
2The modality notation is intentionally omitted as the original NPs are designed for unimodal inputs [13, 14].
Figure 1: Model diagram of MNPs: DCM refers to Dynamic Context Memory, MBA refers to Multimodal Bayesian Aggregation, and RBF refers to radial basis function.
Processes (ANPs) [28] leveraged the scaled dot-product cross-attention [60] to create target-specific context representations that allocate higher weight on the closer context points:
$$
r _ {*} (C, T _ {X}) = \underbrace {\operatorname {S o f t m a x} \left(T _ {X} C _ {X} ^ {T} / \sqrt {d}\right)} _ {A \left(T _ {X}, C _ {X}\right) \in \mathbb {R} ^ {N _ {T} \times N _ {C}}} \underbrace {\operatorname {e n c} _ {\rho} \left(\operatorname {c a t} \left[ C _ {X} ; C _ {Y} \right]\right)} _ {\mathbb {R} ^ {N _ {C} \times d _ {e}}} \tag {1}
$$
where  $A(T_X, C_X)$  is the attention weight with  $T_X$  as the query,  $C_X$  is the key, the encoded context set is the value, and  $d$  is the input dimension of  $T_X$  and  $C_X$ . The resulted target-specific context representation has been shown to enhance the expressiveness of the task representation [28, 53, 29, 9, 47, 67].
The training procedure of NPs often randomly splits a training dataset to the context and the target sets (i.e.,  $N_{C} + N_{T} = N_{train}$ ). In the inference stage, a context set is provided for the test/target samples (e.g., a support set in few-shot learning or unmasked patches in image completion). While the context set for the test dataset is given for these tasks, NPs also have been applied to other tasks where the context set is not available (e.g., semi-supervised learning [62] and uncertainty estimation for image classification [25]). In those tasks, existing studies have used a context memory during inference, which is composed of training samples that are updated while training [25, 62]. It is assumed that the context memory effectively represents the training dataset with a smaller number of samples. We build upon these previous studies by leveraging the concept of context memory in a more effective manner, which is suitable for our task.
Note that the existing NPs such as ETP [25] can be applied to multimodal classification by concatenating multimodal features into a single unimodal feature. However, in Section 5.1, we show that this approach results in limited robustness to noisy samples.
3 Multimodal Neural Processes
The aim of this study is to generalise NPs to enable multimodal uncertainty estimation. In order to achieve this goal, there are significant challenges that need to be addressed first. Firstly, it is essential to have an efficient and effective context memory for a classification task as described in Section 2. Secondly, a systematic approach of aggregating multimodal information is required to provide unified predictions. Finally, the model should provide well-calibrated predictions without producing overconfident estimates as described in Section 1.
Our MNPs, shown in Figure 1, address these challenges with three key components respectively: 1) the dynamic context memory (DCM) (Section 3.1), 2) the multimodal Bayesian aggregation (MBA) (Section 3.2), and 3) the adaptive radial basis function (RBF) attention (Section 3.3). From Section 3.1 to 3.3, we elaborate detailed motivations and proposed solutions of each challenge, and
Section 3.4 outlines the procedures for making predictions. Throughout this section, we refer the multimodal target set  $T^{M} = (\{T_{X}^{m}\}_{m=1}^{M}, T_{Y}) = \{\{x_{i}^{m}\}_{m=1}^{M}, y_{i}\}_{i=1}^{N_{T}}$  to the samples from training and test datasets and the multimodal context memory  $C^{M} = \{C_{X}^{m}, C_{Y}^{m}\}_{m=1}^{M}$  to the context set.
3.1 Dynamic Context Memory
Motivation In NPs, a context set must be provided for a target set, which however is not always possible for non-meta-learning tasks during inference. One simple approach to adapt NPs to these tasks is to randomly select context points from the training dataset, but its performance is suboptimal as randomly sampling a few samples may not adequately represent the entire training distribution. Wang et al. [62] proposed an alternative solution by introducing a first-in-first-out (FIFO) memory that stores the context points with the predefined memory size. Although the FIFO performs slightly better than the random sampling in practice, the performance is still limited because updating the context memory is independent of the model's predictions. Refer to Appendix C.1 for comparisons.
Proposed Solution To overcome this limitation of the existing context memory, we propose a simple and effective updating mechanism for the context memory in the training process, which we call Dynamic Context Memory (DCM). We partition context memory  $\{C_X^m,C_Y^m\}$  of  $m^{th}$  modality into  $K$  subsets ( $K$  as the number of classes) as  $\{C_X^m,C_Y^m\} = \{C_{X,k}^m,C_{Y,k}^m\}_{k = 1}^K$  to introduce a class-balance context memory, with each subset devoted to one class. Accordingly,  $N^{m} = N_{K}^{m}\times K$  where  $N^{m}$  is the number of context elements per modality (i.e., the size of  $\{C_X^m,C_Y^m\}$ ) and  $N_K^m$  is the number of class-specific context samples per modality. This setting resembles the classification setting in [13], where class-specific context representations are obtained for each class.
DCM is initialised by taking  $N_K^m$  random training samples from each class of each modality and is updated every mini-batch during training by replacing the least "informative" element in the memory with a possibly more "informative" sample. We regard the element in DCM that receives the smallest attention weight (i.e.,  $A(T_X,C_X)$  in Equation (1)) during training as the least informative one and the target point that is most difficult to classify (i.e., high classification error) as a more informative sample that should be added to DCM to help the model learn the decision boundary. Formally, this can be written as:
$$
C _ {X, k} ^ {m} [ i ^ {*},: ] = T _ {X} ^ {m} [ j ^ {*},: ], \quad k \in \{1, \dots K \}, \quad m \in \{1, \dots M \} \tag {2}
$$
$$
i ^ {*} = \underset {i \in \{1, \dots , N _ {K} ^ {m} \}} {\operatorname {a r g m i n}} \frac {1}{N _ {T}} \sum_ {t = 1} ^ {N _ {T}} \underbrace {A \left(T _ {X} ^ {m} , C _ {X , k} ^ {m}\right)} _ {\mathbb {R} ^ {N _ {T} \times N _ {K} ^ {m}}} [ t, i ], j ^ {*} = \underset {j \in \{1, \dots , N _ {T} \}} {\operatorname {a r g m a x}} \frac {1}{K} \sum_ {k = 1} ^ {K} \left(T _ {Y} [ j, k ] - \widehat {T} _ {Y} ^ {m} [ j, k ]\right) ^ {2} \tag {3}
$$
where  $[i, :]$  indicates the  $i^{th}$  row vector of a matrix,  $[i, j]$  indicates the  $i^{th}$  row and the  $j^{th}$  column element of a matrix, and  $\widehat{T}_Y^m$  is the predicted probability of the  $m^{th}$  modality input.  $i^*$  selects the context memory element that receives the least average attention weight in a mini-batch during training, and  $j^*$  selects the target sample with the highest classification error. To measure the error between the predictive probability and the ground truth of a target sample, one can use mean squared error (MSE) (Equation (3)) or cross-entropy loss (CE). We empirically found that the former gives better performance in our experiments. Refer to Appendix C.1 for ablation studies comparing the two and the impact of  $N^m$  on its performance.
The proposed updating mechanism is very efficient as the predictive probability and the attention weights in Equation (3) are available with no additional computational cost. Also, in comparison to random sampling which requires iterative sampling during inference, the proposed approach is faster in terms of inference wall-clock time since the updated DCM can be used without any additional computation (refer to Appendix C.1 for the comparison).
3.2 Multimodal Bayesian Aggregation
Motivation With DCM, we obtain the encoded context representations for the  $m^{th}$  modality input as follows:
$$
r ^ {m} = \operatorname {e n c} _ {\phi} ^ {m} \left(c a t \left[ C _ {X} ^ {m}; C _ {Y} ^ {m} \right]\right) \in \mathbb {R} ^ {N ^ {m} \times d _ {e}}, \quad s ^ {m} = \operatorname {e n c} _ {\psi} ^ {m} \left(c a t \left[ C _ {X} ^ {m}; C _ {Y} ^ {m} \right]\right) \in \mathbb {R} ^ {N ^ {m} \times d _ {e}} \tag {4}
$$
where  $\phi$  and  $\psi$  are the encoders' parameters for the  $m^{th}$  modality. Next, given the target set  $T_X^m$ , we compute the target-specific context representations with the attention mechanism:
$$
r _ {*} ^ {m} = A \left(T _ {X} ^ {m}, C _ {X} ^ {m}\right) r ^ {m} \in \mathbb {R} ^ {N _ {T} \times d _ {e}}, \quad s _ {*} ^ {m} = A \left(T _ {X} ^ {m}, C _ {X} ^ {m}\right) s ^ {m} \in \mathbb {R} ^ {N _ {T} \times d _ {e}} \tag {5}
$$
where the attention weight  $A(T_X^m, C_X^m) \in \mathbb{R}^{N_T \times N^m}$  can be computed by any scoring function without loss of generality. Note that a single  $i^{th}$  target sample consists of multiple representations from  $M$  modalities  $\{r_{*,i}^m = r_*^m [i,:] \in \mathbb{R}^{d_e}\}_{m=1}^M$ . It is important to aggregate these multiple modalities into one latent variable/representation  $z_i$  for making a unified prediction of the label.
Proposed Solution Instead of using a deterministic aggregation scheme, we propose Multimodal Bayesian Aggregation (MBA), inspired by [61]. Specifically, we view  $r_{*,i}^{m}$  as a sample from a Gaussian distribution with mean  $z_{i}$ :  $p(r_{*,i}^{m}|z_{i}) = \mathcal{N}\left(r_{*,i}^{m}|z_{i},\mathrm{diag}(s_{*,i}^{m})\right)$  where  $s_{*,i}^{m} = s_{*}^{m}[i,:] \in \mathbb{R}^{d_{e}}$  is the  $i^{th}$  sample in  $s_{*}^{m}$ . Additionally, we impose an informative prior on  $z_{i}$ :  $p(z_{i}) = \prod_{m=1}^{M}\mathcal{N}(u^{m},\mathrm{diag}(q^{m}))$  with the mean context representations of  $u^{m} \in \mathbb{R}^{d_{e}}$  and  $q^{m} \in \mathbb{R}^{d_{e}}$ , which assign uniform weight across the context set as follows:
$$
u ^ {m} = \frac {1}{N ^ {m}} \sum_ {i = 1} ^ {N ^ {m}} \operatorname {e n c} _ {\theta} ^ {m} \left(\operatorname {c a t} \left[ C _ {X} ^ {m} [ i, : ]; C _ {Y} ^ {m} [ i, : ] \right]\right), q ^ {m} = \frac {1}{N ^ {m}} \sum_ {i = 1} ^ {N ^ {m}} \operatorname {e n c} _ {\omega} ^ {m} \left(\operatorname {c a t} \left[ C _ {X} ^ {m} [ i, : ]; C _ {Y} ^ {m} [ i, : ] \right]\right) \tag {6}
$$
where  $\theta$  and  $\omega$  are the encoders' parameters for the  $m^{th}$  modality. Note that the encoders in Equation (4) and (6) are different because they approximate different distribution parameters.
Lemma 3.1 (Gaussian posterior distribution with factorised prior distribution). If we have  $p(x_i|\mu) = \mathcal{N}(x_i|\mu, \Sigma_i)$  and  $p(\mu) = \prod_{i=1}^{n} \mathcal{N}(\mu_{0,i}, \Sigma_{0,i})$  for  $n$  i.i.d. observations of  $D$  dimensional vectors, then the mean and covariance of posterior distribution  $p(\mu|x) = \mathcal{N}(\mu|\mu_n, \Sigma_n)$  are:
$$
\Sigma_ {n} = \left[ \sum_ {i = 1} ^ {n} \left(\Sigma_ {i} ^ {- 1} + \Sigma_ {0, i} ^ {- 1}\right) \right] ^ {- 1}, \quad \mu_ {n} = \Sigma_ {n} \left[ \sum_ {i = 1} ^ {n} \left(\Sigma_ {i} ^ {- 1} x _ {i} + \Sigma_ {0, i} ^ {- 1} \mu_ {0, i}\right) \right] \tag {7}
$$
As both  $p(z_{i})$  and  $p(r_{*,i}^{m}|z_{i})$  are Gaussian distributions, the posterior of  $z_{i}$  is also a Gaussian distribution:  $p(z_{i}|r_{*,i}) = \mathcal{N}\left(z_{i}|\mu_{z_{i}},\mathrm{diag}(\sigma_{z_{i}}^{2})\right)$ , whose mean and variance are obtained by using Lemma 3.1 as follows:
$$
\sigma_ {z _ {i}} ^ {2} = \left[ \sum_ {m = 1} ^ {M} \left(\left(s _ {*}, i\right) ^ {\ominus} + \left(q ^ {m}\right) ^ {\ominus}\right) \right] ^ {\ominus}, \mu_ {z _ {i}} = \sigma_ {z _ {i}} ^ {2} \otimes \left[ \sum_ {m = 1} ^ {M} \left(r _ {*}, i ^ {m} \otimes \left(s _ {*}, i ^ {m}\right) ^ {\ominus} + u ^ {m} \otimes \left(q ^ {m}\right) ^ {\ominus}\right) \right] \tag {8}
$$
where  $\varnothing$  and  $\otimes$  are element-wise inverse and element-wise product respectively (see Appendix A for proof).
We highlight that if the variance  $s_{*,i}^{m}$  of a modality formed by the target-specific context representation is high in  $\sigma_{z_i}^2$ ,  $q^{m}$  formed by the mean context representations dominates the summation of the two terms. Also, if both  $s_{*,i}^{m}$  and  $q^{m}$  are high for a modality, the modality's contribution to the summation over modalities is low. By doing so, we minimise performance degradation caused by uncertain modalities (see Section 5.1 for its robustness to noisy samples). Refer to Appendix C.2 for ablation studies on different multimodal aggregation methods.
3.3 Adaptive RBF Attention
Motivation The attention weight  $A(T_X^m, C_X^m)$  in Equation (3) and (5) can be obtained by any attention mechanism. The dot-product attention is one of the simple and efficient attention mechanisms, which has been used in various NPs such as [28, 53, 9, 47]. However, we have observed that it is not suitable for our multimodal uncertainty estimation problem, as the dot-product attention assigns excessive attention to context points even when the target distribution is far from the context distribution. The right figure in Figure 2a shows the attention weight in Equation (1) with  $A(x_{OOD}^m, C_X^m)$  where  $x_{OOD}^m$  is a single OOD target sample (grey sample) far from the context set (red and blue samples). This excessive attention weight results in overconfident predictive probability for OOD samples (see the left figure of Figure 2a), which makes the predictive uncertainty of a classifier hard to distinguish the ID and OOD samples.
(a) Dot-product
(b) RBF without  $\mathcal{L}_{RBF}$
(c) RBF with  $\mathcal{L}_{RBF}$
Figure 2: Predictive probability of upper circle samples (blue) is shown in the left column with (a) the dot-product attention, (b) the RBF attention without  $\mathcal{L}_{RBF}$ , (c) and the RBF attention with  $\mathcal{L}_{RBF}$ . Upper circle samples (blue) are class 1, lower circle samples (red) are class 2, and grey samples are OOD samples. The attention weight  $A(x_{OOD}^m, C_X^m)$  of an OOD sample across 100 context points is shown in the right column. The summarised steps of training MNPs are shown in (d). Refer to Appendix B for the experimental settings.
Input: Multimodal data  $T^{M}$  as the target set, the number of input modalities  $M$ , and the number of classes  $K$ .
Initialise: Multimodal context memory  $C^M$  by randomly sampling  $T^M$ .
for Mini-batch do
for Modality  $m = 1$  to  $M$  do
Context representations  $r^m$ ,  $s^m \gets$  Equation (4)
Attention weight  $A(T_X^m, C_X^m) \gets$  Equation (9)
Target-specific context representations  $r_{*}^{m}, s_{*}^{m} \gets$  Equation (5)
Mean context representations  $u^{m}$ $q^{m}\gets$  Equation (6)
Unimodal predictive probability  $\hat{T}_Y^m\gets$  Equation (13)
Unimodel prediction loss  $\mathcal{L}_{T_v}^m\gets$  Equation (14)
Unimodal supervised contrastive loss  $\mathcal{L}_{CL}^{m}\gets$  Equation (10)
Gaussian posterior distribution with aggregated multimodal representations  $p(Z|R^{*}(C^{M},T_{X}^{M}))\gets$  Equation (8)
Unified predictive probability  $\widehat{T}_Y\gets$  Equation (12) Optimise with  $\mathcal{L}$
for Label class  $k = 1$  to  $K$  do
Update context memory  $C_{X,k}^{m} \gets$  Equation (2)-(3)
(d) Pseudocode of MNPs
Proposed Solution To address the overconfident issue of the dot-product attention, we propose an attention mechanism based on RBF. RBF is a stationary kernel function that depends on the relative distance between two points (i.e.,  $x - x'$ ) rather than the absolute locations [65], which we define as  $\kappa^m(x, x') = \exp\left(-\frac{1}{2} ||\frac{x - x'}{(l^m)^2}||^2\right)$  where  $||\cdot||^2$  is the squared Euclidean norm, and  $l^m$  is the lengthscale parameter that controls the smoothness of the distance in the  $m^{th}$  modality input space. The RBF kernel is one of the widely used kernels in GPs [64, 65], and its adaptation with DNNs has shown well-calibrated and promising OOD detection [40, 41, 58, 24]. Formally, we define the attention weight using the RBF kernel as:
$$
A \left(T _ {X} ^ {m}, C _ {X} ^ {m}\right) = \operatorname {S p a r s e m a x} \left(G \left(T _ {X} ^ {m}, C _ {X} ^ {m}\right)\right) \tag {9}
$$
where the elements of  $G(T_X^m, C_X^m) \in \mathbb{R}^{N_T \times N^m}$  are  $[G]_{ij} = \kappa^m (T_X^m [i,:], C_X^m [j,:])$ , and Sparsemax [39] is an alternative activation function to Softmax. It is defined as  $\text{Sparsemax}(h) := \underset{p \in \Delta^{K-1}}{\operatorname{argmax}} || p -$