SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$h||^2$  where  $\Delta^{K - 1}$  is the  $K - 1$  dimensional simplex  $\Delta^{K - 1} \coloneqq \{p \in \mathbb{R}^K | \mathbb{1}^\mathrm{T}p = 1, p \geq 0\}$ . Here we use Sparsemax instead of the standard Softmax because Sparsemax allows zero-probability outputs. This property is desirable because the Softmax's output is always positive even when  $\kappa^m(x,x') = 0$  (i.e.,  $||x - x'||^2 \to \infty$ ) leading to higher classification and calibration errors (see Appendix C.3 for ablation studies).
The lengthscale  $l^m$  is an important parameter that determines whether two points are far away from each other (i.e.,  $\kappa^m(x, x') \to 0$ ) or close to each other (i.e.,  $0 < \kappa^m(x, x') \leq 1$ ). However, in practice,  $l^m$  has been either considered as a non-optimisable hyperparameter or an optimisable parameter that requires a complex initialisation [24, 59, 58, 42, 65, 56]. To address this issue, we propose an adaptive learning approach of  $l^m$  to form a tight bound to the context distribution by leveraging the supervised contrastive learning [27]. Specifically, we let anchor index  $i \in T_{ind} \equiv \{1, \dots, N_T\}$  with
negative indices  $N(i) = T_{ind} \backslash \{i\}$  and positive indices  $P(i) = \{p \in N(i) : T_Y[p, :] = T_Y[i, :]\}$ . Given an anchor sample of the target set, the negative samples refer to all samples except for the anchor sample, while the positive samples refer to other samples that share the same label as the anchor sample. We define the multimodal supervised contrastive loss as:
$$
\mathcal {L} _ {C L} ^ {M} = \frac {1}{M} \sum_ {m = 1} ^ {M} \mathcal {L} _ {C L} ^ {m} = \frac {1}{M} \sum_ {m = 1} ^ {M} \sum_ {i = 1} ^ {N _ {T}} - \frac {1}{| P (i) |} \times \sum_ {p \in P (i)} \log \frac {\exp \left(\kappa^ {m} \left(T _ {X} ^ {m} [ i , : ] , T _ {X} ^ {m} [ p , : ]\right) / \tau\right)}{\sum_ {n \in N (i)} \exp \left(\kappa^ {m} \left(T _ {X} ^ {m} [ i , : ] , T _ {X} ^ {m} [ n , : ]\right) / \tau\right)} \tag {10}
$$
where  $|P(i)|$  is the cardinality of  $P(i)$ , and  $\tau$  is the temperature scale. This loss encourages higher RBF output of two target samples from the same class and lower RBF output of two target samples from different classes by adjusting the lengthscale. In addition to  $\mathcal{L}_{CL}^{M}$ , a  $l_{2}$ -loss is added to form the tighter bound by penalising large lengthscale. Overall, the loss term for our adaptive RBF attention is  $\mathcal{L}_{RBF} = \mathcal{L}_{CL}^{M} + \alpha * \frac{1}{M} \sum_{m=1}^{M} ||l^{m}||$  with the balancing coefficient  $\alpha$ .
We show the difference between the RBF attention without  $\mathcal{L}_{RBF}$  (see Figure 2b) and the adaptive RBF attention with  $\mathcal{L}_{RBF}$  (see Figure 2c). It can be seen that the decision boundary modelled by the predictive probability of the adaptive RBF attention is aligned with the data distribution significantly better than the non-adaptive one. For ablation studies with real-world datasets, see Appendix C.4.
3.4 Conditional Predictions
We follow the standard procedures of Gaussian process classification to obtain predictions [65, 24, 42, 19] where we first compute the predictive latent distribution  $p(f(T_X^M)|C^M,T_X^M)$  as a Gaussian distribution by marginalising  $Z = \{z_{i}\}_{i = 1}^{N_{T}}$ :
$$
p \left(f \left(T _ {X} ^ {M}\right) \mid C ^ {M}, T _ {X} ^ {M}\right) = \int p \left(f \left(T _ {X} ^ {M}\right) \mid Z\right) p \left(Z \mid R ^ {*} \left(C ^ {M}, T _ {X} ^ {M}\right)\right) d Z \tag {11}
$$
where  $T_X^M = \{T_X^m\}_{m=1}^M$ ,  $R^*(C^M, T_X^M) = \{r_{*,i}\}_{i=1}^{N_T}$ , and  $p(f(T_X^M)|Z)$  is parameterised by a decoder. Then, we obtain the predictive probability  $\widehat{T}_Y$  by:
$$
\widehat {T} _ {Y} = \int \operatorname {S o f t m a x} \left(p \left(f \left(T _ {X} ^ {M}\right)\right)\right) p \left(f \left(T _ {X} ^ {M}\right) \mid C ^ {M}, T _ {X} ^ {M}\right) d f \left(T _ {X} ^ {M}\right) \tag {12}
$$
Similarly, we can obtain the unimodal predictive latent distribution  $p(f(T_X^m) | \{C_X^m, C_Y^m\}, T_X^m)$  and the unimodal predictive probability  $\widehat{T}_Y^m$  for the  $m^{th}$  modality as:
$$
\widehat {T} _ {Y} ^ {m} = \int \operatorname {S o f t m a x} (p \left(f \left(T _ {X} ^ {m}\right)\right)) p \left(f \left(T _ {X} ^ {m}\right) \mid \left\{C _ {X} ^ {m}, C _ {Y} ^ {m} \right\}, T _ {X} ^ {m}\right) d f \left(T _ {X} ^ {m}\right) \tag {13}
$$
We minimise the negative log likelihood of the aggregated prediction and the unimodal predictions by:
$$
\mathcal {L} _ {T _ {Y}} = - \mathbb {E} _ {T _ {Y}} \left[ \log \widehat {T} _ {Y} \right] - \frac {1}{M} \sum_ {m = 1} ^ {M} \mathcal {L} _ {T _ {Y}} ^ {m} = - \mathbb {E} _ {T _ {Y}} \left[ \log \widehat {T} _ {Y} \right] - \frac {1}{M} \sum_ {m = 1} ^ {M} \mathbb {E} _ {T _ {Y}} \left[ \log \widehat {T} _ {Y} ^ {m} \right] \tag {14}
$$
Since Equations (11)-(13) are analytically intractable, we approximate the integrals by the Monte Carlo method [44]. The overall loss for MNPs is  $\mathcal{L} = \mathcal{L}_{T_Y} + \beta * \mathcal{L}_{RBF}$  where  $\beta$  is the balancing term. Refer to Figure 2d for the summarised steps of training MNPs.
Neural Processes CNP and the latent variant of CNP that incorporates a latent variable capturing global uncertainty were the first NPs introduced in the literature [13, 14]. To address the under-fitting issue caused by the mean context representation in these NPs, Kim et al. [28] proposed to leverage the attention mechanism for target-specific context representations. This approach has been shown to be effective in subsequent works [53, 29, 9, 47, 67, 22]. Many other variants, such as SNP [51, 67], CNAP [48], and MPNPs [5] were proposed for common downstream tasks like 1D regression, 2D image completion [28, 14, 15, 10, 61, 26, 20], and image classification [13, 62, 25]. However, none of these studies have investigated the generalisation of NPs to multimodal data. This study is the first to consider NPs for multimodal classification and its uncertainty estimation.
Multimodal Learning The history of multimodal learning that aims to leverage multiple sources of input can be traced back to the early work of Canonical Correlation Analysis (CCA) [21]. CCA learns the correlation between two variables which was further improved by using feed-forward networks by Deep CCA (DCCA) [2]. With advances in various architectures of DNNs, many studies on multimodal fusion and alignment were proposed [4, 12, 55, 6]. In particular, transformer-based models for vision-language tasks [36, 52, 1] have obtained great attention. Nonetheless, most of these methods were not originally intended for uncertainty estimation, and it has been demonstrated that many of them exhibit inadequate calibration [16, 43].
Multimodal Uncertainty Estimation Multimodal uncertainty estimation is an emerging research area. Its objective is to design robust and calibrated multimodal models. Ma et al. [37] proposed the Mixture of Normal-Inverse Gamma (MoNIG) algorithm that quantifies predictive uncertainty for multimodal regression. However, this work is limited to regression, whereas our work applies to multimodal classification. Han et al. [17] developed TMC based on the Dempster's combination rule to combine multi-view logits. In spite of its simplicity, empirical experiments showed its limited calibration and capability in OOD detection [24]. Jung et al. [24] proposed MGP that combines predictive posterior distributions of multiple GPs by the product of experts. While MGP achieved the current SOTA performance, its non-parametric framework makes it computationally expensive. Our proposed method overcomes this limitation by generalising efficient NPs to imitate GPs.
Apart from measuring the test accuracy, we assessed our method's performance in uncertainty estimation by evaluating its calibration error, robustness to noisy samples, and capability to detect OOD samples. These are crucial aspects that a classifier not equipped to estimate uncertainty may struggle with. We evaluated MNPs on seven real-world datasets to compare the performance of MNPs against four unimodal baselines and three multimodal baselines.
To compare our method against unimodal baselines, we leveraged the early fusion (EF) method [3] that concatenates multimodal input features to single one. The unimodal baselines are (1) MC Dropout (MCD) [11] with dropout rate of 0.2, (2) Deep Ensemble (DE) [32] with five ensemble models, (3) SNGP's GP layer [35], and (4) ETP [25] with memory size of 200 and a linear projector. ETP was chosen as a NP baseline because of its original context memory which requires minimal change of the model to be used for multimodal classification.
The multimodal baselines consist of (1) Deep Ensemble (DE) with the late fusion (LF) [3] where a classifier is trained for each modality input, and the final prediction is obtained by averaging predictions from all the modalities, (2) TMC [17], and (3) MGP [24]. For TMC and MGP, we followed the settings proposed by the original authors. In each experiment, the same feature extractors were used for all baselines. We report mean and standard deviation of results from five random seeds. The bold values are the best results for each dataset, and the underlined values are the second-best ones. See Appendix B for more detailed settings.
5.1 Robustness to Noisy Samples
Experimental Settings In this experiment, we evaluated the classification performance of MNPs and the robustness to noisy samples with six multimodal datasets [24, 17]. Following [24] and [17], we normalised the datasets and used a train-test split of 0.8:0.2. To test the robustness to noise, we added zero-mean Gaussian noise with different magnitudes of standard deviation during inference (10 evenly spaced values on a log-scale from  $10^{-2}$  to  $10^{1}$ ) to half of the modalities. For each noise level, all possible combinations of selecting half of the modalities (i.e.,  $\binom{M}{M/2}$ ) were evaluated and averaged. We report accuracy and expected calibration error (ECE) for each experiment. Please refer to Appendix B for details of the datasets and metrics.
Results We provide the test results without noisy samples in Table 1, 2, and 5 and the results with noisy samples in Table 3. In terms of accuracy, MNPs outperform all the baselines in 5 out of 6 datasets. At the same time, MNPs provide the most calibrated predictions in 4 out of 6 datasets, preserving the non-parametric GPs' reliability that a parametric model struggles to achieve. This shows that MNPs bring together the best of non-parametric models and parametric models. Also,
Table 1: Test accuracy  $(\uparrow)$ .
Method	Dataset
Handwritten	CUB	PIE	Caltech101	Scene15	HMDB
MCD	99.25±0.00	92.33±1.09	91.32±0.62	92.95±0.29	71.75±0.25	71.68±0.36
DE (EF)	99.20±0.11	93.16±0.70	91.76±0.33	92.99±0.09	72.70±0.39	71.67±0.23
SNGP	98.85±0.22	89.50±0.75	87.06±1.23	91.24±0.46	64.68±4.03	67.65±1.03
ETP	98.75±0.25	92.33±1.99	91.76±0.62	92.08±0.33	72.58±1.35	67.43±0.95
DE (LF)	99.25±0.00	92.33±0.70	87.21±0.66	92.97±0.13	67.05±0.38	69.98±0.36
TMC	98.10±0.14	91.17±0.46	91.18±1.72	91.63±0.28	67.68±0.27	65.17±0.87
MGP	98.60±0.14	92.33±0.70	92.06±0.96	93.00±0.33	70.00±0.53	72.30±0.19
MNPs (Ours)	99.50±0.00	93.50±1.71	95.00±0.62	93.46±0.32	77.90±0.71	71.97±0.43
Table 2: Test ECE (↓).
Method	Dataset
Handwritten	CUB	PIE	Caltech101	Scene15	HMDB
MCD	0.009±0.000	0.069±0.017	0.299±0.005	0.017±0.003	0.181±0.003	0.388±0.004
DE (EF)	0.007±0.000	0.054±0.010	0.269±0.004	0.036±0.001	0.089±0.003	0.095±0.003
SNGP	0.023±0.004	0.200±0.010	0.852±0.012	0.442±0.004	0.111±0.063	0.227±0.010
ETP	0.020±0.002	0.051±0.009	0.287±0.007	0.096±0.002	0.045±0.008	0.100±0.010
DE (LF)	0.292±0.001	0.270±0.009	0.567±0.006	0.023±0.002	0.319±0.005	0.270±0.003
TMC	0.013±0.002	0.141±0.002	0.072±0.011	0.068±0.002	0.180±0.004	0.594±0.008
MGP	0.006±0.004	0.038±0.007	0.079±0.007	0.009±0.003	0.062±0.006	0.036±0.003
MNPs (Ours)	0.005±0.001	0.049±0.008	0.040±0.005	0.017±0.003	0.038±0.009	0.028±0.006
Table 3: Average test accuracy across 10 noise levels (↑).
Method	Dataset
Handwritten	CUB	PIE	Caltech101	Scene15	HMDB
MCD	82.15±0.17	76.08±0.61	64.65±0.77	73.45±0.11	48.97±0.33	42.63±0.08
DE (EF)	82.16±0.18	76.94±0.82	65.53±0.20	73.99±0.19	49.45±0.35	41.92±0.06
SNGP	72.46±0.41	61.27±1.24	56.52±0.69	56.57±0.17	38.19±1.86	37.49±0.42
ETP	75.85±0.31	73.99±1.12	62.64±0.35	66.62±0.08	46.17±0.58	38.28±0.23
DE (LF)	95.63±0.08	76.16±0.28	67.69±0.35	81.85±0.14	50.13±0.27	43.01±0.19
TMC	82.44±0.15	74.19±0.69	62.18±0.80	71.77±0.22	42.52±0.29	36.61±0.30
MGP	97.66±0.12	85.48±0.25	90.97±0.19	92.68±0.23	65.74±0.56	67.02±0.21
MNPs (Ours)	98.58±0.10	88.96±1.98	93.80±0.49	92.83±0.18	74.14±0.35	64.11±0.15
Table 4: Test accuracy (↑), ECE (↓), and OOD detection AUC (↑).
Method	Test accuracy ↑	ECE ↓	OOD AUC ↑
SVHN	CIFAR100
MCD	74.76±0.27	0.013±0.002	0.788±0.022	0.725±0.014
DE (EF)	72.95±0.13	0.154±0.048	0.769±0.008	0.721±0.014
SNGP	61.51±0.30	0.020±0.003	0.753±0.026	0.705±0.024
ETP	74.42±0.20	0.032±0.003	0.780±0.011	0.695±0.007
DE (LF)	75.40±0.06	0.095±0.001	0.722±0.016	0.693±0.006
TMC	72.42±0.05	0.108±0.001	0.681±0.004	0.675±0.006
MGP	73.30±0.05	0.018±0.001	0.803±0.007	0.748±0.007
MNPs (Ours)	74.92±0.07	0.011±0.001	0.872±0.002	0.786±0.005
Table 5: Speed-up (×folds) measured by the ratio of wall-clock time per epoch of MNPs against MGP.
Dataset	Training	Testing
Handwritten	1.62	2.66
CUB	1.41	2.12
PIE	2.88	5.37
Caltech101	1.56	3.30
Scene15	1.89	2.63
HMDB	2.03	3.27
CIFAR10-C	1.17	2.16
MNPs provide the most robust predictions to noisy samples in 5 out of 6 datasets achieved by the MBA mechanism. Both unimodal and multimodal baselines except MGP show limited robustness with a large performance degradation.
In addition to its superior performance, MNPs, as shown in Table 5, are also faster than the SOTA multimodal uncertainty estimator MGP in terms of wall-clock time per epoch (up to  $\times 5$  faster) measured in the identical environment including batch size, GPU, and code libraries (see Appendix B for computational complexity of the two models). This highly efficient framework was made possible
by DCM that stores a small number of informative context points. It also highlights the advantage of using efficient DNNs to imitate GPs, which a non-parametric model like MGP struggles to achieve.
5.2 OOD Detection
Experimental Settings Following the experimental settings of [24], we trained the models with three different corruption types of CIFAR10-C [18] as a multimodal dataset and evaluated the OOD detection performance using two different test datasets. The first test dataset comprised half CIFAR10-C and half SVHN [46] samples, while the second test dataset comprised half CIFAR10-C and half CIFAR100 [31] samples. We used the Inception v3 [54] pretrained with ImageNet as the backbone of all the baselines. The area under the receiver operating characteristic (AUC) is used as a metric to classify the predictive uncertainty into ID (class 0) and OOD (class 1).
Results Table 4 shows test accuracy and ECE with CIFAR10-C and OOD AUC against SVHN and CIFAR100. MNPs outperform all the baselines in terms of ECE and OOD AUC. A large difference in OOD AUC is observed which shows that the proposed adaptive RBF attention identifies OOD samples well. Also, we highlight MNPs outperform the current SOTA MGP in every metric. A marginal difference in test accuracy between DE (LF) and MNPs is observed, but MNPs achieve much lower ECE (approximately 8.6 folds) with higher OOD AUC than DE (LF).
In this study, we introduced a new multimodal uncertainty estimation method by generalising NPs for multimodal uncertainty estimation, namely Multimodal Neural Processes. Our approach leverages a simple and effective dynamic context memory, a Bayesian method of aggregating multimodal representations, and an adaptive RBF attention mechanism in a holistic and principled manner. We evaluated the proposed method on the seven real-world datasets and compared its performance against seven unimodal and multimodal baselines. The results demonstrate that our method outperforms all the baselines and achieves the SOTA performance in multimodal uncertainty estimation. A limitation of this work is that despite the effectiveness of the updating mechanism of DCM in practice, it is not theoretically guaranteed to obtain the optimal context memory. Nonetheless, our method effectively achieves both accuracy and reliability in an efficient manner. We leave developing a better updating mechanism for our future work. The broader impacts of this work are discussed in Appendix D.
[1] H. Akbari, L. Yuan, R. Qian, W.-H. Chuang, S.-F. Chang, Y. Cui, and B. Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 24206-24221. Curran Associates, Inc., 2021.
[2] G. Andrew, R. Arora, J. Bilmes, and K. Livescu. Deep canonical correlation analysis. In S. Dasgupta and D. McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 1247-1255, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
[3] T. Baltrusaitis, C. Ahuja, and L.-P. Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2):423-443, 2019. doi: 10.1109/TPAMI.2018.2798607.
[4] G. Bhatt, P. Jha, and B. Raman. Representation learning using step-based deep multi-modal autoencoders. Pattern Recognition, 95:12-23, 2019. ISSN 0031-3203. doi: https://doi.org/10.1016/j.patcog.2019.05.032.
[5] C. Cangea, B. Day, A. R. Jamasb, and P. Lio. Message passing neural processes. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning, 2022.
[6] S. D. Dao, D. Huynh, H. Zhao, D. Phung, and J. Cai. Open-vocabulary multi-label image classification with pretrained vision-language model. In 2023 IEEE International Conference on Multimedia and Expo (ICME), pages 2135-2140. IEEE, 2023.
A Lemma and Proof
For the comprehensiveness of proof, we duplicate Lemma 3.1 here.
Lemma A.1 (Gaussian posterior distribution with factorised prior distribution). If we have  $p(x_i|\mu) = \mathcal{N}(x_i|\mu, \Sigma_i)$  and  $p(\mu) = \prod_{i=1}^{n} \mathcal{N}(\mu_{0,i}, \Sigma_{0,i})$  for  $n$  i.i.d. observations of  $D$  dimensional vectors, then the mean and covariance of posterior distribution  $p(\mu|x) = \mathcal{N}(\mu|\mu_n, \Sigma_n)$  are:
$$
\Sigma_ {n} = \left[ \sum_ {i = 1} ^ {n} \left(\Sigma_ {i} ^ {- 1} + \Sigma_ {0, i} ^ {- 1}\right) \right] ^ {- 1}, \quad \mu_ {n} = \Sigma_ {n} \left[ \sum_ {i = 1} ^ {n} \left(\Sigma_ {i} ^ {- 1} x _ {i} + \Sigma_ {0, i} ^ {- 1} \mu_ {0, i}\right) \right] \tag {15}
$$
$$
p (\boldsymbol {\mu} | \boldsymbol {x}) \propto \prod_ {i = 1} ^ {n} \frac {1}{\sqrt {(2 \pi) ^ {D} | \Sigma_ {i} |}} \exp \left(- \frac {1}{2} (x _ {i} - \boldsymbol {\mu}) ^ {T} \Sigma_ {i} ^ {- 1} (x _ {i} - \boldsymbol {\mu})\right) \times
$$
$$
\frac {1}{\sqrt {(2 \pi) ^ {D} | \Sigma_ {0 , i} |}} \exp \left(- \frac {1}{2} (\mu - \mu_ {0, i}) ^ {T} \Sigma_ {0, i} ^ {- 1} (\mu - \mu_ {0, i})\right) \tag {16}
$$
$$
\propto \exp \left[ - \frac {1}{2} \left(\sum_ {i = 1} ^ {n} \left(\mu - x _ {i}\right) ^ {T} \Sigma_ {i} ^ {- 1} \left(\mu - x _ {i}\right) + \left(\mu - \mu_ {0, i}\right) ^ {T} \Sigma_ {0, i} ^ {- 1} \left(\mu - \mu_ {0, i}\right)\right) \right] \tag {17}
$$
$$
\propto \exp \left[ - \frac {1}{2} \left(\mu^ {T} \left(\sum_ {i = 1} ^ {n} \left(\Sigma_ {i} ^ {- 1} + \Sigma_ {0, i} ^ {- 1}\right)\right) \mu - 2 \mu^ {T} \left(\sum_ {i = 1} ^ {n} \left(\Sigma_ {i} ^ {- 1} x _ {i} + \Sigma_ {0, i} ^ {- 1} \mu_ {0, i}\right)\right)\right) \right] \tag {18}
$$
$$
= \frac {1}{\sqrt {(2 \pi) ^ {D} | \Sigma_ {n} |}} \exp \left(- \frac {1}{2} (\mu - \mu_ {n}) ^ {T} \Sigma_ {n} ^ {- 1} (\mu - \mu_ {n})\right) \tag {19}
$$
where we dropped constant terms for clarity. From Equation (18) and (19), we can see that:
$$
\Sigma_ {n} ^ {- 1} = \sum_ {i = 1} ^ {n} \left(\Sigma_ {i} ^ {- 1} + \Sigma_ {0, i} ^ {- 1}\right), \quad \Sigma_ {n} ^ {- 1} \mu_ {n} = \sum_ {i = 1} ^ {n} \left(\Sigma_ {i} ^ {- 1} x _ {i} + \Sigma_ {0, i} ^ {- 1} \mu_ {0, i}\right) \tag {20}
$$
$$
\Sigma_ {n} = \left[ \sum_ {i = 1} ^ {n} \left(\Sigma_ {i} ^ {- 1} + \Sigma_ {0, i} ^ {- 1}\right) \right] ^ {- 1}, \quad \mu_ {n} = \Sigma_ {n} \left[ \sum_ {i = 1} ^ {n} \left(\Sigma_ {i} ^ {- 1} x _ {i} + \Sigma_ {0, i} ^ {- 1} \mu_ {0, i}\right) \right] \tag {21}
$$
If we use Lemma A.1 with diagonal covariance matrices for  $p(r_{*,i}^{m}|z_{i}) = \mathcal{N}\left(r_{*,i}^{m}|z_{i},\mathrm{diag}(s_{*,i}^{m})\right)$  and  $p(z_i) = \prod_{m = 1}^{M}\mathcal{N}\left(u^m,\mathrm{diag}(q^m)\right)$ , we can obtain the posterior distribution of  $\mathcal{N}\left(z_i|\mu_{z_i},\mathrm{diag}(\sigma_{z_i}^2)\right)$  as follows:
$$
\sigma_ {z _ {i}} ^ {2} = \left[ \sum_ {m = 1} ^ {M} \left(\left(s _ {*}, i} ^ {m}\right) ^ {\varnothing} + \left(q ^ {m}\right) ^ {\varnothing}\right) \right] ^ {\varnothing}, \mu_ {z _ {i}} = \sigma_ {z _ {i}} ^ {2} \otimes \left[ \sum_ {m = 1} ^ {M} \left(r _ {*}, i} ^ {m} \otimes \left(s _ {*}, i} ^ {m}\right) ^ {\varnothing} + u ^ {m} \otimes \left(q ^ {m}\right) ^ {\varnothing}\right) \right] \tag {22}
$$
where  $\varnothing$  is the element-wise inversion, and  $\otimes$  is the element-wise product.
In this section, we outline additional details of the experimental settings including the datasets (Appendix B.1), hyperparameters of the models used (Appendix B.2), metrics (Appendix B.3), and a brief analysis of computational complexity of MGP and MNPs (Appendix B.4). For all the experiments, we used the Adam optimiser [30] with batch size of 200 and the Tensorflow framework. All the experiments were conducted on a single NVIDIA GeForce RTX 3090 GPU.
B.1 Details of Datasets
Synthetic Dataset Figure 2 shows the predictive probability and the attention weight of different attention mechanisms. Here, we describe the dataset and the settings used for the demonstrations.
We generated 1,000 synthetic training samples (i.e.,  $N_{train} = 1,000$ ) for binary classification by using the Scikit-learn's moon dataset with zero-mean Gaussian noise ( $std = 0.15$ ) added. The test samples were generated as a mesh-grid of 10,000 points (i.e.,  $100 \times 100$  grid with  $N_{test} = 10,000$ ). The number of points in the context memory  $N^{m}$  was set to 100. In this demonstration, we simplified the problem by setting  $M = 1$  which is equivalent to the unimodal setting and illustrated the difference in attention mechanisms.
Robustness to Noisy Samples Dataset In Section 5.1, we evaluated the models' robustness to noisy samples with the six multimodal datasets. The details of each dataset are outlined in Table 6. These datasets lie within a feature space where each feature extraction method can be found in [17].
Table 6: Multimodal datasets used for evaluating robustness to noisy samples.
Dataset
Handwritten	CUB	PIE	Caltech101	Scene15	HMDB
# of modalities	6	2	3	2	3	2
Types of modalities	Images	Image&Text	Images	Images	Images	Images
# of samples	2,000	11,788	680	8,677	4,485	6,718
# of classes	10	10	68	101	15	51
OOD Detection Dataset We used CIFAR10-C [18] which consists of corrupted images of CIFAR10 [31]. 15 types of corruptions and five levels of corruption for each type are available for the dataset. Following [24], we used the first three types as multimodal inputs with different levels of corruption (1, 3, and 5).
B.2 Details of Models
In our main experiments, four unimodal baselines with the early fusion (EF) method [3] (MC Dropout, Deep Ensemble (EF), SNGP, and ETP) and three multimodal baselines with the late fusion (LF) method [3] (Deep Ensemble (LF), TMC, and MGP) were used. In this section, we describe the details of the feature extractors and each baseline.