Uncertainty Estimation for Multi-view Data: The Power of Seeing the Whole Picture
Authors: Myong Chol Jung
He Zhao
Joanna Dipnall
Belinda Gabbe
Lan Du
Venue: NeurIPS 2022 Accept (2022)

Abstract: Uncertainty estimation is essential to make neural networks trustworthy in real-world applications. Extensive research efforts have been made to quantify and reduce predictive uncertainty. However, most existing works are designed for unimodal data, whereas multi-view uncertainty estimation has not been sufficiently investigated. Therefore, we propose a new multi-view classification framework for better uncertainty estimation and out-of-domain sample detection, where we associate each view with an uncertainty-aware classifier and combine the predictions of all the views in a principled way. The experimental results with real-world datasets demonstrate that our proposed approach is an accurate, reliable, and well-calibrated classifier, which predominantly outperforms the multi-view baselines tested in terms of expected calibration error, robustness to noise, and accuracy for the in-domain sample classification and the out-of-domain sample detection tasks

Uncertainty Estimation for Multi-view Data: The Power of Seeing the Whole Picture
Myong Chol Jung Monash University david. david.jume jungomonash.h.....ommmoo edu
He Zhao
Joanna Dipnall
CSIRO's Data61
Monash University
zhao@ieee.... org
jo. dipnall aamaalmmmoaahh..mmmmm edu
Belinda Gabbe Monash University
Lan Du
Monash University
belinda. www.eggaaabbbbbbeeeeeo.o edu
lan. du@monash. edu
Abstract
Uncertainty estimation is essential to make neural networks trustworthy in real- world applications. Extensive research efforts have been made to quantify and reduce predictive uncertainty. However, most existing works are designed for unimodal data, whereas multi-view uncertainty estimation has not been sufficiently investigated. Therefore, propose new multi-view classification framework for better uncertainty estimation and out-of-domain sample detection, where we associate each view with uncertainty-avee tions all the views in a principled way. The experimental results with real-world datasets demonstrate that our proposed approach is accurate, reliable, and well- calibrated classifier, which predominantly outperforms the multi-view baselines tested in terms of expected calibration error, robustness noise, and accuracy for the in-domain sample classification and the out-of-domain sample detection t
classifier and combine the predie-
Introduction
Reliable uncertainty estimation is critical for deploying deep learning models in number of domains such as medical imaging diagnosis (35) or autonomous driving [8]. Even with accurate predictions, domain experts still raise questions of how trustworthy the models are [39]. For example, when model's prediction contradicts domain expert's opinion, the quantification of the uncertainty the model's predictions can help determine model's reliability and justify model use.
Recently, uncertainty estimation of neural networks has been an active research area, where many methods of quantifying uncertainty in predictions have been proposed [44] 9 40 50
32 24]. The majority of existing work focuses on uncertainty estimation for unimodal data. However, in many practical problems, data can exhibit in multi-views multi-modalities, For example, LiDAR, radar, and RGB cameras can simultaneously capture complementary information about a scene [49], and computed tomography (CT) scans and x-ray images can be analyzed together diagnose disease 3. Trustworthy uncertainty estimation with multi-view or multi-modalities data important because the challenges it faces may differ from unimodal setting (e.g., maintaining accurate predictions with one of the input views' domain shifted). Despite the success existing work on unimodality, modelling and estimating uncertainty for multi-view data remain less explored question [1].
"Correponndig author
2 We provide our at httppp///httt..cc.aaaraaaa/aa/ae
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

39
Caie
00D
000
07 Pampiaan
63
(a) Input view
(b) Input view 2
(c) SNGP
(d) MGP (Ours)
Cass
Class
(e) Noisy view (NV) (f) SNGP with NV
(g) MGP with NV
Figure I: Visualization on a synthetic multi-view moon dataset. Top row: the dataset has two views and two classes (e.g.. blue upper circles in (a) and (b) two views of Class 1), and an OOD class (grey); (c) and (d) are the predictive probability surfaces of SNGP and our MGP. Bottom row: A new noisy view (e) is added to the data; (f) and (g) are the predictive probability surfaces of SNGP and our MGP with the noisy view. The darker the region is (i.e., dark blue), the lower the probability of heing class 1. Since SNGP is unimodal model, input views are fused into unimodal dataset. The difference between (c) and (f) shows SNGP cannot correctly capture the input shape in the presence of noise, MGP, however, is robust to noise, shown by minimal difference between (d) and (g).
A way of solving this problem is to fuse multi multi-modalities into one modality and directly apply existing unimodal methods. However, even the sate-offtthe-art unimodal model (e.g.. SNGP [29] can be Without
prone to noise if one of the views in multi-view dataset is noisy, as shown in Figure T
the noisy view, unimodal models can produce accurate and confident predictions nearby the training domain. However, with the noisy view, the predictions become uncertain for samples even close to the training domain (see Figure 3 We also show that the existing multi-view classifiers (e.g.. TMC III have limited capacity to detect oft-oan (OOD) samples in our experiments (see Table 4.
To this end, we propose the Multi-view Gaussian Process (MGP) that is a tailored framework providing intrinsic uncertainty estimation for classification of multi-view/modal data. Specifically, MGP consists of dedicated Gaussian process (GP) expert for each view whose predictions are aggregated by the product of expert (PoE). In our proposed method, there is natural way of capturing uncertainty by measuring the distance between training set and test samples in the reproducing kernel Hilbert space (RKHS). The contributions of our method can be summarized as follows:
1. We propose
uncertainty estimation framework with GPs for multi-view data, which is an under-explored yet increasingly important problem in safety-critical applications.
2. The framework provides better uncertainty estimation through product of expert model, providing more robustness dealing with noise and hetter capacity of detecting OOD data.
We develop an effective variational inference algorithm approximate multi-view posterior distributions in a principled way.
4. We conduct comprehensive and extensive experiments on both synthetic and real-world data, which show that our method achieves the sate-of-the-aas performance for uncertainty estimation of multi milti-v data.
2 Muti-view Gaussian Process
Given training data X = {X1, X2. Xv) where is the number of views, each view consists of training set of N samples X,, { sample consists of V views {Kw,)) (e.g.. the CUB dataset consists of images as the first view and captions as the second view) and y, is the data sample's ground-truth label shared across the views. Without loss of generality, a multiview classification or regression problem can be formulated as
L.V and labels (y:).r In other words, the 2t data

---

predicting y, given testing samples [xxvv=== In this paper, we propose the Multi-view Gaussian Process (MGP). novel framework for villi-view/modal data, where in nutshell we first apply GP to each view of the data and then combine in principled way the predictions from all the GPs as a unified prediction by using the product of expert (PoE) [28].
2.1
GP for Individual View
For each view, we consider a multiclass classification problem with C' classes. We set C' independent Gaussian priors over latent function (.) with zero-mean and N N covariance matrix KNN whose element is Kij = k(Zn,.. x. where k(-;-) is a kernel function. The radial basis function (RBF) which is commonly used in the GP literature [51 33 is selected in this paper as the covariance where oi is the signal variance and by is the length-scale for each GP which are parameters to be optimized.
function. It is defined as lo(2, æ²) o, exp
[2 20g
To bypass the limitations of standard GPs 22 namely high computational cost (NN)) and inconve- nience of applying stochastic gradient descent (SGD), we propose leverage the sparse variational GP (SVGP) [28] [4 [15] (3), which is detailed as follows. With SVGP, we introduce M (M < N) inducing points Z. representing the training samples of view o with a smaller number of points, and the inducing variable
is the latent function evaluated at the inducing points (i.e., where both z and u. are random variables to be optimized. Similar to the Gaussian prior set for the latent function, joint prior can be set as:
fo( (Z.))
fo N (0,
KNN KNM KNM
(1)
uv
KMM
where we use fa to indicate (X...
for notation convenience. The use of inducing points can reduce
the computational cost ((M))
28 We outline the likelihood of GPs in Section 2.2 and the
posterior in Section 2.3
2.2 GPs for Multi-view Data with PoE
Product of Experts (PoE) With one GP expert for each view, we propose to combine the GP Specifically, we aggregate
experts into unified prediction by using the PoE mechanism [7 28 7 posterior distributions of individual views by:
x(f(,y) x II((f
For Gaussian posteriors with mean Mr and covariance E.,, the aggregation using Equation forms the unified posterior's 's mean and covariance expressed as:
a En.zo
E.
E
ΣΣ Σ
Dirichlet-based Likelihood In order to apply Equation (3) to a multi-view problem, the latent function fo in each view should refer to the same observable variable (i.e., ((a(,, c) cannot be combined with (Vcll, e) for a 4 c). However, in GP classification, the latent function is a non- observable nuisance function that squashed through sigmoid or softmax function estimate labels [5I], which is not necessarily the same for every independent view. We alleviate this problem by reparameterizing the class labels regression labels by:
y: = (x...) + &~,00,,)))
where ÿJ is the transformed label and / is the noise parameter fixed for all views. Since ÿ. and o are shared across the views, we ensure that fo (xr..) refers to the same variable. By using the log-normal distribution, the Gaussian likelihood can be used in the log space as p(u. f..) Aff.. o?).
To transform the class labels to regression labels, we propose to adopt representing the class probability n, [T,1, ... c over Dirichlet distribution with the categorical likelihood 3
p(u:)a:) Cat (T;))) where
Dir(a:)
Jr.c
Ti.
where Die ~ Gamma(Qr,
(4)
Σ == Ij.
3

---

where a; = |a,,1 0s,2, .... Ci is the concentration parameters, the shape parameter for Gamma distribution ai,e- and the scale parameter for Gamma distribution Gamma distribution in (4) with Ii.c Jognormall TF,e) by moment matching:
1. We approximate the
ai.c exp (#i. + e)), ai,c = (exp (d7,,) - 1) exp (2ij,c +%,o)
Thus, the transformed labels and the noisy parameter are expressed in terms of the concentration parameters:
(S) ag if Vi.c and ai, ag if Vi,c 0 with the me-hot label Vi.c. de is parameter to prevent the noise parameter from converging to infinity. See Appendix for the impacts of Og the model performance. Compared with other transforming methods such Platt scaling (88. the used Dirichlet likelihood compromises classification accuracy less and requires no post-hoc calibrations after training.
t².- log (1/a + 1), Vi.e log Oii,c - //.e/2
where ai,c
2.3 Training of the Proposed Framework
Given the priors from Section 2.1 and the Gaussian likelihood from Section 2.2 the goal of training our framework is to estimate a posterior distribution via variational inference (VI) [14 [5] 4) By using Equation 2 we propose an aggregated variational distribution for all the views as:
q (f) x II (ffe)
(6)
where ((f.) is the variational distribution for each view that approximates the true posterior. We define (fo) as:
(ff)))) (f.((u,))
((f.) = P
((f))==
((u,) du..
where p (folu,)
(u..) ) is the conditional prior from Equation (T.. and q(u.) ) is the marginal variational distribution of [(m,, S.) with optimizable model parameters m. and S.. The analytical solution of 7 is provided Appendix. VI seeks to minimize the following Kullback-Leibler divergence (KL) between the true posterior and variational distributions:
KL E (f)(
(8)
where ye fūr
Lemma (Additive Property of KL. Divergence). Ifx = [xi, ... ,xn] EX p(x) = II. p(x,) and ((x) = II. ((x;), we have:
KL [p(x))l((>)) = Σ KL (p(xx)
(9)
Theorem 2 (KL Divergence with PoE). With Equations and (6) we have:
KI. (I. x x.)) |
Σ KL.
(10)
According Theorem 2 the VI for the PoE splits the VI of each expert/view. For the th view, the VI minimizes KL (pp(). Øe)], which can be turned into the maximization of the evidence lower bound (ELBO):
ELBO,, = Σ B(ss.)) [log (logp((j...... ((....lf... f...)) - B KL (q. (pp),,)lp(..)))
(11)
where / is a parameter to control the KL term, similar to [66, which can be interpreted as a are provided in Appendix.
regularization term. Proofs of Equation
In order apply SGD, we match the expectation stochastic gradient of the expected log likelihood term to the full gradient by multiplying the number of batches to the log likelihood term in Equation The overall loss for all experts is:
15
C
ELBO.,
(12)
The training steps are summarized in Algorithm

---

Algorithm 1: Learning MGP
Input: V views of training data
X = {X, xx)== where each view JNN
Algorithm 2: Inference of MGP
has N samples of X. {
Input: V views of testing data
and y = {yi J
X, [X,,ee=
Transform: Reparameterize ye by (5)
for to V do
for minibatch do
2
Compute gf.. by 13
for v =1 V do
Calculate 7 X 6, by 17
Compute (ffv) ) by 7
end for
Calculate ELBO, by
5 Aggregate ppoz (f.) (.) by 16
end for
Output: IE' [T.cc] and V 1
of class
Sum ELBOS by 12
probability by 14
SGD update (lu, oe Ze m, So 1V
end for
2.4 Inference on Test Samples
Given test samples X, varitional distribution as:
(X the predictive distribution p(f=,vlūee is estimated by the
p(S..(le) q(fo,o) = f p(S.,oluu, (...))))))) du.
(13)
where p(ff,,/uu) can formed by the joint prior distribution similar to Equation TT) (see Appendix for full derivation). Similar to Equation ( aggregate the predictive distributions to form qPoE (f.) that is sampled to approximate Gamma-distrbbueed samples which in the end form the posterior of Dirichlet distribution follows:
exp (Jia,))
E [Traec]
=
E, exp (figh)) TroE (fea))
V [ =
exp (Firne) ΣS exp (fig.))
E (Tt,e) (poE(fic.))
(14)
Equation (14 can be approximated with the Monte Carlo method. See Appendix for the impacts of the number of Monte Carlo samples on classification performance and inference time. The aggregated predictive distribution can also be weighted by each expert's 's predictive distribution by:
(PoE(f.) x (I(ff.))) (X...)
(15)
2
where 7(X.,)) is the weight controlling the influence of each expert the aggregated prediction. The mean and covariance of (PoE (f+) with 7( (X.,,) are:
μo Eun((X..) Ew. Ew E
(16)
In our experiments, we use negative entropy predictive distribution:
(X.,,) - (g((..)))
(17)
Note that the original PoE [17] in Equation (6) is recovered (X.,,) 1. The intuition behind
choosing negative entropy is that the experts with lower posterior entropy, which means the lower uncertainty, gain more contribution to aggregated predictions. Please note that other choices of
7(X.,) can also be applied such as the difference in entropy from prior to posterior 6 and negative predictive variance [7]. We obtain the better empirical results with negative entropy, the choice of function is flexible. The inference steps are summarized in Algorithm 2
5

---

3 Related Work
Uncertainty Estimation with GP GP has been one of the gold standards of uncertainty estimation because of GP's high sensitivity to domain shift. One of the common ways to implement GP with deep learning models is to place GP the output layer top of extracted features. The features often extracted from deterministic deep neural networks [46 29 [5], Bayesian neural networks (26), or graph data [30]. Similarly, MGP builds on these approaches and be combined with various feature extractors. However, our method differs from all of above studies because these studies designed for unimodal data. However, MGP is multi-view GP. Other variants utilizing kernel learning for uncertainty estimation include deep GP [27 and RBF network [45].
Multi-view Learning Multi-view and multimodal learning aim for various downstream tasks by leveraging multiple data sources that describe the same event or object. Canonical correlation analysis (CCA) [8] holds a long history, which finds a common representation of multiple sources II 47). Similarly, contrastive learning builds the common representation by forming positive pairs and negative pairs [42 12 Also, view-specific representations are learned to make models robust to missing input view [52]. Recently, it has been theoretically and empirically shown that vision-languaa models outperform unimodal models [19 25 2n. Other methods include gradient- blending 48) and hierarchical metric learning [53]. Despite these extensive studies in multi-view and multimodal learning, most of them not mainly designed for uncertainty estimation.
Multi-view Uncertainty Estimation Few studies have been designed and evaluated for multi-view and multimodal uncertainty estimation. Multimodal regression with mixture of normal-invers Gamma distributions yields promising uncertainty estimation and predictions with the real-world data [3I]. However, this method is designed for regression, which differs from our method for classification. The closest study to ours the trusted multi-view classifier (TMC) which combines evidence from different views by using Dempster's 's combination rule [II]. However, the Dempsterd's combination rule ignores predictions of confficting views, which an undesirable property especially in high-risk applications [I] 20 In addition, our experiments show that TMC is overconfident about the OOD samples.
4 Experiments
4.1 Synthetic Dataset
To illustrate predictive behaviours of baselines and MGP, we constructed multi-view synthetic dataset with the Scikit-learn's 's moon dataset by scaling the data with three different scaling factors (the same dataset used in Figure I.. Each view consists of 1,000 data points formed as two half circles: upper circle (class 1) and lower circle (class 2). In each view, data points share the same relative locations with the same labels. Note that the points in the third view overlap each other, representing a noisy view.
Deep Ensemble [23] with late fusion 22 (DE(LF)), TMC, and MGP comprised of dedicated classifiers for every view (view 1, 2, and 3 in Figure (). and the predictions made in the views are combined single prediction represented as multi- view in Figure Since SNGF is a single-view classifier, data points of all the views were concatenated as input. The results of SNGP are shown in Figure For experimental details, see Appendix.
One of the benefits of having a multi-view uncertainty estimator is that the prediction accuracy of an ideal multi-view classifier remains high even if one of the input views does not provide meaningful information. This feature can be achieved by assigning lower weights to the views with high uncertainties when combining predictions. High uncertainty here refers unconfident predictions with a uniform class probability across classes.
Figure 2 shows that the combined predictions of DE(LF) are moderately affected by the noise because the predictions are averaged across all the views. TMC and MGP, on the other hand, are not affected
https://scikitt Lear://sscciittttee...ee http:///ww................/////a
datasets make moons.
html
To provide fair comparison, the same feature extractor without spectral normalization is used for DE(LF), TMC, MGP, and SNGP. See Appendix for results of SNGP with spectral normalization.

---

@ View (P)
(b) View (P)
(c) View 3 (P)
(d) Multi-view (P)
(e) Multi-view (U)
to
(I) View 1 (P)
(g) View (P)
(h) View 3 (P)
(i) Multi-view (P)
( Mullii-viw (U)
(k) View 1 (P)
(i) View 2
(m) View (P) Multi-view (P)
(o) Multi-view (U)
Figure 2: Predictive probability surface (P) of class 1 and uncertainty estimation surface (U) from DE(LF) (a)-(e), TMC (f-(), and MGP (k)-(o) trained with upper circle (class 1) and lower circle
(class 2) synthetic multi-view dataset. Grey data points are OOD samples. View 3 is intentionally made be noisy. Uncertainty surfaces for individual views are plotted Appendix.
by the noisy view (View 3). The majority of areas in the noisy view show high uncertainty (i.e., class probability close to 0.5) which have minimal impact on combining predictions because TMC and MGP are aware of uncertainty of each view. However, SNGP's prediction is heavily impacted by the noisy view as shown in Figure Ic If SNGP were trained without any noisy view, it can properly estimate class probability (Figure Ii This difference between Figure Ic and fillustrates the degrading effect of noisy view to single-view classifiers.
Although TMC is robust noise, it produces overconfident predictions at the regions far from decision boundaries (see Figure 21). As a result, OOD samples are indistinguishable from in- domain samples (see Figure 2 This is mainly caused by lack of distance awareness which MGP and SNGP have in common due to GP (29). Figure 2o show that MGP is well aware of the distance between the training domain and OOD by allocating high uncertainty at the OOD samples where uncertainty is calculated by the sum of variance across all classes.
4.2 Robustness to Noise
Experimental Settings We used six multi-view datasets [1] (Handwritten, CUB, PIE, Caltech 101, Scene15, and HMDB) with train-test split of 0.8:0.2. For testing robustness to noise, we added Gaussian noise of zero mean to half of the views for each dataset, following the experimental setting in [TI] In order to have the same impact of noise to all views, normalized each view first and then added the noise because the range of raw data of each view varies significantly We increased the noise standard deviation from 0.01 to 10. To report test results invariant to selecting which views to add the noise, ran all combinations of selecting noisy views (i.e., (v/2) configurations) and report its average for each noise level.
Compared Methods
We selected three single-view baselines and two multi-view baselines. For single-view baselines, we used early fusion (EF) technique 2 by concatenating multi-view features
'TMC's authors added the noise first and then normalized the noisy input. The results of this setting are reported in Appendix.

---

1,0
0.8
MCD
0.3
Acamey
DE (RF)
Accimnt
Accumay
SNGPP
8.50
0.4
TMC
0.4
0.25
MGP (Ours)
0.00
10
10
10
10
10
10
10
Noise std
Noise
Noise sod
(a) Handwritten
(b) CUB
(c) PIE
0.75
0.6
0.6
Accuncy
Aucmncy
Accuneyy
0.50
0.4
0.4
0.2
0.0
(d) Caltech101
(c) Scene15
(f) HMDB
Figure 3: Domain-shift test accuracy where Gaussian noise added to half of the views.
Table 1: In-domain test accuracy 1
Dataset
Method
Handwritten
CUB
PIE
Caltech101
Scene
HMDB
MC Dropout
99.25=00000
92.33=1.09
91.32:+0.22
92.95=0.299
71.75=0.25
71.68::0.36
DE (EF)
99.20=#011
93.1610.70
91.76++0.3
92.99=:0.09
72.70=0.39
71.67+0.23
SNGP
98.85=0.22
89,50=0.75
87.06+1.23
91.244:046
64.68=4.03
67.65+1.03
DE (LF)
99.25=10.00
92.33+0.700
87.21+0.66
92.97=0.13
67.05=0.38
69.9810.36
TMC
98. 18.10
91.17=0.46
91.18+1.72
91.63::.28
67.68=0.27
65.17
MGP (Ours)
98.60-10.14
92.33-10070
0.99
93.00-L0.33
70.00-00.53
7..30-10.19
Table 2: In-domain BCE 4
Dataset
Method
Handwritten
CUB
PIE
Caltech101
Scene15
HMDB
MC Dropout
0.009:10.000
0.069::0.017
0.299-+0.005
0.017+00003
0.181=0.003
0.388=0.004
DE (EF)
0.007:10.000
0.054:10.010
0.269:10.004
0.03610.001
0.089-0.003
0.095-0.003
SNGP
0.023:40.004
0.200=10.010
0.852+0.012
0.442-1..044
0.111=0.063
0.227=0.010
DE (LF)
0.292:10.001
0,270:10.009
0.567=10.00
0.02311.002
0.019=00005
0.270=0.003
TMC
0.013
0.141:10.002
0.072-10.011
0.06810.002
0.180=0.004
0.594=0.008
MGP (Ours)
0.006+0.004
0.008
0.079+0.0007
0.009+0.003
0.062 0.006
0.036-0.000 0.003
into single feature. The selected single-view haselines are as follows: MC Dropout
9 with dropout
rate of 0.2, Deep Ensemble (DE) [23] with 5 models, and SNGP's GP layer
29 The multi-view
baselines are following: Deep Ensemble (DE) with late fusion (LF) technique 2 where one model
was trained for one view and predictions of all views were combined single prediction by averaging the predictions, and TMC IT.. For details of model settings, see Appendix.
Results For every metric, mean and standard deviation from five runs with different random seeds reported. The results of multi-view classifiers are generated from the combined predictions. In Table and 2 we provide test accuracy and test expected calibration error (ECE) [0O] without adding noise (the definition of ECE provided in Appendix). outperforms DE(LF) over 4 out of 6 datasets and TMC over all the datasets. Also, the test ECE of MGP outperforms both DE(EF) and DE(LF) over all the datasets and TMC over out of datasets. Sshww the test accuracy with respect to the standard deviation of Gaussian noise, representing the domain-shift test accuracy. This highlights that MGP robust to noise while the accuracy of others degrades significantly. We also report the average accuracy in Table
terms of accuracy, MGP

---

Table Average test accuracy with Gaussian noise (std from 0.01
10) added to half of the views.
Dataset
Method
Handwritten
CUB
PIE
Caltech101
Scene
HMDB
MC Dropout
82.15+0.117
76.08+0.61
64.65+00.7
73.45+0.11
48.97+0.33
42.63++0.08
DE (EF)
82.16=10118
76.94-0..2
65.53:1.200
73.99:0.19
49.45=0.35
41.9211.06
SNGP
72.46=0.41
61.27=1.24
56.52:1.69
56.57=0.17
38.19=1.86
37.4910.42
DE (LF)
95.63=10.08
76.1610.22
67.6910.33
81.8510.14
50.13=0.27
43.0110.19
TMC
82.4410.15
74.19:0.69
62.18:1.80
71.77::.22
42.52:10.29
36,6110.30
MGP (Ours)
97.66+0011
85.48+0.22
90.97+0.19
92.68+0.23
65.74+0.56
67.02+-021
Table 4: Out-of-domail detection results with CIFARIO-C
OOD AUROC f
Method
Test accuracy
ECE.
SVHN
CIFAR100
MC Dropout
74.76110.27
0.013:0.002
0.788-0.022
0.725
DE (EF)
72.95:10.13
0.15410.048
0.76910.008
0.72110.014
SNGP
61.51100.30
0.020:10.003
0.753-100066
0.705:10.024
DE (LF)
75.40:10.06
0.095=0.001
0.722=00016
0.693+0.000
TMC
72.42=00.05
0.108=0.001
0.681=0.004
0.675+0.006
MGP (Ours)
73.30+0.05
0.018=0.001
0.803=0.007
0.748
4.3 OOD Samples Detection
Experimental Settings Similar to experimental settings in [29]. we investigated OOD detection test with CIFARIO-C [3]. SVHN (36). and CIFAR [22). We used CIFARIO-C as a multi-view dataset which is corrupted version of CIFARIO with 15 different types of corruption and 5 severity levels. The first three corruption types were selected as three multi-view inputs with severity levels of 1, 3, and 5 respectively in order to have variety of noise levels. Each view was trained with CIFAR10-C and tested with SVHN and CIFAR 100 as OOD detection tests. Detecting SVHN samples easier task since SVHN is distinct from CIFAR IO-C. and detecting CIFAR 100 samples is more difficult task since CIFAR100 looks similar to CIFARIO-C. For all methods, we used the Inception v3 [4T] pre-trained with ImageNet the feature extractor without further training it. Details of experimental settings are reported in Appendix.
Results Table 4 shows in-domain test accuracy, ECE for CIFARIO-C, and OOD results with SVHN and C0FAR100. MGP's OOD results significantly outperform the others with comparable test accuracy and ECE. Especially, MGP outperforms TMC over all the metries. Note that the difference in ECE and OOD AUROC between TMC and MGP is considerably larger than the difference in test accuracy. The similar pattern is observed when DE(LF) and MGP are compared, where test accuracy of DE(LF) is slightly higher than MGP, but MGP outperforms DE(LF) with ECE and OOD AUROC. This highlights that although multi-view baselines can provide accurate predictions with in-domain samples, their calibration and uncertainty estimation could be limited, which aligns with To validate that the predictive uncertainty of MGP could identify OOD samples, also provide the predictive uncertainty with both testing sets of SVHN and CIFAR 100 in Figure
37
34 29
45
Conclusion
In this work, we have proposed a new multi-view classification framework for better uncertainty estimation and out-of-domain sample detection, where we associate each view with an uncertainty- aware GP classifier and combine the predictions of all the views with the PoE mechanism. With both the synthetic and the real-world data, we empirically demonstrated that our method is robust to domain-shift and aware of OOD samples, outperforming other baselines ECE and OOD detection scores. Our method is not limited to a particular feature extractor and can be attached on top of existing feature extractors. possible limitation of our work is that the weight term that balances predictive distributions is introduced in sub-optimal way. We leave optimizing it as future work.

---

SVHN
CIFAR100
2
Densily
2
CIFARIO-C
Density
CIFARIO-C
0.00
0.00
0.25
0.50
0.75
0.25
0.50
0.75
Uncertainty
Uncertainty
(a) SVHN
(b) CIFAR 100
Figure 4: Uncertatinty density MGP with OOD testing sets: (a) SVHN and (b) CIFARIO0.
Acknowledgments and Disclosure of Funding
Acknowledgments We would like to thank reviewers for their time and effort review this paper. We sincerely appreciate the comments that enhanced and extended our work.
Funding This work part of the Predicting fracture outcomes from clinical registry data using artificial intelligence supplemented models for evidence-informed treatment (PRAISE) study. This study is funded by the National Health and Medical Research Council of Australia Ideas Grant (NHMRC- APP2003537).
References
[1] G. Andrew, R. Arora, J. Bilmes, and K. Livescu. Deep canonical correlation analysis. In Proceedings of the 30th International Conference on Machine Learning, pages 1247-1255, 2013.
[2] T. Baltrušaitis, C. Ahuja, and L.-P. Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, (1 41(2):223443, 2019. doi: 10. 109/TPAMM.2018..22 27996777
[3] A. Bhandary, G. A. Prabhu, V. Rajinikanth, K. P. Thanaraj. S. C. Satapathy, D. E. Robbins, C. Shasky, Y.-D. Zhang, J. M. R. Tavares, and N. M. Raja. Deep-learning framework to detect lung abnormality study with chest x-ray and lung ct images. Pattern Recognition Letters, doi: htpp:///doi.og/II1 pateo/o.
129:271 1788, 2020. ISSN 0167-8655.
[4] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 11(511))859-877, 2017. doi: 10.1080/ 29221449.0017.28855
[5] Bradshaw, G. G. Matthews, and Z. Ghahramani. Adversarial examples, uncertainty, and transfer testing robustness in gaussian process hybrid deep networks. arXiv preprint mwXivv/707.000 02000000.476... 2017.
[6] Y. Cao and D. Fleet. Generalized product of experts for automatic and principled fusion of gaussian process predictions. arXiv preprint arXiv: 1400.1177. 2014.
[7] S. Cohen, R. Mbuvha, T. Marwala, and M. Deisenroth. Healing products of Gaussian process experts. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 2068-2077. PMLR, 13-18 Jul 2020.
[8] D. Feng, L. Rosenbaum, and K. Dietmayer. Towards safe autonomous driving: Capture uncertainty in the deep neural network for lidar 3d vehicle detection. 2018 21st International Conference Intelligent Transportation Systems (ITSC), pages 3266-32662 3273, 2018. 1109/TSSC.2188. 2....99811188
doi: 10.
[9] Y. Gal Z. Ghahramani. Bayesian convolutional neural networks with bernoulli approximate variational inference. arXiv preprint
2015.
10

---

[10] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks, In D. Precup and W. Tch, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321-1330. PMLR, 06-11 Aug 2017.
[11] Z. Han, C. Zhang, H. Fu, and J. T. Zhou. Trusted multi-view classification. In International Conference on Learning Representations, 2021.
[12] K. Hassani and A. H. Khasahmadi. Contrastive multi-view representation learning on graphs. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages 41 4126. PMLR, 13-18 Jul 2020.
[13] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference Learning Representa- tions, 2019.
[14] Hensman, N. Fusi, and N. D. Lawrence. Gaussian processes for big data. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, UAI' 13, page 282-290, Arlington, Virginia, USA, 2013. AUAI Press.
[15] J. Hensman, A. Matthews, and Z. Ghahramani. Scalable Variational Gaussian Process Clas- sification. In Lebanon and S. V. Vishwanathan, editors, Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceedings of Machine Learning Research, pages 351-360, San Diego, California, USA, 09-12 May 2015. PMLR.
[16] I. Higgins, L. Matthey, A. Pal, C. P. Burgess, X. Glorot, M. M. Botvinick, S. Mohamed, and A. Lerehner. 8-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[17] G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput., 14(8)1777--1800, aug 2002. ISSN 0899-76677 doi: 10. 10.1162/089976667761113301
[18] H. Hotelling. Relations between two sets of variates. Biometrika, 28(3/4) 1936. ISSN 00063444.
[19] Y. Huang, C. Du. Z. Xue, X. Chen, H. Zhao, and L. Huang. What makes multi-modal learning better than single (provably). In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 10944-10956. Curran Associates, Inc., 2021.
[20] A. Jøsang. Belicf fusion. In Subjective Logic, volume 3, chapter 12, pages 207-236. Springer, 2016.
[21] W. Kim, B. Son, and I. Kim. Vilt: Visioneaand-anguuage transformer without convolution or region supervision. ICML, pages 5583-5594, 2021.
[22] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[23] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, page 6405-6416, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 0781510860949
[24] Lee, M. Humt, J. Feng, and R. Triebel. Estimating model uncertainty of neural networks in sparse information form. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5702-5713, PMLR, 13-18 Jul 2020.

---

[25] Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 9705. Curran Associates, Inc., 2021.
[26] Y. Li. S. Rao, A. Hassaine, R. Ramakrishnan, D. Canoy, G. Salimi Khorshidi, M. Mamouci, Lukasiewicz, and K. Rahimi. Deep bayesian gaussian processes for uncertainty estimation in electronic health records. Scientific reports, 111):1113, 2021.
[27] J. Lindinger, D. Reeb, C. Lippert, and B. Rakitsch. Beyond the mean-field: Structured deep gaussian processes improve the predictive uncertainties. In H. Larochelle, M. Ranzato, R. Had- sell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 8498-8509. Curran Associates, Inc., 2020.
[28] H. Liu, Y.-.. Ong, X. Shen, and J. Cai. When gaussian process meets big data: A review of scalable gps. IEEE Transactions on Neural Networks and Learning Systems, 31(11):44554443, 2020. doi: 10. 109/1N99NNNN...
[29] J. Liu, Z. Lin, S. Padhy, D. Tran, T. Bedrax Weiss, and B. Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7498-7512. Curran Associates, Inc., 2020.
[30] 7.-Y. Y. Liu, S.-Y. Li, S. Chen, Y. Hu, and S.-J. Huang. Uncertainty aware graph gaussian process for semi-supervised learning- Proceedings of the AAAI Conference on Artificial Intelligence, 34 (04):4957 4964, Apr. 2020. doi: 1009/aaai. ht09/aaai.v3440444 v34i04.5934.
[31] H. Ma, Z. Han, C. Zhang, H. Fu, J. T. Zhou, and Q. Hu. Trustworthy multimodal regression with mixture of normal-inverse gamma distributions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 6881-6892 Curran Associates, Inc., 2021.
[32] A. Malinin and M. Gales. Predictive uncertainty estimation via prior networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 7047-7058, Red Hook, NY, USA, 2018. Curran Associates Inc.
[33] D. Milios, R. Camoriano, P. Michiardi, L. Rosasco, and M. Filippone. Dirichlet- gaussian processes for large-scale calibrated classification. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Besachi, and R. Garnett, editors, Advances in Neural Information Pro- cessing Systems, volume 31. Curran Associates, Inc., 2018.
[34] J. Mukhoti, A. Kirsch, J. van Amersfoort, P. H. Torr, and Y. Gal. Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty. arXiv preprint ar arXiv:2 1582, 2021.
[35] Nair, D. Precup, D. L. Amold, and Arbel. Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. Medical Image Analysis, 59:101557, 2020. ISSN 1361-8415. doi: https://doi.org/// wttp//....1..0... mew.a.//mmmmaaa...me 2019 1015557
[36] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in nat- ural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[37] Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and J. Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[38] Platt et Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
12

---

[39] A. G. Roy, S. Conjeti, N. Navab, and C. Wachinger. Bayesian quicknat: Model uncertainty deep whole-brain segmentation for structure- wise quality control. Neurolmage, 195:11-22. 2019. ISSN 1053-8119. doi: https://doi.or//100 wttp///...///////../../ (016///.eeurommage 2019.03.441.000
[40] M. Sensoy, L. Kaplan, and M. Kandemir. Evidential deep learning to quantify classification uncertainty. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bisaa-Bincch, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[41] C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna. Rethinking the inception archi- tecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2016.
[42] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. In A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, editors, Computer Vision - ECCV 2020, pages 776-794, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58621
[43] M. Titiis. Variational learning of inducing variables in sparse gaussian processes. In D. van Dyk and M. Welling, editors, Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Research, pages 567-574, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16-18 Apr 2009. PMLR.
[44] M. Valdenegro-Torg Deep sub-ensembles for fast uncertainty estimation in image classification. arXiv preprini 1910.0810, 2019.
[45] J. Van Amersfoort, L. Smith, Y. W. Tch, and Y. Gal. Uncertainty estimation using a single deep deterministic neural network. In H. D. IIII and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 0690-9700 PMLR, 13-18 Jul 2020.
[46] van Amersfoort, L.. Smith, A. Jesson, O. Key, and Y. Gal. On feature collapse and deep kernel learning for single forward pass uncertainty. arXiv preprint arXiv:2102. 2021.
[47] W. Wang, R. Arora, K. Livescu, and J. Bilmes. On deep multi-view representation learning. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1083-1092, Lille, France, 07-09 Jul 2015. PMLR.
[48] W. Wang, D. Tran, and M. Feiszli. What makes training modai classification networks hard? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[49] Z. Wang, Y. Wu, and Q Niu. Multi-sensor fusion in automated driving: A survey. IEEE Access, 8:2847-2868, 2020. doi:
(000/AACCSSS2019.2990
[50] Y. Wen, D. Tran, and J. Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020.
[51] C. K. Williams and C. E. Rasmussen. Gaussian Processes for Machine Learning. The MIT Press, 2006.
[52] C. Zhang, Z. Han, cui, H. Fu, J. T. Zhou, and Q. Hu. Cpm-nets: Cross partial multi-view networks. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[53] H. Zhang, V. M. Patel, and R. Chellappa. Hierarchical multimodal metric learning for multi- modal classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Our contributions are supported by Section and
Section 4
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Appendix.
(b) Did you include complete proofs of all theoretical results? See Appendix. [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] See Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section Jactionnaand Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4 and Appendix.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "wFKzo6WqPSF", "reviewer": "Reviewer_2rPi", "summary": "This paper proposed Multi-view Gaussian Process (MGP) for classification with multi-view and multi-modal data. The MGP modeled each view of data using Gaussian Process (GP) whose classification results are aggregated using product of expert (PoE). Variational inference (VI) is used for model learning (estimation of posterior distribution of latent function $f_v$) and inference (estimation of class probability given test data). Experiment with synthetic data and multiple real-world dataset shows that proposed method achieves comparable in-distribution classification accuracy and better robustness to noise than other state-of-the-art methods that also considers uncertainty in data. The proposed method also showed better performance in out-of-distribution detection (trained with CIFAR10-C and detect samples from SVHN and CIFAR100). ", "questions": "1. How is the $M$ value determined in Eq. (1) and subsequent experiments?\n2. In algorithm 1 step 3, how is $q(f_v)$ computed by Eq. (7)? Is there an analytical solution to the integration?\n3. In OOD experiment, how is the three views used for training selected? Can we expect similar results if we attempt training and evaluation with a different choice of views such as in a manner of k-fold cross validation?\n4. Can the authors provide some discussion on the computation cost vs. accuracy given Monte Carlo estimation is used for Eq. (14) during inference?\n5. In Figure 3, the performance of TMC appears to be comparable to DE (EF) but worse than DE (LF), which is different from what reported in [10], which also had a much wider range of noise magnitude. Can the authors comment on what may contribute to this difference?\n6. Typo: Line 69 'applyinh' should be 'applying'", "limitations": "The author provided a discussion on societal impacts in the supplementary materials. I have no concern on the limitations and potential negative societal impact of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The paper extended GP to model multi-view data and utilize PoE for combining results of different views. The overall framework seems to be novel. The adoption of VI provides a principled way to solve the learning and inference problems. Though this is not groundbreaking as it has been adopted for GP in the past such as [14].\n2. Empirical evaluation on multiple real-world datasets show that the proposed method is much more robust than other uncertainty-aware methods in robustness to noise and out-of-distribution data.\n\n## Weakness\n\n1. There is no discussion on how uncertainty is actually computed in MGP. In addition, although the authors did provide illustration in Figure 2 on uncertainty values, there is no equivalent illustration or analysis on real datasets. \n2. Although the proposed method to improve the robustness and ability to detect OOD samples of classification model. It comes with a cost of reduced in-domain classification accuracy (at least not as competitive as some other state-of-the-art e.g. DE). It is unclear whether this is a limitation of the classification model (or representation power) or the limitation of the proposed inference framework.\n3. The metrics Expected Calibration Error (ECE) is introduced without definition or citation. It is preferred to at least describe the intuition and add citation to the metrics. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RqzxuG86_UQ", "reviewer": "Reviewer_LsN2", "summary": "This paper proposes a new uncertainty estimation framework with Gaussian Process for multi-view/modal data. The framework is basically a product-of-expert model. The model is trained via a variational inference algorithm to approximate the multi-view posterior distributions. Experiments on both synthetic and real-worl data are conducted. ", "questions": "The whole part of methodolgy needs to be clarified and improved. Please see the problems in Weaknesses.", "limitations": "The limitations and potential negative societal impact are not explained in the paper.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n\n# Weaknesses\n1. The paper is hard-to-follow, especially in Sec. 2. [L63] The latent function $f_v$ is not defined. Inferring from the context, the dimension of $f_v$ is $N$, i.e., the number of training samples. However, in many real-world dataset, $N$ can be ten thousand to one million, in which case the covariance is intractable. [L66] The kernel function indicates that the inputs $x_{v,i}$ are real numbers, while in practice the inputs are often high dimensional images or discrete tokens (like captions). [L114] What is the $c$ in $\\tilde{y}_{i, c}$? After reading the whole section, the reviewer cannot find the origin of **Gaussian Process** and thinks the term is inappropriate.\n2. The experiments are not convictive. There is no metrics for synthetic dataset. The rationality of synthetic dataset needs to be justified. Why can the two moon curves be considered as two views of the samples from the same class? Why are the red and blue classes not symmetric? \n\nTypos:\n[title] Uncertainty Estimation for Multi-View Data\n[L69] applyinh -> applying\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MTma_fiPvRY", "reviewer": "Reviewer_7qYz", "summary": "This paper is about uncertainty quantification for multi-view inputs, where multiple views are input to a model and a decision (classification or regression) must be made by combining decisions made with each individual input/view.\n\nThe authors propose a new kind of Gaussian Process, the Multi-View GP, that works by combining predictions from individual views with a Product of Experts (PoE) and training/estimating the posterior predictive distribution using variationa inference. The main purpose seems to be robustness to noise in some of the views, and out of distribution detection.\n\nThe contributions of this work are:\n- A concept for uncertainty quantification on multi-view data using Gaussian Processes and Variational Inference.\n- The proposed framework improves on the state of the art for multi-view/multi-modality classification under noise corrupted inputs, calibration, and out of distribution detection, on several datasets.\n- Multi-view posterior distributions are approximated using variational inference in a theoretically grounded way, and predictions for each view are inversely weighted to their uncertainty, meaning that only the most certain views contribute the most to the final output of the model.", "questions": "- How was Product of Experts chosen? [27] shows that there are other options, like Mixture of Experts, and Naive Local Experts, so was there an ablation using these other choices to combine individual views?\n- Would it be possible to also use Deep Ensembles (+ some approximate Bayesian NN) instead of Gaussian Processes for each of the views, while keeping the variational posterior distribution?", "limitations": "The supplementary material contains a nice section on societal impacts, mostly about the approximation to uncertainty which could be misleading, and  to raise awareness of the importance of uncertainty quantification for safety and trustability of AI systems.\n \nThere are some obvious limitations that could be mentioned or discussed, such as that it provides an approximation (due to approximate GP, variational inference, etc), and that the formulation is made for classification. The authors mention that the weight computation can be suboptimal, which I believe is true.\n", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper deals with the problem of multi-view/multi-modal inputs to machine learning models, in particular Gaussian Processes, combined with uncertainty quantification. As the authors say, this specific combination (multi-view + uncertainty) is often not well explored. I believe the ideas presented in this paper are novel, in particular the use of Gaussian Processes for multi-view/multi-modal data, and the results show that this combination is robust to noise.\n- This work is very well written, the ideas flow very clearly and it was a pleasure to read. I only have some minor comments below.\n- The formulation used in the paper makes sense. First there is one Gaussian Process for each input view/modality, and the predictions are combined using Product of Experts. The joint predictive distribution is approximated using variational inference, in a very standard framework by maximizing the ELBO. At prediction time, the different views/modalities are weighted by the negative entropy of the predictive distribution, which gives more weight to more certain views/modalities. The whole formulation is very well theoretically grounded and connected to the literature.\n- Overall the baselines selected are appropriate for comparison. I have no doubts about the evaluation, but in Weaknesses I mention some additional baselines that could be a good comparison.\n- One experiment uses 6 common multi-view datasets, and adds Gaussian noise to half of the views, simulating a case of corrupt data or sensor failure. In this experiment, the proposed method MGP clearly outperforms all baselines in terms of accuracy as noise is increased, but more notably MGP seems to be very robust to noise in some of the views, with minor drops in accuracy as the noise standard deviation is increased.\n- Another experiment tests out of distribution detection in a synthetic multi-view dataset based on CIFAR10-C, where views are different corruptions of the dataset, and SVHN/CIFAR100 are used as out of distribution datasets. In this setup, for the in-distribution data, MGP has the best calibration error, no significant degradation in accuracy, and best accuracy when corrupting half views with Gaussian Noise. For the OOD detection setting, MGP has the best AUC.\n- The paper has a very nice and detailed supplementary material with proofs and additional results and details.\n- Overall I believe that this paper is well executed, the ideas are novel, and the results show that it is a very good improvement over the state of the art. I believe the paper should be accepted.\n\nWeaknesses\n- One major weakness is that the proposed method seems only to be defined for classification, but I believe it is possible to extend it to regression, but the paper does not evaluate or describe the formulation for a regression setting.\n\n~~- I think in terms of baselines, some methods like deep ensembles could also  be trained in a similar way to weight each view prediction with the negative entropy, in order to show if this mechanism is important to improve performance, particularly in noisy views. A deep ensemble could also in theory be combined with Bayes by Backprop and be trained using the Variational approximation, not sure if this would work but the authors could mention if this is viable and why it was not tried.~~\n\nMinor issues\n- In Fig 1 and 2, it would be nice to have a proper legend for the colorbar on the right, indicating what exactly each heatmap is representing.\n- I think the paper is using the ECE metric without defining it, I am aware it is the Expected Calibration Error, and is clearly defined in the supplementary, but some indication of what it is, like a reference, should be in the main paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept