Calibrating Transformers via Sparse Gaussian Processes
Authors: Wenlong Chen
Yingzhen Li
Venue: ICLR 2023 poster (2023)
Keywords: Transformers
Gaussian processes
Bayesian neural networks
uncertainty estimation
variational inference

Abstract: Transformer models have achieved profound success in prediction tasks in a wide range of applications in natural language processing, speech recognition and computer vision. Extending Transformer’s success to safety-critical domains requires calibrated uncertainty estimation which remains under-explored. To address this, we propose Sparse Gaussian Process attention (SGPA), which performs Bayesian inference directly in the output space of multi-head attention blocks (MHAs) in transformer to calibrate its uncertainty. It replaces the scaled dot-product operation with a valid symmetric kernel and uses sparse Gaussian processes (SGP) techniques to approximate the posterior processes of MHA outputs. Empirically, on a suite of prediction tasks on text, images and graphs, SGPA-based Transformers achieve competitive predictive accuracy, while noticeably improving both in-distribution calibration and out-of-distribution robustness and detection.

Published as a conference paper at ICLR 2023
CALIBRATING TRANSFORMERS VIA SPARSE GAUS- SIAN PROCESSES
Wenlong Chen & Yingzhen Li Imperial College London (wenlong chennn2, yingzhen. eeweeiieeeeiiiiie impirpppraaa ac..c.
ABSTRACT
Transformer models have achieved profound success in prediction tasks in a wide range of applications in natural language processing. speech recognition and com- puter vision. Extending Transformer's 's success to safety-critical domains requires calibrated uncertainty estimation which remains under-explored. To address this, we propose Sparse Gaussian Process attention (SGPA), which performs Bayesian inference directly in the output space of multi-head attention blocks (MHAs) in transformer calibrate its uncertainty. It replaces the scaled dot dot-product oper- ation with a valid symmetric kernel and uses sparse Gaussian processes (SGP) techniques to approximate the posterior processes of MHA outputs. Empirically, on a suite of prediction tasks on text, images and graphs, SGPA-based Transform- ers achieve competitive predictive accuracy, while noticeably improving both in- distribution calibration and out-of-distribution robustness and detection.
INTRODUCTION
Significant improvements have been made for accuracies in prediction tasks for computer vision, speech recognition and natural language processing using deep learning (He al.) 2015 Graves itt al. 2013 Vaswani et al. 2017). In particular, Transformers (Vaswani et al. 2011) based on multi head attention (MHA) have gained popularity recent years. With Transtormers being deployed in many downstream applications (Vaswani et al.) 2017: Dosovitskiy et al.) 2021] Brown et al.) 2020). it is crucial to prevent poor robustness which often comes from erratic outputs with high confidence from these models (Guo et al., 2017b; Mukhoti et al.) 2020). This requires calibrated uncertainty quantification for Transformers which is much less well-studied at the time of this work, and it raises concerns about using Transformers for safety-critical tasks which require rational and risk-averse decision making under uncertainty.
Regarding uncertainty quantification, Bayesian inference is a powerful and principled framework build probabilistic models for rational prediction and decision-making under uncertainty (Gal, 2016). Significant progress is observed for applying (approximate) Bayesian inference methods to quantify uncertainty fully-connected, convolutional and recurrent neural networks (Blundell et al. 2015:: Gal & Ghahramani, 2016); [Zhang et al. [019 Ritter et al.) (021). Initial efforts have been made on extending these techniques to Transformers but with mixed results (Tran et al. 2019 Xue et al. 2021).On On the other hand, Gaussian processes (GPs) are gold standard methods for tasks requiring reliable function- 2020). Researchers have proposed to integrate deep learning ideas to GP model design, including deep kernel learning (Wilson et al. 2016) and deep GPs (Damianou & Lawrence 2013) Salimbeni
uncertainty estimates (Rasmussen & Williams 2006 Wilson et al.)
& Deisenroth] 2017). Still these models have yet to be scaled to modern deep learning tasks such as large-scale image classification and language modelling.
In this work, we propose sparse Gaussian process attention (SGPA), a novel uncertainty quantifica- tion technique for attention- models (e.g.. Transformers). by leveraging techniques from sparse variational Gaussian processes (SVGP) (Snelson & Ghahramani 2005 Hensman et al.) (013) for improved uncertainty estimates. Our work presents the following insights and contributions:
Our key observation is that kernel-based attention (Tsai et al. 2019) is equivalent to the posterior mean of an SVGP. This inspires extend SVGP to Transformers for uncertainty estimation.

---

Published as a conference paper at ICLR 2023
FL
mi)
R
(0,
(me)) x
(m)) 22))
xW
#WC
1/- 1.7
#W
(AAU Norm
kernel
Nemory
kemni atteriin
G201
Food Paneard
reduction
Gg(-)"
Ga(g) presious layer
oulput froms presious layer
cupput
(a) Vanilla Transformer
(b) Standard SGPA (ours)
(c) Decoupled SGPA (ours)
Figure 1: Illustration of one head (h) of multi-head self attention in one layer of (a) vanilla Trans-
former, (b) Transformer based on standard SGPA and (c) Transformer based on decoupled SGPA.
The resulting Transformer based on our SGPA approach can be viewed as a sparse deep GP (Salimbeni & Deisenroth 2017) with deep kernel in for each GP layers.
We address the computational inefficiency issues of a naive extension of SVGP to multi-head self-attention with decoupled inducing points techniques (Salimbeni et 2018), making SPGA scalable to deep learning tasks that Transformers are applied to.
Empirically, on variety of vision, NLP and graph prediction tasks and compared with base- lines, SGPA-based Transformers improve considerably over in-distribution calibration, out- of-distribution (OOD) robustness, and OOD detection, while achieving competitive accuracy against Transformers with standard (Vaswani et al. 2017) or kernel attention (T'sai et al. 2019)
2 BACKGROUND
Attention mechanism, first introduced in Graves et al. (2013), has become the bulding block for Transformer models. In this work, we consider Transformers using multi-head self-altention (MHSA) as in Vaswani al. (2017); Dosovitskiy al. (202T). Here, briefly review MHSA and sparse variational Gaussian process, based on which our method is developed.
2.1 MULTI-HEAD SELF-AITENTION (MHSA)
Given T queries q Txd, keys & Txd, (dg dg) and values v E TRx', dot-product attention Vaswani et al. 2017) is computed as follows, using a nonlinear activation function wt
F= (gkk )v.
(1)
The dot product (gk' measures the similarities between the queries and keys. For self attention the keys are simply set be equal to the queries, i.e., k q. Transformers use multi-head self-attention (MHSA) which modifies dot-product self-attention as follows. Assume // attention heads are in use. Then given 7 inputs s E Rd'xd, to the MHSA block. first project them the queries for each head with a projection matrix W E xd,; q" sshh We obtain the keys ke and values v accordingly by projections using matrices W Rdd,xd and W Rdd xd, respectively. Typically we use the same d, = dy = d, for all the heads. Then the head's output F, is obtained by plugging q², h and v in eq. Lastly the attention outputs from each head is combined as follows with the output projection matrix WF E TR (Hd.)))(HHd))
(2) In Transformers multiple layers of MHSA may be in use, where the output of the (1 1 1)th MHSA layer is further processed by a non-linear function G parameterised by an MI.P obtain the input to the Ith MHSA layer, i.e., s' = Go (F'-1). See Figure la for an illustration of an MHSA block in Transformer model (excluding the combination projection step of eq. ()).
F = concat F1, Fn)
2.2 SPARSE VARIATIONAL GAUSSIAN PROCESS (SVGP) WITH DEEP KERNEL
Gaussian process (GP) Rassussen Williams 2006) a distribution over function f with infinite-dimensionall index set X (domain of f). In Bayesian inference framework, GP prior
2

---

Published as a conference paper at ICLR 2023
is specified with mean function (often set to zero) and covariance function parameterized by kernel function Ke(, (-..) (with hyperparameters ч)). Specifically, the marginal distribution of function
values f evaluated on any finite number of inputs X [=1, xNn] 2n E X' is Gaussian:
Prior: p(f) ~ GP(O, ko(())) =
x(fx))= = (0, Kxx), [Kx]a, = (a; mz)) (3)
Given training data (X, y), with a Gaussian likelihood p(ylf) (f. I), the posterior process is also a GP, and the posterior predictive distribution of I evaluated at the test inputs X is:
X. 4) = XX
+ to" Kx-x- Xx-x(Xxx + "xxx)) (4)
Unfortunately, with non-Gaussian likelihoods (e.g.. for classification) or when the number of training datapoints N is large, the posterior process is intractable. Still we can approximate the posterior process with a GP, and a popular approach is sparse variational Gaussian process (SVGP) (Titsias 2009 Hensman et al. 2013), which
small number of M inducing points (Z, u) = [= Um) mm=1 to summarise the training data and, to some degree, replaces the terms involving X, y in eq. with the inducing points.
A detailed introduction of SVGP is provided in Appendix B.I in short it utilises the property of GP to augment the prior p(f, u X, Z). which is Gaussian with zero mean and covariance matrix as a kernel matrix computed on [X, Z]. and define the approximate posterior process as:
p u. Z, X X, y) x (lll))((f"" f/u, Z, x <f( f, Z, X X) p(f" flu, Z,X", X ()(), g(u) (mm,, S.).
(5)
Notice that the exact posterior and the approximate posterior share the conditional distribution p(f", flu, ,, This simplifies the evidence lower-bound (ELBO) objective for optimis- ing the variational parameters mu, S. and the kernel hyperparameters &i to
= LELBO = Ex (o)ggxx))/oog(()))
K L(g(() (p)()))(((Zz))
(6)
Since q(u) and p(u Z) are both Gaussian, the second term can be evaluated analytically. For non- Gaussian likelihoods, we resort
Carto estimation for computing the first term. In predic- tion, the approximate posterior predictive distribution of f evaluated on test inputs X becomes:
[ plf flu, Z, X", X))((()))))) X)q(u)dudf
qf 2)
=N(Kx x- z K K ma: Xx + Kx-z
Xzz)KZ2KZx*
z2
Note that the computations of both the ELBO (eq.6)) and the approximate posterior predictive distribution (eq.(7) require matrix inversion of K zz only. Since we usually use a small number of inducing points (M < N), the computational cost of SVGP (O(N + M²) is significantly lower than the O(N3) cost in full GP resulting from the inversion of Kxx (c.f. 44.
One way to take advantage of the expressiveness of DNN in GP is to parameterize the kernel func- tion using DNN, so that the network weights become part of the hyperparameters of a deep kernel Willson et al. 2016). Given regular base kernel, such as RBF kernel (-ase(-,)), we can first map the inputs X to feature space using a DNN, ho( then apply the base kernel to the DNN features corresponding to the inputs: Kdeep(
Kbase (ho h-).
SPARSE GAUSSIAN PROCESS ATTENTION
We propose Sparse Gaussian Process Attention (SGPA) to perform approximate Bayesian inference for Transformer- based models. The key idea is to replace the softmax operation in scaled dot- product attention (Vaswani et al. 2017) with kernel (Tsai al. 2011). and connect the resulting attention to the mean of an SVGP. This insight allows us to apply SVGP equations for uncertainty estimation, and we further introduce decoupled inducing points to improve computational efficiency.
3.1 ATTENTION AS THE MEAN OF A SPARSE VARIATIONAL GAUSSIAN PROCESS
Standard Transformers use attention blocks based on scaled dot fot-product Vaswani et al. 2017
Given queries q. keys l and value v, the scaled dot- dro-prr (SDP) attention is given as follows:
qk
SDP-Attention
F = softmax
V
(8)
3

---

Published as a conference paper at ICLR 2023
where de is the dimension of keys. Since attention involves measuring the similarity between q and k. Tsai et al. (2019) generalised SDP-Attention by replacing softmax gh -dd in eq. with a kernel gram matrix K qt: ( (Kattlss = K (q. k;)) computed using a valid symmetric kernel K(:'), for which we refer to it as kernel attention or K-Attention for short:
K K-Attention:
F= Kg...
Recall the posterior mean of SVGP in 7 is m = K Xxz when evaluated on training inputs (X* = X). Now we reparameterise the variational mean parameter of SVGP as [v],a == K Kz for each dimension (d) of v, and define the queries and keys as the input locations and inducing point locations: q = X, k == Z. By doing so, equivalence can be identified between the posterior mean of an SVGP and each dimension of the output of kernel attention block.This allows us to extend the toolbox of Gaussian processes and their scalable approximations for quantifying uncertainty in Transformers in the following sections.
3.2 STANDARD SGPA & ITS INEFFICIENCY FOR SELF-ATTENTION
Observing the equivalence between K-Attention and SVGP mean, natural idea for uncertainty estimation is to apply SVGP techniques for approximate posterior variance computations. In detail, we introduce a set of variational covariance parameters S E Txxxxd (with T as the number of keys/inducing inputs), and optimise them using the ELBO (eq. ( This procedure returns the mean and covariance for each dimension (d) of the posterior attention output as:
(10) In this way, we fit an SVGP for each dimension of attention outputs independently: for each dimen- sion d, an SVGP given as eq. 4 is fitted using the same kernel, but with different variational mcan (V)).)) and covariance ([S]..d) parameters. We name this approach as standard SGPA and provide
K m. = (g [V],d Σn = Kgg + Kg [S).,aK kk K2k Kkg-
a visualisation of the operations in Figure 1b
Unfortunately, standard SGPA becomes computationally inefficient when applied to Transformers based on multi-head self-attention. In each attention layer the keys for each head h, k², are obtained by passing the output from previous layer through neural network. Moreover, the projection ma- trices for queries and keys need to be tied (i.e., W = W := wh to obtain a valid symmetric kernel (Tsai et al. 2019). As a result, the queries and keys in a self-attention layer are the same, more importantly they are input-dependent i.e., ka s s and they vary the input sequence to the Transformer changes. Therefore, to extend standard SVGP framework (cq. (T)) to self-attention, the covariance parameters 5th need to be input-dependent well to accommodate the varying inducing inputs k.. A naive idea would parameterise Shh by linear projection, e.g., [ s sW for one head where L is the Cholesky factor of Sh (see Figure lb). This will ineur memory cost of O per head even if we tie the variational covariances across output dimensions, and a run-time cost of (TT3) per head per input sequence for inverting KR as k² is input-dependens Therefore standard SGPA is both memory and run-tinn time inefficient especially for long input sequences.
3.3 IMPROVING TIME & MEMORY EFFICIENCIES VIA DECOUPLED SGPA
We propose to address the aforementioned inefficiency issues by extending the orthogonally decou- pled sparse Gaussian process approximation (Salimbeni et al. 2018) to self-attention. In addition to input-dependent (or "amortised") keys/inducing inputs k², which we will call k from now on, for each head h, also incorporate another My number of "global" keys/inducing inputs k that are shared across all input sequences. The main idea is to compute the variance of sparse GP using the global keys only, so that the variational parameters for the Shh matrix become independent the input sequences. Indeed, following the derivations presented in Appendix [B22] we can compute the mean and covariance for each output dimension (d) of each head as (we drop the superscript h here for more concise notation):
m. = KKyh. [Valed - Kgk, S.n.ullaaa.e + Kyh, (aladd
(11)
Σn = Kag + K Kyk, K, ((Sg)azd - KRYM,)) KK,,a, k,k,, K
where V E M, Sy E My Myxdu are the variational parameters associated with the global keys ky. and v. E xxd, is computed via projection v. = sW,. We name this approach as decoupled SPGA which is illustrated in Figure
4

---

Published as a conference paper at ICLR 2023
Compared to standard SGPA (eq. where k. in decoupled SGPA is the same as k in stan- dard SGPA), we see that the posterior mean of decoupled SGPA also involves two extra terms to take into account the effect of global inducing points. But more importantly, the posterior vari- ance of the two SGPA methods differ only in the keys/inducing inputs in use (input-dependent keys hd versus global keys k and this brings in the key advantange of decoupled SGPA. As the posterior covariance in (I) only involves the global inducing points, the variational co- variance no longer needs to be input-dependent, and (the Cholesky factor of) S can be param- gg eterised freely. Now the number of parameters for the covariance part of order of O(M3) (vs O in standard SPGA), and the computation of matrix inversion pays a one-off cost of O(M)) (vs O(T3) for every input sequence). Notice that global inducing points Mg. and in practice we find Mg O ((TTr)) ) is usually sufficient, where Taog is the average length of training input sequences. Table | we summarise time complex- ity (with batch size B) and the additional memory (number of parameters) required for SGPA in one head of a Transformer. We also include maximum likelihood estimation (MLE) for ref- erence (note that memory complexity for MLE does not depend imput sequence length T). As the time and mem- signifi-
free to choose the number of
ory savings cant, we mainly evalu- ate decoupled SGPA in our experiments, and in the rest of the main text we will refer to decoupled SGPA as SGPA for short.
Table I: Complexity comparison for standard and decoupled SGPA.
Model
Time
Additional Memory
MLE
O
Standard SGPA
O(BTa)
2
Decoupled SGPA
M, + MA)
O(M))
3.4 TRANSFORMER BASED ON DECOUPLED SGPA
So far we have presented SGPA methods for uncertainty quantification in multi-head self-attention module. When applied to Transformer models, multiple layers of attention blocks are in use, and in the following we describe the construction of Transformer model based decoupled SGPA. Note that, as SGPA is equivalent to a sparse GP, the Transformer model presented below can be viewed as a sparse approximation to a deep GP (Damianou & Lawrence 2013 Salimbeni & Deisenroth 2017) with deep kernel in each layer.
Our Transformer architecture mostly follows the one in Vaswani et al.
2017 The input to the ith We first process the input
SGPA layer is the output from previous SGPA layer Fi-1 E p Txd
with a non-linear mapping Gøt: : .Rdd' + Rdd,, and then perform projections to obtain the queries, amortised & global keys and values. Specifically, we have for each head h::
g.h = -k-h GF ol
k/gh = Go W qk
v
- = GF
(12)
where Zitt E M2 are global inducing locations of the /th layer defined on the same space as Fl-1, Then we apply a base kernel --ase(:-)) to compute the kernel matrices. This is equivalent to using a deep kernel defined on the space of F!-1, and the parameters of G are viewed as the hyperparameters of the deep kernel. Lastly, with variational parameters " associated with the global inducing locations Z//h, we can obtain m and Σ using equation 11 We then propagate uncertainty to the next layer by generating samples of output for each head using the reparameterization trick as in (Salimheni & Deisenroth] (2017):
[F] m/ 1.h + E/ Ith ed"
Ed ~ (0,
(13)
The final output F' E & Txd of this SGPA layer is obtained by linear combination in the same way as in standard Transformers (see cq.(2)
The ELBO objective for training the variational & kernel parameters is derived following deep GP and additive GP (Duvenaud et al.) 201T) approaches. The key idea is that as each head in MHSA with SGPA is a (sparse) GP, the final output F' can also be viewed GPs, which is again a GP (Duvenaud et al., 2011). This allows us to perform variational approxi- mations on each of the heads before the final combination instead of using direct approximation on the F' process (Sun et al. 2021). Assuming the approximate posterior q for ( { === factorises over h the corresponding EL BO with input sequence F := X is (derivations in Appendix B.3):
weighted summation of (sparse)
!!
5

---

Published as a conference paper at ICLR 2023
LELBO = E ge( r9,0l lo [F')]
H
(14)
E OF (M (KL(qQuunn I. lkg lbgh [lp'))lp(uuuu 1h 1 F F()
In practice, resort Monte-Carlo to estimate LELBO with samples of function values generated iteratively passing through each layer using the reparameterization trick (eq.
4 EXPERIMENTS
We evaluate SGPA on prediction tasks across modalities, with the following experimental set-up.
Datasets: CIFARIO & CIFAR100 (image classification (Krizhevsky et al. 2009), CV tasks); CoLA (linguistic acceptability prediction (Warstadt et al. 2019), NLP task) and IMDB (senti- ment analysis, (Maas al. 2011). NLP task).
Network architectures: We use Vision Transformers (VIT (Dosovitskiy et al. 2021) l)) for CV tasks. For kernel attention we use the exponential kernel (Tsai et al., 2019) and the ARD-RBF kernel (Rasmussen & Williams| 2006) for NLP and CV tasks respectively. Scaled dot-product (SDP) attention based Transformers are also evaluated. As in Tsai et al. (2019), we find kernel attention tends to outperform SDP attention in most tasks considered, thus we do include the results of SDP attention in the main text. These results can be found in the tables in Appendix
Baselines: We compare our approach with the following "single-model" methods: maximum likelihood estimation (MLE), Bayesian inference methods including mean-field variational infer- ence (MFVI, (Blundell al. 2015) Monte-Cart Dropout (MCD, (Gal & Ghahramani (2016), Kronecker-factored last layer Laplace approximation (KFLLLA) (Kristiadi et al., (2020), and Spectral-normalizer Neural Gaussian Process (SNGP) (Liu et al. 2020). For tasks where a val- idation set is used, we also consider temperature scaling (TS) (Go et al.) 2017a) and use the validation set as the calibration set. For CV tasks, we also consider ensemble methods: we com- pare SGPA ensemble (SGPAE) with deep ensemble (DE) (Lakshminarayanan et al.) (017). We don't consider ensemble models in NLP tasks since we use different (ainn(valid)-aee splits in different runs for them.
Evaluations & metrics: We consider three evaluation set-ups: in-distribution performance, out- of-distribution (OOD) robustness and OOD detection. The metrics on test set include predictive accuracy metrics for each task, uncertainty calibration metrics such negative predictive log- likelihood (NLL), expected calibration error (ECE) and maximum calibration error (MCE) (Guo) et al. 2017b We report the two standard errors for each metric obtained from 5 inde- pendent runs. For OOD detection tasks we consider the area under the ROC & precision-recall curves (AUROC & AUPR, respectively), and we report the average ranks in terms of AUROC and AUPR over all of the OOD detection tasks for each method.
For fair comparisons, within each task, all the models are trained using the same architecture and optimisation setting. All the models are trained from scratch without pre-training. We include the experimental details in Appendix E Results in tables also presented in Appendix G
4.1 IN-DISTRIBUTION CALIBRATION
We report the evaluation results for in-distribution test data image classification (CIFARIO & 100, without data augmentation), sentiment analysis (IMDB), and linguistic acceptability (CoLA) tasks in the first, second, third and fourth row of Figure respectivel. Here for the CoLA dataset, predictive accuracy is measured by Matthew correlation coefficient (MCC) (Matthews) 1975) instead of accuracy, as in Warstadt et al. (2019).
All "single-model" calibration methods considered tend to improve the calibration, except for sen- timent analysis, where KFLLLA fails in the sense that it achieves worse calibration even than MLE (although KFLLLA achieves best calibration for linguistic acceptability (CoLA), its performance is unstable across tasks). Although MFVI tends achieve the lowest calibration errors, it severely un- derfits the data in all the experiments. This is undesirable, as improvement in calibration should not come price of noticeable drop in predictive correctness. As counter example, achieve

---

Published as a conference paper at ICLR 2023
MFV1
M03
KFILLA
SHCP
50PA
DE
50PAE
64
472
1.04
KLL
14006
0.33
1.0
13.5
Accuracy acy
-01
MCC
ECE
MELTA
XM0
2911
MOE
R0L
Figure 2: Test set accuracy (or MCC for CoLA) & calibration metrics of Transformers or ViTs trained on CIFARIO (Ist row), CIFARI00 (2nd row), IMDB (3rd row) and CoLA (4th row).
perfect calibration in terms of zero ECE by predicting marginal class probability, but this prediction is useless in practice. For image classification on CIFARI00, KFLLLA achieves lower ECE than SGPA, however, it achieves worse NLL and the worst MCE among all the methods. Overall, SGPA achieves the best performance when compared with the other single-model" baselines: it consis- tently achieves better calibration across all tasks while maintaining competitive (or even better, on IMDB) predictive accuracy. Compared with "single-model" methods, both ensemble methods, DE and SGPAF, achieve much better predictive accuracy. SGPAE noticeably outperforms DE in terms of calibration while maintaining competitive predictive accuracy.
4.2 ROBUST PREDICTION ON OUT-OF-DISTRIBUTION DATA
Next we evaluate the performance of SGPA under distribution shift for both the linguistic accept- ability task (CoLA) and the image classification tasks (CIFAR10& & CIFAR100). The OOD data for CoLA is introduced by the same authors of Warstadt et al. (2019), while for the CIFAR datasets, we use corrupted CIFAR datasets (CIFAR 10-C and CIFARTO0-CI (Hendrycks Dietterich) 2019) as the OOD data, which contains noisy CIFAR images with different types of distortions introduced to their clean counterparts at different skew intensities. Note that we don't consider TS OOD tasks in this and the next subsection since a Frequentist method, it is proposed to calibrate the uncertainty on in-distribution data only.
We report in Figure MCC and calibration metrics for the OOD test on CoLA. The observations similar with the in-distribution test: SGPA outperforms MLE, MCD, and SNGP in terms of NLL and calibration errors while achieving improved accuracy. MFVI and KFLLLA achieve lower
30PA
2.5
20
0.4
14.
BH1
MEP
MALLA
MALLA
M011
MALLA
M41
RCP
SALLA
JWP
MCC
NLL
ECE
MCE
Figure 3: Test set MCC & calibration metrics for OOD test set of Transformers trained on COLA.

---

Published as a conference paper at ICLR 2023
06
SCRME
NPV
M00
KFLLLA
SHCP
SUPA
0.32
1.-
34
Accuracy ve slae intensity
NL. ve akov intenaityy
ECE vs swaw intent ity
sken intensity
Figure 4: Test accuracy calibration metries on CIFARIO-C (top row) and CIFAR100-C (bot- tom row) against skew intensity of corruption for ViTs trained on corresponding clean data.
calibration errors but they achieve worse predictive accuracy than SGPA. In particular, MFVI again underfits the data.
For OOD robustness test on image classification, we compute metries against skew intensity on the corrupted CIFAR datasets and report them in Figure 4 Again the same story: among "single- model" methods, SGPA outperforms MLE, MCD and SNGP terms of calibration without hurting accuracy. MFVI achieves lower calibration errors than SGPA but it pays the price of underfitting especially when the skew intensity small. The performance of KFLLLA seems to be not as sta- ble SGPA. For CIFAR SGPA achieves better calibration than KFLLLA. For CIFAR100 KFLLA achieves the best NLL and ECE but the worst MCE. Ensemble methods again achieve bet- ter accuracy than "single-model" methods and SGPAE still outperforms DE in terms of calibration while achieving similar accuracy.
4.3 OUT-OF-DISTRRBUTION DETECTION
Lastly we consider OOD detection tasks on Transformer models trained for image classifi- cation, further evaluate the quality of uncertainty estimates. Here use predictive en- tropy to score each input (from either in-distribution or OOD data) and make decisions of "in/out" the entropy is smaller/greater than a specified threshold. Using different thresh- olds for this detection task allows to compute both the receiver operator characteristic (ROC) and the precision-recall (PR) curves, and we use the area under the curves, i.c. AU- ROC and AUPR, for performance evaluations. For each of the CIFAR datasets, sider the other CIFAR dataset, SVHN and mini-ImageNNt as OOD datasets so that we con- struct 6 OOD detection tasks in total. For each method, here report its average ranks in terms of AUROC and AUPR 6 tasks in Table 2 Ensemble "single-
Table 2: Average ranks different methods in terms of AU- ROC and AUPR over 6 OOD detection tasks
methods outperform model" methods with SGPAE achieving the best performance. Among "single-mode!" meth- ods, apart from KFLLLA which achieves the best performance, SGPA outperforms all the other baselines. For comparison within each task, Figure 5 and 66 in Appendix A where the values of AUROC and AUPR for all methods are plotted for each task.
Model
avg. rank (AUROC) avg. rank (AUPR).
MLE
6.1667
5.5000
MFVI
7.0000
7.1667
MCD
5.6667
5.6667
KFLLLA
4.3333
4.5000
SNGP
5.3333
5.3333
SGPA
4.5000
4.6667
DE
1.6667
2,0000
SGPAE
1.3333
1.1667
4.4 SUMMARY ADDITIONAL RESULTS
Additional experiments reported in Appendix where summarise the main results as follows.
8

---

Published as a conference paper at ICLR 2023
In appendix C.1 we find that in addition to the parameter inefficiency problem, Transformers based on standard SGPA also suffer from underfitting Compared with decoupled SGPA, stan- dard SGPA achieves significantly worse accuracy CIFARIO classification task.
report results of image classification with data augmentation. While MFVI and SNGP underfit the data, both the accuracy and calibration improve for the other methods. The performance difference between SGPA and other "single- model" baselines (except MFVI and SNGP) becomes smaller as a result of strong regularisations from data augmentation. SG- PAE again achieves the best performance.
In appendix C.2.
report results of graph property regression with ZINC dataset (Dwivedi et al. 2020 For this task, SGPAE achieves the best performance and SGPA outperforms the other Single-model baselines.
In appendix C
5 RELATED WORK
Bayesian Transformers. Tran et al. (2019) and Xue et al. (2021) propose to perform approximate posterior inference using MFVI in weight space for a subset of layers in Transformers. However, in our experiments we find this type of approaches underfits the data. This pathology of underfitting is
theoretically confirmed for weight space MFVI (Foong et al. 2020; Coker et al. (022). Another line of research proposes to perform VI over the attention matrices directly (Fan et al. 2020; Cinquin (ct al. (021). However, Fan et al. (020) only considers finetuning with variational attention, and (Cinquin et al., (021) only considers experiments on synthetic simple datasets with shallow net- works and the variational distribution fitted over the attention weights are shared across data, which might be too restrictive for complex problems. Moreover, they find that Hata-dependen variational distribution over attention weights can even hurt the performance of their approaches. [iu et al. (220)) consider performing Bayesian inference directly over the Transformer output by fitting GP over the last layer output (Bradshaw et al. 2017). This approach can be viewed as using a GP model with a deep kernel defined by the Transformer. Instead, SGPA fits a deep GP so that uncertainty is propagated through each attention layer of the Transformer. In addition. Liu et al. (2020) propose to preserve the distance awareness property for the deep kernel. Note that this restance-preserving trick is orthogonal to ours and can also be easily integrated into SGPA.
Related GP methods. The ELBO of SGPA is similar to that in Sun et al. 2021) which also propose to independently approximate the posterior for each additive component in an additive GP (Duve- naud et al., 2011). The difference is in the kernel design: Sun et al. (2021) aim to decompose a given kernel function into orthogonal "kernel basis", while in SGPA we consider the same type of kernel for each attention head but with different kernel hyperparameters. Our approach is also related to sparse within sparse Gaussian process (SWSGP) (Tran et al.) Jafrasteh et al. 2022) which al- lows adaptive inducing points for each data point (similar to the input-dependent keys k2 in SGPA). This connection between SGPA and SWSGP is further discussed in appendix D
CONCLUSION AND FUTURE WORK
We have proposed SGPA to directly perform approximate Bayesian inference over the output of attention blocks in Transformers. Compared with other baselines, showed Transformers based on SGPA achieve better balance between predictive accuracy and calibration. Furthermore, the im- proved quality of uncertainty estimation provided by SGPA has been proved useful in maintaining robustness under distribution shift and in out of distribution detection. Future work will inves- tigate the following directions. First, masked pre-training (Delvin et al. 2019). which has been proved crucial for downstream tasks for standard Transformers, may also improve the performance of Transformers based on SGPA. In this work, we are not able to consider pretraining due to the high computational cost, and since SGPA replaces scaled dot-product with a valid kernel, there is no existing pre-trained backbone that can be directly used for the downstream fine-tuning tasks. Second, many tasks using Transformers, such as neural machine translation, require autoregressive prediction using an mucoder-decoded based architecture (Vaswani et al., 2017 Brown et al. 2020), and we will adapt SGPA to the decoder as well. Lastly, will investigate the introduction of hidden mapping distance preserving trick (Liu et al,, (020) to SGPA, which has been shown to be useful in regularizing the parameters in deep kernel learning.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We would like to thank Harrison B. Zhu and Zijing Ou at Imperial College London for their valuable comments the manuscript. We also would like to thank the anonymous reviewers: their valuable feedback and helpful discussions during the rebuttal helped improve the paper.
ETHICS STATEMENT
We believe that this work in its current status has minimal ethical implications: this research in- volves no human subjects and no data domain where privecydisermiinatoo/iiinses is concern. However, as with most research in machine learning, new methods could be used on datasets and domains where privcyyliscimnnatonnaannsss is concerning cause negative ethical impacts, but we do not think SGPA is more concerning than any other method in this regard.
REPRODUCIBILITY STATEMENT
Example code can be found at: httt:s://g.ttbb..om///e/e/////// perimental set-up can be found in Appendix
Details for the ex-
REFERENCES
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In International Conference on Machine Learning, 2015.
John Bradshaw, Alexander G de G Matthews, and Zoubin Ghahramani. Adversarial examples, uncertainty, and transfer testing robustness in gaussian process hybrid deep networks. arXiv preprint arXiv: 10XX::1007... 70777007722442 02476, 2017.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv:2005. 14165, 2020.
Ching-An Cheng and Byron Boots. Variational inference for gaussian process models with linear complexity. In Advances in Neural Processing Information Systems, 2017.
Tristan Cinquin, Alexander Immer, Max Horn, and Vincent Fortuin. Pathologies in priors and in- ference for bayesian transformers. In NeurIPS 2021 "I can't believe it's not better" workshop, 2021.
Beau Coker, Wessel P. Bruinsma, David R. Burt, Weiwei Pan, and Finale Doshi-Velez Wide mean- field variational bayesian neural networks ignore the data. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
Andreas C. Damianou and Neil D. Lawrence. Deep gaussian processes. In International Conference on Artificial Intelligence and Statistics, 2013.
Jacob Delvin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, and Sylvain Gelly. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
David K. Duvenaud, Hannes Nickisch, and Carl Rasmussen. Additive gaussian processes. In Advances in Neural Information Processing Systems, 2011.
10

---

Published as a conference paper at ICLR 2023
Vijay Prakash Dwivedi, Chaitanya K Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00962 2020.
Xinjie Fan Fan, Shujian Zhang, Bo Chen, and Mingyuan Zhou. Bayesian attention modules. In Advances Neural Information Processing Systems, 2020.
Andrew Y. K. Foong, David Burt, Yingzhen Li, and Richard Turner. On the expressiveness of ap- proximate inference bayesian neural networks. In Advances Neural Information Processing Systems, 2020.
Yarin Gal. Uncertainty in deep learning. PhD dissertation, University of Cambridge, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation:
Representing model uncertainty in deep learning. In International Conference Machine Learning, 2016.
Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. Hybrid speech recognition with deep bidirectional Istm. In 2013 IEEE workshop automatic speech recognition and understanding, pp. 273-278. IEEE, 2013.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, 2017a.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, 2017b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arXiv:1512.03335 2015.
D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
James Hensman, Neil D. Lawrence, and Nicol'o Fusi. Gaussian processes for big data. In The Conference on Uncertainty Artificial Intelligence, 2013.
Bahram Jafrasteh, Carlos Villacampa-Calva and Daniel Hernandez-LLobtt. Input dependent sparse gaussian processes. In International Conference Machine Learning, 2022.
Jinwoo Kim, Tien Dat Nguyen, Sconwoo Min, Sungjun Cho, Moontae Lee, Honglak Lec, and Seunghoon Hong. Pure transformers powerful graph learners. arXiv, abs/2207.02500 2022. URL
2200 02505
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just bit, fixes overconfidence in relu networks. In International Conference on Machine Learning, 2020.
A. Krizhevsky, V. Nair, and G Hinton. Cifar-10 10 and cifar- 100 datasets. 2009. URL https:
// www.. C8 wtt.ssttoooooooeeeeereeoooo/oeee
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 2017.
Jeremiah Zhe Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax- Reiis, and Balaji Lakshmi- narayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 2020.
A. L. Maas, R. F. Daly, P. T. Pham, A. Y. Huang, D.; Ng, and C. Potts. Learning word vectors for sentiment analysis. NAACL-HLT, 2011.
B. W. Matthews. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2): 442-451, 1975.

---

Published as a conference paper at ICLR 2023
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip HS Torr, and Puneet K Dokania. Calibrating deep neural networks using focal loss. In Advances in Neural Information Processing Systems, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vitito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Conference of the North American Chapter of the Association for Computational Linguistics, 2018.
Carl Edward. Rasmussen and Christopher KI Williams. Gaussian Processes for Machine Learning. 2006. ISBN-13 978-0-262-18253-4
Hippolyt Ritter, Martin Kukla, Cheng Zhang, and Yingzhen Li. Sparse uncertainty representation in deep learning with inducing weights. Advances in Neural Information Processing Systems, 34: 2515-6523 2021.
Hugh Salimbeni and Mare Deisenroth. Doubly stochastic variational inference for deep gaussian processes. In Advances in Neural Information Processing Systems, 2017.
Hugh Salimbeni, Ching-An Cheng, Byron Boots, and Mare Deisenroth. Orthogonally decoupled variational gaussian processes. In Advances in Neural Information Processing Systems, 2018.
Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. In Advances in Neural Information Processing Systems, 2005.
Shengyang Sun, Jiaxin Shi, A. G. Wilson, and Roger Grosse. Scalable variational gaussian processes via harmonic kernel decomposition. International Conference on Machine Learning, 2021.
Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificial Intelligence and Statistics, 2009.
Hugo Touvron, Matthicu Cord, and Herve Jegou. Deit ii:: Revenge of the vit. arXiv preprint arXiv:2204.07118, 2022.
Dustin Tran, Michael W. Dusenberry, Mark van der Wilk, and Danijar Hafner. Bayesian layers: A module for neural network uncertainty. arXiv.1812.03973. 2019.
Gia-Lac Tran, Dimitrios Milios, Pietro Michiardi, and Maurizio Filippone. Sparse within sparse gaussian processes using neighbor information. In International Conference on Machine Learning, 2021.
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis Phils-ph Morency, and Ruslan Salakhutdinov. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In EMNLP, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
A. Warstadt, A. Singh, and S. R. Bowman. Neural network acceptability judgments. In Transactions of the Association for Computational Linguistics, 2019.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning- In International Conference Artificial Intelligence and Statisties, 2016.
James T. Wilson, Viacheslav Borovitskiy, Alexander Terenin, Peter Mostowsky, and Mare Peter Deisenroth. Efficiently sampling functions from gaussian process posteriors. International Conference Machine Learning, 2020.
12

---

Published as a conference paper at ICLR 2023
Boyang Xue, Jianwei Yu, Junhao Xu, Shansong Liu, Shoukang Hu, Zi Ye, Mengzhe Geng, Xun- ying Liu, and Helen Meng. Bayesian transformer language models for speech recognition. arXiv:2102.04754, 2021.
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning. In International Conference on Learning Representations, 2019.
13

---

Published as a conference paper at ICLR 2023
A
AUROC AND AUPR FOR OUT-OF-DISTRIBUTION DETECTION TASKS
VLE
WFVI
VCD
KFLLLA
SN3P
S0PA
DE
SOPAE
7.6
790
0.78
AUROC
0.74
0.76
2.675
M.E
8PV1
MG3
KPLLLA
SHOP
SEP
DE
83MC
M-11
KPLLLA
0M27
50H
DE
53ME
MV1
KPLLA
SKCP
0.72
D. 945
AUPR
0.70
940
916
M.E
RP1)
MOP KPILLA SREP SEPA CIFAR10
16.
50Mc
RLE
MS GPLLLA
1927
500A
08 524E
0.E
WV1 PG PILLA AN2
691 DE
GIFAR100
CIFARI0 SVHN
CIFAR10
MINI IMAGENET
Figure 5: AUROC (top) and AUPR (bottom) for OOD detection using ViTs trained CIFARIO.
SGPAE
0.70
0.6
AUROC
0.70
0.66
0.00
KALLL
PG
8.0
0.54
0.92
AUPR
775
790
56
CI CIF ORTO
CIFAR100 V
CIFAR100
WINI IMAGENET
Figure AUROC (top) and AUPR (bottom) for OOD detection using ViTs trained on CIFAR100.
B DERIVATIONS
B.1 ELBO DERIVATIONS FOR SVGP
Here we review the derivation of ELBO for standard SVGP (Titsias 2009
Hensman et al. 2013
With M inducing points pairs ((zm; Um m= the prior distribution [f, u
Kxx Kzx
Kxz
p(f, u X, 2) = N(O,,
(15)
Kzz
With prior conditional matching assumption (see eq. 55). the approximate posterior conditional dis- tribution of function values f for training inputs X given inducing points u is the same as the prior conditional distribution:
qffuu, X) p(f (u, Z, X).
(16)
14

---

Published as a conference paper at ICLR 2023
Under prior conditional matching assumption, "if,u/Z,X p(flu. Z, (.)(). where q(u)
Nmm,, Su). Suppose the observation likelihood is plylf), ELBO can simplified as follows:
ply. f. u 2 x
LELBO z,
{(I u Z, X)
ply f (p(ffu 2, X Xjp(ulZ)
p( Z, X (q((u)
= p(flu, =/ Z, Low loggly)))ff + (( log p(ulZ) g(u) / Zx
= 20011xx22 -Lig(llllii)
(17)
Here (f|x, = f p( Z, X (uu)dd is Gaussian and is given as:
(f Z) NN(Kx2 Kxx Kxz X2z)K72K2x) (18)
With Gaussian likelihood, the first term in ELBO can be evaluated analytically. Otherwise we estimate the first term using Monte-Carlo samples divergence between two Gaussian distributions. Thus, it admits closed form:
q(f|x, Z). The second term is KL-
- - TT( + my K zz m log
Kzzz SS
K [L(g())ll((u2))
M).
(19)
In standard SGPA the ELBO objective remains almost the same, except that as the variational mean reparameterised K mzz the mean of xf Z) becomes K kxzv, and the quadratic term in (L(g(u)) p Z)) becomes T Kzzv.
B.2 ORTHOGONALLY DECOUPLED SVGP
The orthogonally decoupled SVGP (Salimbeni et al. 2018) can be interpreted as SVGP (Titsias) 2009) with a structured variational distribution the inducing points. Two sets inducing points are in use: {(ze (1² Ua
(m²) [mm)] ) and [(z) (1m) ug (mm) M, Consider a structured Gaussian variational
distribution over = Ualg ua Ug. with variational mean and covariance given follows:
K K2,2, K- Z, mg + ma
mu
mg
(20)
K Kz. K z. B
K 1
2, 2 (S, K2,z,
2,2, Kz,z K K
2, K2.Z,KZ,, Sq)
S,
S, K Z, Z.
Kz,z. Z.
S
Plugging the above and S. in eq. eq.18 18 can obtain the posterior distribution of orthogo- nally decoupled SVGP for ff after canceling some terms, which is a Gaussian with mean my and covariance Egg given
m, (Kxz. kxz, K2, K2, (K.2, Kz. K K2 2, Kz,2,) 'mm + Kx 2, K 2 mg
Σ, = Kxx + Kx2, K 2 (S, Kz, 1 2,2, K 22
(21)
1 we further reparameterize (K..2. kz,2, 2, 2, K Z., - mm as Va and K-1 2,2, my as Vg. then we arrive at the final expressions used decoupled SGPA (see eq.
K Kxz, yg: Eイ Kxx Kxz, K 2, (S, K z, K Z, 2,
m Kxz, Kxz, Kn
(22)
K Kz,x.
B.3 ELBO FOR TRANSFORMERS BASED ON SGPA
An L-layer Transformer based on SGPA a deep GP (Damianou & Lawrence 2013) and we train it using the doubly stochastic variational inference framework (Salimbeni & Deisenroth 2017). For
15

---

Published as a conference paper at ICLR 2023
each imput sequence FD X, the joint distribution for Y (F
M==1 (Wolig 1=1,k=1 is:
lif L.H
p(Y ((Y,(F') Im a 11=1 1
th L.H
(23)
p PYYF F p
u aLigh alighhi F' p((1))(((utd pllllaag Ih-1
1 1h-1:
(khha F²
1=1
where ( 1H [Kg 1 II pl aug Ik-. F since we assume the prior for h==1 inducing points factorizes across heads in each layer. Note here the amortised keys kith depend on F1-1 in deterministic manner, therefore we drop the amortised key terms in the conditioning.
Assuming prior conditional matching (i.e, q(F') h== F'-1) p(F [u 1 F²-1)),
I.H the joint approximate posterior for {F)) (1aug]1=1.h=1 is:
L.H ((FF) ( (La FF)
II Huubg M F' boltubugg
1. ME F'-1),
(24)
where (((u 1H (kL.) 1
kg Jhm F² II (( whth Fl since we let the approximate
1.h 1 distribution for {MaLg h=1 also factorises across heads.
The ELBO derived in a similar manner as the single-layer GP case (eq. and again the condi- tional distribution terms in q and p cancel with each other. This simplifies the ELBO
LELBO E 1(((F') (((F') (ulb Mar.i (Fo) log PYY: . (E L.N I, FF))
Waughi==1hs
((FF M= (Waug)i=1,te 1= FO))
LI
E y(Fr FO, (r. |oggl( FT)
L.
H
E O 1o (1'
(KLgguuang F1- Mll in
F(-1))]
1=1 h=1
(25)
where
((F'FF (hihh
) )
IIF I j=1
1H 1h Fi- (i--))(((un j.h 1 (((uuu Thm Tamilhk Thm1: Fu-l)dubug 1411: dFL:l11
(26)
Both terms in the ELBO can be estimated using samples generated iteratively through each layer using the reparameterization trick. For the second "regularisation" term, the KL-divergence within the expectation admits simplified form independent decoupled SVGP is fitted:
assume for each attention output dimension (d), an
K LL(guu
J.h. K. F (l-")ll((u L.h I F F'-1))
2 Σ Σ
([v I
l. (K Kea K ki Mii
K-1 with with K
[ Ky lv Ii.d [vl IT
k/ wia
K. Rg ks
d=1
+ Tr((St J...
K b/gh whhngh)) log ([St)))).d al + log K k MeA.g MgJ.
(27)
where the total number of attention output dimensions and M, is the number global inducing points for each head.
16

---

Published as a conference paper at ICLR 2023
C ADDITIONAL EXPERIMENTS
C.I EMPIRICAL COMPARISON BETWEEN STANDARD SGPA AND DECOUPLED SGPA
In our preliminary experiments, we compare performance of ViT based on standard SGPA versus decoupled SGPA for image classification on CIFARIO without data augmentation. We also consider decoupled SGPA based on Cheng Boots (017). There no difference in the expressiveness of the basis functions between decoupled SGPA based on Cheng & Boots (2017) and decoupled SGPA based on [Salimbeni et al. 2018 (the version shown in the main text). However, Salimbeni et al. (0188) tends to demonstrate faster convergence due to the orthogonal decomposition of the basis in the mean function. The posterior mean and covariance formula for decoupled SGPA based on Cheng & Boots (0017) is given as follows:
ma = Kkk. [Validd
(28)
E = Kag + K Kak, Ke ((Sglazdd Knyh, Kky,,, K kgk, Kk,9
Table 3 shows VIT based on standard SGPA and decoupled SGPA based on Cheng & Boots (2017)
achieve worse performance than decouple SGPA based on Salimbeni et (2018). In particular, standard SGPA considerably underfits the data. Therefore we only consider decoupled SGPA based on Salimbeni al. (2018) for the rest of the experiments.
Table Test set accuracy and NLL of ViTs based on standard and two variants of decoupled SGPA, trained on CIFARIO without data augmentation.
Model
Accuracy
NLL
Standard SGPA
0.6435=0.0009 0.7513++0.004 0.7787=0.0024
1.0159=000003 0.7877+000005 0.6968+0000022
Decoupled SGPA (Cheng & Boots 2017)
Decoupled SGPA
(Salimbeni et al.
2018)
C.2 IMAGE CLASSIFICATION WITH DATA AUGMENTATION
For ViTs trained on CIFARIO and CIFART00 with data augmentation, we report results of in- distribution calibration and some "single-model" methods can achieve lower ECE or MCE than DE and SGPAE in some cases, DE and SGPAE consistently outperform them in terms of accuracy and NLL. SGPAE again achieves the best overall performance. Among "single- methoos,, MFVI still underfits the data, but for the other methods, data augmentation improves the model performance with SNGP achieving rel- atively low accuracy. The difference between SGPA and other "single-model" baselines becomes smaller, perhaps due to the strong regularisation from data augmentation. Still, SGPA performs more robustly as it generally returns smaller error bars when compared to TS, MCD, and KFLLLA.
robustness in Figure 7 and respectively. Although
In table 4 we report for each method its average ranks in terms of AUROC and AUPR over 6 OOD detection tasks. Ensemble methods again outperform "single-modes methods with SGPAE achiev- ing the best performance. Among "single-model" methods, SGPA achieves the best performance terms of AUROC while KFLLLA achieves the best in terms of AUPR. In Figure 9 and 10 in Appendix A we further plot the values of AUROC and AUPR for all methods within each task.
C.3 GRAPH PROPERTY REGRESSION WITH ZINC DATASET
For graph property regression, assume Laplace likelihood with trainable scale parameter b (i.e., the density of the obaservation likelihood is g(vlf) 25 exp 1op--ll-a ls-/1 -app-ll111, where f is the scalar function value output by the Transformer). We compute mean-absooute-error (MAE), root-mean- square error (RMSE) and negative-log-llkeliood (NLL) to evaluate the models, with results pre- sented in Figure 11 Moreover, we use predictive variances as scores and evaluate OOD detection performance in Figure 12 Note that MLE is useless for OOD detection in this case since it produces homogeneous predictive variances for all instances. We use a synthetic OOD dataset generated from test set: for each test instance, we remove the existing edges from the adjacency matrix and add edges between nodes that are not originally connected. Within "single-model" methods, SGPA and
17

---

Published as a conference paper at ICLR 2023
Table 4: Average ranks of different methods (trained with data augmentation) terms of AUROC and AUPR over OOD detection tasks
Model
avg. rank (AUROC)
avg. rank (AUPR)
MLE
6.1667
6.0000
MFVI
8.0000
8.0000
MCD
4.1667
4.5000
KFLLLA
4.3333
4,0000
SNGP
6.3333
6.3333
SGPA
4.0000
4.1667
DE
1.8333
1.8333
SGPAE
1.1667
1.1667
MCD achieve much better results than MLE and MFVI. For this task, the difference in performance between SGPA and MCD is negligible. However, when compared to MCD, SGPA performs more robustly as returns smaller error hars. Ensemble methods outperform "single-model" methods in OOD detection with SGPAE achieving the best result. Interestingly, for in-distribution calibration, they achieve worse performance than SGPA and MCE.
D LIMITATION AND CONNECTION WITH SPARSE WITHIN SPARSE GP
Unlike standard sparse GPs where the inducing points are shared across all inputs, SGPA consists of set of input-dependent (or "amortised") inducing locations {k(,) and the corresponding vari- ational parameters, which means we are using a different mean function for each input sequence.
WV1
MD2
KFLLLA
SHGP
SIPA
DC
SGPAL
0.8
DIFARTO
0.7
000
0.08
VAL
SULA
0.003
0.40
0.6
8.55
0.100
0.0
400
LMD
47. EMIP
Acour ady
Figure 7: Test set accuracy and calibration metries of ViTs trained CIFAR10 (top row) and CIFAR100 (bottom row) with data augmentation.
2.75
0.31
6.31
4.0
0.4
1221
ity
Intensity
Figure 8: Accuracy and calibration metrics on CIFARIO-C (top row) and CIFAR100-C (bottom row) for ViTs trained corresponding clean data with data augmentation.
18

---

Published as a conference paper at ICLR 2023
VLE
WFVI
MCD
KFLLLA
SNGP
SGPA
DE
SGPAE
00
85
AUROC
75
0. 820
70
0. 775
KO
BYLLL SHCP
527AC
MDD KILLLA
LILLA
92A
0.80
0.57
0.990
0.925
0.90
AUPR
3.670
0.50
0.89
M.E 8P1)
MJO KALLL
SHOP
MSD KELULA
0.90
MED KRLLAA 001A
CIFAR10 10
CIFART00
GIFARIO - SVHN
CIFARTO - MINI IMAGNET
Figure 9: AUROC (top) and AUPR (bottom) metrics for OOD detection using ViTs trained on CIFARIO with data augmentation.
VLE
WFVI
MCD
KFLLLA
SNGP
SGPA
DE
SGPAE
0.50
0.70
0.78
AUROIC
0.76
0.77
25
K.E
8F11
MO
KPLLLA
SHP
SEPA
DE. 53he
1L2
8/11
MLD
KPLLLA
S907
50FA
DE
5242
0.8
WV1
PG
RILLA
OKP
9.65
AUPR
67
0,9
58
K.E
RF11
MOD 4F
SREP MEFA
DE
507cc
RLE
W1V1
50FA FIRAR0 3VHN
MLD LILLLA
8907
02 5244
0.E
WT11
MGB
LILLA OKP
691A
0
CIFAR100
GIF AR10
0IFAR000
MINI IMAGENET
Figure 10: AUROC (top) and AUPR (bottom) metrics for OOD detection using ViTs trained CIFARIO0 with data augmentation.
Consequently, SGPA based Transformers can not perfectly model the correlation between input se- quences. Instead, they can only provide marginal uncertainty for each input sequence. Nevertheless, empirically we found that correlation might not be critical in applications such as text or image classification.
One way to explain SGPA is to consider it as sparse-within-sparse Gaussian process (SWSGP) (Tran et al., 202T Jafrasteh al. 2022). which allows adaptive inducing points for each input. Suppase the index set (in our case the embedding space) is and p(Z) is distribution over M-element subset of x (ic. each random draw will give us M inducing locations from X). The joint prior and approximate posterior become pff, u, Z) = p(f]u))(( Z) Z), and q(u, Z) (uu(Z)) (p(2))p(2))) Z), respectively. In Tran et al. (2021), for each input æ. they propose to select its M- nearest neighbours taken from the training inputs as inducing locations, so that g( 2(Z) is a delta distribution conditioned on æ, and q(u/ Z) is the marginal variational distribution over function values evaluated at the selected inducing locations. In contrast, for each input sequence æ, in layer I, the inducing locations used by Transformer based on SGPA consist both input-dependent ones (fkt." 1/1 Th=1). which are obtained from using neural network as in Jafrasteh et al. (2022). and global inducing locations (kl 1h-1
( which are shared across all input sequences.
inducing points used during test time may not be encountered in training. Instead, we rely the learned neural network to amortise them (Jafrasteh et al.) (022). Therefore the fitted mean function may not be consistent when test sequence includes tokens far away from the tokens in training sequences (as v. may not be consistent). Empirically, we found
Note that the
19

---

Published as a conference paper at ICLR 2023
NLE
SGPAE
60
55
1.0
1.1
50
0.9
1.0
0.45 45
0.8
0.9
0. 40
SGPA
DE
SGPAE
MAE +
RMSE +
NLL
Figure 11: Test set regression error and NL. metrics for Transformers trained on ZINC.
SGPA
SGPAE
1. 0
0. 7
0.
0. 6
0.55
0. 4
0.
MFVI
MCD
SGPA
DE
SGPAE
MFVI
MCD
SGPA
DE
SGPAE
AUROC f
AUPR
Figure 12: AUROC and AUPR metrics for OOD detection using Transformers trained on ZINC.
this inconsistency issue to be minor for in-distribution test sequences. Furthermore, argue that for OOD inputs, although the fitted posterior mean might be unreliable, the uncertainty still increases since the posterior covariance is fully determined by the global inducing points which exhibits inconsistency issue. Intuitively, the global keys // ===,,==1 shared across all input sequences,
I.M
play similar role the inducing locations in standard SVGP: they summarise the training set but focus on the uncertainty behaviour only. As result, the posterior variance in eq. eq.(1) (TT) still increases for queries that less similar to the global keys measured by the kernel. By propagating the uncertainty through each layer, we can still obtain increased uncertainty for input sequences that very different from the training data, that users still be notified "when the model does not know" (Gal 2016). However, unlike standard kernels such as RBF, it's less obvious how deep kernel measures similarity and some advantages Bayesian inference still lose as we have no idea what prior (or inductive bias) the deep kemne represents. Therefore, in our future work, we will investigating how to inject meaningful inductive bias into deep kernels. For datasets where Euclidean distance is meaningful,
adopt the distance-preserving techniques in SNGP (Liu) ctt al. 2020) to enforce inductive bias into deep kernels. However, for other types of datasets, it still remains an interesting open question that we are keen to explore.
E EXPERIMENTAL DETAILS
Training settings shared across experiments. For MLE and MCD, we initially considered both dropout rates 0.1 and 0.2, but we found models trained with dropout rate 0. consistently outper- formed models trained with dropout rate 0.2 in terms of accuracy in preliminary experiments for image classification. Therefore, we decided to use dropout rate 0.1 for methods except MFVI. For each layer,
mean-pooling strategy. The non-lineat mapping GJi at each attention layer is pa- rameterised by a 2-layer MLP as in Vaswani et al. (2017). For models with kernel- based attentions, exponential kernel for sentiment analysis and linguistic acceptability, (Tsai al. 2019):
20

---

Published as a conference paper at ICLR 2023
k(x,, x²) 03
03 expl exp( m and we use ARD-RBF kernel (Rasmussen & Williams 2006 for
of
image classification and graph property regression: k(ar, x²) of exp( - Σド= (0--2))
where
D is the dimension of x and x', of is the output variance, and o, is the length-scale for the j-th dimension. For MFVI, MCD, SNGP and SGPA, predictive uncertainty is estimated using 10 Monte Carlo samples.
Initialization:
we train all our models from scratch (i.e. all parameters are randomly initialized without any pretraining). Apart from the global inducing points parameters, the rest of the pa- rameters are the same as in standard transformers. and are initialized via the default method of the dcep learning platform (we use Pytorch (Paszke et al. 2019)). Each dimension of global induc- ing locations and the global variational mean are randomly initialized with standard Gaussian. For the Cholesky factor used to parameterize the global variational covariance, we randomly initialize each element in the lower triangular part and each element in the log-diagonal with standard Gaussian.
Optimization: all the models are trained using ADAM optimiser (Kingma & Ba 2015). 5), and for each input sequence in a batch, we only draw one sample to estimate the HI BLBO (eq. 14). In our experiments, observe optimization issue: the ELBO is consistently minimized during the training without significant spikes.
Sentiment analysis with IMDB (Maas et al. 2011). We consider 5 different splits, each includes 35,000 training, 5,000 validation, and 10,000 test instances. The maximum number of tokens in each input sequence is 512. Architecture-wise we use a Transformer with I MHSA layer and 8 attention heads, same embedding dimension and hidden dimension of 128. For SGPA we use 50 global inducing points for each head. We train all the models (except post-hoc methods, TS and KFLLLA) for 20 epochs with batch-size 32 and with a initial learning rate 0.001 which decays linearly 0.0001. The best model is selected based on the validation accuracy computed in every training epoch.
Linguistic acceptability with CoLA (Warstadt et al. 2019). The 516 OOD samples provided by the original dataset are used to assess the models' OOD robustness. Within each of the 5 independent runs, the remaining 9,078 in-distribution samples are randomly split into 7,262 training and 1,816 in-distribution test instances. We use a Transformer with 2 MHSA layers, each with 4 attention heads, an embedding dimension of 128 and hidden dimension of 256. For the input embeddings, we use EL.M-style representation (Peters et al. 2018). For SGPA we use 5 global inducing points for each head. We train all models (except post-hoc method KFIILIA) for 50 epochs with batch-size 32 and with a initial learning rate 0.0005 which decays linearly to 0.00001 and we use the model from the final epoch for evaluation.
Image classification with CIFARIO and CIFARI00 (Krizhevsky et al. 2009). For both CIFARIO and CIFAR 100 datasets, we randomly split the original training set into 4,5000 training and 5,000 validation instances, and test on the original 10,000 test instances. The input images are tokenised with a patch size of 4 x 4. For CIFAR1O without data augmentation, we use a VIT (Dosovitskiy let al. (021) with 5 MHSA layers, each with 4 attention heads and a hidden dimension of 128. For all the other experiments, we use a ViT with 6 layers 4 attention heads, a hidden dimension of 256. We train all models (except post-hoc methods, TS and KFLLLA) except SGPA for 600 epochs with batch-size 100 and with initial learning rate of 0,0005 which decays linearly to 0. 0.00001. For SGPA, we use 32 global inducing points for each head, and we use the parameters from the 100th epoch of MLE to initialize the deep kernel hyperparameters, and continue training for 500 epochs. The best model is selected based on the validation accuracy computed every 10 epochs. For experiments with data augmentation, we consider the same data augmentation strategy (3-Augment) as in Touvron et al. (022)).
Graph property regression with ZINC (Dwivedi et al., 2020). The results are presented in which are averaged from 3 independent runs. We use the same split as in (Dwivedi et al. 2020). resulting in 10,000 training, 1,000 validation, and 1,000 test instances. Instead of applying graph- specific modifications to the network architecture, we use the feature engineering technique pro- posed in Kim et al.) (2022) to transform each graph into sequence of input embeddings. We consider Transformers with 8 layer and 8 attention heads, same embedding dimension and hidden dimension of 80. For SGPA use 10 global inducing points for each head. We train all models for 500 epochs
21

---

Published as a conference paper at ICLR 2023
with batch-size 64 and with an initial learning rate 0.0004 which decays linearly to 0.000002. The best model selected based on the validation accuracy computed at the end of every 10 epochs.
F RUNNING TIME
We analyse the wall-clock training and inference time of SGPA here. In Table 5 we present the computational time for single batch the inference stage with 10 Monte Carlo samples for CoLA (batch size = 227) and CIFAR10 (batch size = 200) (results obtained using a single Nvidia GTX 2080 Ti GPU card). For SGPA, first pay one-off cost of inverting the kernel matrices related to global inducing points k Once this is done, we treat them as constant matrices and plug them in eq. eq.I1). When generating samples that are passed to the next layer, we diagonalize the covariance (Ex) eq. 11 to avoid the costly computation of the Cholesky factor of E
The computational cost depends on the number of global inducing points used. For CoLA, we only used 5 global inducing points for each head, and the relative difference between inference times for MCD and SGPA is less than that for CIFARIO, where we use 32 global inducing points for each head. is noteworthy that we haven't done extensive hyperparameter tuning for the number of global inducing points. It is likely that SGPA can still work well with smaller number of global inducing points. For example, for CIFARIO, we recently trained an SGPA model with 16 global inducing points for each head, and we did not see considerable performance drop (Accuracy: 0.7790, NLL: 0.7259, ECE: 0.0119, MCE: 0.0819). In this case, the inference time can be further reduced from 0.986 to 0.807.
Table 5: The computational time (in s) for a single batch at the inference stage with 10 Monte Carlo samples for CoLA (batch size 227) and CIFARIO (batch size 200) (results obtained using single Nvidia GTX 2080 Ti GPU card).
MCD
MFVI
SGPA
CoLA
1.839
1.882
2.358
CIFARIO (32 k:g)
0.234
0.395
0.986
CIFARIO (16 kg)
0.234
0.395
0.807
In Table we present the training time (in s) of one epoch for SGPA and MLE on CoLA (batch size 32) and CIFAR1O (batch size 100) (results obtained using single Nvidia GTX 2080 GPU card):
Table 6: The training time (in s) of one epoch for SGPA and MLE on CoLA (batch size 32) and CIFAR1O (batch size 100) (results obtained using single Nvidia GTX 2080 Ti GPU card).
MLE
SGPA
CoLA
17.834
20.089
CIFARIO (32 kg)
30.593
138.609
CIFARIO (16 kg)
30.593
109.298
G RESULTS IN TABLES
We present numerical results stannast error) for all experiments in tables.
We show in-distribution results in Tables 7 to
13
We show OOD robustness results Tables to 30
We show OOD detection results in Tables 3 to 35
22

---

Published as a conference paper at ICLR 2023
Table 7: In-distribution performance: sentiment analysis with IMDB
Model
Accuracy
NLL
ECE
MCE
SDP
MLE
18806=00.0066 0.3267++00006
0.0009
MLE
0.8822=0.0019 0.3202:0000136
0.0519=000055
0.1568=0.0273
TS
0.8022=0.0019
0.3202+0.0033
0.0519=0.0055
0.0008=0.0004
Kernel
MFVI
0.8799=0.0022
0.3476++002300
0.0585+00.106
0.1293=0.0271
MCD
0.8017=0.0014
0.2986:10.0070
0.0285+0000088
0.0910=0.0218
KFLLLA
0.8822=0.0019 0.4366:10.0015
0.1905+000003
0.2263+0.0048
SNGP
0.8083=0.0013
0.3499=000000
0.0646=0.0037
0.1141=000106
SGPA
0.8075=0.0004
0.2823+0.0027
0.0215#0.0034
0.0647=0.0105
Table 8: In-distribution performance; linguistic acceptability with CoLA
Model
MCC
NLL
ECE
MCE
SDP
MLE
25.0408:10.222
L.897610.0223
0:265410.0033
0.388710.0230
MLE
26.0722:10:4300 2.0083.00.0322
0.2643
0..002:00.0217
MFVI
21.0864.1 1.0809:0..099
0.2018:0000800
0.266:10.0113
Kernel
MCD
26.439910049999 0.9121#0.0154
0.211310.0049
0.323310.0066
KFLLLA
26.0295=0.4050 0.6023=0.0077
0.0241=0.0044
0.1063#0.0204
SNGP
27.8715+1.2588
1.2075
0.3230+0.0146 0.2025+000066
SGPA
27.2380
0.9070+0.0207
0.2076+0.0082
0.3062+0.0097
Table 9: In-distribution performance: image classification for CIFAR10 without data augmentation
Model
Accuracy
NLL
ECE
MCE
SDP
MLE
0.743910.0015
2.2940H0.0277
0.043710.0004
0.4509
MLE
0.7811# 0.0019
1.3395+0.0035
0.038110.0004
0.372410.0103
TS
0.7811=0.0019 0.825410.0077
0.0230:10.0004
0.249
Kernel
MFVI
0.7202=0.0024 0.7995=#00003
0.0058+t0.0009
0.0664+000064
MCD
0.7910=0.0015 0.7789=0.0063
0.0159:+0.0004
0.1501+0.0055
KFLLLA
0.7052-0.0019 0.7201
0.0151+0.0020
0.1497+0.0217
SNGP
0.0032-0.0025
0.7534
0.0156
0.1443 0.0049
SGPA
0.7087=0.0024
0.6062+0.0032
0.0100+0.0004
0.0825+0.0066
DE
0.8235
0.6934
0.0110
0.0699
SGPAE
0.8172
0.5657
0.0054
0.0579
Table 10: In-distribution performance: image classification for CIFARIO with data augmentation
Model
Accuracy
NLL
ЕCE
MCE
SDP
MLE
0.8898=0.0009
0.6073=0.0330
0.0163:1..005
0.3250=10.0118 18
MLE
0.0042
0.4700
0.006811000006
0.0951
TS
0.8442-0.0057
0.4552+0.0176
0.0040 + 0.0007
0.0532 + 0.0077
Kernel
MFVI
0.6967=0.0022 0.864310.0078
0.00801:0.0013
0.0635-10.0083
MCD
0.8437=0.0074
0..591
0.0024+0.0002
0.0429+000009
KFLLLA
0.8445=0.0057 0.4554+0.0058
0.0032+t0.0004
0.0570++0.0090
SNGP
0.8286=0.0118
0.4012-0.0351
0.00201100001
0.0098
SGPA
0.8475-000048
0.0012-
0.0020+100001
0.0395-10.0037
DE
0.8654
0.4062
0.0034
0.0724
SGPAE
0.8691
0.3885
0.0028
0.0421
23

---

Published as a conference paper at ICLR 2023
Table 11: In-distribution performance: image classification for CIFAR100 without data augmenta-
tion
Model
Accuracy
NLL
ECE
MCE
SDP
MLE
0.4893=0.0022 5.7677#0.0492
0.0086::0.0001
0.0066=000007
MLE
0.5216=0.0100 4.3898+000029
0.0076+0.0002
0.5696=000003
TS
0.5216=0.0100 2.5863+0.0527
0.0057=000001
0,4067=000032
Kernel
MFVI
0.4117=0.0015
2.4195+00.013
0.0025+10.0000
0.0055=0.0057
MCD
0.5041=0.0096 2.7122-+0.0299
0.0045+00.000
0.2651
KFLLLA
0.5084=0.0103 3.2984+0.0210
0.0031#0.0002
0.7690=0.0159
SNGP
0.4715=0.0039 2.8296=10.0338
0.0051
0.3028=000082
SGPA
0.5002=000071
2.5643+000813
0.0042++0.0001
0.2463
DE
0.5815
2.4887
0.0039
0.1381
SGPAE
0.5700
1.9434
0.0020
0.0704
Table 12: In-distribution performance: image classification for CIFARI00 with data augmentation
Model
Accuracy
NLL
ECE
MCE
SDP
MLE
0.5984=0.0024
1.8743+:020322
0.0036:000006
0.2792=000512
MLE
0.6053=0.0048 1.525410.0195
0.0021
0.1578=0.0159
TS
0.6053=0.0048
1.4512::0.0511
0.0007+0.0001
0.0825=0.0109
Kernel
MFVI
0,4496=0.0045 2.1132::0.0105
0,0013:000003
0.1339=0.0056
MCD
0.6149-0.0074 1.4255
0.0007++00001
0.0066
KFLLLA
0.5994=0.0038 0.0005+10.0001 1.0000
0.0474=0.0071
SNGP
0.5645=0.0106 1.5839+00.0416
0.0005+0000000
0.0424=0.0060
SGPA
0.0050
1.4486+0000055
0.0010+10.0000
0.0612
DE
0.6471
1.3023
0.0008
0.0986
SGPEA
0.65
1.2761
0.0007
0.0593
Table 13: In-distribution performance: graph property regression with ZINC dataset
Model
МAE
RMSE
NLL
SDP
MLE
0.573311.000 0084
1.145910.0139
1.1459=0.0139
MILE
4..0344
1.0977+0.0869
1.0977=0.0869
Kernel
MCD
0,438540.0163
0.83694100300
0.9264=0.0317
MFVI
0.5702+10.0 6.066
1.1050+0.0115
1.1683=0.0101
SGPA
0.434110.0012
0.819940.0195
0.9319-0.0242
DE
0.4711
0.8635
1.0949
SGPAE
0.4557
0.8479
1.0118
Table 14: OOD robustness: linguistic acceptability with CoLA dataset
Model
MCC
NLL
ECE
MCE
SDP
MLE
19.4835+2.0234
2.057410.0516
0.29141:00009
0.491310.0418
MIE
18.6467+1.6116
2.3565+0.0974
0.0009
0.3864+0.0303
MFVI
16.6363+11.5793
1.134910.0445
0.2184:10.0035
0.0085+0.0151
Kernel
MCD
20.4774+1.4727 1.011410.0234
0.2370-000077
0.3439+0.0177
KFLLLA
18.6250+1.6333 0.6299:000006
0.0361+0.0065
0.0876+0.0065
SNGP
23.6410+11.5986 1.3070+4.0412
0.2537+0.0049
0.3828
SGPA
22.9190
0.951410.0261
0.2257-0.0069
0.0099
24

---

Published as a conference paper at ICLR 2023
Table 15: Accuracy of CIFARIO-C for ViT's trained clean data without data augmentation.
Skew Intensity
Model
2
5
SDP
MLE
0.7120+000011
0.6004+000001
0.6359-0.0014
0.5883++0.0017
0..054+0.0015
MLE
0.725410.0019 0.6632::0.0023
0.6029=000023
0.549410.0008
0.4863-10.003
MFVI
0.6606+0.0093 0.6109=0.0083
0.5730=000077
0.5266++0.0033
0.4778:+0.006
Kernel
MCD
0.7327+-00013 0.6717+0.0015
0.6229=0000116
0.5603++0000019
0,4982++0.0022
KFLLLA
0.7186+0.0014 0.6580+0.0035
0.6092++0.0088
0.5462++0.0022
0.4825+00000
SNGP
0.7281+000004 0.6676+0.0025
0.6208+0.0002)
0.5622++0.00200
0.5006++0.001
SGPA
0.7175+0000388
0.6059+0.0002
0.6067+0.0040
0.5466+0.0033
0.4832:+0.0046
DE
0,7700
0,7089
0.6585
0.5908
0,5246
SGPAE
0.7516
0.6911
0.6401
0.5779
0.5173
Table 16: Accuracy of CIFARIO-C for ViTs trained clean data with data augmentation.
Skew Intensity
Model
2
4
5
SDP
MIE
0.8516=0.0010
0.81.3:=0.0011
0.7752=000010
0.7165:50.0013
0.6500==000012
MLE
0.794410.0061 0,749910.0074
0.709011.0082.
0.6465:10.0085
0.58071:0.000 0094
MFVI
0.6005110.0016 0.6222-10.022
0.587511.0022
0.537710.0032
0.4913 0000
Kernel
MCD
0.7939110.0068
0.7480+10.0082
0.7052110.005
0.6415:10.0992
0.075110.0097
KFLLLA
0.7949+10.0030
0.7508+10.0037
0.7100
0.6480+:0.0043
0.582110.006
SNGP
0.7783+0.0062
0.7336+000069
0.6932:10.0069
0.6314+:0.0062
0.5710=00.007
SGPA
0.791010.0027
0.742610.0021
0.6946110.0075
0.635610.0033
0.5002-10.0000
DE
0.8237
0.7819
0.7428
0.6795
0,6130
SGPAE
0,8105
0.7646
0.7220
0.6581
0.5914
Table 17: Accuracy of CIFAR100-C for ViTs trained on clean data without data augmentation.
Skew Intensity
Model
2
5
SDP
MILE
0.3947+-00022
0.3484+00.0018
0..013
0.2805++0.0013
0.2402++00000
MLE
0.445610.0075 0.3769100005S
0.337310.0049
0.283510.0034
0.233910..00 0029
MFVI
0.338010.0020 0.2832-10.0022
0.2540=10.0018
0.2155+00.0014
0.1803-10.0015
Kernel
MCD
0.452610.0074 0.3840::0.006
0.3446:0.0052
0.2913-40.0037
0.2403
KFLLLA
0.4350+1.0070 0.3656++000053
0.3251=0.0045
0.2717++0.0030
0.2229++0.002
SNGP
0.4053++0.0000
0.3455++000014
0.3093=+0.0015
0.2624+t0.0015
0.2203++0.001
SGPA
0.4472+0000050
0.3764+0.0035
0.3371+000027
0.2828++0.002
0.2317+0.00022
DE
0.5054
0.4289
0.3858
0.3234
0.2690
SGPAE
0.4848
0.4104
0.3675
0,3087
0,2539
Table 18: Accuracy of CIFAR100-C for VITs trained clean data with data augmentation.
Skew Intensity
Model
2
5
SDP
MLE
0,538410.0030
0,485410000038
0.446410.0041
0.393010.0037
0.3410=1000099
MLE
0.5327+0000188 0.4700+00.0018
0..067
0.3690+00.0017
0.0134+0.0013
MFVI
0.39911:0.0019 0.35351:0.0011
0.3224+1.0019
0.28361:0.0017
0.24441:0.001 0.0044
Kerne
MCD
0.5397110.0020 0.4775:10.0018
0.4349:10.0015
0.3777+:0.0011
0.3213+0.0010
KFLLLA
0.5199++0.0019
0.4577+0.0018
0.4048=+000017
0.3581+00.004
0.3035+000004
SNGP
0.4979
0.4440+0.00035
0.4042++0.0034
0.3534+000009
0.3066+0.0022
SGPA
0.5189+E0.0054
0.4603+0.0010
0.4181+000013
0.3625++0.0014
0.3061 +0.0014
DE
0.5742
0.5104
0.4649
0.4039
0,3449
SGPAE
0.5592
0.4936
0.4476
0.3908
0.3316
25

---

Published as a conference paper at ICLR 2023
Table 19: NLL of CIFARIO-C for ViTs trained on clean data without data augmentation.
Skew Intensity
Model
2
4
5
SDP
MLE
2.4011000202
3.1620F0.0202
1.0031
4.384310.0233
5.
5.2626-F0.0287
MLE
I790661000179 2.3460:H0.0321
2.8899
3.6899:L0.0544
4.5003=10.0811
MFVI
0.9761=0.0274 1.1345::00067
1.2737=0.0259
1.7005+0.0237
1.7038=+0.0222
Kernel
MCD
1.146610.0099
1.5153+10.016
18892:10.0234
2.4692:L0.0315
3.1086:10.0431
KFLLLA
0.9573+10.0496
1.2092:10.0577
1.435=0.0068
1.7889-10.0377
2.1537:0.1198
SNGP
L00799+0.0066
1.3611
L6$70=1000129
2.0807:1.00136
2.0071
SGPA
0.9602
1.2340:1002113
14972-10.0283
L8951
2.3812-40.0586
DE
0.9317
1.2424
1.5716
2.0911
2.6373
SGPAE
0.7816
0.9985
1.2125
1.5365
1,9399
Table 20: NLL of CIFARIO-C for ViTs trained on clean data with data augmentation.
Skew Intensity
Model
2
5
SDP
MLE
0.830610.04222
1.0712+0.0559
1.065550.0753
L87955:0.100
2.5810::0.157
MLE
0.636410.0140 0.7960
0.961410.0296
1.2404E0.0416
1.88181000538
MFVI
0.9616=40.044 1.0706:+000061
1.1802=0.0098
1.3612++0.010
1.5629=40.0248
Kernel
MCD
0.6067++0.0180 0.7579++0.0236
0.91444+0.0304
1.1707+00.063
1,4977+000420
KFLLLA
0.6104+000080 0.7565+00.010
0.9044++0.014 44
0.1496+0.0183
1,4487+000225
SNGP
0.6439-t0.0187 0.7874+000011
0.9304=+0.0225
1.17734.00023
1.4671 +225
SGPA
0.620410.0055
0.7739:10.00$2
0,9298=0000047
1.1873+40.0118
1,5058 0226
DE
0.5641
0,7028
0.844
1.0737
1,3595
SGPAE 0.5230
0.6503
0.7814
1.0027
1.2835
Table 21: NLL of CIFAR100-C for ViTs trained on clean data without data augmentation.
Skew Intensity 3
Model MLE
2
4
SDP
7.4649-000045
8.5233+0.0524
9.1964+0.0527
10.4061=0.0555
1..5400=0000-49
MIE
5.5126++0.0578 6.7077+0.0416
7.5503+0.0237
9.0093-000333
10.5647-+0.0660
MFVI
3.0138+0.0132
3.4762+0.0136
3.7921+0.0135
4.3295-000153
4.9042+0.0200
Kernel
MCD
3.8996
4.8412+0.0178
5.5397:10.069
6.7984=000091
8.2454-1 0.099
KFLLLA
3,4366=20.0122
3.5660=000160
3.6457#0.0152
3.7645=0.0135
3.8862++0.01 +0001
SNGP
3.6404-10.0297
4.3067+0.030
4.7994+0.0446
5.6242=000502
6.5029-+0.00
SGPA
3.6960
4.518810.1401
5.1522-10.1510
6.3096-0.1788
7.6306-10.2166
DE
3.1930
4,0004
4.6072
5.7357
6.9765
SGPAE 2.8407
3.5393
4,0466
4,9890
6,0803
Table 22: NLL of CIFARI00-C for ViTs trained on clean data with data augmentation.
Skew Intensity
Model
2
5
SDP
MLE
2.2593+0.2276
2.6313+00.235
1.978610.3144 0.3148
3.5393+0.3720
4. 40.444
MLE
1.969510.0167
2.372140.0260
2.7423-0.0346
3.3555+0.0500
3,981340.0602
MFVI
2.3501+10.0033 0.002440.084
2.7571+0.0112
3.0582-10.0123
3.3805+:0.014
Kerne
MCD
1.833710.0049 2.187410.0073
2.5076:10.0112
3.0263:10.0205
3.61961100073
KFLLLA
1.8645-10.0992 1.2008110.099
2.493511.0009
2.9465-10.0112
3.4418
SNGP
1.916710.0161
2.206710.0162
2.475310.0163
2.91661000033
1..096
SGPA
1.9307-0000075
2.315410.0104
2.0093
3.2105+00.0151
3.8575+L0.0180
DE
1.7354
2.0787
2.3857
2.8802
3.4604
SGPAE
1.6583
1.9839
2.2802
2.7592
3.2976
26

---

Published as a conference paper at ICLR 2023
Table 23: ECE of CIFARIO-C for ViTs trained clean data without data augmentation.
Skew Intensity
Model
2
5
SDP
MLE
0.0091+000002
0.0564+000002
0.0627-0.000
0.0713++000003
0.0811+0.0005
MLE
0.0425:000003 0.0530:+0.0005
0.0617=0.0005
0.07341:000066
0.0846:10.0000
MFVI
0.0077+000017 0.0122:+0.0021
0.0170=000021
0.0243++:00022
0.0316=+0.0021
Kernel
MCD
0.0242:+0.0002 0.0028++00000
0.0405=0.0004
0.0515++000004
0.0621+0.0006
KFLLLA
0.0226+0.0022 0.0305+0.0002
0.0374++0.0028
0.0472++0.0331
0.0566+0.0034
SNGP
0.0233+0.0002 0.0317+0.000
0.0389+0.0003
0.0487+000002
0.0588 +0.0000
SGPA
0.0078+000006
0.0007
0.0330=+0.0088
0.0432:t0.0008
0.0010
DE
0.0160
0,0227
0.0291
0.0389
0.0484
SGPAE
0.0088
0.0136
0.0191
0.0282
0.0368
Table 24: ECE of CIFAR1O-C for ViTs trained clean data with data augmentation.
Skew Intensity 3
Model
2
4
5
SDP
MLE
0.021920.0007
0.0275+000009
0.0337=0.0011
0.0436:=0.0013
0.0551#0.0015
MLE
0.011011.0007
0.0156::0.000 0008
0.0208=0.0011
0.0299110.0014
0.0400110.0017
MFVI
0.008611.0007 0.0106::0.0000 0002
0.014410.0005
0.02261100007
0.029110.0011
Kernel
MCD
0.00501000003 0.0088
0.013410.0008
0.0225:10.0011
0.032410.0013
KFLLLA
0.0069-10.0003
0.0109::000003
0.0155
0.0240+1000066
0.03341:0.000 0.0008
SNGP
0.0049::00002
0.0087++00004
0.0132::0.0005
0.0225++0.0006
0.0314+:0.000
SGPA
0.0053-10.0003
0.00901000004
0.0136:100005
0.0225:10.0009
0.0320=100010
DE
0.0074
0.0085
0.0112
0.0181
0.0256
SGPAE 0.0047
0.0061
0.0090
0.0158
0.0239
Table 25: ECE of CIFAR100-C for ViTs trained on clean data without data augmentation.
Skew Intensity
Model
2
5
SDP
MIE
0.0004+000000
0.0113++000000
0.00100000000
0.0127++0.0000
0.0135+0.0000
MLE
0.009010.0001 0.01031000000
0.011110.0001
0.012310.0000
0.013410.000
MFVI
0.004310.0000 0.00541000000
0.006210.0000
0.0073+10.0001
0.008310.0000
Kernel
MCD
0.0063-10.0000
0.0077+000000
0.0085=000000
0.0099:10.0000
0.0112+0.000 0.0000
KFLLLA
0.0022++0.0002 0.0016+000001
0.0013=0.0001
0.0010++0.0001
0.0008+0.0000
SNGP
0.006
0.0078+0.0001
0.0085=000001
0.0097++0.0001
0.0107+0.0000
SGPA
0.0061+000000
0.0074+0.0001
0.0083+0.0001
0.0096+0.0001
0.0109+00.0001
DE
0.0051
0.0063
0.0072
0.0086
0.0099
SGPAE 0.0042
0.0054
0.0062
0.0076
0.0090
Table 26: ECE of CIFAR100-C for ViTs trained on clean data with data augmentation.
Skew Intensity
Model
2
5
SDP
MLE
0.00431:0.00 0006
0.0050+10.0007
3.0057=0.0007
10068$10.0007
0.00781000007
MLE
0.0031+:0.002
0.0040++0.00002
0.0049:000000
0.0061+:0.000
0.0072++0.0002
MFVI
0.0012:10.0001
0.0012+:0.000
0.0015=100000
0.0024:10.0000
0.00311:0.00 0.00
Kernel
MCD
0.0017110.0001
0.0025110.0001
0.0033:10.0001
0.0045:1000001
0.0057-:0.000
KFLLLA
0.0013++0.0001
0.0020+0000001
0.0027+0.0001
0.0038++0.0001
0.0049++0.000
SNGP
0.001 1+0000000
0.0017+0000000
0.0025+00.000
0.0036+0.00000
0.0046+0.0000
SGPA
0.0020++00001
0.0029+0.0001
0.0037+000001
0.0050++0.0011
0.0062+00.000
DE
0.0012
0.0017
0.0024
0.0036
0.0047
SGPAE
0.0012
0.0017
0.0024
0.0035
0.0047
27

---

Published as a conference paper at ICLR 2023
Table 27: MCE of CIFARIO-C for ViTs trained clean data without data augmentation.
Skew Intensity
Model
2
5
SDP
MLE
0.466610.0021
0.4840::0.0026
0.5041=0.0011
0.528010.0019
0.553440.0000
MLE
0.4038:1.0022 0.4362:+0.0022
0.4655=0.0000
0.5051+:00033
0.5442:+0.005
MFVI
0.0598:1..0112
0.0852::0.0150
0.1125=0.0147
0.15861:0.0177
0.201
Kernel
MCD
0.1902:10.0021 0.2402:10.0022
0.2807+0.0026
0.3413:10.006
0.3949110.00 100401
KFLLLA
O. 0.1977
0.2.62+000024
0.2729+-002.38
0.3248+00.1220
0.3733+00.0200
SNGP
0.04+
0.2337+0..001
0.2710+000001
0.3229+
0.3798 +..09%+
SGPA
0.1273++00002
000801+0.0046
0.2212++0.0061
0.2807++0.0057
0.3340++0.0047
DE
0.1051
0,1402
0.1813
0.2476
0,3091
SGPAE
0.0570
0.0816
0.1173
0.1757
0.2208
Table 28: MCE of CIFARIO-C for ViTs trained clean data with data augmentation.
Skew Intensity
Model
2
5
SDP
MLE
0.3500110.00 0125
0.3649::0:0126
0.387010.0012
0.41861000122
0.4561 10.0129
MIE
0.1188+000093
0.1451+0.0091
0.0776=0000090
0.2341+0.0009
0,2886+00,0090
MFVI
0.0772++000048
0.0965+0.0022
0.1283+000052
0.1903++000006
0.2261+00.015 +0.000
Kernel
MCD
0.0566+0000032
0.0083+000031
0.1110++0.008
0.1720+0.0062
0.2298+000005
KFLLLA
0.0808+10.003 0.1035-::0002 0.028
0. 0.00==000030
0.1845::000053
0.2396=10..005
SNGP
0.0557:10.0013 0.0797=000029
0.1093=0.0009
0. 0.17644::00040
0.2318=0.0004
SGPA
0.0569+E0.0022
0.0781+000026
0. 0.1050=0.0033
0. 0.1601=+0.0054
0.2167
DE
0.0868
0.0855
0.1003
0.1334
0.1705
SGPAF
0.0627
0.0659
0.0882
0.1262
0.1776
Table 29; MCE of CIFAR100-C for ViTs trained clean data without data augmentation.
Skew Intensity
Model
2
5
SDP
MILE
0.6608+000014
0.6852+00.014
0.7009 1..0022000022
0.7542+00.011
MLE
0.60371000033 0.642811000031
0.664910:0027
0.70371000007
0.738510.0010
MFVI
0.1925=00.0311 0.2594110.0339
0.3069=000055
0.380940.004
0.4562+00.027
Kernel
MCD
0.3424+10.042
0.4103+:00047
0.4554=1.0045
0.5255++0.005
0.5902+00.0338
KFLLLA
0.7691+00.016
0.763240.0181
0.7512=000194
0.7248++0.0166
0.6699++0.0228
SNGP
0.3631++00004
0.4201 0.4201+0.0044
0,4624--0.0339
0.5177++0.0043
0.5749+0.0083
SGPA
0.32171000057
0.388610000833
0.4353+0.0099
0.5047110.083
0.5726110.0772
DE
0.1767
0.2257
0.2655
0.3522
0.4202
SGPAE 0.1247
0.1766
0.2209
0.3142
0.3965
Table 30: MCE of CIFARI00-C for ViTs trained on clean data with data augmentation.
Skew Intensity
Model
5
SDP
MLE
0.3142-:0.0470
0.3487=0.0058
0.381610.0451
0.4308-10.0412
0.48.9)+0.0038
MLE
0.22371:000136
0.2650+0.0022
0.309810.0131
0.3733::0.0117
+0.0140
MFVI
0.13301:0.000 0083
0.1173-10.064
0.127510.0051
0.1751 0006
0.2208110.006
Kernel
MCD
0.0995110.0000
0.141810.0067
0.184510.0067
0.2568110.008
0.3240-10.088
KFLLLA
0.0893-10.004
0.1278110.0048
0.168910.0057
0.2355100.006
0.2999
SNGP
1.063610.0012
0.0972-10.0013
0.1377=0.0015
0.2007+00.001
0.2620=10.0012
SGPA
0. 0.1120+0.0022
0.158240.0034
0.2062+00.040
0.2776++0.002.
0.3455+0.0030
DE
0.0920
0.0971
0.1220
0.1906
0.2444
SGPAE
0.0722
0.0888
0.1088
0.1766
0.2291
28

---

Published as a conference paper at ICLR 2023
Table 31: AUROC and AUPR metrics for OOD detection using ViTs trained on CIFARIO without data augmentation.
OOD: CIFARTOO
OOD: SVHN
0OD: MINIMMAGENET
Model
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
SDP
MLE
0,68934000018
0.6433=000023
0.69834000109
0.220600.0074
0.7115+000018
09219=000005
MLE
0.7208=0000029 0,674::000099
0.7477:0001118
0:851820.0088
0.7325-0:0116
0.930=000004
MPVI
0.717740.005 0,6722:0000500
0.70591100115
0.79441000134
0.7531=0.0036
0.9407=0.0008
Kemel
MCD
0,7426=+0.0015 0.7046+400000 0016
0.6860+0.01388
0.8045+0.0058 0083
0.7443=0.0019
0.9366=0.005
KFLLLA
0.7228=0.00556
0.6889=:0.0555
0.75991:0.0105
0.8345+0.0069
0.7855=0.0046
0.9487-0.015
SNGP
0.7391
0.7023++0.000 0029
0.7135+0.0125
0.8258+0.0088
0.7091=000050
0.9341-0.0015
SGPA
0.7498=000010 0.7111=0.0022
0.7198:0.0085
0.8125
0.0001 000007
0.9382-0.0004
DE
0.7757=+0.0000 0,7423-+-00000
0.7819++00000
0.8512+00.000 0000
0. 0,8007 000000
0.9520-0.0000
SGPAE
0.7872:10.0000
0.748240.0000 0.000
0.7600
0. 0.00000000000 0.8619+000000
0.9520=0.0000
Table 32: AUROC and AUPR metrics for OOD detection using ViTs trained on CIFARIO with data
augmentation.
OOD: CIFARIOU
ODD: SVHN
OOD: MINIMAGENET
Model
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
SDP
MIE
0.8455 4000007
0.7780-:0.0013
0.0038+0000000
0.9308+0.0015
0.0075+0.0009
0.958500.005
MLE
0.7908
0.752310.0028
0.8748:10.0023
0.933810.0015
000022
0.555-000005
MFVI
0.7009=000044
0.6530=:0.004
0.76524:0.019
0.8468#0.0109
0.7506=0.0073
0.9379-0.0027
Kernel
MCD
0.7995 0038
0.76141:000300
0.8754::0.000 0.0400
0.930410.0029
0.8282:0.0029
0.9595=0.0007
KFLLLA
0.7989 0041
0.7617=0.0036
0,8884+0.0000 0036
0,9415+0.0022
0.8273=000034
0000070
SNGP
0.7889 0.0092
0.7543=1.0104
0.87761
0.932610.0032
0.812510.0772
0.9551=0.0020
SGPA
0.8007:400001
0.7630-4-0 0013
0.8746++0.006
0.9281+000000
0.8319=0000144
0.9600=0.0004
DE
0,8230
0,7869=0.0000 0000
0.0195+:0.0000
0.9575=0000000
0,00000000000
0.9666=0.0000
SGPAE
0.8264:+0.0000
0,7917+400000
0,9004++0.000
O. 8.06--0000000 0.9452++0.0000
0.9677-0.0000
Table 33: AUROC and AUPR metrics for OOD detection using ViTs trained on CIFAR100 without
data augmentation.
OOD: CIFARIO
OOD: SVHN
OOD: MINIMAGEENT
Model
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
SDP
MLE
0.6091110.0023
0.575610.00000
0.6242:00.0044
0.785610.0015
0.6512:10.0188
0.9012-0.0007
MIJE
0.6355
0.5963--000055
0.6725+-001111
0,8171 +0.012-
0.6582-00.005
Kemel
MFVI
0.6277:10000
0,5900-10.003 0037
0.6242::0.0108
0.7729+100007
0.6496:10.0054
0.9051-0.0024
MCD
0.6729 0.0069
0.6310++0.007
+.0054+0.0057
0.7597++0.0038
0.6854
0.9155
KFLLLA
0.6297=+0.0059 0,5920++0.0071
0.6733+-000199
1.8101+0000086
0.6915=000057
0.9194-0.0177
SNGP
0.6448:10.0016 0.6080:10.0020
0.676410.0096
0.8093:10.0065
0.6823:0.0025
0.9161-0.0099
SGPA
0.6712++0.0052 0.6297-+0.0057
0.6454+0.0147
0.7854+0.0099
0.6911+0000028
0.9170-0.009
DE
0.6848:10.0000 0.6377:10.0000
0.7388
0.842510.0000
0.739410.0000
0.9300-0.0000
SGPAE
0.6925+0000000
0.6461++00000
0.6961+00.000
0.8213+000000
0.7.90
0.9304 0.0000
Table 34: AUROC and AUPR metrics for OOD detection using ViTs trained on CIFAR100 with
data augmentation.
OOD: CIFARIO
OOD: SVHN
OOD: MINIMAGENNT
Model
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
SDP
MLE
0.6778:000022
0:6312:000000
0.76524000093
0.876950.0055
0.7430:0.0065
0.946-00.020
MIE
4.68994400022
0.6086+00.002
0.7570++0.0056
0.8670+0.0025
0.7036
0.9374-0.001
Kemel
MEVI
0.6403:10.004 0.5924:10.003
0.74851000106
0.864010.0006
0.726910.0099
0.9317-0.0030
MCD
0.7047:10.0025
0.6068:10.0029
0.7972+10.040
0,8885 1000003
0.0000
0.9400-0.0088
KFLLLA
0.6924:100001
0.6510:10.0022
0.816410.0044
0.9104 0.003
0.7726: 0008
0.9435-0.0024
SNGP
0.6813:10.0034
0.638810.0034
0.7616110.0090
0.8701 10.0052
0.7086:10.0048
0.0067=0.0014
SGPA
0.7056:10.0112 0.6586::0.0111
0.81201:0.0335
0.89711000002
0.7055
0.9430=0.0044
DE
0.7171110.0000 0.6718:10.0000
0.8459=10.0000
0.9173-10.008
0. 0.000000.0000
0.0517-0.0000
SGPAE
0.7198:10:0000
0.6719:0.0000
0.86861000000
0.929811000000
0.815200.0000
0.9506=0.0000
Table 35: AUROC and AUPR metrics for OOD detection using Transformers trained on ZINC.
Model MCD
AUROC
AUPR
SDP
0.456610.0072
0.48891000029
MFVI
0.725440.0632 0.6821#0.0512
Kernel
MCD
0.87971100039 0.8629#100071
SGPA
0.8968--0.0317 0.8705+00.0335
DE
0.9783+0.0000 0.9697+0.0000
SGPAE
0.9884+000000
0.972740.0000
29


Reviews:
[{"review_id": "umXYzp5Vv3d", "reviewer": "Reviewer_o7ME", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new methodology to introduce uncertainty quantification in Transformer models. This probabilistic approach is based on Gaussian processes. The driving idea is to exploit the similarity between the scaled dot-product (Vaswani, 2017) and the Kernel-attention (Tsai, 2019) with the posterior mean of sparse Gaussian processes (SVGP), to obtain posterior approximations in the attention heads of transformers. Despite the computational cost, which is reduced using global and amortized \"keys/inducing-points\", the methodology shows a significant performance which is compared with other baselines based on probability in several datasets for different tasks.", "summary_of_the_review": "The paper is valuable and relevant for the community, as it builds a new door to introduce probabilistic methods into Transformers via the equivalences in kernel-attention heads. The methods brought are perhaps revisited and there are not of extreme novelty, but the ideas presented for the solution, the equivalences and their connection are still important. The experiments are thorough and I believe the results presented. However, there are some weaknesses and missing details that make me decrease my score for strong acceptance to weak acceptance. If these ones are clarified, I would be glad to raise my score.\n\n", "strength_and_weaknesses": "**Strengths:** The main strength of the paper is on the identifiable equivalences. Find the simplification of transformers models as certain parallelisation of multi-head self-attention modules, where some scaled dot-product operations are similar to the core linear-operators of GPs when considering kernels is an important point. Additionally, exploiting this equivalence and building a new uncertainty estimation framework on top of the transformer's architecture is also nice. The paper provides a lot of details to make sure that is clear where the linking points between SVGPs and Transformers join. Experimental results are somehow clear and provide a fair comparison which the chosen baselines.\n\n**Weaknesses:** In my opinion, there are perhaps several equivalences and connections (i.e. with multi-output GPs and Deep GPs) that are introduced but not really developed or consistently analysed. For example in the paragraph before Section 3.2, the following is added:\n\n> (...) multi-output SVGP (...) and the output of a kernel attention block\n\nI see and appreciate the similarity between K-attention and the posterior mean of SVGP, but the MO-SVGP connection with multi-head attention is not entirely clear to me if I think in papers like (Alvarez, 2008). What I want to remark is that MOGP with SVGPs seems a bit more complicated to me, and does not seem that easy to say that both are equivalent... At least I would like to see this justified.\n\nSimilarly, for Deep GPs, the connection is done, but no experiments or further derivation is added on this line. That for me is a flaw of the paper, which I think is not that difficult to solve...\n\n**Questions:**\n\n*[Q1].* How expensive is to train wrt Eq. (14). The use of Monte-Carlo in expectations for SVGPs is in general not a good idea (long training, not very scalable wrt dimensionality, noisy evaluation on the likelihood model), so I am trying to figure out what would be the effect here? Could the authors clarify a bit on this point?\n\n*[Q2].* Thinking in terms of how difficult is to bring posterior computations to standard NNs, and how the \"Gaussian nature\" is broken in Deep GPs when one layer after the other is connected passing GPs through, I am somehow surprised that SVGPs fit that well in multiple layers of the Transformer without any of these problems. Are non-linearities or multiple layers affecting the approximation considered? \n\n**References:**\n\n(Alvarez, 2008) https://proceedings.neurips.cc/paper/2008/file/149e9677a5989fd342ae44213df68868-Paper.pdf\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rgENjYHOQbz", "reviewer": "Reviewer_PkUZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a Sparse Gaussian Process Attention (SGPA) for the Transformer architecture. The SGPA is used instead of the classical scaled dot-product attention and directly performs an approximation of the Bayesian inference in the attention blocks in Transformers. In addition, the authors provide decoupled SGPA to overcome the time and memory consumption limitations of the standard SGPA. This approach is evaluated on image classification, graph property regression, and NLP classification tasks. The authors show that this method is able to produce better calibrated predictions. The SGPA also helps to produce better uncertainty estimates and shows improvements in the OOD detection task in CV.", "summary_of_the_review": "This paper provides a theoretically derived method that helps to calibrate prediction and capture better uncertainty estimates for Transformers models. The authors make a significant contribution with minor weaknesses that can be solved. \n\n[After the review] The response by authors resolved part of my concerns and I decided to increase the score.", "strength_and_weaknesses": "Strength:\n* This paper is the first to consider a Gaussian process in attention block and extends SGPA in Transformer for uncertainty estimation.\n* The authors carefully provide limitations of the proposed method and propose decoupled SGPA which overcomes the time and memory consumption issues.\n* Derivations of the evidence lower-bound objective (ELBO) for optimizing the variational parameters in SGPA and of the decoupled SGPA for the Transformer's attention are provided.\n* This method shows significant improvement in in-distribution calibration quality and OOD detection performance while achieving competitive accuracy against the standard Transformers.\n* The results are well-presented and clearly discussed.\n\nWeaknesses:\n* Not mentioned other approaches that consider stochastic evaluation in the attention block for UE with Transformers. For example, [1] uses stochastic self-attention with the Gumbel-Softmax trick.\n* It would be useful to see the results of OOD detection in NLP tasks, for example, for the following tasks: [2], [3].  \n* It is not clear how exactly the variational parameters are calculated in the decoupled SGPA in equation (11).\n* It is not clear what method refers to MLE in the Experiments section. Is it a method that leverages maximum probability? If no, please, add a reference or exact formula.\n* It would be interesting to see the actual computational time of the inference stage of the proposed method with 10 Monte Carlo samples in comparison with the classical MC dropout.\n* The improvement of the proposed method over the MLE baseline in the OOD detection task is not stable taking into account the cost of computation.\n\nTypos:\n* On page 5 after equation (12): “viewd” -> “viewed” \n* On page 8 in section 4.4: “kernel based” -> “kernel-based”\n\nReferences\n[1] Jiahuan Pei, Cheng Wang, and György Szarvas. 2022. Transformer uncertainty estimation with hierarchical stochastic attention. In AAAI 2022.\n[2] Larson, S. et al. An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction. In EMNLP 2019.\n[3] Gangal, V. et al. Likelihood Ratios and Generative Classifiers for Unsupervised Out-of-Domain Detection in Task Oriented Dialog. In AAAI 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kaZ7q8INj-", "reviewer": "Reviewer_GcWL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors suggest an efficient implementation of GP for the attention layers of transformers. Through a few experiments they argue it helps in improving the accuracy, the covariate-shift robustness, the calibration and the out-of-distribution detection performance when fine-tuning transformers on smaller scale datasets.", "summary_of_the_review": "The paper is interesting, but it either needs a better motivation or much stronger experimental evidence. If either (or both) these issues are improved, I am happy to rise the score.\n\n--------------------\nPOST-REBUTTAL AND DISCUSSION WITH OTHER REVIEWERS\nI increase my score to 5. \n\n- Although it is difficult to interpret the meaning of GPs over self-attention, the additional comparisons with strong baselines (both deterministic and non-deterministic) like TS, KFAC-LLLA, SNGP and Deep Ensembles shows the method is at least empirically effective\n- The justification provided for the usage of GPs on self-attention is intuitive, and although still not completely supported and questionable given the lack of an understanding of how uncertainties compose through the layers of a network in the presence of additional components (normalization layers, non-linearities, skip-connections etc.), I think it would be unfair considering this to be a limitation of this specific work, as it is a known problem of Bayesian Deep Learning.\n- The usefulness of the method might depend on the fact Transformers don't converge to a good minimum without enough pre-training data on such small datasets. The technique can be considered effective probably also because it somehow regularizes the training procedure. Therefore, the improve the impact of the paper, it would be important to show the proposed method can be effective also when fine-tuning pre-trained models. \n\n\n", "strength_and_weaknesses": "Strenghts:\n- The paper is clearly written, easy to follow\n- The inclusion of preliminaries is nicely done and can help people not familiar with the topic \n- The method seems to positively affect not only the uncertainty properties but also the accuracies  \n- The authors perform an adequate selection of the type of experiments to perform\n- The idea of developing a computationally efficient GP implementation for attention is interesting \n\nWeaknesses:\n- Q1: Some very important baselines seem to be missing, and the used baselines are generally acknowledged to be extremely weak (MCD and MFVI are notorious for being outperformed by much better SOTA methods). I would suggest integrating your experiments (1) for accuracies on in-distribution and covariate-shift datasets, it's important to evaluate against methods that have been observed to produce improved accuracies (and also improved calibration and out-of-distribution performance) like PixMix [1], RegMixup [2], Deep Ensembles [3] (and optionally any of its more computationally efficient variants like Snapshot ensembles [4] or BatchEnsemble), (2) for calibration experiments, as a bare minimum Temperature Scaling [5] should be considered too, and it would be good to compare with more sophisticated calibration techniques too, (3) for out-of-distribution detection performance techniques like G-ODIN [6] and VOS [7]. Also, since your methodology falls in the Bayesian inference with GPs, it would be good to at least compare with KFAC-LLLA [8], but I understand it might be tricky to apply SNGP, DUE and other GP-based techniques to transformers. \n- Q2: The motivation of the methodology is not clear. This is a general problem of Bayesian Deep Learning, but it's not clear why having GP on the self-attention should meaningful. While works like SNGP and KFAC-LLLA motivate their usage of (approximations) of GP with a distance-aware uncertainty estimation narrative (though it is questionable that euclidean distances are meaningful at all), I don't feel like the paper provides a clear motivation of why introducing GPs in a few parts of the network should be theoretically meaningful. The paper does not even provide aneddotical evidence that the uncertainty estimates the GP provide in the self-attention can actually detect ambiguous features of the input.\n- Important comment on Q1 and Q2: if Q2 cannot be addressed due to the limitations of Bayesian DL, working on Q1 and showing the effectiveness of your method at least empirically would convince me to rise the score. \n- Q3: The paper is not really about calibrating for pre-training but for fine-tuning. Modern approaches to fine-tuning are quite sophisticated and might affect uncertainty estimates or the usefulness of the proposed method. Did you consider advanced fine-tuning techniques?  \n\nImprovements (Marginal):\n- Probably the paper should emphasize more it's about fine-tuning and not really calibrating large-scale pre-training\n- The NLL is known to be a questionable metric for calibration, but a lot of literature still reports it \n\n\n\n[1] https://arxiv.org/abs/2112.05135\n[2] https://arxiv.org/abs/2206.14502\n[3] https://arxiv.org/abs/1612.01474\n[4] https://arxiv.org/abs/1704.00109\n[5] https://arxiv.org/abs/1706.04599\n[6] https://arxiv.org/abs/2002.11297\n[7] https://arxiv.org/abs/2202.01197\n[8] https://arxiv.org/abs/2002.10118", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RBMzatWInQ", "reviewer": "Reviewer_3Jj8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to formulate the mult-head attention operation used in transformer models as a sparse gaussian process. The work shows how the computation of such a GP attention mechanism can be sped up by relying on inducing points and standard approaches for increasing Gaussian process scalability. Multiple such attention operations can be stacked by sampling between layers. In the experiments of the paper, the proposed model shows comparable performance in terms of accuracy, but scores better in terms of NNL and calibration metrics on small vision and graph datasets. The baselines used in the paper where Maximum likelihood, mean field VI in weight space and Monte Carlo Dropout.", "summary_of_the_review": "The approach is interesting and to some degree novel. While the results look promising, further experiments are necessary to completely convince me. The model is never trained on a data set where there are external reports on the current maximal performance achievable. The rather low performance on CIFAR10 and 100 is in this context slightly alarming. If the authors, can show me references where a transformer model reached similarly low performance or if they can demonstrate the utility of their method on a data set where the performance of transformers is already well quantified I will be happy to increase my score accordingly.", "strength_and_weaknesses": "**Strengths:**\n\n * The paper is mostly clearly written and the motivation of the method is easy to follow and explained well.\n * The approach is to some degree new and the experiments look promising. \n\n**Weaknesses:**\n\n * In the introduction of the method, it would be beneficial to differentiate the application of Gaussian processes for interpolation compared to classification. Both approaches are common in machine learning, yet in one case the goal is to condition on the observations of the instance in order to predict values at unknown query locations whereas a classification GP predicts the output of an instance on the test data, while conditioning on the training data. The former is closer to what the authors are proposing in their work.\n * The work does not compare against approaches relying on deterministic uncertainty estimates while citing papers from that field (for instance Liu et al., NeurIPS 2020). This is particularly relevant in the OOD experiment.\n * The performance of all models is relatively low on the datasets presented. For instance, 77% accuracy is far from SOTA on CIFAR10. This questions the usefulness of the experiments on these data. While the trends compared to baseline approaches are generally ok, it is difficult to rely on these results when models are not applied on dataset/model combinations where there is existing data. I would suggest to at least run one comparison on imagenet, where sufficient performance measurements of vision transformers are available in other papers.\n * The approach is very similar to Set Transformers, which use a similar idea to speed up the computations. A reference would be appropriate.\n * Utilization of `[]` in the equations is slightly confusing. I would recommend to discard these or indicate that they solely are used to keep the subscripts separated in the text.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster