Exploring the Uncertainty Properties of Neural Networks’ Implicit Priors in the Infinite-Width Limit
Authors: Ben Adlam
Jaehoon Lee
Lechao Xiao
Jeffrey Pennington
Jasper Snoek
Venue: ICLR 2021 Poster (2021)
Keywords: Deep Learning
Uncertainty
Infinite-Width Limit
Neural Network Gaussian Process
Bayesian Neural Networks
Gaussian Process

Abstract: Modern deep learning models have achieved great success in predictive accuracy for many data modalities. However, their application to many real-world tasks is restricted by poor uncertainty estimates, such as overconfidence on out-of-distribution (OOD) data and ungraceful failing under distributional shift. Previous benchmarks have found that ensembles of neural networks (NNs) are typically the best calibrated models on OOD data. Inspired by this, we leverage recent theoretical advances that characterize the function-space prior of an infinitely-wide NN as a Gaussian process, termed the neural network Gaussian process (NNGP). We use the NNGP with a softmax link function to build a probabilistic model for multi-class classification and marginalize over the latent Gaussian outputs to sample from the posterior. This gives us a better understanding of the implicit prior NNs place on function space and allows a direct comparison of the calibration of the NNGP and its finite-width analogue. We also examine the calibration of previous approaches to classification with the NNGP, which treat classification problems as regression to the one-hot labels. In this case the Bayesian posterior is exact, and we compare several heuristics to generate a categorical distribution over classes. We find these methods are well calibrated under distributional shift. Finally, we consider an infinite-width final layer in conjunction with a pre-trained embedding. This replicates the important practical use case of transfer learning and allows scaling to significantly larger datasets. As well as achieving competitive predictive accuracy, this approach is better calibrated than its finite width analogue.

Published as a conference paper at ICLR 2021
EXPLORING THE UNCERTAINTY PROPERTIES OF NEU- RAL NETWORKS' IMPLICIT PRIORS IN THE INFINITE-
WIDTH LIMIT
Ben Adlam*
Jachoon Lee"
Lechao Xiao
Jeffrey Pennington
Jasper Snoek
Google Brain
(adlam, jaehlee, xle, jpennin, jsnoek @google.com com
ABSTRACT
Modern deep learning models have achieved great success in predictive accuracy for many data modalities. However, their application to many real-world tasks is restricted by poor uncertainty estimates, such as overconfidence on out-of- distribution (OOD) data and ungraceful failing under distributional shift. Previous benchmarks have found that ensembles of neural networks (NNs) are typically the best calibrated models on OOD data. Inspired by this, we leverage recent theoretical advances that characterize the function-space prior an infinitely-wide NN as Gaussian process, termed the neural network Gaussian process (NNGP). We use the NNGP with softmax link function to build a probabilistic model for multi-class classification and marginalize sample from the posterior. This gives us a better understanding of the implicit prior NNs place function space and allows direct comparison of the calibration of the NNGP and its finite-width analogue. We also examine the calibration of previous approaches classification with the NNGP, which treat classification problems as regression to the one-hot labels. In this case the Bayesian posterior is exact, and we compare several heuristics to generate a categorical distribution over classes. We find these methods are well calibrated under distributional shift. Finally, we consider an infinite-width final layer in conjunction with a pre-trained embedding. This replicates the important practical use case of transfer learning and allows scaling to significantly larger datasets. As well as achieving competitive predictive accuracy, this approach is better calibrated than its finite width analogue.
the latent Gaussian outputs
INTRODUCTION
Large, neural network (NN) based models have demonstrated remarkable predictive performance on test data drawn from the same distribution as their training data, but the demands of real world applications often require stringent levels of robustness novel, changing, or shifted distributions. Specifically, we might ask that our models are calibrated. Aggregated over many predictions, well calibrated models report confidences that are consistent with measured performance. The Brier score (BS), expected calibration error (ECE), and negative log-likeli likelihoooo (NL.L) are common measurements of calibration (Brier 1950, Naeini al., 2015 Gneiting & Raftery 2007).
Empirically, there are many concerning findings about the calibration of deep learning techniques, particularly on out-of-distribution (OOD) data whose distribution differs from that of the training data. For example, [MacKay (1992) showed that non-Bayesian NNs are overconfident away from the training data and Hein et al. (2019) confirmed this theoretically and empirically for deep NNs that use ReLU. For in-distribution data, post-hoc calibration techniques such as temperature scaling tuned on a validation set (Platt et al. 1999 Guo et al. 2017) often give excellent results; however, such methods have not been found to be robust shifted data and indeed sometimes even reduce calibration on such data (Ovadia et al. 2019). Thus finding ways to detect or build models that produce reliable probabilities when making predictions on OOD data is a key challenge.
Authors contributed equally to this work. Work done member of the Google Al Residency program https:///.co/aarrsideeee))

---

Published as a conference paper at ICLR 2021
Sometimes data can be only slightly OOD shift from the training distribution gradually over time. This is called dataset shift (Quionero-Camee Candela et al. 2009) and is important in practice for models dealing with seasonality effects, for example. While perfect calibration under arbitrary distributional shift is impossible, simulating plausible kinds of distributional shift that may occur in practice at different intensities can be useful tool for evaluating the calibration of existing models. A recently proposed benchmark takes this approach (Ovadia et al. 2019). For example, using several kinds of common image corruptions applied at various intensities, the authors observed the degradation in accuracy expected of models trained only on clean images (Hendrycks & DI. etterich, 2019 Mu Gilmer, 2019), but also saw very different levels of calibration, with deep ensembles (Lakshminarayanan et al. 2017) proving the best.
Calibration metries. There many common metrics used to evaluate the calibration of model. Probably most common is the negative log-liielihood (NLL), which is a proper scoring rule and whose units, nats, are a coherent unit for information entropy. The Brier score (BS) from Brier (1950) is also a proper scoring rule, but its scale is harder to interpret and its use of the squared loss makes it sensitive to in/frequent events. We also consider the expected calibration error (ECE) of Naeini et al. (2015), which is not proper scoring rule. In particular, uniformly random predictions on a balanced, classification dataset will produce an ECE of However, since measures the absolute difference between predicted and observed probabilities, the scale of ECE is easier to interpret. Finally, since the previous metrics require labels, we consider the average confidence and entropy on OOD data, where the confidence is the probability assigned to the most likely class and the entropy is the entropy of the predicted distribution over labels. Ovadia et al. (2019) has more discussion on calibration metrics.
Bridging Bayesian Learning and Neural Networks. In principle, Bayesian methods provide a promising way to tackle calibration, allowing us to define models with and infer under specific aleatory and epistemic uncertainty. Typically, the datasets on which deep learning has proven successful have high SNR (Signal-to-Noise Ratio), meaning epistemic uncertainty is dominant and model averaging is crucial because our overparameterized models are not determined by the training data. Indeed Wilson (2020) argues that ensembles are a kind of Bayesian model average.
Ongoing theoretical work has built bridge between NNs and Bayesian methods Neal 1994a Lee
et al. 2018 Matthews et al. 2018b), by identifying NNs as converging to Gaussian processes in the limit of very large width. Specifically, the neural network Gaussian process (NNGP) deseribes the prior on function space that is realized by an i.i.d. prior over the parameters. The function space prior is GP with specific kernel that is defined recursively with respect to the layers. While the many heuristics used in training NNs may obfuscate the issue, little is known theoretically about the uncertainty properties implied by even basic architectures and initializations of NNs. Indeed theoretically understanding overparameterized NNs is major open problem. With the NNGP prior in hand, it is possible to disambiguate between the uncertainty properties of the NN prior and those due to the specific optimization decisions by performing Bayesian inference. Moreover, it is only in this width limit that the posterior of a Bayesian NN can be computed exactly.
1.1 SUMMARY OF CONTRIBUTIONS
This work is the first extensive evaluation of the uncertainty properties of infinite-width NNs. Unlike previous work, we construct valid probabilistic model for classification tasks using the NNGP, Le. label's prediction is always a categorical distribution. We perform neural network Gaussian process classification (NNGP-C) using softmax link function exactly mirror NNs used in practice. Note that prior work on the NNGP has also used softmax functfnn, but considered approximate inference using inducing points on MNIST (Garriga-Alonso et al., 2019). We perform detailed comparison of NNGP-C against its corresponding NN on clean, OOD, and shifted test data and find NNGP-C to be significantly better calibrated and more accurate than the NN.
Next, we evaluate the calibration of neural network Gaussian process regression (NNGP-R) on both UCI regression problems and classification on CIFARIO. As the posterior of NNGP-R is a multivariate normal and so not categorical distribution, heuristic must be used to calculate confidences for classification problems. On the full benchmark of Ovadia et al. (2019). we compare several such heuristics, and against the standard RBF kernel and ensemble methods. We find the calibration of NNGP-R to be competitive with the best results reported in Ovadia et al. (2019). However in the process of preparing our findings for publication, newer strong baselines have been
2

---

Published as a conference paper at ICLR 2021
reported that we do not compare against
Finally, we consider NNs whose last layer only is infinite by taking a pre-trained embedding and using infinite- width final layer (abbreviated as NNGP-LL). This mirrors an important practical use case for practitioners unable to retrain large models from scratch, but looking to adapt one to their particular data, which could have relatively few samples. We compare the calibration of NNGP-LL with a multi-layer FC network, fine-tuning of the whole network, and the gold standard ensemble method, and again we find NNGP-LL to have competitive calibration. While NNGP-LL is not principled Bayesian method, it allows scaling to much larger datasets and potential synergy with recent advances in un- and semi-supervised learning due to the pre-training of the embedding, making this method applicable to many real-world uncertainty critical applications like medical imaging.
In addition, code for some of our experiments and precomputed kernels are also publicly available for any further analysis by the research community
2
BACKGROUND
Neal 1994b) identified the connection between infinite-widre width NNs and Gaussian processes, showing that the outputs of a randomly initialized one-hidden layer NN converge to a Gaussian process as the number of units in the layer approaches infinity. Let 2 (xx describe the 7th pre-activation following a linear transformation in the 1hh layer of a NN. At initialization, the parameters of the NN are independent and random, so the central-limit theorem can be used to show that the pre-activations become Gaussian with zero mean and a covariance matrix K(xx, = (T)(())
Knowing the distributions of the outputs, one can apply Bayes theorem to compute the posterior distribution for new observations, which we detail in Sec. classification and Sec. for regression. Moreover, when the parameters evolve under gradient flow to minimize a squared loss, the output distributior remains a GP but with a different kernel called the neural tangent kernel (NTK) (Jacot jet al. 2018; Lee et al. 2019). This allows us to derive the exact formula for ensemble training of infinite width networks; see Sec. A for more details. While we also analyzed the uncertainty properties of the NTK in our experiments, we observed that the results mirror those of the NNGP.
These observations have recently been significantly generalized to include NNs with more than one hidden layer (Lee et al. 2018 Matthews et al. 2018b) and with a variety of architectures, including weight-sharing (CNNs) (Xiao et al. 2018, Novak al., 2019b; Garriga-Alonso 2019), skippconeections, dropout (Schoenholz et al. 20177 Pretorius et al. (2019), batch norm (Yang al. (2019). weight-tying (RNNs) Yang 2019). self attentino (Hron et al., 22020), graphical networks (Du et al.] 2019). and others. In this work, we focus on FC and CNN- Vec NNGPs, whose kernels are derived from fully-connected networks and convolution networks without pooling respectively (see Sec. A precise definitions). When it is required, we prepend FC- or C- to the NNGP to distinguish between these two variants. We use the Neural Tangents library of Novak al. (2019a) automate the transformation of finite-width NNs to their corresponding Gaussian processes.
al.
Another line of work has connected NNs and GPs by combining them in a single model. Hinton & Salakhutdinov (2088) trained a deep generative model and fed its output in to a GP for discriminative tasks. Wilson et al. (201 consider a similar approach, creating a GP whose kernel is given as the composition of nonlinear mapping (e.g. a NN) and spectral kernel. They scale this approach to larger datasets in Wilson et al. (2016a) using variational inference, but Tran et al. (218)) found this method to be poorly calibrated.
3 FULL BAYESIAN TREATMENT OF CLASSIFICATION WITH NEURAL KERNELS
is common to interpret the logits of NN once mapped through softmax as a categorical distribution over the labels for each point. Indeed cross entropy loss is sometimes motivated as the KL divergence between the predicted distribution and the observed label. Similarly, while the
attps:
httcps://iithub.co wttp://..cooogggeeeeeeeeeoe httpoo///shhooocoooggoooeeeerree www.ea/www.......eeeeeeeeeeeeeee http:///iiiiiiiiiieeeeettttttttt
Note that K is independent of due to the independence between 2 and 2/y for şf f. In other words, the covariances of the GP have a Kronecker factorization K. & lz, where e is the number of classes.
"The source of randomness comes from random initialization.
3

---

Published as a conference paper at ICLR 2021
initialization scheme used for NN's parameters is often chosen for optimization reasons, it can also be thought of as prior. This implicit prior over functions and over the distribution of labels has effects, despite the decision of most common training algorithms in deep learning to forgo explicitly trying to find its posterior. In this section, we take seriously this implicit prior and utilize the simple characterization it has over logits in the infinite width limit to define probabilistie model for mutli-class classification as
(w/x) ~ (oftmax(f(()))) where f~ (P(O,
(1)
and KC is the NNGP kernel. In words, Eq. 1 states that the latent space is a Gaussian process with mean 0 and kernel function IC and the softmax function is used to transform the latent space into a distribution over classes for each point x. If this prior is a correct model for the data generation process, then the posterior is optimal for inference. Thus, by avoiding heuristic approaches to inference, we are able to directly evaluate the statistical properties of the prior. Then by comparing this to more standard gradient- based training methods, we can understand their effect on calibration.
For training data (X,)),, ) the posterior on a test point x can be found by marginalizing out the latent space. Denote the latent vector for the whole training data with F == f(X) and the test point with f f= [xx), and then let F denote their concatenation. Then the posterior for the test label can be calculated by integrating over the posterior on the latent space of the test point:
plyla, (,,, =
softmax(())((loxxx.))))
(2)
Note that ((fl is marginal of the latent vectors' posterior distribution on all points, i.e.
(FFR,XX)) = (p x)
(3)
which, after neglecting the normalizing denominator, is proportional to the prior in the latent space and the probabilities under the softmax link functionnnnnnnn N(F; O, II, softmax((F))))) See Williams Rasmussen| (2006) for more details. We generate samples from the joint posterior distribution of and F' using elliptical slice sampling (ESS) (Murray al. 2010).
Note that the latent space dimension in the datasets we consider substantial, which makes inference with ESS computationally intensive. Specifically, each step of ESS requires sampling from the prior and thus a matrix multiplication of the Cholesky decomposition of the kernel matrix (whose computa- tional time is cubic in the dataset size but can be precomputed once), and the convergence times are longer for larger systems. Moreover, while ESS is hyperparameter free, we found its convergence is affected by the kernel hyperparameters. We release open-sourced JAX based implementation of GPC based on ESS described here.
Given these computational constraints on both tuning hyperparameters and running ESS, we focus our attention on FC and simple CNN-Vec kernels, which are much faster to compute than pooling based kernels Since we wish make like-for-like comparisons between the Gaussian process and the corresponding NN, this restricts us to simplistic NN architectures that are far from state-of-the the-art. We find this acceptable as the goal of this section is relative comparison and not to obtain optimal absolute performance
To tune the kernel hyperparameters (weight and bias standard deviations, the activation function, the depth, and the kernel jitter) we used the Google Vizier service (Golovin et al., (017b) with a budget of 250 trials and selected the setting with the best log-likelihood on a validation set. We use the same hyperparamters for the NN to make a direct comparison of the prior. The additional hyperparameters required for the NN like width and training time, were also tuned using Vizier. We also compared against the NN performance when all its hyperparameters NN improved but the calibration results were broadly similar. See the supplement for additional details. Note that while optimizing hyperparamters based validation set likelihood is nonstandard for Bayesian methods, we do so here to make the fairest comparison to NNs, whose hyperparamters are normally optimized in this way.
tuned, and found the accuracy of the
$ For reference, the relatively efficient Myrtle kernel of Shankar 2020) takes approximately 12,000 V100 GPU hours to compute for CIFAR10 and CIFAR for single hyperparameter configuration. 'Making a fair comparison between finite versus infinite-witth network is subtle, requiring careful thought controlling for architecture and training methods (Lee al. 2020
4

---

Published as a conference paper at ICLR 2021
C-NNGP-C: CIFARIO
1.0
ACC.
1.0
ECE
Clean
0.8
Average eccurac
0.8
Fog
Donser
SVHN
Fog
0.6
Fog
Fag
0.4
0.4
Fog
0.2
0.2
0
0.0
0.0
2
0.2
0.6
0.8
Confidence
Corruption strength
Confidence
Figure Investigating the calibration of Gaussian process classification with CNN-GP kernels. (left) Histogram of the confidence of the posterior distribution for each test point. We compare the (CIFARIO) and an OOD test
C-NNGP-C and finite width CNN on an in-distribution (SVHN). C-NNGP-C shows lower confidence and higher entropy (see Fig, (2) on both test sets compared the CNN, which has very high confidence on many points as indicated by the spike around 1. On the clean data the CNN's overconfidence hurts its calibration, as it achieves worse ECE and BS than C-NNGP-C. (middle) The performance both models, C-NNGP-C is solid and CNN dashed, under increasing distributional shift given by the CIFARIO fog corruption from Hendrycks & Dietterich (2019). The accuracy of the CNN and C-NNGP-C are comparable as the shift intensity increases, but C-NNGP-C remains better calibrated throughout. (right) We bin the test set into ten bins sorted by confidence, and we plot mean confidence against mcan accuracy. C-NNGP-C remains closer y than the CNN, which shifts toward increasing overconfidence.
Table 1: Comparing NNs and against the equivalent NNGP-C on CIFAR- 10 and evaluated on several test sets. We observe that the NNGP-C outperforms its parametric NN counterpart on every metric. We see particularly significant improvements in ECE and NLL, implying that NNGP-C is considerably better calibrated. For complete results on other intensities of Fog corruption see Table
Data
Metric
FC-NN
FC-NNGP-C
CNN
C-NNGP-C
ECE
0.209
0.072
0.283
0.031
Brier Score
0.711
0.629
0.685
0.519
CIFARIO
Accuracy
0,486
0,518
0.576
0.609
NLL
17383
14007
21786
11215
BCE
0.178
0.098
0.271
0.042
Brier Score
0.707
0.655
0.685
0.542
Fog
Accuracy
0.471
0.496
0.561
0,590
NLL
16702
14671
19716
11755
ECE
0,279
0.057
0.420
0.134
Brier Score
0.961
0.847
1.052
0.846
Fog
Accuracy
0.241
0.250
0.287
0,289
NLL
26345
20694
33891
22014
Mcan Confidence
0.537
0.335
0.718
0,463
SVHN
Entropy
1.230
1.840
0.733
1.403
Mean Confidence
0.651
0.398
0.812
0.474
CIFAR100
Entropy
0.944
1.663
0.493
1,420
In our experiments, we compare NNGP-C and a finite NN of the same architecture on in-distribution, shifted, and OOD test data when both models are trained using the standard CIFARIO training set. We consider CIFARI00, which contains classes not present in CIFAR 10, and SVHN, which consists of images of house numbers, as two examples of OOD data. The five levels of fog corruptions from Hendrycks & Dietterich (2019), which distorts the CIFAR 10 test set with a fog-like effect, are used as increasing amounts distributional shift in Ovadia al. (2019). Our main findings for NNGP-C, summarized in Fig. and Table show that NNGP-C is well calibrated and outperforms corresponding NN. This indicates that, rather than the prior induced by the random initialization scheme, the
training of the NN is partly responsible for its poor calibration.
4 REGRESSION WITH THE NNGP
As observed in Sec. inference with NNGP-C is challenging as the posterior is intractable. this section, consider Gaussian process regression using the NNGP (abbreviated as NNGP-R), which

---

Published as a conference paper at ICLR 2021
is defined by the model
(y/z) ~ f(z) + where f~ (C).,
(4)
where 1C is the NNGP kernel and ~ NO, o.) ) is an independent noise term. One major advantage of NNGP-R is that the posterior is analytically tractable. The posterior at test point æ has Gaussian distribution with mean and variance given by
p(x)= = K(x,, (2,
and
0²(x) - K(x, x) - (x, X (2, (X(,,))))))
x),, (5)
where ()) is the training set of inputs and targets respectively and K. = K o Since I. has Kronecker factorization, the complexity of inference is ((( rather than x(c")x("). For regression problems where E R. the variance describes the model's uncertainty about the test point. Note that while this avoids the difficulties of approximate inference methods like MCMC, computation of the kernel inverse means running time scales cubically with the dataset size.
4.1 BENCHMARK ON UCI DATASETS
We perform non-linear regression experiments proposed by Hernández-L Lobato & Adams (015)).
which is standard benchmark for evaluating uncertainty of Bayesian NNs. We all datasets except for Protein and Year. Each dataset is split into 20 train/test folds. We report our result in Table comparing against the following strong baselines: Probabilistic BackPropagation with the Matrix Variate Gaussian distribution (PBP-MV) (Sun et al.) 2017), Monte-Carlo Dropout (Gal Ghahramani] 2016) evaluated with hyperparameter tuning as done Mukhoti et al. (2018) and Deep Ensembles (Lakshminarayanan et al., (017). We also evaluated standard GP with RBF kernel KRAF(x, x = 3 exp (->l x x (I) for comparison.
Instead of maximizing train NL.I for model selection, we performed hyperparameter search on validation set (we further split the training set so that overall train/valid/test split is 80/10/10), as commonly done in NN model selection and in the BNN context applied in Mukhoti et al. (011)) For NNGP-R, the following hyperparameters were considered. The activation function is chosen from (ReLU, Erf), number of hidden layers among [1, 4. 16], o from [1, 2, 4], of from [0., 0.09, 1.0], readout layer weight and bias variance are chosen either same as body or (a of) = (1, For the GP with RBF kernel we evaluated y E {10* E [-5, -4, ....3..] 3]} and 8 E {10* k e
[-3, --3.-4..... ... 3. 3.). For both experiments, de np. logspae (-6, 4, 20).
We found that NNGP-R can outperform and remain competitive with existing methods in terms of both poot-menn-oe wawaan--uuuuederrree (RMSE) and that NNGP-R achieves the lowest RMSE on the majority (5/8) of the datasets and competitive NLL. When using the NTK as the GP's kernel instead, we broadly similar results.
likkllioood (NLL). In Table observe
4.2 CLASSIFICATION AS REGRESSION
Formulating classification regression often leads to good results, despite being less princi- pled (Rifkin et al. 2003) Rifkin & Klautau| 2004). By doing so, we can compare exact inference for GPs to trained NNs well-studied image classification tasks. Recently, various studies of infinite NNs have considered classification as regression tasks, treating the one-hot labels as indepen- dent regression targets (e.g. Lee al. (2018); Novak al. (2019b); Garriga--Aonso al. )2019)).
Predictions are then obtained as the argmax of the mean in Eq. (5) i.c. arg max, u((x)x
However, this approach does provide confidences corresponding the predictions. Note that the posterior gives support to all of Idd including points that are known to be impossible. Thus, heuristic is required to extract meaningful uncertainty estimates from the posterior Eq. (5) even though these confidences will not correspond to the Bayesian posterior of any model.
Following Albert & Chib 1993):
Girolami & Rogers 2006), we produce a categorical distribution
for each test point z, denoted Pa, by defining
K II k=1
(i) == Plys max ma.......... ..... vall / 1(i argmax, ,U;)
p(un (x. D)
While training NNs with MSE loss is more challenging, peak performance can be competitive with cross- entropy loss (Lewkowyez et al. 2020 Hui& Belkin 2020 et al. 2020).

---

Published as a conference paper at ICLR 2021
Table 2: Result for regression benchmark on UCI Datasets. Note -r reports the standard error around estimated mean for 20 splits. We compare to strong baselines: PMP-MV (Sun et al. 2017). MC-Dropout from Mukhoti et (2018) and Deep Ensembles (Lakshminarayanan et al. 2017
Average RMSE Test Performance
Datasel
(m, d)
PBP-MV
Dropout
Ensembles
RBF
FC-NNGP-R
Boston Housing
(506, 13)
3.11 + 0.15
2.90 + 0.18
3.28 + 1.00
3.24 + 0.21
3.07 + 0.24
Concrete Strength
(1030, 8)
5.08 + 0.14
4.82 - 0.16
6.03 - 0.58
5.63 + 0.24
5.25 + 0.20
Energy Efficiency
(768,
0.45 + 0.01
0.54 + 0.06
2.09 + 0.29
0.50 0.01
0.57 + 0.02
Kin8nm
(8192, 8)
0.07 = 0.00
0.08 0.00
0.09 + 0.00
0.07 + 0.00
0.07 0.00
Naval Propulsion
(11934, 16)
0.00 0.00
0.00 1 0.00
0.00 = 0.00
0.00 0.00
0.00
0.00 0.00
Power Plant
(9568, 4)
3.91 0.04
4,01 0.04
4.11 0.17
3.82 0.04
3.61 0.04
Wine Quality Red
(1588, 11)
0.64 0.01
0.62 0.01
0.64 0.04
0.64 + 0.01
0.57 0.01
Yacht Hydrodynamics
(308,
0.06
0.67
1.58
0.60 + 0.07
0.41 0.04
Average Negative Log- Likelihood Test Performance
Boston Housing
(506, 13)
2.54 1: 0.08
2.40 1 0,04
2,41 1 0.25
2,63 +: 0.09
2,65 0.13
Concrete Strength
(1030, 8)
3.04 + 0.03
2,93 + 0.02
3.06 + 0.18
3.52 + 0.11
3.19 + 0.05
Energy Efficiency
(768, 8)
1.01 0.01
1.21
1
0.78 + 0.06
1
Kin8nm
(8192, 8)
-1.28 +
-1.14
-1.11 + -1.11 -1.20
-1.15
Naval Propulsion
(11934, 16)
-4.55 0.06
4.45
-5.63
-10.07 1 0.01
-10.01
Power Plant
(9568, 4)
2.78 0.01
2.80
2.79+
2.94 + 0.01
2.77 + 0.02
Wine Quality Red
(1588, 11)
0.97 0.01
0.93 0.01
0,94
-0.78 + 0.07
-0.98 + 0.06
Yacht Hydrodynamics
(308, 6)
1.64 0.02
1.25 0.01
1.18 0.21
0.49 + 0.06
1.07 = 0.27
where (yi,..., Ma) is sampled from the posterior for (z. and we used the independence of the posterior for each class Note that we also treat the predictions on different test points independently. In general, Eq. (6) does not have analytic expression, and resort Monte-Carlo estimation. We refer readers to the supplement for comparison to other heuristics (e.g. passing the mean predictor through softmax function and pairwise comparison). While this is heuristic,
find it is well calibrated (see Fig. 2 This is perhaps because the posterior Eq. ( still represents substantial model averaging, and most uncertainty high-SNR cases is epistemic rather than aleatory.
4.3 BENCHMARK ON CIFAR10
We examine the calibration of NNGP-R on increasingly corrupted images of CIFARIO-C (Hendrycks) Dietterich 2019) using the benchmark of Ovadia et al. (2019). The results are displayed Fig. While FC-NNGP is similar to the standard RBF kernel, C-NNGP outperforms both in terms of calibration and accuracy. Moreover, we find that at severe corruption levels, C-NNGP actually outperforms all methods in Ovadia et al. (2019) (compare against their Table G1) in BS and ECE.
0.7
Method
0.25
RBF
0.8
Acsuscy
0.20
FC-NNGP-R
0.5
0.15
C-NNGP-R
0.4
0.10
0.3
0.2
0.00
Valid
Test
2
Vald
Test
2
3
Corruption Intensity
Cornuption Intensity
Figure 2: Uncertainty metrics across shift levels on CIFARIO using NNGP-R. CNN kernels perform best as well as being more robust to corruption. See the supplement for numerical values at each quartile well as comparison to the NTK (Fig. (3).. All methods remain well calibrated for all intensities of shifts, with C-NNGP-R performing best, and significantly better than methods in Ovadia al. 2011); contrast against their Fig. 2 and $4.
'Implied directly by the NNGP's ' Kronecker structure and treating the lahels as independent targets.
7

---

Published as a conference paper at ICLR 2021
5 BAYESIAN OR INFINITE- WNFINITEWWIDTH LAST LAYER
As we have seen NNGP-C and NNGP-R are remarkably well-calibrated. However, obtaining high performing models can be computationally intensive, especially for large datasets. NNGP-C and NNGP-R have running times that are cubic in the dataset size, due to computation of the kernel's s Cholesky decomposition. with NNGP-C suffering additionally from potentially slow convergence of MCMC. Moreover, performant NNGP kernels require substantial compute to obtain (Novak et al. 2019b Arora et al. 2019; Novak et al.) 2019a Li et al. 2019) in contrast to training a NN to similar accuracies. Moreover, even though the most performant NNGP kernels are SotA for kernel method (Shankar et al. 2020), they still under-perform SotA NNs by a large margin. To combine the benefits of the NNGP and NNs, obtaining models that both performant and well calibrated, we propose stacking an infinite-widd witth sub-network on top of a pre-trained NN. More precisely, we use features obtained from a pre- pre-trained model as inputs to the NNGP. As such, the outputs of the combined network are drawn from a GP and we may use Eqs. ( and ( for inference. We refer to this as neural network Gaussian process last-layer (NNGP-LL). Mathematically, the model is
y~ f(g(z)) EE, where f ~ GP(.,
where g is pre-trained embedding, I. is the NNGP kernel, and (O, σ.) is a noise term. For the implementation of this method, we imagine that is an input to the algorithm. The computation of the kernel requires precomputing the embeddings of all data points, which are then considered as inputs to the NNGP-R algorithm, resulting in running time that is cubic in the training set size (see Sec. F The main advantage is that the embedding g allows for great expressive power which can be coupled with computationally cheaper kernels. This method draws inspiration from HHinton & Salakhutdinov (2008); Wilson al. (2066b); |Bradshaw et al. (217). but with a multi-layer NNGP kernel. Note we are specifically interested in the calibration properties and so the innovations in computational efficiency in Wilson et al. (001bb) are complementary to our work.
This setup mirrors important use case in practice, e.g. for customers of cloud ML services, who may use embeddings trained on vast amounts of coo--ommin-sseciiie model to their specific use case. This fine tuning consists of either fitting logisitic regression layer or deeper NNs using the embeddings obtained from their data, or perhaps training the whole NN generating the embedding by simply initializing with the pre-trained weights (Kornblith et al. 2019). These strategies allow practitioners to obtain highly accurate models without substantial data or computation. However, little is understood about the calibration of these transfer learning approaches.
data, and then fine-tune this
We consider the EfficientNet-B3 (Tan & Le 2019) embedding from TF-Hut and TF Keras Applic- tions that is trained on ImageNet (Deng et al. (200)), and perform our evaluations on CIFARIO 10 and its corruptions. We mainly use a multi-layer FC-NNGP as the top sub-network since its kernel very fast to compute and the final FC layer of EfficientNet-B3 removes any spatial structure that might be exploited by convolutions. However, we also explore other kernel types (linear [T CNNS with pooling, and self-attention layers) in Fig. by using earlier layers of EfficienNNet-B.
We compare our method with other popular last-layer methods for generating uncertainty estimates (Vanilla logisitic regression, using a deep NN for the last layers, temperature scaling (Platt et al.. 1999: Guo et al.) 2017). MC-Dropout (Gal & Ghahramani 2016). ensembles of several last- deep NNs). In the supplement we further investigate these results with a WideResNet (Zagoruyko train from scratch, using the initialization method in Dauphin & Schoenholz (2019), which achieves good test performance on CIFAR-10 without BatchNorm (loffe & Szegedy 2015). This allows to compare against the gold standard ensemble method.
& Komodakis 2016) that
We find that in the transfer learning case, ensembles are quite ineffective alone. The best previous method is given by combining MC-Dropout, temperature scaling, and ensembles. However, this is still bested by NNGP-LL (see Table 33. We also examine the effect of the fine-tuning dataset size on calibration and performance, Remarkably, NNGP-LL is able achieve accuracy and calibration comparable to ensembles with as few as 1000 training points.
ntttps:/ sttps://wwir censor flow org/hub
10 inttpe://wwer zensor Flow.oowwooroooooo wotg/ppi_dooooooo sa___ddoa/pyyhhhp phonn C/ C/keeas////eee pateras//ppl..e applications "In Riquelme et al. (2018). changing the last layer of a network to a Bayesian linear regression layer was useful for bandit tasks where calibrated uncertainty are important.
8

---

Published as a conference paper at ICLR 2021
1.0
LINEARELL
0.4
FO-NGGP LL
Acciracy
CANGINLL
0.8
EGB
SELF-ATIN-RRGGLL
0.2
0.4
0.0
0.2
Valid
Test
5
Valid
Test
Corupttio Intensity
Corruption Intersity
Figure 3: Uncertainty metrics across shift levels on CIFARIO using NNGP-LL with EfficientNNt-B3 embedding. We observe that more complex neural kernels (convolution and self-attention NNGPs) can have higher performance while being more robust corruptions compared replacing last layer to Bayesian linear regression (Riquelme et al. 2018). See Fig, for the comparison to the NTK.
Table 3: NNGP-LL with EfficientNet-B3: fine tuned on CIFARIO as an embedding and evaluated over all CIFAR1O 10 corruptions and intensities. We show the quartiles over all corrupted variants for several different last-layer methods of obtaining uncertainties. Vanilla denotes fine tuned network, Ensembles refers to Lakshminarayanan et (2017) and Ens/Drp/l refers combining Ensembles, MC-Dropout (Gal & Ghahramani] (016) and temperature scaling (Guo et al. 2017). Center column baseline where last-layer replaced with an RBF kernel instead of NNGP. Right column groups are NNGP-LL results. More specifically, we train a three-layer FC network with dropout using imput from the embedding from EfficientNet-BB and report results in the middle columns. The rightmost columns show that NNGP-LL can be well-calibrated with very little training data (100, 1K, 5K, 10K, and full dataset are shown). See Fig. S5 for fine-grained box plots for each corruption level.
Method
Vanilla
Ensembles
Ens/DrpTT
RBF-IL
100
IK
5K
10K
NNGP-LL
Brier Score (25th)
0.230
0.218
0.182
0.178
0.363
0.256
0.218
0.203
0.173
Brier Score (50th)
0.351
0.331
0.265
0.274
0.448
0.346
0.308
0.288
0.271
Brier Score (75th)
0.521
0.511
0.410
0.392
0.572
0.478
0.436
0.409
0.397
NLL (25th)
0.913
0.823
0.382
0.389
0.797
0.562
0.474
0.455
0.367
NLL (50th)
1.517
1.411
0.569
0.597
0.991
0,746
0.682
0.655
0,586
(75th)
2.662
2,492
0.932
0.880
1.326
1.075
0.972
0.921
0.905
ECE (25th)
0.104
0.098
0.016
0.020
0.023
0.044
0.018
0.019
0.017
ECE (50th)
0.160
0.154
0.028
0.030
0.040
0.062
0.023
0.027
0.025
ECE (75th)
0.247
0.243
0.079
0.066
0.081
0.104
0.042
0.057
0.044
Accuracy (75th)
0.869
0.875
0.875
0.879
0.742
0.825
0.851
0.860
0.884
Accuracy (50th)
0.802
0,812
0.813
0.809
0.674
0.758
0.784
0.798
0.813
Accuracy (25th)
0.714
0.719
0.718
0.719
0.582
0.663
0.689
0.704
0.722
DISCUSSION
In this work, we explored several methods that exploit neural networks' implicit priors over functions in order to generate uncertainty estimates, using the corresponding Neural Network Gaussian Process (NNGP) as a means to harness the full distribution over function-space in the infinite-width limit, Using the NNGP, and also examined heuristics for generating confidence estimates when classifying via regression. Across the board, we found that the NNGP provides good uncertainty estimates and generally delivers well-calibrated models even on OOD data. We found that NNGP-R is competitive with SOTA methods on the UCI regression task and remained calibrated Despite their good calibration properties, as pure kernel methods, NNGP-C and NNGP-R cannot always compete with modern NNs terms of accuracy. Adding an NNGP to the last-layer of pre-trained model (NNGP-LL). allowed us to simultaneously obtain high accuracy and improved calibration. Moreover, we found NNGP-LL to be simple and efficient way to generate uncertainty estimates with potentially very little data, and that it outperforms all other last-layer methods for generating uncertainties we studied. Overall, we believe that the infinite-width limit provides a promising direction to improve and better understand uncertainty estimates for NNs.
performed fully Bayesian classification (NNGP-C) and regression (NNGP-R)
for severe levels of corruption.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENT
We would like to thank Roman Novak, Jascha Sohl-Dickstein, and Florian Wenzel for insightful discussions and feedback.
REFERENCES
James H Alhert and Siddhartha Chib. Bayesian analysis of binary and polychotomous response data. Journal of the American statistical Association, 88(422):669-00 1993.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with infinitely wide neural net. In Advances in Neural Information Prucessing Systems. 2019.
John Bradshaw, Alexander G de Matthews, and Zoubin Ghahramani. Adversarial examples, uncertainty, and transfer testing robustness in gaussian process hybrid deep networks. arXiv preprint arXiv:1707.02477 2017.
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 1--1).1-3. 1950.
Yann N. Dauphin and Samuel S. Schoenholz. Metainit: Initializing learning by learning to initialize. In Advances in Neural Information Processing Systems, 2019.
J. Deng, W. Dong, R. Socher, L.. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Puttern Recognition, pp. 248-255, 2009.
Simon Du, Kangcheng Hou, Russ R Salakhutdinov, Barnabas Poczos, Ruosong Wang, and Keyulu Xu. Graph neural tangent kernel: Fusing graph neural networks with graph kernels. In Advances in Neural Information Processing Systems, pp. 5724-5734, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout a bayesian approximation: Representing model uncertainty in deep lcarning. In international conference on machine learning, pp. 1050-1059, 2016.
Adrià Garriga-Alonse Laurence Aitchison, and Carl Edward Rasmussen. Deep convolutional networks as shallow gaussian processes. In International Conference on Learning Representations, 2019.
Mark Girolami and Simon Rogers. Variational bayesian multinomial probit regression with gaussian process priors. Neural Computation, 1898):1790-1888 1817, 2006.
Tilmann Gneiting and Adrian E Raftery. Strietly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-778 2007.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D Sculley. Google vizier: service for black-box optimization. Proceedings of 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1487-1495. ACM, 2017a.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D Scullcy. Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1487-1495, 2017b.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, 2017.
Trevor Hastie and Robert Tibshirani. Classification by pairwise coupling. In Advances in neural information processing systems, pp. 507-513, 1998.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference on Learning Representations, 2019.
José Miguel Hernández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International Conference Machine Learning, pp. 1861-1869, 2015.
Geoffrey E. Hinton and Ruslan R Salakhutdinov. Using deep helief nets to learn covariance kemness for gaussian processes. In Advances in neural information processing systems, pp. 1249-1256, 2008.
10

---

Published as a conference paper at ICLR 2021
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite width attention networks. In International Conference on Machine Learning (ICML), 2020.
Like Hui and Mikhail Belkin. Evaluation neural architectures trained with square loss vs cross-entropy in classification tasks, arXiv preprint arXiv:2006.07:22 2020.
Sorgey loffe and Christian Szegedy. Batch normalization: Accelcrating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 1502.03167 2015.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, 2018.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2262-266271 2019.
Balaji Lakshminarayanan. Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances Neural Information Processing Systems. 2017.
Jachoon Lec, Yasaman Bahri, Roman Novak, Sam Schoenholz, Jeffrey Pennington, and Jascha Sohl-dickstein. Deep neural networks as gaussian processes. In International Conference on Learning Representations, 2018.
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advonces in neural information processing systems, pp. 8570-8581, 2019.
Jaehoon Lee. Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: empirical study. Advances in Neural Information Processing Systems, 33, 2020.
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Art Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02211 2020.
Zhiyuan Li, Ruosong Wang, Dingli Yu, Simon S Du, Wei Hu, Ruslan Salakhutdinov, and Sanjecv Arora. Enhanced convolutional neural tangent kernels. arXiv preprint arXiv: 1910090,,,,,,,,,, 2019.
David J.C. MacKay. The evidence framework applied to classification networks. NEURAL COMPUTATION, 4: 720-736, 1992.
Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. arXiv preprint 1804 11271, 2018a.
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard F. Turner, and Zoubin Ghahramani. Gaussian
process behaviour in wide deep neural networks. International Conference on Learning Representations, 2018b. URL /ttttss://ppeeeee///
net fooum? id=H1 H1---nnhhh -GGgCC
Norman Mu and Justin Gilmer. Mnist-c: A robustness benchmark for computer vision. arXiv preprint arXiv: 02337, 2019.
Jishnu Mukhoti, Pontus Stenetorp, and Yarin Gal. On the importance of strong baselines in bayesian deep learning. arXiv preprint Xrviv 1.09385, 2018.
lain Murray, Ryan Prescott Adams, and David JC MacKay, Elliptical slice sampling. 2010.
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Radford M. Neal. Priors for infinite networks (tech. rep. no. crg-tr-94-1). University of Toronto, 1994a.
Radford M. Neal, Bayesian Learning for Neural Networks. PhD thesis, University of Toronto, Dept. of Computer Science, 1994b.
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander Alemi, Jascha Sohl-Dickstein, and Samuel S Schocnholz. Neural tangents: Fast and casy infinite neural networks in python. arXiv preprint arXiv: 1et.. 2019a.
Roman Novak, Lechao Xiao, Jachoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. International Conference Learning Representations, 2019b.
11

---

Published as a conference paper at ICLR 2021
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your modcl's uncertainty? evaluating predictive uncertainty under dataset shift. In NeurlPS, 2019.
John Platt et al. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances Neural Information Processing Systems, pp. 3360- 3368, 2016.
Arnu Pretorius, Herman Kamper, and Steve Kroon, On the expected behaviour of noise regularised deep neural networks as gaussian processes, arXiv preprint Stti::9910.05660 2019.
Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence. Dataset Shift Machine Learning. The MIT Press, 2009. ISBN 70051.
Ryan Rifkin and Aldeharo Klautau. In defense of one-vs-all classification. Journal of machine learning research, 101an):001141,, 2004.
Ryan Rifkin, Gene Yeo, Tomaso Poggio, et al. Regularized least-squares classification. Nato Science Series Sub Series III Computer and Systems Sciences, 190:131-154, 2003.
Carlos Riquelme, George Tucker, and Jasper Snock. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. In International Conference Learning Representations, 2018.
Samuel Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. International Conference on Learning Representations, 2017.
Vaishaal Shankar, Alex Chengyu Fang, Wenshuo Guo, Sara Fridovich-Keil, Ludwig Schmidt, Jonathan Ragan- Kelley, and Benjamin Recht. Neural kernels without tangents, ArXiv, 0es/2003.0223377 2020.
Shengyang Sun, Changyou Chen, and Lawrence Carin. Learning structured weight uncertainty in bayesian neural networks. In Artificial Intelligence and Statistics, pp. 1292, 2017.
Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 1946, 2019.
Gia-Lac Tran, Edwin V Bonilla, John P Cunningham, Pietro Michiardi, and Maurizio Filippone. Calibrating deep convolutional gaussian processes. arXiv preprint 1805 180.102205 2018
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. volume 2. MIT press Cambridge, MA, 2006.
Andrew G Wilson, Zhiting Hu, Russ R Salakhutdinov, and Eric P Xing. Stochastic variational deep kernel learning. In Advances in Neural Information Processing Systems, pp. 2586-2594, 60.
Andrew Gordon Wilson. The case for bayesian deep learning. arXiv preprint arXiv:2001.10995 2020.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric Xing. Deep kernel learning. Artificial Intelligence and Statistics, pp. 370-378, 2016b.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstent Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks. In International Conference on Machine Learning. 2018
Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. In Advances in Neural Information Processing Systems. 2017.
Greg Yang. Tensor programs Wide feedforward or recurrent neural networks of any architecture are gaussian processes. arXiv preprint arXiv:1910.12478, 2019.
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl Dickstein, and Samuel Schoenholz. A mean field theory of batch normalization. arXiv preprint arXiv:1902.08 08299999..........000909992922 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference, 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv 2017.
12

---

Published as a conference paper at ICLR 2021
Supplementary Material
A DETAILED DESCRIPTION OF THE NNGP AND NTK
In this section, we describe the FC-NNGP and the C-NNGP, Most of the contents are adopted from Lee et al. [ Novak et al. 2019b); Lee et al. (2019), which we refer readers to for more technical details.
{x (a.y) E D} and y/ {v: : (z.y) E D) denote the inputs and labels, respectively. Consider follecconnected ford-forward network with L hidden layers with widths ni, for = 1,... L and readout layer with #*+1 = K. For each E "R"", we use h (x), a² (x) E Th to represent the pre- and post-activation functions at layer with input x. The recurrence relation for a feed-forward network is defined as
NNGP: Let D S R R denote the training set and
hh4t z'W'+1 + D
W b/
w/y /mi
and
(S1)
a441 = o (1++1
0
where ø is a point-wise activation function, W R and 6 E R 1 are the weights and biases, why and 6ÿ are the trainable variables, drawn i.d.d from a standard Gaussian wly, o4 10, at initialization, and o. and of are weight and bias variances.
As the width of the hidden layers approaches infinity, the Central Limit Theorem (CLT) implies that the outputs at initialization {f(==) converge to a multivariate Gaussian in distribution. Informally, this occurs because the pre-activations at each layer weights and bias), and thus become a Gaussian random variable themselves. See Poole et al.) (2016); Schoenholz al. (2017): [ee al. 20188: Xiao et al. (2018); Yang & Schoenholz (2017) for more details, and Matthews et al. (2018a) 8a) Novak et al. (001b) for a formal treatment.
sum of Gaussian random variables (the
Therefore, randomly initialized neural networks are in correspondence with a certain class of GPs (hereinafter referred to as NNGPs), which facilitates a fully Bayesian treatment of neural networks (Lee et al. 2018; Matthews et al. 2018b). More precisely, let f denote the i-th output dimension and K. denote the so-plel kernel function (of the pre-activation) of the outputs in the infinite width setting,
lim Kid (x,,x)) 2
IE (C)( (=)')
(S2)
then f(X²) ~ ((X, where Kid (xc, x²) denotes the covariance between the i-th output of x and j-th output of x', which can be computed recursively (seeLeee et al. (2018) $2.3). For test input x E Xr. the joint output distribution / ([z, X)) is also multivariate Gaussian. Conditioning on the training samples, f(x)= = V, the distribution of fx)) X,
)/ is also a Gaussian N' (μ(z), of (x)),
p)x)= K(x,, Y,, o² (x) K(x, .)) K(x, - x)², and where K. = x) This is the posterior predictive distribution resulting from exact Bayesian inference in an infinitely-we neural network.
(S3)
The above arguments can be extended to convolutional architectures Novak et ( (019b)) By taking the number of channels in the hidden layers to infinity simultaneously. the outputs of CNNs also converge weakly to a Gaussian process (C-NNGP). The kernel of the C-NNGP takes into account the correlation between pixels in different spatial locations and can also be computed exactly via a recursively formula; e.g., see Eq, (7) in (Novak et al., 2019b $2.2). Note that for convolutional architectures, there two canonical ways of collapsing image-shaped data into logits. One is to vectorlize the image to a one-dimensional vector (CNN-Ver and the other is to apply a global average pooling to the spatial dimensions (CNN-GAP). The kernlls induced by these two approaches are very different and so are the C-NNGPs. We refer the readers to Section 3.2 of (Novak et al., [019b] $3) for more details. In this paper, we have focused mostly on vectorization since is more efficient to compute.
C-NNGP:
There is also a correspondence between self-attention mechanisms and GPs. Indeed, for multi-head attention architectures, the number of heads and the number of features tend to infinity, the outputs of an attention model also converge to a GP (Hron et al. 2020). We refer the readers to Hron al. (2020) for technical details.
ANTN-NNGP:

---

Published as a conference paper at ICLR 2021
NTK: When neural networks are optimized using continuous gradient descent with learning rate on mean squared error (MSE) loss, the function evaluated on training points evolves as
Oefe(x)) = (1)(()) (f(()) -y)
(S4)
where (X) is the Jacobian of the output fa evaluated at X' and O, = (x)) is the Neural Tangent Kernel (NTK). In the infinite-wied -iitt limit, the NTK remains constant (θ; = +)
throughout training (Jacot et al. 2018). Thus the above equation is reduced to constant coefficient
ODE
0,5((2) = - (
(S5)
and the time-evolution of the outputs of unseen input X7 can be solved in closed form Gaussian with mean and covariance
= (1-
y
(S6)
E(Ar, Nr) KC (2r, Xr) 0(xr, 210 e--e- K c e81) 6 0 (X' Xt) () ee-e K (7) h...).
(S7)
Note that the randomness of the solution is the consequence of the random initializaaton f(X) N(O, K((,
Finally, the above arguments do rely the choice of architectures and could likewisely define CNTK and ATTN-NTK, the NTK for CNNs and attention models, respectively.
2

---

Published as a conference paper at ICLR 2021
B ADDITIONAL FIGURES FOR NNGP-C
In this section, we show some additional plots and results comparing NNGP-C against standard NNs. Recall that the NNGP-C model inference was carried out by elliptical slice sampling ( (Murray et al. (010), which is hyperparameter free, but there are hyperparameter associated with the NNGP and NNs. Here, we mainly address the method of hyperparameter tuning considered the main text, where fixed the hyperparameters that common to both the NNGP and the NN, then only tuned the additional NN hyperparameters. We show results for tuning all of the NN's hyperparameter from scratch.
Additional Tuning details. For any tuning of hyperparameters, split the original training set of CIFARIO into a 45K training set and a 5K validation set. All models were trained using the 45K points, and we then selected the hyperparameters from the validation set performance. We introduced a constant that multiples the whole NNGP kernel, or equivalently scales the whole latent space vector the last layer bias and weight standard deviations FC-NNGP-C, the activation function was tuned over {ReI.U. erf), the weight standard deviation was tuned over [0.1, 2.0] on linear scale, the bias standard deviation was tuned over [0.1, 0.5] on linear scale, the kernel scale was tuned over [10-2 100] on logarithmic scale, the depth was tuned over {1, 3, 4, 5). and the diagonal regularizer was tuned over (0., 0.01] on a linear scale.
called this constant the kernel scale. For
For the FC-NN, there are additional hyperparameters: the learning rate, training steps, and width. For the NN, we considered two types of tuning. Either, as in the main text, the hyperparameters that are shared with the NNGP are fixed and the additional hyperparameters are tuned, or, as we present in the supplement, all of the NN's hyperparameters are tuned from scratch. In either case, the activation, the weight standard deviation, the bias standard deviation, the kernel scale, and the depth were tuned as above. The learning rate was tuned over [10- -4, 0.1] on a logarithmic scale, the total training steps was tuned over [10*, 107] logarithmic, and the width was tuned over {64, 128, 256, 512].
For the C-NNGP-C, all hypermaramters were treated as for the FC-NNGP-C case, except depth which was limited {1, 2}. For the CNN, we again considered the two types of tuning: either fixing common hyperparameters or retuning all hyperparameters from scratch. The CNN's learning rate tuned over [10-4, 0.1] logarithmic scale, the total training steps tuned over [217 222 on a logarthimic scale, and the width was tuned over {64, 128, 256, 512}.
l'Our implementation of ESS in JAX that is suitable for GPUs and TPUs can be found here: ht:pa: 7/git ttsb.tho.//oooo..... moon/googii www.gooooooooooooooooooooooooooe p-reearrhhgoog /ttpo//aaa/aa///ee/ea////////// untorrrtttt
3

---

Published as a conference paper at ICLR 2021
1.0
8-
NNGP-C: CIFARIO
1.0
FC FC-NN: CIFARIO
NNGP-C: SVHN
Average accuror
0.8
Clcan
Fog
Deraity
FC EC:NN:55 SVHN
0.6
Acc.
Fog
ECE
0.6
0.4
Fog
0.2
0.0
0.2
0.4
0.6
0.8
1.0
Clean
2
Corruption strength
NNGP C: CIFARIO
1.0
NNGP-C SVHN
FC-NN: CIFARIO
0.8
Averace accuracy
0.8
Clean
Fog
Density
EC:NN: SVHN
0.6
Acc. ECE
0.6
4-
B5
0.4
Fog
0.2
0.0
0.2
0.4
0.6
0.8
1.0
Clean
2
0.2
0.8
Corruption strength
Deneity
C NNGP SVHN CNN: CIFAR10
CNN: SVHIN
C-NNGP-C: CIFARIO
1.00
0.75
Acc.
BS
ECE
Average accuracy
0.8
Fog
Glean
Fog
0.6
Fog
Fog
0.4
Fog
2-
0.25
0.2
0.00
0.2
0.4
0.6
0.8
1.0
Clean
5
0.0
0.2
0.4
0.6
0.8
1.0
Confidence
Corruption strength
Confidence
Figure S1: Investigating the calibration of Gaussian process classification with NNGP kernels as in Fig.[] where find similar results. (left column) Histogram of the confidence of the posterior distribution for each test point. We compare the NNGP-C and finite width NN on an identically distributed test set (CIFARIO) and OOD test (SVHN). (middle column) Performance, NNGP- solid and NN dashed, under increasing distributional shift given by the CIFAR 10 fog corruption. (right column) We bin the test set into bins sorted by confidence. and we plot mean confidence against mean accuracy. (top row) We compare FC-NNGP-C against a FC-NN with the same hyperparameters. (middle row) We compare FC-NNGP-C against FC-NN, where all of the NN's hyperparameters are tuned independently with Vizier. (bottom row) We compare C-NNGP-C against a CNN, where all of the CNN's hyperparameters are tuned independently with Vizier.
4

---

Published as a conference paper at ICLR 2021
2.5
2.5
NNGP-C: CIFAR1O
NNGP-C: CIFARIO
NNGP-C: SVHN
NNGP-C C: SVHN
2.0
2.0
FC-NN: CIFARIO
FC-NN: CIFARIO
Density
1.5
FC-NN: SVHN
Density
1.5
FC-NN: SVHN
1.0
1.0
0.5
0.5
0.0
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Entropy
Entropy
3.0
3.0
C-NNGP-C: CIFAR10 C-NNGP-C: SVHN CNN: CIFAR10
C-NNGP-C: CIFAR10
2.5
2.5
C-NNGP-C: SVHN
CNN: CIFAR10
2.0
CNN: SVHN
2.0
CNN: SVHN
Density
1.5
De
1.5
1.0
1.0
0.5
0.5
0.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0 Entropy
1.5
2.0
Entropy
Figure S2: We compare NNGP-C and finite width NN identically distributed test set (CIFAR10) and an OOD test set (SVHN). We plot a histogram of the entropy of the predicted distribution for each example in the set. (top row) Fully connected network architectures. (bottom row) Convolutional architectures. (Ieft column) Same hyperparamers, (right column) All of the CNN's hyperparameters are tuned independently.

---

Published as a conference paper at ICLR 2021
Table S1: Performance of NÑs, where all of the NN's hyperparameters are tuned independently with Google Vizier hyperparameter tuner (Golovin al. 2017a). In addition we show results on all Fog corruptions that were omitted from Table
Data
Metric
FC
CNN
(same)
NN (all tuned)
NNGP-C
NN (same)
NN tuned)
NNGP-C
ECE
0.209
0.217
0.072
0.283
0.265
0.031
Brier Score
0.711
0.721
0.629
0.685
0.605
0,519
CIFARIO
Accuracy
0.486
0.478
0.518
0.576
0.632
0.609
NLL
17383
17967
14007
21786
21160
11215
ECE
0.178
0.187
0.098
0.271
0,258
0.042
Brier Score
0.707
0.726
0.655
0.685
0.618
0.542
Fog
Accuracy
0.471
0.458
0.496
0.561
0.617
0.590
NLL
16702
17372
14671
19716
19484
11755
ECE
0.177
0.189
0.073
0.286
0.287
0.018
Brier Score
0.758
0.781
0.711
0.747
0.712
0.620
Fog 2
Accuracy
0.416
0.396
0.425
0.507
0.552
0.519
NLL
18026
18930
16233
20822
21657
13819
ECE
0.202
0.215
0.039
0.320
0.333
0.03
Brier Score
0.822
0.846
0,759
0.836
0.816
0.694
Fog 3
Accuracy
0.358
0.336
0.363
0.445
0.480
0.445
NLL
20092
21247
17638
23752
25703
16019
ECE
0.236
0.241
0.026
0.352
0,381
0.071
Brier Score
0.881
0.902
0.797
0.914
0.924
0.758
Fog
Accuracy
0.307
0.292
0.311
0,385
0.409
0.380
NLL
22589
23746
18867
27055
30844
18353
ECE
0.279
0.282
0.057
0.420
0.472
0.134
Brier Score
0.961
0.975
1.052
1.104
0.846
Fog 5
Accuracy
0.241
0,232
0.250
0.287
0,301
0.289
NLL
26345
27493
20694
33891
41058
22014
Conf.
0.537
0.542
0.335
0.718
0.794
0.463
SVHN
Entropy
1.230
1.208
1.840
0.733
0.524
1.403
Conf.
0.651
0.654
0.398
0.812
0.847
0.474
CIFAR100
Entropy
0.944
0.930
1.663
0.493
0.394
1.420

---

Published as a conference paper at ICLR 2021
C COMPARISON OF HEURISTICS FOR GENERATING CONFIDENCES FROM NNGP-R
In Secs. Haand we utilized heuristic to generate confidence from exact GPR posterior distribution. Here denote the heuristic described in Eq. (6) as exact confidence, which is the probability of class probit being maximal under an independent multivariate Gaussian distribution. We consider two more heuristics. One is denoted pairwise, where take confidence to be proportional to the probability that the i-th class probit is larger than other probits in pairwise fashion, i.e.
ox 2z. #
II p(u: X,)) j
II j
Mi 0
(S8)
+
where (-) is Gaussian cumulative distribution function. In order to obtain confidence, normalize by the sum so that the heuristic confidence sums up to 1. This is following the one-vs-one multiclass classification strategy Hastie & Tibshirani (998).
We note that, we introduce temperature scaling with temperature T by replacing posterior variances of = To².
(S9)
Another heuristic denoted softmax, where we apply the softmax function to the posterior mean:
:=00):=((1)) ets/ /
(S10)
eusiiV
In this case, the posterior variance is not used to construct the heuristic confidences.
comparison for these dhree-different heuristics for C-NNGP-R is shown in Fig. S4 with and without temperature scaling. We note that exact and pairwise heuristics remain well calibrated without temperature scaling. However with temperature scaling the softmax heuristic can competitive to other heuristics. In Sec. and 5 focused the exact heuristic.
Table $2: Quartiles of Brier score, Segative-log-llikeiboee and ECE over all CIFAR10 corruptions for methods in Fig.
Method/Metric
RBF
FC-NNGP-R C-NNGP-R
Myrtle-NNG--R
Brier Score (25th)
0.568
0.569
0.435
0.330
Brier Score (50th)
0.580
0.586
0.464
0.460
Brier Score (75th)
0.599
0.613
0.515
0.760
Gaussian NLL (25th)
0.147
0.612
0.108
0.763
Gaussian NLL (50th)
0.270
0.830
0.457
2.740
Gaussian NLL (75th)
0.447
1.099
1.027
6.933
NLL (25th)
1.331
1.351
1.002
0.783
NLL (50th)
1.363
1.398
1.079
1.050
NLL. (75th)
1.415
1.466
1.178
1.773
ECE (25th)
0.048
0.030
0.022
0.035
ECE (50th)
0.052
0.039
0.046
0.085
ECE (75th)
0.069
0.065
0.071
0.203
Accuracy (75th)
0.573
0.574
0.683
0.770
Accuracy (50th)
0.566
0.561
0.659
0.678
Accuracy (25th)
0.549
0.541
0.628
0.432

---

Published as a conference paper at ICLR 2021
0.7
IT
0.6
Accuracy
0.5
Method
0.4
RBF
C-NNGP-R
FC-NNGP-R
C-NTK-R
0.3
FC-NTK-R
Valid
Test
2
3
4
Corruption Intensity
0.3
ECE
0.2
0.1
Valid
Test
Corruption Inlensity
1.0
0.9
Brier Score
0.8
0.5
0.4
Valid
Test
Corruption Intensity
Negative Log Likellhood
Valid
Test
Corruption Intensity
Figure S3: Uncertainty metrics across shift levels CIFAR10 using NNGP-R and NTK-R. CNN kernels perform best well as being more robust to corruption. The NTK corresponding the same architecture shows similar robustness properties.
8

---

Published as a conference paper at ICLR 2021
NNGP-R Confidence Metrics Temperature Scaling
MNGP-R A Confidence Matics without Temperature Booling
0.50
0.1
Valid
Test
Test
Comupttin Intensly
Comuptinn Inteniity
Mathod
0.8
EXECT
Breer
0.4
Valid
Teat
3
Vaiid
Test
2
Comuption Intansity
Compuption Intensity
Figure S4: Comparison of NNGP-R based confidence measures on CIFAR10 corruptions using C-NNGP-R. Left column uses temperature scaling based on validation whereas right column uses T = 1. We see that softmax confidence requires adjusting temperature which is equivalent to modifying prior variance be calibrated whereas exact and pairwise heuristic confidence calibrated without needing to modify the prior variance.

---

Published as a conference paper at ICLR 2021
D ADDITIONAL FIGURES FOR NNGP-LL
The results in the main text focused a fixed embedding (see Table and show additional results for this as a box-plot in Fig. $S here. However, it is also common in practice to tune all the embedding's 's weights and simply initialize their pre-triained values. We explore this setting in the supplement in Table S3 and Fig. [S6]by considering the EfficientNet-B3 embedding and fine tuning it CIFARIO. We also show results comparing the FC-NNGP-LL with using the standard RBF kernel for the same purpose (see Fig. S7
1.0
0.8
Accunecy
Mothod
0.6
DRP
0.4
R3F
NNGP
100
Vald
Test
Comuuttin Intensity
0.8
0.6
0.4
0.2
0.0
Vald
Test
Comuuiton Intensity
Biter Score
1.0
0.5
Vald
Test
Coruuption Intansity
Figure S5: Uncertainty metries across corruption levels on CIFARIO using NNGP-LL with EfficientNet-B3 embedding. Baseline NNs are trained on CIFARIO with parameters of body network fixed. See Table for quartile comparison and Fig. 166and Table S3 for comparison with fine tuning of all the embedding's weights. Fig. S7 compares use of NNGP and RBF kernel for NNGP-LL settings.
D.1 A COMPARISON OF ENSEMBLE AND NNGP-LL ON WIDERESNET
To compare the NNGP-LL method against the gold standard ensemble method, we train a WideResnet 28-10 on CIFAR-10 from scratch with different random initialization. The model is trained using Metalnit (Dauphin & Schoenholz 2019). Delta-Orthogonal (Xiao et al. (018)), mixup (Zhang et al. 2017) and without BatchNorm (loffe & Szegedy 2015). The model achieves about 94% accuracy on the clean test set. See Table (S4 and Fig. [S8 for the comparison. We find the ECE for NNGP-LL is very competitive, even with small dataset size, compared baseline methods including ensembles.
E COMPLEXITY OF NNGP-LL
There are two main steps in applying NNGP-LL. One is computing the NNGP kernel K. and the other is inference. For NNGP-LL, we note that the computational of the kernel has the added cost of inference using the pretrained NN on the whole dataset, which adds cost that is linear in dataset size but with a potentially large constant factor. For inference, we need to invert the whole kernel matrix of size c|X' x c. the computational complexity is usually ((e 2x))) ((2))x113) where e is the number of classes and | is the training set size. However, since has a Kronecker factorization, the complexity is reduced to (()2))).
10

---

Published as a conference paper at ICLR 2021
Table S3: Quartiles of Brier score, negative-log- log-iiilioooodd and ECE over all CIFARIO corruptions for methods in Fig. usjugin EfficientNet-Bi embedding. The first five columns are the same as the methods in Table whereas the last two columns use the same embedding architecture but where all the weights are fine tuned (FT) CIFARIO. 10. See Fig. S6 for the corresponding box-plot.
L.I. T Scaling
Ens Drp
I.I.
LL Ens
NNGP-IL
FT-NN FT-NNGP-LIL
Brier Score (25th)
0.230
0.191
0.218
0.182
0.173
0.083
0.081
Brier Score (50th)
0.351
0.281
0.331
0.265
0.271
0.155
0.153
Brier Score (75th)
0.521
0,422
0.511
0,410
0.397
0.341
0.361
NLL (25th)
0.913
0.406
0.823
0.382
0.367
0.166
0.196
NLL (50th)
1.517
0.612
1.41
0,569
0.586
0.320
0.374
NLL (75th)
2.662
0,988
2.492
0.932
0.905
0.730
0.868
ECE (25th)
0.104
0.016
0.098
0.016
0.017
0,005
0.012
ECE (50th)
0.160
0.030
0.154
0.028
0.025
0.015
0.022
ECE (75th)
0.247
0,092
0.243
0.079
0.044
0.051
0.046
Accuracy (75th)
0.869
0.869
0.875
0.875
0.884
0,945
0.947
Accuracy (50th)
0.802
0.802
0.812
0.813
0.813
0.894
0.896
Accuracy (25th)
0.714
0.714
0.719
0.718
0.722
0.758
0.748
1.0
Accurscy
0.8
Method
0.6
NNGP-LL
0.4
ENS
FT
Test
Cerruption Intensity
0.8
0.6
ECE
0.4
0.2
0.0
Valid
Test
Corruption Intenaity
Score
1.0
0.5
0.0
Valid
Test
Corruption Intensity
Figure S6: Uncertainty metrics corruption levels on CIFARIO using NNGP-LL with EfficientNet-B3 embedding. Baseline NNs are either last layer (LL) trained on CIFARIO with parameters for body networks fixed or where all the weights are fine tuned (FT). Quartile comparisons can be found in Table (S3
The complexity of computing the NNGP kernels is architecture dependent, For notational simplicity, assume the cardinality of the training and test sets are and n.
11

---

Published as a conference paper at ICLR 2021
Method
Aocursyy
0.8
REFOPLLLR
NMGPLL-
0.6
Brier
0.5
0.4
0.0
0.0
Vaiid
Tasz
Vali
Tist
Vald
Test
Commuttion Intensity
Cormuttion Intonsity
Comuption intersity
Figure S7: Comparison of NNGP-LL using NNGP head vs RBF GP head on EfficientNet-B3 embedding. While there are slight advantage of using NNGP head over RBF-GP overall they provide very similar benefits as corruption intensity increase.
1.0
0.8
Acouncy
0.6
Mathod
0.4
VANILAA
NNGP-LL 100
0.2
ENSEMBLES
SCCLINN
NNGP LL 1K
Vali
Test
Coruuttin Intensity
0.8
0.6
868
0.0
Coruptiin Intensity
Biler Score
1.0
0.0
Tost
Cormuption Intansily
Figure S8: Uncertainty metrics across corruption levels on CIFAR10 using NNGP-LL with Metalnit embedding. Baseline NNs are compared with vanilla training, temperature scaling and ensembles.
E. FCN
The FCN kernel is relatively cheap to compute since it depends only on the dot product between inputs. It takes O(L(mn + n²))) time, where L is the number of layers of the FCN. In practice, the whole CIFAR-10 kernel could obtained within minute.
E.2 CNN-VEC
The CNN-VEC kernel is slightly more expensive compute. Each entry K(I,x²) x²) is function of (I., <')) : E [d]} where d is the total number of pixels in one image (e.g. d = 32 * 32 for CIFAR- 10). Computing K(x,, æ²) takes O(Ld) time and the whole kernel takes O(mmn + n Ld) time. In practice, the whole CIFAR-10 10 kernel could be obtained in about 80 minutes.
E.3 CNN-GAP AND ATTENTION
CNN-GAP and Attention kernels are very expensive to compute. Each entry KC(x, x²) is a function of ( 22,) b [d]. The complexity for one entry is O(d L) and O(mm + n dd² L for the whole kernels. For a 10 layers Myrtle kernel, it takes about 400 GPU hours (using NVIDIA Tesla V100) to compute the upper triangle part for the whole CIFAR10 10 (where d 322). However, for the last layer
12

---

Published as a conference paper at ICLR 2021
Table S4: Quartiles of Brier score, gegative-lo-likellnne ECE and accuray over all CIFARIO
corruptions for Metalnit Embedding NNGP-LL and Metalnit trained networks in Fig. S8
Method/Meetic
Vanilla
T Scaling
Ensembles
100
IK
NNGP-LL
Brier Score (25th)
0.165
0.164
0.141
0.256
0.162
0.152
Brier Score (50th)
0.247
0.244
0.210
0.366
0.257
0.241
Brier Score (75th)
0.397
0.410
0.356
0.567
0.415
0.399
NLL (25th)
0.382
0.391
0.328
0.562
0.381
0.360
NLL (50th)
0,553
0.576
0.483
0.799
0.599
0.570
NLL (75th)
0.864
1.040
0.781
1.214
0.967
0.925
ECE (25th)
0.046
0.025
0.044
0.018
0.011
0.012
ECE (50th)
0.062
0.051
0.066
0.044
0.017
0.017
ECE (75th)
0.079
0.126
0.077
0.101
0.030
0.039
Accuracy (75th)
0.895
0.895
0.916
0.824
0.889
0.896
Accuracy (50th)
0.840
0.840
0.866
0.738
0.821
0.834
Accuracy (25th)
0.726
0.726
0.761
0.586
0.703
0.716
06
LINDAR-LL.
Vettod
LINEA--L
Nachod
0.4
POWNOPLLL
24-
PONISAL
Tal:
0.0
0.0
0.0
0.0
0.4
0.4
0.2
Companntturrtl
Gomuzion
5000
0.75
2.77
0.00
0.00
0.6
0.05
Compton
Comptton Imoondly
Compattinn Iraerdit
Figure S9: Comparison between NNGP-LL and NTK-LL with varying embeddings and neural kernels.
method, the images are down- lown-sampled to smaller 8 x 8 image-patches. The whole kernel could be obtained about 400/(44) 1.6 GPU hours.
13

---

Published as a conference paper at ICLR 2021
F COMPARISON WITH MYRTLE ARCHITECTURE
While FCN or CNN-Vec NNGP performance is not as competitive modern CNNs on CIFARIO classification tasks, recently Shankar et al. (2020) has shown that the Myrtle architecture along with 7CA regularized preprocessing can obtain highly performant kernels. Teee et al. (2020) has shown that this generally true for pooling kernels.
Here we evaluate uncertainty properties of Myrtle-10 10 NNGP. Note that this kernll is extremely compute intensive (see SM and evaluation on the full CIFARIO and CIFARIO-C requires about 12,000 GPU hours (using NVIDIA Tesla V100s). We release these precomputed kernels for any further analysis by the research community
We observe that NNGP-R with the Myrtle-10 10 kernel is the Ieast robust under corruption when compared to other kernel architectures (Fig. [S10). However, comparing to the finite Myrtle network trained via gradient descent, observe that NNGP-R method method when average over 10 random seeds (Fig. SII). While we consider these results preliminary and requiring further investigation, it suggests there may be necessary tradeoff between the accuracy and calibration obtainable by NNGP-R.
robust the deep ensemble
We also plot the singular values of the training kernels (10k training samples) for three different architectures: FCN, CNN-VEC and Myrtle (Fig. S12). Note that the Myrtle kernel is much more ill-conditioned than the other two kernels.
08
02
10ld
Test
Compption Ineratyy
0.8
Mathod
0.6
R:F
FC-NNGE
C/NNGP R
0.4
0.2
0.0
Text
Compption
1.50
1.25
0.00
Compttio
Negashen bog bood
15
10-
Compption Imeratyy
Figure S10: Uncertainty metrics across shift levels on CIFARIO using NNGP-R compared with Myrtle-10 10 architecture (Shankar et al.) 2020). Here we compare NNGP-R Fig. to the pooling based Myrtle- 10 kernel with ReLU activation. All methods except for the Myrtle architecture remain well calibrated for all intensities of shifts. See Table S2 for detailed numbers.
httpa: //g1 http::/ch......ooooooooooooooeea
14

---

Published as a conference paper at ICLR 2021
0.8
Acouracy
0.6
0.4
Method
0.2
FINITE MYRTLE (ENS/T)
MYYRTE-NNGPA
Valid
Test
3
Corruption Intensity
0.8
0.6
ECE
0.2
0.0
Valid
Test
Corruption Intensity
1.5
Brier Score
1.0
0.5
Valid
Test
3
Corruption Intensity
Figure S11: Uncertainty metrics across shift levels on CIFARIO comparing Myttl-NNGP-R and deep ensemble of finite Myrtle- 10 networks.
Myrtle FCN CNN-VEC
Singular Value
107
10a
10-2
10
10a
10t
102
10a
10*
Index
Figure S12: Singular values of the NNGP kemels of a 10k training set for three architectures: 3-layer fully-connected network (FCN), 10-layer convolutional network with a vectorization readout layer (CNN-VEC) and 10-layer myrtle network (Myrtle).
15


Reviews:
[{"review_id": "x6lvC3wE6_I", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "update 1: I thank the authors for the rebuttal. My questions and concerns were appropriately addressed. However, other reviewers raised some concerns regarding the experimental set-up that I have missed. Thus, I will keep my score as is -- 7: good paper, accept.\n\nSummary:\n\nThe submission studies Gaussian process models with the infinite-width neural network kernels. In particular, through a series of experiments, the paper investigates their uncertainty properties and answers the question “how calibrated are the predictive uncertainties for in-distribution/out-of-distribution inputs?”: i) a comparison between GP classification with the infinite-width neural network kernels and finite width neural network classification was provided to test the calibration, (ii) a study of these kernels on regression tasks where the GP posterior can be obtained exactly, (iii) a study of using GPs with these neural network kernels on features extracted from a pre-trained network, aka, deep kernel learning of Wilson et al with neural network kernels. Overall, the performance of these GP variants is promising and competitive to the best existing methods.\n\n\nAssessment:\n\nThe paper is very clear and is an interesting read. The experiments are well-designed and the comparison to relevant state-of-the-art methods was provided. Whilst the theoretical contribution is on the light side (no new models or algorithms proposed in this submission), the results from the experiments outweighs this weakness and thus make this paper potentially relevant to the large Bayesian deep learning/GP community. I am leaning toward \"accept\" and willing to increase the score if my questions below could be clarified.\n\n1. This point is rather philosophical. The calibration metrics are great and they can be used for out of distribution detection. Perhaps this is discussed in previous work but it is not clear to me how these scores can be used to compare methods. For log likelihood, Grosse et al pointed out that a difference of xx nats is significant. Can we quantitatively do the same for these metrics, i.e. the ideal classifier would produce xxx, or a useful threshold for practical use is xxx, or the differences between the whisker plots in figures 2 and 3 are significant. Additionally, are these uncertainties calibrated enough for downstream tasks such as active learning, continual learning, bandits/reinforcement learning etc?. At the risk of asking too much for a conference paper, perhaps since this submission is about understanding the uncertainty of these GP models, an experiment or two on this aspect could be included.\n\n2. I think the difference between the models/inference schemes considered here could be made clearer so that readers could get a clear picture of what methods are being compared to what alternatives. My understanding is that there are the following configuration of models/inf methods -- top: inference, left: net width\n\nnet width  Bayesian   point (ML/MAP) \n---------------------------------------------------\nfinite               (1)                  (2)\n\ninfinite           (3)                  (4)\n\n(1) is hard to compute exactly. Approximate Bayesian is not calibrated enough. Though there are approximate sampling methods that can scale to the network size considered in this submission but these were not considered.\n(2) is cheap and thus often used in practice, but not calibrated.\n(3) exact solution can be obtained for regression with Gaussian observation noise, but expensive. Approximate inference/sampling is often required. This submission shows this configuration is better calibrated than existing solutions for (1) and (2).\n(4) is not considered in the experiments provided. It might be interesting to consider this.\n\n3. Why not directly optimising the marginal likelihood (in the regression case)? If I understand correctly, Lee et al used this for their MNIST regression experiment?\n\n4. It would be interesting to compare the infinite net (GP) performance with wide-net performance or narrow-net performance, to understand what regime of the networks is being used and what causes the performance difference across model/inf method configurations: is that because of the model, or is that because of the inference method?. Matthews et al had some comparisons like this using BNN with HMC and GP regression.\n\n5. clarity: For figure 1 (left) and those in the appendix, it’s very hard to see the overlapping bars, maybe just plot the curves connecting the top of the bars? For figure 2 (middle) the limits of the y-axis could be changed. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "N4LHtd8I6e9", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After rebuttal: I am uneasy about the overstated claims made in section 2. That the architectures are small should really be mentioned more prominently. However reviewers #1 and #2 make a good case that what matters are the improvements presented in Section 5. Thus, I reluctantly recommend acceptance.\n\n# Paper summary\n\nThe authors empirically investigate the calibration performance of NN-GPs in CIFAR10 and several UCI data sets, in three forms:\n- Bayesian inference for the NN-GP function-space prior, through a softmax link function\n- Heuristics to convert the classification-as-regression posterior into class probabilities\n- As a \"head\" to a pre-trained network\n\nGenerally, they find the NN-GPs are competitive with the _examined_ alternatives.\n\n# High-level comments\n\nI think empirical work like this paper is important: we Bayesians like to justify ourselves using calibration, but unless the beautiful Bayesian methods are *actually* calibrated, they are not useful. This paper mainly explores some properties of existing algorithms that are not understood, which is great.\n\n## Table 1's architectures are too small (2 layer CNN?)\n\nHowever, I will criticise the methodology. The architecture of the NNs used in Table 1 is specified in the supplement, and it is not pretty: the CNN is restricted to have only 1 or 2 layers (Appendix B)q. The convolution filter shape is unspecified, so I'll assume it is 3x3 like most modern CNNs. Because these networks are so shallow, the receptive field of a given convolutional location is not even close to the full input image.\n\nThis architecture is simpler than even the 5-layer 1998 LeNet , and is very far from the 2015 ResNet (He et al.) that is a reasonable CIFAR10 baseline; let alone the current state of the art. It is also much simpler than the networks considered for kernel architectures by Shankar et al. (2020). \n\nGiven how far these architectures are from things used in practice that have decent CIFAR10 performance, I'm not sure we can extrapolate much from the data in Table 1. Perhaps the good calibration and superior performance of the kernel will stop being true once we go to 18 or 20 layers, or once we add mean-pooling to the CNN.  We know that the superior performance of the kernel stops being true at a certain point for classification-as-regression, and I see no reason to think it would be different for \"proper\" Bayesian inference. (In fact, in my experience, variational inference for GP classification with these kernels has worse accuracy than classification-as-regression).\n\nIt is true that calculating the kernel matrix for more complicated CNNs is much more expensive. However, you could do that experiment without tuning the hyperparameters with Vizier. Since the complexity of MCMC once you have the kernel matrix is the same, you should be able to do it. I suspect the accuracy will be much better for the modern CNNs without tuning, than for the 2-layer CNNs you found (the calibration may be better or worse).\n\nAt the very least, the caveat that the CNNs are tiny should be clearly stated in Section 3 (and not hidden in the appendix).\n\n## Ignores similar work doing Bayesian linear regression on top of a NN base\n\nThe paper \"Deep Bayesian bandits showdown\" (https://arxiv.org/abs/1802.09127) compares various Bayesian deep learning methods on a bandit task, where calibrated uncertainty is crucial. As a result, most of them fail. The one that seems to come out on top is to change the last layer of a network to a Bayesian linear regression layer. This is equivalent to the method shown in Section 5 of this paper (except this paper sometimes uses a kernel, that is not just the linear regression kernel). It should be at least acknowledged, and ideally it would be included in Table 3.\n\nTable 3 should also acknowledge Bradshaw et al. (2017, https://arxiv.org/abs/1707.02476), which adds an RBF kernel on top of a neural network. All of these should perform similarly to NNGP-LL, especially given how close the RBF kernel is to the NNGP in Table 2.\n\n## Unclear that NNGPs have better uncertainty in Table 2\n\nAnd that's okay! But it should be a bit more emphasized than \"competitive in NLL\" just before Section 4.2. Also, the non-standard training for RBF kernels (selecting hyperparameters on a validation set, instead of maximising the train likelihood) may hurt their performance in Table 2.\n\nThat NN-GPs are good in accuracy at UCI data sets was noted by Arora et al. (https://openreview.net/forum?id=rkl8sJBYvH), though only for classification\n\n## In conclusion: overstated claims\n\nI think this paper is good, but it overstates how good the methods evaluated turn out to be. It paints a rosy picture of everything, like calibration in NNs is now solved, but that is not true. However, I do think this paper is a meaningful contribution, that puts together a few ideas that were lying around and shows that you can use them to make progress in calibration in NNs.\n\nThe authors should clearly caveat in section 3 that the results may not extrapolate to bigger CNNs or, ideally, add a data point with bigger CNNs. The authors should compare to similar algorithms that will be competitive in Table 3.\n\n# Minor points\n\nTop of pg. 2: SNR -> show acronym.\n\nbefore section 1.1: \"It is possible to disambiguate between the uncertainty properties of the NN prior and those due to the specific optimization decisions by performing Bayesian inference\". What do you mean? Transforming the prior into a posterior using Bayesian updating is one such decision. Also, because you're approximating the prior, deciding which Bayesian inference algorithm to use matters.\n\nSection 3: \"by avoiding heuristic approaches to inference, we are able to directly evaluate the prior\". I realise now that by this you mean the same as what I asked above, whatever it is. But the phrasing would be clearer with \"we are able to directly evaluate the [properties conferred by the] prior\". The fact is, you can evaluate the prior density for any weight setting fairly trivially!\n\nPage 3, top: \"under gradient flow to minimize a loss, the output distribution remains a GP\". This is only true for squared loss, as far as I know.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5VdYhkcpuNv", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: the authors have greatly improved the figures and tables, and expanded the captions, removing my major concern about clarity. I have improved my rating. I not not share the objection of reviewers 5 and 4 about the small size of the CNN, resulting in an inferior baseline in section 3, as the approach is still impractical and mostly a proof of concept that should encourage further research. What matters is that improvements in section 5 are built upon a SOTA baseline, inspired by section 3.\n\nThe stated goal of this paper is very ambitious: how NNGPs provide better confidence prediction, in terms of calibration, OOD data  and distributional shift.\n\nAfter an abstract that is somehow confusing, Introduction and Background  are clear and comprehensive.\nNext the authors “describe” their NNGP and try an impressive number of architectures over several datasets, however, assembling the paper seems to have been an hasty business where many clarity issues have been left unresolved, and these are severe, especially for reader such as myself who are not an expert in GP.\nThe key innovation presented in section 5: the NNGP is just added as a calibration layer on top of a pretrained NN.  It could be of high significance for a practitioner like myself interested in added methods improving the calibration of an existing model, but the authors set aside computational complexity issues and do not give an implementation. There are also serious clarity issues.\nResults look excellent, but in the way they are reported in figures and tables, there are too many questions, inconsistencies and readability issues to be sure.\n\nIn summary, this seems to be an unfinished write-up where the rich material needs to be better organized with a clear presentation flow, and tables and figures need serious improvements. It has the potential for an excellent publication, proposing a novel solution to an important problem, with rich technical and experimental material.\n\nDetailed comments:\n\nFor a reader such as myself who is not an expert in GP but interested in their application to calibration and OOD data, the general introduction in Eq.(1) and (2) makes no sense. I was not even able to parse them and I had to read section 2.4 of (Lee et al 2018) to fully understand how the posterior on a test point is computed. Further down, the authors mention in passing  two critical issues that make NNGP so expensive  without proper explanation:\n-\tHyperparameter tuning using Vizier: what are the hyperparameters?\n-\tComputational cost: they mention it is O(N^3) In the number examples because of the Cholesky decomposition (within what operation? Inversion? reference?). This seems extreme as I see an O(N^2) complexity in other work (Lee et al.)\nNote that section A in the appendix did not provide more clarity.\n\nStandard RBF method: can you give a reference?\n\nI spent some time looking for experiments that account for dataset shift before realizing ‘fog’ perturbations represent dataset shift. Same issue with the OOD data: it just appears in Figure 1 and table 1 without proper analysis.\n\nFigure 1: the coloring scheme of the left diagram is not readable. Where is the figure showing entropy?\n\nTable 1 is dropped without any description of the metrics and the datasets:\n-\tThe reader is left guessing what the ‘Fog’ corruptions are, as they are neither introduced nor described, except for a mention of “CIFAR fog corruption” in a table caption. One later learns those are the same as Ovadia shift levels.\n-\tWhat is “accuracy”? The reported number for CIFAR10 is 0.487: is this 48.7%? SOTA is more than 90%.\n-\tAre SVHN and CIFAR100 introduced as OOD? Define mean confidence and entropy? No analysis of results here.\n\nFigure 2 is not readable or comparable to Ovadia et al and should be made larger (maybe remove the Brier Score?):\n-\tLike in table 1, accuracy is in the 0.6 range when it should be 0.9\n-\tECE should range between 0 and .2, not 0 and .8\n\nTable 3: \n-\twhat is the number of examples in NNGP-LL last column? 50K?\n-\tI have never seen quartiles used for comparison and I do not understand why they are used, as the authors do not seem to exploit the added information. They should be either explained/exploited or removed (recommended to make space for better explanations elsewhere)\n\n\nAppendix: lots of good material there, some of which could be moved to the main paper (and some material in the main paper should be moved to the appendix, in particular some rows in table 1 and 3)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2sw3ggn3TWt", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an interesting paper which evaluates the calibration of NNGPs, including around OOD detection.\n\nMajor points:\n1. It is worth noting that in the appendix for arXiv:1808.05587, they explicitly use the full categorical log-likelihood with softmax probabilities (albeit with inducing points), and they find that the resulting model is well-calibrated.  I would be surprised if the use of the Categorical-softmax did not appear in other papers.\n\n2. I am a bit confused about the notion of an \"ensemble\" of NNGPs.  To my mind, that would imply a number of different NNGPs with different hyperparameters (e.g. depth).  I'm not sure what is being done here, but it appears to be equivalent to a single NNGP?\n\n3. However, my primary concern is around the strength of the NN baselines. They find that the performance of their CNNs and C-NNGPs with CIFAR-10 is very similar, with around 60% accuracy.  For C-NNGPs,  60% accuracy is pretty standard, but for NNGPs, it is really, really bad: with very little effort it is possible to get 90% accuracy, and with a little further tuning, you can get to 95%.  Even when trying to do a fair comparison to infinite networks, other researchers have found 80% accuracy in finite networks (arXiv:1810.05148).  This poor performance brings into question the relevance of their results.  At the very least, the paper needs:\n    * A detailed description of why the NN's performance is so poor.\n    *Additional comparisons against stronger NN baselines.\n\n4. It isn't clear what the infinite width layers give you that's better than either using Bayesian linear regression over the output weights, or deep kernel learning.\n\nMinor points:\n1. Figure 1 (left) is completely unreadable.  Either use separate plots, or use a line (rather than an area) to denote the density.\n2. Quote NLL per datapoint, not for full dataset.\n3. Exactly what is in table 3 is very unclear: does it aggregate over corruptions?  What are 1k etc. (all of this should be in the caption).\n4. Table S1. \"confidence on a specific test point being 1.0 to machine precision\".  The entropy in that case is zero, and any standard, numerically stable implementation should give that.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an empirical study focusing on Bayesian inference on NNGP - a Gaussian process where the kernel is defined by taking the width of a Bayesian neural network (BNN) to the infinity limit. The baselines include a finite width BNN with the same architecture, and a proposed GP-BNN hybrid (NNGP-LL) which is similar to GPDNN and deep kernel learning except that the last-layer GP has its kernel defined by the width-limit kernel. Experiments are performed on both regression and classification tasks, with a focus on OOD data. Results show that NNGP can obtain competitive results comparing to their BNN counterpart, and results on the proposed  NNGP-LL approach provides promising supports on the hybrid design as to combine the best from both GP and deep learning fields.

Although the proposed approach is a natural extension of the recent line of work on GP-BNN correspondence, reviewers agreed that the paper presented a good set of empirical studies, and the NNGP-LL approach, evaluated in section 5 with SOTA deep learning architectures, provides a promising direction of future for scalable uncertainty estimation. This is the main reason that leads to my decision on acceptance.

Concerns on section 3's results on under-performing CNN & NNGP results on CIFAR-10 has been raised, which hinders the significance of the results there (since they are way too far from expected CNN accuracy). The compromise for model architecture in order to enable NNGP posterior sampling is understandable, although this does raise questions about the robustness of posterior inference for NNGP in large architectures.
Decision: Accept (Poster)