Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes
Authors: Liam Hodgkinson
Chris van der Heide
Fred Roosta
Michael W. Mahoney
Venue: Submitted to ICLR 2023 (2023)
Keywords: double descent
Gaussian processes
Bayesian statistics

Abstract: The quality of many modern machine learning models improves as model complexity increases, an effect that has been quantified—for predictive performance—with the non-monotonic double descent learning curve. Here, we address the overarching question: is there an analogous theory of double descent for models which estimate uncertainty? We provide a partially affirmative and partially negative answer in the setting of Gaussian processes (GP). Under standard assumptions, we prove that higher model quality for optimally-tuned GPs (including uncertainty prediction) under marginal likelihood is realized for larger input dimensions, and therefore exhibits a monotone learning curve. After showing that marginal likelihood does not naturally exhibit double descent in the input dimension, we highlight related forms of posterior predictive loss that do. Finally, we verify empirically that our results hold for real data, beyond our considered assumptions, and explore unusual consequences involving synthetic covariates.

Under review as a conference paper at ICLR 2023
MONOTONICITY AND DOUBLE DESCENT IN UNCER- TAINTY ESTIMATION WITH GAUSSIAN PROCESSES
Anonymous authors Paper under double-blind review
ABSTRACT
The quality of many modern machine learning models improves as model com- plexity increases, an effect that has been quantified for predictive performance- with the oon-oootooiic double descent learning curve. Here, we address the over- arching question: is there an analogous theory of double descent for models which estimate uncertainty? We provide partially affirmative and partially negative an- swer in the setting of Gaussian processes (GP). Under standard assumptions, we prove that higher model quality for optimally-tuned GPs (including uncertainty prediction) under marginal likelihood is realized for larger input dimensions, and therefore exhibits a monotone error curve. After showing that marginal likelihood does not naturally exhibit double descent in the input dimension, we highlight re- lated forms of posterior predictive loss that do exhibit non-monotonicity. Finally, we verify empirically that our results hold for real data, beyond our considered assumptions, and we explore consequences involving synthetic covariates.
INTRODUCTION
With the recent success of overparameterized and nonparametric models for many predictive tasks in machine learning (ML), the development of the corresponding uncertainty quantification (UQ) has unsurprisingly become topic of significant interest. Naive approaches for forward propagation of error and other methods for inverse uncertainty problems typically apply Monte Carlo methods under Bayesian framework (Zhang 2021). However, the large-scale nature of many ML problems of interest results in significant computational challenges. One of the most successful approaches for solving inverse uncertainty problems is the use of Gaussian processes (GP) (Williams & Rasmussen] 2006). This is now frequently used for many predictive tasks, including time-series analysis (Roberts) ctt al. 2013) and classification (Williams & Barber 1998 Williams & Rasmussen 2006). GPs are also fundamental for Bayesian optimization (Hebbal et al. 2019 Frazier 2018), although extending Bayesian optimization into high-dimensional settings remains challenging (Binois & Wycoff 2021))
Although the theoretical understanding of the predictive capacity of high- ditensiienii ML mod- els continues to advance rapidly, a parallel rigorous theory for UQ is comparatively lagging. The prominent heuristic in modern ML that larger models will typically perform better has become al- most axiomatic. However, it is only more recently that this heuristic has become represented in the theory through the characterisation of benign overfitting (Bartlett et al.) (2020). particular, the double descent curve extends the bias-varianee tradeoff curve to account for improving performance
with higher model complexity (Belkin al. 2019) Wang et al. 202T Derezinski al. 2020b) (see
Figure right)). Typically, these arguments involve applications of random matrix theory (Edelman
& Rao 2005 Paul & Aue 2014), notably the Marchenko-Pare -astur law, concerning limits of spectral distributions under large data/large dimension regimes.
While the predictive performance of ML models can improve as model size increases, not clear whether or not the same is true for predictions of model uncertainty. Several common measures of model quality which incorporate inverse uncertainty quantification are Bayesian in nature, the most prominent of which are the marginal likelihood and various forms of posterior predictive loss. It is well-known that Bayesian methods can perform well high dimensions (De Roos et al. (202T). even outperforming their -iw-meensiioal counterparts when properly tuned (Wilson & Izmailov (020). To close this theory-practin gap, an analogous formulation of double descent curves in the setting of uncertainty quantification is desired. Marginal likelihood and posterior distributions are often

---

Under review as a conference paper at ICLR 2023
Performance Metric
Error Curve
Optimal 7 eqn.
Marginal Likelihood / Free Energy 3
Monotone (Thm.
Posterior Predictive L Loss
Double Descent (Prop.
Posterior Predictive NLL
Double Descent (Prop.
f v/1 |
Table I:
Behavior of UQ performance metrics and optimal posterior temperature 7
intractable for arbitrary models (c.g., Bayesian neural networks (Goan & Fookes 2020 However,
their explicit forms are well known for GPs (Williams & Rasmussen 2006).
GPs are nonparametrie, and most of the kernels used in practice induce dimenti-ad feature spaces, so model complexity can be difficult to quantify (although some notions of kernel dimen- sion have been proposed (Zhang) 2005 Alaoui & Mahoney 2015)). Nevertheless, it is generally expected that accurately fitting GP to data lying in bigher-dimensione spaces requires training on larger dataset. This curse of dimensionality has been justified using error estimates (von Luxburg & Bousquet 2004), and verified empirically (Spigler et al. 2020). However, under appropriate se- tups, predictive performance has been demonstrated to improve with larger input dimension (Liu et al. 2021). Here, we consider whether the same is true for the marginal likelihood and posterior predictive metries. Our main results (see Theorem and Proposition [1] are summarized follows.
For two optimally regularized scalar GPs, both fit to a sufficiently large set of iid normalized and whitened input-output pairs, the better performing model under marginal likeli- hood is the one with larger input dimension.
Monotonicity:
Double Descent: For sufficiently small temperatures, GP posterior predictive metrics exhibit dou- ble descent if and only if the mean squared error for the corresponding kernel regression task exhibits double descent (see Liang & Rakhlin (2020): Liu et al. (202)) for sufficient conditions).
Figure [ illustrates characteristics of monotone and double descent error curves. Along the way, we identify optimal choices of temperature (which can be interpreted as noise in the data) under a tempered posterior setup see Table [ for summary. Our results highlight that the common curse of dimensionality heuristic can be bypassed through an empirical Bayes procedure. Furthermore, the performance of optimally regularized GPs (under several metrics), can be improved with additional covariates (including synthetic ones). Our theory supported by experiments performed on real large datasets. Additional experiments, including the effect of ill-conditioned inputs, alternative data distributions, and choice of underlying kernel, are conducted in Appendix A. Details of the setup for each experiment listed in Appendix G.
BACKGROUND
2.1 GAUSSIAN PROCESSES
A Gaussian process is stochastic process f on Rd such that for any set of points . Te E Rd [f(z1)..... f(rx)) )) is distributed as multivariate Gaussian random vector (Williams & Rasmussen, 2006 $2.2). Gaussian processes are completely determined by their mean and covariance functions: if for any v, x' E Rd Rf(()) = m(x) and (x:) fxx') k(x,, ('r,"), x'), then we say that ~ gP(mm, k). Inference for GPs informed by Bayes' rule: letting (X,, Y)) denote collection of
id input-output pairs, we impose the assumption that Y; f(X,) ) + & where each G; (0,
error
erTor
input dimension
input dimension
Figure I: Illustrations of monotone (left) and double descent (right) error curves.
2

---

Under review as a conference paper at ICLR 2023
yielding Gaussian likelihood ff) (2my) n//
n/2 exp(
f(x))l?). The parameter ||
24
y is the temperature of the model, and dictates the perceived accuracy of the labels. For example, taking y + 0+ considers a model where the labels are noise-free.
For the prior, assume that GP(O,A for fixed covariance kernel i and regularization parameter À > 0. While other mean functions m can be considered, in the sequel we will consider the case where m = O. Indeed, if m * O, then one can instead consider V; = Y; - m (X)), so that Y; = f(X.) + E, and the corresponding prior for f is zero-mean. The Gram matrix Kx E R for X has elements X k(,,, Xj)- Let æ = (.1,-... .Tm) denote a collection of N points in Rd f(x)= = (f( elements Ky = k(z,, 2z) and k xz),
and denote by K E Rmmx and hi R the matrices with
Given this setup, we are interested in several metrics which quantify the uncertainty of the model. The posterior predictive distribution (PPD) of (ax given X, (Williams & Rasmussen 2006 pg. 16)
f(x) ~
where f(x)) = =: + xy and E(x) = K k + - This defines a posterior predictive distribution p² the GP f given X, Y (so X,Y p²). If we let y (yI..... (y.) denote a collection of m associated test labels corresponding to our test data æ, the posterior predictive I loss (PPL2) is the quantity
fxr, y) = Ry-r (mx) y/|² 1f(x)) yll + (r-((((((x))
(1)
Closely related is the posterior predictive negative log- bog-likel (PPNLL), given by
L(x, y|X, = -y log = (1f(()) - y11² + 2X7 tr tr(())))) + log(2 (2)
2.2 MARGINAL LIKELIHOOD
The fundamental measure of model performance in Bayesian statistics is the marginal likelihood (also known as the partition function in statistical mechanics). Integrating the likelihood over the prior distribution provides probability density of data under the prescribed model. Evaluating this density the training data gives an indication of model suitability before posterior inference. Hence, the marginal likelihood is Zn Ey~-p(Y [f X). A larger marginal likelihood is typically understood as an indication of superior model quality. The Bayes free energy F, = - log Z, is interpreted analogue of the test error, where smaller F,, is desired.
The marginal likelihood for a Gaussian process is straightforward to compute: since Y; = f(X.) + under the likelihood, and (f(X,)) gP(( k)), we have Y| NO, ((0,0-11 X + 7T), and hence the Bayes free energy is given by (Williams & Rasmussen, 2006, eqn. (2.30))
A/ A ( 'xx) under the GP prior
F AY (Kx + X²1)
let +yy1) log 2
In practice, the hyperparameters X, are often tuned to minimize the Bayes free energy. This is an empirical Bayes procedure, and typically achieves excellent results (Krivoruchko & Gribov 2019).
The PPNLL can be linked the marginal likelihood through cross-validation measures. Let T be uniform on {1,..., k) and let T be a random choice of le indices from {1.....,) n. (the test ser). Let T {1, (I,..., nt denote the corresponding training set. The leave-k-out cross- validation score under the PPNLL is defined by (X, EL(XT, Yr, (Y Yy). The Bayes free energy the sum of all leave-k-out cross-validation scores (Fong Holmes (020). that is k== = F F. can be interpreted as the average cross-validation score. Similar connections can also be formulated in the PAC-Bayes framework (Germain et 2016).
(), Therefore, the mean Bayes free energy (or mean free energy for brevity)
2.3 BAYESIAN LINEAR REGRESSION
One of the most important special cases of GP regression is Bayesian linear regression, obtained by taking knn (z, x') x x'. As a special case of GPs, our results apply to Bayesian linear regres- sion, directly extending double descent analysis into the Bayesian setting. By Mercer's Theorem
3

---

Under review as a conference paper at ICLR 2023
Williams & Rasmussen 2006, $4.3.1), realization of GP f has series expansion in terms of the eigenfunctions of the kernel k. As the eigenfunctions of kin are linear, f ~ GP X (ii)) and only if
f(x) w =
~~((0
More generally, if x TRd + R2 is a finite-dimensional feature map, then [(x) = w (x), ~ (O, ( is GP with covariance kernel (a, y) =(x) o(y). This is the weight-space in- terpretation of Gaussian processes. In this setting, the posterior distribution over the weights satisfies p²(w)= = p( X, Y) & exp(- 2/1 o(X)=12 =//ll(l²²)
and the marginal likelihood becomes
2 - f X(]], w) w)T(w)dur 7m/2 (2 ((mdd) R
14/2
/ e - 1
1| dw,
(4)
where (X) (o(x))) Ruxp Under this interpretation, the role of À a regularization
parameter is clear. Note also that if X - n/7 for some > O, then the posterior p (w(w) depends on as (e' (w)) (excluding normalizing constants). This is called a tempered posterior, if y < 1, the posterior cold, and it warm whenever 7 >> 1.
RELATED WORK
Double Descent and Multiple Descent. Double descent is observed phenomenon in the error curves of kernel regression, where the classical (U-shaped) bias-variance tradeoff in underparame- terized regimes is accompanied by curious monotone improvement in test error as the ratio c of the number of datapoints to the ambient data dimension increases beyond c = 1. The term was popularized in Belkin et al. (2018b 2019). However, had been observed in earlier reports (Do- briban& & Wager, 2018 Loog et al., 2020), and the existence of such non- non-monotonic behavior as function of system control parameters should not unexpected, given general considerations about different phases of learning that are well-known from the statistical mechanics of learning (Engell & den Broeck 2001] Martin & Mahoney, 2017). An early precursor to double descent analysis came in the form of the Stein effect, which establishes uniformly reduced risk when some degree of regularisation is added (Strawderman, (02T). Stein effects have been established for kernel regres- sion in Muandet et al.] (2014): Chang et al. (2017). Subsequent theoretical developments proved the existence of double descent error curves on various forms of linear regression (Bartlett et al. 2020
Tsigler & Bartlet 2020 Hastie et al., 2022, Muthukumar et al., 2020), random features models
(Liao et al. 2020: Holzmüller 2020), kernel regression (Liang & Rakhlin 2020 Liu et al. 2021}
two-layer neural networks (Mei & Montanari 2022). and classification tasks (Frei al. 2022 Wang
ct al. 2021). For non-asymptotic results, subgaussian data is commonly assumed, yet other data dis- tributions have also been considered (Derezinski et al. (0206). Double descent error curves have also been observed in nearest neighbor models (Belkin et al. 2018a), decision trees (Belkin et al.) 2019), and -fate-of-tthe-an neural networks (Nakkiran et al. (021I. More recent developments have identified large number of possible curves in kernel regression (Liu et al.) [02T]. including triple descent (Adlam & Pennington] 2020 d' Ascoli et al., 2020) and multiple descent for related volume- based metrics (Derezinski et al.) 2020a). Similar to our results, optimal choice of regularization parameter can negate the double descent singularity and result in a monotone error curve (Liu et al.) 2021 Nakkiran et al., 2020 Wu & Xu 2020). While there does not appear to be clear consensus on a precise definition of "double descent," for our purposes, we say that an error curve (t) ex- hibits double descent if it contains single global maximum away from zero at t", and decreases monotonically thereafter. This encompasses double descent as it appears in the works above, while excluding some misspecification settings and forms of multiple descent.
Learning Curves for Gaussian Processes. The study of error curves for GPs under posterior predictive losses has a long history (see Williams & Rasmussen (2006) $7.3) and Viering Loog (22TT). However, most results focus on rates of convergence of posterior predictive loss in the large data regime n - 20. The resulting error curve is called a learning curve, as it tracks how fast the model learns with more data (Sollich) 1998 Sollich & Halees) 2002: Le Gratiet & Garnier 2013). Of particular note are classical upper and lower bounds on posterior predictive loss (Opper & Vivarelli 1998 Sollich & Halees 2002: Williams & Vivarelli 2000), which are similar in form to counterparts in the double descent literature (Holzmüller) 2020). For example, some upper bounds have been obtained with respect to forms of effective dimension, defined in terms of the Gram matrix
4

---

Under review as a conference paper at ICLR 2023
10a
Mean Free Energy
1.47
3.16
1.46
3.15
1.45
3.13
1.43
3.11
3.10
1.42
3
log:o Dimension
Figure 2: Error curves for mean Bayes free energy n- FF; for synthetic data under linear (top) and Gaussian (bottom) kernels, with X = X (left; monotone decreasing) and À = 0.01 (right;
increases at higher input dimensions).
(Zhang 2005 Alaoui & Mahoney 2015). Contraction rates in the posterior have also been examined Lederer et al. (209)). In our work, we consider error curves over dimension rather than data, but we note that our techniques could also be used study learning curves.
Cold Posteriors. Among the most surprising phenomena encountered in Bayesian deep learning the cold posterior effect (CPE): the performance of Bayesian neural networks (BNNs) appears to improve for tempered posteriors when ?y + 0- This presents challenge for uncertainty prediction: taking 7 + 0 concentrates the posterior around the maximum a posteriori (MAP) point estimator, and so the CPE implies that optimal performance is achieved when there is little or no predicted uncertainty. First observed in Wenzel et al. (2020), several authors have since sought explain the phenomenon through data curation (Aitchison (2020). data augmentation (Izmailov et al. [02T]] Fortuin et al., (02T). and misspecified priors (Wenzel al. 2020), although the CPE can still arise in isolation of each of these factors (Noci et 2021). While our setup is too simple to examine the CPE at large, we find some common forms of posterior predictive loss are optimized as 7 -> 0+.
4 MONOTONICITY IN BAYES FREE ENERGY
In this section, we investigate the behavior of the Bayes free energy using the explicit expression in 3). First, to facilitate our analysis, we require the following assumption on the kernel k.
Assumption. The kernel l is formed by function R + IR that is continuuslly differentiable on (0, oo) and is one of the following two types:
(I) Inner product kernel: continuously differentiable in neighbourhood of zero, with x' (0) > 0. Let
k(æ, x') = k<œT /d/d) for x, Rdd where three-times
x'()),
8 =(11 x(0) (0))
(II) Radial basis kernel: k(x, x') ={[1 x' x/12//d) for xr, x'' E Rdd. where k. is three-times continuously differentiable on (0, 00), with k. (2) < 0. Let
- 3 =(0) 2 (2) xx()).
5

---

Under review as a conference paper at ICLR 2023
0.99
0.99
Mean Free Energy
Mean Free Energy
101
6.1
2
0.01
10a
10a
Dimension
Dimension
Figure 3: Error curves for mean Bayes free energy under the CIFAR10 dataset; linear (left) and Gaussian (right) kernels; À X"; curves for real data match Figure 2 (left).
This assumption allows for many common covariance kernels used for GPs, including polynomial kernels k(xx (c + xT /d)r, the exponential keree k(r, x') = exp(x' Tx x2/d),, the Gaussian kernel k(x, x') exp(-1la x' 2/||2/////// /d), the multiquadric kernel k(x, x²) = (c || x' ////// the
inverse multiquadric k(x, x²) = (c +
x /d) /d) kernels, and the Matérn kernels
k(x,==)
(2 (()) "| 2'1l" K, (1|== 1'll)
(where K2 is the Bessel-K function). Different bandwidths choice of k. However, it does exclude some of the more recent and sophisticated kernel families, e.g., neural tangent kernels. Due result of El Karoui (2010), the Gram matrices of kernels satisfying this assumption exhibit limiting spectral behavior reminiscent of that for the linear kernel, k(x,x') c c z'/d. Roughly speaking, from the perspective of the marginal likelihood, can treat GPs as Bayesian linear regression.
also be incorporated through the
In line with previous work on double descent curves (Belkin et al. 2019). our objective is vestigate the behavior of the marginal likelihood with respect to model complexity, which is often given by the number of parameters parametric settings (d' d'cool et al., 2020 Derezinski et al., 2020b, Hastie et al.) 2022)). GPs are non- paraamaeteric, and while notions of effective dimension do exist (hang 2005 Alaoui & Mahoney 2015), it is common to instead consider the input dimension in place of the number of parameters in this context (Liang & Rakhlin 2020
Liu et al. 2021).
For our theory, we first consider the "best-case scenario," " where the prior is perfectly specified and its mean function is used to generate Y: Y m(X,) + c,. where each e; is iid with zero mean and unit variance. By a change of variables, we can assume (without loss generality) that m = O, so that Y; e, and is therefore independent of X. To apply the Marchenko- Pastur law from random matrix theory, we consider the large dataset large input dimension limit, where n and scale linearly that d/n - be independent zero-mean random vectors with unit covariance. Under this limit, the sequence of mean Bayes entropies n- -11F for each n 1, 2,... ..., converges in expectation over the training to quantity F% which is more convenient to study. Our main result is presented in Theorem I the proof is delayed to Supplementary Material.
oo). The inputs
assumed to have been whitened and to
Theorem 1 (Limiting Bayes Free Energy). Let XI.X..... be independent and identically dis- tributed zero-mean random vectors in IRd with unit covariance, satisfying E [|| < 400 for some 8 > O. For euch n = 1,2,... ..., let F7 denote (3) applied to X = (X:) and Y (Y:)
with each Y2 ~ N(O0, 1). If d + Do such that d/n + C (0, oo), then
F = lim EEF
well-defined. In this case,
(a) x = /// for some > 0, there exists optimal temperature > which minimizes F which given by
-c- - (B + + 5B + +))2 4c.
(5)
the kernel & depends X such that a is constant in X and 3 = 3ad for 30 E (0, 1), then

---

Under review as a conference paper at ICLR 2023
0.99
Mean Free Energy
10a
0.2
2
log10 Dimension
Figure 4: Error curves for mean Bayes free energy under real data with Gaussian (left); repeated data (center); and zeroed data (right), under the linear kernel and = X* Only adding synthetic non-zero iid covariates improves model performance.
(b) 1f E (0, - 80), there exists unique optimal X > 0 minimizing F% satisfying
x aalcc++ 1)(7 + So) + /(c 1)2 + 4c(y + Bb)
(6)
dl (7 Bn)3)
if 2 11 3o, then no such optimal X* exists.
(c) For any temperature O (0, oo).
Bo, at X = X", F is monotone decreasing in c. E
The expression for the asymptotic Bayes free energy FL is provided in the Supplementary Material. To summarize, first, in the spirit of empirical Bayes, there exists an optimal X for the Gaussian prior which minimizes the asymptotic mean free energy. Under this setup, the choice of À which maximizes the marginal likelihood for a particular realization of X, Y will converge almost surely X n,d d - Similar to Nakkiran et al. (2020): Wu & Xu (2020), we find that model performance under marginal likelihood improves monotonically with input dimension when À = X for fixed amount of data. Indeed, for large n, d, IEFF nF& and
d/n, so Theorem [ implies that the expected Bayes free energy decreases (approximately) monotomically with the input dimension, provided fixed and the optimal regularizer X is chosen.
Discussion of assumptions. The assumption that the kernel scales with À is necessary using our techniques, as X cannot be computed explicitly otherwise. This holds for the linear kernel, but most other choices of be made to satisfy the conditions of Theorem [ by taking k(z) + "x(T)), for appropriately chosen bandwidth r = r7)). For example, for the quadratic kernel, this gives k(x,x') = (> 1/2 + x1/2x x²)2. Effectively, this causes the regularization parameter to scale empirically
non-linearly in the prior kernel. Even though this is required for our theory, demonstrate this monotonicity also holds under the typical setup where does not change with X. In Figure 2 we plot the mean free energy for synthetic Gaussian datasets of increasing dimension at both optimal and fixed values of À for the linear and Gaussian kernels. Since n. is fixed, in line with Theorem [6] the curves with optimally chosen À decrease monotonically with input dimension, while the curves for fixed À appear to increase when the dimension is large. Note, however, that the larger 3 for the Gaussian kernel induces significant regularizing effect. A light CPE appears for the Gaussian kernel when À is fixed, but does not seem to occur under X*
While the assumption that m O may appear too restrictive, in Appendix B, we show that m is necessarily small when the data is normalized and whitened. Consequently, under a zero-mean prior, the marginal likelihood behaves similarly our assumed scenario. This translates well in practice: under similar setup to Figure 2 the error curves corresponding to the linear and Gaussian kernels under the whitened CIFAR10 benchmark dataset (Krizhevsky & Hinton 2009) exhibiting the predicted monotone behavior (Figure
Synthetic covariates. Since Theorem I implies that performance under the marginal likelihood
improve as covariates added, it is natural ask whether improvement will be seen if

---

Under review as a conference paper at ICLR 2023
0.90
Posterior Predictive
10a
10a
10a
0.001
log10 Dimension
Figure 5: Posterior predictive L loss error curves for synthetic data exhibiting perturbed / tempered double descent under the linear kernel with 0.01/7 (left), and À X (right).
the data is augmented with synthetic covariates. To test this, we considered the first 30 covariates of the whitened CT sl iee dataset obtained from the UCI Machine Learning Repository (Graf al., 2011), and we augmented them with synthetic (iid standard normal) covariates; the first 30 covariates repeated; and zeros (for more details, see Appendix A). While the first of these scenarios satisfies the conditions of Theorem [ the second two do not, since the new data cannot be whitened such that its rows have unit covariance. Consequently, the behavior of the mean free energy reflects whether the assumptions of Theorem [ satisfied: only the data with Gaussian covariates exhibits the same monotone decay. From practical point of view, a surprising conclusion reached: after optimal regularization, performance under marginal likelihood can be further improved by concatenating Gaussian noise to the input.
DOUBLE DESCENT IN POSTERIOR PREDICTIVE Loss
In this section, we will demonstrate that, despite the connections between them, the marginal like- lihood and posterior predictive loss can exhibit different qualitative behavior, with the posterior predictive losses potentially exhibiting a double descent phenomenon. Observe that the two forms of posterior predictive loss defined in ( and (2) can both be expressed in the form
L = ci (() vl1 c2(A,,) ((()(E(()) + ca(7).
MSE
volume
The first term is the mean-squared error (MSE) of the predictor J. and well-studied object in the literature. In particular, the MSE can exhibit double descent, or other types of multiple descent error curves depending on k, both ridgeless (Holzmüller) 2020 Liang & Rakhlin 2020) and general (Liu et al. 2021) settings. On the other hand, the volume term has the uniform bound Etr(E(x)) < m El(x. x. so provided 02 is sufficiently small, the volume term should have little qualitative effect. The following is immediate.
Proposition 1. Assume that the MSE [[)(()) 2 for Gaussian inputs and labels y converges to an error curve E(c) that exhibits double descent as n + oo with = d(n) satisfying d(n) /n E (0, o.). If there exists function X(7) such that x))7),)//((11) - 0 as 7 - 0+, then for any E > O, there exists an error curve E(c) exhibiting double descent, a positive integer N, and No and N. CC/c1 El at atdd= d(n) and X X(7).
Yo 0 such that for any 0
For posterior predictive L loss, in the tempered posterior scenario where X u/n, the MSE remains constant 7, while c2/01 7/pu. Since the predictor depends only on μ4, the optimal in the tempered posterior scenario is realised as + 0+. In other words, under the posterior predictive 2 loss, the best prediction of uncertainty is none at all. This highlights trivial form of CPE for PPL2 losses, suggesting it may not be suitable examine the linear kernel case; similar experiments for more general kernels are conducted in the Supplementary Material. In Figure (leff), kernel on synthetic Gaussian data by varying while keeping i fixed. We find that the error curve exhibits double descent when < 2p. The corresponding plot for the CIFAR1 O dataset is shown
UQ metric. Here we shall empirically
plot posterior predictive L loss under the linear
8

---

Under review as a conference paper at ICLR 2023
Posterior Predictive L??
0.99
Posterior Predictive L
0.00
10-2
10-2
0.01
10:3
0.01
10a
10a
Dimension
Dimension
Figure 6: PPI.2 loss under the linear kernel with X = /y (left) and CIFARIO dataset; curves for real data match Figure
X* (right) on the
in Figure b[eeft), demonstrating that this behavior carries over real data. Choosing X (the optimal À according to marginal likelihood) reveals more typical set of regularized double descent curves; this is shown in Figure rrihht) for synthetic data and Figure 6 righht for the CIFARI dataset. This is due to the monotone relationship between the volume term and X, hence the error curve inherits its shape from the behaviour X (see Appendix in the Supplementary Material).
In contrast, this phenomenon is not the case for posterior predictive negative log-likelihood. In- deed, letting À = 14/y and optimizing the expectation of ( in y. the optimal y = m [.]]]]]f()) y||². The expected optimal PPNLL is therefore
EE logp(ulf x) mm[ + log(2xB]] b1|²)| + trr))(((EE(x))) (7)
Otherwise, the PPNLL displays similar behavior to PPL., the related linearly.
CONCLUSION
Motivated by understanding the uncertainty properties of prediction from GP models, have ap- plied random matrix theory arguments and conducted several experiments to study the error curves of three UQ metrics for GPs. Contrary to classical heuristics, model performance under marginal likelihood/Bayes free energy improves monotonically with input dimension under appropriate reg- ularization (Theorem D However, Bayes free energy does exhibit double descent. Instead, posterior predictive loss inherits a double descent curve from non-UQ settings when the variance the posterior distribution sufficiently small (Proposition [T]. While our analysis was conducted under the assumption of a perfectly chosen prior mean, similar error curves appear to hold under small perturbations, which always holds for large whitened datasets. Although our contributions are predominantly theoretical, our results also have some noteworthy practical consequences:
Tuning hyperparameters according to marginal likelihood is essential ensuring good perfor- mance in higher dimensions, and completely negates the curse of dimensionality.
When using T losses as UQ metrics, care should be taken view of the CPE. As such, we do recommend the use of this metric in lieu of other alternatives.
Our experiments suggest that further improvements beyond the optimisation of hyperparameters may be possible with the addition of synthetic covariates, although further investigation is needed before such procedure can be universally recommended.
In light of the surprisingly complex behavior on display, the fine-scale behavior our results demon- strate, and surprising absence of UQ metrics in the double descent literature, encourage in- creasing adoption of random matrix techniques for studying UQ / Bayesian metrics in double de- scent contexts and beyond. There numerous avenues available for future work, including the incorporation of more general kernels (c.g., using results from Fan & Wang (2020) to treat neural tangent kernels, which commonly used approximations for large-width neural networks).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ben Adlam and leffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and multi-scale theory of generalization. In International Conference on Machine Learning, pp- 74-84. PMLR, 2020.
Laurence Aitchison. A statistical theory of cold posteriors in deep neural networks. In International Conference on Learning Representations, 2020.
Ahmed Alaoui and Michael W Mahoney. Fast randomized kernel ridge regression with statistical guarantees. Advances in Neural Information Processing Systems, 28, 2015.
Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117749):3063-30070 2020.
Mikhail Belkin, Daniel J. Hsu, and Partha Mitra. Overfitting perfect fitting? risk bounds for classification and regression rules that interpolate. Advances in neural information processing systems, 31, 2018a.
Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to under- stand kernel learning. In International Conference on Machine Learning, pp. 541-549, PMLR, 2018b.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias- variance trade-off. Proceedings of the National Academy of Sciences, 116(32):5849-15588 2019.
Mickael Binois and Nathan Wycoff. survey on high-dimensional Gaussian process modeling with application to Bayesian optimization. arXiv preprint arXiv:2/11.0504/ 2021.
Wei-Cheng Chang, Chun-Liang Liang Li, Yiming Yang, and Barnabás Póczos. Data-driven random fourier features using stein effect. Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 1497-1503, 2017.
Stéphane d' Ascoli, Levent Sagun, and Giulio Biroli. Triple descent and the two kinds of overfitting: Where why do they appear? Advances Neural Information Processing Systems, 33:3058- 3069, 2020.
Filip De Roos, Alexandra Gessner, and Philipp Hennig, High-dimensional Gaussian process infer- ence with derivatives. International Conference on Machine Learning, pp. 2535-2545. PMLR, 2021.
Michal Derezinski, Rajiv Khanna, and Michael W. Mahoney. Improved guarantees and a multiple- descent curve for Column Subset Selection and the Nystrom method. Annual Advances in Neural Information Processing Systems 33: Proceedings of the 2020 Conference, pp. 4953 4964, 2020a.
Michal Derezinski, Feynman T. Liang and Michael W. Mahoney. Exact expressions for double descent and implicit regularization via surrogate random design. Advances in neural information processing systems, 33:5152-5164 2020b.
Edgar Dobriban and Stefan Wager. High-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics, 46(1):247-279, 2018.
Alan Edelman and N. Raj Rao. Random Matrix Theory. Acta numerica, 14:233-291 297, 2005.
Noureddine El Karoui. The spectrum of kernel random matrices. The Annals of Statistics, 38(1):1- 50, 2010.
Andreas Engel and Christian P. L. Van den Broeck. Statistical mechanics of learning. Cambridge University Press, New York, NY, USA, 2001.
Zhou Fan and Zhichao Wang. Spectra of the conjugate kernel and neural tangent kernel for linear- width neural networks. Advances in Neural Information Processing Systems, 33:7710-7721, -7711, 2020.
10

---

Under review as a conference paper at ICLR 2023
Edwin Fong and Chris C. Holmes. On the marginal likelihood and cross-validation Biometrika, 107(2):489 496, 2020.
Vincent Fortuin, Adrià Garriga-Alonoo, Florian Wenzel, Gunnar Rätsch, Richard Turner, Mark van der Wilk, and Laurence Aitchison. Bayesian neural network priors revisited. arXiv preprint arXiv:2102.06571, 2021.
Peter I. Frazier. tutorial on Bayesian optimization. arXiv preprint arXiv: 2018.
Spencer Frei, Niladri S. Chatterji, and Peter L. Bartlett. Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. arXiv preprini arXiv:2202.055928, 2022.
Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. PAC-Bayesian theory meets Bayesian inference. arXiv preprint arXiv:1605.086666 2016.
Ethan Goan and Clinton Fookes. Bayesian neural networks: An introduction and survey. In Case Studies in Applied Bayesian Data Science, pp. 45-87. Springer, 2020.
Franz Graf, Hans-Peter Peter Kriegel, Matthias Schubert, Sebastian Pölsterl, and Alexander Cavallaro.
2D image registration in CT images using radial image descriptors. International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 607-614. Springer, 2011.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949-99) 2022.
Ali Hebbal, Loic Brevault, Mathieu Balesdent, El-Ghazali Talbi, and Nouredine Melab. Bayesian optimization using deep Gaussian processes. arXiv preprint arXiv: 2019.
David Holzmüller. On the universality of the double descent peak in ridgeless regression. In Inter- national Conference on Learning Representations, 2020.
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What Bayesian neural network posteriors really like? In International Conference Machine Learning, pp. 4629-4640. PMLR, 2021.
Konstantin Krivoruchko and Alexander Gribov. Evaluation of empirical Bayesian kriging. Spatial Statistics, 32:100368, 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Loic Le Gratiet and Josselin Garnier. Asymptotic analysis of the learning curve for Gaussian process regression. Machine learning, 98(3):407-433, 2015.
Armin Lederer, Jonas Umlauft, and Sandra Hirche. Posterior variance analysis of Gaussian pro-
cesses with application average learning curves. arXiv preprint arXiv:1906.0140% 2019.
Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel "ridgeless" regression can gener- alize. The Annals of Statistics, 48(3):1329-133 2020.
Zhenyu Liao, Romain Couillet, and Michael W. Mahoney. A random matrix analysis of random
Fourier features: beyond the Gaussian kernel, a precise phase transition, and the corresponding double descent. Advances in Neural Information Processing Systems, 33:13939- 2020.
Fanghui Liu, Zhenyu Liao, and Johan Suykens. Kernel regression in high dimensions: Refined anal- ysis beyond double descent. International Conference on Artificial Intelligence and Statistics, pp. 649-657. PMLR, 2021.
Marco Loog, Tom Viering, Alexander Mey, Jesse H. Krijthe, and David M.J. Tax. A brief prehistory of double descent. Proceedings of the National Academy of Sciences, 117:20)::0225-06626 2020.
Charles H. Martin and Michael W. Mahoney, Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior. Technical Report Preprint: arXiv:1710.0955.. 2017.
11

---

Under review as a conference paper at ICLR 2023
Song Mei and Andrea Montanari. The generalization enror of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667-766, 2022.
Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, Arthur Gretton, and Bernhard Schölkopf. Kernel mean estimation and stein effect. In International Conference on Machine Learning. pp. 10-18. PMLR, 2014.
Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpo- lation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory, 1(1): 67-83, 2020.
Preetum Nakkiran, Prayaag Venkat, Sham M. Kakade, and Tengyu Ma. Optimal regularization can mitigate double descent. International Conference on Learning Representations, 2020.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021 (12) 2021.
Lorenzo Noci, Kevin Roth, Gregor Bachmann, Sebastian Nowozin, and Thomas Hofmann. Dis- entangling the roles of curation, data-augmentation and the prior in the cold posterior effect. Advances in Neural Information Processing Systems, 34, 2021.
Manfred Opper and Francesco Vivarelli. General bounds on Bayes errors for regression with Gaus- sian processes. Advances in Neural Information Processing Systems, 11, 1998.
Debashis Paul and Alexander Aue. Random matrix theory in statistics: A review. Journal of Statistical Planning and Inference, 150:1-29, 2014.
Stephen Roberts, Michael Osborne, Mark Ebden, Steven Reece, Neale Gibson, and Suzanne Aigrain. Gaussian processes for time-series modelling. Philosophical Transactions of the Royal
Society A: Mathematical, Physical and Engineering Sciences, (71(1984):201 10550, 2013.
Peter Sollich. Learning curves for Gaussian processes. Advances in neural information processing systems, 11, 1998.
Peter Sollich and Anason Halees. Learning curves for Gaussian process regression: approximations and bounds. Neural Computation, 14(6):1393-142) 2002.
Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods: empirical data versus teacher-student paradigm. Journal of Statistical Mechanics: Theory and Experiment, 2020(12):124001 2020.
William E Strawderman. On charles stein's contributions to (in) admissibility. The Annals of Statis- fics, 4934):1823-11855. 2021.
2021.
Alexander Tsigler and Peter L. Bartlett. Benign overfitting in ridge regression. arXiv preprint arXiv:2009. /4286, 2020.
Tom Viering and Marco Loog. The shape of learning curves: a review. arXiv preprint arXiv:2/03. 10948, 2021.
Ulrike von Luxburg and Olivier Bousquet. Distance-based based classification with lipschitz functions. J. Mach. Learn. Res., 5(Jun):669-663 2004.
Ke Wang, Vidya Muthukumar, and Christos Thrampoulidis, Benign overfitting in multiclass classi- fication: All roads lead to interpolation. Advances in Neural Information Processing Systems, 34, 2021.
Florian Wenzel, Kevin Roth, Bastiaan S. Veeling, Jakub Swiętkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the Bayes posterior in deep neural networks really? arXiv preprint arXiv:2002.02405, 2020.
Christopher K. I. Williams and David Barber. Bayesian classification with Gaussian processes. IEEE Transactions pattern analysis and machine intelligence, 134 1998.
12

---

Under review as a conference paper at ICLR 2023
Christopher K. L. Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
Christopher K. I. Williams and Francesco Vivarelli. Upper and lower bounds on the learning curve for Gaussian processes. Machine Learning, 40(1):77-102. 2000.
Andrew G. Wilson and Pavel Izmailov. Bayesian deep learning and probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697-4770,, 2020.
Denny Wu and Ji Xu. On the optimal weighted lg regularization in overparameterized linear regres- sion. Advances in Neural Information Processing Systems, 13:10112-10113,, 2020.
Jiaxin Zhang. Modern Monte Carlo methods for efficient uncertainty quantification and propagation:
survey. Wiley Intendisciplinary Reviews: Computaaional Statistics, 13(5):e1539 2021.
Tong Zhang. Learning bounds for kernel regression using effective data dimensionality. Neural Computation, 17(9):2077-200 2005.
13


Reviews:
[{"review_id": "s1GWQ3Et68O", "reviewer": "Reviewer_Qeva", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the question that considers if the marginal likelihood and the posterior predictive losses exhibit a monotone error curve (as other learning models do) and double descent in the input dimension. For that reason, the paper introduces the necessary conditions for proving the results and verify empirically on synthetic problems that this is indeed true or not. ", "summary_of_the_review": "The paper has several flaws that make me doubt about its quality to be accepted in its current format. ", "strength_and_weaknesses": "**Strengths:** The paper adequately addresses the literature around the considered question. The formulation of the problem around the marginal likelihood of GPs and predictive posterior quantities seems correct to me and the obtained theoretical result seems clear to me. Some connections are built, for instance with Fong & Holmes, 2020 and the analysis with respect to the obtained result for the regularisation hyperparameter $\\lambda$ and temperature $\\gamma$ is also clear.\n\n**Weaknesses:** The manuscript presents several weaknesses that I want to remark in the following lines:\n\n- *On GPs, Bayesian statistics and clarity:* Despite the correctness of the question addressed and the considered ideas, I think that the presentation of the GP model plus Bayesian metrics, i.e. posterior predictive and log-posterior predictive density, is far from being clear or easily understandable. This is problematic as this has been longly studied in the recent literature since Rassmussen & Williams (2006), and there are simpler and clearer ways to make the presentation of such model. Example can easily found in the recent literature. Additionally, some use of notation is not entirely positive for me, for instance, the regularisation parameter $\\lambda$ appears in the whole manuscript, when it could be simply added as a kernel scale parameter, right? \n\n- *Imprecise claims and sentences:* I find somehow some imprecise comments and claims in the paper that make me doubt or be unsure of the technical claims around the GP analysis. For instance:\n\n> In practice, the hyperparameters (...) are often tuned to minimize the Bayes free energy. This is an\nempirical Bayes procedure, and typically achieves excellent results (Krivoruchko & Gribov, 2019)\n\nExhaustive efforts for training GP models (including tuning of hyperparameters) based on the log-marginal likelihood have been done in the last 15 years. Sentences as the previous one, are somehow surprising to me. Even if the Bayes free energy is defined as the negative log-marginal likelihood...\n\n- *Connections with CV, but no further analysis:* The results presented by Fong & Holmes (2020) were extremely revelant for the community and particularly for the connection between log-marginal likelihood, log-predictive scores and average k-fold-out cross validation (CV). This connection is mentioned before Section 2.3. and where the average $n^{-1}F_n$ is first described. However, to my eyes, this seems extremely related to the Theorem 1, but no mentions or links to Fong & Holmes (2020) are added. I really do not perceive much of a difference between the two results, and make me think if some this analysis is not partially covered in the previous work.\n\n- *Analysis in a very reduced scenario:* The theoretical result is interesting to me, but seems quite reduced or constrained for GPs, when it seems could be considered from a wider perspective of more general Bayesian models. Additionally, I find the practical utility missing or at least not mentioned or properly discussed.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_K9RyNIw-A", "reviewer": "Reviewer_Qcnt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "There has recently been renewed interest in further studying the theoretical foundations influencing the quality of increasingly parametric models under conditions of larger dataset sizes and higher dimensionality. This paper focus on the latter case, with particular focus on how conditions observed for learning curves on predictive performance carry over to metrics that instead describe uncertainty quantification (UQ). By focusing on non-parametric Gaussian process models, the authors illustrate how increased dimensionality impacts the quality of uncertainty estimates returned by such models. With some assumptions on the choice of kernel used and the nature of additional dimensions added to existing data, the authors are able to formalize how the aforementioned UQ metrics are expected to change as input dimensionality is increased. These claims are verified using a variety of experiments on synthetic and real-world datasets.", "summary_of_the_review": "I consider this to be a solid paper, which delivers interesting theoretic insight that could inspire further research in this direction. I do think that the paper could benefit from certain concepts being better illustrated overall however, for which some rewriting may be necessary.", "strength_and_weaknesses": "- There is always great value in papers that contribute towards demystifying how widely-used models behave under controlled set-ups, and the insights provided in this paper are expected to be of particular interest to researchers working on Gaussian processes and Bayesian deep learning models. \n- I particularly appreciated how the authors considered different measurements of uncertainty quantification, and illustrated how the expectations for how learning improves with added dimensionality vary in each case. While this entails that some of the findings are less definitive, the work comes across as more complete as a result of their inclusion.\n- Although the findings rely on making certain assumptions on the choice of kernel, I believe that the findings are still sufficiently broad, and it makes sense to leave further extensions, such as adapting to NTKs, for future work.\n- One aspect I would have liked to see more of is how the findings of this submission influence future work on Gaussian processes and other Bayesian models? Are there any specific takeaways which might guide the design of models yielding improved estimates with better uncertainty calibration? \n- Furthermore, although the authors highlight the importance of additional covariates being Gaussian for some of the derived theorems to hold, it wasn’t clear whether there are also any actionable learnings practitioners could take from this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SeGzRizVLW", "reviewer": "Reviewer_ZFnV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In their manuscript, \"Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes\", the authors consider the topic known in machine learning as 'double descent' which speaks to an important question for anyone using spectral representations of Gaussian processes for predictive modelling: how many features should I use?  A particular emphasis is given here to the performance of these models in terms of uncertainty quantification. Theoretical results are derived and numerical examples are given in some well- and miss- specified settings.\n", "summary_of_the_review": "The topic is of importance and the results seem correct, but there is missing a strength of argument for the value of the insights given by these results. \n", "strength_and_weaknesses": "The authors demonstrate a strong knowledge of the recent literature in this field.  I have not checked the proofs (though I would happily do so if a revised manuscript is submitted); however, the conditions supposed and the theorem statements themselves are consistent with what I what expect from a proof in their field (i.e., address the natural questions such as differentiability at the origin for the chosen kernels).\n\nHowever, I struggle to see the value of the theorems given for understanding the real world behaviour of Gaussian process models, owing to the limitations of the problem set up: \n- foremost: Unless I have gravely misinterpreted the problem description, the \"best-case scenario\" (page 6) assumed is that all the data are generated under an iid noise model.  In this case, any improvement that an estimator that correlates the prediction to new locations against data from currently observed locations can give over the naive model (no correlation; modulo that potentially induced by learning a pooled value for the scale of e_i) is the domain of the original Stein effect (Stein 1956).  This has been examined previously for the case of RFF kernel approximations by (e.g.) Muandet et al. 2014 and Chang et al. arXiv:1705.08525).  For these particular Stein effect results to be valuable the authors would need to explain---either theoretically, or just heuristically but with convincing numerical examples---why the results for an iid noise model should be of value to practitioners trying to model highly correlated (essentially, 'functional') data.\n- second: the restriction to a setting in which estimation of a data-driven bandwidth is not included in the empirical Bayesian solution feels like an important limitation relative to the frequency of this additional calibration step in practice.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject