The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
Authors: Geoff Pleiss
John Patrick Cunningham
Venue: NeurIPS 2021 Poster (2021)
Keywords: neural networks
deep gaussian processes
bayesian neural networks
neural network priors
infinite width limits

Abstract: Large width limits have been a recent focus of deep learning research: modulo computational practicalities, do wider networks outperform narrower ones? Answering this question has been challenging, as conventional networks gain representational power with width, potentially masking any negative effects. Our analysis in this paper decouples capacity and width via the generalization of neural networks to Deep Gaussian Processes (Deep GP), a class of nonparametric hierarchical models that subsume neural nets. In doing so, we aim to understand how width affects (standard) neural networks once they have sufficient capacity for a given modeling task. Our theoretical and empirical results on Deep GP suggest that large width can be detrimental to hierarchical models. Surprisingly, we prove that even nonparametric Deep GP converge to Gaussian processes, effectively becoming shallower without any increase in representational power. The posterior, which corresponds to a mixture of data-adaptable basis functions, becomes less data-dependent with width. Our tail analysis demonstrates that width and depth have opposite effects: depth accentuates a model’s non-Gaussianity, while width makes models increasingly Gaussian. We find there is a “sweet spot” that maximizes test performance before the limiting GP behavior prevents adaptability, occurring at width = 1 or width = 2 for nonparametric Deep GP. These results make strong predictions about the same phenomenon in conventional neural networks trained with L2 regularization (analogous to a Gaussian prior on parameters): we show that such neural networks may need up to 500 − 1000 hidden units for sufficient capacity - depending on the dataset - but further width degrades performance.

The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
Geoff Pleiss Columbia University 1aap2162@colubiaan
John Cunningham Columbia University ipe2181 @columbia.edd edu
Abstract
Large width limits have been a recent focus deep learning research: modulo com- putational practicalities, do wider networks outperform narrower ones? Answering this question has been challenging. as conventional networks gain representational power with width, potentially masking any negative effects. Our analysis in this paper decouples capacity and width via the generalization of neural networks to Deep Gaussian Processes (Deep GP), a class of nonparametric hierarchical models that subsume neural nets. In doing so, we aim to understand how width affects (standard) neural networks once they have sufficient capacity for given mod- eling task. Our theoretical and empirical results on Deep GP suggest that large width be detrimental to hierarchical models. Surprisingly, nonparametrie Deep GP converge to Gaussian processes, effectively becoming shallower without any increase in representational power. The posterior, which corresponds to a mixture of data-adaptable basis functions, becomes less data- dependent with width. Our tail analysis demonstrates that width and depth have opposite effects: depth accentuates a model's non- Son--ausianntt. while width makes models increasingly Gaussian. We find there is "sweet spot" that maximizes test performance before the limiting GP behavior prevents adaptability, occurring at width or width for nonparametric Deep GP. These results make strong predictions about the same phenomenon in conventional neural networks trained with I.2 regularization (analogous to a Gaussian prior on parameters): we show that such neural networks may need to 500 1000 hidden units for sufficient sapacity-dependee on the dataset-but further width degrades performance.
prove that even
Introduction
Research has shown that deeper neural networks tend to more expressive and efficient than wider networks under variety of metrics [c.g. 21, 63, 67, 70, 74, 75, 78, 83]. Nevertheless, there is resurgent interest in wide models due in part to empirical successes [e.g. 92] and theoretical analyses of limiting behavior. When randomly initialized to create distribution over functions, neural networks converge to Gaussian processes (GP) as width increases. This result, first proved for 2-layer networks [69]. has been extended to deeper networks [56, 64), convolutional networks [38, 71], and other architectures [50, 88]. A similar limit exists for gradient-trained networks, which behave increasingly like kernel machines under the neural tangent kernel [e.g. 6, 8, 28, 39, 52, 57, 89].
While these limits simplify analyses, there something unsettling about reducing neural networks to kernel methods. Neal [69, p. 161] describes the GP limit as "disappointing." noting that "infinite networks do not have hidden units that represent 'hidden features' ... often seen [as the] interesting aspect of neural network learning." Recent work indeed shows that learned hierarchical features can be exponentially more efficient than the fixed shallow representations of kernels [e.g. 4, 5, 8, 11, 13, 21, 41, 42, 60, 91]. At the same time, wider networks more accurately model complex functions
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

[44]. Thus, wide limits appear to confound opposing phenomenon: increased capacity makes them more expressive, yet the loss of hierarchical features seems make them less expressive. This may explain the mixed empirical performance of limiting models: outperforming finite width models in some scenarios [e.g. 9, 38, 39, 58], yet falling short on more complex tasks [e.g. 8, 11, 35, 57, 81].
This paper aims to decouple these effects of large width. Our goal is to understand the inductive biases of wide networks, after a network has "sufficient" capacity for given modeling task. We ask: If we control for the effects of increased capacity, what-if any-value remains in wide networks?
To achieve this control, note that typical neural network layer corresponds to a finite basis, where elementwise nonlinearities transform each hidden feature into a single basis function. In order to decouple width from capacity, one could generalize these layers so that each nonlinearity produces any number of basis functions; if each hidden feature gives rise to an infinite and universal basis, then hidden layers would have infinite representational capacity regardless of width. This generalization is in fact well-studied class of hierarchical models-Deep Gaussian Processes (Deep GP) [19, 24, 26, 27, 30, 32, 46, -9e-w standard neural net layers are replaced with vector-valued Gaussian processes. Indeed, typical neural networks are a degenerate Deep GP subclass [1, 2, 33, 72].
We therefore have generalization of neural networks where capacity is controlled, from which we can glean insights about conventional networks that have sufficient representational power for a given modeling task. Surprisingly, despite using Gaussian processes as the primary hierarchical component, we prove that Deep GP converge to (single-layer) GP in their infinite width limit (Thm. 1). Troubling implications immediately ensue: large width is strictly detrimental to Deep GP, as the limiting model collapses to a shallower version of itself. We support this theorem with an analysis of neural network and Deep GP posteriors, which become less adaptable as width increases. Specifically, we show that the posterior mean corresponds to mixture of functions drawn from data-dependent (and thus adaptive) reproducing kernel Hilbert spaces, formalizing the above claim from Neal [69]. As width increases, this mixture collapses to the data-independent kernel of the limiting GP, implying that wider models have less feature learning. Finally, we present novel tail analysis which indicates that width and depth have opposite effects: depth accentuates non- -oon-aassiaatty sharpening peaks and fattening tails, whereas width increases Gaussianity (Thms. and 3).
Our theoretical results hold for Deep GP and conventional (parametric) neural networks alike. Experiments confirm that- after a model achieves sufficient capacity! wwidt can become harmful to model fit and performance. For nonparametric Deep GP, a width of or 2 often achieves the best performance. Neural networks- hidden units before achieving optimal accuracy. Nevertheless, for Bayesian neural networks and conventional (optimized) neural networks trained with L2 regularization, performance degrades after a certain width. On small datasets (N < 1000) with low dimensionality, we find that models with S 16 hidden units achieve best test set performance. On larger datasets like CIFARIO, this "sweet spot" occurs later (at x 500 hidden units for sufficiently deep models), yet performance degrades beyond this width. We note that these trends do not necessarily hold for models that do not have a probabilistic interpretation-ii. optimized neural networks trained without (or nearly without) I.2 regularization. Nevertheless, our findings suggest that narrower models have better inductive biases, and wide models perform well in spite of not because of large width.
of their parametric nature-naturea natura-naturally require more
no-not
2 Setup
2.1 Related Work
Effects of width. Works have shown that, given finite parameters, deeper models are more expressive than wider models [63, 67. 74, 75, 83]. Similarly to our work, Aitchison [2] recognises the link between finite neural networks and Deep GP, and argues that finite neural networks have flexibility in the tap-layer representation that is absent in the infinite-width limit. Halverson et al. [45] draw a connection to quantum field theory to argue that neural networks become "simpler" near their infinite-width limit. In the non non- non-probabilistic setting, it is worth noting that wide models have been shown to have favorable optimization landscapes [7, 28, 59, 70, 82] and are resistant to overfitting via double descent [12, 20, 68]. Our work controls for these factors by examining nonparametric hierarchical models with exact Bayesian inference, and thus does not disagree with these other works.
We offer a formal notion of "sufffcient capacity" in Appx. B.5.

---

Infinite width limits have received renewed interest in Bayesian [38, 50, 57, 64, 69, 71, 88] and con-Baysiian [6, 8, 22, 28, 43, 52, 57, 65, 89, 90] settings. Most of these works show that neural networks converge to kernel methods, though recent work suggests that this limiting behavior can be avoided with different parameterizations [e.g. 22, 43, 65, 90]. Similarly to Lee al. [57], our Deep GP limit analysis sequentially increases the width of each layer, though we hypothesize similar proof exists where the width of all layers increases simultaneously (akin to [64]).
Deep GP are introduced by Damianou and Lawrence [27]. A large portion of Deep GP research has thus far focused on scalable approximate inference methods [19, 24, 25, 32, 46. 72, 79, 85]. Though prior work has studied tail properties of neural networks [84, 93] and Deep GP with RBF kernels [62], our work is-to the best of our knowlege-thh first general result for Deep GP tails. Duvenaud et al. [33] and Dunlop et al. [29] investigate pathological behaviors that arise with depth, while grrawal et al. [1] note that "bottlenecked" Deep GP have better performance and correlations among predictive tasks. Our work complements these analysis by characterizing the effects of width.
Connections between Deep GP and neural networks. Many researchers have noted connections between neural networks and Deep GP [e.g. 24, 31, 36, 61]. Duvenaud al. [33] suggest that infinitely-wide neural networks with intermediate bottleneck layers are nonparametric Deep GP. Agrawal al. [1] formalize this connection, but note that not all Deep GP can be constructed from bottlenecked neural networks (see Appx. H). In contrast to these prior works, we avoid reducing Deep GP to neural networks, and instead reduce neural networks to degenerate Deep GP.
2.2 A Covariance Perspective on Gaussian Process Limiting Behavior
To decouple the effects of increasing width and capacity, we first prove a new result about GP limits for a more general class of models, including Deep GP well typical neural networks. This result forms a necessary foundation for the subsequent theorems that are main contribution of this work. To begin. note that the proof technique introduced by Neal [69] and extended by others [38, 50, 56, 64, 71, 88] relies on the multivariate central limit theorem, which requires a model with additive structure. Deep GP do not generally decompose in an additive manner, so we establish more general proof technique. For simplicity, we first present it in the context of neural networks, and then extend a more general class of models.
Consider the 2-layer neural network (f (x)), with fi RD + RR and f2 R + R:
(()) = WTW () + Ab, fz(-) = 1 VH. W2 (.) + Bb2.
(() is an elementwise nonlinearity, ,3 is positive constant, and W1, bı, W2, and b2 are i.i.d. Normal. With randomly initialized parameters, (2) : R - IR is a prior distribution over functions, and this distribution converges a GP in the infinite width limit [69].
Lemma 1. The neural network defined in Eq. (1) is a Gaussian process if and only if for any finite set of inputs X = [x1,.... , xN] x the conditional prior covariance (X,WW, [f2,, is almost surely equal to the marginal prior covariance Fx,,x [f2[]]] where fz X A
(xxxxxx)
Proof. By definition, ff (-)) is a GP if and only if fz X is multivariate Gaussian for any X. From Eq. (1), we have p(fp X, W, b1) = (OO ()),,, where [Kw,bb (X, X)]is = 82 + H. T (W x; + áb,)* 3 a ( x, + {b,) is the appropriate kernel Gram matrix. Using Jensen's inequality, we have lower bound on the characteristic function of fz X:
E [exp (it T fe)] = E f
Texp (it T bb]]
(law of total expectation)
Wr.b,
& W.W.b
= -Ew,b,
exp (- 1t K. (X, X)t)
(char. func. of a Gaussian)
2 exp
-JJt 1T E Kw.bb, (X,x)]t)
(convexity of exp)
Wi.b;
This lower bound happens to be the characteristic function of (0, Ew,,b. Kw,, (X,X)): Since exp is strictly convex. the characteristic function of fz X equals the Gaussian lower bound Vt if and only if p(Kw((WW)) b, (X, X) bi) Eqx, n.x,,ar b, [ett is constant with probability 1. D
3

---

Seeing that 1 oWW x; + {b1) a ( x, + {b1) becomes a.s. constant as H1 + oc, Lemma re-establishes the result of Neal [69] (see Appx. E.1). Critically, unlike Neal's proof. Lemma 1 neither relies on the central limit theorem nor requires (2(fi to be neural network; holds p(f2 f (x1), (xw))) is Gaussian. Therefore, we can generalize it to a larger class of models:
H.
Lemma 2. Let (z(((()) RRD TR be a hierarchical model where f2()) : IR - R. is GP and f((): RDD - IK is a random vector- function (including multilayer hierarchical model). Then f-) is GP if and only if (F.)x,f(() [2 = Er, x xff2f]] a.s. for all X = [x1..... x
The covariance perspective from Lemmas 1 and 2 is revealing about GP limits. As ( [f2[]] converges to 1x [f2ff]]] the model output becomes less and less dependent on f-).. In other words, (-) loses its hierarchical nature. We reiterate that Lemma 2 has no requirements about fe)
transitioning from a finite to infinite basis, nor does it require (fi (-)) to have additive structure. We
demonstrate its generality in the next section with surprising- and mubblig--impliattinnn
3 Deep Gaussian Processes Collapse to Shallow Gaussian Processes
Deep GP [19, 26, 27, 79] are hierarchical models where layers
are (vector-valued) GP:
DGP(x) = fr. .... o (x),
- = (). (H) ()), 1(0()) lid gP (o (2)
H; is the width of the Th GP layer, and the output dimensions of each f.).) are independent. By using GP as the primary hierarchical building blocks, Deep GP are generally nonparametric and, assuming the GP layers use universal kernels [66], have infinite representational capacity (see Appx. B.1).
Deep GP versus GP. Deep GP seek to offer more expressivity: conventional single-layer though also nonparametric-arr are inherently limited by the choice of the prior covariance function [19, 79]. For example, a GP with a RBF covariance is not suitable for data with discontinuities or sharp changes. However, stacking two RBF GP together-fzll (-)) (-)) f, (x) can encode a warping of x that "smoothes" the input data for fz(-) (as we will show in Fig. 1). Empirically, Deep GP have been shown offer much more accurate predictive posteriors than standard GP [e.g. 17, 24, 26, 27, 30, 46, 79].
overcome this limitation, since
Deep GP versus neural networks. (Bayesian) feed-forward neural networks are strict subclass of Deep GP, albeit a degenerate one [2, 61, 72]. The first neural network layer is a GP with linear kemel,, while subsequent layers are GP with the kernel k(z, z') 3 + H Σ (2x) (2;). A neural network, unlike other Deep GP, does not have infinite capacity. Put loosely, a single neural network hidden unit corresponds to a single basis, while in general single Deep GP unit corresponds to potentially-infint basis. See [1, 2, 24, 31, 33. 72] and Appx. B.2 3.2 for more discussion on this connection. The critical takeaway is that all of our Deep GP results apply to neural networks as well.
3.1 Wide Deep GP are Gaussian Processes
Having established model where width does not effect capacity, we now establish what remaining effects width has. Empirical evidence suggests that the choice of width impacts Deep GP predictions [19, 46). In practice it is common to make Deep GP as wide as comparably-sized neural networks; Salimbeni and Deisenroth [79] for example train Deep GP with 2 30 units per layer.
Surprisingly, here we prove that-in in the limit of infinite width- Width-Deep GP collapse to single-layer Gaussian processes. Our proof relies the conditional covariance analysis of the previous section. the GP layers have non- nathpatoolooicll covariance functions² the Deep GP conditional covariance becomes almost surely constant with width (see Lemma 3, Appx. E). Combining this with Lemma 2:
Theorem Let fo o.... fj (x) be a zero-mean Deep GP (Eq. 2), where each layer satisfies Assump- tions and 2 (non-pation pathooloool prior covariances that scale with dimensionality- see Appx. E.3). Then lim 1000
lim os Jt. o ...... o fi (x) converges in distribution to a (single-layer) GP.
2 Any textbook kernel (isotropic kernels, dot product kernels, etc.) any covariance function with a Fourier Fourier-Steiljes representation is "nou- pothoooholo Appx. E.3 for formal assumptions.

---

Posteriot F'b (Wklth
Posterior (Wilth 16)
Posterior (Wilth 256)
Posterior (Wilth o0)
falf
Prateriur
1.0
0.5
0.0
0.5
10
100
0.5
0.0
0.5
10
10
6.5
0.0
0.5
10
10
0.5
6.0
0.5
10
Postoriorr Kern. (Widch
Praterior Korn. (Width 16)
Posterior Kem. (Width 250)
Posterior Kem. (Width o0
1.0
0.5
0.0
0.5
1.0
-.0
-0.5
0.0
0.5 1.0
-1.
-0.5
3.0
0.5
1.0
-1.0 -0.6
0.0
0.5
1.0
Figure 1: Top: Posterior 2-layer RBF Deep GP to noisy step function. A width-1 Deep GP the discontinuity at 0. As width increases, the Deep GP converges to GP with stationary covari- ance unable to the step. Bottom: Average posterior covariance E,, (x) (x))ly [ky (fi (x), fi (x'))] The width posterior covariance is non-stationary, with little covariance around x 0. As width increases, the posterior covariance becomes stationary (as seen by the kernel's 's constant diagonals).
(See Appx. E for proof.) The implications of Thm. are paradoxical and unsettling. Deep GP motivated as more powerful model than standard GP. However, as we make the model wider, we arrive back where we started- Gaussian process (although with different prior covariance).
neural network gains representational power its limit, transitioning from a finite-basis model to nonparametric model. The Deep GP limit on the other hand has no additional representational power, since Deep GP are already universal approximators at any width. (Indeed this fact motivates their control.) The only difference between finite and infinite width Deep GP is the prior distribution itself: transitioning from Gan-Sausiin to Gaussian with increasing width. In the next section, we investigate how this transition affects model performance.
4 Large Width Limits the Adaptability of Hierarchical Posteriors
Even with Thm. 1 and its troubling suggestions, it is not immediately clear exactly what is lost in the infinite-widtt limit. Here, we quantify specific differences the predictive capabilities of narrow versus wide models. In particular, analyze Deep GP/neural network posterior distributions, rather than focusing on a single model trained through optimization. We show that these posteriors correspond to mixture of dala-dependent adaptable bases; however, width increases this mixture collapses to the (data- seaa--nnddddeentt basis the limiting GP. This result formalizes the often vague notion of feature learning, and demonstrates that it is indeed lost in kernel limits.
Hierarchical posteriors correspond to data-adaptable bases. Consider the (finitewwidth) layer Deep GP (-),, where kn (..-) -) and ka(:,-) -) are the covariance functions of f-()) and fa(-). (xx)] and fg & [/2(f] (x1)),.... f2 (xx))] Let x" be test input, and let fi and fz equal fi (x*) and fe (x*) (see Fig. 5 in Appx. B.3 for a graphical model). Crucially, fo and fž only depend on F and ff through the covariances
Given training data X,y, define F, A (x1),,.......
(2(F,, F1),
k2 (f), and (ff ff) (which we abbreviate as K2, kg, and kg)"):
p(fe K2) N K2). p(f2 kg, K2, ~ N'' (k2 K, fg. Ky K2 K, K,'K;)
By D-separation (e.g. 16, Ch. 8], we can factorize the posterior distribution as: p(fi, f2, K2, kü, k y) p(fī f2, K2, kg*) p(22 K2, See derivation in Appx. B.3, Applying the factorization in Eq. (3), the posterior mean is:
ky ly).
(3)
E [
k
K, [g]
k; K
(4)
K
Kayy
K,
6/K
E
Σ.v as ka(ff (xx)
(5)
(x(x)) (xc)).....
(x
5

---

where the second line follows from K₂ and kj being deterministic given fx(x"), , fy (x1)..... fi (xN).
The term inside the Eq. (5) expectation a function from the reproducing kernel Hilbert space (RKHS)
defined by l2 (-) (-), fi
(-)). We can thus interpret this expectation as an infinite mixture of functions from different Hilbert spaces. Because the mixture distribution (x,((x)) (x)). (xxx y)
depends on y. Eq. (5) is an adaptive data-dependent mixture of RKHS.
Adaptability is lost in the Gaussian process limit. What happens to Eq. (5) as f2((( becomes Gaussian process in the limit of infinite-wwid width? Recall from Lemma 2 that the conditional prior covariance becomes deterministic as (2() converges to a GP. In other words. the prior and posterior distributions over K₂ and kç become atomic: p(K2, kš) = p(Kg, k2j y) 8 Kmm kim where Kiim and km are shorthand for [B[ and [ff respectively. Eq. (4) thus collapses to:
lim 400
E E by
[
E
[kg [K]aa] [a] = Di=1 0g klmm (X,, (x,;x)) x"),
(6)
8 "[Kinkki
which is no longer mixture of functions from different RKHS. It is instead a function from single RKHS (that of the limiting GP prior). 3 In other words, while Deep GP (and neural networks) perform kernel learning (or feature learning) to adapt to training data, this ability is lost with large width.
Example. Consider a Deep GP with RBF covariances k1 (x, x') = exp (-1|x - -x112((()))) and As we show in Appx. G, this Deep GP converges to a GP with Atm x, x') ) = (exp(-eplll))) - x x2112/(2D) - 1). Note that this limiting covariance is stationary and is ill-equipped
(2 (x), (x')) = exp (-11 [ (x) - fi (x))
model the data step Fig. However, because (i(.) is nonlinear, (2 (x), (x)) is nonstationary. Fig. I (top left) shows that the width-1 Deep GP posterior accurately models this data. The posterior covariance Er, (x).6 (x')ly [Ky (f(),, f (x))) (bottom left) features long-ram -range correlations near x = #1 and short-range correlations near x = O. As width increases, we lose this nonstationarity and the posterior becomes a worse fit.
5 The Difference Between Width and Depth: A Tail Analysis
Our work so far has troubling implications for large width. On the other hand, empirical evidence has shown that depth improves Deep GP performance- as it does for neural nets [e.g. 46, 72, 79] (though pathologies can emerge [29, 33]). Through a novel tail analysis, we show that width makes Deep GP priors more Gaussian, while depth makes them less Gaussian. In other words, width and depth have opposite effects on Deep GP tails, results that again also apply to typical neural networks.
Deep GP/neural networks are sharply peaked and heavy tailed. The proof technique used in Lemma I can be used to similarly bound the moment generating function of Deep GP marginals:
( t E K2(L,,, 2
O [c=x]] (7)
E
E
2 cxp
E g0,(O,
F1
where Kmm = -E, [E2f2 = EF, (i),F)) Generalizing these bounds to deeper models, we have:
Theorem 2. Let fr.o... be zero-mean Deep GP. Given finite set of inputs X [xi,..., xN]. define fo Then, p(f, = 0) Ng = 0; 0, K.m).
(x ..... .......f..... (f
E [L, L), and define K Er, [
Theorem 3. Let t E R.N. Using the same setup, notation, and assumptions as Thm. 2, the odd moments of t tf are zero and the even moments larger than 2 are super-Gaussie i.e. 3)"] all even r 2 4. Moreover, if ((())))) is bounded almost everywhere, the moment generating function R,, [exp(t fr.) exists and is similarly super-Gaussian.
Er, [tt fr."²] 2 Eg--N(,....)(
(See Appx. F for proofs.) Thm. 2 states that Deep GP marginals are more sharply peaked than a moment-matched Gaussian, while Thm. states that they are also more heavy tailed.
Increasing depth leads to sharper peaks and heavier tails. To understand how depth affects this tail behavior, we examine the Jensen gap in Eq. (7). Consider a 3-layer Deep GP fa (f, (-)). If we
"To rigorously argue that the infinite-width posterior collapses in this way, we can invoke Proposition from Hron et al. [49]. See Appx. B.4 for details.

---

Depth=2, Wibhe1
Depth=2 Wiith=2
Depthe2,
Depshm2, Widthe4
Lim.
0.330
0.290
0.200
0.200
Depth=3. Wxlhe1
Deplim3, Wkith=2
Depthe3, Widthe3
Depth=3, Widthe4
Lim. GP
0.120
0.080
8.
0.040
0.010
0.001
Figure 2: Marginal densities p(3/,,32 1/2 X1 X2) for zero-mean Deep GP of various depths and widths on the N 2 dataset X 0.5, X2 0.5. All 2-layer models have the same second moments (covariance is that of the 3-layer width = 1 RBF-RBF-RBF Deep GP), Left to right: width increases, marginals become increasingly Gaussian, tails become thinner, and the peak at [y1 y2l 0 loses density. Top to bottom: depth increases, tails become fatter, and the peak becomes sharper.
extend Eq. (7) to 3-layer models, we see that the Jensen gap cascades:
E
E F.
Kgt
E F.
Ks]t)
exp t
E F2
Kg
MGF 3-layer Deep GP marginal
MGF 2-layer Deep GP marginal
MGF of o Er, [EF,IF, IF, [K<]])
where K3 is short for (F(F2(F((X)) (2 (X))). The middle term is the moment generating func- tion of 2-layer Deep GP marginal (where the second layer has covariance (r, The right-most term is the moment generating function of (single-layer) Gaussian. Generalizing this cascade, we see that deeper models are more heavy-tailed. A similar analysis on the characteristic function shows that the peak the prior mean also becomes sharper with depth (see Appx. F).
(-)))
Adding additional layers to Deep GP will change the model's prior covariance, and thus the effects of depth cannot solely be explained by tail analysis [29, 33]. Nevertheless, if we control for this change in covariance, we indeed see that depth leads to heavier tails. In Fig. we compare 2-layer and 3-layer Deep GP. The 3-layer models use GP layers with additively-deccoppssng RBF covariances, while the 2-layer models use layers constructed to match the 3-layer models' prior covariance (see Appx. H for construction details). The N 2 marginal densities for the 3-layer models (bottom row) are more stretched than the 2-layer densities (top row). We further confirm these effects in Appx. D.
Increasing width leads to flatter peaks and Gaussian tails. Conversely, consider what happens when we make the model wider. We define the sequence of increasingly wide 2-layer Deep GP:
AP() lid gP [0, l (;.)), 11())
DGP(M) (-)
(8)
un CP k2(
(GP( (-) is a width-m Deep GP, where the second layer decomposes additively over the m dimensions. By linearity of expectation, each model in the sequence shares the same prior covariance: EJDGP(1) (x) DGP(1) (x') = EJDGP(?) DGP(2) OxP) ... A Kim (x, x'). Though each model has the same marginal covariance, the conditional covariance B IF, ff2f, becomes increasingly concentrated around K X) as increases. This consequentially shrinks the Jensen gap Eq. (7), and so the Deep GP marginals become increasingly Gaussian. We again visualize this effect in Fig. 2, which depicts marginal densities from 2-layer and 3-layer Deep GP of various width (see Appx. H for details). Compared with the limiting
Σm (f( p(1)
(right), the width-1 densities (left) appear sharper near [0, 0] and more stretched at the tails. As width increases, the peaks and tails look increasingly Gaussian (see also Fig. 7 in Appx. D). In this sense, width has the opposite effect depth-deeper marginals are less Gaussian, while wider marginals more Gaussian.
7

---

Yacht
Boston
Energy
Concrete
Power
Poliele 1000, 26)
Tim. GP Logg Tike.
0.50
[N 201, D=
IN 578, = 13)
[N = 516. D =
(N = 772. D =
[N = 1000, D 4)
Lim, GF
Log. Lik.
0.25
SED
9.00
Kennel Fir-
Lim. GP Fit
500
256
256
16
250
250
Width
Width
Width
Width
Wiith
Width
Test set log likelihood (LL) of 2-layer Deep GP (and neural networks) regression as function of width (higher is better). Numbers are shifted so that 0 corresponds to the limiting GP log likelihood. Narrow models achieve the best log likelihood, and performance degrades with width. Bottom: Fit of the posterior kernel (-), (-)) on the training data, as measured by Gaussian log marginal likelihood (higher is better). corresponds to the limiting GP log marginal likelihood. Fit becomes increasingly worse with width.
Figure 3: Top:
Table I:
Test set log likelihood (LL) of Deep GP regression as a function of depth (higher is better). Depth refers the limiting GP. For each dataset, the models are constructed to have the same first and second moments, Unlike width, deeper models generally have better performance.
Depth
(N 291, D = 6)
Yacht
(N 378, D = 13)
Boston
(N = 576, D 8)
Energy
(N =72, D 8)
Concrete
(N
1000, D= 4) (V = 1000, /2 = 26)
Power
PolTele
-0.532
-0.890
-0.477
-0.663
-0.249
-0.176
-0.520
-0.684
-0.434
-0.573
-0.260
-0.381
-0.482
-0.609
-0.383
-0.620
-0.251
-0.318
6 Experiments
6.1 Regression with Deep GP and Bayesian Neural Networks
To isolate the effects of width and depth, each experiment compares Deep GP/Bayesian neural networks that share the same first and second prior moments, and the Deep GP models use GP layers with universal kernels. To remove any potential side effects from approximate inference methods, we sample Deep GP/neural network posteriors using NUTS [48] and do not any stochastic inducing point [46, 79] or finite basis [24] approximations. This inference is costly and scales cubically with N; therefore, we subsample all training datasets N < 1000. See Appx. H for experimental details.
Effect of width. We compare 2-layer Deep GP of various width on 6 regression datasets from the UCI dataset repository [10] (see Appx. for 3-layer results). The first GP layers use a RBF kernel for the prior covariance, while the second layers use sum of one-dimensional RBF covariance functions. We additionally compare against the limiting (single-layer) GP with the same prior covariance (Lim. GP). For each dataset, we choose hyperparameters that maximize the Lim. GP log marginal likelihood. In Fig. 3 (top row) we see a near- menotmont performance degradation as width increases. The width optimum may represent the "sweet spot" for Deep GP width, but it may instead be side-effect of inference difficulties for width our theory predicts, width is detrimental to Deep GP predictive performance.
models (see Appx. D for a control experiment). Regardless, as
We repeat the experiment for 2-layer neural networks (and 3-layer models in Appx. D). where here the Lim. GP corresponds to the arc-cosine kernel [23, 56]. Fig. 3 indicates an optimal width with regards test set log likelihood, usually between 8 16 hidden units. We expect this optimum exists (and differs from the Deep GP optimum) because narrow models have too few basis functions for these datasets. Nevertheless, after sufficient capacity, width harmful to Bayesian neural networks.
Adaptable versus non-adaptable RKHS. One way to measure the "fit" of a kernel k(.,.) on regression training dataset X, y is the Gaussian log marginal likelihood Nogg((y:: (y; O, K(X, oo)) where o² is observational noise parameter [c.g. 77]. To demonstrate how Deep GP/neural network
8

---

MNIST (NV 500)
MNIST (N 50XID)
MNIST (N = 30000)
CIFARIO (N
50000)
JL NN, Adam
31. NN, Adam La
3L NN, Adam L2
ResNet, Adam Lg
0.99
0.89
Accuray
0.966
Accuracy
0.88
0.98
0.85
0.87
0.95
Lamiom
L2=10-1
0.97
L2=10
0.80
Depth
0.86
1.3110
Deptho244
0.94
10
10P
10 10
10a
10
10P
10
10
10
10a
Width
Width
Width
Width
Figure 4: Effect of width on standard (non-Bayesian) neural networks. Shaded regions depict standard error. Left: 3-layer MLP trained on subsets of MNIST. With large values of 1.2 regularization, model performance is maximized when width < 1.000. For small values of L2 regularization (e.g. 10 10-5, which corresponds to a prior of (0, 20,000) on the parameters), there is little accuracy loss with increasing width. It is possible that theory does not apply to models with little L2 regularization which have little Bayesian interpretation. Right: Wide ResNet models (8-layer and 1 14-layer variants) trained on CIFAR-10. For both depths, accuracy is optimal when width S 500.
posteriors correspond to adaptable RKHS mixtures, the bottom row of Fig. 3 plots the "kernel fit" of (.). fi() (-)) for posterior samples of f (-) (see Eq. 5). A higher fit corresponds to a model that is better adapted to the dataset X. y. We that narrower Deep GP almost universally achieve better kernel fit than wider Deep GP, which converge to the same fit as the limiting GP. (Standard deviations, depicted by shaded regions, are generally imperceptible.) Bayesian neural networks achieve best "kernel fit" at 8 - 16 hidden units, and then converge to the limiting Deep GP with further width.
Effect of depth, controlling for covariance. Table displays Deep GP test set log likelihood as a function of depth. Again, we isolate the tail effects of depth by ensuring that all models share the same first and second moments. We construct GP and 2-layer Deep GP that match the moments of a 3-layer width = 2 Deep GP with RBF covariances, and we use hyperparameters that maximize the limiting GP marginal likelihood for each dataset. Note that computing the limiting covariance of 2 layer models involves intractable integrals that we approximate with quadrature (see Appx. G). Our findings confirm that-in this controlled setting-depth unlike width improves test set performance.
6.2 Standard (Optimized, Non-Bayesian) Neural Networks
We now turn to standard (optimized, oon-Bayesian) neural networks. While our theoretical results primarily apply to full posteriors over models, our goal is to see if our theory can also be predictive in "real world" neural networks without a Bayesian treatment. There is reason to believe that our theory should be applicable in these settings, since standard neural network training with L. regularization equivalent to maximum posteriori inference with Gaussian priors. To that end, we ensure some correspondence between these experiments and our Bayesian experiments. In particular, we measure the effects of width on networks with fixed values of L2 regularization, which corresponds to a fixed prior on neural network parameters. Additionally, models are trained without data augmentation, as data augmentation does not have a probabilistic interpretation [51].
Fig. 4 (left) depicts test set accuracy for increasingly wide models trained on MNIST [55]. Each network is a MLP with 3 layers (i.e. 2 hidden layers). Following the GP-limiting neural network construction in Eq. (I), we scale the outputs of layer l by 1/ H We measure the effect of width over networks with various L2 regularization constants (10- 5 10-6 6 10-7, and 10-8) which respectively correspond to priors of (0, 2), (0, 20), (0, 200), and (00 2000) when N 50,000. We train these sequences on various-sized subsets of the training data (N 500, N 5,000. and N = 50,000). From this figure we can observe several phenomena. For larger values of L2 regularization, we see a distinct maximum in accuracy, typically around width R 1,000. For smaller values of 1.2 regularization, wider models tend to perform better (and indeed, for this dataset/model combination it appears that less regularization tends to be beneficial to overall performance). We would note that these low regularization constants correspond to arguably unreasonable parametric
In other words, we do not consider the regularization constant to be a hyperparameter that we optimize over for the purposes of these experiments.

---

priors like (0, 2000), and so a Bayesian interpretation of these models may not be applicable. In such settings, is more likely that the interpolation analysis of Belkin et al. [12] is better model of performance, since this analysis explicitly focuses on the low-regularization setting.
Fig. 4 (right) depicts 8- and 14-layer ResNets [47] trained the CIFAR-10 dataset [54]. We use the hyperparameters from the original ResNet paper, which have been shown to be efficacious on both narrow and wide variants of ResNet models [92]. (This training procedure uses a L2 coefficient of 10-4, which corresponds to a prior of N (O, 0.2) for each parameter when N = 50,000.) For both depths, we observe that performance optimal when width is between 500 and 1,000. While is possible that different hyperparameters may yield different outcomes, these results indeed suggest that large width can adversely affect standard neural networks once sufficient capacity is reached.
7 Discussion
This paper shows that, across typical neural networks (with L2 regularization), Deep GP, and Bayesian neural networks, large width can be detrimental to model performance.
Even with these results, we can ask when width might be desirable? First, we note that our results analyze exact posteriors or MAP solutions, and does not focus on practical considerations with regards to obtaining these solutions. We do not consider the effect that width might have on approximate inference methods, which are commonly used with Bayesian neural networks and Deep GP in practice [e.g. 18, 34, 79J. For conventional neural networks, poor conditioning and non-convexity make it challenging to obtain a MAP solution. The optimization dynamics-which depend on numerous factors like learning rates, initializations, and choice of optimizer- may be improved by width, as wider models tend to have more favorable optimization landscapes [7, 28, 59, 70, 82). Consequentially, wider models may obtain better performance due to these practical considcrations.
Secondly-as as noted in Sec. while we notice detrimental effects of width on neural networks with a Bayesian interpretation (i.e. inferring parameter posterior or optimizing parameters with L2 regularization), we do not see these effects when such an interpretation does not exist (i.e. optimizing parameters with almost no L2 regularization). Our theoretical findings assume that layers are conditionally Gaussian, and different priors may have different effects. We note that much of the preliminary works on NTK assume no explicit regularization during training [28, 52, 57] (with the notable exception of Wei et al. [86]). and so our findings may be at odds with the empirical findings around these models [c.g. 9, 38, 39J. Moreover, recent work has proposed (non- nonn-Baysiian) infinite-wiee width constructions that avoid any limiting kernel behavior [e.g. 22, 43, 65, 90], and so findings would not apply to these models. We emphasize that our results do not conflict with these prior works, but rather reflect different perspective. The models we study correspond prior on parameters, and so relaxing this correspondence may lessen the consequences of width that observe. Nevertheless, our results suggest that the inductive bias of width may be harmful, even if these undesirable effects can be avoided via careful construction.
Gaussian
Finally, it is worth considering when one might still choose a conventional shallow GP over deep model. An often-touted benefit of Gaussian processes is the ability to encode prior domain knowledge via the choice of covariance function. In Appx. C, prove that certain prior covariances cannot be expressed by adaptable hicrarchical models. For example, a Deep GP that is composed of stationary GP layers cannot model anti-correlations priori (Thm. 4, Appx. C), whereas (single-layen stationary GP can have positive and negative prior covariances. Nevertheless, Deep GP are capable of modeling many common covariance functions, including the RBF, Matérn, and rational quadratic kernels. In Appx. C we demonstrate a 2-layer Deep GP construction of any width that is capable of producing prior covariances that match most isotropic kernels (Thm. 5, Appx. C). In other words, a Deep GP can match the first and second moments of most GP, while also offering an adaptable posterior.
Acknowledgments and Disclosure of Funding
We would like to thank Elliott Gordon-Rodrigue for his help with the proofs. This work was supported by the Simons Foundation, McKnight Foundation, the Grossman Center, and the Gatsby Charitable Trust.
10

---

References
[1] D. Agrawal, Papamarkou, and Hinkle. Wide neural networks with bottlenecks are deep Gaussian processes. Journal of Machine Learning Research, 21(175):1-66, 2020.
[2] L. Aitchison. Why bigger not always better: on finite and infinite neural networks. ICML, 2020.
[3] L. Aitchison, A. Yang, and S. W. Ober. Deep kernel processes. In JCML, 2021.
[4] Z. Allen-Zhu and Y. Li. What can ResNet learn efficiently, going beyond kerness? In NeurIPS, 2019.
[5] Z. Allen-Zhu and Y. Li. Backward feature correction: How deep learning performs deep learning. arXiv preprint 200: 2001:20000044133 04413, 2020.
[6] Z. Allen-Zhu, Y. Li, and Y. Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. In NeurlPS, 2019.
[7] S. Arora, S. Du, W. Hu, Z. Li, and R. Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In ICML,, 2019.
[8] S. Arora, S. S. Du, Hu, Z. Li, R. Salakhutdinov, and Wang. exact computation with an infinitely wide neural net. In NeurlPS, 2019.
[9] S. Arora, Du, Li, Salakhutdinov, R. Wang, and Yu. Hamessing the power of infinitely wide deep nets on small-data tasks. In ICLR, 2020.
[10] A. Asuncion and Newman. UCI machine learning repository, 2007. URL hwtp:h/ttthhhhhiieeeeee uci /edu/m/// index. php.
[11] Y. Bai and D. Lee. Beyond linearization: On quadratic and higher-order approximation of wide neural networks. ICIRR 2020.
[12] M. Belkin, D. Hsu, Ma, and Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 118632):1584--155844 2019.
[13] Y. Bengio, O. Delalleau, and N. Le Roux. The curse of dimensionality for local kernel machines. 2005.
[14] P. Billingsley. Convergence of probability measures. John Wiley & Sons, 2013.
[15] E. Bingham, P. Chen, M. Jankowiak, F. Obermeyer, N. Pradhan, Karaletsos, R. Singh, P. Szerlip,
P. Horsfall, and N. D. Goodman. Pyro: Deep universal probabilistic programming. The Journal of Machine Learning Research, 20(1):973 978, 2019.
[16] C. M. Bishop. Pattern recognition and machine learning. Springer, 2006.
[17] Blomqvist, Kaski, and Heinonen. Deep convolutional Gaussian processes. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2019.
[18] C. Blundell. J. Cornebise, Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In ICML, 2015.
[19] T. Bui, D. Hernández-Lobaed regression using approximate expectation propagation. In JCML, 2016.
Beernandez-Lobace Li, R. Turner. Deep Gaussian processes for
[20] Y. Cao and Q Generalization bounds of stochastic gradient descent wide and deep neural networks. In NeurlPS, 2020.
[21] M. Chen, Y. Bai, D. Lec, T. Zhao, H. Wang, Xiong, and R. Socher. Towards understanding hicrarchical learning: Benefits neural representations. In NewrlPS, 2020.
[22] I.. Chizat and F. Bach. On the glohal convergence of gradient descent for parr-paaaeeerrized models using optimal transport. NewrIPS, 2018
[23] Y. Cho and L. Saul. Kemel methods for deep learning. In NeurlPS, 2009.
[24] K. Cutajar, E. V. Bonilla, P. Michiardi, and M. Filippone. Random feature expansions for deep Gaussian processes. In JCML, 2017.
[25] Z. Dai, A. C. Damianou, J. González, and N. D. Lawrence. Variational auto unco-edccded deep Gaussian processes. In ICLR, 2016.

---

[26] A. Damianou. Deep Gaussian processes and variational propagation of uncertainty. PhD thesis, University of Sheffield, 2015.
[27] A. Damianou and N. Lawrence. Deep Gaussian processes. In AISTATS, 2013.
[28] S. Du, Lee, H. Li, L. Wang, and X. Zhai. Gradient descent finds global minima of deep neural networks. In ICML, 2019.
[29] M. M. Dunlop, M. A. Girolami. A. M. Stuart, and A. L. Teckentrup. How deep are deep Gaussian processes? Journal of Machine Learning Research, 19(54):1-46, 2018.
[30] V. Dutordoir, M. Wilk, A. Artemev, and J. Hensman. Bayesian image classification with deep convolutional Gaussian processes. In AISTATS, 2020.
[31] V. Dutordoir, J. Hensman, M. van der Wilk, C. H. Ek, Z. Ghahramani, and N. Durrande. Deep neural networks as point estimates for deep Gaussian processes. In NeurlPS, 2021.
[32] V. Dutordoir, Salimheni, E, Hambro, J. McL. cod, F. Leibfried, A. Artemev, M. van der Wilk, J. Hens- man, M. P. Deisenroth, and S. John. GPflux: A library for deep Gaussian processes. arXiv preprint vrXiv:2104.05674 2021.
[33] D. Duvenaud, O. Rippel, R. Adams, and Z. Ghahramani. Avoiding pathologies in very deep networks. In AISTATS, 2014.
[34] A. Y. Foong, D. R. Burt, Y. Li, and R. E. Turner. On the expressiveness of approximate inference in bayesian neural networks. In NewrIPS, 2020.
[35] S. Fort, G. K. Dziugaite, M. Paul, S. Kharaghani, D. M. Roy, and S. Ganguli. Deep learning versus kernel lcarning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kemnel. In NeurlPS, 2020.
[36] Y. Gal and Z. Ghahramani, Dropout as Bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
[37] J. R. Gardner, G. Pleiss, K. Q. Weinberger, D. Bindel, and A. G. Wilson. GPyTorch: Blackbox matrix- matrix Gaussian process inference with GPU acceleration. In NeurlPS, 2018.
[38] A. Garriga-Alonso, C. F. Rasmussen, and L. Aitchison. Deep convolutional networks as shallow Gaussian processes. In ICLR, 2019.
[39] M. Geiger, A. Jacot, S. Spigler, F. Gabriel, L. Sagun, S. d' Ascoli, G. Biroli, C. Hongler, and M. Wyart. Scaling description of generalization with number of parameters in deep learning. Journal of Statistical Mechanics:
Theory and Experiment, 2020(2), 2020.
[40] M. G. Genton. Classes of kereels for machine learning; statistics perspective. Journal of Machine Learning Research, (Dec)):29-3313 2001.
[41] B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari. Limitations of lazy training of two-layers neural networks. In NeurlPS, 2019.
[42] B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari. When do neural networks outperform kernel methods? In NeurlPS, 2020.
[43] E. Golikov. Towards general thcory of indinite-width limits of neural classifiers. In ICML, 2020.
[44]
Goodfellow, Bengio, A. Courville, and Bengio. Deep learning. MIT press Cambridge, 2016.
[45] J. Halverson, A. Maiti, and K. Stoner. Neural networks and quantum field theory. Machine Learning: Science and Technology, 2(3), 2021.
[46] M. Havasi, J. M. Hernández-Lobenr and J. J. Murillo-Fuentee Inference in deep Gaussian processes using stochastic gradient Hamiltonian Monte Carlo. In NeurlPS. 2018
[47] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[48] M. D. Hoffman and A. Gelman. The no-u-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(3):1593-16 2014.
[49] J. Hron, Y. Bahri, R. Novak, J. Pennington, and J. Sohl-Dickstedi Exact posterior distributions of wide bayesian neural networks. In JCML Workshop Uncertainty and Robustness in Deep Learning, 2020.
12

---

[50] J. Hron, Y. Bahri, J. Sohl-Dickstein, and R. Novak. Infinite attention: NNGP and NTK for deep attention networks. In ICML, 2020.
[51] P. Izmailov, S. Vikram, M. D. Hoffiman, and A. G. Wilson. What are bayesian neural network posteriors really like? In ICML, 2021.
[52] A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kemel: Convergence and generalization in neural networks. In Neur/PS, 2018.
[53] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[54] A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[55] Y. LeCun, Botto,, Y. Bengio, Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86( (66(11):22722332 1998.
[56] Lee, Bahri, Novak, Schoenholz, Pennington, and Sohl-DDckstein. Deep neural networks as Gaussian processes. In ICLR, 2018.
[57] Lee, Xiao, S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In NewrlPS, 2019.
[58] Lee, Schoenholz. Pennington, B. Adlam, L. Xiao, Novak, and Sohl-Dickstein. Finite versus infinite neural networks: an empirical study, In NeurlPS, 2020.
[59] H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein. Visualizing the loss landscape of neural nets, In NewrIPS, 2018.
[60] Y. Li, T. Ma, and H. R. Zhang. Learning over per-parametrized two-layer neural networks beyond NTK. In COLT, 2020.
[61] C. Louizos and M. Welling. Structured and efficient variational deep learning with matrix Gaussian posteriors. In ICML, 2016.
[62] C.-K. Lu, C.-H. Yang, Hao, and P. Shafto. Interpretable deep Gaussian processes with moments, In AISTATS, 2020.
[63] Z. Lu, H. Pu, F. Wang, Z. Hu, and Wang. The expressive power of neural networks: A view from the width. In NeurlPS, 2017.
[64] A. d. G. Matthews, M. Rowland, J. Hron, R. Turner, and Z. Ghahramani. Gaussian process behaviour in wide deep neural networks. In ICLR, 2018.
[65] S. Mei, A. Montanari, and P.-M. Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7665 E7671, 2018.
[66] C. A. Micchelli, Xu, and H. Zhang. Universal kernels. Journal of Machine Learning Research, 7(12). 2006.
[67] G. Montúfar, Pascanu, Cho, and Y. Bengio. the number of linear regions of deep neural networks. In NeurlPS, 2014.
[68] P. Nakkiran, G. Kaplun, Y. Bansal, Yang, Barak, and Sutskever. Deep double descent: Where bigger models and more data hurt. In ICLR, 2020.
[69] R. M. Neal. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995.
[70] Q Nguyen and M. Hein. The loss surface of deep and wide neural networks. In ICML, 2017.
[71] R. Novak, L. Xiao, J. Lee. Y. Bahri, G. Yang. J. Hron. D. A. Abolafia, J. Pennington, and J. Sohl- -iccksisttnin. Bayesian deep convolutional networks with many channels are Gaussian processes. In ICLR, 2019.
[72] S. W. Ober and L. Aitchison. Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes. In /CML, 2021.
[73] A. Paszke, S. Gross, F. Massa, A. Lerer. J. Bradbury, G. Chanan. T. Killeen, Z. Lin, N. Gimelshein. L. Antiga, et al. PyTorch: An imperative style, figh-performanc deep learning library. In NeurlPS, 2019.
[74] B. Poole. S. Lahiri, M. Raghu, J. Schl-Dickstein and S. Ganguli. Exponential expressivity in deep neural networks through transient chaos. In NeurlPS, 2016.
13

---

[75] M. Raghu, B. Poole. J. Kleinberg, S. Ganguli, and J. Sohl-Dicksteii On the expressive power of deep neural networks. In ICML, 2017.
[76] A. Rahimi, B. Recht, et al. Random features for large-scale kernel machines. In NeurlPS, 2007.
[77] C. E. Rasmussen and C. Williams, Gaussian processes for machine learning, volume 1. MIT Press, 2006.
[78] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. Fitnets: Hints for thin deep nets. ICLR, 2015.
[79] H. Salimbeni and M. Deisenroth. Doubly stochastic variational inference for deep Gaussian processes. In NeurIPS, 2017.
[80] I. Schoenberg. Metric spaces and completely monotone functions. Annals of Mathematics, pages 811 8411 1938.
[81] V. Shankar, A. Fang, W. Guo, S. Fridovich-Keill J. Ragan-Kelley, L. Schmidt, and B. Recht. Neural kernels without tangents. JCML, 2020.
[82] Soltanolkotabi. A. Javanmard, and J. D. Lee. Theoretical insights into optimization landscape wver-parameterized shallow neural networks. IEEE Transactions on Information Theory, 65(2):742-769, 2018.
[83] M. Telgarsky. Benefits of depth in neural networks. In COLT, 2016.
[84] M. Vladimirova, J. Verbeek, P. Mesejo, and J. Arbel. Understanding priors in Bayesian neural networks at the unit level. In ICMI,, 2019.
[85] Y. Wang. M. Brubaker, B. Chaib-Draa, and R. Urtasun. Sequential inference for deep Gaussian process. In AISTATS, 2016.
[86] C. Wei, J. Lee, Q. Liu, and T. Ma. Regularization matters: Generalization and optimization of neural nets vs their induced kernel. In NeurlPS, 2019.
[87] A. Yaglom. Correlation theory of stationary and related random functions. Springer Series in Statistics, New York, 1987.
[88] G. Yang. Tensor programs I: Wide feedforward recurrent neural networks of any architecture gaussian processes. NeurIPS, 2019.
[89] G. Yang. Tensor programs II: Neural tangent kemee for any architecture, arXiv preprint arXiv:2006.1544 2020.
[90] G. Yang and J. Hu. Tensor programs IV: Feature learning in infinite-width neural networks. In ICML, 2021.
[91] G. Yehudai and Shamir. On power limitations of random features for understanding neural networks. In NeurlPS, 2019.
[92] S. Zagoruyko and Komodakis. Wide residual networks. In BMVC, 2016.
[93] A. Zavatone-Vän Veth and C. Pehlevan. Exact priors finite neural networks. NeurIPS, 2021.
Checklist
1) For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 7.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appx.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2) If you including theoretical results...
14

---

(a) Did you state the full set of assumptions of all theoretical results? [Yes] Assumptions are listed in the theorem/lemma statements, well in Appx. E.3. (b) Did you include complete proofs of all theoretical results? [Yes] See Appx. C, E and
3) If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main exper- imental results (either the supplemental material or as a URL)? [Yes] Code is in supplementary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appx. H.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] Our results for optimized networks are run 3-5 times, and indicate as such in the results figures. Due compute resource constraints we only have one HMC run for each experiment in Sec. 6.1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appx. H.
4) If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] In Secs. 6.1 and 6.2 we supply citations for all datasets. In Appx. H we supply citations for all software
packages.
(b) Did you mention the license of the assets? [No] To the best of knowledge, none of the datasets we use have licenses. We mention that all software packages we use are open source.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] We use commonly-availabe public datasets that, to the best of our knowledge, do not pose any privacy or content concerns.
5) If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "sL1PozogBAg", "reviewer": "Reviewer_dRMh", "summary": "     This paper provides a narrative of composite Gaussian process models. In specific the paper provides insights into the interplay between depth and width in the context of capacity and representative power. The paper further draws conclusions on the implications that this has for composite finite basis function models such as neural networks.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a very nice paper that I thoroughly enjoyed reading. The paper really tries to tell a story through related work and provides easy to follow mathematical derivations of the main concepts with plenty of intuition.\n\n     My main question regarding the generalisation of the results from Deep GPs to neural networks. While it is true that Deep GPs subsumes neural networks and as is shown in [1] there exists an equivalence class in parametrisation between a sparse Deep GP and a neural network. However, the main difference comes in the difference between the marginal likelihood and the objective function used to train the neural network. It would be great if the authors could comment on this and what the implications of this is in interpreting the results in the paper for neural networks.\n     \n     While this paper wants to study the model in isolation from the inference mechanism I think the results implies something quite interesting with respect to sparse DGPs, the approximate model is actually more useful compared to the model it aims to approximate. The same type of behaviour can be seen with a VAE. I think this would be an interesting idea for the authors to discuss and what this actually means for the conclusions we should draw from inference and how we should design approximate inference schemes for Deep GPs.\n     \n     In summary I think this is a lovely written paper providing a nice narrative to many concepts that people have had an intuitive understanding of with respect to Deep GPs. Making these mathematically concrete and providing a narrative grounded in related work is important to move these methods forward with understanding beyond empirical results. The authors have done a good job of separating the material in the main paper from the detailed explanations in the appendix. For me it was crucial to have the verbose derivations in Appendix F,G to solidify my understanding from the paper. I would be happy to increase my score, however I feel (as is reflected by my confidence score) that I am less well versed in the related work that this paper builds upon so there might be things that I have missed.\n\n     Even though this paper has *a lot* of references already, one paper that the authors might like is [2]. There are quite a few connections between these two works that you might find interesting for future work.\n     \n     1) Dutordoir, V., Hensman, J., Wilk, M. v. d., Ek, C. H., Ghahramani, Z., & Durrande, N., Deep neural networks as point estimates for deep gaussian processes, CoRR, (),  (2021). \n     2) Halverson, J., Maiti, A., & Stoner, K., Neural networks and quantum field theory, Machine Learning: Science and Technology, 2(3), 035002 (2021).  http://dx.doi.org/10.1088/2632-2153/abeca3", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "LuYgk-fnH7A", "reviewer": "Reviewer_PVcF", "summary": "The paper argues that deep Gaussian Processes with large widths and more than 2 layers have undesirable properties. In particular, the paper shows that firstly, deep GP prior in the GP limit is essentially shallow, and secondly, deep GP posterior then loses data adaptivity in the hidden layers. The paper further examines the role of depth and argues that the deep GP prior, if not equal to a GP, has heavy-tailed behaviors.\n\n==================================\nPost-rebuttal:\n\nThanks for the reply and clarification!\n\nWhile I agree that very large width hurts in the context of deep GP as shown in the paper, I also agree with other reviewers that the message that very large width hurts generally is overly strong. The literature has a mixed bag of results concerning widths -- theoretically or empirically, so a general statement should be validated very carefully.\n\nI'd like to keep my score.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The subject of the paper is timely, and the conclusions are interesting. In a way, it is not surprising that the deep GP prior becomes shallow in the GP limit; specifically Lemma 1 is expectable since intuitively a mixture of zero-mean Gaussians, if equal to a Gaussian, should comprise of same Gaussian components. (To be fair, Lemma 2 is more general than this.) What is interesting in the paper is the interpretation that in the hidden layers, a certain hierarchical structure is lost and hence so is data adaptivity in the posterior. This message is a strong and noteworthy one.\n\nYet this important message about the posterior is where I do not find it satisfying yet. In particular, the argument around (6) potentially ignores some important limit exchange issue, technically speaking. The conceptual problem I have in mind is the following. It is true that in the prior the kernel converges pointwise to a degenerate limit as the width increases. However this doesn’t exclude the possibility that in the process of computing the posterior (or in general, a process that involves fitting the training data), the movement of the weight of the first layer explodes with large width. In such scenario, it is unclear if in the posterior, the kernel can still be close to the degenerate limit (in an appropriate sense such that (6) holds).\n\nAnother issue — of less importance in my personal opinion — is that of the experimental results in Section 6.2. This section aims to argue against large widths in contexts beyond GP. However the way I read the results in Figure 4, they say that the performance does not necessarily increase monotonically with width. I feel even more mixed when contrasting this figure with other papers, such as [51]. The notion of “sufficient capacity” is not defined, nor is it illustrated numerically. I have a similar concern regarding Table 1, where the message that deeper is better is not clear.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "7E2fPJYDaRZ", "reviewer": "Reviewer_h2Yp", "summary": "The authors show that in the infinite width limit, deep GPs becomes a single-layer GP, generalizing the result that deep neural networks at initialization become a GP in the infinite width limit. The authors show that increasing depth of the GP makes the distribution of the final layer of deep GPs deviate from being Gaussian at finite width. The authors claim that this simplification implies that large widths are detrimental to neural network performance beyond some point.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\nThe authors study the effect of the width and depth of deep GPs (a generalization of deep networks at initialization) and discuss the implications of the depth and width on the distribution of the output layer conditioned on the input features. This is an interesting generalization of the results that deep neural networks in the infinite limit become GPs at initialization.\n\n### Quality\nThe quality of the theoretical analysis of the work is good. The experiments conducted on neural networks, and the authors' assertion that larger widths hurts neural network performance (because they make networks into GPs) has the following flaws:\n\n1-0. In order to conduct a fair comparison of the performance of two neural networks with different width, one needs to conduct hyper-parameter search of basic parameters. At a minimum, it is standard to at least conduct a hyper-parameter search of the learning rate, which the authors do not do. This brings into question the validity of figure 4, because, for example, the performance of a width 5000 network might be worse than the performance of a width 500 network, because the fixed learning rate the authors used is more optimal for the width 500 network.\n\n1-1.  For example, it is very possible that the drop-off of performance with respect to the width of the network for MNIST shown in figure 4 can be related to stability issues, as the maximum stable learning rate for wider networks tend to be lower. We can't know for sure, since the authors have not conducted a learning rate sweep for each width.\n\n2-0. Besides the fact that the experiments conducted to support the author's assertion that \"very large\" widths hurt neural network performance has not been carefully conducted, even if one accepts the assertion that there exists an optimal width for performance, the reason the authors provide for it is not consistent with the existing literature.\n\n2-1. If the authors' diagnosis for width hurting performance, i.e., that the neural network collapses to a GP in that limit, is correct, it must be the case that the performance of a neural-network GP (NNGP) cannot exceed that of a neural network for the MNIST tasks the authors explored.\n\n2-2. This can be refuted by the experiments presented in work the authors cite, namely Lee et al. [49]. In table 1 and table 2 of their paper, the authors present the result of numerous experiments they conduct where they demonstrate examples where the NNGP performance exceeds that of a trained neural network. For example, the authors report an NNGP (obtained as the infinite width limit of a 3-layer network) that achieves 96.93% test accuracy on 5k MNIST, better than what the authors report in figure 4.\n\n### Clarity\nThe theoretical analysis itself has been presented in a clear and consistent way. The connection between the theoretical results and the implication it has to the performance of neural networks has not been clearly presented, if presented at all. The authors try to imply that the fact that the infinite limit reduces the neural network at initialization to a GP (a well-known fact) is the reason that very wide networks should eventually under-perform networks with \"optimal-width,\" but there is hardly an argument being made. In fact, Lee et al. [49] has systematically explored this problem with extensive experiments and have showed that the GP limit often out-performs trained neural networks, disproving the authors' claim.\n\n### Significance\nThe paper contains some interesting analysis on deep GPs, but their claims connecting their theoretical results to the performance of actual neural networks are not consistent with experimental results in the literature.\n\n---\n\n### Summary of Author Correspondence\n\nAfter discussion with the authors, the authors have agreed to make the following changes to their paper:\n1. They have run additional experiments to show how widening the width of trained-NNs affect the NN performance, and agreed to provide the training parameters used.\n2. They have agreed to clarify that their theoretical and experimental results on deep GPs cannot be directly be applied to trained-NNs, but is suggestive in nature.\n\nMy initial rating of the paper for a clear rejection (3) was based on a (perhaps misunderstood) assessment on my part that the paper is confusing the reader by implying that the Gaussianity of wide DGP marginals is what should be attributed to the performance loss of trained-NNs at very large widths, the evidence of which I found to be unconvincing. Given that the authors have agreed to clear up such confusion, I believe my initial rating to be too harsh now. At the same time, it has been clarified that the most interesting part of the paper to me, the consequence of the authors' work on trained-NNs, is indeed of a speculative nature. I thus have adjusted my rating to a marginal reject (5).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not have useful comments in this aspect.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "3aSVDjN6_Kc", "reviewer": "Reviewer_zKHo", "summary": "The paper shows that in the limit of infinite width a Deep Gaussian Process (DGP) converges to an ordinary Gaussian Process (GP). Indeed, in the limit of infinite width the covariance function of each hidden GP layer becomes deterministic (as a consequence of the strong law of large numbers). This allows proving that the marginal prior covariance  of the DGP becomes gaussian (Theorem1). The  consequence of this theorem is that as the width increases DGP models lose their ability to adapt the hidden covariance functions to the input data (section 4).\nThis theoretical result is supported by numerical experiments (section 6.1).\nThe paper also addresses the convergence of a DGP to a GP with increasing depth and width (section 5).\nSince it is known that deep neural networks (DNN) approach a GP as they are made wider and wider,  Theorem 1 is used to  claim that a very large width can be detrimental to the expressivity of a DNN (and not only a DGP). \n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is  well written and the proofs of the theorems/lemmas are, to my understanding, correct. The connections with the relevant literature are also discussed, with an important exception (see below). \n\nI have a concern related to the meaningfulness of the extension of the results to standard neural networks.  The test accuracy reported in Figure 4, rightmost panel, is significantly lower than the accuracy obtained with non-Bayesian approaches on CIFAR10. (see for example https://paperswithcode.com/sota/image-classification-on-cifar-10?tag_filter=3) This is for sure partially due to the lack of data augmentation (which is acceptable in a numerical experiment) but, in my opinion, also to the training protocol. If I understood correctly, the same weight decay is applied for every model. What would happen if one would tune this decay optimally? Maybe, as stated by the authors “.. by changing the weight decay parameter, may lessen the consequences of width that we observe” (line 322-323). Indeed, setting the weight decay at initialization amounts simply  to set the inverse variance of the gaussian prior on the network weights. It is well known empirically that smaller models require larger weight decay than wider models. It seems thus natural to choose a gaussian prior with small variance i.e. more restrictive for narrower networks than for wider ones. \n\nThe fact that the author(s) have not optimized their training protocol might partially explain the (possible) discrepancy with other works: see for example  https://arxiv.org/abs/1901.01608 (which should be cited and discussed), ref 49, where it is explicitly shown that width improves accuracy *in practice*.\n\nIndeed, the quality of an architecture is not only in its theoretical expressivity, but, even more, in its “trainability”. Even a large  width perceptron is, as has been well known since a long time, extremely “expressive”, but it cannot be trained.  The importance of trainability is in my opinion overlooked in an otherwise interesting and relevant paper.\n\nIn short, I think that either the detrimental effect of width is demonstrated using state-of-the-art training protocols, or the strong claims on the importance of Theorem 1 for standard DNNs should be weakened. \n\n The code used for the experiments should be publicly released for the reproducibility of the results\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In my opinion the limitations are not sufficiently discussed, in particular concerning  the connection of the results to the theory of DNNs. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper leverages an analysis of deep Gaussian processes to argue that an excess increase in the width of a neural network can degrade performance. The question of whether or not neural networks benefit from increased width is an important and unresolved question in the literature, and the reviewers and I agree that this paper provides an additional and important perspective on the topic that will be of interest to the NeurIPS community. While some reviewers found the experimental results unconvincing and the conclusions somewhat speculative, others found the framework to be illuminating and to provide new perspectives on this important problem. Overall, I do not expect this or any paper to fully and unambiguously resolve all questions relating to the benefits/drawbacks of large width networks, and I believe this paper provides novel and useful insights that shed light on this important problem. Therefore, I recommend acceptance.
Decision: Accept (Poster)