The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective

1 Introduction . Research has shown that deeper neural networks tend to be more expressive and efficient than wider networks under a variety of metrics [ e.g . 21 , 63 , 67 , 70 , 74 , 75 , 78 , 83 ] . Nevertheless , there is resurgent interest in wide models due in part to empirical successes [ e.g . 92 ] and theoretical analyses of limiting behavior . When randomly initialized to create a distribution over functions , neural networks converge to Gaussian processes ( GP ) as width increases . This result , first proved for 2-layer networks [ 69 ] , has been extended to deeper networks [ 56 , 64 ] , convolutional networks [ 38 , 71 ] , and other architectures [ 50 , 88 ] . A similar limit exists for gradient-trained networks , which behave increasingly like kernel machines under the neural tangent kernel [ e.g . 6 , 8 , 28 , 39 , 52 , 57 , 89 ] . While these limits simplify analyses , there is something unsettling about reducing neural networks to kernel methods . Neal [ 69 , p. 161 ] describes the GP limit as “ disappointing , ” noting that “ infinite networks do not have hidden units that represent ‘ hidden features ’ . . . often seen [ as the ] interesting aspect of neural network learning. ” Recent work indeed shows that learned hierarchical features can be exponentially more efficient than the fixed shallow representations of kernels [ e.g . 4 , 5 , 8 , 11 , 13 , 21 , 41 , 42 , 60 , 91 ] . At the same time , wider networks can more accurately model complex functions 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . [ 44 ] . Thus , wide limits appear to confound opposing phenomenon : increased capacity makes them more expressive , yet the loss of hierarchical features seems to make them less expressive . This may explain the mixed empirical performance of limiting models : outperforming finite width models in some scenarios [ e.g . 9 , 38 , 39 , 58 ] , yet falling short on more complex tasks [ e.g . 8 , 11 , 35 , 57 , 81 ] . This paper aims to decouple these effects of large width . Our goal is to understand the inductive biases of wide networks , after a network has “ sufficient ” capacity for a given modeling task . We ask : If we control for the effects of increased capacity , what—if any—value remains in wide networks ? To achieve this control , we note that a typical neural network layer corresponds to a finite basis , where elementwise nonlinearities transform each hidden feature into a single basis function . In order to decouple width from capacity , one could generalize these layers so that each nonlinearity produces any number of basis functions ; if each hidden feature gives rise to an infinite and universal basis , then hidden layers would have infinite representational capacity regardless of width . This generalization is in fact a well-studied class of hierarchical models—Deep Gaussian Processes ( Deep GP ) [ 19 , 24 , 26 , 27 , 30 , 32 , 46 , 79 ] —where standard neural net layers are replaced with vector-valued Gaussian processes . Indeed , typical neural networks are a degenerate Deep GP subclass [ 1 , 2 , 33 , 72 ] . We therefore have a generalization of neural networks where capacity is controlled , from which we can glean insights about conventional networks that have sufficient representational power for a given modeling task . Surprisingly , despite using Gaussian processes as the primary hierarchical component , we prove that Deep GP converge to ( single-layer ) GP in their infinite width limit ( Thm . 1 ) . Troubling implications immediately ensue : large width is strictly detrimental to Deep GP , as the limiting model collapses to a shallower version of itself . We support this theorem with an analysis of neural network and Deep GP posteriors , which become less adaptable as width increases . Specifically , we show that the posterior mean corresponds to a mixture of functions drawn from data-dependent ( and thus adaptive ) reproducing kernel Hilbert spaces , formalizing the above claim from Neal [ 69 ] . As width increases , this mixture collapses to the data-independent kernel of the limiting GP , implying that wider models have less feature learning . Finally , we present a novel tail analysis which indicates that width and depth have opposite effects : depth accentuates non-Gaussianity , sharpening peaks and fattening tails , whereas width increases Gaussianity ( Thms . 2 and 3 ) . Our theoretical results hold for Deep GP and conventional ( parametric ) neural networks alike . Experiments confirm that—after a model achieves sufficient capacity1—width can become harmful to model fit and performance . For nonparametric Deep GP , a width of 1 or 2 often achieves the best performance . Neural networks—because of their parametric nature—naturally require more hidden units before achieving optimal accuracy . Nevertheless , for Bayesian neural networks and conventional ( optimized ) neural networks trained with L2 regularization , performance degrades after a certain width . On small datasets ( N ≤ 1000 ) with low dimensionality , we find that models with ≤ 16 hidden units achieve best test set performance . On larger datasets like CIFAR10 , this “ sweet spot ” occurs later ( at ≈ 500 hidden units for sufficiently deep models ) , yet performance degrades beyond this width . We note that these trends do not necessarily hold for models that do not have a probabilistic interpretation—i.e . optimized neural networks trained without ( or nearly without ) L2 regularization . Nevertheless , our findings suggest that narrower models have better inductive biases , and wide models perform well in spite of —not because of—large width . 2 Setup . 2.1 Related Work . Effects of width . Works have shown that , given finite parameters , deeper models are more expressive than wider models [ 63 , 67 , 74 , 75 , 83 ] . Similarly to our work , Aitchison [ 2 ] recognises the link between finite neural networks and Deep GP , and argues that finite neural networks have flexibility in the top-layer representation that is absent in the infinite-width limit . Halverson et al . [ 45 ] draw a connection to quantum field theory to argue that neural networks become “ simpler ” near their infinite-width limit . In the non-probabilistic setting , it is worth noting that wide models have been shown to have favorable optimization landscapes [ 7 , 28 , 59 , 70 , 82 ] and are resistant to overfitting via double descent [ 12 , 20 , 68 ] . Our work controls for these factors by examining nonparametric hierarchical models with exact Bayesian inference , and thus does not disagree with these other works . 1We offer a formal notion of “ sufficient capacity ” in Appx . B.5 . Infinite width limits have received renewed interest in Bayesian [ 38 , 50 , 57 , 64 , 69 , 71 , 88 ] and non-Bayesian [ 6 , 8 , 22 , 28 , 43 , 52 , 57 , 65 , 89 , 90 ] settings . Most of these works show that neural networks converge to kernel methods , though recent work suggests that this limiting behavior can be avoided with different parameterizations [ e.g . 22 , 43 , 65 , 90 ] . Similarly to Lee et al . [ 57 ] , our Deep GP limit analysis sequentially increases the width of each layer , though we hypothesize a similar proof exists where the width of all layers increases simultaneously ( akin to [ 64 ] ) . Deep GP are introduced by Damianou and Lawrence [ 27 ] . A large portion of Deep GP research has thus far focused on scalable approximate inference methods [ 19 , 24 , 25 , 32 , 46 , 72 , 79 , 85 ] . Though prior work has studied tail properties of neural networks [ 84 , 93 ] and Deep GP with RBF kernels [ 62 ] , our work is—to the best of our knowlege—the first general result for Deep GP tails . Duvenaud et al . [ 33 ] and Dunlop et al . [ 29 ] investigate pathological behaviors that arise with depth , while Agrawal et al . [ 1 ] note that “ bottlenecked ” Deep GP have better performance and correlations among predictive tasks . Our work complements these analysis by characterizing the effects of width . Connections between Deep GP and neural networks . Many researchers have noted connections between neural networks and Deep GP [ e.g . 24 , 31 , 36 , 61 ] . Duvenaud et al . [ 33 ] suggest that infinitely-wide neural networks with intermediate bottleneck layers are nonparametric Deep GP . Agrawal et al . [ 1 ] formalize this connection , but note that not all Deep GP can be constructed from bottlenecked neural networks ( see Appx . E ) . In contrast to these prior works , we avoid reducing Deep GP to neural networks , and instead reduce neural networks to degenerate Deep GP . 2.2 A Covariance Perspective on Gaussian Process Limiting Behavior . To decouple the effects of increasing width and capacity , we first prove a new result about GP limits for a more general class of models , including Deep GP as well as typical neural networks . This result forms a necessary foundation for the subsequent theorems that are a main contribution of this work . To begin , note that the proof technique introduced by Neal [ 69 ] and extended by others [ 38 , 50 , 56 , 64 , 71 , 88 ] relies on the multivariate central limit theorem , which requires a model with additive structure . Deep GP do not generally decompose in an additive manner , so we establish a more general proof technique . For simplicity , we first present it in the context of neural networks , and then extend it to a more general class of models . Consider the 2-layer neural network f2 ( f1 ( x ) ) , with f1 : RD → RH1 and f2 : RH1 → R : f1 ( · ) = W > 1 ( · ) + βb1 , f2 ( · ) = 1√H1w > 2 σ ( · ) + βb2 . ( 1 ) σ ( · ) is an elementwise nonlinearity , β is a positive constant , and W1 , b1 , w2 , and b2 are i.i.d . Normal . With randomly initialized parameters , f2 ( f1 ( · ) ) : RD → R is a prior distribution over functions , and this distribution converges to a GP in the infinite width limit [ 69 ] . Lemma 1 . The neural network defined in Eq . ( 1 ) is a Gaussian process if and only if—for any finite set of inputs X = [ x1 , . . . , xN ] —the conditional prior covariance Ef2|X , W1 , b1 [ f2f > 2 ] is almost surely equal to the marginal prior covariance Ef2|X [ f2f > 2 ] , where f2 | X , [ f2 ( f1 ( x1 ) ) , . . . , f2 ( f1 ( xN ) ) ] . Proof . By definition , f2 ( f1 ( · ) ) is a GP if and only if f2 | X is multivariate Gaussian for any X . From Eq . ( 1 ) , we have p ( f2 | X , W1 , b1 ) = N ( 0 , KW1 , b1 ( X , X ) ) , where [ KW1 , b1 ( X , X ) ] ij = β2+ 1H1σ ( W > 1 xi+βb1 ) > σ ( W > 1 xj +βb1 ) is the appropriate kernel Gram matrix . Using Jensen ’ s inequality , we have a lower bound on the characteristic function of f2 | X : E f2|X [ exp ( it > f2 ) ] = E W1 , b1 [ E f2|X , W1 , b1 [ exp ( it > f2 ) ] ] ( law of total expectation ) = EW1 , b1 [ exp ( − 12t > KW1 , b1 ( X , X ) t ) ] ( char . func . of a Gaussian ) ≥ exp ( − 12t > E W1 , b1 [ KW1 , b1 ( X , X ) ] t ) . ( convexity of exp ) This lower bound happens to be the characteristic function ofN ( 0 , EW1 , b1 [ KW1 , b1 ( X , X ) ] ) . Since exp is strictly convex , the characteristic function of f2 | X equals the Gaussian lower bound ∀t if and only if p ( KW1 , b1 ( X , X ) |W1 , b1 ) = Ef |X , W1 , b1 [ f2f > 2 ] is a constant with probability 1 . Seeing that 1H1σ ( W > 1 xi + βb1 ) > σ ( W > 1 xj + βb1 ) becomes a.s. constant as H1 →∞ , Lemma 1 re-establishes the result of Neal [ 69 ] ( see Appx . E.1 ) . Critically , unlike Neal ’ s proof , Lemma 1 neither relies on the central limit theorem nor requires f2 ( f1 ( · ) ) to be a neural network ; it holds if p ( f2 | f1 ( x1 ) , . . . , f1 ( xN ) ) is Gaussian . Therefore , we can generalize it to a larger class of models : Lemma 2 . Let f2 ( f1 ( · ) ) : RD → R be a hierarchical model where f2 ( · ) : RH1 → R is a GP and f1 ( · ) : RD → RH1 is a random vector-valued function ( including a multilayer hierarchical model ) . Then f2 ( f1 ( · ) ) is a GP if and only if Ef2|X , f1 ( · ) [ f2f > 2 ] = Ef2|X [ f2f > 2 ] a.s. for all X = [ x1 , . . . , xN ] . The covariance perspective from Lemmas 1 and 2 is revealing about GP limits . As Ef2|X , f1 ( · ) [ f2f > 2 ] converges to Ef2|X [ f2f > 2 ] the model output becomes less and less dependent on f1 ( · ) . In other words , f2 ( f1 ( · ) ) loses its hierarchical nature . We reiterate that Lemma 2 has no requirements about f2 ( f1 ( · ) ) transitioning from a finite to infinite basis , nor does it require f2 ( f1 ( · ) ) to have additive structure . We demonstrate its generality in the next section with surprising—and troubling—implications .

Summary: The paper argues that deep Gaussian Processes with large widths and more than 2 layers have undesirable properties. In particular, the paper shows that firstly, deep GP prior in the GP limit is essentially shallow, and secondly, deep GP posterior then loses data adaptivity in the hidden layers. The paper further examines the role of depth and argues that the deep GP prior, if not equal to a GP, has heavy-tailed behaviors.  ================================== Post-rebuttal:  Thanks for the reply and clarification!  While I agree that very large width hurts in the context of deep GP as shown in the paper, I also agree with other reviewers that the message that very large width hurts generally is overly strong. The literature has a mixed bag of results concerning widths -- theoretically or empirically, so a general statement should be validated very carefully.  I'd like to keep my score.