Rethinking Parameter Counting: Effective Dimensionality Revisited

1 INTRODUCTION . Parameter counting pervades the narrative in modern deep learning . “ One of the defining properties of deep learning is that models are chosen to have many more parameters than available training data . In light of this capacity for overfitting , it is remarkable that simple algorithms like SGD reliably return solutions with low test error ” ( Dziugaite and Roy , 2017 ) . “ Despite their massive size , successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance ” ( Zhang et al. , 2017 ) . “ Increasing the number of parameters of neural networks can give much better prediction accuracy ” ( Shazeer et al. , 2017 ) . “ Scale sensitive complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization ” ( Neyshabur et al. , 2018 ) . “ We train GPT-3 , an autoregressive language model with 175 billion parameters , 10× more than any previous non-sparse language model ” ( Brown et al. , 2020 ) . The number of model parameters explicitly appears in many modern generalization measures , such as in Equations 20 , 51 , 52 , 56 , 57 , 59 , and 60 of the recent study by Jiang et al . ( 2020 ) . Phenomena such as double descent are a consequence of parameter counting . Parameter counting even permeates our language , with expressions such as over-parametrization for more parameters than data points . But parameter counting can be a poor description of model complexity , model flexibility , and inductive biases . One can easily construct degenerate cases , such as predictions being generated by a sum of parameters , where the number of parameters is divorced from the statistical properties of the model . When reasoning about generalization , over-parametrization is besides the point : what matters is how the parameters combine with the functional form of the model . Indeed , the practical success of convolutional neural networks ( CNNs ) for image recognition tasks is almost entirely about the inductive biases of convolutional filters , depth , and sparsity , for extracting local similarities and hierarchical representations , rather than flexibility ( LeCun et al. , 1989 ; Szegedy et al. , 2015 ) . Convolutional neural networks have far fewer parameters than fully connected networks , yet can provide much better generalization . Moreover , width can provide flexibility , but it is depth that has made neural networks distinctive in their generalization abilities . In this paper , we gain a number of insights into modern neural networks through the lens of effective dimensionality , in place of simple parameter counting . Effective dimensionality , defined by the eigenspectrum of the Hessian on the training loss ( equation 2 , Section 2 ) , was used by MacKay ( 1992a ) to measure how many directions in the parameter space had been determined in a Bayesian neural network . There is immense value in revisiting effective dimensionality in the context of modern deep learning . We demonstrate that effective dimensionality can be used to explain phenomena such as double descent and width-depth trade-offs in architecture specification ( Section 5 ) . We also show that effective dimensionality provides a straightforward , scalable , and promising metric for generalization in modern deep learning , comparing to PAC-Bayes flatness and path-norm measures , two of the most successful measures in the recent study by Jiang et al . ( 2020 ) ( Section 6 ) . We additionally show how effective dimension can explain why subspace compression methods for neural networks ( Izmailov et al. , 2019 ; Li et al. , 2018 ) can be so effective in practice , demonstrating function-space homogeneity as we move in directions given by eigenvectors corresponding to the smallest eigenvalues of the Hessian ( Section 4 ) . We connect this finding with Bayesian Occam factors and minimum description length frameworks , providing an interpretation of effective dimensionality as model compression ( Section 4.3 ) . Moreover , we show that despite a seeming lack of determination in parameter space , a neural network can be relatively well-determined in function space ( Section 4 ) . Consider Figure 1 , where we see that once a model has achieved low training loss , the effective dimensionality , computed from training data alone , replicates double descent behaviour for neural networks . Models that are wider have both lower effective dimensionality and better generalization . Alternatively , in Figure 2 we see that width and depth determine effective dimensionality in different ways , though both are related to numbers of parameters . Remarkably , for models with low training loss ( above the green partition ) , the effective dimensionality closely tracks generalization performance for each combination of width and depth . We also see that wide but shallow models overfit , while depth helps provide lower effective dimensionality . When two models have the same training loss they can be viewed as providing a compression of the training data at the same fidelity , in which case the model with the lower effective dimensionality , and thus provides the better compression ( Section 4.3 ) — capturing more regularities — will tend to generalize better . In particular , effective dimension should be used to compare models with similarly low values of training loss . In this regime , we see that ED closely tracks generalization for both double descent and width-depth trade-offs . 2 POSTERIOR CONTRACTION , EFFECTIVE DIMENSION , AND THE HESSIAN . We consider a model , typically a neural network , f ( x ; θ ) , with inputs x and parameters θ ∈ Rk . We define the Hessian as the k × k matrix of second derivatives of the loss , Hθ = −∇∇θL ( θ , D ) , where D is the training data . To begin , we describe posterior contraction , effective dimensionality , and connections to the Hessian . 2.1 THE HESSIAN AND THE POSTERIOR DISTRIBUTION We begin by providing a simple example explaining the relationship between the posterior distribution over the model ’ s parameter , the amount of posterior contraction from the prior , and the Hessian of the negative log posterior . Figure 3 shows the prior and posterior distribution for a Bayesian linear regression model with a single parameter , with predictions generated by parameters drawn from these distributions . As expected , we see that the variance of the posterior distribution is significantly reduced from that of the prior ; we call the difference of the variance between the posterior and the prior the posterior contraction of the model . More specifically , as shown in Figure 3 , the arrival of data increases the curvature of the loss ( negative log posterior ) at the optimum . This increase in curvature of the loss that accompanies certainty about the parameters leads to an increase in the eigenvalues of the Hessian of the Growth in eigenvalues of the Hessian of the loss corresponds to increased certainty about parameters . 2.2 POSTERIOR CONTRACTION AND EFFECTIVE DIMENSIONALITY . When combined with the functional form of a model , a distribution over parameters p ( θ ) induces a distribution over functions p ( f ( x ; θ ) ) . The parameters are of little direct interest — what matters for generalization is the distribution over functions ( e.g . the right panel of Figure 3 ) . As parameter distributions concentrate around specific values we expect to generate less diverse functions , the behavior seen in Figure 3 . We show in Appendix E that we can describe posterior contraction in Bayesian linear regression , y ∼ N ( Φβ , σ2I ) , with isotropic Gaussian prior , β ∼ N ( 0 , α2Ik ) , as ∆post ( Φ ) = α 2 N∑ i=1 λi λi + α−2 , ( 1 ) where λi are the eigenvalues of Φ > Φ/α2 , the Hessian of the log likelihood , which is dependent on both the model and the training data . We refer to the summation in equation 1 as the effective dimensionality of Φ > Φ/α2 . We generalize equation 1 , defining the effective dimensionality of a symmetric matrixA ∈ Rk×k as Neff ( A , z ) = k∑ i=1 λi λi + z , ( 2 ) in which λi are the eigenvalues ofA and z > 0 is a regularization constant MacKay ( 1992a ) .1 Typically as neural networks are trained we observe a gap in the eigenspectrum of the Hessian of the loss ( Sagun et al. , 2017 ) ; a small number of eigenvalues become large while the rest are near zero . In computing effective dimensionality , eigenvalues much larger than z contribute a value of approximately one to the summation , and eigenvalues much smaller than z contribute a value of approximately zero . 1In discussing model generalization , we use effective dimensionality as short form for the effective dimensionality of the Hessian of the loss of a trained model . Therefore , the effective dimensionality explains the number of parameters that have been determined by the data , which corresponds to the number of parameters the model is using to make predictions . In comparing models of the same parameterization that achieve low loss on the training data , we expect models with lower effective dimensionality to generalize better — which is empirically verified in Figures 1 and 2 . The intuition built using Figure 3 carries through to this approximation : as the eigenvalues of the Hessian increase , the eigenvalues of the covariance matrix in our approximation to the posterior distribution shrink , further indicating contraction around the MAP estimate . Practical Computations For large neural networks computing the eigenvalues and eigenvectors of the Hessian of the loss is nontrivial . We estimate effective dimensionality by computing the dominant eigenvalues using the Lanczos algorithm implemented in GPyTorch , since many of the eigenvalues are typically close to zero and do not significantly contribute to the estimate ( Gardner et al. , 2018 ) . Hessian vector products as implemented in PyTorch take three backward passes ; In our experiments , we compute 100 Hessian vector products to produce 100 eigenvalues , so that the cost is about 1.5× the cost of standard training 2 . As a heuristic , one can set z using the connection with the prior variance α2 and ` 2 regularization , or to measure the number of relatively large eigenvalues . We use a value of z = 1 in equation 2 for all experiments , and show in Figure 4 that effective dimensionality for model comparison is highly robust to different values of z over the range of networks with near zero training loss in Figure 2 . For neural networks , the Hessian can have negative eigenvalues ( e.g. , Sagun et al. , 2017 ; Ghorbani et al. , 2019 ) ; however , these negative eigenvalues are in practice extremely small in magnitude compared to the positive ones , and do not practically impact the computations of effective dimensionality . The Hessian ( and its effective dimensionality ) is not invariant to re-parameterizations ( e.g . ReLU rescaling and batch normalization ) ( MacKay , 2003 , Chapter 27 ) . For this reason we assume a fixed parameterization , as is the case in practice , and compare between models of the same parameterization . 3 RELATED WORK . MacKay ( 1992a ) used effective dimensionality to measure posterior contraction in Bayesian neural networks . Effective dimensionality has also been used for measuring generalization error of kernel methods ( Caponnetto and Vito , 2007 ) . Various connections between flatness and generalization have been explored via Occam factors ( MacKay , 2003 ; Smith and Le , 2018 ) and minimum description length ( Hinton and Van Camp , 1993 ; Achille and Soatto , 2018 ) . Nakkiran et al . ( 2020 ) found generalization gains as neural networks become overparameterized , showing the double descent phenomenon ( e.g. , Belkin et al. , 2019a ; Nakkiran et al. , 2020 ) that occurs as the width increases in residual and convolutional neural networks . Flatness has also be considered in the PAC-Bayes literature ( e.g. , Dziugaite and Roy , 2017 ; Jiang et al. , 2020 ) , with Jiang et al . ( 2020 ) showing that PAC-Bayesian measures of flatness , in the sense of 2In Appendix I we provide an example of the insensitivity of effective dimensionality to the number of eigenvalues used . insensitivity to random perturbations , perform well relative to many generalization bounds . Zhou et al . ( 2018 ) used PAC-Bayesian compression arguments to construct non-vacuous bounds . Our work shows that effective dimensionality can shed light on a number of phenomena in modern deep learning , including double descent , width-depth trade-offs , and Bayesian subspace inference , while providing a straightforward and compelling generalization metric , relative to several of the highest performing metrics in Jiang et al . ( 2020 ) . We provide an extended discussion of historical perspectives in Appendix D .

Summary: In this article, the authors revisited the idea of *effective dimensionality* as a complexity measure for large-scale machine learning systems, and in particular, modern deep neural networks. Theoretical arguments were provided for linear and generalized linear models (Theorem 4.1 and 4.2). Connections were made between the proposed effective dimensionality and the double descent phenomenon, width-depth trade-off, function-space homogeneity, and other generalization measures in the literature. Experiments on linear models as well as deep networks (ResNet18) were provided to support the effectiveness of the proposed metric.