A Constructive Prediction of the Generalization Error Across Scales

1 INTRODUCTION . With the success and heightened adoption of neural networks for real world tasks , some questions remain poorly answered . For a given task and model architecture , how much data would one require to reach a prescribed performance level ? How big a model would be needed ? Addressing such questions is made especially difficult by the mounting evidence that large , deep neural networks trained on large-scale data outperform their smaller counterparts , rendering the training of high performance models prohibitively costly . Indeed , in the absence of practical answers to the above questions , surrogate approaches have proven useful . One such common approach is model scaling , where one designs and compares small-scale models , and applies the obtained architectural principles at a larger scale ( e.g. , Liu et al. , 2018 ; Real et al. , 2018 ; Zoph et al. , 2018 ) . Despite these heuristics being widely used to various degrees of success , the relation between the performance of a model in the small- and large-scale settings is not well understood . Hence , exploring the limitations or improving the efficiency of such methods remains subject to trial and error . In this work we circle back to the fundamental question : what is the ( functional ) relation between generalization error and model and dataset sizes ? Critically , we capitalize on the concept of model scaling in its strictest form : we consider the case where there is some given scaling policy that completely defines how to scale up a model from small to large scales . We include in this context all model parameters , such that traversing from one scale ( in which all parameters are known ) to another requires no additional resources for specifying the model ( e.g. , architecture search/design ) . We empirically explore the behavior of the generalization error over a wide range of datasets and models in vision and language tasks . While the error landscape seems fairly complex at first glance , we observe the emergence of several key characteristics shared across benchmarks and domains . Chief among these characteristics is the emergence of regions where power-law behavior approximates the error well both with respect to data size , when holding model size fixed , and vice versa . Motivated by these observations , we establish criteria which a function approximating the error landscape should meet . We propose an intuitive candidate for such a function and evaluate its quality , both in explaining the observed error landscapes and in extrapolating from small scale ( seen ) to large scale ( unseen ) errors . Critically , our functional approximation of the error depends on both model and data sizes . We find that this function leads to a high quality fit and extrapolation . For instance , the mean and standard deviation of the relative errors are under 2 % when fitting across all scales investigated and under 5 % when extrapolating from a slimmed-down model ( 1/16 of the parameters ) on a fraction of the training data ( 1/8 of the examples ) on the ImageNet ( Russakovsky et al. , 2015 ) and WikiText-103 ( Merity et al. , 2016 ) datasets , with similar results for other datasets . To the best of our knowledge , this is the first work that provides simultaneously : • A joint functional form of the generalization error landscape—as dependent on both data and model size—with few , interpretable degrees of freedom ( section 5 ) . • Direct and complete specification ( via the scaling policy ) of the model configuration attaining said generalization error across model and dataset sizes . • Highly accurate approximation of error measurements across model and data scales via the functional form , evaluated on different models , datasets , and tasks ( section 6 ) . • Highly accurate error prediction from small to large model and data ( section 7 ) . We conclude with a discussion of some implications of our findings as a practical and principled tool for understanding network design at small scale and for efficient computation and trade-off design in general . We hope this work also provides a useful empirical leg to stand on and an invitation to search for a theory of generalization error which accounts for our findings . 2 RELATED WORK . Model scaling : A number of studies have explored the effect of model scaling on performance . For instance , image classification networks can be scaled by depth ( number of layers ; He et al. , 2016 ) or width ( number of channels ; Zagoruyko & Komodakis , 2016 ; Howard et al. , 2017 ) . More recently , Tan & Le ( 2019 ) demonstrated how scaling width , depth , and input resolution has combined positive effects larger than scaling each factor in isolation . However , this relationship has yet to be quantified in a predictive form – by how much will error change with model scaling ? In this work , we focus on finding a constructive functional form for determining the model given a specified performance . Data scaling : It has long been recognized that more data improves performance , and various studies report such trends in both computer vision ( e.g. , Zhu et al. , 2012 ; Sun et al. , 2017 ) and language processing tasks ( e.g. , Banko & Brill , 2001 ; Talmor & Berant , 2019 ) . A number of prior studies observed power-law relations between the generalization error and training data size ( Cho et al. , 2015 ; Miceli Barone et al. , 2017 ; Johnson et al. , 2018 ) . Most relevant to our work , Hestness et al . ( 2017 ) explored the effect of data size on the generalization error in vision , language , and speech tasks , and observed a strikingly consistent power-law behavior in a large set of experiments . However , while these studies point to the empirical existence of a power law in terms of data , they do not offer tools for predicting the performance given a specified model . Nor do they offer low-cost methods to specify the model configuration which would attain the power law with data dependency . Indeed , Hestness et al . had to search over models and their configurations at large scale to exhibit their findings , incurring prohibitive computational costs . In contrast , we demonstrate a constructive recipe , where we directly predict the test performance at large scale and specify the full model configuration which attains it ( with no need for large-scale search ) , given performance at small scale . Predicting model performance : Since training models at full data/model scale may be computationally prohibitive , a line of work tries to predict the performance of a given model on a given dataset , without training the model , for example by using a bank of previously trained models , dataset , and their associated performances ( Istrate et al. , 2019 ) . Others have proposed to estimate performance on small data ( Klein et al. , 2017 ) or model sizes ( Zoph et al. , 2018 ; Real et al. , 2019 ) in the context of neural architecture search ( NAS ) . In this case , the small-scale evaluation is used to compare models at small cost , to expedite the search process ; see Elsken et al . ( 2019 ) for a recent survey . Our work complements previous approaches by demonstrating a functional form that can predict large-scale performance from small-scale measurements . Moreover , our method may be integrated in NAS , addressing some of its current limitations ( as discussed in section 8 ) . Theoretical error bounds : Much attention has been given to theoretical explanations of the generalization capabilities of deep neural networks ( Neyshabur et al. , 2017a ; b ; Allen-Zhu et al. , 2018a ; b ; Arora et al. , 2018 ) . While fully engaging with this literature is beyond our scope , we note that recent studies have derived bounds involving power-law dependencies in both model ( Yarotsky , 2018 ) and data size ( Liang et al. , 2019 ) . We leave it as an open question for future work to find theoretical explanations for the empirical behavior and the functional form we investigate in this work . 3 EXPERIMENTAL SETUP . Notation : Let Dn = { xi , yi } ni=1 denote a labeled ( training ) dataset with n samples or datapoints . Let fm denote a neural network whose size is the number of parameters m , such that ŷ = fm ( x ) is the predicted label . Let ( n , m ) be the generalization error as a function of n and m , measured by a performance metric ( e.g. , top-1 accuracy or cross-entropy loss ) on a held-out test set . We refer to this error function as the error landscape . 3.1 SCALING POLICIES . Dataset scaling : We wish to scale datasets while preserving the original distribution . For image classification , we uniformly subsample all classes by a constant ratio , thus preserving the relative sample size per class . We limit the maximal sub-sampling to avoid eradicating any class . For language modeling , where the number of classes ( vocabulary items ) has a very long tail distribution , we randomly sample sentences such that the total number of sampled words will be a certain fraction of the original dataset . Table 1 reports the data scales we use . In all tasks the held-out test set remains untouched for evaluating the error . Model scaling : We are critically interested in a method where moving across scales is defined by some scaling function , such that no additional significant computation would be incurred . We thus consider the case where the model architecture is given and the model size determines how to scale it . For instance , one may scale width ( number of channels in convolutional networks , hidden state size in recurrent networks ) , depth ( number of layers ) , do compound scaling ( Tan & Le , 2019 ) , or more generally define a function tying the model degrees of freedom and size . We focus primarily on width scaling in our experiments ; the model scales are reported in Table 1 . We also perform selected depth scaling to demonstrate flexibility with respect to the scaling method . 0 ( er r ) Hyper-parameters : For similar reasons we wish to avoid hyper-paramater search at large scales , and thus avoid the temptation to tune hyper-parameters accordingly ( learning rate , regularization , etc. ) . Therefore , we hold all hyper-parameters fixed . This enables us to construct a functional form that fits the error landscape and can be used to predict the error across scales while completely defining the model attaining it . We consider pros and cons of this approach in the discussion ( section 8 ) . 3.2 TASKS , MODELS , OPTIMIZERS AND DATASETS . We experiment with both vision and language tasks . We use 6 benchmark datasets for image classification and 3 for language modeling . For image classification , we train ResNet ( He et al. , 2016 ) and WRN models ( Zagoruyko & Komodakis , 2016 ) with stochastic gradient decent ( SGD ) . In section 6.2 we explore the effect of varying architectures and optimizers for a fixed task ( CIFAR100 ) , adding VGG16 ( Simonyan & Zisserman , 2014 ) and DenseNet ( Huang et al. , 2017 ) models trained with both Adam ( Kingma & Ba , 2015 ) and SGD . For language modeling , we train AWD-LSTM ( Merity et al. , 2018 ) and Transformer-XL models ( Dai et al. , 2019 ) with SGD and Adam optimizers respectively . Summary statistics are shown in Table 1 , along with the range of explored scales . Appendix A gives additional information .

Summary: This paper proposes a functional form to model the dependence of generalization error on a held-out test set on model and dataset size. The functional form is derived based on empirical observations of the generalizing error for various model and dataset sizes (sections O1, O2, and O3) and on certain necessary criteria (C1, C4 and C5). The parameters of the function are then fit using linear regression on observed data. The authors show that the regressed function \(\epsilon(m,n)\) is able to predict the generalization error for various \(m\) and \(n\) reasonably accurately.