A Constructive Prediction of the Generalization Error Across Scales

1 INTRODUCTION . With the success and heightened adoption of neural networks for real world tasks , some questions remain poorly answered . For a given task and model architecture , how much data would one require to reach a prescribed performance level ? How big a model would be needed ? Addressing such questions is made especially difficult by the mounting evidence that large , deep neural networks trained on large-scale data outperform their smaller counterparts , rendering the training of high performance models prohibitively costly . Indeed , in the absence of practical answers to the above questions , surrogate approaches have proven useful . One such common approach is model scaling , where one designs and compares small-scale models , and applies the obtained architectural principles at a larger scale ( e.g. , Liu et al. , 2018 ; Real et al. , 2018 ; Zoph et al. , 2018 ) . Despite these heuristics being widely used to various degrees of success , the relation between the performance of a model in the small- and large-scale settings is not well understood . Hence , exploring the limitations or improving the efficiency of such methods remains subject to trial and error . In this work we circle back to the fundamental question : what is the ( functional ) relation between generalization error and model and dataset sizes ? Critically , we capitalize on the concept of model scaling in its strictest form : we consider the case where there is some given scaling policy that completely defines how to scale up a model from small to large scales . We include in this context all model parameters , such that traversing from one scale ( in which all parameters are known ) to another requires no additional resources for specifying the model ( e.g. , architecture search/design ) . We empirically explore the behavior of the generalization error over a wide range of datasets and models in vision and language tasks . While the error landscape seems fairly complex at first glance , we observe the emergence of several key characteristics shared across benchmarks and domains . Chief among these characteristics is the emergence of regions where power-law behavior approximates the error well both with respect to data size , when holding model size fixed , and vice versa . Motivated by these observations , we establish criteria which a function approximating the error landscape should meet . We propose an intuitive candidate for such a function and evaluate its quality , both in explaining the observed error landscapes and in extrapolating from small scale ( seen ) to large scale ( unseen ) errors . Critically , our functional approximation of the error depends on both model and data sizes . We find that this function leads to a high quality fit and extrapolation . For instance , the mean and standard deviation of the relative errors are under 2 % when fitting across all scales investigated and under 5 % when extrapolating from a slimmed-down model ( 1/16 of the parameters ) on a fraction of the training data ( 1/8 of the examples ) on the ImageNet ( Russakovsky et al. , 2015 ) and WikiText-103 ( Merity et al. , 2016 ) datasets , with similar results for other datasets . To the best of our knowledge , this is the first work that provides simultaneously : • A joint functional form of the generalization error landscape—as dependent on both data and model size—with few , interpretable degrees of freedom ( section 5 ) . • Direct and complete specification ( via the scaling policy ) of the model configuration attaining said generalization error across model and dataset sizes . • Highly accurate approximation of error measurements across model and data scales via the functional form , evaluated on different models , datasets , and tasks ( section 6 ) . • Highly accurate error prediction from small to large model and data ( section 7 ) . We conclude with a discussion of some implications of our findings as a practical and principled tool for understanding network design at small scale and for efficient computation and trade-off design in general . We hope this work also provides a useful empirical leg to stand on and an invitation to search for a theory of generalization error which accounts for our findings . 2 RELATED WORK . Model scaling : A number of studies have explored the effect of model scaling on performance . For instance , image classification networks can be scaled by depth ( number of layers ; He et al. , 2016 ) or width ( number of channels ; Zagoruyko & Komodakis , 2016 ; Howard et al. , 2017 ) . More recently , Tan & Le ( 2019 ) demonstrated how scaling width , depth , and input resolution has combined positive effects larger than scaling each factor in isolation . However , this relationship has yet to be quantified in a predictive form – by how much will error change with model scaling ? In this work , we focus on finding a constructive functional form for determining the model given a specified performance . Data scaling : It has long been recognized that more data improves performance , and various studies report such trends in both computer vision ( e.g. , Zhu et al. , 2012 ; Sun et al. , 2017 ) and language processing tasks ( e.g. , Banko & Brill , 2001 ; Talmor & Berant , 2019 ) . A number of prior studies observed power-law relations between the generalization error and training data size ( Cho et al. , 2015 ; Miceli Barone et al. , 2017 ; Johnson et al. , 2018 ) . Most relevant to our work , Hestness et al . ( 2017 ) explored the effect of data size on the generalization error in vision , language , and speech tasks , and observed a strikingly consistent power-law behavior in a large set of experiments . However , while these studies point to the empirical existence of a power law in terms of data , they do not offer tools for predicting the performance given a specified model . Nor do they offer low-cost methods to specify the model configuration which would attain the power law with data dependency . Indeed , Hestness et al . had to search over models and their configurations at large scale to exhibit their findings , incurring prohibitive computational costs . In contrast , we demonstrate a constructive recipe , where we directly predict the test performance at large scale and specify the full model configuration which attains it ( with no need for large-scale search ) , given performance at small scale . Predicting model performance : Since training models at full data/model scale may be computationally prohibitive , a line of work tries to predict the performance of a given model on a given dataset , without training the model , for example by using a bank of previously trained models , dataset , and their associated performances ( Istrate et al. , 2019 ) . Others have proposed to estimate performance on small data ( Klein et al. , 2017 ) or model sizes ( Zoph et al. , 2018 ; Real et al. , 2019 ) in the context of neural architecture search ( NAS ) . In this case , the small-scale evaluation is used to compare models at small cost , to expedite the search process ; see Elsken et al . ( 2019 ) for a recent survey . Our work complements previous approaches by demonstrating a functional form that can predict large-scale performance from small-scale measurements . Moreover , our method may be integrated in NAS , addressing some of its current limitations ( as discussed in section 8 ) . Theoretical error bounds : Much attention has been given to theoretical explanations of the generalization capabilities of deep neural networks ( Neyshabur et al. , 2017a ; b ; Allen-Zhu et al. , 2018a ; b ; Arora et al. , 2018 ) . While fully engaging with this literature is beyond our scope , we note that recent studies have derived bounds involving power-law dependencies in both model ( Yarotsky , 2018 ) and data size ( Liang et al. , 2019 ) . We leave it as an open question for future work to find theoretical explanations for the empirical behavior and the functional form we investigate in this work . 3 EXPERIMENTAL SETUP . Notation : Let Dn = { xi , yi } ni=1 denote a labeled ( training ) dataset with n samples or datapoints . Let fm denote a neural network whose size is the number of parameters m , such that ŷ = fm ( x ) is the predicted label . Let ( n , m ) be the generalization error as a function of n and m , measured by a performance metric ( e.g. , top-1 accuracy or cross-entropy loss ) on a held-out test set . We refer to this error function as the error landscape . 3.1 SCALING POLICIES . Dataset scaling : We wish to scale datasets while preserving the original distribution . For image classification , we uniformly subsample all classes by a constant ratio , thus preserving the relative sample size per class . We limit the maximal sub-sampling to avoid eradicating any class . For language modeling , where the number of classes ( vocabulary items ) has a very long tail distribution , we randomly sample sentences such that the total number of sampled words will be a certain fraction of the original dataset . Table 1 reports the data scales we use . In all tasks the held-out test set remains untouched for evaluating the error . Model scaling : We are critically interested in a method where moving across scales is defined by some scaling function , such that no additional significant computation would be incurred . We thus consider the case where the model architecture is given and the model size determines how to scale it . For instance , one may scale width ( number of channels in convolutional networks , hidden state size in recurrent networks ) , depth ( number of layers ) , do compound scaling ( Tan & Le , 2019 ) , or more generally define a function tying the model degrees of freedom and size . We focus primarily on width scaling in our experiments ; the model scales are reported in Table 1 . We also perform selected depth scaling to demonstrate flexibility with respect to the scaling method . 0 ( er r ) Hyper-parameters : For similar reasons we wish to avoid hyper-paramater search at large scales , and thus avoid the temptation to tune hyper-parameters accordingly ( learning rate , regularization , etc. ) . Therefore , we hold all hyper-parameters fixed . This enables us to construct a functional form that fits the error landscape and can be used to predict the error across scales while completely defining the model attaining it . We consider pros and cons of this approach in the discussion ( section 8 ) . 3.2 TASKS , MODELS , OPTIMIZERS AND DATASETS . We experiment with both vision and language tasks . We use 6 benchmark datasets for image classification and 3 for language modeling . For image classification , we train ResNet ( He et al. , 2016 ) and WRN models ( Zagoruyko & Komodakis , 2016 ) with stochastic gradient decent ( SGD ) . In section 6.2 we explore the effect of varying architectures and optimizers for a fixed task ( CIFAR100 ) , adding VGG16 ( Simonyan & Zisserman , 2014 ) and DenseNet ( Huang et al. , 2017 ) models trained with both Adam ( Kingma & Ba , 2015 ) and SGD . For language modeling , we train AWD-LSTM ( Merity et al. , 2018 ) and Transformer-XL models ( Dai et al. , 2019 ) with SGD and Adam optimizers respectively . Summary statistics are shown in Table 1 , along with the range of explored scales . Appendix A gives additional information .

Summary: This work proposes a functional form for the relationship between <dataset size, model size> and generalization error, and performs an empirical study to validate it. First, it states 5 criteria that such a functional form must take, and proposes one such functional form containing 6 free coefficients that satisfy all these criteria. It then performs a rigorous empirical study consisting of 6 image datasets and 3 text datasets, each with 2 distinct architectures defined at several model scales, and trained with different dataset sizes. This process produces 42-49 data points for each <dataset, architecture> pair, and the 6 coefficients of the proposed functional form are fit to those data points, with < 2% mean deviation in accuracy. It then studies how this functional form performs at extrapolation, and finds that it still performs pretty well, with ~4.5% mean deviation in accuracy, but with additional caveats.