Learning Curves for Analysis of Deep Networks

A learning curve models a classifier ’ s test error as a function of the number of training samples . Prior works show that learning curves can be used to select model parameters and extrapolate performance . We investigate how to use learning curves to analyze the impact of design choices , such as pretraining , architecture , and data augmentation . We propose a method to robustly estimate learning curves , abstract their parameters into error and data-reliance , and evaluate the effectiveness of different parameterizations . We also provide several interesting observations based on learning curves for a variety of image classification models . 1 INTRODUCTION . What gets measured gets optimized . We need better measures of learning ability to design better classifiers and predict the payoff of collecting more data . Currently , classifiers are evaluated and compared by measuring performance on one or more datasets according to a fixed train/test split . Ablation studies help evaluate the impact of design decisions . However , one of the most important characteristics of a classifier , how it performs with varying numbers of training samples , is rarely measured or modeled . In this paper , we refine the idea of learning curves that model error as a function of training set size . Learning curves were introduced nearly thirty years ( e.g . by Cortes et al . ( 1993 ) ) to accelerate model selection of deep networks . Recent works have demonstrated the predictability of performance improvements with more data ( Hestness et al. , 2017 ; Johnson & Nguyen , 2017 ; Kaplan et al. , 2020 ; Rosenfeld et al. , 2020 ) or more network parameters ( Kaplan et al. , 2020 ; Rosenfeld et al. , 2020 ) . But such studies have typically required large-scale experiments that are outside the computational budgets of many research groups , and their purpose is extrapolation rather than validating design choices . We find that a generalized power law function provides the best learning curve fit , while a model linear in n−0.5 , where n is the number of training samples ( or “ training size ” ) , provides a good local approximation . We abstract the curve into two key parameters : eN and βN . eN is error at n = N , and βN is a measure of data-reliance , revealing how much a classifier ’ s error will change if the training set size changes . Learning curves provide valuable insights that can not be obtained by single-point comparisons of performance . Our aim is to promote the use of learning curves as part of a standard learning system evaluation . Our key contributions : • Investigate how to best model , estimate , characterize , and display learning curves for use in classifier analysis • Use learning curves to analyze impact on error and data-reliance due to network architecture , depth , width , fine-tuning , data augmentation , and pretraining Table 1 shows validated and rejected popular beliefs that single-point comparisons often overlook . In the following sections , we investigate how to model learning curves ( Sec . 2 ) , how to estimate them ( Sec . 3 ) , and what they can tell us about the impact of design decisions ( Sec . 4 ) , with discussion of limitations and future work in Sec . 5 . 2 MODELING LEARNING CURVES . The learning curve measures test error etest as a function of the number of training samples n for a given classification model and learning method . Previous empirical observations suggest a functional form etest ( n ) = α + ηnγ , with bias-variance trade-off and generalization theories typically indicating γ = −0.5 . We summarize what bias-variance trade-off and generalization theories ( Sec . 2.1 ) and empirical studies ( Sec . 2.2 ) can tell us about learning curves , and describe our proposed abstraction in Sec . 2.3 . 2.1 BIAS-VARIANCE TRADE-OFF AND GENERALIZATION THEORY . The bias-variance trade-off is an intuitive and theoretical way to think about generalization . The “ bias ” is error due to inability of the classifier to encode the optimal decision function , and the “ variance ” is error due to variations in predictions due to limited availability of training samples for parameter estimation . This is called a trade-off because a classifier with more parameters tends to have less bias but higher variance . Geman et al . ( 1992 ) decompose mean squared regression error into bias and variance and explore the implications for neural networks , leading to the conclusion that “ identifying the right preconditions is the substantial problem in neural modeling ” . This conclusion foreshadows the importance of pretraining , though Geman et al . thought the preconditions must be built in rather than learned . Domingos ( 2000 ) extends the analysis to classification . Theoretically , the mean squared error ( MSE ) can be modeled as e2test ( n ) = bias 2 + noise2 + var ( n ) , where “ noise ” is irreducible error due to non-unique mapping from inputs to labels , and variance can be modeled as var ( n ) = σ2/n for n training samples . The ηn−0.5 term appears throughout machine learning generalization theory . For example , the bounds based on hypothesis VC-dimension ( Vapnik & Chervonenkis , 1971 ) and Rademacher Complexity ( Gnecco & Sanguineti , 2008 ) are both O ( cn−0.5 ) where c depends on the complexity of the classification model . More recent work also follows this form . We give some examples of bounds in Table 2 without describing all of the parameters because the point is that the test error bounds vary with training size n as a function of n−0.5 , for all approaches . 2.2 EMPIRICAL STUDIES . Some recent empirical studies ( e.g . Sun et al . ( 2017 ) ) claim a log-linear relationship between error and training size , but this holds only when asymptotic error is zero . Hestness et al . ( 2017 ) model error as etest ( n ) = α + ηnγ but often find γ much smaller in magnitude than −0.5 and suggest that poor fits indicate need for better hyperparameter tuning . This raises an interesting point that sample efficiency depends both on the classification model and on the efficacy of the optimization algorithm and parameters . Johnson & Nguyen ( 2017 ) also find a better fit with this extended power law model than by restricting γ = −0.5 or α = 0 . We find that , by selecting the learning rate through validation on one training size and using the Ranger optimizer ( Wright , 2019 ) , we can achieve a good approximate fit with γ = −0.5 and best fit with −0.3 < γ < −0.7 . In the language domain , learning curves are used in a fascinating study by Kaplan et al . ( 2020 ) . For natural language transformers , they show that a power law relationship between logistic loss , model size , compute time , and dataset size is maintained if ( and only if ) each is increased in tandem . We draw some similar conclusions to their study , such as that increasing model size tends to improve performance especially for small training sets ( which surprised us ) . However , the studies are largely complementary , as we study convolutional nets in computer vision , classification error ( instead of logistic loss ) , and a broader range of design choices such as effects across depth , width , data augmentation , pretraining source , architecture , and dataset . Also related , Rosenfeld et al . ( 2020 ) model error as a function of both training size and number of model parameters with a five-parameter function that accounts for training size , model parameter size , and chance performance . A key difference in our work is that we focus on how to best draw insights about design choices from learning curves , rather than on extrapolation . As such , we propose methods to estimate learning curves and their variance from a relatively small number of trained models . 2.3 PROPOSED CHARACTERIZATION OF LEARNING CURVES FOR EVALUATION . A classifier ’ s performance can be characterized in terms of its error and data-reliance , or how quickly the error changes with training size n. With e ( n ) = α + ηnγ , we find that γ = −0.5 provides a good local approximation but that fitting γ significantly improves leave-one-size-out RMS error and extrapolation accuracy , as we detail in Sec . 4 . However , α , η , and γ can not be meaningfully compared across curves because the parameters have high covariance with small data perturbations , and comparing η values is not meaningful unless γ is fixed and vice-versa . We propose to report error and sensitivity to training size in a way that can be derived from various learning curve models and is insensitive to data perturbations . The curve is characterized by error eN = α + ηN γ and data-reliance βN at N , and we typically choose N as the full dataset size . Noting that most learning curves are locally well approximated by a model linear in n−0.5 , we compute data-reliance as βN = N−0.5 ∂e∂n−0.5 ∣∣ n=N = −2ηγNγ . When the error is plotted against n−0.5 , βN is the slope at N scaled by N−0.5 , where the scaling was chosen to make the practical implications of βN more intuitive . This yields a simple predictor for error when changing training size by a factor of d : e ( d ·N ) = eN + ( 1√ d − 1 ) βN . ( 1 ) Thus , by this linearized estimate , asymptotic error is eN −βN , a 4-fold increase in data ( e.g . 400→ 1600 ) reduces error by 0.5βN , and using only one quarter of the dataset ( e.g . 400→ 100 ) increases the error by βN . For two models with similar eN , the one with a larger βN would outperform with more data but underperform with less . Note that ( eN , βN , γ ) is a complete re-parameterization of the extended power law , with γ + 0.5 indicating the curvature in n−0.5 scale . 3 ESTIMATING LEARNING CURVES . We now describe the method for estimating the learning curve from error measurements with confidence bounds on the estimate . Let eij denote the random variable corresponding to test error when the model is trained on the jth fold of ni samples ( either per class or in total ) . We assume { eij } Fij=1 are i.i.d according to N ( µi , σ2i ) . We want to estimate learning curve parameters α ( asymptotic error ) , η , and γ , such that eij = α + ηn γ i + ij where ij ∼ N ( 0 , σ2i ) and µij = E [ eij ] = µi . Sections 3.1 and 3.2 describe how to estimate mean and variance of α and η for a given γ , and Sec . 3.3 describes our approach for estimating γ . 3.1 WEIGHTED LEAST SQUARES FORMULATION . We estimate learning curve parameters { α , η } by optimizing a weighted least squares objective : G ( γ ) = min α , η S∑ i=1 Fi∑ j=1 wij ( eij − α− ηnγ ) 2 ( 2 ) wherewij = 1/ ( Fiσ2i ) . Fi is the number of models trained with data size ni and is used to normalize the weight so that the total weight for observations from each training size does not depend on Fi . The factor of σ2i accounts for the variance of ij . Assuming constant σ 2 i and removing the Fi factor would yield unweighted least squares . The variance of the estimate of σ2i from Fi samples is 2σ 4 i /Fi , which can lead to over- or underweighting data for particular i if Fi is small . Recall that each sample eij requires training an entire model , so Fi is always small in our experiments . We would expect the variance to have the form σ2i = σ 2 0 + σ̂ 2/ni , where σ20 is the variance due to random initialization and optimization and σ̂ 2/ni is the variance due to randomness in selecting ni samples . Indeed , by averaging over the variance estimates for many different network models on the CIFAR-100 ( Krizhevsky , 2012 ) dataset , we find a good fit with σ20 = 0.2 . This enables us to estimate a single σ̂ 2 parameter from all samples e in a given learning curve as a least squares fit and also upper-bounds wij < = 5 even if two models happen to have the same error . This attention to wij may seem fussy , but without such care we find that the learning curve often fails to account sufficiently for all the data in some cases .

Summary: This paper advocates for studying the effect of design choices in deep learning via their effect on entire *learning curves* (test error vs num samples N), as opposed to their effect only for a fixed N. This is a valid and important message, and it is indeed an aspect that is often overlooked in certain domains. However, although this paper addresses an important issue, there are methodological concerns (described below) which prevent me from recommending acceptance. In summary, the paper oversimplifies certain important aspects in both the setup and the experiments.