Year: 2020

Abstract:
A learning curve models a classifier’s test error as a function of the number of training samples. Prior works show that learning curves can be used to select model parameters and extrapolate performance. We investigate how to use learning curves to analyze the impact of design choices, such as pretraining, architecture, and data augmentation. We propose a method to robustly estimate learning curves, abstract their parameters into error and data-reliance, and evaluate the effectiveness of different parameterizations. We also provide several interesting observations based on learning curves for a variety of image classification models.

text: A learning curve models a classifier’s test error as a function of the number of training samples. Prior works show that learning curves can be used to select model parameters and extrapolate performance. We investigate how to use learning curves to analyze the impact of design choices, such as pretraining, architecture, and data augmentation. We propose a method to robustly estimate learning curves, abstract their parameters into error and data-reliance, and evaluate the effectiveness of different parameterizations. We also provide several interesting observations based on learning curves for a variety of image classification models.

heading: 1 INTRODUCTION
text: What gets measured gets optimized. We need better measures of learning ability to design better classifiers and predict the payoff of collecting more data. Currently, classifiers are evaluated and compared by measuring performance on one or more datasets according to a fixed train/test split. Ablation studies help evaluate the impact of design decisions. However, one of the most important characteristics of a classifier, how it performs with varying numbers of training samples, is rarely measured or modeled.
In this paper, we refine the idea of learning curves that model error as a function of training set size. Learning curves were introduced nearly thirty years (e.g. by Cortes et al. (1993)) to accelerate model selection of deep networks. Recent works have demonstrated the predictability of performance improvements with more data (Hestness et al., 2017; Johnson & Nguyen, 2017; Kaplan et al., 2020; Rosenfeld et al., 2020) or more network parameters (Kaplan et al., 2020; Rosenfeld et al., 2020). But such studies have typically required large-scale experiments that are outside the computational budgets of many research groups, and their purpose is extrapolation rather than validating design choices. We find that a generalized power law function provides the best learning curve fit, while a model linear in n−0.5, where n is the number of training samples (or “training size”), provides a good local approximation. We abstract the curve into two key parameters: eN and βN . eN is error at n = N , and βN is a measure of data-reliance, revealing how much a classifier’s error will change if the training set size changes. Learning curves provide valuable insights that cannot be obtained by single-point comparisons of performance. Our aim is to promote the use of learning curves as part of a standard learning system evaluation.
Our key contributions:
• Investigate how to best model, estimate, characterize, and display learning curves for use in classifier analysis
• Use learning curves to analyze impact on error and data-reliance due to network architecture, depth, width, fine-tuning, data augmentation, and pretraining
Table 1 shows validated and rejected popular beliefs that single-point comparisons often overlook. In the following sections, we investigate how to model learning curves (Sec. 2), how to estimate them (Sec. 3), and what they can tell us about the impact of design decisions (Sec. 4), with discussion of limitations and future work in Sec. 5.

heading: 2 MODELING LEARNING CURVES
text: The learning curve measures test error etest as a function of the number of training samples n for a given classification model and learning method. Previous empirical observations suggest a functional form etest(n) = α + ηnγ , with bias-variance trade-off and generalization theories typically indicating γ = −0.5. We summarize what bias-variance trade-off and generalization theories (Sec. 2.1) and empirical studies (Sec. 2.2) can tell us about learning curves, and describe our proposed abstraction in Sec. 2.3.

heading: 2.1 BIAS-VARIANCE TRADE-OFF AND GENERALIZATION THEORY
text: The bias-variance trade-off is an intuitive and theoretical way to think about generalization. The “bias” is error due to inability of the classifier to encode the optimal decision function, and the “variance” is error due to variations in predictions due to limited availability of training samples for parameter estimation. This is called a trade-off because a classifier with more parameters tends to have less bias but higher variance. Geman et al. (1992) decompose mean squared regression error into bias and variance and explore the implications for neural networks, leading to the conclusion that “identifying the right preconditions is the substantial problem in neural modeling”. This conclusion foreshadows the importance of pretraining, though Geman et al. thought the preconditions must be built in rather than learned. Domingos (2000) extends the analysis to classification. Theoretically, the mean squared error (MSE) can be modeled as e2test(n) = bias
2 + noise2 + var(n), where “noise” is irreducible error due to non-unique mapping from inputs to labels, and variance can be modeled as var(n) = σ2/n for n training samples.
The ηn−0.5 term appears throughout machine learning generalization theory. For example, the bounds based on hypothesis VC-dimension (Vapnik & Chervonenkis, 1971) and Rademacher Complexity (Gnecco & Sanguineti, 2008) are both O(cn−0.5) where c depends on the complexity of the classification model. More recent work also follows this form. We give some examples of bounds in Table 2 without describing all of the parameters because the point is that the test error bounds vary with training size n as a function of n−0.5, for all approaches.

heading: 2.2 EMPIRICAL STUDIES
text: Some recent empirical studies (e.g. Sun et al. (2017)) claim a log-linear relationship between error and training size, but this holds only when asymptotic error is zero. Hestness et al. (2017) model error as etest(n) = α + ηnγ but often find γ much smaller in magnitude than −0.5 and suggest that poor fits indicate need for better hyperparameter tuning. This raises an interesting point that sample efficiency depends both on the classification model and on the efficacy of the optimization algorithm and parameters. Johnson & Nguyen (2017) also find a better fit with this extended power law model than by restricting γ = −0.5 or α = 0. We find that, by selecting the learning rate through validation on one training size and using the Ranger optimizer (Wright, 2019), we can achieve a good approximate fit with γ = −0.5 and best fit with −0.3 < γ < −0.7. In the language domain, learning curves are used in a fascinating study by Kaplan et al. (2020). For natural language transformers, they show that a power law relationship between logistic loss, model size, compute time, and dataset size is maintained if (and only if) each is increased in tandem. We draw some similar conclusions to their study, such as that increasing model size tends to improve performance especially for small training sets (which surprised us). However, the studies are largely complementary, as we study convolutional nets in computer vision, classification error (instead of logistic loss), and a broader range of design choices such as effects across depth, width, data augmentation, pretraining source, architecture, and dataset. Also related, Rosenfeld et al. (2020) model error as a function of both training size and number of model parameters with a five-parameter function that accounts for training size, model parameter size, and chance performance.
A key difference in our work is that we focus on how to best draw insights about design choices from learning curves, rather than on extrapolation. As such, we propose methods to estimate learning curves and their variance from a relatively small number of trained models.

heading: 2.3 PROPOSED CHARACTERIZATION OF LEARNING CURVES FOR EVALUATION
text: A classifier’s performance can be characterized in terms of its error and data-reliance, or how quickly the error changes with training size n. With e(n) = α + ηnγ , we find that γ = −0.5 provides a good local approximation but that fitting γ significantly improves leave-one-size-out RMS error and extrapolation accuracy, as we detail in Sec. 4. However, α, η, and γ cannot be meaningfully compared across curves because the parameters have high covariance with small data perturbations, and comparing η values is not meaningful unless γ is fixed and vice-versa.
We propose to report error and sensitivity to training size in a way that can be derived from various learning curve models and is insensitive to data perturbations. The curve is characterized by error eN = α + ηN
γ and data-reliance βN at N , and we typically choose N as the full dataset size. Noting that most learning curves are locally well approximated by a model linear in n−0.5, we compute data-reliance as βN = N−0.5 ∂e∂n−0.5 ∣∣ n=N
= −2ηγNγ . When the error is plotted against n−0.5, βN is the slope at N scaled by N−0.5, where the scaling was chosen to make the practical implications of βN more intuitive. This yields a simple predictor for error when changing training size by a factor of d:
e(d ·N) = eN + ( 1√ d − 1 ) βN . (1)
Thus, by this linearized estimate, asymptotic error is eN −βN , a 4-fold increase in data (e.g. 400→ 1600) reduces error by 0.5βN , and using only one quarter of the dataset (e.g. 400→ 100) increases the error by βN . For two models with similar eN , the one with a larger βN would outperform with more data but underperform with less. Note that (eN , βN , γ) is a complete re-parameterization of the extended power law, with γ + 0.5 indicating the curvature in n−0.5 scale.

heading: 3 ESTIMATING LEARNING CURVES
text: We now describe the method for estimating the learning curve from error measurements with confidence bounds on the estimate. Let eij denote the random variable corresponding to test error when the model is trained on the jth fold of ni samples (either per class or in total). We assume {eij}Fij=1 are i.i.d according to N (µi, σ2i ). We want to estimate learning curve parameters α (asymptotic error), η, and γ, such that eij = α + ηn γ i + ij where ij ∼ N (0, σ2i ) and µij = E[eij ] = µi.
Sections 3.1 and 3.2 describe how to estimate mean and variance of α and η for a given γ, and Sec. 3.3 describes our approach for estimating γ.

heading: 3.1 WEIGHTED LEAST SQUARES FORMULATION
text: We estimate learning curve parameters {α, η} by optimizing a weighted least squares objective:
G(γ) = min α,η S∑ i=1 Fi∑ j=1 wij (eij − α− ηnγ)2 (2)
wherewij = 1/(Fiσ2i ). Fi is the number of models trained with data size ni and is used to normalize the weight so that the total weight for observations from each training size does not depend on Fi. The factor of σ2i accounts for the variance of ij . Assuming constant σ 2 i and removing the Fi factor would yield unweighted least squares.
The variance of the estimate of σ2i from Fi samples is 2σ 4 i /Fi, which can lead to over- or underweighting data for particular i if Fi is small. Recall that each sample eij requires training an entire model, so Fi is always small in our experiments. We would expect the variance to have the form σ2i = σ 2 0 + σ̂ 2/ni, where σ20 is the variance due to random initialization and optimization and σ̂ 2/ni is the variance due to randomness in selecting ni samples. Indeed, by averaging over the variance estimates for many different network models on the CIFAR-100 (Krizhevsky, 2012) dataset, we find a good fit with σ20 = 0.2. This enables us to estimate a single σ̂
2 parameter from all samples e in a given learning curve as a least squares fit and also upper-bounds wij <= 5 even if two models happen to have the same error. This attention to wij may seem fussy, but without such care we find that the learning curve often fails to account sufficiently for all the data in some cases.

heading: 3.2 SOLVING FOR LEARNING CURVE MEAN AND VARIANCE
text: Concatenating errors across dataset sizes (indexed by i) and folds results in an error vector e of dimension D = ∑S i=1 Fi. For each d ∈ {1, · · · , D}, e[d] is an observation of error at dataset size nid that follows N (µid , σ2id) with id mapping d to the corresponding i. The weighted least squares problem can be formulated as solving a system of linear equations denoted by W 1/2e = W 1/2Aθ, where W ∈ RD×D is a diagonal matrix of weights Wdd = wd, A ∈ RD×2 is a matrix with A[d, :] = [1 nγd ], and θ = [α η]
T are the parameters of the learning curve, treating γ as fixed for now. The estimator for the learning curve is then given by θ̂ = (W 1/2A)+W 1/2e = Me, where M ∈ R2×D and + is pseudo-inverse operator. We compute a mean curve using
θ = E[θ̂] = ME[e] = Mµ (3) where µ ∈ RD with µ[d] = µ̂id computed by empirical estimate as ∑Fi j=1 eij/Fi.
The covariance of the estimator is given by
Σθ̂ = MΣeM T (4)
where Σθ̂ ∈ R2×2 and Σe ∈ RD×D is the covariance of e, where Σe[d1, d2] = σ2id1 if id1 = id2 and 0 otherwise. We compute our empirical estimate of σ2i as described in Sec. 3.1.
Since the curve is given by e(n) = [1 nγ ]θ, the mean curve can be computed as
e(n) = [1 nγ ]θ = α+ ηnγ . (5)
The 95% bounds at any n can be computed as e(n)± 1.96× σ̂(n) with
σ̂2(n) = [1 nγ ] Σθ̂ [ 1 nγ ] (6)
where α̂ and η̂ are the empirical estimates of α and η. These confidence bounds reflect the variance in error measurements, assuming the parameterization is capable of fitting the true mean.
3.3 ESTIMATING γ
We search for γ that minimizes the weighted least squares objective with an L1-prior that slightly encourages values close to 0.5. Specifically, we solve
min γ∈(−1,0)
G(γ) + λ|γ + 0.5| (7)
by searching over γ ∈ {−0.99, ...,−0.01} with λ = 5 for our experiments.

heading: 4 EXPERIMENTS
text: We describe our implementation details in Sec. 4.1, apply learning curves to gain insights about error and data-reliance in Sec. 4.2, and validate our choice of learning curve parameterization and fitting weights used in the least squares objective in Sec. 4.3.

heading: 4.1 IMPLEMENTATION DETAILS
text: We use Pytorch-Lightning (Falcon, 2019) for our implementation with various architectures, weight initializations, data augmentation, and linear or fine-tuning optimization.
Training: We train models with images of size 32×32 for CIFAR (Krizhevsky, 2012) and 224×224 for Places365 (Zhou et al., 2017) with a batch size of 64 (except for Wide-ResNet101 and WideResNeXt101, where we use a batch size of 32 and performed one optimizer step every two batches). For each experiment setting, we conduct a learning rate search on a subset of the training data and choose the learning rate with the highest validation accuracy, and use it for all other subsets. We determine each fold’s training schedule on a mini-train/mini-val split of 2:1 on the train set. Each time the mini-val error stops decreasing for some epochs (“patience”), we revert to the best epoch and decrease the learning rate to 10%, and we perform this twice. Then we use this optimal mini-train learning rate schedule and ending epoch to train on the whole fold. The patience is ∝ 1/√n, and is 5 at the n = 400 samples/class for CIFAR100/Places365 and 15 at the largest training size for other smaller datasets. We use a weight decay value of 0.0001. We use the Ranger optimizer (Wright, 2019), which combines Rectified Adam (Liu et al., 2019), Look Ahead (Zhang et al., 2019), and Gradient Centralization (Yong et al., 2020). In early experiments, we found Ranger to lead to lower error and to reduce sensitivity of hyperparameters, compared to vanilla SGD or Adam (Kingma & Ba, 2015).
Backbone Architecture: We use the default Pytorch implementations of all of the following architectures: AlexNet (Krizhevsky et al., 2012), ResNet-18, ResNet-50, ResNet-101 (He et al., 2015b), ResNeXt-50, ResNeXt-100 (Xie et al., 2016), VGG16 BN (Simonyan & Zisserman, 2014), Wide-ResNet-50, and Wide-ResNet-101 (Zagoruyko & Komodakis, 2016). For each architecture, we modify the last layer to match the same number of classes as the test dataset with Kaiming initialization (He et al., 2015a).
Number of Training Examples: To compute learning curves for CIFAR and Places365, we vary the number of training examples per class, partition the train set, and train one model per partition. For CIFAR100 (Krizhevsky, 2012), we use {25, 50, 100, 200, 400} training examples per class, and the number of models trained for each respectively is {16, 8, 4, 2, 1}. Similar to Hestness et al. (2017), we find training sizes smaller than 25 samples per class are strongly influenced by bounded error and deviate from our model. For Places365 dataset, we use {25, 50, 100, 200, 400, 1600} training examples per class and {16, 8, 4, 3, 3, 1} models each. For other datasets (Fig. 6), we use {20%, 40%, 80%} of the full data and train {4, 2, 1}models each. We hold out 20% of data from the original training set for testing (a validation set could also be used if available) to discourage metafitting on the test set. For example, we hold out 100 samples per class from the original CIFAR100 training set and perform hyperparameter selection and training on the remaining 400 samples.
Pretraining: When pretraining is used, we initialize models with pretrained weights learned through supervised training on ImageNet or Places365, or MOCO self-supervised training on ImageNet (He et al., 2020). Otherwise, weights are randomly initialized with Kaiming initialization.
Data Augmentation: For CIFAR, we pad by 4 pixels and use a random 32× 32 crop (test without augmentation), and for Places365 we use random-sized crop (Szegedy et al., 2015) to 224×224
and random flipping (center crop 224×224 test time). For remaining datasets, we follow the preprocessing in Zhai et al. (2020) that produced the best results when training from scratch.
Linear vs. Fine-tuning: For “linear”, we only train the final classification layer, with the other weights frozen to initialized values. All weights are trained when “fine-tuning”.
4.2 LEARNING CURVE COMPARISONS
We plot the fitted learning curves and confidence bounds, with observed test errors as black points. The legend displays γ, error eN , and data reliance βN with N = 400. The x-axis is in scale n−0.5 (n in parentheses), but bestfitting γ is used in all cases.In all plots, n denotes number of samples per class except Fig. 6 where n is the total number of samples. A vertical bar indicates n = 1600, which we consider the limit of accurate extrapolation from curves fit to n ≤ 400 samples. All points are used for fitting, except in Fig. 9b n = 1600 is held out to test extrapolation.
Network architecture: Advances in CNN architectures have reduced number of parameters while also reducing error over the range of training sizes. On CIFAR100,
AlexNet has 61M parameters; VGG-16, 138M; ResNet-50, 26M; ResNeXt-50, 25M; and ResNet101, 45M. Fig. 1 shows that each major advance through ResNet reduces both data-reliance and e400, while ResNeXt appears to slightly reduce e400 without change to data-reliance.
Pretraining and fine-tuning: In Fig. 2 we see that, for linear classifiers, pretraining leads to a huge improvement in e400 though with a moderate increase in data-reliance. When fine-tuning, the pretraining greatly reduces datareliance β400 and also reduces e400. Pretraining clearly improves performance with smaller training sizes. However, we cannot draw conclusions about bias because extrapolated asymptotic error is not reliable, and the full story is complicated. On an object detection task, He et al. (2019) find that, with long learning schedules, randomly initialized networks approach the performance of pretrained networks (for the CNN backbone), even with finite data. Experiments by Zoph et al. (2020), also on object detection, show that pretraining can sometimes harm performance when strong data augmentation is used. Ko-
rnblith et al. (2019) show that fine-tuned pretrained models outperform randomly initialized models on many datasets, but the gap is often small and narrows as data size grows. All agree that pretraining is at least important for providing a warm start that greatly reduces the training time, but whether it introduces bias (i.e. asymptotic error) likely depends on the tasks, domains, and optimization settings.
Pretraining data sources: In Fig. 3, we compare pretraining strategies: random, supervised on ImageNet or Places365, and self-supervised on ImageNet (MOCO by He et al. (2020)). All initializations have similar extrapolated error at n = 1600, but different data-reliance. Self-supervised MOCO leads to lower e400 and β400 compared to Places365 pretraining. Supervised pretraining on ImageNet has the lowest e400 and β400. We suspect that the γ = −0.67 and higher extrapolated asymptotic error may be due to measurement noise and suboptimal hyperparameter selection.
Network depth, width, and ensembles: The classical view is that smaller datasets need simpler models to avoid overfitting. In Figs. 4a, 4b, we show that, not only do deeper networks have better potential at higher data sizes, their data reliance does not increase (nearly parallel and drops a little for fine-tuning), making deeper networks perfectly suitable for smaller datasets. For linear classifiers (Fig. 4b), the deeper networks provide better features, leading to consistent drop in e400. The small jump in data reliance between Resnet-34 and Resnet-50 may be due to the increased last layer input size from 512 to 2048 nodes. When increasing width, the fine-tuned networks (Fig. 4c) have reduced e400 without much change to data-reliance. With linear classifiers (Fig. 4d), increasing the width leads to little change or even increase in e400 with slight decrease in data-reliance. Rosenfeld et al. (2020) show that error can be modeled as a function of either training size, model size, or both. Modeling both jointly can provide additional capabilities such as selecting model size based on data size, but requires many more experiments to fit the curve.
An alternative to using a deeper or wider network is forming ensembles. In Figure 4e, we find that while an ensemble of six ResNet-18’s (each 11.7M parameters) improves over a single model, it has higher e400 and data-reliance than ResNet-101 (44.5M), Wide-ResNet-50 (68.9M), and WideResNet-101 (126.9M). Three ResNet-50’s (each 25.6M) underperforms Wide-ResNet-50 on e400 but outperforms for small amounts of data due to lower data reliance.
Data Augmentation: One may expect that data augmentation acts as a regularizer with reduced effect for large training sizes, or even possibly negative effect due to introducing bias. However, Fig. 5 shows that data augmentation on Places365 reduces error for all training sizes with little or no change to data-reliance when fine-tuning. e(n) with augmentation roughly equals e(1.8n) without it, supporting the view that augmentation acts as a multiplier on the value of an example. For the linear classifier, data augmentation has little apparent effect due to low data-reliance, but the results are still consistent with this multiplier.
Additional datasets: In Fig. 6, we verify that our learning curve model fits to multiple other datasets (chosen from natural tasks in Zhai et al. (2020)), comparing fine-tuned vs. linear with Resnet-18. For these plots only, n is the total number of samples. The γ values are estimated from data, but the prior has more effect here due to fewer error measurements.
We see fine-tuning consistently outperforms linear, though the difference is most dramatic for Sun397. Pretraining provides large benefits across datasets.

heading: 4.3 EVALUATION OF LEARNING CURVES MODEL AND FITTING
text: We validate our learning curve model using leave-one-size-out prediction error, for example, predicting empirical mean performance with 400 samples per class based on observing error from models trained on 25, 50, 100, and 200 samples. We consider various choices of weighting schemes (w’s in Eq. 2) and estimating different parameters in a general form of the learning curve given by e(n) = α+ηnγ +δn2γ . Note that setting δ = 0 yields the learning curve model described in Sec. 3.
Weighting Schemes. In the Fig. 7 table, we compare three weighting schemes across 16 classifiers: wij = 1 is unweighted; wij = 1/σ2i is weighted by estimated size-dependent standard deviation; wij = 1/(Fiσ 2 i ) makes the total weight for a given dataset size invariant to the number of folds. On average our proposed weighting performs best with high significance compared to unweighted. The p-value is paired t-test of difference of means calculated across all dataset sizes.
Model Choice. We consider other parameterizations that are special cases of e(n) = α+ηnγ+δn2γ . The table in Fig. 7 shows that the parameterization used for our experiments outperforms the others, in most cases with high significance, and achieves a very good fit with R2 of 0.998.
Model Stability. We test stability and sample requirements by repeatedly fitting curves to four resampled data points for a model (Resnet-18, no pretraining, fine-tuned, tested on Places365). Based on estimates of mean and standard deviation, one point each at n = {50, 100, 200, 400} is sampled and used to fit a curve, repeated 100 times. Parentheses in legend show standard deviation of estimates of eN , βN , and γ. Our preferred model extrapolates best to n = 1600 and n = 25 while retaining stable estimates of of eN and βN , but predicted asymptotic error α varies widely.
Appendix C shows similar estimates of eN and βN by fixing γ = −0.5 and fitting only α and η on the three largest sizes (typically n = {100, 200, 400}), indicating that a lightweight approach of training a few models can yield similar conclusions.

heading: 5 LIMITATIONS AND FUTURE WORK
text: Limitations: Our work in this paper is limited to classification loss, and our model does not account for small sample effects where chance performance is a major factor. Although our proposed eN and βN are stable under perturbations and different learning curve parameterizations, the asymptotic error α and exponent γ parameters of the learning curve are unstable, and our confidence interval does not account for γ variance. Unstable α means that little can be concluded about asymptotic performance, though eN −βN can stand in as a measure of large-data performance. Unstable γ may mean that conclusions are subject to the hyperparameter selection and optimization method.
Future work: Do the hyperparameters such as learning rate, schedule, and weight decay determine γ, or something else? It appears that γ < −0.5 is accompanied by high α and/or η. Should γ = −0.5 for a well-trained system? Answering these questions could lead to improved training and evaluation methodologies. It would also be interesting to investigate learning curve models for small training size, other losses and prediction types, more design parameters and interactions, and impact of imbalance in class distribution.
Appendix A offers extended discussion. Appendix B provides a guide to fitting, displaying, and using learning curves. Appendix C contains a table of learning curves for all of our experiments and compares eN and βN produced by two learning curve models.

heading: 6 CONCLUSION
text: We investigate learning curve models for analyzing classifier design decisions. We find an extended power law provides the best fit across many different architectures, datasets, and other design parameters. We propose to characterize error and data-reliance with eN and βN , which are stable under data perturbations and can be derived from different learning curve models. Our experiments lead to several interesting observations about impacts of pretraining, fine-tuning, data augmentation, depth, width, and ensembles. We anticipate learning curves can further inform training methodology, continual learning, and representation learning, among other problems, and hope to see learning curves become part of a standard classification evaluation.

heading: A EXTENDED DISCUSSION
text: Evaluation methodology is the foundation of research, impacting how we choose problems and rank solutions. Large train and test sets now serve as the fuel and crucible to refine machine learning methods. The current evaluation standard of using fixed i.i.d. train/test sets has supported many classification model improvements, but as machine learning broadens to continual learning, representation learning, long-tail learning, and so on, we need evaluation methods that better reflect the uncontrollable, unpredictable, and ever-changing world. By characterizing performance in terms
of error and data-reliance, we can provide a more complete understanding of model design and training size impact than single-point error. With that perspective, we discuss the limitations of our experiments and directions for future work.
• Cause and impact of γ: We speculate that γ is largely determined by hyperparameters and optimization rather than model design, so conclusions are conditioned on particular training parameters. This presents an opportunity to identify poor training regimes and improve them. Intuitively, one would expect that more negative γ values are better (i.e. γ = −1 preferable to γ = −0.5), since the curve is curve O(nγ), but we find the highmagnitude γ tends to come with high asymptotic error, indicating that the efficiency comes at cost of over-commitment to initial conditions. We speculate (but with some disagreement among authors) that γ = −0.5 is an indication of a well-trained curve and will generally outperform curves with higher or lower γ, given the same classification model. It would be interesting to examine the impact of hyperparameter selection and optimization method on γ.
• Small training sets: Error is bounded and, with small training sets, classifier performance may be modeled as transitioning from random guess to informed prediction, as shown by Rosenfeld et al. (2020). We do not model performance with very small training size, partly to keep our model simple, partly because small training performance can be easily measured empirically, and partly because performance with small training size is highly variable depending on the sample. However, studying performance with small training sizes could be interesting, particularly to determine whether design decisions have an impact at the small size that is not apparent at larger sizes.
• Losses and Prediction types: We analyze multiclass classification error, but the same analysis could likely be extended to other losses and prediction types. For example, Kaplan et al. (2020) analyze learning manifolds of cross-entropy loss, which is unbounded, of language model transformers. Problems like object detection or grounding sometimes have relatively complex evaluation measures, such as average precision after accounting for localization and label accuracy, but test evaluation of the same losses used for training should still apply. Sun et al. (2017) show an approximately log-linear behavior between mean intersection of union semantic segmentation error as a function of number of training samples.
• More design parameters and interactions: The interaction between data scale, model scale, and performance is well-explored by Kaplan et al. (2020) and Rosenfeld et al. (2020), but it could also be interesting to explore interactions, e.g. between class of architecture (e.g. VGG, ResNet, EfficientNet (Tan & Le, 2019)) and some design parameters, to see how ideas such as skip-connections, residual layers and creating bottlenecks influence performance. More extensive evaluation of data augmentation, representation learning, optimization and regularization methods would also be interesting.
• Unbalanced class distributions: In most of our experiments, we use equal number of samples per class. Further experimentation is required to determine whether class imbalance impacts the form of the learning curve.

heading: B USER’S GUIDE TO LEARNING CURVES
text: B.1 USES FOR LEARNING CURVES
• Comparison: When comparing two learners, measuring the error and data-reliance provides a better understanding of the differences than evaluating single-point error. We compare curves with eN and βN , rather than directly using the curve parameters, because they are more stable under data perturbations and do not depend on the parameterization, instead corresponding to error and rate of change about n = N . eN −βN can be used as a measure of large-sample performance.
• Performance extrapolation: A 10x increase in training data can require a large investment, sometimes millions of dollars. Learning curves can predict how much performance will improve with the additional data to judge whether the investment is worthwhile.
• Model selection: When much training data is available, architecture, hyperparameters, and losses can be designed and selected using a small subset of the data to minimize the extrapolated error of the full training set size. Higher-parameter models such as in Kaplan et al. (2020) and Rosenfeld et al. (2020) may be more useful as a mechanism to simultaneously select scale parameters and extrapolate performance, though fitting those models is much more computationally expensive due to the requirement of sampling error/loss at multiple scales and data sizes. • Hyperparameter validation: A poor fitting learning curve (or one with γ far from −0.5)
is an indication of poor choice of hyperparameters, as pointed out by Hestness et al. (2017).
B.2 ESTIMATING AND DISPLAYING LEARNING CURVES
Use validation set: We recommend computing learning curves on a validation set, rather than a test set, according to best practice of performing a single evaluation on the test set for the final version of the algorithm. All of our experiments are on a validation set, which is carved from the official training set if necessary.
Generate at least four data points: In most of our experiments on CIFAR100, we train a 31 models: 1 on 400 images, 2 on 200 images, 4 on 100 images, 8 on 50 images, and 16 on 25 images. Each trained model provides one data point, the average validation error. In each case, the training data is partitioned so that the image sets within the same size are non-overlapping. Training multiple models at each size enables estimating the standard deviation for performing weighted least squares and producing confidence bounds. However, our experiments indicate that learning curves are highly stable, so a minimal experiment of training four models on the full, half, quarter, and eighth-size training set may be sufficient as part of a standard evaluation. See Fig. 8 It may be necessary to train more models if attempting to distinguish fine differences.
Set hyperparameters: The learning rate and learning schedule are key parameters to be set. We have not experimented with changes to weight decay, momentum, or other hyperparameters.
Fit learning curves: If more than one data point is available for the same training size, the standard deviation can be estimated. As described in Sec. 3, we recommend fitting a model of σ2i = σ 2 0 + σ̂2/n, where σ20 . σ 2 0 is the variance due to randomness in initialization and optimization. The fitting is not highly sensitive to this parameter, so we recommend setting σ20 = 0.01 and fitting σ̂ to observations, since estimating both from experiments to generate a single learning curve introduces high variance and instability.
Display learning curves or parameters: As in this paper, learning curves can be plotted linearly with the x-axis as n−0.5 and the y-axis as error. We choose this rather than log-linear because it reveals prediction of asymptotic error and yields a linear plot when γ = −0.5. Since space is often a premium, the learning curve parameters can be displayed instead, as illustrated in Table 3.

heading: C TABLE OF LEARNING CURVES
text: Table 4 shows experimental settings and fit parameters for learning curves under two parameterizations. We can see that similar e400 and β400 values are obtained when fixing γ = −0.5 and fitting to errors with only three training sizes (RMS difference in e400 and γ400 are 0.42 and 0.95, respectively). This means that learning curves can be fit and compared without training a large number of additional models.

heading: D DRAFT ADDITIONAL CHANGES TO INCLUDE IN FINAL VERSION
text: This section contains preliminary results and text requested by reviewers that will be carefully integrated into the main document in final revision.
D.1 OPTIMIZATION EXPERIMENTS
In Fig. 9, we show results on Cifar10 when training ResNet-18 using four different optimization methods: Ranger (Wright, 2019), Adam (Kingma & Ba, 2015), stochastic gradient descent (SGD) w/ momentum, and SGD w/o momentum. Similarly to our experiments with Cifar100, we use 80% of the standard training set for training and validation (4000 examples per class) and the remaining 20% for testing. With pretraining, all methods perform similarly, but when training from scratch (no pretraining), Ranger outperforms with lower e4000 and β4000. SGD without momentum performs the worst and is least consistent across folds.
D.2 TEXT FOR SECTION 2.2
However, if the classifier parameters are functions of n, then γ may deviate from−0.5. For example, Tsybakov (2008) shows that a kernel density estimator (KDE) with fixed bandwidth h has MSE bounded by O( 1nh ), but when the bandwidth is set as a function of n to minimize MSE, the bound becomes O(n− 2β
2β+1 ) where β is the kernel order. In our experiments, all aspects of our model are fixed across training size when estimating one learning curve, except learning schedule, but it should be noted that error bounds and likely the learning curve parameters depend on both the classifier form and which parameters vary with n.
D.3 OTHER PLANNED IMPROVEMENTS
• Experiments to include WRN-28-10 (or similarly effective Wide ResNet model) on Cifar100 to show that learning curve methodology applies and experimental findings hold for high-performing models
• Discussion to clarify that experiments serve to exemplify use of learning curves and make interesting observations, but more extensive study of each design parameter is warranted. Also discuss any other concerns/limitations raised by reviewers.

Review summaries:
This paper advocates for studying the effect of design choices in deep learning via their effect on entire *learning curves* (test error vs num samples N), as opposed to their effect only for a fixed N. This is a valid and important message, and it is indeed an aspect that is often overlooked in certain domains. However, although this paper addresses an important issue, there are methodological concerns (described below) which prevent me from recommending acceptance. In summary, the paper oversimplifies certain important aspects in both the setup and the experiments.

In this paper, the authors first propose a simple weighted least squares method to compute the "learning curve" (error plotted against dataset size) , where error is modelled with the form error = alpha - eta*n^gamma, for parameters alpha, eta, gamma. Gamma is taken to be - 0.5, while alpha, eta are estimated from the data. This also allows an estimate of "data reliance", in essence the slope of error wrt dataset size, computing how much error decrease is dependent on dataset size.