Learning Curves for Analysis of Deep Networks
Authors: Derek Hoiem
Tanmay Gupta
Zhizhong Li
Michal M Shlapentokh-Rothman
Venue:  (2021)
Keywords: learning curve
deep network
analysis
asymptotic error
learning efficiency
power law

Abstract: A learning curve models a classifier’s test error as a function of the number of training samples. Prior works show that learning curves can be used to select model parameters and extrapolate performance. We investigate how to use learning curves to analyze the impact of design choices, such as pre-training, architecture, and data augmentation. We propose a method to robustly estimate learning curves, abstract their parameters into error and data-reliance, and evaluate the effectiveness of different parameterizations. We also provide several interesting observations based on learning curves for a variety of image classification models.

Under review as a conference paper at ICLR 2021
LEARNING CURVES FOR ANALYSIS OF DEEP NET- WORKS
Anonymous authors Paper under double-blind review
ABSTRACT
A learning curve models a classifier's error function of the number of training samples. Prior works show that learning curves can be used to select model parameters and extrapolate performance. We investigate how to learn- ing curves to analyze the impact of design choices, such as pretraining, architec- ture, and data augmentation. We propose method to robustly estimate learning curves, abstract their parameters into enror and data-reliance, and evaluate the effectiveness of different parameterizations. We also provide several interesting observations based on learning curves for variety of image classification models.
INTRODUCTION
What gets measured gets optimized. We need better measures of learning ability to design better classifiers and predict the payoff of collecting more data. Currently, classifiers are evaluated and compared by measuring performance on one more datasets according to fixed train/test split. Ablation studies help evaluate the impact of design decisions. However, one of the most important characteristics of classifier, how it performs with varying numbers of training samples, is rarely measured or modeled.
In this paper, we refine the idea of learning curves that model error as a function of training set size.
Learning curves were introduced nearly thirty years (e.g. boCets et al. (1993) to accelerate model selection of deep networks. Recent works have demonstrated the predictability of performance
improvements with more data (Hestness et al. 2017: Johnson & Nguyen 2017 Kaplan et al.. 2020
Rosenfeld et al., 2020) or more network parameters (Kaplan et al. 2020 Rosenfeld et al. 2020).
But such studies have typically required large-scale experiments that are outside the computational
budgets of many research groups, and their purpose is extrapolation rather than validating design choices. We find that a generalized power law function provides the best learning curve fit, while model linear in 0.5 where is the number of training samples (or "training size"), provides good local approximation. We abstract the curve into two key parameters: eN and 8x. ex is error at n N, and 8x is measure of data-reliance, revealing how much classifier's error will change the training set size changes. Learning curves provide valuable insights that cannot be obtained by single-point comparisons of performance. Our aim is to promote the use of learning curves as part of standard learning system evaluation.
Our key contributions:
Investigate how to best model, estimate, characterize, and display learning curves for use in classifier analysis
Use learning curves to analyze impact on error and data-reliance due network architec- ture, depth, width, fine-tuning, data augmentation, and pretraining
Table shows validated and rejected popular beliefs that single-poin comparisons often overlook. In the following sections, we investigate how model learning curves (Sec. 2). how estimate them (Sec.B) 3. and what they can tell us about the impact of design decisions (Sec. 4h, with discussion of limitations and future work in Sec.

---

Under review as a conference paper at ICLR 2021
Popular
pucs
guass
ligunes
Prestriiiing on similar domuine naiely alwzys helps compueed in training scratch.
Pre-training, even similas dousins, introdues his that would harm formance with large enough training lan inperissed truiniig pertocms better than supervised gre caining small dataees.
Finsstaming the cuiire actwokk (vs. just the clasiliation layct) is cnly helptul if the taiiing set lage.
lncrasiin sctvork depri, Increasing nenvork, depti, when fine tuning. more helpful for larger trining sees than smaller ones. lncrasing nenwork deprh, if the backbone frozn, is more heiptul for smaller training sets than largr ones. Increasing depth Date ougnemationn is ruggly oquivalent to using a on times larper training set for some
line
performance small training
ovedly complex model.
Bh
Hb
improves more than ensembles smaller networks with
of parameters.
Table I: Deep learning quiz! We encourage our readers to judge each claim T (true) F (false), and then compare to our guesses and results. In the results column, "T" means the experiments are consistent with the belief, "FF" for inconsistent, and "?" for hard say.
2 MODELING LEARNING CURVES
The learning curve measures test error Ctest as a function of the number of training samples n for given classification model and learning method. Previous empirical observations suggest functional form Stest (n) = a + yn²,, with bias-variance trade-off and generalization theories typi- cally indicating 7 0.... We summarize what bias variance trade-off and generalization theories (Sec. 1. and empirical studies (Sec. 2.2) can tell us about learning curves, and describe our pro- posed abstraction in Sec. 2.3
2.1
TRADE-OFF AND GENERALIZATION THEORY
The bias-variance trade-off is an intuitive and theoretical way to think about generalization. The "bias" is error due inability of the classifier to encode the optimal decision function, and the "variance" is error due to variations in predictions due to limited availability of training samples for parameter estimation. This is called trade-off because classifier with more parameters tends to have less bias but higher variance. Geman et al. (1992) decompose mean squared regression error into bias and variance and explore the implications for neural networks, leading to the conclusion that "identifying the right preconditions is the substantial problem in neural modeling". This con- clusion foreshadows the importance of pretraining, though Geman et al. thought the preconditions must be built in rather than learned. Domingos (000) extends the analysis to classification. Theo- retically, the mean squared where "noise" is irreducible error due to non-unique mapping from inputs to labels, and variance can be modeled as var(n) o² n for n training samples.
(MSE) be modeled as etea (n) bias? + noise + var(n).
The 70.50.5 term appears throughout machine learning generalization theory. For example, the bounds based on hypothesis VC-dimension (Vapnik & Chervonenkis 1971) and Rademacher Com- plexity (Gnecco& & Sanguineti 2008) are both O(en -u.s) where e depends on the complexity of the classification model. More recent work also follows this form. We give some examples of bounds in Table 2 without describing all of the parameters because the point is that the test error bounds vary with training size function of n -0.5 for all approaches.
Table 2: Generalization bound examples: The bounds each predict generalization error increasing function of n UU.5 Note: variable notation is consistent only within each line, except n.
Work
Key Variables
Bound
Neyshabur al.
2018
network depth
08 Ba B In(dhjm ||; 3 TWa3T ///)/))))))
2017
spectral complexity Rg compressibility to parameters with discrete values based on analysis of stability with margin
Mllla log(max, hy) + +n -..)))
zim o (n 0.5 loggr) o (n-03/7)
Arora et al.
2018
Bousquet & Elssceeff
2002
2

---

Under review as a conference paper at ICLR 2021
2.2 EMPIRICAL STUDIES
Some recent empirical studies (e.g. Sun et al. (2017) claim a log-linear relationship between error and training size, but this holds only when asymptotic enror is zero. Heesteess al. (2017) model error as étest (n) = a + ynx? but often find 7 much smaller in magnitude than - 0.55 and suggest that poor fits indicate need for better hyperparameter tuning. This raises interesting point that sample efficiency depends both on the classification model and on the efficacy of the optimization algorithm and parameters. Johnson
Johnnooo & Nguyen (017) also find better fit with this extended power law model than by restricting = -0.5 or a = O. We find that, by selecting the learning rate through validation one training size and using the Ranger optimizer (Wright 2019). we can achieve a good approximate fit with y = -0.5 and best fit with 0.3 < < 0.7.
In the language domain, learning curves are used in fascinating study by Kaplan et al. (2020). For natural language transformers, they show that power law relationship between logistic loss, model size, compute time, and dataset size is maintained if (and only if) each is increased in tandem. We draw some similar conclusions to their study, such as that increasing model size tends to improve performance especially for small training sets (which surprised us). However, the studies are largely complementary, as we study convolutional nets in computer vision, classification error (instead of logistic loss), and broader range of design choices such effects across depth, width, data aug- mentation, pretraining source, architecture, and dataset. Also related, Rosenfeld et al. (2020) model error function of both training size and number of model parameters with five- wwe-parameter function that accounts for training size, model parameter size, and chance performance.
A key difference in work is that we focus how best draw insights about design choices from learning curves, rather than on extrapolation. As such, we propose methods to estimate learning curves and their variance from relatively small number of trained models.
2.3 PROPOSED CHARACTERIZATION OF LEARNING CURVES FOR EVALUATION
A classifier's performance can be characterized in terms of its error and data-reliance, or how quickly the error changes with training size n. With e(n) = ynn?, we find that 7 = -0.5 provides good local approximation but that fitting significantly improves leave-one-size-out RMS error and extrapolation accuracy, we detail in Sec. 4 However, a, 7,, and 7 cannot be meaningfully compared across curves because the parameters have high covariance with small data perturbations, and comparing r) values is not meaningful unless y is fixed and vice-versa.
We propose to report error and sensitivity to training size in a way that can be derived from various learning curve models and is insensitive to data perturbations. The curve is characterized by error eN a + 7yN77 and data-reliance 3N at N, and we typically choose N as the full dataset size. Noting that most learning curves locally well approximated by model linear in n -0.5 we compute data-reliance as 3N = --555 de
0.r n=N N7. When the error is plotted against -U.5 NN is the slope at N scaled by N-5.5 where the scaling was chosen to make the practical implications of By more intuitive. This yields a simple predictor for error when changing training size by a factor of d:
e(d N)= eN + (/2-1) BN-
(1)
Thus, by this linearized estimate, asymptotic error is ex - BN. a 4-fold increase in data (e.g. 400 + 1600) reduces error by 0.53r, and using only one quarter of the dataset (e.g. 400 100) increases the error by 3y. For two models with similar ex, the one with larger BN would outperform with more data but underperform with less. Note that (eN, BN, 7) is a complete re-parameterization of the extended power law, with 7 + 0.5 indicating the curvature in n -0.5 scale.
3 ESTIMATING LEARNING CURVES
We now describe the method for estimating the learning curve from error measurements with confi- dence bounds the estimate. Let eij denote the random variable corresponding to test error when the model is trained on the 5th fold of n, samples (either per class or in total). We assume fe are i.d according to N ((us of). We want to estimate learning curve parameters a (asymptotic er- ror), J/. and 7, such that eij-
mm, + Gij where 6ij AO of) and Pij RB(es] = /4-
3

---

Under review as a conference paper at ICLR 2021
Sections 3.1 and 3.2 describe how to estimate mean and variance of and r7 for given % and Sec. 3.3 describes our approach for estimating ?-
3.1
WEIGHTED LEAST SQUARES FORMULATION
We estimate learning eurve parameters {a, ni by optimizing a weighted least squares objective:
$
P.
(()) min 0.7] (=1
ΣΣ
Wii (eij - a - mm²))
where w,j = =/(Foof)) F; is the number of models trained with data size n, and is used to normalize the weight so that the total weight for observations from each training size does not depend on F-- The factor of of accounts for the variance of Eij. Assuming constant of and removing the Fj factor would yield unweighted least squares.
The variance of the estimate of of from F; samples is 20 / Fú, which lead over- under- weighting data for particular i if F; is small. Recall that each sample eij requires training an entire model, so F; is always small in our experiments. We would expect the variance to have the form /n,, - where of is the variance due to random initialization and optimization and 8² //n is the variance due to randomness in selecting n, samples. Indeed, by averaging over the variance estimates for many different network models on the CIFAR- (Krizhessky 2012) dataset, we find good fit with of 0.2. This enables a given learning curve as a least squares fit and also upper-bounds Wij <= even two models happen to have the same error. This attention Wij may seem fussy, but without such that the learning curve often fails to account sufficiently for all the data in some cases.
of = 60"
estimate single o² parameter from all samples in
find
3.2 SOLVING FOR LEARNING CURVE MEAN AND VARIANCE
Concatenating errors across dataset sizes (indexed by i) and folds results in an error vector of dimension = D F.. For each (1..... D}. e[d] is an observation of error dataset size n4, that follows N (p)) with ia mapping d to the corresponding i.
The weighted least squares problem can be formulated as solving system of linear equations de- noted by / = 11/2AAB where W E DDD a diagonal matrix of weights Was = wd. A IR Dx2 is a matrix with A(d, :] = [1 m3l. and 0 = [a 77 are the parameters of the learn- ing curve, treating ? as fixed for now. The estimator for the learning curve then given by ( +)+
(W
= Me, where M E 2x and + is pseudo-inverse operator.
We compute a mean curve using
0
0 MEje] Mμ
(3)
where μ E IR with [d[] = ممa computed by empirical estimate as Efn Fz./F.
The covariance of the estimator is given by
Σg = M
(4)
where Σg 2 and Ee RDD is the covariance of e, where Eg [d[d,, d2] = o2
if id, = ids
and otherwise. We compute our empirical estimate of of as described in Sec. 3.1
Since the curve is given by e(n) = [1 n² 0, the mean curve can be computed as e (n) [1 n2 0 + 17n".
(5)
The 95% bounds at any can be computed as Z(n) + 1.96 x ((n) with
1 n²
22(m) = n n
(6)
where à and vj are the empirical estimates of a and y,. These confidence bounds reflect the variance in error measurements, assuming the parameterization is capable of fitting the true mean.
4

---

Under review as a conference paper at ICLR 2021
3.3 ESTIMATING y
We search for that minimizes the weighted least squares objective with an Ll-prior that slightly encourages values close 0.5. Specifically, we solve
min ((7) A/y + 0.5| 7E(-1,0)
by searching over 7 E {-0.99, 00.01) with for our experiments.
EXPERIMENTS
We describe our implementation details in Sec. 4.1 apply learning curves to gain insights about enror and data-reliance in Sec. 4.2 and validate our choice of learning curve parameterization and fitting weights used the least squares objective Sec. 4.3
4.1 IMPLEMENTATION DETAILS
We use Pytorch- Lighhtin Falcon 2019 for our implementation with various architectures, weight initializations, data augmentation, and linear or fine-tuning optimization.
Training: We train models with images of size 32 x2 for CIFAR (Krizhevsky 2012) and 224 x 224 for Places365 (Zhou et al. 2017) with batch size of 64 (except for Wide-ResNett 01 and Wide- ResNeXt101, where we use batch size of 32 and performed one optimizer step every two batches). For each experiment setting, we conduct a learning rate search on a subset of the training data and choose the learning rate with the highest validation accuracy, and use it for all other subsets. We determine each fold's training schedule on a mini-train/miii-val split of 2:1 on the train set. Each time the mini-val error stops decreasing for some epochs ("patience"),, epoch and decrease the learning rate to 10%, and we perform this twice. Then we use this optimal mini-train learning rate schedule and ending epoch to train on the whole fold. The patience is x 1/Vn, and is 5 at the n 400 samples/class for CIFAR 100/Places365 and 15 at the largest training size for other smaller datasets. We use weight decay value of 0.0001. We use the Ranger optimizer (Wright (2019), which combines Rectified Adam (Liu et al., 2019), Look Ahead (Zhang 2019), and Gradient Centralization (Yong et al. 2020)). In early experiments, we found Ranger to lead to lower error and to reduce sensitivity of hyperparameters, compared to vanilla SGD or Adam (Kingma & Ba 2015).
revert to the best
Backbone Architecture: We use the default Pytorch implementations of all of the following architectures: AlexNet (Krizhevsky al. 2012). ResNet-18, ResNet-50, ResNet-101 (He et al. 20155), Res5eX ResNeXt- (Xie et 20116)))) VGG16 (Simonyan & Zisserman 20144). Wide-ResNet-50, and Wide-ResNeet-OI (Zagoruyko Komodakis (2016). For each architecture, we modify the last layer to match the same number of classes as the test dataset with Kaiming initialization (He al. 2015a).
Number of Training Examples: To compute learning curves for CIFAR and Places365, we vary the number of training examples per class, partition the train set, and train one model per partition. For CIFARI00 (Krizhevsky) 2012), we use (25, 50, 100, 200, 400} training examples per class, and the number of models trained for each respectively is {16, 8, 4, 2, 1). Similar Hestness et al. (017). we find training sizes smaller than 25 samples per class are strongly influenced by bounded error and deviate from our model. For Places365 dataset, we use (25, 50, 100, 200, 400, 1600) training examples per class and {16,8, 8, 3, 1} models each. For other datasets (Fig. 6). we use (20%, 40%, 80%) of the full data and train (4, 1} models each. We hold out 20% of data from the original training set for testing (a validation set could also be used if available) to discourage meta- fitting on the test set. For example, we hold out 100 samples per class from the original CIFAR100 training set and perform hy hyperparameter selection and training on the remaining 400 samples.
When pretraining is used, we initialize models with pretrained weights learned through supervised training on ImageNet or Places365, or MOCO self-supervised training on Ima- geNet (He et al. 2020). Otherwise, weights are randomly initialized with Kaiming initialization.
Pretraining:
Data Augmentation: For CIFAR, we pad by 4 pixels and use a random 32 x 32 crop (test without augmentation), and for Places365 we random-sized crop (Szegedy al. 2015 to 224x224
5

---

Under review as a conference paper at ICLR 2021
and random flipping (center crop 224 test time). For remaining datasets, we follow the pre- processing in Zhai et al. (2020) that produced the best results when training from scratch.
Linear vs. Fine-tuning: For "linear", we only train the final classification layer, with the other weights frozen to initialized values. All weights are trained when "fine-tuning".
4.2 LEARNING CURVE COMPARISONS
We plot the fitted learning curves and confidence bounds, with observed test errors as black points. The legend dis- plays y, eπrrr eN, and data reliance BN with N 400. The x-axis is in scale n -0.5 (n in parentheses), but best- fitting y is used in all cases.In all plots, n denotes number of samples per class except Fig. 6 where n is the total number of samples. A vertical bar indicates = 1600, which we consider the limit of accurate extrapolation from curves fit to n 400 samples. All points are used for fitting, except in Fig. 9b n = 1600 is held out to test extrapolation.
deeMe: 2508 deol- -632
VGG 103-) true BlA Jun 9.20 viaa)
Pachee tram 5623 -422 y ---mmm
Baitato (02220
EMU
14]
1900
(182
(40)
06
Network architecture: Advances in CNN architectures Figure 1: Architecture (w/ finetuning) have reduced number of parameters while also reducing error over the range of training sizes. On CIFARIO0, AlexNet has 61M parameters; VGG-16, 138M; ResNet-50, 26M; ResNeXt-50, 25M; and ResNet- 101, 45M. Fig. [ shows that each major advance through ResNet reduces both data-reliance and e400. while ResNeXt appears to slightly reduce e400 without change to data-reliance.
<(n) +---n
<(n)
100
25.20
-0.5
(aa -651
No Prate Fixtare (eam 27.5
0.41)
Firstuss (taw -T,00: 12.8s
Poek: Unee frem 32400 Pas 8881 7
0.99)
Prite: Livoe frew 2151; Saw - 4,00: 6.38)
Paes: Freesre tras 18.6 Als 75
-0.7)
Pretr: Fertase (eae $4.) Aao - 7.50)
0.35
(409)
(100)
(99)
(25)
(N)
(400)
(100)
(25)
[19]
(a) Transfer: Imagenet Cifar100
(b) Transfer: Imagenet to Places365
Figure 2: Pretraining and fine-tuning with Res-Net-18
Pretraining and fine-tuning: In Fig. 2 we see that, for linear classifiers, pretraining leads to a huge improvement in 6100 though with moderate increase in data-reliance. When fine-tuning, the pretraining greatly reduces data- reliance $0on and also reduces e400- Pretraining clearly improves performance with smaller training sizes. How- ever, cannot draw conclusions about bias because ex- trapolated asymptotic error is not reliable, and the full story is complicated. On object detection task, He eet al. (2019) find that, with long learning schedules, ran- domly initialized networks approach the performance of pretrained networks (for the CNN backbone), even with finite data. Experiments by Zoph et al. (2020), also ob- ject detection, show that pretraining can sometimes harm performance when strong data augmentation is used. Ko-
1330 Thau 2015 125a.y
035
14]
Figure 3: Pretraining sources (test 10f)

---

Under review as a conference paper at ICLR 2021
mbblth et al. (2019) show that fine-tuned pretrained models outperform randomly initialized models on many datasets, but the gap is often small and narrows as data size grows. All agree that pretraining is at least important for providing warm start that greatly reduces the training time, but whether it introduces bias (i.c. asymptotic error) likely depends on the tasks, domains, and optimization settings.
Pretraining data sources: In Fig. 3 we compare pretraining strategies: random, supervised on ImageNet Places365, and self-supervised on ImageNet (MOCO by He et al. (2020)). All initial- izations have similar extrapolated error at n = 1600, but different data-reliance. Self-supervised MOCO leads to lower e400 and $400 compared to Places365 pretraining. Supervised pretraining on ImageNet has the lowest 6400 and 8200- We suspect that the 7 = -0.67 and higher extrapolated asymptotic error may be due to measurement noise and suboptimal hyperparameter selection.
v(t) +-
Rejeet 15.06 Aln -057)
Reizae 50).
Roiret 16-77 434 -0730
Rejeet
92
-30
290
(462)
(103)
(40
(400)
(200)
[45]
[23]
(a) Depth: finctune
(b) Depth: linear
Emsende) Aua anlyy oll rove 11th Juach
150
3.5
540
(c) Width: fineture
(d) Width: lincar
(0) Ensemble: finctune
Figure 4: Depth, width, and ensembles on Cifar 100.
Network depth, width, and ensembles: The classical view is that smaller datasets need simpler models to avoid overfitting. In Figs. 4a| 4b we show that, not only do deeper networks have better potential at higher data sizes, their data reliance does not increase (nearly parallel and drops little for fine-tuning), making deeper networks perfectly suitable for smaller datasets. For linear classifiers (Fig. (4b). the deeper networks provide better features, leading consistent drop in e400- The small jump in data reliance between Resnet-34 and Resnet-50 may be due to the increased last layer input size from 512 to 2048 nodes. When increasing width, the fine-tuned networks (Fig. 44c have reduced 6400 without much change to data-reliance. With linear classifiers (Fig. 4db, increasing the width leads to little change even increase in 6400 with slight decrease in data-reliance. |Rosenelld et al.) (2020) show that error can be modeled as a function of either training size, model size, or both. Modeling both jointly can provide additional capabilities such as selecting model size based on data size, but requires many more experiments to fit the curve.
An alternative to using deeper or wider network is forming ensembles. In Figure 4el we find that while an ensemble six ResNet-18's (each 11.7M parameters) improves over a single model, has higher e400 and data-reliance than ResNet-101 (44.5M), Wide-ResNet-500 (68.9M), and Wide- ResNet- 101 (126.9M). Three ResNer-50's (each 25.6M) underperforms Nede-ResNett50 on 6400 but outperforms for small amounts of data due lower data reliance.
7

---

Under review as a conference paper at ICLR 2021
100
(a) w/o Pretrain; Finetune
(b) Pretrain; Finetune
(c) Pretrain; Linear
Figure 5: Data augmentation on Places365.
Data Augmentation: One may expect that data augmentation acts a regularizer with reduced effect for large training sizes, or even possibly negative effect due to introducing bias. However, Places365 reduces error for all training sizes with little
Fig. shows that data augmentation change to data-reliance when fine-tuning. e(n) with augmentation roughly equals e(1.8n) without it, supporting the view that augmentation acts multiplier the value of an example. For the linear classifier, data augmentation has little apparent effect due to low data-reliance, but the results still consistent with this multiplier.
Additional datasets: In Fig. 6 we verify that our learning curve model fits to multiple other datasets (chosen from natural tasks in Zhai et al.] (2020)). comparing fine-tuned vs. linear with Resnet-18. For these plots only, is the total number of samples. The y values are estimated from data, but the prior has more effect here due to fewer error measurements.
Cane
Saltt bet la
We see fine-tuning consistently outperforms linear, though the difference is most dramatic for Sun397. Pretraining provides large benefits across datasets.
(4)
(1000)
(200))
2010
Figure 6: Additional datasets
4.3 EVALUATION OF LEARNING CURVES MODEL AND FITTING
We validate our learning curve model using leave-one-size-our prediction error, for example, pre- dicting empirical mean performance with 400 samples per class based on observing error from models trained on 25, 50, 100, and 200 samples. We consider various choices of weighting schemes (w's in Eq. 2 and estimating different parameters in a general form of the learning curve given by e(n) a am 2n Note that setting 8 =0 yields the learning curve model described in Sec.
Weighting Schemes. In the Fig. ttalle, we compare three weighting schemes across 16 classifiers: Wij is unweighted; Way = T/a? is weighted by estimated size-dependent standard deviation; wij = 1/(Foo²)) makes the total weight for a given dataset size invariant to the number of folds. On average our proposed weighting performs best with high significance compared unweighted. The p-value is paired t-test of difference of means calculated across all dataset sizes.
Model Choice. We consider other parameterizations that special cases of e(n) a++7m 7 2 The table in Fig. shsows that the parameterization used for our experiments outperforms the others, in most cases with high significance, and achieves very good fit with R² of 0.998.
Model Stability. We test stability and sample requirements by repeatedly fitting curves to four resampled data points for model (Resnet-18, pretraining, fine-tuned, tested Places365). Based on estimates of mean and standard deviation, one point each at n = {50, 100, 200, 400} is sampled and used fit curve, repeated 100 times. Parentheses in legend show standard deviation of estimates of ex, Ix, and 7. Our preferred model extrapolates best to n = 1600 and n 25 while retaining stable estimates of of ex and BN, but predicted asymptotic error varies widely.
8

---

Under review as a conference paper at ICLR 2021
Appendix C shows similar estimates of eN and By by fixing Y -0.5 and fitting only and on the three largest sizes (typically n = {100, 200, 400)), indicating that a lightweight approach of training a few models can yield similar conclusions.
RMSF
Params
Weights
50
100
200
400
avg
p-value
0.998
0.999
2.38
0.83
0.69
0.54
1.08
1.10
0.06
0.998
2.66
0.86
0.79
0.50
1.26
1.21
0.008
0.4
0.988
0,
0.999
2.89
0.74
0.68
0.56
0.94
1.16
0.05
a,1), 6,7
0.999
3.46
0.74
0.70
0.59
1.00
1.30
0.02
Figure
Learning curve model and weights validation. See text for explanation.
5 LIMITATIONS AND FUTURE WORK
I Limitations: Our work in this paper is limited to classification loss, and our model does not account for small sample effects where chance performance is a major factor. Although our proposed en and By are stable under perturbations and different learning curve parameterizations, the asymptotic error a and exponent " parameters the learning curve are unstable, and our confidence interval does not account for variance. Unstable means that little be concluded about asymptotic performance, though e - 3N can stand in as a measure large-data performance. Unstable y may mean that conclusions are subject to the hyperparameter selection and optimization method.
Future work: Do the hyperparameters such as learning rate, schedule, and weight decay determine -0.5 is accompanied by high and/or Should y = -0.5 for well-trained system? Answering these questions could lead to improved training and evaluation methodologies. It would also be interesting to investigate learning curve models for small training size, other losses and prediction types, more design parameters and interactions, and impact of imbalance in class distribution.
7, or something else? It appears that
Appendix A offers extended discussion. Appendix B provides a guide fitting, displaying, and using learning curves. Appendix (ccetdiiiConnninss a table of learning curves for all of our experiments and compares en and 3N produced by two learning curve models.
6 CONCLUSION
We investigate learning curve models for analyzing classifier design decisions. We find extended power law provides the best fit across many different architectures, datasets, and other design param- eters. We propose to characterize error and data-reliance with eN and BN. which are stable under data perturbations and can be derived from different learning curve models. Our experiments lead to several interesting observations about impacts of pretraining, fine-tuning, data augmentation, depth, width, and ensembles. We anticipate learning curves can further inform training methodology, con- tinual learning, and representation learning, among other problems, and hope to see learning curves become part a standard classification evaluation.
REFERENCES
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for 2018. URL htttp://arxiv:
deep nets via compression approach. CoRR, org/ httg/bb/18000 05296
Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Specttally-norrmliized margin bounds for neural networks. CoRR, a 706.08498, 2017. URL. 08498

---

Under review as a conference paper at ICLR 2021
O. Bousquet and A. Elisseeff. Stability and generalization. Journal of Machine Learning Research, 2:499-526, 2002.
Corinna Cortes, L. D. Jackel, Sara A. Solla, Vladimir Vapnik, and John S. Denker. Learning curves: Asymptotic values and rate of convergence. In NIPS, 1993.
Pedro Domingos. A unified bias-variance decomposition. In ICML,
WA Falcon. Pytorch lightning. GitHub. Note: htt:://giiilaaccooo////otttiiits lightming, 3, 2019.
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural Computation, 4:1-58, 01 1992. doi: 10.1162/neco 1992.......9....11...
Giorgio Gnecco and Marcello Sanguineti. Approximation error bounds via rademacher's complex- ity. 2(4), 2008.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Icarnnn for image recog- nition. CoRR, abs/1512.03388 2015b. URL htLhttp::/ wttp:xxwww..///////.22....
Kaiming He, Ross Girshick, and Piotr Dollar. Rethinking imagenet pre-training. pp- 4917-49699 10 2019. doi: 10. 00.1109/CCV.2011.00088
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9733 2020.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianine- jad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. CoRR, 2017.
Mark Johnson and Dat Quoe Nguyen. How much data is enough? Predicting how accuracy varies with training data size. Technical report, 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
S. Korblithh J. Shlens, and Q. V. Le. Do better imagenet models transfer better? In CVPR, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097- 1105. Curran Associates. Inc.. 2012. URL
www.: hhaaaaaaw..lssii.aaaaaaoo..aeaea
pdf
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.0320 03265,, 2019.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A PAC-bayesian approach to spectrally-normaiizze margin bounds for neural networks. International Conference on Leam- ing Representations, 2018. httt:p::///oeeeeeeiieeeeee/////e
10

---

Under review as a conference paper at ICLR 2021
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In International Conference on Learning Representa- tions, 2020. URL
net/ http::/www......ru....aaaaaaraaa
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556. 2014. UtLhtt:: http://aaxxxv.ooaa/////// 15556
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In IEEE International Conference Computer Vision, pp. 943-852,220 2017.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanct, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Mingxing Tan and Quoc V Le. EfficientNet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.11946. 2019.
Alexandre B. Tsybakov. Introduction Nonparametric Estimation. Springer Publishing Company, Incorporated, 1st edition, 2008. ISBN 0387790519.
V. N. Vapnik and A. Ya. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications, 2800, 1971. doi: 10. 16025. URL /ltkk wttpa///.....///////////////////
Less Wright. New deep learning optimizer, ranger: Synergistic combination of radam + lookahead for the best of both. Github http:///ettblooooeeeee////////ee Optriiizeettit 08 2019.
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated resid- ual transformations for deep neural networks. CoRR, abs/1611.05453 2016. URL http:
611311
Hongwei Yong, Jianqiang Huang, Xiansheng Hua, and Lei Zhang. Gradient centralization: A new optimization technique for deep neural networks. ArXiv, abs/2004.01461, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. CoRR, abs/1605.07146, 2016.
URL
07146
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski. Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint 1rXiv:1910.0486: 2020.
Michael Ruogu Zhang, James Lucas, Geoffrey E. Hinton, and Jimmy Ba. Lookahead optimizer: k steps forward, step back. NeurlPS, 2019.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Barret Zoph, Golnaz Ghiasi, Tsung-Yi Yi Lin, Yin Cui, Hanxiao Liu, Ekin D. Cubuk, and Quoc V. Le. Rethinking pre-training and self-training, 2020.
A EXTENDED DISCUSSION
Evaluation methodology is the foundation of research, impacting how we choose problems and rank solutions. Large train and test sets now serve as the fuel and crucible to refine machine learning methods. The current evaluation standard of using fixed i.idd. train/test sets has supported many classification model improvements, but machine learning broadens continual learning, repre- sentation learning, long-tail learning, and so on, we need evaluation methods that better reflect the uncontrollable, unpredictable, and ever-changing world. By characterizing performance in terms
11

---

Under review as a conference paper at ICLR 2021
of error and data-reliance, training size impact than single-pone error. With that perspective, we discuss the limitations of our experiments and directions for future work.
provide more complete understanding of model design and
Cause and impact of y: We speculate that yy is largely determined by hyperparameters and optimization rather than model design, conclusions conditioned particular training parameters. This presents an opportunity to identify poor training regimes and improve them. Intuitively, one would expect that more negative Y values are better (i.e. Y preferable to = -0.5), since the curve is curve O(n7), ). but we find the high- magnitude 7 tends to come with high asymptotic error, indicating that the efficiency comes at cost of over-commitment to initial conditions. We speculate (but with some disagreement among authors) that y -0.5 is an indication of well-trained curve and will generally outperform curves with higher or lower y, given the same classification model. It would be interesting examine the impact of hyperparameter selection and optimization method
Error is bounded and, with small training sets, classifier performance may be modeled as transitioning from random guess to informed prediction, as shown by Rosenfeld et al. (2020). We do not model performance with very small training size, partly to keep our model simple, partly because small training performance can be easily mea- sured empirically, and partly because performance with small training size is highly vari- able depending on the sample. However, studying performance with small training sizes could be interesting, particularly determine whether design decisions have an impact at the small size that is not apparent at larger sizes.
Small training sets:
We analyze multiclass classification error, but the same analysis could likely be extended to other losses and prediction types. For example, Ka- plan et al. (2020) analyze learning manifolds of cross-entropy loss, which is unbounded, of language model transformers. Problems like object detection or grounding sometimes have relatively complex evaluation measures, such as average precision after accounting for localization and label accuracy, but test evaluation of the same losses used for training should still apply. [Sun et al. (2017) show an approximately log-linear behavior between mean intersection of union semantic segmentation error as a function of number of training samples.
Losses and Prediction types:
More design parameters and interactions: The interaction between data scale, model scale, and performance is well-explored by KKallan al. (2020) and Rosenfeld al. (2020), but it could also be interesting to explore interactions, e.g. between class of architecture (e.g. VGG, ResNet, EfficcentNet (Tan Ie (019g) and some design parameters, to see how ideas such as skip-connections, residual layers and creating bottlenecks influence per- formance. More extensive evaluation of data augmentation, representation learning, opti- mization and regularization methods would also be interesting.
Unbalanced class distributions: In most of our experiments, we use equal number of sam- ples per class. Further experimentation is required to determine whether class imbalance impacts the form of the learning curve.
B USER'S GUIDE TO LEARNING CURVES
B. USES FOR LEARNING CURVES
When comparing two learners, measuring the error and data-reliance pro- vides a better understanding of the differences than evaluating single-point error. We com- pare curves with ex and 88, rather than directly using the curve parameters, because they are more stable under data perturbations and do not depend on the parameterization, instead corresponding error and rate of change about N. ex By can be used measure of large-sample performance.
Comparison:
Performance extrapolation: A 10x increase in training data can require large invest- ment, sometimes millions of dollars. Learning curves can predict how much performance will improve with the additional data judge whether the investment worthwhile.
12

---

Under review as a conference paper at ICLR 2021
Model selection: When much training data is available, architecture, hyperparameters, and losses can be designed and selected using small subset of the data to minimize the extrap- olated error of the full training size. Higher-paramete (2020) and Rosenfeld et al. (2020) may more useful as a mechanism to simultancously select scale parameters and extrapolate performance, though fitting those models is much more computationally expensive due to the requirement sampling error/loss at multiple scales and data sizes.
models such as in Kaplan et al.
Hyperparameter validation: A poor fitting learning curve (or one with far from - 00.5) is an indication of poor choice of hyperparameters, as pointed out by Hestness al. (2017)
B.2 ESTIMATING AND DISPLAYING LEARNING CURVES
We recommend computing learning curves on validation set, rather than test set, according best practice of performing single evaluation on the test set for the final version of the algorithm. All of our experiments are on validation set, which is carved from the official training set if necessary.
Use validation set:
Generate at least four data points: In most of our experiments on CIFAR 100, we train a 31 models: on 400 images, 2 on 200 images, 4 on 100 images, 8 50 images, and 16 on 25 images. Each trained model provides one data point, the average validation error. In each case, the training data is partitioned that the image sets within the same size are non-overlapping. Training multiple models at each size enables estimating the standard deviation for performing weighted least squares and producing confidence bounds. However, our experiments indicate that learning curves are highly stable, so a minimal experiment of training four models on the full, half, quarter, and eighth-size training set may be sufficient as part of standard evaluation. See Fig. may be necessary to train more models if attempting to distinguish fine differences.
100
E11e
000
S: di 16 U.8. 8:33)
ovti (8.16,0 0.06)
(0.12) (12, bu)
0.25
(430)
1100)
[45)
(25)
(10)
Figure 8: Stability under sparse measurements: Sampled learning curves for Places365 fine-tuned without pretraining shown for four different learning curve parameterizations. In each case, means and standard deviations (shown by error bars) are estimated for n 50, n 100, n 200, 400, using all the data points shown white circles. Then, 100 times, sample one point each from a Guassian distribution and fit a learning curve to the four points. In parantheses, the legend shows the standard deviation of eN, 8N, and 7. Note that the parameterization of {a, 77, 7 extrapolates best to lower and higher data sizes while still producing stable estimates of e and BN- Asymptotic error, however, varies widely.
Set hyperparameters: have not experimented with changes weight decay, momentum, other hyperparameters.
The learning rate and learning schedule are key parameters to be set. We
Fit learning curves: If more than one data point is available for the same training size, the standard deviation be estimated. As described in Sec. 3 recommend fitting a model of o? of + 8²/n, where o- og is the variance due to randomness in initialization and optimization. The fitting is not highly sensitive to this parameter, observations, since estimating both from experiments to generate a single learning curve introduces high variance and instability.
recommend setting of 0.01 and fitting â to
13

---

Under review as a conference paper at ICLR 2021
Display learning curves parameters: As in this paper, learning curves can be plotted linearly with the x-axis as n 00.5 and the y-axis as error. We choose this rather than log-lineat because reveals prediction of asymptotic eтror and yields linear plot when y -0.5. Since space is often a premium, the learning curve parameters can be displayed instead, as illustrated in Table
Table 3: Results: modelı and modelz have similar percent test error when training on the full set. Fitting learning curve the validation set, we see that model2 has higher data-reliance, may outperform for larger training sets. This is a hypothetical example to illustrate use of learning curves in table.
EN
3N
model
25.3 %
4.6
-0.36
model2
25.2 %
8.4
-0.47
C TABLE OF LEARNING CURVES
Table 4 shows experimental settings and fit parameters for learning curves under two parameteriza- tions. We can see that similar 440 and Bamo values are obtained when fixing 7 -0.5 and fitting to errors with only three training sizes (RMS difference in evoo and Yauo are 0.42 and 0.95, respec- tively). This means that learning curves can be fit and compared without training large number of additional models.
Table 4: Experiment settings and parameters: We show the datasets, architectures, settings, and learning rate (set by mini-train/val) used to train and test our classifiers Next, we show the param- eters fit using the extended power law model e(n) = + 7 Next to that, we show the model resulting from setting 7 = 0.5 and fitting to only the three training sizes with highest n.
calenial poweer bee
Home paato
datout
Ena-tone?
data ang"
Irailate
No Pex: Liwer
CIFAR
Recett 18
SIK
Randon
No
Ves
0.
78:51
12011
76 20
1,30
75.06
26-12
18:36
1:31
No Frer: Peaaee
Raadom
You
3.00
18.25
11:11
336.13
Linar
CHNM
S1K
66.25
$2.42
12.64
5.11
Padto Freter:
CIFAR
Ro
11.1M
agchee
Yes
Yes
0.00
12:45
196.0) 19
JUS)
15:26
11.00
156/25
(8.9)
7.54
PISTRAIN JUH ACPS
Pros: Lawe
Pacss
Boast-
Raadom
01,99
13.100
0.50
33.15
11T.4
-0.36
$1,00
12.00
203.00
$7.57
13.15
Potr: Liany
Pares
Roce 18
187K
No
Yei
0.003
$5.41
$3.99
56.11
4116
56.11
16.95
$9.05
355
PRETRAIN PLACES MOC0000AMM
No
CHEAR
Roces
Roadonn
You
0.
543-46
0..5
17.66
30.05
9.18
153
33.70
Purt of Bragrau
CIEAR
Rocas
You
000
15.11
178.6
14.25
1.5
15.5
09.08
1.5
Past ee bragean vias XOCO
CIAAR
Reasi-50
25.60
Brajshhs 34DCD)
Yoa
Yo
00003
1477
112.00
-85
18.7%
11.21
1007
210
30.61
13.55
TIPPIIJ JIT
Bo
CIRAR
Roassi5
13:700
Yi
Yo
0.00:
12:45
194.1
-U37
13:26
1.25
11.39
13575
11.51
3.54
CIEAR
Rocat 90
Byr Wagghan ageleee
You
1511
166
16.55
135
13.5
18:00
110
DEPURLLINEA INEAK
S1K
65.26
4035
11.42
2135
Becren
CIEAR
Rocas 34
FIK
Bragehin
No
000
21.75
06.56
1/11
2011
06.55
10.00
140
CINR
Roazi-001
258K
N6
03003
4.21
15.56
5.17
3006
59.2
23.55
4.55
Recrer-0 50
CIEAR
Rocas-9) 90
Bragehin Nci
104
0000
15:11
17341
1607
16.55
135
13.5
0638
18:00
1.00
Wilchtee--552
Nomes 101
CIBAR
Ro
44.500
03003
1001
196.4
4062
1457
500
605
11122
1451
536
SeW Retren 101
CIEAR
Beagehan
106
00001
116.1
11:35
131
116.36
13.66
531
WIDTIL LINEAR
Ieagefee
112.0
SKW de Sexres00
CIDM
Vide Reean
20.%
No
56.05
35.77
24.5
4.5
Recken-011 101
CHEAR
Boc bot
Bragehon
No
2003
11:7
1621
14.00
4.7
Pacze
13.320
Roadon Ruadon
350
0.500
35.74
11834
-0.26
53.01
12.44
44.99
243.00
$7.57
13.15
Relees18 18 Mha
Paces
13.77
704
No
0.003
116.2
60.75
11.01
258-6
$3.24
Pacze
Boasi-18
InaadNei
07335
23.00
4000
5.73
0.500
504
183.00
Rearn 18 Traa- Aug
Plaks
Rocws 18
1876
Bragehin
No
Yee
0003
05.55
(001
130
16111
T6.55
19.95
1.55
Bomasi-l whw Dalaaaaa
Paace
Boxsh-1
18%
Ieaadhii
No
0.000
55,42
50:42
ABCHIFIICT RISS JNCCIPA
CIFAR
VCG--BB
135 AM
Iragchei
0003
-0.5
1911
1000
18:.1
19:13
3.06
10
15.11
119.11
Baxbban Sucto 48)
CIDAR
Baclalto 50113440
25102
Bragehon
1365
16.05
191
11.00
100.99
16.45
5.15
Rechher INI
CIEAR
Rocas mol
Bragehon
106
Yel
0.000
11:41
1067
11.95
1.0
5.05
117.22
1161
536
18
23.730
BraseNon
Yoa
-U57
19.36
1.3
11.27
19875
CITAR
0.50
176.36
15.01
136
Lolledd50
CIEAR
Rossi30 30
25
No
Yo
0.00
15.11
178.11
JU67
15:35
4.0
15.5
1936
19:0
4.50
14

---

Under review as a conference paper at ICLR 2021
D DRAFT ADDITIONAL CHANGES TO INCLUDE IN FINAL VERSION
This section contains preliminary results and text requested by reviewers that will be carefully inte- grated into the main document in final revision.
D.1 OPTIMIZATION EXPERIMENTS
<(6)
100
Tanger (rasa Fana
Ranger (ruaa 2.56 Piao
550 (n 630 Ape -0.5
5G0 (rake 7.7h
Haw 2555 -0335)
SG0
5G00
%
044
(2500)
(025)
(278)
(15)
(198)
(70]
(K)
(2540)
(625)
(278)
(157)
(100)
[90]
without Pretrain
(b) with ImageNet Pretrain
Figure 9: Optimization on Cifar10 with ResNet-18.
In Fig. 9 we show results Cl when training ResNet-18 using four different optimization methods: Ranger [Wright] (019), Adam (Kingma & Ba 2015), stochastic gradient descent (SGD) w/ momentum, and SGD w/o momentum. Similarly our experiments with Cifarl00 we use 80% of the standard training set for training and validation (4000 examples per class) and remaining 20% for testing. With pretraining, all methods perform similarly, but when training from scratch (no pretraining), Ranger outperforms with lower 64000 and 34000- SGD without momentum performs the worst and is least consistent across folds.
D.2 TEXT FOR SECTION 2.2
However, the classifier parameters are functions of n, then y may deviate from -0.5. For example, Tsybakov (2008) shows that kernel density estimator (KDE) with fixed bandwidth h has MSE bounded by O wh ), but when the bandwidth is set as a function of n to minimize MSE, the bound
becomes (n IJTT) where 3 is the kernel order. In our experiments, all aspects of our model fixed across training size when estimating one learning curve, except learning schedule, but should be noted that error bounds and likely the learning curve parameters depend on both the classifier form and which parameters vary with n.
D.3 OTHER PLANNED IMPROVEMENTS
Experiments to include WRN-28-10 (or similarly effective Wide ResNet model) on Cifar- 100 to show that learning curve methodology applies and experimental findings hold for high-performing models
Discussion to clarify that experiments serve to exemplify use of learning curves and make interesting observations, but more extensive study of each design parameter is warranted. Also discuss any other concern/limitations raised by reviewers.
15


Reviews:
[{"review_id": "uFQYo2VFfwn", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper advocates for studying the effect of design choices in deep learning via their effect on entire *learning curves* (test error vs num samples N), as opposed to their effect only for a fixed N. This is a valid and important message, and it is indeed an aspect that is often overlooked in certain domains. However, although this paper addresses an important issue, there are methodological concerns (described below) which prevent me from recommending acceptance. In summary, the paper oversimplifies certain important aspects in both the setup and the experiments.\n\nConcerns:\n1. My main concern is that the discussion of learning curves ignores the effect of model size. Prior work (including Kaplan 2020 and Rosenfeld 2020) has shown that learning curves exhibit quantitatively different behavior when models are overparameterized vs. underparameterized. In particular, learning curves are only known to exhibit clean power-law behavior when model-size is not the bottleneck (e.g. if model-size is scaled up correspondingly to data size). There is no discussion of the model-size issue in the present work. This may be problematic, since data from small N are used to extrapolate to large N, but the model size is held fixed. \nConcretely: a full discussion of how to evaluate and interpret learning curves should account for the effect of model-size.\n\n2. The curve-fitting procedure is non-standard, and produces some questionable extrapolations.\nThis is concerning because one of the stated contributions of this paper is to propose an experimental methodology. Specifically: \n\nA. If the true parametric form is a power-law with some \\gamma != 0.5, why are the learning curves plotted assuming \\gamma=0.5 (Figure 1)? In the regression estimate (Equation 7), why is the exponent \\gamma encouraged to be close to 0.5?\nNote that the theoretical justification for \\gamma=0.5 (Table 2) is weak -- it only includes parametric bounds. Non-parametric rates are in general different from \\gamma=0.5.\n\n\nB. Several of the curves in Figure 1 predict cross-overs which we do not expect to occur at N=infty. For example, Figure 1g predicts that an ensemble of 6 ResNet18s will be better than 1 ResNet50 at N=\\infty, which we do not expect.\n\nC. In general, the curves are extrapolated from only 5 points -- it would be more convincing to see more data-sizes tested.\n\n3. Regarding experimental setup and conclusions:\n\nA. Why are there experiments for CIFAR-100 but not CIFAR-10? Most of the current experiments have high error rates (~20%), so it would have been nice to see how the curve-fits perform down to low error-rates (< 5%) as we would see on CIFAR-10.\n\nB. The claim that \"pretraining does not bias the classifier\" is too strong to be supported by the experiments. Certainly this does not hold for any arbitrary pre-training dataset, but perhaps it holds for \"natural\" pre-training datasets close to the ones tested here. In general, several of the experimental claims are too strong in this way -- they make universal statements, but are only tested in a few limited respects. Further experiments would give more evidence to these claims. For example, it is speculated on pg 11 that \\gamma does not depend much on the model architecture. Does this continue to hold for MLPs? (Only convnets are tested in this paper).\n\n\nSummary: The motivation of this paper is very good, but the proposed experimental methodology is somewhat lacking. This paper would be much improved by more thorough experiments and analysis, and more nuanced discussion of the experimental conclusion.\n\n\nComments/clarifications which do not affect the score:\nWhy are the experiments done using the Ranger optimizer? Would any conclusions differ if we use standard optimizers (SGD/Adam)?\nI would suggest moving Section 3.2 to the appendix, since the mechanics of least squares is likely familiar to readers. This would open more space for further discussion of Figure 1 experiments.\n\n---\nEdit after rebuttal: Changed score from 5 to 6 (see below)", "extra_scores": {}, "extra_text": {}}, {"review_id": "eaRrDkbQ-75", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThe authors conduct an investigation of learning curves on image classification models to understand tradeoffs between error and dataset size across different design choices. These learning curves help clarify the relationship between error and data reliance as a function of choices such as depth, width, pre-training and fine-tuning.\n\n### Comments\n* The introduction and motivation for the work is clear and well written.\n* I found the work well-contextualized within existing work.\n* The learning curve is a nice way to understand tradeoffs in design choices for a given model.\n* A single choice of optimizer is used, and learning curves could conceivably vary between different optimizers. It would be good to explore this further.\n* The experiments are thorough and have interesting conclusions that are applicable to researchers / practitioners e.g. deeper networks are suitable for smaller datasets.\n\n### Recommendation / Justification\nI vote to accept the paper. The authors do an excellent job of motivating the importance of learning curves and are systematic about their experimentation and analysis.\n\n### Minor feedback\n* Missing header and page number on first page\n* I like the T/F quiz. I think it'd help to put more detailed descriptions of some of the procedures e.g. \"Fine-tuning the entire network is only helpful if the training set is large\" is vague if you do not quantify helpful nor define an alternative (such as fine-tuning just the final layer).\n* typo:  \"pseudeo-inverse\" \n* appendix discussion on \"impact of \\gamma\" $\\gamma = 0 - 0.5$ would probably be clearer as $\\gamma \\in (0, 0.5)$", "extra_scores": {}, "extra_text": {}}, {"review_id": "9CksysmVHb4", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper uses experimental measurements and empirical curve-fits of learning curves to study their interaction with training protocols such as transfer learning and data augmentation. They define the learning curve to be the test error as a function of training set size. \n\nI appreciate the authors' goal of relating the scaling behavior of learning curves with common choices in NN training. Deep learning models are often complex and hard to model from first principles. Trying to understand and design deep learning models using scaling laws, empirical measurements, and power-law fits seems like a great idea. However, I am not entirely sure if the conclusions of this paper are convincing enough to be sufficiently useful to the ICLR readership. For example:\n\na) The authors conclude that \"Pre-training on similar domains nearly always helps compared to training from scratch.\", in agreement with their initial guess before they ran their experiments.  As far as I can tell, they reached this conclusion because one model they plot in Figure 1d, which was trained from random initializations, does worse on average than the 3 models they finetuned, all of which were pre-trained on larger datasets. It has already been reported in literature that pre-training on a similar dataset does not always outperform training from scratch [1 (already cited), 2, 3]. In fact, Ref. [3] has shown that depending on the data-augmentation settings, pre-training can significantly hurt final performance compared to training from scratch. Furthermore, Ref. 2 found that pre-training will not be helpful for certain dataset pairs, even if the target dataset has \"similar\" classes as the source dataset (e.g. cars in FGVC cars and ImageNet). I find the conclusion from Ref. 2 more convincing and precise, since they have evaluated transferring to 12 different datasets, using 16 different architectures, compared to the 2 different target datasets in this paper, and only 1 architecture (authors mention that they trained 8 different architectures, but am I understanding correctly that Figure 1d only includes a single architecture that is not pre-trained?)\n\nb) Building on item (a), I am not sure if the model trained without pre-training is representative of what researchers would use. Accuracies are not reported in the text, but it reads as though the \"No Pretr\" model achieves around 72% accuracy on CIFAR-100 (Fig 1d). It is relatively easy to achieve above 80% with a standard architecture such as WideResNet-28-10. Other work cited above have already seen that pre-training is more likely to do better than models trained from scratch if the models trained from scratch are not trained as optimally as they could be (common culprits are not training for long enough, not using sufficient regularization etc.) For this reason, in order to claim something as strong as  Popular Beliefs #1, it is important to have a good baseline for the model trained from scratch (note that I am not asking for a state-of-the-art results here, but a result that is comparable to models trained in literature). (One reason could potentially be that the authors trained on 40k samples of CIFAR-100 instead of the commonly used 45k, but it is still easy to get 80%+ with a standard training protocol initialized to random weights on 40k samples of CIFAR-100). \n\nc)  I worry that similar issues exist for other conclusions of the paper. For example, the conclusions about \"Increasing the network depth\" are very interesting, but I am not convinced that the paper has the experiments to justify them. It is crucial to optimize the regularization techniques for each width and depth separately, before one can make a statement about whether increasing the depth or width is generally helpful or not. \n\nI hope that the authors continue this work, expand and improve their experimental setup. Relating learning-curves to models that are of interest to the community, with performance that match what is reported in literature, would be a great first step. This way, the readers can easily judge if the connections observed in the paper would be applicable to the standard training protocols. A next step could be to show that using the insights that are developed in this paper, the authors can achieve a result that was not possible without their insights.    \n\n[1] He, Kaiming, Ross Girshick, and Piotr Dollár. \"Rethinking imagenet pre-training.\" Proceedings of the IEEE international conference on computer vision. 2019.\n[2] Kornblith, Simon, Jonathon Shlens, and Quoc V. Le. \"Do better imagenet models transfer better?.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2019.\n[3] Zoph, Barret, et al. \"Rethinking pre-training and self-training.\" arXiv preprint arXiv:2006.06882 (2020).", "extra_scores": {}, "extra_text": {}}, {"review_id": "7LcJfHro2ke", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors first propose a simple weighted least squares method to compute the \"learning curve\" (error plotted against dataset size) , where error is modelled with the form error = alpha - eta*n^gamma, for parameters alpha, eta, gamma. Gamma is taken to be - 0.5, while alpha, eta are estimated from the data. This also allows an estimate of \"data reliance\", in essence the slope of error wrt dataset size, computing how much error decrease is dependent on dataset size.\n\nThe authors then perform an extensive experimental evaluation on varying sized subsets of CIFAR-100 and Places365, across multiple different neural architectures and varying choices of finetuning, pretraining, linear classifier (frozen feature training) varying architecture size and data augmentation. They fit learning curves on these different empirical configurations, estimating data-reliance (along with extrapolating error), finding interesting conclusions such as finetuning outperforming linear classifiers even on small datasets, and larger architectures actually improving data reliance even in small data settings.\n\nBoth the learning curve computations and empirical evaluations are interesting and I recommend accepting this paper. \n\nHowever, I would strongly suggest changing the layout of the paper, in particular, splitting up Figure 1 into several figures to better emphasize some of the different takeaways. (E.g. perhaps section 3.2, which consists of recapping weighted least squares could be moved to the supplementary.) As it is, it is very difficult to follow the main takeaways from the different experiments, and even the insights given by fitting the learning curve (and computing data-reliance.)\n\nIf the authors can provide such a revised version, I would definitely consider further increasing my score.\n\nMinor comments:\n\n-- I appreciate the open acknowledgement of some of the limitations of the method\n\n-- I also liked the summary table (deep learning quiz) summarizing some of the conclusions of Figure 1 (which would have been hard to absorb otherwise)\n\n-- Are there assumptions about the dataset/task that must be made for fitting learning curves (and predicting data reliance) to work well? For example, what if the model is trained to memorize random labels?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the relationship between test error as a function of training set size and various design choices of neural network training. Overall all of the reviewers are excited about the prospect of relating error curves to neural network design choices, but different reviewers complain about the rigor of empirical evaluation and the accuracy of conclusions given limited data points. I agree with reviewers on both points, i.e., the paper studies different design choices, but does not do a thorough job studying those design choices. Moreover, it is not clear what aspects of the study are directly related to error curves vs. a standard correlation study done in prior work, e.g. in "Do better ImageNet models transfer better?" for usefulness of ImageNet pre-training. So, overall, I believe not only the empirical evaluation needs improvement, but also the story needs refinement. I am looking forward to seeing this paper published in other ML venues.
Decision: Reject