Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper uses experimental measurements and empirical curve-fits of learning curves to study their interaction with training protocols such as transfer learning and data augmentation. They define the learning curve to be the test error as a function of training set size. 

I appreciate the authors' goal of relating the scaling behavior of learning curves with common choices in NN training. Deep learning models are often complex and hard to model from first principles. Trying to understand and design deep learning models using scaling laws, empirical measurements, and power-law fits seems like a great idea. However, I am not entirely sure if the conclusions of this paper are convincing enough to be sufficiently useful to the ICLR readership. For example:

a) The authors conclude that "Pre-training on similar domains nearly always helps compared to training from scratch.", in agreement with their initial guess before they ran their experiments.  As far as I can tell, they reached this conclusion because one model they plot in Figure 1d, which was trained from random initializations, does worse on average than the 3 models they finetuned, all of which were pre-trained on larger datasets. It has already been reported in literature that pre-training on a similar dataset does not always outperform training from scratch [1 (already cited), 2, 3]. In fact, Ref. [3] has shown that depending on the data-augmentation settings, pre-training can significantly hurt final performance compared to training from scratch. Furthermore, Ref. 2 found that pre-training will not be helpful for certain dataset pairs, even if the target dataset has "similar" classes as the source dataset (e.g. cars in FGVC cars and ImageNet). I find the conclusion from Ref. 2 more convincing and precise, since they have evaluated transferring to 12 different datasets, using 16 different architectures, compared to the 2 different target datasets in this paper, and only 1 architecture (authors mention that they trained 8 different architectures, but am I understanding correctly that Figure 1d only includes a single architecture that is not pre-trained?)

b) Building on item (a), I am not sure if the model trained without pre-training is representative of what researchers would use. Accuracies are not reported in the text, but it reads as though the "No Pretr" model achieves around 72% accuracy on CIFAR-100 (Fig 1d). It is relatively easy to achieve above 80% with a standard architecture such as WideResNet-28-10. Other work cited above have already seen that pre-training is more likely to do better than models trained from scratch if the models trained from scratch are not trained as optimally as they could be (common culprits are not training for long enough, not using sufficient regularization etc.) For this reason, in order to claim something as strong as  Popular Beliefs #1, it is important to have a good baseline for the model trained from scratch (note that I am not asking for a state-of-the-art results here, but a result that is comparable to models trained in literature). (One reason could potentially be that the authors trained on 40k samples of CIFAR-100 instead of the commonly used 45k, but it is still easy to get 80%+ with a standard training protocol initialized to random weights on 40k samples of CIFAR-100). 

c)  I worry that similar issues exist for other conclusions of the paper. For example, the conclusions about "Increasing the network depth" are very interesting, but I am not convinced that the paper has the experiments to justify them. It is crucial to optimize the regularization techniques for each width and depth separately, before one can make a statement about whether increasing the depth or width is generally helpful or not. 

I hope that the authors continue this work, expand and improve their experimental setup. Relating learning-curves to models that are of interest to the community, with performance that match what is reported in literature, would be a great first step. This way, the readers can easily judge if the connections observed in the paper would be applicable to the standard training protocols. A next step could be to show that using the insights that are developed in this paper, the authors can achieve a result that was not possible without their insights.    

[1] He, Kaiming, Ross Girshick, and Piotr Dollár. "Rethinking imagenet pre-training." Proceedings of the IEEE international conference on computer vision. 2019.
[2] Kornblith, Simon, Jonathon Shlens, and Quoc V. Le. "Do better imagenet models transfer better?." Proceedings of the IEEE conference on computer vision and pattern recognition. 2019.
[3] Zoph, Barret, et al. "Rethinking pre-training and self-training." arXiv preprint arXiv:2006.06882 (2020).
review_title: The conclusions reached by the authors are not justified by their experiments.

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors first propose a simple weighted least squares method to compute the "learning curve" (error plotted against dataset size) , where error is modelled with the form error = alpha - eta*n^gamma, for parameters alpha, eta, gamma. Gamma is taken to be - 0.5, while alpha, eta are estimated from the data. This also allows an estimate of "data reliance", in essence the slope of error wrt dataset size, computing how much error decrease is dependent on dataset size.

The authors then perform an extensive experimental evaluation on varying sized subsets of CIFAR-100 and Places365, across multiple different neural architectures and varying choices of finetuning, pretraining, linear classifier (frozen feature training) varying architecture size and data augmentation. They fit learning curves on these different empirical configurations, estimating data-reliance (along with extrapolating error), finding interesting conclusions such as finetuning outperforming linear classifiers even on small datasets, and larger architectures actually improving data reliance even in small data settings.

Both the learning curve computations and empirical evaluations are interesting and I recommend accepting this paper. 

However, I would strongly suggest changing the layout of the paper, in particular, splitting up Figure 1 into several figures to better emphasize some of the different takeaways. (E.g. perhaps section 3.2, which consists of recapping weighted least squares could be moved to the supplementary.) As it is, it is very difficult to follow the main takeaways from the different experiments, and even the insights given by fitting the learning curve (and computing data-reliance.)

If the authors can provide such a revised version, I would definitely consider further increasing my score.

Minor comments:

-- I appreciate the open acknowledgement of some of the limitations of the method

-- I also liked the summary table (deep learning quiz) summarizing some of the conclusions of Figure 1 (which would have been hard to absorb otherwise)

-- Are there assumptions about the dataset/task that must be made for fitting learning curves (and predicting data reliance) to work well? For example, what if the model is trained to memorize random labels?
review_title: A simple, easily usable method to predict learning curves and data reliance, with interesting insights on transfer and data augmentation

final_score: rating: 7: Good paper, accept
review_content: ### Summary
The authors conduct an investigation of learning curves on image classification models to understand tradeoffs between error and dataset size across different design choices. These learning curves help clarify the relationship between error and data reliance as a function of choices such as depth, width, pre-training and fine-tuning.

### Comments
* The introduction and motivation for the work is clear and well written.
* I found the work well-contextualized within existing work.
* The learning curve is a nice way to understand tradeoffs in design choices for a given model.
* A single choice of optimizer is used, and learning curves could conceivably vary between different optimizers. It would be good to explore this further.
* The experiments are thorough and have interesting conclusions that are applicable to researchers / practitioners e.g. deeper networks are suitable for smaller datasets.

### Recommendation / Justification
I vote to accept the paper. The authors do an excellent job of motivating the importance of learning curves and are systematic about their experimentation and analysis.

### Minor feedback
* Missing header and page number on first page
* I like the T/F quiz. I think it'd help to put more detailed descriptions of some of the procedures e.g. "Fine-tuning the entire network is only helpful if the training set is large" is vague if you do not quantify helpful nor define an alternative (such as fine-tuning just the final layer).
* typo:  "pseudeo-inverse" 
* appendix discussion on "impact of \gamma" $\gamma = 0 - 0.5$ would probably be clearer as $\gamma \in (0, 0.5)$
review_title: Official Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper advocates for studying the effect of design choices in deep learning via their effect on entire *learning curves* (test error vs num samples N), as opposed to their effect only for a fixed N. This is a valid and important message, and it is indeed an aspect that is often overlooked in certain domains. However, although this paper addresses an important issue, there are methodological concerns (described below) which prevent me from recommending acceptance. In summary, the paper oversimplifies certain important aspects in both the setup and the experiments.

Concerns:
1. My main concern is that the discussion of learning curves ignores the effect of model size. Prior work (including Kaplan 2020 and Rosenfeld 2020) has shown that learning curves exhibit quantitatively different behavior when models are overparameterized vs. underparameterized. In particular, learning curves are only known to exhibit clean power-law behavior when model-size is not the bottleneck (e.g. if model-size is scaled up correspondingly to data size). There is no discussion of the model-size issue in the present work. This may be problematic, since data from small N are used to extrapolate to large N, but the model size is held fixed. 
Concretely: a full discussion of how to evaluate and interpret learning curves should account for the effect of model-size.

2. The curve-fitting procedure is non-standard, and produces some questionable extrapolations.
This is concerning because one of the stated contributions of this paper is to propose an experimental methodology. Specifically: 

A. If the true parametric form is a power-law with some \gamma != 0.5, why are the learning curves plotted assuming \gamma=0.5 (Figure 1)? In the regression estimate (Equation 7), why is the exponent \gamma encouraged to be close to 0.5?
Note that the theoretical justification for \gamma=0.5 (Table 2) is weak -- it only includes parametric bounds. Non-parametric rates are in general different from \gamma=0.5.


B. Several of the curves in Figure 1 predict cross-overs which we do not expect to occur at N=infty. For example, Figure 1g predicts that an ensemble of 6 ResNet18s will be better than 1 ResNet50 at N=\infty, which we do not expect.

C. In general, the curves are extrapolated from only 5 points -- it would be more convincing to see more data-sizes tested.

3. Regarding experimental setup and conclusions:

A. Why are there experiments for CIFAR-100 but not CIFAR-10? Most of the current experiments have high error rates (~20%), so it would have been nice to see how the curve-fits perform down to low error-rates (< 5%) as we would see on CIFAR-10.

B. The claim that "pretraining does not bias the classifier" is too strong to be supported by the experiments. Certainly this does not hold for any arbitrary pre-training dataset, but perhaps it holds for "natural" pre-training datasets close to the ones tested here. In general, several of the experimental claims are too strong in this way -- they make universal statements, but are only tested in a few limited respects. Further experiments would give more evidence to these claims. For example, it is speculated on pg 11 that \gamma does not depend much on the model architecture. Does this continue to hold for MLPs? (Only convnets are tested in this paper).


Summary: The motivation of this paper is very good, but the proposed experimental methodology is somewhat lacking. This paper would be much improved by more thorough experiments and analysis, and more nuanced discussion of the experimental conclusion.


Comments/clarifications which do not affect the score:
Why are the experiments done using the Ranger optimizer? Would any conclusions differ if we use standard optimizers (SGD/Adam)?
I would suggest moving Section 3.2 to the appendix, since the mechanics of least squares is likely familiar to readers. This would open more space for further discussion of Figure 1 experiments.

---
Edit after rebuttal: Changed score from 5 to 6 (see below)
review_title: review

Meta-review: This paper studies the relationship between test error as a function of training set size and various design choices of neural network training. Overall all of the reviewers are excited about the prospect of relating error curves to neural network design choices, but different reviewers complain about the rigor of empirical evaluation and the accuracy of conclusions given limited data points. I agree with reviewers on both points, i.e., the paper studies different design choices, but does not do a thorough job studying those design choices. Moreover, it is not clear what aspects of the study are directly related to error curves vs. a standard correlation study done in prior work, e.g. in "Do better ImageNet models transfer better?" for usefulness of ImageNet pre-training. So, overall, I believe not only the empirical evaluation needs improvement, but also the story needs refinement. I am looking forward to seeing this paper published in other ML venues.
Decision: Reject