Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper explores the relation among the generalization error of neural networks and the model and data scales empirically. The topic is interesting, while I was expecting to learn more from the paper, instead of some well-known conclusions. If the paper could provide some guidance for model and data selection, that would be an interesting paper for the ICLR audience.  For instance, how deep should a model be for a classification or regression task? What is the minimum/maximum layers of a deep model? How much data is sufficient for a model to learn? What is the minimum/maximum size of the data set? Do we really need a large data set or just a subset that covers the data distribution? What's the relation between the size of a model and that of a data set? By increasing the depth/width of a neural network, how much new data should be collected for achieving a reasonable performance? How about the gain of the task performance? 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper proposes a functional form to model the dependence of generalization error on a held-out test set on model and dataset size. The functional form is derived based on empirical observations of the generalizing error for various model and dataset sizes (sections O1, O2, and O3) and on certain necessary criteria (C1, C4 and C5). The parameters of the function are then fit using linear regression on observed data. The authors show that the regressed function \(\epsilon(m,n)\) is able to predict the generalization error for various \(m\) and \(n\) reasonably accurately.

Major Points:
- While the current experiments are a good start, I do not think they are extensive enough to count as strong evidence for the  power-law form of \(\epsilon(m,n)\). I would ideally like to see results on more optimizers, at the very least for Adam, even if for fixed hyper-parameters. As far as I understand, this involves only minor changes in the code since reasonable hyperparameters required for the convergence of Adam have been extensively studied. If the form still holds true then the results from this work can be more reliably used for small-scale network development and in making trade-off choices (as discussed in section 8).
- Given the current form of the paper, the abstract and introduction should be modified to reflect the fact that only limited architectures and optimizers were experimented with, and the claims of the paper are not experimentally validated in general.

Minor Points:
 
- It would be nice if more network architectures were analysed (such as VGG and DenseNets). 
- It would be nice if different stopping criteria were analysed.
- It would greatly benefit the reader if eq. 5 were expanded.
 
Overall, I think this is a well written paper and provides good insight into the behaviour of the error landscape as a function of model and dataset size. The paper’s primary drawback is the restrictive setting under which the experiments are performed. Therefore, I am not convinced that the power-law form of the generalization error would hold when the experimental settings are marginally different (like when using the Adam optimizer or a VGG-like architecture). I think this work would have much greater impact if the authors can show that the power-law form holds for a larger variety of architectures and optimizers thus allowing researchers to more confidently incorporate the results of this work into the design and training deep neural networks.

Rebuttal Response
I would like to thank the authors for their response. The results of additional experiments as described in Section 6.2 and in Figure 5 do indeed provide stronger evidence of the power-law form of the error function. In light of this, I have changed my original rating. 


review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This work proposes a functional form for the relationship between <dataset size, model size> and generalization error, and performs an empirical study to validate it. First, it states 5 criteria that such a functional form must take, and proposes one such functional form containing 6 free coefficients that satisfy all these criteria. It then performs a rigorous empirical study consisting of 6 image datasets and 3 text datasets, each with 2 distinct architectures defined at several model scales, and trained with different dataset sizes. This process produces 42-49 data points for each <dataset, architecture> pair, and the 6 coefficients of the proposed functional form are fit to those data points, with < 2% mean deviation in accuracy. It then studies how this functional form performs at extrapolation, and finds that it still performs pretty well, with ~4.5% mean deviation in accuracy, but with additional caveats.

Decision: Accept. This paper states 5 necessary criteria for any functional form for generalization error predictor that jointly considers dataset size and model size, then empirically verifies it with multiple datasets and architectures. These criteria are well justified, and can be used by others to narrow down the search for functions that approximate the generalization error of NNs without access to the true data distribution, which is a significant contribution. The empirical study is carefully done (e.g., taking care to subsample the dataset in a way that preserves the class distribution). I also liked that the paper is candid about its own limitations. A weakness that one might perceive is that the coefficients of the proposed functional form still needs to be fit to 40-ish trained NNs for every dataset and training hyperparameters, but I do not think this should be held against this work, because a generalization error predictor (let alone its functional form) that works for multiple datasets and architecture without training is difficult, and the paper does include several proposals for how this can still be used in practice.
(Caveat: the use of the envelope function described in equation 5 (page 6) is not something I am familiar with, but seems reasonable.)

Issues to address:
- Fitting 6 parameters to 42-49 data points raises concerns about overfitting. Consider doing cross validation over those 42-49 data points, and report the mean of deviations computed on the test folds. The extrapolation section did provide evidence that there probably isn't /that/ much overfitting, but cross validation would directly address this concern.
- In addition, the paper provides the standard deviation for the mean deviations over 100 fits of the function as the measure of its uncertainty, but I suspect that the optimizer converging to different coefficients at different runs isn't the main source of uncertainty. A bigger source of uncertainty is likely due to there being a limited amount of data to fit the coefficients to. Taking the standard deviation over the deviations measured on different folds of the data would be better measure of uncertainty.

Minor issues:
- Page 8: "differntiable methods for NAS." differentiable is misspelled.
review_title: Official Blind Review #3

Meta-review: The paper presents a very interesting idea for estimating the held-out error of deep models as a function of model and data set size. The authors intuit what the shape of the error should be, then they fit the parameters of a function of the desired shape and show that this has predictive power. I find this idea quite refreshing and the paper is well written with good experiments. Please make sure that the final version contains the cross-validation results provided during the rebuttal.
Decision: Accept (Poster)