Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper characterized the scaling of generalization error with respect to the number of training samples $D$ and parameters $P$ for certain smooth estimators in two regimes: the variance-limited regime and the resolution-limited regime; this divide is somewhat analogous to the parametric vs. nonparametric estimation. The authors also argued that the large-width and large-sample limit have similar scaling behavior. The findings are supported by empirical evidence. 
main_review: Neural scaling laws have become a popular topic in machine learning, so the theoretical understanding, which this submission attempts to address, is an important research problem. However, I have the following concerns:  

**(i)** Many important prior works (related to the results presented in this submission) were not cited. The authors should discuss the difference from and improvement upon these papers. In particular, 
1. If we do not take optimization into account, then the so-called "scaling law" of neural network has already been extensively studied in the nonparametric literature. For example, [Schmidt-Hieber] and [Suzuki] derived the learning rates of deep neural network in fitting certain target functions; such analysis is usually divided into two parts, the estimation error, which scales like $D^{-\alpha}$, and the approximation error, which scales with $P^{-\beta}$, where the exponents $\alpha, \beta$ depend on the problem dimensionality and smoothness of the target function. 
2. If we restrict ourselves to linear/kernel models, then power-law generalization error rates are fairly standard and well-known results (e.g., see [Caponnetto and De Vito]). Moreover, for neural networks in the kernel regime, the role of optimization (i.e., the number of training steps) can also be incorporated in the analysis, as in [Nitanda and Suzuki]. 
3. For the random features model, the "dual" relation between the model width and training set size has been rigorously studied in [Ghorbani et al.] and [Mei et al.]. Note that [Ghorbani et al.] has been publicly available for more than 2 years, so I don't think the current submission can be treated as concurrent work. 

- Caponnetto and De Vito, FOCM 2007. https://link.springer.com/article/10.1007/s10208-006-0196-8. 
- Schmidt-Hieber, Annals of Statistics 2020. https://arxiv.org/pdf/1708.06633.pdf. 
- Suzuki, ICLR 2018. https://arxiv.org/pdf/1810.08033.pdf.
- Ghorbani et al., Annals of Statistics 2021.  https://arxiv.org/pdf/1904.12191.pdf.
- Suzuki and Nitanda, ICLR 2021. https://arxiv.org/pdf/2006.12297.pdf.
- Mei et al., 2021. https://arxiv.org/pdf/2101.10588.pdf.

**(ii)** The theoretical results are somewhat underwhelming, and some discussions lack rigor. 
1. It is not clear whether the $O(1/D)$-fluctuation is relevant in practical neural network training, because it omits other dependencies. For example, if we naively couple the finite-sample (stochastic) gradient descent with the population counterpart, then without additional assumptions, the difference between the two trajectories can blow up exponentially in time. Hence this bound would be uninformative beyond the early phase of training. Moreover, this does not provide any insight on the scaling law of training speed.  
2. The smoothness parameters in Theorem 2 and 3 are treated as constants independent to $D$ or $P$. Why wouldn't the Lipschitz constant of the learned interpolator depend on $D$, especially in the noisy setting? 
3. Many findings in Section 2.3 and 2.4 have little to do with actual neural network. For example, while a wide model can be "frozen" in the kernel regime under certain initialization, it is not clear why a trained network in the $D\gg P$ regime would resemble a random features model with similar spectral properties. 
4. For the replica computation, the authors should elaborate on the difference from the following papers, both of which derived power-law decay of generalization error for kernel regression.
- Spigler et al., J. Stat. Mech. 2020. https://arxiv.org/pdf/1905.10843.pdf.
- Bordelon et al., ICML 2020. https://arxiv.org/pdf/2002.02561.pdf.

**Additional Comments**

- For Section 2.3.2, the authors should comment on when one can expect such power-law decay of the kernel spectra to hold true. Intuitively this may not happen when $D\asymp d$ (for some notion of intrinsic dimensionality $d$) -- in this regime the kernel eigenvalues are usually bounded away from 0. 
summary_of_the_review: I evaluated this submission primarily as a theory paper; in my opinion the current theoretical contribution is below the acceptance bar, for reasons listed above. I am happy to adjust my score if the authors can address my concerns.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considered Variance-Limited Regime and Resolution-Limited Regime to explain the explored neural scaling law. In the Variance-Limited Regim, one fixed one of the D and P and in the Resolution-Limited Regime, one of the D and P is effectively infinite. The paper aims to provide a parametric rate scaling law in the Variance-Limited Regime  and a non-parametric scaling law in the Resolution-Limited Regime.
main_review: The decomposition of Variance-Limited Regime and Resolution-Limited Regime is somehow interesting. However, the reviewer find out it's another kind of bias-variance decomposition. The variance limited regime happens when the model size is constrained, if the rademencher complexity is bounded, then you have the 1/sqrt(data number) convergence, it's the variance in the statistics setting. The  Resolution-Limited Regime in the paper is the data is infinite and then the rate is the approximation rate.


For the Variance-Limited Regime, the statement is not clear. As I assume, the paper assumes the neural network lies in the  neural tangent kernel regime. Otherwise, the network weights are correlated, you can't do any concentration on width.  For the dataset scaling, the author claims "For every value of the weights, the training loss, thought of as a random variable over draws of a training set of size D, concentrates around the population loss, with a variance which scales as O(1/D)". This is wrong and is taught in statistical learning 101 course. The learned network is not independent with the data, so you can't use the concentration inequality, that's why the empirical process technique is introduced.(i.e. a uniform bound of all possible neural network is needed here (foundations of machine learning, Theorem 2.13 and chapter 3) .) Also the rate is not O(1/D) but is O(1/sqrt(D)) due to the  central limit theorem/Chernoff bound.

For the Resolution-Limited Regime, the 1/d scaling law is actually not surprising. The 1/d scaling law is actually the same as the non-parametric statistics [6]  and there is a line of research using a neural network to achieve this rate[1-5]. The bound the paper achieve is the same as the approximation rate obtained by [1-4]. They achieved N^{-s/d} approximation error for the s-holder function, and is standard and well-known in the literature. For the kernel setting, the nonparametric rate is from the Eigen decay  is also well known, see references in [5].

In section 2.3.1,the L(D,P)-L(D) should be O(1/sqrt{P}) but not O(1/P) due to central limit theorem/Chernoff bound,  similar errors appear many times in the paper as I think. In the case of linear models **with l2 loss**, I agree the variance will become O(1/P), but it's not the case for general model.

The paper is somehow interesting, however, the reviewer discovered that the author has little knowledge in basic statistics, as for example bias-variance decomposition, parametric rate (1/sqrt{N} variance fluctuation), and non-parametric rate (1/d scaling law). The notation in the paper is not well-defined and most of the "theorems" don't state the full set of assumptions of all theoretical results and even don't state the rigorous setting of the theorems. This doesn't seem like a rigorous paper but like a blog post. (For an example, in section 2.1 the expectation is an expectation on a different variable but the author use the same notation.)


[1] Schmidt-Hieber J. Nonparametric regression using deep neural networks with ReLU activation function[J]. The Annals of Statistics, 2020, 48(4): 1875-1897.

[2] Farrell M H, Liang T, Misra S. Deep neural networks for estimation and inference[J]. Econometrica, 2021, 89(1): 181-213.

[3] Suzuki T, Nitanda A. Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space[J]. arXiv preprint arXiv:1910.12799, 2019.

[4] Chen M, Jiang H, Liao W, et al. Efficient approximation of deep relu networks for functions on low dimensional manifolds[J]. Advances in neural information processing systems, 2019, 32: 8174-8184.

[5] Nitanda A, Suzuki T. Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime[J]. arXiv preprint arXiv:2006.12297, 2020.

[6] Tsybakov A B. Introduction to Nonparametric Estimation[J]. Springer series in statistics, 2009: I-XII,1-214.
summary_of_the_review: The paper is not well-written and theoretical results are not well-stated. The paper also igonres standard results in the statistical results, bias-variance decomposition vs the four regimes and the non-parametric rates. The reviewer also find out that the author lack of knowledge in basic statistical machine learning theory. Reviewer suggests the author write down the model considered (random feature, neural tangent kernel regime NN?) rigorously and state all the notation and theorem in detail. 
The empirical experiment in the paper is interesting and meets the bar of a top conference, however the "theory" part is confusing, not novel and even something can be wrong.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a phenomenon called scaling laws of test loss for neural networks. In a nutshell, the test loss decreases by following a power law curve (exponent) with respect to width or data set size, while the other being fixed.  Specifically, the variance limited scaling follows simply from the existence of a well-behaved infinite data or infinite width limit, while the resolution-limited regime can be explained by
positing that models are effectively resolving a smooth data manifold. The proofs are for linear models (random feature) and pretrained models. Authors also observe several empirical relationships between datasets and scaling exponents: super-classing image tasks does not change exponents, while changing
input distribution (via changing datasets or adding noise) has a strong effect.
main_review: The paper proposes a phenomenon called scaling laws of test loss for neural networks. In a nutshell, the test loss decreases by following a power law curve (exponent) with respect to width or data set size, while the other being fixed.  Specifically, the variance limited scaling follows simply from the existence of a well-behaved infinite data or infinite width limit, while the resolution-limited regime can be explained by
positing that models are effectively resolving a smooth data manifold. The proofs are for linear models (random feature) and pretrained models. Authors also observe several empirical relationships between datasets and scaling exponents: super-classing image tasks does not change exponents, while changing
input distribution (via changing datasets or adding noise) has a strong effect.

Overall, the paper studies a very interesting phenomenon. It's quite interesting to see such power law with respect to width and data set sizes with the other being fixed. Such good combination of interesting phenomenon with theoretical explanation is a great fit for ICLR. Hence, an accept has been recommended. 

However, there are several concerns remaining.

1. It's counter-intuitive to see the accuracy follows power law for width. In the limit, it becomes the kernel regime, and we all know NTK is different from neural network. In this case, there must be finite width which perform better than infinite width? Then this is a counter-example to your phenomenon?

2. What's the practical different between variance limited and resolution limited? These two terms are bit confusing.

3. The data set size is reasonbale, but does not consider if we add more rare examples? If rare examples are of measure 0, then it's not reflected in your theory, but can make a big difference in practice?

4. In some cases, I find smaller network perform better (say width 32), larger width cannot work. How do you explain this?
summary_of_the_review: Overall, the paper studies a very interesting phenomenon. It's quite interesting to see such power law with respect to width and data set sizes with the other being fixed. Such good combination of interesting phenomenon with theoretical explanation is a great fit for ICLR. Hence, an accept has been recommended.  However, authors need to clarify a few concerns.
review_title: null

Meta-review: This paper investigates the scaling laws of neural networks with respect to the number of training samples $D$ and parameters $P$ for some estimators in two regimes: the variance-limited regime and the resolution-limited regime. The theoretical results are supported by some numerical experiments.

Unfortunately, the paper has several critical issues, in particular, in its novelty and technical correctness. 
1. The theoretical analyses lack much of their rigor. The assumptions and problem setups are not precisely introduced. Accordingly, the statement of each theorem is presented in an inaccurate way. Moreover, some theoretical consequences contain technical flaws (e.g., $1/P$ should be replaced by $1/\sqrt{P}$ without an appropriate assumption on the loss function such as strong convexity and smoothness).  
2. Many of the presented results are already known in the literatures. It is unfortunate that the authors did no cite relevant existing literatures and did not discuss its novelty compared with the existing work. 

For those reasons, this paper lacks its novelty and the quality of the paper is not sufficient to be accepted.  
I recommend the authors to thoroughly survey the literature of the statistical learning theory from classic nonparametric regression analyses to recent advances on overparameterization.
Decision: Reject