Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a piecewise defined "broken power law functional" that can accurately fit the performance of neural networks when quantities like the amount of compute used for training, number of model parameters, or training dataset size varies across several orders of magnitudes. The accuracy of the functional is verified on many different tasks. Compared with existing power laws, the proposed functional can fit non-monotonic behaviors like the double-descent phenomenon
strength_and_weaknesses: Strength: 

The broken power law functional can fit the performance of neural networks over the change of its size more accurately. It can work on more networks, problems, and metric of performances.

Weaknesses:

Compared with existing power laws, the proposed functional has more free parameters. Hence, it is not surprising that it can fit the scaling behavior more accurately. To show the power of this proposed functional, the authors need to show that it can give more accurate predictions to the performance of neural networks than the power laws, or the prediction works in a larger range of network size (or other factors). As shown in the numerical results in the appendix, it seems in many cases the functional does not provide good prediction to the green points even though those points only span less than one order of magnitude in the horizontal axis. 
clarity,_quality,_novelty_and_reproducibility: Clarity: The writing is clear and easy to follow.

Quality: The proposed functional is carefully tested on many different settings. The quality of the numerical study is high.

Novelty: The broken power law functional is novel in the studying of neural scaling law. It has been used in other fields like astrophysics.

Reproducibility: I did not check the reproducibility of the experiments.
summary_of_the_review: The paper proposes a new functional that can more accurately fit the scaling behavior of neural networks. The concern is that the accuracy of the functional comes from the flexibility provided by undetermined parameters, rather than characterizing the true pattern behind the scaling behaviors.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new functional form for neural scaling laws that corrects for two weaknesses in previous functions: (1) that they could only model strict monotonic behaviour, and (2) they could not express inflection points. The function is intuitive and well-designed and outperforms previous scaling law functions on learning curve fitting, sometimes by orders of magnitude. These are evaluated on many vision and language tasks to demonstrate better accuracy than other methods.

strength_and_weaknesses: Strengths:
- The paper has many extensive experiments and the function is intuitive and can be easily implemented

Weaknesses:
- The authors remark on a limitation which I wish that they numerically investigated. Just how many points do we need to fit this function? Simple neural scaling laws can be fit with even less than 10 points, whereas since this function has lots of parameters, I suspect it can be challenging. It would be nice to run some ablations on this, especially since it is an easy test.

- How do we determine $n$ when forming a BNSL? This seems to be a parameter that must be specified a priori and not learned. Is there intuition in terms of how we can go about selecting it? Is there a consequence for selecting n too large or too small?
clarity,_quality,_novelty_and_reproducibility: Overall the paper is clearly written and the functional form has intuitive value that addresses weaknesses of previous forms. The paper also includes details on how to train the predictor.

It might be helpful to provide more intuition on equation (1), e.g., with another equation that describes the relationship in a log-log form.
summary_of_the_review: Overall the paper presents a simple, intuitive function to train and demonstrates its value extensively on both vision and language tasks. Although the paper identifies limitations, it would be nice to also evaluate the degree of the limitations, especially since they are not too hard to test.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose using an functional form of smoothly broken power law namely broken neural scaling laws (BNSL) for extrapolating and generalizing to model scaling behaviors across diverse set of upstream and downstream tasks such zero-shot, prompted, and fine-tuned settings of unsupervised language, vision, reinforcement learning, and arithmetic. The results shows that using BNSL is significantly more accurate in extrapolating the scaling laws based on dataset size and model complexity. These benefits are driven from two important characteristics of BNSL to model non-monotonic transitions in scaling behavior (e.g in double descent) or and inflection points (e.g. in delayed and sharp transitions in arithmetic). 
strength_and_weaknesses: The strengths of this paper include but not limited to clarity of the idea presented in presenting a formula to model scaling behavior neural networks in learning tasks is very clear and somehow intuitive idea. Second, this paper is extremely concise and the basic mathematical idea behind the paper sounds solid and intuitive. Another strength is significant enhancement in the performance of the scaling law in neural networks, at least compared to other M1 to M4 methods. 

Some of the weaknesses are presented in the A.3 section. Although the results show stronger performance compared to other proposed formulations, the figures show significant deviations in performance. This is not necessarily a weakness however it is worthy of addressing in the paper. In addition, the performance for cases with more monotonic extrapolation sounds reasonable in some simpler cases however in more complex nonlinear cases, the duration of predictions sound too short to give any notion of the performance. My concern is the number of test points has been cherry picked based on the strength of this approach. In addition, the practicality of modeling scaling behavior is way more valuable when we can predict a certain behavioral patterns sooner than later. In other words, modeling with fewer datapoints and extrapolating on more points than is presented. Finally, I am curious how does the scaling formula is compared to other methods aside from direct formulations such as predictive models and machine learning extrapolations. The reason for my notion is my observation in the A.3 that this formulation misses some basic locally linear extrapolations in some cases. For example, in Fig.6 the results sounds like a simple linear regression in log-log plot models the scaling behavior better and the BNSL actually is underestimating the scaling behavior. This also intensifies my previous concern that longer extrapolation horizon might show even higher and more staggering deviations between results and the BNSL prediction. 
clarity,_quality,_novelty_and_reproducibility: The paper is very clear and the language is standard language in academia. 

The mathematical language of the paper could have been more rigorous. There is not many propositions and theorems that define the theoretical analysis the scalings, the formulation limitations, or boundaries of the problem. The quality of the appendices are low because it lacks discussion and is mainly focused on experimental plots. 

The experimental aspect of the paper is high quality as it is experimented on multiple task settings and famous datasets. In comparison to Alabdulmohsin et al (2022) the novelty is less than average as proposing a new functional form for scaling neural networks has been around in multiple previous works. The mentioned paper Also in comparison, the suggested formulation by Hernandez et al. (2021) proposes more specific case of fine-tuning and limited to dataset size. 



  
summary_of_the_review: The paper has clear and solid and intuitive idea and observations, however, limited presentation to support the theory behind the scaling law. The presentation is not rigorous in terms of theoretical foundation and too much dependent on the intuitive explanation of the formula's property. The experimental results can be improved and acceptable by including longer extrapolation horizon to show the true capability of this functional form for more sophisticated extrapolation (not just more sophisticated modeling) and this functional form can be compared to non-functional scaling laws (the reviewer were not able to find precedent to non-functional forms of scaling laws in recent literature). To the reviewer, proposing one functional form for all the task settings and networks is not convincing. There might be the case that different scaling formulas can be applied to different neural networks and deeper analysis of the scaling formula in combination with the network properties gives better view of the scaling behavior.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: A new neural scaling law is proposed that can predict and extrapolate to high-order scale regimes.

Strengths:
- Extensive empirical evidence is provided for the law
- The method is compared appropriately with the existing power-law models. 
- The paper is very easy to follow and is very well organized.

Weaknesses: 
- In almost all experimental results the [Alabdulmohsin et al. 2022] (M4) results seem to be as good on predictive performance as the method proposed. 
- The limitation raised by the reviewers related to the inaccurate results for example Caltech101, is not negligible. 
- Having a lot more hyperparameters in the scaling law as opposed to previous work is not ideal in this space, as performing ground truth experiments could be extremely expensive.

The paper is borderline, as there are multiple open concerns (some outlined above) that must be addressed about this new scaling law. However, based on my discussions with the reviewers, I believe that the paper can attract some useful discussions at the conference on neural scaling laws and, thus, vote for acceptance. 

P.S. The claims must be toned down; Especially in the abstract and the intro as there are evident limitations observed where the introduced law failed.
justification_for_why_not_higher_score: The paper has some serious shortcomings and limitations that must be iterated before becoming of higher impact. Details are provided in the meta-review summary. 
justification_for_why_not_lower_score: The work can potentially attracts constructive discussions at the conference. Details are provided in the meta-review summary.
Decision: Accept: poster