Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper contributes a solid work with a new recipe to estimate scaling laws reliably from learning curves, which outperforms previous work on a variety of tasks. The paper also shows that in order to validate scaling law parameters, we should base on extrapolation of data points instead of interpolation of data points. The paper also contributes a benchmark dataset consists of 90 tasks, which I believe it is an important contribution to the community.


strengths_and_weaknesses: Pros:
- The paper is well-written for most parts.
- The new recipe is ingenious to me.
- The experiment section is solid. It covers a lot of different tasks with clear results supporting the benefits of using M4, the proposed method.
- Overall the paper contains materials that are very nice contribution to the field.

Cons:
- I don't think it is clear in the paper while the new recipe works that well compared to M2. Is it because M4 contains M2 as a special case or is it because M4 has a sigmoid like shape, which avoids failures of M2 as in Figure 2 left? I think this needs to be elaborate more.

- It is also not clear what exactly it means to be "extrapolation". In the introduction, the authors said that: ``Given the estimated scaling law parameters, one can then extrapolate by predicting the performance f(x) for large values of x". However, in the experiment section, it was explained that: "we divide the learning curve into two splits: (1) one split used for training the scaling law estimators, and one split used for evaluating extrapolation performance. We set the cutoff between the two splits to be equal to xmax/2, where xmax is the maximum value of x". With that, I am not sure the second split, which used for evaluating extrapolation performance reflects what was stated in the statement (i.e. large values of x).

Note: Equation 3 is the one that I could not follow. Perhaps a more explanation on this would help readers a lot.

Typo:

of which four seem to be more prominent -> four seem???

This is consistent with the fact that scaling exponents often satisfy c -> satisfies?

-------
Update: I wanted to thanks authors for answering my questions. I still have an issue with extrapolation and hopefully the authors will put some more efforts on explaining that in the paper. However, my overall impression is that the recipe is ingenious to me and it seems to work. So I just raised my score from 6 to 7. 

questions: (see the strengths and weakness)
limitations: n/a
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper argues that predicting the benefit of scale should be conducted based on learning the learning curve 'extrapolation' rather than 'interpolation'. In addition, based on this, this work proposes an estimator model for the scaling laws, which extrapolates more accurately than previous methods, and in various experiments, the proposed estimator indeed shows better performance.
strengths_and_weaknesses: ### Strengths

- I believe the task dealt with in this paper will be able to give a significant impact in the era where scaled-up DNN models show surprising performance.
- To support the claim of this work, extensive experiments with various models, datasets, and experimental environments are conducted.

### Weaknesses

- There are several concerns about the experiments and I listed the questions below.
- I guess the derivation of $\mathcal{M}_4$ arises from the motivation part, and the motivation part says an accurate model that extrapolates well would produce a linear curve, however, I was not able to get why it is true.
- It was a little confusing when I first read the paper, so I recommend providing the range of $\beta, c$ in the Introduction to help readers imagine the shape of the graphs.
questions: - Isn’t it natural to derive the $\mathcal{M}_4$ based on the empirical observations rather than the synthetic experiment as in Section 3?
- I’m not sure why the first experiments were conducted in a few-shot setting instead of the pre-training task itself.
- In figure 3, does the x-axis, examples seen, means the number of data examples that are fed during training instead of the size of the dataset? Then, is it the realistic experimental setting where the interpolation region is set around $3\cdot10^8$ examples seen, which is the same as the number of data samples of JFT-300M?
- The observation that the $\mathcal{M}_4$ tends to yield estimates of the scaling exponents that are larger in absolute magnitude than in other methods means what?
- I’m confused that, for example in Figure 3, shouldn’t the difference in $c$ be reflected as different linear functions with different slopes rather than different curves in log-log plots?
limitations: This paper does not address the limitations and considerations about the negative societal impact.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose an extrapolation-based fitting for better estimating the scaling properties of deep learning models (e.g., the larger model lowest Val/Test loss). The author compares interpolation methods, e.g.,  power-law fitting, with the proposed extrapolation method on different tasks image classification, NMT, language modeling, tasks from BIG-Bench, and architecture type and sizes. The results show that the proposed extrapolation method is more accurate in predicting the model's performance. 
strengths_and_weaknesses: Strengths
- The experiments to support the claims are extensive and explore multiple modalities and architectures. 
- The paper is well written, well-motivated, and easy to follow. 
- Achieving better prediction in terms of model scaling (e.g., size, data size) is important in many scenarios, once for all neural architecture search. 



questions: - The loss shown in the appendix is quite important to highlight the difference between different methods. Why not include it in the main paper?
limitations: No this is missing in the paper.
review_title: null

Meta-review: metareview: The submission proposes a new method for deriving scaling laws for deep neural networks. Previous work has fit curves to datapoints based on interpolation. The paper argues convincingly that extrapolation is much more important - i.e. how well the function will predict future points if we trained longer. The authors derive a new approach to estimating scaling laws, and demonstrate that it outperforms existing proposals at predicting future points on a range of vision and language tasks. The paper notices a clear issue with current approaches to scaling laws, the intuition behind their approach is clear, and the experimental results are convincing - so I recommend acceptance. 
confidence: Certain
Decision: Accept