(Mis)Fitting Scaling Laws: A Survey of Scaling Law Fitting Techniques in Deep Learning

Authors: Margaret Li, Sneha Kudugunta, Luke Zettlemoyer
Primary area: foundation or frontier models, including LLMs
Keywords: survey, scaling laws, large language models, foundation models

Abstract:
Modern foundation models rely heavily on using scaling laws to guide crucial training decisions. Researchers often extrapolate the optimal architecture and hyper parameters settings from smaller training runs by describing the relationship between, loss, or task performance, and scale. All components of this process vary, from the specific equation being fit, to the training setup, to the optimization method. Each of these factors may affect the fitted law, and therefore, the conclusions of a given study. We discuss discrepancies in the conclusions that several prior works reach, on questions such as the optimal token to parameter ratio. We augment this discussion with our own analysis of the critical impact that changes in specific details may effect in a scaling study, and the resulting altered conclusions. Additionally, we survey over 50 papers that study scaling trends: while 45 of these papers quantify these trends using a power law, most under-report crucial details needed to reproduce their findings. To mitigate this, we we propose a checklist for authors to consider while contributing to scaling law research.

Decision: accept (poster)
Reviewer scores: 5, 5, 8, 5

Reviews and discussion:
<thread>
creation date:
1729880308419

thread_type:
official_review

<thread_object>
creator: reviewer_pbpk

<content>
summary:
The paper presents a survey on the fitting of scaling laws, and argue that current practices are lacking in scientific rigor. Apart from an extensive survey, the authors presents a reproducibility checklist, and compare 51 papers to this checklist. They generally find that important details are underreported, e.g. the method to calculate model parameters might not be given. They also provide a replication study of Hoffman, using data extracted from the paper PDF and data they’ve collected themselves. Here they find that subtle choices in the curve fitting can result in significantly different conclusions.

strengths:
- Scaling laws is an important topic, and scientific rigor here can benefit the research community.
- The authors illustrate how subtle choices in the curve fitting can cause significant results
- Section 7 is great.

weaknesses:
- Significant parts of the papers are dedicated to a survey. I’m not sure survey papers are the right fit for ICLR main track.
- There are not so many empirical results.

questions:
- could you provide explicit recommendation regarding how to perform the curve fitting? I think this is different from a checklist which allows reproduction. 
- Could you expand section 7?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730189745449

thread_type:
official_review

<thread_object>
creator: reviewer_kn4j

<content>
summary:
This work revisits scaling laws and factors influencing the results reported in recent papers. The paper takes a "review approach", outlining some of the debates in recent literature as well as common strategies. To draw conclusions and emphasize their points, the last pages are dedicated to an in-depth analysis by the authors on small to moderate size transformers.

strengths:
I really like this paper and think it is of great value for the community to "recap" scaling law results and provide a critical discussion, complemented by experiments showing which factors matter when choosing a scaling law. It was really a pleasant read, quality of writing is good and the motivation clear.

weaknesses:
One could maybe claim the paper is not too constructive, as it shows that choices (optimizer, fitting method, lr annealing, data) matter when fitting a scaling law: there is no correct answer. However, this conclusion demystifies the topic, which I like very much: there is no magic, just common choices and "usual" results. This said, there are a couple of very minor points.

1) Proposing a checklist is helpful, but, as the authors themselves seem to hint, the number of factors to account for is potentially infinite. What about Adam beta2? What about weight decay? What about hybrid algorithms? What about qk norm and new tricks? The reality this paper points out is that, indeed, such choices matter, and I do not think any checklist can be conclusive.

2) section 7.1: why did you decide to set alpha=beta?

3) The paper lacks a bit of conclusions: what should researchers do? should we trust scaling laws? what are the things that hold true despite changing the setting? Is there some practical rule for scaling that holds approximately in your experiments? (would have been interesting alpha and beta)

typo spot: "was was" in the abstract, repetition.

questions:
-
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659645275

thread_type:
official_review

<thread_object>
creator: reviewer_vpgv

<content>
summary:
The authors survey a large corpus of papers that involve scaling laws, and find that many papers underreport necessary details for reproducibility, which they demonstrate with experiments that demonstrate a large variability depending on those exact choices of details. They propose a checklist for authors to consider when publishing scaling laws.

strengths:
The paper gives a good overview of many papers on scaling laws, and nicely categorizes the important steps: functional form, training setup, data(points) extraction, curve fitting. The checklist provides a clear way of reproducibility and quality assessment of scaling experiments. I think the topic of scaling law studies is important and relevant, and the writing is clear.

weaknesses:
The main concern for me is the following: what is the main goal the authors are trying to convey? To me, there are two obvious takeaways, which is 1) changes to the scaling law setup can change the results drastically, and 2) previous papers very much underreport crucial details. However, both of these things are rather clear already to the community and also illustrated by published papers: for example, point 1) is shown by Porian et al., and point 2) is a broader critique of reproducibility problems, which (unfortunately) is a generic problem. I do not see a clear and actionable interpretation beyond that. For instance, how do the different choices of fitting actually affect the scaling laws? (The assessment is mostly just “the results vary dramatically” — but how?) What should I as a researcher now do for my future scaling studies, having read your paper, beyond using the checklist? Are there clearly ‘wrong’ or ‘right’ choices? Was there a most predictive scaling law (e.g. when you leave out some experiments as a validation set)?

To be clear, I very much believe there is merit in a survey or pointing out these problems; as it stands, however, the paper is foremost “just” a survey, and I am not convinced this merits publishing at the conference.

Some additional comments: 
 * The paper template says ICLR 2024
 * The Figures are unfortunately of low quality (very pixelated), especially considering the fact that it’s natural to zoom in to compare the many lines and details. I suggest the authors include the pdf forms for proper rendering.

questions:
I have already listed direct questions in the section above, and I would be open to discuss this in the rebuttal. I hope the authors see the comments to be constructive, and can clarify or improve the distinctive value of the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731073929515

thread_type:
official_review

<thread_object>
creator: reviewer_phzz

<content>
summary:
This paper surveyed/ more than 50 papers about the scaling law of language models. Authors discussed different aspects of scaling law including fitting forms, model training, data extraction, and fitting optimization. Based on that, authors provided a checklist, which helps to transparent settings for reproducible results in future research. Experiments also were conducted to verify their replication and analyses.

strengths:
This paper has several strengths:
- This paper considers a timely topic, the scaling law of language models. Understanding this topic will help to effectively train LLMs, avoiding resource overuse. 
- Authors discussed the discrepancies in experiment settings of different papers and empirically verified it. Results are aligned with previous works.
- Authors open-sourced their code to reproduce results which benefits the community since the source code is usually absent from previous papers.

weaknesses:
Despite these strengths, this paper has several weaknesses:
- The scale of the model in experiments is not big enough. Authors consider only models with less than 400M params, ignoring the existence of larger models with billions of params. 
- The writing in some parts of the main paper causes confusion. E.g., Section 5 is about data extraction after training, I was confused by which kind of data could be extracted.

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732994331597

thread_type:
official_comment

<thread_object>
creator: area_chair_kcve

<content>
comment:
Dear reviewers, and authors:

Thank you all for engaging with the review process.
Several reviewers were uncertain about whether surveys / meta-analyses are appropriate for ICLR.
Several points of guidance below:
* Since there is no official policy on whether surveys are in-scope for ICLR, this matter should be handled on a case-by-case basis.
* Historically, ICLR has been welcoming to papers that are valuable and of interest to the community, even if they are outside the traditional scope of ML conferences. For example, the einops paper (ICLR 2022 Oral) was not a research paper: https://openreview.net/forum?id=oapKSVM2bcj
* The current paper appears to be closer to a “meta-analysis” work than a purely-survey paper. The authors have introduced a taxonomy of the major sources of inconsistency in the literature, and have also made efforts to independently reproduce experiments from prior works.
* Thus, we encourage reviewers to score this paper based on: (1) correctness, and (2) whether the ICLR community would benefit from reading this work. Note that surveys and meta-analyses cannot be judged by the same criteria as research papers — for example, novelty of claims or experiments is less of a factor. For surveys and meta-analyses, however, it is important that they are comprehensive within their stated scope.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work is a survey/meta-analysis of more than 50 recent papers on scaling laws in deep learning; it identifies key methodological decisions involved in scaling law research (functional form, model training, data, etc) and explains inconsistencies in the literature through this lens. The work also conducts its own experiments to explore how significantly conclusions can vary depending on experimental design choices.

Scaling laws is a very active research area, yet it has not reached consensus on methodological best-practices: different papers make slightly different choices of how to measure & fit laws, and these choices can sometimes affect the conclusions. Moreover, as noted by this work, many papers omit crucial experimental details which impact scaling laws. It is thus nontrivial for researchers to understand the current status of scaling laws by reading the literature, especially for those who are not aware of the “folklore”. 
Reviewers appreciated the work this paper does to clarify the field; some quotes from reviewers:
* “It is of great value for the community to "recap" scaling law results and provide a critical discussion” (Reviewer kn4j)
* “I very much believe there is merit in a survey or pointing out these problems; as it stands, however, the paper is foremost “just” a survey, and I am not convinced this merits publishing at the conference.” (Reviewer vpGv)

In the reviewer discussion, the main concern preventing a higher score was the fact that this paper is primarily a survey, and it is unclear whether ICLR accepts such works.
There is no official policy on whether surveys are in-scope for ICLR. 
However, historically ICLR has been welcoming to papers that are valuable and of interest to the community, even if they are outside the traditional scope of ML conferences (e.g. the einops paper, ICLR 2022 Oral: https://openreview.net/forum?id=oapKSVM2bcj ).
In this specific case, having read both the paper and the review discussion, I recommend acceptance for the following reasons:
* It is clear that the work is valuable for the community (especially newcomers), and clarifies the state of the active but not-yet-mature research field of scaling laws.
*  The work is not purely a survey; it includes aspects of meta-analysis wherein it identifies sources of inconsistencies in prior work. This goes beyond simply surveying known results.
* I believe this type of work should be encouraged at ICLR: deep learning is a fast-moving field, and there should be incentives to occasionally reflect on recent progress and coalesce results into a collective understanding.