Can Language Models Discover Scaling Laws?
Authors: Haowei Lin
Haotian Ye
Wenzheng Feng
Quzhe Huang
Yujun Li
Hubert Lim
Zhengrui Li
Xiangyu Wang
Jianzhu Ma
Yitao Liang
James Zou
Venue: ICLR 2026 Poster (2026)
Keywords: scaling law; agent; LLM
Primary area: foundation or frontier models, including LLMs
TL;DR: Using LLM agents to discover scaling laws for LLMs

Abstract: Discovering scaling laws for predicting model performance at scale is a fundamental and open-ended challenge, mostly reliant on slow, case specific human experimentation. To investigate the potential for LLMs to automate this process, we collect over 5,000 experiments from existing literature and curate seven diverse scaling law discovery tasks. While existing agents struggle to produce accurate law formulas, this paper introduces SLDAgent, an evolution-based agent that co-optimize the scaling law model and the parameters, enabling it to autonomously explore complex relationships between variables. For the first time, we demonstrates that SLDAgent can automatically discover laws that exhibit consistently more accurate extrapolation than their established, human-derived counterparts across all tasks.
Through comprehensive analysis, we elucidate why these discovered laws are superior and verify their practical utility in both pretraining and finetuning applications. This work establishes a new paradigm for agentic scientific discovery, showing that AI systems can understand their own scaling behavior, and can contribute novel and practical knowledge back to the research community.

Published as a conference paper at ICLR 2026
CAN LANGUAGE MODELS DISCOVER SCALING LAWS?
Haowei Lin 1.3+
Haotian Y
Wenzheng Feng"
Quzhe Huang
Yujun Li
Hubert Lim'
Zhengrui Li
Xiangyu Wang'
Jianzhu Ma
Yitao Liang's James Zou"
PPeking University
Stanford University
Wizard Quant
Tsinghua University
Website
HuggingFace
Github
ABSTRACT
Discovering scaling laws for predicting model performance at scale is funda- mental and open-ended challenge, mostly reliant on slow, case specific human ex- perimentation. To investigate the potential for LLMs to automate this process, we collect over 5,000 experiments from existing literature and curate eight diverse scaling law discovery tasks. While existing agents struggle produce accurate law formulas, this paper introduces SLDAgent, an evolution- based agent that co- optimize the scaling law model and the parameters, enabling it to autonomously explore complex relationships between variables. For the first time, we demon- strates that SLDAgent can automatically discover laws that exhibit consistently more accurate extrapolation than their established, human-derived counterparts across tasks. Through comprehensive analysis, we elucidate why these discov- ered laws are superior and verify their practical utility in both pretraining and fine- tuning applications. This work establishes a new paradigm for agentic scientific discovery, showing that AI systems understand their scaling behavior, and can contribute novel and practical knowledge back to the research community.
INTRODUCTION
Scaling laws are fundamental to the development of foundation models, including Large Language Models (LLMs) (Achiam et al., 2023; Bi et al., 2024), Vision-Language Models (VLMs) (Achiam al., 2023), and far beyond. A core objective is predict performance for models training at large scales, based on experiments at smaller scales (Kaplan et al., 2020). This predictive capability is crucial for determining optimal model configurations (Hoffmann al., 2022), selecting the best pre-trained models for fine-tuning (Lin et al., 2024), and identifying ideal hyperparameters such as batch size and learning rate (L.i al., 2025). The scope of modern scaling laws has expanded significantly beyond the classical Chinchilla Law (Hoffmann et al., 2022), which expresses pre- training loss L as function model size and dataset size using a power-law relationship (L = Ou + 0 + 0s/ Dot, More diverse and emerging scenarios, including scaling behaviors for supervised fine-tuning (Lin et al., 2024; Zeng et al., 2025), Mixture-of Experrss (MoE) mod- els (Krajewski et al., 2024), vocabulary size (Tao et al., 2024), domain mixture (Ye et al., 2024), and inference parallelism (Chen et al., 2025a), have emerged rapidly with the development of LLMSs.
One of the fundamental challenges in scaling law discovery (SLD) lies in the combination of sym- bolism and open-endedness. symbolic and mathematical law formula is required due to the uni- versality and generalizability, yet the underlying formulation space infinite and a priori knowledge of the problem is necessary. Consequently, most scaling law discoveries in new scenarios are manual and case- specific,,.. where human experts propose mathematical models to characterize the relation- ship between scaling variables within the system based on intuition and experience. However, this human-centric approach is often inefficient, labor-intensive, and sub-optimal, requiring numerous cycles of hypothesis generation and experimentation. Furthermore, it is constrained by humans' understanding and capacities in analyzing complex relationships that involve multiple variables.
Recent progress in LLMs has enabled a variety of practical agentic workflows (Yang et al., 2025; Soni et al., 2025; Zhang et al., 2025). Building on this, the notion of an "AI scientist" has emerged:
"Equal contribution. Correspondence: linhaowei 0pku. haotianye@atanoordccom

---

Published as a conference paper at ICLR 2026
Damiini2.5-Fiaaa
GPT-6
GerriiiS-Prr
Cludi-Sorral-45
Aug. R (nigher better)
0.8
0.748
0.85
Human
c4 miri
SLD/gent
Caude--aaiiu-.55
0.508
0.500
0.517
0.906
0.501
0.382
0.348
0.317
0.-
0.169
S SLD (OPEs)
Gonts (SPT (P
si SLDgger (od-mio))
SLDAgest (o-Somat-4.)
(0-3)-Pu))
Codeot (OPTS))
SuDggent (-Haan-dd
SLOApert (0.2 (ilesi)
Homan
Openraae (SP (SPTsS)
Cleude Cade (R-Sooul-d.))
swit Aggent (SPT.))
Clande Clssn (C-iaalts 4
Geniei (0:3
Creaxt (ot (ok-min))
Openlennts (oP
Aldte (SPt
Tarmsna 4 (GPP (SPPE.))
Geevin (o- Flash)
SL
Figure I: Overview of the scaling law discovery performance on SLDBench of various Al agents, our proposed SLDAgent, and human baselines. Performance is measured by the average prediction score (R²) across all tasks, where higher values indicate better fit on unseen test sets Our SLDAgent (hatched bars) consistently enhances the performance of its underlying LLMs, achieving top ranks and outperforming all other baseline agents and human experts. Breakdown of scores individual tasks can be found in Sec. 4.
systems that generate hypotheses, run analyses, and iterate with minimal human input (e.g., dis- covering new drugs, developing machine learning models, drafting papers) (Chan et al., 2024b; Lu et al., 2024; Swanson et al., 2024). Yet these systems struggle with open-ended problem formulation, principled experiment design, and robust long-horizon execution (Xie et al., 2025). Scaling law dis- covery is precisely such an open-ended scientific discovery task, and as we empirically demonstrate below, existing agentic systems fall short in designing scaling laws that are superior to human- derived counterparts. This motivates us to ask a pivotal question: can LLM-based agents discover the scaling laws that govern their own behavior more efficiently and accurately than humans?
To rigorously answer the question, we curate SLDBench, a comprehensive scaling law discovery testbed constructed from over 5,000 LLM training experiments from existing scaling law discovery literature. SI.DBench contains eight distinct tasks, where each task requires identifying a symbolic expression that maps input variables to target prediction, such that its extrapolation accuracy on an unseen, scaling test set is accurate. The experimental results are given, without needing to formulate the target variables or execute the experiments. Unlike conventional benchmarks with binary success criteria, SLDBench emphasizes open-endedness: the objective is clear, continuous, and computed directly from extrapolation data (e.g., R²) without any learned reward model, but the ground solution remains unknown to human experts, and minor gain could result in substantial benefits in practical model developments. Because existing laws are designed based on iterative human research, they set high standard that agentic systems lag behind on difficult SLD tasks. However, these same human-derived laws can also give poor predictions under challenging scenarios.
To fully uncover the capabilities of LLMs, inspired by the human research process and recent studies on evolution-based agents (Gao et al., 2025b), we design an LLM-based agent named SLDAgent. Intuitively, SLDAgent keeps pushing the boundary of its solution evolutionarily by co-optimizing two functions based on previously proposed solutions: (1) a generator that proposes a law expres- sion and (2) an optimizer that fits the coefficients on seen data efficiently and robustly. This co- optimization process will be iterated for certain steps and the optimal proposal will be returned.
Our comprehensive evaluation reveals that SL.DAgent achieves SOTA performance across all SLD- Bench tasks, consistently outperforming human experts and existing agent baselines, and can boost the capacities of all LLMs it is paired with. This superiority stems from the agent's 's ability to evolve and optimize the law formulations. As case studies, its discovered laws introduce more conceptu- ally sound principles, such as unifying data sources under a single scaling exponent, and ensuring principled asymptotic behavior be accurate. The practical utility of these laws is demonstrated in two critical applications: enabling direct, analytical optimization of pre- pre-training hyperparameters to near-optimal state, and successfully identifying the best fine- fine-tuning checkpoints.
2

---

Published as a conference paper at ICLR 2026
In summary, this work establishes new paradigm for agentic scientific discovery, demonstrating for the first time that an Al agent can automate the discovery of scaling laws superior to human- derived counterparts. We introduce SLDAgent, novel evolution-based agent that achieves state- of-the-art, superhuman performance, alongside S1.BBecch, the first comprehensive henchmark for this task. Through rigorous analysis, show that the laws discovered by SL.AAeett are not just more accurate but fundamentally more principled, and we validate their practical utility in various applications. We have open-sourced our entire framework, providing tangible path toward AI systems that can autonomously understand and accelerate their own development.
2 SLDBENCH
General formulation. Scaling law discovery (SLD) requires the prediction of scaling law, in- cluding its mathematical formula and instantiated parameters, based on give set of observed trails. Specifically, the input of the problem is dataset
D {(x,, ji, v:))
where each of the m data contains:
(1) a vector of n feature variables x, = (xx)) ,....,,,, x (n)) that vary across trials (c.g., parameter count, batch size, training token length);
(2) vector (usually I-dimension) of target y, to be predicted (c.g., loss, perplexity, accuracy), this the value of interest that the proposed scaling law should output:
(3) a control index j: E C identifying the experimental setting (e.g., a particular model). All settings are required to fit the same scaling law model, but the parameters for different settings can different.
The output of SLD should be: (1) a symbolic expression fo x x y. mapping input features to a target prediction ú using parameters 0, and (2) group of parameters {0, Ieec that are fitted specifically for each control setting. The objective of SLD that for any setting j E C, the proposed law parameterized by 0j can predict the target y = Sa (x) accurately in unseen, extrapolated x.
Example. Consider classical pre-training scaling laws, where x = DN denotes model size and dataset size, and the target is the training loss L. Different control settings correspond to different corpora (e.g., books, code, news) architectures (e.g., decoder-only, incoder-decodee RNN). Human experts have proposed the formula T. 0g + 0g /N/N% + 0g/ Dd, where N, D is the input features, and f {0,, 01, Og 0g 04) coefficients that be fitted using standard optimization methods (e.g., BFGS (Fletcher, 2000), SGD (Amari, 1993)) for each j.
Datasets. Following the above paradigm, we cu- rate concrete scaling law tasks from comprehensive surveys and literature such as Sengupta et al. (2025) and filter the ones with available open-sourced data. total of eight tasks are included in SL.Bencc (we will continuously expand our dataset), and detailed statistics are presented in Table I: (1) Parallel Scal- ing Law (parallel) examines the effect of paral- lelism P and model size N on the language model- ing loss. This method involves creating P augmen- tations of an input and aggregating the correspond- ing outputs to generate a final response, conceptu- ally similar to the Best-of-N approach (Chen et al., u_shape
Table 1: Tasks of SL.Deench and the number of (un)seen datapoints. T. loss; N: #parameters; D: dataset size; r: domain proportions; E: #experts; #unique tokens; I: learning rate; & batch size;
B: brier score; C: FLOPs.
Task Scenario
Seen
# Unseen
Target
parallel vocab_size _ize
12
L(N,P)
1,080
120
I.N.V.D)
sit
504
42
L(D)
domain_mix
80
24
(La(r))
moe
193
28
L(N,
constraain
161
21
(.U)
Irábez
2.702
117
L,
389
127
B(C)
2025a). (2) Vocabulary Scaling Law (ocaa_sizee) models the unigram-normaliinn size V. and dataset size D (Tao et al., 2024). (3) Supervised Finetuning Scaling Law (sft) models the finetuning loss based on the dataset size D (Lin et al., 2024). (4) Domain Mixture Scaling Law (domain mix) models the pre- pre-training loss for each domain based on its proportion in the training mixture (Ye et al., 2024). (5) Mixture of Experts Scaling Law (moe) models the loss in relation to
loss function of the non-vocabulary model size N. vocabulary
3

---

Published as a conference paper at ICLR 2026
Expression subroutine
scalinng law, (in,, 8):
X_ X_aug = np. (Xs (X, 9, 1., acis=1)
X_aug. shape[1] [1]
Init aniaaalizza alaiziztionn
Database
LLM
Evaluator
e, p 0[:n], 8[m:]
return rp. np.sun(c X_aug * P. axis:1)
Optimization subroutine
parent_ subrotines, inspirations = database.
prompt but Id proppp (parent___t_bbraaa subrouutiees inspirations) difr
def fit__sclling Lam(X, Y):
wwt..generaee(prone
= [1 +1 lanbda (x,)): (x-y)+
shild__sbroutties
diff)
nase evaluuttor exocuto subroutinas)
lanbda 8: ise(Yy waw. (xn,
database.add (chilld subrout riss)
e_init rp.random..and22 n)
return winferize(loss, 8. init nethode BFOS
Program Evolution
The SLDAgent System, Left: Each candidate program consists of two subroutines: an Expression that defines the symbolic model (x),, and an Optimization routine that fits its parameters 0 to data. Right: The evolutionary loop. An LLM mutates a parent program sampled from database. The resulting child evaluated and inserted back into the database, continually improving the population.
Figure 2:
the network size N and the number of experts E (Krajewski et al., 2024). (6) Data Constrained Scaling Law (d_ consttraii) models the pre-training loss as function of network size N, dataset size D, and the number of unique tokens U in the dataset. (7) Learning Rate and Batch Size Scaling Law (1r&bsz) models the pre-training loss based on learning rate I, batch size b, dataset size D, and network size N. This law is adapted from the Step Law (Li et al., 2025) that predicts the optimal learning rate Ir* and batch size bsz" given D and N. (8) U-shaped Scaling Law (u_ (u_shape) models momoonotoni function of compute (measured in FLOPs C... This task is adapted from Wu & Lo (2024) as an adversarial extrapolation scenario.
the performance (measured by Brier score B)
Execution environment and metrics. For each task, we hold out extrapolation test set from the dataset by selecting data corresponding to the largest model or dataset sizes. This closely im- itates the practical scaling law cases. The evaluation environment is sandbox terminal with a minimal set of pre-installed Python packages (scikit-learn, pandas, and datasets). Agents have no network access during evaluation. The agent's goal is to produce Python script named law.py, which must contain the discovered scaling law defined within function with a specified signature. Upon the agent's 's completion, our evaluation script automatically computes the coefficient of determination (R²) on the test set. A perfect law should give R² close to 1. Several tasks involve multiple control indices j. requiring the prediction a scaling law with a distinct set of coefficients for each experimental setting j E C. These include parallel (across two pre-training datasets), sft (across three SFT datasets and pre-trained models), domain fomain_mix mix (across four different mod- els), and shape (across nine datasets). Details can be found in Sec. A.2 and B.
Specialty of SLDBench. The symbolism and open-endedness of SLDBench make it distinct from existing benchmarks in various ways. Traditional symbolic regression tasks (Udrescu & Tegmark, 2020; Shojaee et al., 2025) focus on rediscovering pre-known formula, thus they are most synthetic and not truly open-ended; Conversely, while agentic benchmarks like MLE-Bench (Chan et al., 2024b) involve open-ended problem-solvine to produce generalizable symbolic discovery. SLDBench provides rigorous testbed for deeper form of agentic scientific intelligence by uniquely combining the challenge of symbolic reasoning with demand for abstract, multi-context generalization.
their goal is to automate engineering workflows, not
3 SLDAGENT: EVOLUTIONARY SOLVERS FOR SCALING LAW DISCOVERY
Intuitively, SL.Beench features a series of open-ended scientific discovery problems where the ob- jective is clear, computable, continuous, and unbiased (improving R² to 1 in our case), while the perfect solution remains unknown even to human experts. Similar to other problems that fall into the same category, such as optimizing matrix multiplication efficiency (Fawzi et al., 2022) im- proving the Ramsey number bound (Campos et al., 2023), the solution of these problems requires generations of efforts that continuously improving upon existing proposals. Consequently, a single round of proposal, no matter how impressive it is, is insufficient to push the limits toward the optimal solution, and evolution-basee systems specifically for SLD should be adopted. This motivates us
4

---

Published as a conference paper at ICLR 2026
Table 2:
Performance on SLDBench for agents using GPT-5. Scores are R averaged over five runs. The best and second-best scores for each task are highlighted. "NA" indicates no valid output.
paralleif vocab s size"
SFT1
domain_ nixt moet d. constrain" Irābszt
u. shapet
Avg. R
Aider
0,991
0.132
0.131
0.514
0.119
0.718
-0,659
-0.474
0.184
Terminus-2 1.000
0.533
0.114
0.502
0.186
0.366
-0.754
-0.604
0.168
Mini-SWE-Agent
0,997
0.554
0.853
0.873
0.352
0,903
-0.269
-0,491
0.471
OpenCode
1.000
0.889
0.845
0.960
0.240
0.920
-0,368
-0.480
0.501
OpenHands
1,000
0.182
0,640
0.899
0.466
0.534
-0,909
-0.278
0.317
CodeX
0.999
0.977
0.855
0.933
0.649
0.763
-0.039
-0.740
0.550
Goose
1.000
0.962
0.899
0,944
0.813
0.894
0,280
-0.232
0.695
SIDAgent
1,000
0.987
0,993
0.988
0.773
0.944
0.604
-0.305
0.748
Human
1.000
0.966
0.957
0.671
0,703
0,911
-0,076
-1,000
0.517
to design SLDAgent, evolutionary coding agent (Gao et al., 2025a) that discovers scaling laws by searching over the space of executable programs.
Overview. As illustrated in Figure 2, the core design of SLDAgent is the evolution of a pair of sub- routines, (Expression, Optimization), which together define the model and the fitting function. We adopt this co-evolution design due to the nature of the SLD problem and that a general, problem- agnostic evolution (Novikov al., 2025) faces challenges in fully exploiting the LLM capabilities (see ablations in Sec. B.6). Both functions are implemented by SI.Dggett. The Expression(x. 0) implements a function So x + V. mapping input features x to prediction ý using parameters 0, while the Optimization routine finds the best-fit parameters Ô arg ming C(y,fo(x) for given loss function L, producing a parameterized scaling law ready for extrapolation.
Evolution. The SLDAgent is provided with the training set in its working directory, and it be- gins with baseline program pair, typically a power-law function for Expression and a standard BFGS optimizer for Optimization. This program executed once and stored into the evolutionary database (initially empty), together with its fitness score. At each evolution step, parent program selected from the database via a probabilistic mixture that prioritizes high-scoring programs (ex- ploitation, 70%), diversity (20%), and the top performers (elitism, 10%). This parent, along with several selected berh-performing "inspiration" programs, is contextualized into a structured prompt (see Sec. C.1 for details). The LLM takes in the input and proposes modification of parent's 's code, for instance, by altering the law form, changing the optimizer, or tuning global variables. The re- sulting child program is executed: its Optimization routine fits the Expression dataset, and the R² score on training dataset will be computed. Finally, the new child and its score are added to the database for subsequent generations. The evolutionary loop terminates after fixed budget of iterations. The program with the highest score in the final database is returned as the proposed law. The test set is left untouched throughout the whole evolution.
the given
Implementation remarks. To guide the LLM toward effective laws, the prompt includes task context, selected inspiration programs, and data statistics such value ranges, mean, and variance. Following AlphaEvolve (Novikov et al., 2025), we increase sample diversity and prevent prema- ture convergence by using multi-island strategy (five islands) and MAP-Elites, which structures the population by fitting score, complexity, and novelty. The system is built on the OpenEvolve framework (Sharma, 2025); see Sec. C.2 and D for full prompts and configurations.
4 EXPERIMENTS AND ANALYSIS
4.1 EXPERIMENTAL SETUP
We benchmark SLDAgent again human and existing agents SLDBench. Our comparisons along two perspectives: we first fix the LLM and compare SLDAgent against eight agentic systems, and then we compare SI.DAgett with different LLMS against their native CLI systems.
5

---

Published as a conference paper at ICLR 2026
Table 3: SLDBencc performance provider-specific agents together with reference rows for SLDAgent paired with the corresponding provider models. "G-2.5-Flash" denotes Gemini-2.5-Flassh "G-3-Pro" denotes Periiii---Pro-Previeee "C-Haiku-4.55" denotes Claude-Haiku--e and "C-Sonnet-4.5" denotes Claude-Sonnee 4.5. Scores report R², averaged over five runs. Bold denotes the best value within each model family.
parallelf vacab_aizzs _aizz
SFT1
domain_mixt
noet c_costrriint Irābsz"
n_thaape
Avg. R²
Gemini-CLI (G-2.5 Flash)
0.200
0.485
0.787
0.530
0.273
0.011
-0873
-0.994
0.077
SLDAggett (G-2.5 Flash)
1.000
0.960
0,983
0.991
0.810
0.936
-0.871
-0.758
0.506
Gemini-CLI
0.600
0.578
0.462
0.978
0.833
0.783
-0.332
-0.847
0.382
SLDAgent (G-3-Pro)
1.000
0.982
0.992
0.984
0.766
0.853
0.513
-1.000
0.636
Clande Code (C-Haiku-455)
1. 1.000
0.895
0.894
0.905
0,394
0.778
-0.511
-1.000
0.419
SLDAgent (C- -5
1.000
0.969
0.965
0.980
0.705
0.947
-0.557
-0.754
0.519
Clande Code (C- S.55
0.998
0.962
0.916
0.971
0.858
0.916
0.846
-1.000
0.472
SI.DAgent (.-Sonnet-445)
1.000
0.983
0.981
0.985
0.846
0.959
-0.514
-0.522
0.590
CodeX (o4-mini)
1.000
0.861
0.883
0.553
0.468
0.805
0.773
-1.000
0.349
SI.DAgent (o4-mini)
1.000
0.972
0.994
0.989
0.773
0.917
0.611
-1.000
0.657
CodeX (GPT-5)
0.999
0.977
0.855
0.933
0.649
0.763
-0.039
-0.740
0.550
SLDAgent (GPT-5)
1.000
0.987
0.993
0.988
0.773
0.944
0.604
-0.300
0.748
Human
1.000
0.966
0.957
0.671
0.703
0.911
-0.076
-1.000
0.517
Agent baselines. We use 04-mini as the default LLM for all folowing bascline agents: (1) Aider (Aider-AI, 2025), a CLI pair programmer that edits files in a local Git repository, shows diff's and auto-commits changes; (2) Terminus (Terminal- Bench Team, 2025), a native Python agent that controls a dockerized environment solely via an interactive tmux session by sending keystrokes, enabling fine-grained terminal control; (3) OpenHands (Wang et al., 2025), a general- de- veloper agent that can edit code, run Bash, and perform file operations in a full agentic development environment (v0.55.0); (4) Codex (OpenAI, 2025), OpenAl's cloud notwaare-engineering agent that executes parallel tasks in isolated sandboxes preloaded with the target repository; (5) Open- Code (sst, 2025), an open-source, terminal-based assistant oriented toward local coding workflows; (6) Goose (Block, 2025), local, extensible engineering agent that writes and runs code, orches- trates workflows, and integrates with external tools/APIs; (7) Mini-SWEEAgeet (SWE-agent Team, 2025), a ~100-line agent tailored to repository-level edits and SWE-bench-style tasks (Jimenez et al., 2024a); and finally, Human, the law discovered in the original literature, serving as a strong reference baseline. We refit some laws on our seen data split and recruit a human expert to propose a law for Irābsz, for which no law exists. Further details are presented in Sec. A.3.
LLM baselines. We compare SLDAgent instantiated with different LLMs against the following native command-line code agents: Gemini-2.5-Flasn and CLI (Google, 2025); Blaude-Sonnet-4 and Claude-Haiku-44 using Claude Code (Anthropic, 2025); and 04-mini and GPT-5 using Codex (OpenAI, 2025).
using Gemini-
Evaluation metrics. We report the average R² over 5 runs for each method on the test set of each SLD task in Tables 2 and 3, and postpone the standard deviations in Sec. B.2. Since R² is normalized and has an upper bound 1, the values are also comparable across tasks. We clip the scores between [-1, I] when averaging across tasks to avoid significant influence from single task failure.
4.2 RESULTS ANALYSIS
Superiority of SLDAgent against baseline agents. As shown in Table 2, when benchmarked on the GPT-5 model, SLDAgent achieves an average I22 score of 0.748, outperforming all competing agent architectures, including the next-best Goose (0.695) and CodeX (0.550). Notably, SLDAgent matches the human baseline on the easy parallel task (R² 1.000) and exceeds human perfor- mance on all remaining tasks, yielding the best overall average score (vs. Human 0.517). This result underscores that the agent design (rather than the underlying LLM alone) is decisive: other agents paired with the same GPT-5 model still fail to consistently reach expert-level performance across the full SLDBench task suite.
Pushing the performance toward superhuman levels across model families. We further pair SLDAgent with a range of provider models and compare against the corresponding provider-specific

---

Published as a conference paper at ICLR 2026
CLI agents in Table 3. Across all evaluated model families, SLDAgent consistently improves the average R over the native CLI systems, with gains ranging from +0.100 +0.429 (Gemini-2.55-llash Equipped with GPT-5, SLDAgent matches exceeds the human expert on every task (matching on parallel and exceeding on the other tasks) and achieves the highest overall average (R² 0.748). We note that while SI.DAgent strongly improves performance for smaller models, some difficult tasks (c.g., 1r&bsz for Gemini-2.5-Flash / Claude-Haiku-4.3 can still remain below the human reference, highlighting remaining model-capacity bottlenecks.
to
Task difficulty analysis. The tasks in SL.Beench span wide spectrum of difficulty (Table 2). parallel is easy, with most agents achieving near-perfect R contrast, 1r&bsz and u_ shape particularly challenging: many baseline agents exhibit strongly negative R² (and the human reference also poor on u_shape,, indicating frequent failure modes and high variance. Tasks such moe and d_ connstrain also differentiate agent capabilities: while several systems perform reasonably on some tasks, SLDAgent is the only method that is consistently strong across nearly the full suite, with u_ u_shape remaining the primary outlier. For many SLDBench tasks, the ground-truth scaling laws remain non-obvious, and the prevalence of negative scores for weaker agents suggests that robust discovery requires both strong models and effective agentic search.
4.3 LAW FORMULATION ANALYSIS
An intriguing study is how SLDAgent- laws is different from human-propone laws, and why this leads to better performance. the easy tasks like parallel where simple Chinchilla- like law is sufficient, the slight performance difference is mostly due to optimization choices. While in other tasks, the scaling law formulas are different. Below, we choose the sft and moe tasks to analyze their differences carefully. We postpone the analysis of improved optimization in Sec. B.6. The LLLLAAgenddisoveeree laws analyzed below are the ones with the highest observed data across all LLMs and random seeds.
The sft task. We model the loss I, as a function of SFT dataset size D. Prior work (Lin et al., 2024) and SLDAgent consider the following saturated power-law forms:
do
Po
Human law:
L - 02
SLDAgent law:
L= 1 (D/0g))
D 03'
While the two parameterizations have comparable expressive capacity (both describe diminishing returns toward an asymptote), they differ in how the "effective data size" 03 is incorporated. In the human law, f3 enters additively alongside Dd, which implicitly assigns 0g the same units as Dd and makes its interpretation depend on the exponent. In contrast, the SLDAgent law uses the dimensionless ratio 1D/0a)², so 03 retains the natural unit of dataset size and directly sets the characteristic data scale at which the curve transitions from steep improvement to saturation. This dimensional consistency yields parameters with clearer practical meaning (e.g., how much pre-training knowledge effectively transfers into the SFT stage) and typically improves downstream usability of the fitted law, as analyzed in Sec. 4.5.
The moe task. The moe law describes the loss L function of the number of experts (E) and the number of parameters (N). Krajewski et al. (2024) proposes the following human-designed predictor:
Human law: L cxp( log N + 8, log È + 02 log N log E + 0z):
where É is parameterized transformation of E for numerical stability. In contrast, SLDAgent discovers a more accurate and better-behaved functional form:
81 N
SLDAgent law: L = 1 03
E + 0g No.600 + 06.
This discovered law cleanly separates (i) a primary strameter-driven scaling term N whose contribution is attenuated by the expert-dependent factor + FyE",, capturing diminishing returns from adding experts, and (ii) an additional purameter-dependeer component 0, 0.6602 that follows the same exponent family but with reduced effective scaling rate. Importantly, the additive constant
7

---

Published as a conference paper at ICLR 2026
Table 4: LLM selection results (PearCorr, RelAcc) three datasets in percentage. The best result within the same dataset is bold font, and second best result underlined.
Dataset
Metric
ModelSize
ZeroShot
RectifiedLaw
NIPMettrics
SubTuning
ST.AAgettLaww
PearCorrt RelAcet
-2..8
-12.8
60.1
76.1
79.4
84.6
FLAN
75.6
82.3
100.0
84.7
71.8
100.0
PearCorrt
38.0
6.1
76.4
78.9
90.3
83.6
Wikitext
RelAuct
68.5
84.4
44.0
87.5
100.0
100.0
Gigaword
PearContt
-27.4
-53.2
66.7
80.1
86.3
94.8
RelAcet
77.3
83.3
78.3
86.8
100.0
100.0
8g acts an irreducible loss floor, ensuring realistic extrapolation. In particular, as E - oo the first term vanishes and L approaches 0g N0.602 + 0g: and when the fitted exponent satisfies the typical scaling regime 02 < 0, both N and No.662 decay with increasing N, L further converges to the finite limit 0g as N + cc. The human law, being an exponential of log-linear terms with an interaction log N log E, is far more sensitive under extrapolation and can yield qualitatively different asymptoties depending on fitted signs (e.g., the interaction coefficient is positive, the exponent grows without bound as scale increases). Empirically, the discovered SLDAgent form also fits the observed scaling substantially better than the human proposal (as reflected by the higher R² 0.891 vs. 0.732).
4.4 APPLICATION I: HYPERPARAMETRE OPTIMIZATION FOR PRE- TRE-TRAINNNG
A critical challenge in pre-training LLMs is identifying the optimal learning rate (Ir) and batch size (bsz) for a given model and dataset. Conventional methods often involve extensive hyperpa- rameter sweeps on smaller scales to fit a predictive scaling law. In this process, only the single best-performing configuration from each sweep is typically used to fit law for the optimal hy- perparameters (Ir", bsz") as a function of model size (N) and dataset size (D). For instance, the original StepLaw (Li et al., 2025) ran approximately 3,000 experiments but used only 17 optimal points to fit its law (Ir* = ea D bsz" dD"), discarding the vast majority of experimental data.
In contrast, the SI.Agent discovers a compre- hensive law that models the final loss I. as function of N. D. (r, and bss. This explicit functional form allows us to find the optimal hyperparameters analytically. By treating the loss as a surface over the hyperparameter space, locate the minimum by setting the partial derivatives /0r, aL/Obsz to zero, and solv- ing the resulting system of equations. This calculus-based approach yields a closed-form solution for the optimal /r* and bsz" as func- tions of N and D. The full derivation is detailed in Sec. B.4. We then apply the derived formula on the extrapolation setting following StepLaw: a IB-parameter model trained on 100B tokens, which is at least twice as large as all observed experiments. As shown in fig. 3, the analyti- cally derived optimum is marked by the blue star (Ir 1.77 10-3 3, bsz 402). Training at the nearest available grid point (Ir = 1.95 x 10-3 bsz = 384) yields an actual validation loss of 2.0776, which is very close to the true optimal loss of 2.0762 marked by the red star (relative error 0.067%). This demonstrates that the discovered law is accurate enough to analytically derive near-optimal hyperparameters for a significantly larger scenario, validating its predictive power in practice.
2:300 130
235519
0:377
2049
2.70
127
1199
1472
43809
LLM
ALLOON
1524
2.0700
21335
22000
2,0075
2.2977
2,200 2.00T 2.0000 2,000 2,294
Batchh Size
(Cptiimi Jowy
312
384
2.000
2.280
254
0:134
2.1000
2.357
2,3059
2,0000
2,007
2.300
2,000
2,0034
2,0016
5.0000
CL1244
328
0:1251
2:123
2.1310
2.9900
2.0923
2.7900
2,3055
21000
2:1994
1399
2.1T1
0,200222
0.20323
6,00224
8,000543
00048
500999
5,500077
6,000399
0,000553
500763
50000
500534
Learning Rate
Figure 3: Ground-truth validation-loss heatmap for a IB parameterr LLM trained on 100B tokens.
4.5 APPLICATION II: LLM SELECTION FOR FINE-TUNING
Another challenge for LLM practitioners as discussed Lin et al. (2024) is to select the optimal pre-trained model from model hubs like HuggingFace to fine-tune for specific downstream task. While fine-tuning all candidates on the full dataset and select the best in hindsight is optimal, it is often prohibitively expensive. A more practical strategy is to predict final performance based on
8

---

Published as a conference paper at ICLR 2026
small-scale experiments. Following the methodology of Lin et al. (2024), we fit the sft law small data subset (6.25% of the full dataset) and use it to predict performance on the full dataset, and select the best model based the prediction.
We evaluate the law discovered by SLDAgent (as presented in Sec. 4.3) against five established baseline laws from Zeng et al. (2025): (1) RectifiedLaw (Lin et al., 2024) uses the sft scaling law discovered by human; (2) NLPMetrics (Yang et al., 2022) uses proxy generalization metrics for selection; (3) SubTuning, uses the performance of subset fine-tuning as selection score; (4) ModelSize uses logarithm of the number of model parameters for selection; (5) ZeroShot uses the zero-shot performance on the task data as predictor. LLM selection performance is measured using two metrics from Lin et al. (2024): the Pearson correlation coefficient (PearCorr), which assesses a method's s ability to rank models correctly, and Relative Accuracy (RelAce), which measures the performance of the selected model relative to the best and worst possible choices.
Table 4 shows the results of selecting the optimal LLM from candidates on downstream datasets. On average, SLDAgentLaw achieves a perfect RelAcc of 100.0%, indicating it consis- tently identifies the best-performing LLM across all three datasets, while other methods cannot. Furthermore, it obtains the highest average PearCorr of 87.7%. This highlights its superior abil- ity to accurately rank the candidate models, and show that SLDAgent can discover robust laws for computationally efficient model selection.
RELATED WORKS AND DISCUSSIONS
Coding agents and benchmarks. Terminal-centric coding agents combine program synthesis with execution, editing, package management, and verification loops (Yang et al., 2024; Wang et al., 2024), enabling end-to-end automation of development workflows and research pipelines. Recent systems (e.g., Codex (OpenAI, 2025), OpenHands (Wang et al., 2025), Claude Code (Anthropic, 2025), Gemini CLI (Google, 2025)) have shifted from single-shot code generation to iterative, tool- augmented planning and self-repair (Shinn et al.. 2023), handling tasks such as dependency reso- lution, unit-test authoring, and regression debugging (Google, 2025; Yang et al., 2024; Hu et al., 2025). In parallel, agentic benchmarks have moved beyond toy problems toward realistic, repo- level software engineering (e.g., SWE--ench and variants) (Jimenez et al., 2024a; OpenAI, 2024; Jimenez et al., 2024b; Rashid et al., 2025), ML engineering and ML.Ops tasks (e.g., ML.E-oriented suites) (Chan et al., 2025; Huang et al., 2024), and ssorithm-optiiiization settings that require measurement-driven iteration (Romera-Paredee Paredes et al., 2024; Mankowitz et al., 2023; Sun et al., 2025). Our benchmark follows this trend: it evaluates agents that plan, execute, and reflect inside a real development environment (All Hands AI, 2025; Jimenez et al., 2025) and targets problems of interest to both industry practitioners and the scientific community rather than synthetic puzzles.
Scientific discovery AL growing body of work positions AI agents as assistants or even au- tonomous actors in the scientific discovery loop, as surveyed by recent overviews on scientific AI and self-evolving agents (Gridach et al., 2025; Gao et al., 2025b). Early benchmarks in this area largely emphasized Kaggle-style machine learning engineering (e.g., Science bench) (Chen et al., 2025b; Chan et al., 2024a), while another line of work assessed reproducibility and research implementation fidelity (e.g.. PaperBench, nanoGPT reproduction tasks) (Starace et al., 2025; Zhao et al., 2025). Closest to our setting is LLM-SRBench, which probes symbolic regres- sion for disciplines such as physics, materials, and biology, typically using synthetic or augmented datasets (Shojaee et al., 2025). Our work differs in both settings and domain: we introduce SLD with explicit control groups, emphasize open-ended, real-world experimental traces over synthetic datasets with pre-known laws, and firstly unlock symbolic regression in the domain of AI research.
MLE-
Neural scaling law. Scaling laws have been documented across multiple axes such as: (i) model- size scaling, where pretraining loss and downstream accuracy follow smooth power-law trends as parameters increase (Kaplan et al., 2020; Hestness et al., 2017; Henighan et al., 2020); (ii) data scaling, in which performance improves predictably with the number and diversity of tokens (Ka- plan et al., 2020; Hestness et al., 201 7); (iii) transfer scaling, where downstream performance scales with pretraining compute and data even with modest fine-tuning (Hernandez et al., 2021); (iv) in- context learning scaling, where few-shot and reasoming capabilities strengthen with both parameter count and pretraining tokens (Brown al., 2020); (v) emergent or phase- transition effects, in which

---

Published as a conference paper at ICLR 2026
discrete capabilities appear once scale thresholds crossed (Wei et al., 2022); (vi) alignment and preference-mon monel scaling, tying cowtuuctioo-oolowinn and RLHF quality to the volume and qual- ity of preference data (Ouyang et al., 2022; Bai et al., 2022); (vii) multimodal scaling, extending power-law trends to vasion-language models and revealing data-mixture sensitivity (Radford et al., 2021; Jia et al., 2021; Alayrac et al., 2022); (viii) architecture-awe
scaling, including sparse MoE regimes that increase effective capacity at similar training FLOPs (Fedus et al., 202 and (ix) data quality scaling, which quantifies how higher-quality or more task-aligned tokens substitute for raw volume in determining performance (Penedo et al., 2023; Gadre et al., 2023). This is an actively grown up area and SLDBench will continue to collect more tasks in the future.
DISCUSSION, LIMITATIONS, AND FUTURE WORK
Our work presents a significant step toward agent- based scientific discovery, yet several limitations remain that chart the course for future research:
1. Benchmark Expansion. SI.BBench currently comprises eight scenarios. While these cover varied difficulties, we aim to expand the benchmark with additional tasks to probe a broader spectrum of scientific challenges. Furthermore, we plan adjust the seen/unseen splits in exist- ing scenarios to create more challenging extrapolation settings.
Rediscovery vs. Novel Discovery. The present iteration emphasizes rediscovering known scal- ing laws to verify capability. A pivotal next step is applying our framework to real-world, unex- plored datasets to facilitate genuine novel discoveries. Additionally, SLDBench currently focuses on law discovery (finding formulas for fixed variables). We view variable discovery proposing which new metrics (e.g.. "data mixture") to measure- as a complementary challenge. Extending framework to jointly discover both variables and laws is an ambitious direction for future work.
Overfitting and Self- Verifiiiica,.. critical limitation is the risk of overfitting to the training data. Unlike ceneral-purpose coding agents (c.g.. CodeX), which offer the flexibility to explicitly create train/validation splits or implement custom self-verification loops, SLDAgent currently optimizes directly against the provided data. Evaluating the true validity of discovered law without access to extrapolation test data is inherently difficult. Future iterations must enhance the "self-verrffcation" capabilities of the agent to distinguish between robust physical laws and statistical artifacts.
Specialization vs. Generalization. While SLDAgent's 's specialized design yields ttte-of--the-art performance, this raises open question: general-purpose agents learn to specialize? An exciting research direction investigating methods for general agents to autonomously adapt or learn from the trajectories of successful specialized agents like SI.DAgent.
Summary. In summary, this paper addresses the challenge of scaling law discovery a tradition- ally slow, human-driven process-ae by investigating whether an Al agent can automate this open- ended scientific task. We introduce SLDBench, the first comprehensive benchmark for this prob- lem, and propose SLDAgent, a novel, evolution-based agent designed to navigate the vast symbolic search space of potential formulas. Our extensive experiments demonstrate that SLDAgent achieves superhuman performance, discovering laws that are not only more accurate in extrapolation but also more conceptually sound than established human-derived formulas. By validating the practical util- ity of these discoveries and open-sourcing our framework, this work establishes a new paradigm where AI systems autonomously understand and accelerate their own development.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Aider-AI. Aider: pair programming in your terminal. https://github..cm wttp://tth...oo.ie/eereeeeeee
aider, 2025. GitHub repository; accessed 2025-09-11. 6
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https: /ttpss///aaxxvv ht/pv.xxvvvvvo.... wwtgg/bs///aa 14198. 10
All Hands AI. Custom sandbox openhands docs, 2025. URL httpt://docs. www.:oons...l-.nnn....aaaaaaaa.. dev/ usage/hou-to/ custom- custooo-aandooo-o guide. 9
Shun-ichi Amari. Backpropagation and stochastic gradient descent method. Neurocomputing,
(4-5):185-196, 1993. 3
Anthropic. Claude code overview. https://docs. anthropic. wew///oos/caadeeeeeeeeeeeeeeeeee overview, 2025. Documentation; accessed 2025-09-11. 6, 9,
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. URL https: / /awpvv.ivvvv///////////a///aaava org/ abs/2204 2etp/22240088.. 05862. 10
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chenggi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint S.Niv:2401.0295:5 2024.
Block. goose: a local, extensible engineering agent. httpps:/giithb..oo htwtthub..ommblllooo www.////oo/gooo/eeeoo.ooooooeoee 2025.
GitHub repository; accessed 2025-09-11.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances Neural Information Processing Systems (NeurIPS), 2020. URL https://arxiv.orge harg/abs//0000 2005. 14165.
Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15)3932-3333) 2016. doi: 10. 10.10073ppas.151111 1517388411. 18
Marcelo Campos, Simon Griffiths, Robert Morris, and Julian Sahasrabudhe. An exponential im- provement for diagonal ramsey. arXiv preprint arXiv:2303.09552 2023. 4
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung. Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. Mle- bench: Evaluating machine learning agents on machine learning engineering, arXiv preprint arXiv:2410.0709 07.999, 2024a. doi: 14.5550/aXxi.2441..0000 URL attps://arxiv: org/abs/ abs/ 2410-07095.9 9
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherbum, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, al. Mle-bench: Evaluating machine learn- ing agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024b. 2, 4
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherbum, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mądry. MLEE bench: Evaluating machine learning agents on machine learning engineering. In ICLR, 2025. arXiv:2410.07095.
Mouxiang Chen, Binyuan Hui. Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Jianling Sun, Junyang Lin, and Zhongxin Liu. Parallel scaling law for language models. arXiv preprint arXiv:2505. 10475, 2025a. 1, 21
11

---

Published as a conference paper at ICLR 2026
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu- Ampratwum, Xuhui Huang, Xia Ning, Song Gao. Yu Su, and Huan Sun. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. In Interna- fional Conference on Learning Representations (ICLR), 2025b. doi: 10.48550/a2Xiv.2410.555 05080. arXiv:2410.05080 9
URL aitts://aaxxta org/ abs/2410...../////// 050000000000000.......00
Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Nomera-Paredes Moham- madamin Barekatain, Alexander Novikov, Francisco J R. Ruiz, Julian Schrittwieser, Grzegorz Swirszez, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-53, 2022. 4
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961 2021. URL https:
//arxiv. org/oogg/aa. attpaaas..aa////////aaaaaeaaaaea 03961. 10
Roger Fletcher. Practical methods of optimization. John Wiley & Sons, 2000. 3
Shubham Gadre et al. Datacomp: In search of the next generation of multimodal datasets. arXiv httg//bb//304... 14108.
preprint arXiv:2304. 14108, 2023. URL
Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, et al. A survey of self-evolving agents: On path to artificial super intelligence. arXiv preprint 2002:2250..20046, 2025a. 5
Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan. Hongzhang Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, Hongru Wang, Han Xiao, Yuhang Zhou, Shaokun Zhang, Jiayi Zhang, Jinyu Xiang, Yixiong Fang, Qiwen Zhao, Dongrui Liu, Qihan Ren, Cheng Qian, Zhenghailong Wang, Minda Hu, Huazheng Wang. Qingyun Wu, Heng Ji, Mengdi Wang, et al. A survey self-evolving agents: On path to artificial super intelligence. arXiv preprint arXiv:2507.21046. 2025b. URL
/stps://arxiv.org www.g////////////// 2507 21046. 2,
Google. Gemini cli gemini code assist. gemini-code-and www.cooddassssteeee mosiist/docs, cetpis///ooc//mmmiiiciii.. gemini-cli, 2025.
httpp://deeveooerrs wotpssveelleee..oggleeee.e com/ Documentation; accessed 2025-09-09.
6, 9,
Mourad Gridach, Jay Nanavati, Khaldoun Zine El Abidine, Lenon Mendes, and Christina Mack. Agentic ai for scientific discovery: A survey of progress, challenges, and future directions. arXiv preprint arXiv:2503.0899 08979, 2025. URI. https:ttp/.. http//axxvvvvvvv////////////.ava www.aiiiivirrr.... wwwp:/iwiii...a///////a////////. 0ttp//225300.0..... 08979.
Tom Henighan, Jared Kaplan, Mor Katz, and Mark Chen. Scaling laws for autoregressive generative modeling. In arXiv preprint ar0iv:2010. 14701, 2020. 9
Danny Hermandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv22102.01293 2021. URL https: //arxiv wrggabs//g00..o.. 01293.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprint 1rXiv:1712.00409 2017. 9
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Jardar Uesato, et al. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, volume 35, pp. 10448-30466 2022. 1
Ruida Hu, Chao Peng, Xinchen Wang, Junjielong Xu, and Cuiyun Gao. Repo2Run: Automated building executable environment for code repository at scale. arXiv preprint urXiv:2502. 13681, 2025. 9
Qian Huang, Jian Vora, Perey Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents machine learning experimentation. arXiv preprint arXiv:23 2024. 9
12

---

Published as a conference paper at ICLR 2026
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07977 2024. 27
Chao Jia, Yinfci Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference Machine Learning (ICML), Proceedings of Machine Learning Research (PMLR), 2021. URL https: //arxiv. org/ abwp/a//....///////////////aaa/a 05918. 10
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? In ICLR, 2024a. arXiv:2310.06770. 6,9
Carlos E. Jimenez et al. SWE-bench multimodal: Do Al systems generalize to multimodal github issues? arXiv preprint arXiv:2410.03852 2024b.
Carlos E. Jimenez et al. SWE-bench goes live! arXiv preprint arXiv:2505.234/9 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Chris Hallacy, Jan Leike, Aditya Ramesh, et al. Scaling laws for neural language models. In arXiv preprint arXiv:2001.0836) 2020. 1,9
John Koza. Genetic programming: on the programming of computers by means of natural selec- tion, volume 1. MIT press, 1992. 18
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michal Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóždž, Piotr Sankowski, et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07881, 2024. 1, 4, 2,
William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabrício Olivetti de França, Marco Vir- golin, Ying Jin, Michael Kommenda, and Jason H. Moore. Contemporary symbolic regression methods and their relative performance. arXiv preprint urXiv:2107.14351, 2021. 18
Houyi Li, Wenzhen Zheng. Jingcheng Hu, Qiufeng Wang, Hanshan Zhang, Zili Wang, Shijie Xuyang, Yuantao Fan, Shuigeng Zhou, Xiangyu Zhang, et al. Predictable scale: Part i-optimal hyperparameter scaling law in large language model pretraining. arXiv e-prints, pp. arXiv-2503, 2025. 1, 4, 8, 23
Haowei Lin, Baizhou Huang, Haotian Ye, Qinyu Chen, Zihao Wang, Sujian Li, Jianzhu Ma, Xiaojun Wan, James Zou, and Yitao Liang. Selecting large language model to fine-tune via rectified scaling law, arXiv preprint arXiv.2402.02314, 2024. 1, 3, 7, 21
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using GPT-4 with better human aligmment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511-2522, Singapore, December 2023. Association for Computational Linguistics. doi: 10.
http://..22....mmmmeee.. 153. URL .rg www.orrg///0/// emnlp-main 153/. 18
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scien- tist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.0629) 2024.7 2
Daniel J. Mankowitz al. Faster sorting algorithms discovered using deep reinforcement learning. Nature, 2023. doi: 10. 00.10888/44158800344000004
Niklas Muennighoff Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. In Advances in Neural Information Processing Systems, volume 36, pp. 50358-50377 2023. 22
13

---

Published as a conference paper at ICLR 2026
Alexander Novikov, Ngån Vü, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov. Borislav Kozlovskii, Francisco JR Ruiz. Abbas Mehrabian, et al. Alphaevolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506. 13131, 2025. 5
OpenAL. Introducing SWE-bench verified, 2024.
URL https://oponal.commnddeee
OpenAl. Introducing codex.
cow///nee//nttrroccchggoooooooe 2025. Prod- uct announcement; accessed 2025-09-11 6, 9, 33
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances Neural Information Processing Systems org/abs//////oooooooo ab2/333b.22...... 02155. 10
(NeurIPS), 2022. URL https://arre
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon I1m: Outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01 16, 2023. URL wwttttts///aaiiii wrgg/bbs/23000 01116. 10
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research (PMLR), 2021. URL https //t///axxvvv/v org/ abs/2103, .00020. 10
Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, et al. SWE-polybench: A multi--lanuuage benchmark for repository-level evaluation of coding agents. arXiv preprint arXiv:2504.08703, 2025. 9
Bernardino Romera-Paredee et al. Mathematical discoveries from program search with large lan- guage models. Nature, 2024. doi: 00018//415880003-062222 9
Etienne
Russeil, Fabricio Olivetti de Franca, Konstantin Malanchev, Bogdan Burlacu, Emille Ishida, Marion Leroux, Clément Michelin, Guillaume Moinard, and Emmanuel Gangler. Multiview sym- bolic regression. In Proceedings of the Genetic and Evolutionary Computation Conference, pp- 961-970, 2024. 18
Michael Schmidt and Hod Lipson. Distilling form natural laws from experimental data. Science, 324(5923):81-855 2009. doi: 10.1 2...11//ciiecc 165893. 18
Ayan Sengupta, Yash Goel, and Tanmoy Chakraborty. How to upscale neural networks with scaling law? survey and practical guidelines. arXiv preprint arXiv:2502. 2025. 3
Asankhaya Sharma. Openevolve: an open-source evolutionary coding agent, 2025. URL https:
//github www hotmm//ooeelo///eeeellleeee
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In NeurIPS, 2023.
arXiv:2303.1 1366. 9
Parshin Shojace, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D. Doan, and Chandan K. Reddy. Lim-srbench: A new benchmark for scientific equation discovery with large language models. In International Conference on Machine Learning (ICML), 2025. URL https: ICML 2025 (Oral). 4,
//openreview.c wwwpenrreieew.et/foruueee www/foouumiciiooo http:/forumm/d-SSpPP../W//
Aditya Bharat Soni, Boxuan Li, Xingyao Wang, Valerie Chen, and Graham Neubig. Coding agents with multimodal browsing are generalist problem solvers. arXiv preprint arXiv:2506.03//1, 2025.
sst. Opencode: Ai coding agent, built for the terminal. 2025. GitHub repository; accessed 2025-09-11. 6, 33
14

---

Published as a conference paper at ICLR 2026
Giulio Starace, Oliver Jaffe, Dane Sherbur, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai's ability to replicate ai research. arXiv preprint arXiv:2504.01844, 2025. doi: 10.448550/rXxv.2554..000 2504. 2504.01848. 9
URL https:// //axiv org/abat abs/
Weiwei Sun et al. CO-bench: Benchmarking language model agents in combinatorial optimization. arXiv preprint arXiv:2504.043//0 2025.
Kyle Swanson, Wesley Wu, Nash L Bulaong. John E Pak, and James Zou. The virtual lab: Ai agents design new sars-cov-2 nanobodies with experimental validation. bioRxiv, pp. 2024-11, 2024. 2
SWE-agent Team. mini-swe-agent: the 100-line ai agent. https://githut: wtt.ccoo//////// Senn/WWE-aget. mini seni-sse-agaee agent, 2025, GitHub repository; accessed 2025-09-11. 6, 33
Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies. Advances in Neural Information Processing Systems, 37:114147-114779 2024. 3,
Terminal-Bent Bench Team. Terminus: A sesearch-previere terminal agent. https: //ww. tbench ai/ terminus, 2025. Remotely connects to a Docker env via tmux; accessed 2025-09-11. 6
Silviu-Marian Udrescu and Max Tegmark. AI feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaay2631, 2020. doi: 10. 10.11266scindv..aay26.. 4, 18
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for AI software developers as generalist agents. In International Conference on Learning Representations (ICLR), 2025. URL https: httttp:://oppreeeeee opwrrevieereeeeeeeeeeeeeee net/ wowooddddmddddeeeoo
6, 9,
Xingyao Wang et al. Executable code actions elicit better 11m agents. In Proceedings of the 41st International Conference Machine Learning (ICML)- (arXiv:2402.01030) 2024.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large lan- guage models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL htttp::/openne wwwfiiewnee/foruuee fication. 10
Survey Certification, Outssannin Certi-
Tung-Yu Yu Wu and Pui-Yu Lo. U-shaped and inverted-u scaling behind emergent abilities of large language models. arXiv preprint arXiv:2410.01692. 2024. 4, 20
Qiujie Xie, Yixuan Weng, Minjun Zhu, Fuchen Shen, Shulin Huang, Zhen Lin, Jiahui Zhou, Zilan Mao, Zijie Yang, Linyi Yang, et al. How far are ai scientists from changing the world? arXiv preprint arXiv:2507.2322 2025. 2
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 09388, 2025.
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: agent-computer interfaces enable automated software engineering. arXiv preprint ar57 2024. 9
Yaoqing Yang, Ryan Theisen, Liam Hodgkinson, Joseph E Gonzalez, Kannan Ramchandran, Charles H Martin, and Michael W Mahoney. Evaluating natural language processing models with generalization metrics that do not need access to any training or testing data. arXiv preprint arXiv:2202. 02842, 2022. 9
15

---

Published as a conference paper at ICLR 2026
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint arXiv:2403.16692, 2024. 3,
Xinyue Zeng, Haohui Wang, Junhong Lin, Jun Wu, Tyler Cody, and Dawei Zhou. Lensllm: Un- veiling fine-tuning dynamics for Ilm selection. arXiv preprint 2Xiv:2505.03795 2025. 1, 9,
21
Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin godel machine: Open- ended evolution of self-improving agents. arXiv preprint arXiv:2505. 229544, 2025. 1
Bingchen Zhao, Despoina Magka, Minqi Jiang, Xian Li, Roberta Raileanu, Tatiana Shavrina, Jean- Christophe Gagnon-Audet, Kelvin Niu, Shagun Sodhani, Michael Shvartsman, Andrei Lupu, Alisia Lupidi, Edan Toledo, Karen Hambardzumyan, Martin Josifoski, Thomas Foster, Lucia Cipolina-Kun, Abhishek Charnalia, Derek Dunfield, Alexander H. Miller, Oisin Mac Aodha, Jakob Foerster, and Yoram Bachrach. The automated Ilm speedrunning benchmark: Reproduc- ing nanogpt improvements. arXiv preprint arXiv:2506.2244 22419, 2025. doi: 10448550/r2Xv.25066 22419. URL
orgz//iii.....ooooooeeee abs//bbb.....//////// 22419.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as- judge with MT-bench and chatbot arena. In NeurIPS 2023 Datasets and Benchmarks Track, 2023. URL https: wwwpaarrwroo..ettttoooo/dedeaaee
18
16

---

Published as a conference paper at ICLR 2026
Appendix
Table of Contents
A Additional Details for SLDBench
18
A.1 Why is scaling law discovery good agentic challenge?
18
Seen unseen dataset splits
Human discovered laws prior literature
21
A.4 Human discovery of Gaarning-ratebatttnn scaling law ................................
23
A.5 Data contamination considerations
26
B Additional Experimental Information
28
B.1 Metrics
28
Standard deviation experiments
28
B.3 SLDAggett discovered laws
29
B.4 Closed-form optimizer for är and baz
32
B.5 Baseline agent configs
33
B.6 Ablation study
34
Prompts
C.I Task instructions
36
C.2 SLDAgent prompts
37
D SLDAgent Configuration Details
45
LLM Usage Details
46
17

---

Published as a conference paper at ICLR 2026
A ADDITIONAL DETAILS FOR SLDBENCH
A.1 WHY IS SCALING LAW DISCOVERY A GOOD AGENTIC CHALLENGE?
As novel Al-for-research task, SLD presents a compelling agentic challenge because mirrors the open-ended process of scientific discovery. It is not merely curve fitting black-box optimization. Instead, it demands the formation and testing of scientific hypotheses to uncover concise, mechanis- tic regularities that explain behavior across diverse regimes (Schmidt & Lipson, 2009).
Vast and unstructured search space A core difficulty lies in identifying the unknown functional form of the law. Unlike standard regression where model family (e-g.. linear, polynomial) is pre- specified, symbolic discovery requires searching over a space of expression trees whose size and structure are unknown a priori. Even with modest set of operators like { X, +, pow, log, exp} and few variables, the number of candidate expressions grows combinatorially. rendering exhaus- tive search computationally infeasible. This intractability motivates the use of heuristic, evolution- ary, or sparsity-driver methods (Koza, 1992; La Cava et al., 2021: Brunton et al., 2016). Conse- quently, the problem becomes one of guided exploration rather than passive estimation.
Abstraction across diverse contexts SLDBench elevates this challenge beyond typical symbolic regression (Udrescu & Tegmark, 2020) by requiring the discovery of a universal law that generalizes across multiple experimental contexts, each defined by a set of control variables (See. 2). In the sft task, for example, the goal not to produce 42 independent scaling laws for the 42 experimental groups, but to discover single, unifying functional form whose parameters 0j adapt to each specific context.j. j. Achieving such invariance requires a high level of abstraction (Russeil et al., 2024).
Emulation of the scientific method The discovery process inherently iterative, mirroring the scientific method's cycle of hypothesize -> test - revise. Different workflows are possible (SLDA- gent in Sec. just a proof-of-concept instantiation instead of an necessarily nor sufficiently op- timal workflow); an effective agent can involve exploratory data visualization, symbolic analysis (c.g., examining derivatives and asymptotic behavior) to prune the search space, and rigorous val- idation (e.g., held-out tests or cross- validaanlitt to prevent overfitting (Udrescu & Tegmark, 2020; Brunton et al., 2016). LLMs can act as judges to evaluate the plausibility of candidate expressions with high fidelity human preferences when properly prompted and calibrated (Zheng al., 2023; Liu et al., 2023). Altogether, scaling law discovery demands agentic strategies that integrate search, reasoning, experimental design, and self-evaluation, much like real scientist.
18

---

Published as a conference paper at ICLR 2026
A.2 SEEN AND UNSEEN DATASET SPLITS
We partition each SLDBench dataset into seen (training) and unseen (test) splits. The unseen split is designed to probe extrapolation generalization beyond the training distribution- -atninallyllllyyyyl toward larger models, bigger datasets, more extreme hyperparameters. Below we detail the split protocol for each of the seven tasks unified format.
A.2.1 D_ CONSTRAIN: DATA CONSTRAINED SCALING LAW
Split: 161 seen / 21 unseen.
Dimensions: model parameters N, training tokens D, unique tokens U.
Unseen selection rule: include all points with the largest and second-largest values of either N or D; remaining coordinates follow the original dataset specification.
A.2.2
DOMAIN_MIX. DOMAIN MIXTURE SCALING LAW
Split: 80 seen unseen.
Dimensions: five domain proportions; four model sizes (70M, 160M, 305M, 410M).
Unseen selection rule: we follow the original repository for the seen/unseen split.
LR&BSZ: LEARNING RATE & BATCH SIZE SCALING LAW A.2.3
Split: 2,702 seen 117 unseen.
Dimensions: learning rate I, batch size 6; model size N; data size
Unseen selection rule: restrict to experiments using the largest model (IB parameters) and largest dataset (100B tokens).
A.2.4
MOE: MIXTURE-OF-EXPERR SCALING LAW
Split: 193 seen / 28 unseen.
Dimensions: dense (non-expert) parameters; number of experts; routing-related settings.
Unseen selection rule: include models with 1.31 10 dense parameters, exceeding the training maximum of 3.68 x 10 (+s 3.6x larger); other MoE factors match seen ranges.
A.2.5
SFT: SUPERVISED FINE- TUNNIG SCALING LAW
Split: 504 seen 42 unseen (42 base-model/ dataset pairs).
Dimensions: SFT data size per base-model/aataset pair.
Unseen selection rule: for each pair, unseen uses fixed larger size of 8.19 x 10 tokens; seen covers up to 4.10 10a tokens. This yields a clean data-scale extrapolation.
2OCAB_SIZE: VOCABULARY SCALING LAW A.2.6
Split: 1,080 seen 120 unseen.
Dimensions: vocabulary size V; model size.
Unseen selection rule: fix V 96,300, outside the range [4,100, 64,500] (a 1.5x increase over the seen maximum); model sizes follow the seen set.
A.2.7 PARALLEL: PARALLEL SCALING LAW
Split: 36 seen 12 unseen (two data groups: pile, stack).
Dimensions: model size; degree of parallelism.
Unseen selection rule: use only 8-way parallelism (seen covers 1-4 way). i.c., 2x ex- trapolation in parallel degree; combine with larger model sizes assess joint scaling.
19

---

Published as a conference paper at ICLR 2026
A.2.8 U_SHAPE: U-SHAPED SCALING LAW
Split: 389 seen / 127 unseen (9 data groups for different datasets).
Dimensions: compute in FLOPs; brier score.
Unseen selection rule: We follow the original paper (Wu & Lo, 2024) ensure the final ascending phase is unseen.
20

---

Published as a conference paper at ICLR 2026
A.3 HUMAN DISCOVERED LAWS IN PRIOR LITERATURE
Parallel Scaling Law (Chen et al., 2025a):
A L(N, P) = E + (N + # P))
Explanation.
Variables. N is the non-embedding parameter count; P is the number of parallel streams (independent input transformations run in parallel and then aggregated); A, P. a, are fit parameters.
Intuition. Increasing P reuses the same backbone while adding learnable, diverse pre- fixes and dynamic aggregator over the P outputs, yielding diminishing-returns gains that are logarithmic in P. Empirically, this behaves like scaling the effective parame- ters by Ollog P) at much lower memoryllateccy growth than classic parameter scaling; larger N benefits more from parallelism. Reported fits show positive "parallel bene- fit" coefficient on Stack (Python) vs. Pile, consistent with stronger improvements on reasoning-heavy data.
2. Vocabulary Scaling Law (Tao et al., 2024)':
A
B
C
I(N, V.
+ E D
No
Explanation.
Variables. N is the non-vocabulary (non-embedding) parameter count; V is the tokenizer vocabulary size; D is the training budget (characters/tokens. depending on the normal- ization use); A, B, C, E and exponents a, 3 > are parameters.
Insuition. This extends the standard Chinchilla-style decomposition by adding a vocabu- lary term: larger V reduces tokenization loss (fewer characters per token better segmen- tation) but increases embedding difficulty; the fitted B/V term captures the net effect under fixed compute budget. To compare across different V, they propose a unigram- normalized loss that removes the trivial dependence of cross-entropy on V.
SFT Scaling Law (Lin et al., 2024; Zeng et al., 2025):
A
L(D)
+C
D + B
Explanation.
Variables. D is the fine-tuning set size; A, or, B, C are fit from a few size-loss pairs. Intuition. Fine-tuning exhibits pre-power phase at small D and power phase at larger D. The offset (often interpreted as a "pre-learned data size") rectifies the classic power law function captures both phases and saturates to C as D D-+00 00.
4. Domain Mixture Scaling Law (Ye et al., 2024):
(r(r) = a: + As exp
Σ tijys
Explanation.
Variables. (r1, .r.....,MM) are training mixture proportions M domains (sum to 1); L; is loss on validation domain i; c; is irreducible loss; k; scales the reducible part; lij encodes how training domain transfers validation domain i.
Intuition. Loss varies smoothly and predictably with mixture shares; an exponential of
linear form balances synergy (4z)0)), neutrality (x 0), conflict (< 0). Because Tj E [0, 1] and mixtures interpolate observed settings, simple forms well and allow mixture optimization without running all mixtures.
Models trained with larger V naturally have higher loss. Unigram-normalized loss (Tao 2024) is thus designed normalize the loss w.r.t. ensure fair comparison across different V.
Remarrk:
21

---

Published as a conference paper at ICLR 2026
5. Mixture-of-Experts (MoE) Scaling Law (Krajewski et al., 2024): log L(N, alog + blog E + clog N log d, where E
Paux
Explanation.
Variables. N is non-embedding parameter count; E is the number of experts (or expan- sion rate); É is effective experts transform that captures diminishing returns very small/very large E; a, b, c, d are fit coefficients.
Intuition. In MoE, increasing E' at fixed active parameters improves specialization/rout- ing up to saturation. The interaction log N log É models that the marginal value of more experts depends on N (and vice versa). Related analyses additionally introduce granularity to control expert size and show consistent power-law behavior with respect to
6. Pata-Constrainee Scaling Law (Muennighoff et al., 2023):
L(N,D,U =
+
UN+UNRR
1-exp
U4URD 1-exp
R
Explanation.
Variables. U is the budget of unique tokens (data constraint); is total seen tokens (epochs xU); N is parameters. UN denotes the compute-optimal parameter count for U unique tokens (the "base" capacity). RD and Ry are learned "repeat" hyperparameters controlling how quickly returns decay when repeating data or over-sizing the model, respectively; A, B, E, a. 3 are fit constants.
Intuition. Replace raw D and N with effective data/parameters using exponential half-life: repeated tokens become progressively less valuable, and parameters far beyond the data budget have diminishing returns. When D-U and N=UN, this reduces to the standard Chinchilla-style form.
We refit these laws using their expressions and optimization methods on the seen split of data to produce the results of human baselines.
22

---

Published as a conference paper at ICLR 2026
A.4 HUMAN DISCOVERY OF SHARNING-RATE/BATEE SIZE SCALING LAW
For the Ir&bsz task task-predi pre-training loss as a function of learning rate, batch size, dataset size, and network size-we not aware of an established scaling law in the literature. To construct a strong human baseline, we asked two human experts to derive a predictive law from the same background information and the same training split our agents.
Participants.
Expert A: junior master's student in economics with experience in econometrics.
Expert B: senior robotics master's student with a strong background in machine learning and Python programming.
evaluated both laws on an unseen test set. Expert A obtained It² = 1.1019, while Expert B obtained !t² -0.0756. (Recall that R < indicates worse-than-mean prediction on the test set.) We therefore adopt Expert B's formulation as the canonical Human baseline reported in ?? and Table 3.
Selected Law (Expert B). Expert proposes a hierarchical additive model
A
B ND +
by b b ).
L(D,N,C,))
C Ke to)²
E logb+
D
with scale-dependent optima
to = FNNDS
= D
(2)
Here D is dataset size, N is non-embedding parameter count, l is the learning rate, and b the batch size. (To avoid a symbol clash with the dataset size D, we write Ke for the code parameter D D_lr.)
Why this form is sensible.
Aligned with Step Law. The optima Eq. (2) is exactly in the same form as Step Law (Li et al., 2025), which is proved be good in optima prediction.
Power-law decay in scale. The terms A/ D and B B/N capture the empirically observed diminishing-returey regime as data and model scale increase.
Quadratic well learning rate. The term K&(E - to)2 yields a convex (in E) basin with unique minimizer at l fo- Because &o depends on (D,N) I) via equation 2, the preferred step size adapts smoothly with scale; signs of (7, () determine whether lo increases/de- creases with N and D.
Batch-size with diminishing returns and small-b penalty. The function log + bo/b rises for very small b dommintedd regime), enjoys unique stationary point at b* = by (since 0/Ob = 1/b bo/b² O), and then grows only logarithmically for large b (diminishing returns). The local curvature the optimum is F./b8 > ensuring a well- conditioned minimum if F. > O.
Interpretability and separability. Additivity isolates contributions from data scale, model scale, step size choice, and batch sizing, making the fitted coefficients interpretable and reducing overfitting risk from high-order interactions.
Result. This law is our human baseline and achieved R² -0.0756 on the held-out test set.
import numpy as np
def law (input_ data: list [dict float]], group: str) -> 1ist [dict [str, float]]:
Predicts loss using hierarchical model.
The loss is a sum of terms for data size, param size, learning rate, and batch size.
The optimal learning rate (1r_0) and a batch size parameter (bsz_0) are themselves
23

---

Published as a conference paper at ICLR 2026
modeled functions of data and param size.
unn
PARAMS ____B____UU__ GROUP {
Parameters: [A, alpha, B, beta, C, D, E_new, F, gamma, zeta, G, eta.
"all_data" :ata"; [
262. 1391, 0. ,62 7 7.0285, o. 0.0746, 0. 0.00003,,, 1278.595,
0.0493, 0. 0242 -1 0580, 0. 6498, 0302, 3503 1,
params PARAMS_EE BROUUUPPPRRPPPCEE get- (group, PARAMS BY_RP ["all_data"" A, alpha, B, beta, C, D_lr, E, F, ganma, zeta, G, eta params epa 1e-12
predictions 0]
for pt in imput_data::
Input features
float ["Ir"])
float ("bsz"])
float ("data_size"])
N float (pt
Submodels for optimal ir and bsz offset 10 (N ganma) (D zeta)
eta)
# Loss componenta
term_date data (D ** (-alpha))
term_param (N ** (-beta))
term_Ir D_lr (max(1, eps) 10)
term_ baz
(ma epa)) b0 / max(b, epa))
loss term_ data term_param predictions append ({"1m_losss loss": float (loss)})
term, farm term_baz
return predictions
Alternative Law (Expert A). Expert A proposes an additive power-law backbone with (i) an explicit small-batch regularizer and (ii) a quadratic penalty in log-learning-rate around a scale- dependent optimum:
+ C + D6- + ullog - log e.)
L(D,N,e,0) D
+
(3)
No
where
l, = 2-
(4)
Why this form is sensible.
Power-law backbone. As in equation 1, A/D" and B/N encode diminishing returns with seale.
Explicit small-batch penalty. The Db-* term discourages extremely small batches by making loss large as b + 0, consistent with noise-dominated updates.
Log- log-quadratic in step size. Penalizing (log & log f.)² is scale-invariant: multiplicative misspecification of & incurs symmetric cost (over- vs. under-shooting), The optimum f, scales power of (b. D, N); positive (n, v. μ1) would imply that larger problems favor smaller é, a common empirical heuristic.
Result. This alternative performed worse on the same test set, achieving R² -1.199.
import numpy as np
24

---

Published as a conference paper at ICLR 2026
def 1aw (awiiiuut_daae data: List[dict float]], group: str) list [dict (str, float]]:
unn
Predicts 1m_loss based on a power law with a quadratic term for learning rate deviation.
The optimal learning rate (Ir (Ir_star) is modeled as power-law function of batch size, data size, and parameter count.
Parameters: [A, alpha, B, beta, C, D, gamma, U, k, eta, nu, mu]
PARAMS _B
BeSiS____GRRUUR GROUP {
'all_ _al__daa': [
20.0, 0. 1359, 20.0, 0. 1573, 0.5665, 6. 6.5206, 1.5611, 0.0010, 0. 0.0057, 0. 0.00013, 0.0, 0.6780 }.
params PARAMS BY_
Be__BB__RRRRU GROUP get (group, PARAMS_BY BE__BB__RR_RRRE ['all__atta ']) A, alpha, в, beta, C, D, gamma, U, k, eta, mu, mu params predictions = []
for point in input_data:
Input features
point''lr']
point ['baz']
point [' ['data_size']
N
hhon_embedding.prraae
# Optimal learning rate submodel
1r_star
x (b ** -eta) (D -nu) (N mmu)
Loss components
loss
(A D** (=alpah)
B * N** (-beta)
C +
D * b** (-gamma) +
U (np.log(1)
mp.log(1r_star)) .+2)
predictions. apppriddnn.pppddddeee
loss))
return predictions
Remarks. Both laws decompose improvements from scale (via power laws) and hyperparameter choice (via convex penalties around scale-dependent optima). Expert B's batch-size term has an interior optimum b' = bb(D)) ()) and grows only logarithmically for large which may better match the empirically slow gains from very large batches; the explicit log-domain quadratic on in Expert A is appealingly scale-invariant but, our data, did not generalize as well as the direct quadratic around fo(D,N) Future work could explore interaction terms (e.g., D D N), heteroskedastic residuals, or robust losses for outlier resilience.
25

---

Published as a conference paper at ICLR 2026
A.5 DATA CONTAMINATION CONSIDERATIONS
Because SLDBench is constructed from previously published scaling-law studies, it is important to clarify what kinds of data contamination are possible, what we control for, and how our results should be interpreted.
Threat model. We distinguish three relevant notions of contamination: (i) training-time contami- nation of the underlying LLM, i.e. the model is pre-trained on the original scaling-law papers or their released code/data; (ii) inference cieen contamination, i.e., the agent retrieves the original papers or formulas from the web or other external sources during evaluation; and (iii) rest-split contamination, i... the exact extrapolation datapoints used for evaluation appear in the LLM's pre-training corpus. For large, general-purpose LLMs, (i) and (iii) cannot be ruled out, and SLDBench should therefore be viewed as literuture-wware benchmark analogous to how human scientists build on prior work.
Inference-time isolation. During SLDBench evaluation, all agents interact with a sandboxed ter- minal environment (build upon standard Docker image "python:3. 1.1 that contains only the task-specific training split, and small set of standard Python packages. In our experiments, agents have no browsing or network access during evaluation. Agents are not provided with the original scaling-law papers or any hand-written formulas inside this environment. In particular, they must produce law. py file whose functional form is judged solely by its extrapolation performance on the held-out test split via the evaluation script. Thus, while the underlying L.M weights may encode prior knowledge from the literature, there is no explicit "open book" access to the human baselines at inference time through the benchmark itself.
Training-time contamination and interpretation of results. Because SLDBencch is derived from published work, it is plausible that one or more of the commercial LLMs we use have been pre- trained on related papers, code, or even partial tables of the underlying experiments. We do not claim that our setting is free of such training-time time contamination. Rather, our goal is to evaluate the capabilities of an A/ scientist that, like a human researcher, has already read the existing scaling-law literature and must nonetheless synthesize high-quality, extrapolatable laws on new tasks and splits.
Our main claims are therefore deliberately phrased relative to the strongest human baselines we can construct on SL.Dhench In particular, SL.DAgett is considered "superhuman" in the sense that, given access to the same training split, its discovered laws achieve strictly better extrapolation met- rics than (i) the original noman-designed formulas from the literature and (ii) our additional expert- designed baseline on tasks without an existing law. This remains true even in a literature-awa regime where the underlying LLM may have memorized or partially internalized human proposals.
Evidence against trivial copying of human laws. Although training-time contamination at the level of textual formulas is possible, several observations suggest that SL.Dtgent is not merely copy- ing existing laws: (i) on multiple tasks (e.g., sft and moe discussed in Sec. 4.3), the discovered laws have different functional forms and more principled asymptotic behavior than the human baselines, yet still achieve lower extrapolation error; (ii) the same evolutionary agent architecture consistently improves over diverse underlying LLMS and over their native provider-specific agents, indicating that gains are not due to a single model memorizing a particular formula; and (iii) the agent must fit coefficients on the provided training split and is evaluated on extrapolation points that are never exposed during evolution, exact memorization of individual datapoints (which is typically hard LLMS) is neither necessary nor sufficient to succeed.
for
Taken together, these points support our interpretation that SLDBench primarily measures an agent's ability to organize, adapt, and extend any prior knowledge encoded in the LLM into executable scal- ing laws that generalize well on unseen regimes, rather than its ability to recall specific published formula verbatim.
Future contamination and benchmark longevity. Once SLDBench and this paper are publicly available, future LLMs may be trained directly on the benchmark itself, further increasing the risk of contamination. We view this as an inherent limitation of any public benchmark built from open datasets. To partially mitigate this, we (i) clearly document the literature-awar nature of SLDBench, (ii) provide code to re- generate the seen/unseen splits so that researchers can vary the extrapolation regime, and (iii) plan extend SL.Beecc with additional tasks and splits derived from newer or
26

---

Published as a conference paper at ICLR 2026
unreleased experiments (c.g.. following the practice of LiveCodeBench (Jain et al., 2024)). We also view SLDBench as complementary to private or withheld-eval datasets that can more tightly control for contamination. Overall, we believe that, even under unavoidable training time contamination, SLDBench remains a valuable testhed for agentic scientific discovery in realistic, literature-rich settings.
27

---

Published as a conference paper at ICLR 2026
B ADDITIONAL EXPERIMENTAL INFORMATION
B.1 METRICS
Let yi be the true value and 9; be the predicted value for the i-th sample, for 1,..,, n. The mean of the true values is denoted by y E V/-
R-squared (R²) The coefficient of determination, R², measures the proportion of variance in the dependent variable that is predictable from the model. A value closer indicates a better fit.
Σ 2i=(0 (1/ 0:)²
R²
(5)
1= (16 7)
B.2 STANDARD DEVIATION FOR EXPERIMENTS
Table 5: SLDBench run-to-run variability. Entries are the standard deviation of R over five runs for each task; the last column reports the average STD across tasks.
parallel
vocab_eize
8FT
domain_mis
d_constrain
1rbbsz u_shape Avg STD
Aider (GPT-5)
0,018
0.925
0.924
0.758
0.817
0.312
0.353
0.645
0.594
Temminus-2 (GPT-S)
0.768
0.075
0.751
0.140
0.508
0.452
Mini-SWE-Agemi (GPT-5)
0,004
0.778
0.085
0.028
0.331
0.033
0,525
0.623
0.301
OpenCode (GPT-5)
0.000
0.040
0.051
0.031
0.484
0.008
0.490
0.637
0.218
OpenHands (GPT-5)
0.000
0.008
0.305
0.067
0.001
0.041
0.107
0.590
0.140
CodeX (GPT-5)
0.002
0.006
0.093
0.059
0.329
0.251
0.533
0.520
0.224
Goose (GPT-5)
0.000
0.023
0.087
0.051
0.035
0,021
0.148
0.630
0.124
SI.DAgent (GPT-5)
0.000
0.001
0.000
0.003
0.003
0.012
0.034
0.005
0.007
Gemini-CLI (G-2.5-Flash)
0.866
0.744
0.000
0.767
0.538
0.754
0.105
0.412
0.523
SLDAgent (G-2.5-Flash)
0.000
0.386
0.174
0.000
0.000
0.035
0.005
0.000
0.075
Gemini-CLI (G-3-Pro)
0.000
0.000
0,000
0.009
0.000
0.088
0.354
0.000
0,056
SLDAgent (G-3-Pro)
0.000
0.000
0.000
0.000
0.000
0.016
0.000
0.000
0.002
Claude Code (C- Haiku-4.5)
0.000
0.040
0.059
0.078
0.272
0.170
0.643
0.000
0.158
SLDAgent (C-Haiku-4.5)
0.000
0.028
0.002
0.034
0.005
0.118
0,014
0.002
0.025
Claude Code (C- (C-Sonnet-4.5)
0.004
0.031
0,064
0.001
0.000
0,003
0.183
0.000
0.036
Stgnttent (C-Sonnet-4.5)
0.000
0.002
0.001
0.001
0.004
0.026
0.009
0.060
0.013
CodeX (o4-mini)
0.000
0.000
0.052
0.055
0.000
0.000
0.000
0.000
0.013
SI.DAgent (o4-mini)
0.000
0.001
0.007
0.003
0.003
0.043
0.018
0.004
0.010
28

---

Published as a conference paper at ICLR 2026
B.3 SLDAGENT DISCOVERED LAWS
B.3.1 THE LAW USED FOR HYPERPARAMETER OPTIMIZATION
import пumpy as def scaling. law_ func fanffuunddattt (data. params):
Predict LM loss via richer log- linear scaling law with quadratic and interaction terms.
Inputa:
data_points: array of shape (N,4) with strictly positive columns [ir, baz, data_size, non con_eebedding.paat paraa. size] length-11 array
params:
Returns:
preds: array of shape (N,) predicted lm_loss values un.
np asarray (araaaddddaaappitttaaaa dtype-float)
if X.ndim != 2 or shshape != 4:
raise ValueError (Expecce data_points of shape (N,4), got (X. shape)))""
if an.any(X <= 0):
raise ValueError ("Alll input features must be strictly positive for log <> transforms.".
unpack and log-transform
Ir x[:, 0]
baz = x[:, 1]
x[:, 2]
x[:, 3]
1_lr np.1og (1r)
1_b np- np.log (bsz)
1_D np. 1p.1og (D)
1_P
np.1og(( (P)
beta np. ap.asarra (params, dtype=float) ravel() if beta. [0] != Z. shape[ll] [1]:
raise ValueError (f Exppected (z. shape[1]] params, got (beta. (beta.shape [0]]")
log_pred Z. dot (beta) return np. ep.eep pxpgl__rr))
(N1)
def fit_sscaling_
loss_values)
mp. asarray (dairaaonttt..snssss, dtype=float)
ap.arar (loss_ values, (cype=float).ravvv ravel()
if X.ndim !- or x. shape !- 4:
raise ValueError (f Expeetee data_points of shape (1,4), got (x. sX.apep)"") if y y.ndim !- or X. shape [0] !- sh]p
raise ValueError ("Number loas values match number data >> points.")
if np. any (X <= 0) or up anyany <= 0):
raise ValueError (*All inputs and losses must be strictly positive for log transforms.")
compute logs
1_lr np.log 1X.(X[; 0])
1_b np. log (X[:, 1])
1_D np. np.log (X[:, 2])
1_P np. (X[: 3])
L_y np.1og(9) (
assemble design matrix np.colum__stack(
29

---

Published as a conference paper at ICLR 2026
np. np.ones_ like (l_lr),
1_P, 1_D,
1_b,
1_lr,
1_1ra42,
1_ba=2, 1 1_b,
1. 1_D,
1.P 1_b,
1_D 1_lr
}) shape (N,11)
ridge least-squares
p 2. s.apep [1]
lambda_reg 1e-6
Z.T.dot lambda_reg epe
Z.T.dot (1_y)
beta up. winalg. solvee,, solve(A, b) # (11,)
return beta
B.3.2 THE LAW USED FOR LLM SELECTION
import numpy
def scal wwwllingglaaalaaeeee func (data_points, params):
Predicts fine-tuning loss from data size D using 4-parameter shifted power law:
L(D)
DO) o0)-(-appha)
params [lggA, Log_alpha, logDO, B]
A
= exp (logA)
> 0
alpha exp (log alpha)
DO
exp(logDO)
0 (irreducible plateau)
unp
aawaaaaayddaaaaoinntttt dtype=float)..am rrave())
ap.asarray (params, dtype=float) ravel()
if p.size != 4:
raise ValueError(1 (Exppeetee 4 parameters, got (p.size)")
logA, log_ alpha, logD0, B
ep (1ogA)
alpha np- exp (log_alpha)
DO np. ep.exp (logD0)
return B
ap-pouer(D + DO, -alpha)
def fit_scaling_lat law (adttaa_paidatttta loss_values).
Fits the 4-parameter shifted power law using optimization. unp
ap.asaraayddat (aatadapatttaatttt..aaaaaaa dtype=float) ravel()
asarray values, stype=float).rrvell.
Numeric safeguards epa 1e-8 np.maximum (y, eps) np..mxximu(D, epa)
Initialize paramaters y_min, y_max np min(y), ma) y y_range = max maxxy_mas y_min, eps)
Initialize plateau just below minimum observed loss
30

---

Published as a conference paper at ICLR 2026
B0 (0. y_min, epa)
Small horizontal DO_0 max np. mi)) eps)
for stability
Estimate A and alpha via log-log fit y_shift = y B0 y_shift ap.clip(y_shitt epa, None)
valid (D 0) & (y_shift 0)
np. ap..su (valid) >= logD np. log (D [valid] DO_0)
logY np. np.log (at_ayaattttvliiiiia
slope, intercept
polyfit logY, 1)
alpha0 max max epa)
AO
max (ax wwt.::pppiteereeeeeeeeee
eps)
else: Fallback guesses
alpha0 A0
max(y_max BO, eps)
p0 mp. array (o),,)))))),, mp. log(aaphhh),,
ap.1og(D0_0), BO], dtype-float)
def objective(p):
pred scaling.lax, lar_lllu_uuuffff p) 0):
ap.any
return mp.inf
r x_lin= = (pred - y) / y_range
r_log - np. lo (pred + eps) up. loglo eps)
err r 1in*+2 r_log*+2
np. (err(
bounds [(-20.0, 20,0), (-10.0, 10.0), (-20.0, 20.0), C0.0, y_min)]
res = minimize(
objective, p0, "L-hod bounds-bounds, :riion=("ftool"1ee.. "gtol": 11-99)
return res.x if res. res. success else p0
31

---

Published as a conference paper at ICLR 2026
B.4 CLOSED-FORM OPTIMIZER FOR Ir AND bs2
Model. Let N be the non-embedding parameter count and D the token count. For x = (t(r)) and
y In(bsz), the SLDAgent law for the log loss can be written (constants w.r.t. (æ, v) are grouped into C(N, D)):
log(loss) = C(N, D) + Ps + 84 x + x² + B8 + By xy + So (In N 310 (In D) x. (6)
Minimizing loss is equivalent minimizing the quadratic form above in (x,y).
First-order optimality.
Setting partial derivatives to zero gives a 2 x 2 linear system:
0 Ox
Ba + 2/5. + Bry + B10 In 0,
(7)
0
B3 3yx 23 + Bg In N O.
(8)
Oy
In matrix form,
2
97
I
B34 310 In D
B7
2/g
y/
B3 + Bg In N
H
When A det(H) 43r80 minimizer is unique.
88
0 (true for our fitted model), is positive-definite and the
Closed-form solution.
Solving the system yields
x² -236(3 + 310 In D) + 37(B3 + 3 Nn
(9)
40,86 - 34
y" (-(31 + 810 In D) 2/3 + 1.811
(10)
48,B6 37
Therefore,
Ir f(N, exp(x*)
bsz = =
Instantiation with fitted coefficients.
The fitted parameters are
33 = 0.0595, B1 = 0.1906, fá 0.0098, A6 = 0.0073, 37 = - -.006, 3g = 0.0009, 310 00.0012,
giving A = 2,55 = 0.0196. B = 37 = -0.006, C = 236 = 0.0146, and A = AC B2 0.00025016. Expanding the solution,
x² -C,3 + BBg
C,310
In + B IN = -12.5110 0.070033 In D + 0.213463 In
+
A
A
(11)
B34
Aβ
B310
IN = -9.23329 + 0.028782 In + 0.697314 In A
5"
In
A
(12)
Numeric example. 25.3284 and thus
For N = 1,073,741,824 and D = 10 tokens, In s 20.7944, In D s0
lr exp(x*) 1.7673 10-3
bsz exp(y") 401.79.
In practice we pick the nearest admissible grid point for training: in our experiment this is (Ir, bsz) (1.953 x 10-3, 384).
32

---

Published as a conference paper at ICLR 2026
B.5 BASELINE AGENT CONFIGS
We standardize installation sources and, when available, pin versions to those obtainable on 2025- 09-03. Unless noted, we use upstream defaults and recommended settings.
OpenHands (Wang al., 2025): source All-Hands-A/OOpeeHHand: channel GitHub Release; version 0.55. 0 (released 2025-09-02).
Claude Code (Anthropic, 2025): package Ganchropiccaa//llade--oo channel npm; ver- sion 100.102 (published 2025-09-03).
Codex (OpenAI, 2025): source openai/codex; channel GitHub Release (also available via Home- brew/npm); version 0. 29.0 (released 2025-09-03 08:32).
OpenCode (sst, 2025): source opencode-ai/opeecode: channel GitHub Release; version v0.0.55
(released 2025-07-29).
Mini-SWE-AAgent (SWE-agent Team, 2025): source GitHub Release / PyPI; version v1.10. (released 2025-08-29).
channel
Gemini-CLI (Google, 2025): source gengl---emmiiggiiniiciiii channel GitHub Releases (sta- ble); version v0.3.0 3.0 (stable) as of 2025-09-03 (the stable tag from Tue 2025-09-02).
33

---

Published as a conference paper at ICLR 2026
B.6 ABLATION STUDY
We conduct two key ablation studies to validate the design choices of SLDAgent: one on the co- optimization of the symbolic expression and the fitting procedure, and another on the impact of the number of evolutionary iterations.
Co-optimization of Expression and Fitting Procedure. A key novelty of SLDAgent is its co- optimization of two distinct subroutines: the symbolic Expression defining the law and the Optimization routine for fitting its parameters. assess the importance this design, we com- pare it against an ablation, SLDAgent-law-only, which only evolves the law's expression while using a fixed, standard BFGS optimizer. This setup is analogous to recent methods like LLMSR, which also decouples expression generation from fixed optimization step.
The results in Table clearly demonstrate the superiority of the co-optimization approach. The full SLDtgent achieves higher average R² score (0.848 0.775) and outperforms the law-only vari- ant on five of the seven tasks. The most significant difference is observed on the challenging 1rkbsz task, where Se.DAgent-law-ont predictive law (R²² = 0.194). This highlights that discovering a good symbolic expression is only half the battle; tailoring the optimization procedure the specific law and data is critical for success.
fails (/2² -0.224), while the full SLDAgent finds reasonably
Table 6: Performance comparison of the full SLDAgent against ablation that only evolves the law's pression ("SLDAgent-law LLL-Agent-lawwooly)... law-",ly"), using 04-mini model. Scores are R² averaged over three runs. The best scores for each task are bolded.
parallelf
vocab_size"
SPT1
domain_mixi
noot
i_constrain"
1rbbez]
Avg. R²1
SL.DAgent-lww-- low-only (o4-mini)
1.000
0.947
0.931
0.966
0.835
0.952
-0.224
0.775
SLDAgent (ol-mini)
1.000
0,964
0.994
0.976
0.851
0.961
0.194
0.848
Effect of Evolutionary Iterations. We set the number of evolutionary iterations to 50 as a trade- off between computational cost and performance. To analyze the agent's 's behavior over time, we visualize the best Rt2 score found at each iteration for tasks of varying difficulty; parallel (easy), moe (medium), and 1r&bsz (hard).
As shown in Figure 4, the performance trends confirm the robustness of our evolutionary approach. For the easy parallel task, the agent quickly discovers near near-perfect solution. For the moe task, performance improves steadily and stabilizes at high Rº value. On the most difficult 1rábsz task, the score is volatile in early iterations but trends consistently upward as the agent explores the solution space. Crucially, in all cases, performance either improves or plateaus without any signs of degradation, indicating that the evolutionary search does not overfit to the seen data even without dedicated validation set. This justifies that 50 iterations provide sufficient budget for the agent to converge toward effective solutions.
34

---

Published as a conference paper at ICLR 2026
parallel scaling law checkpoints
1.000
0.994
ru 10
run
run_c
run
run
run_
10
20
30
40
Checkpoint lnteration)
(a) parallel_scall law (Easy)
mac ane_scaaing.o law over checkpaints
0.75
0.25
run
10
0.25
run
run 3
run_5
run_5
-0.75
run_
30
40
Checkpoint (teration)
moe moe_ ing law (Medium)
bsz scaling law
R² over checkpoints
Run
AUn,2
0.25
0.00
Kun
run_9
-0.75
1.000
30
Checkpoint (teration)
(c) 1r br__ssz_scalnn law (Hard)
Figure 4: SLDAgent (o4-mini) progression of the best R score 50 evolutionary iterations for three tasks of varying difficulty. Performance consistently improves stabilizes, demonstrating the effectiveness of the evolutionary search and its robustness against overfitting.
35

---

Published as a conference paper at ICLR 2026
C PROMPTS
C.1 TASK INSTRUCTIONS
You have analyze experimental dataset discover the underlying e- mathematical relationahip, or "scaling 1aw," that governs it.
The dataset located at //app/data" and can be loaded using
"datasets. load_ foaadffoom_deee from_disk())
The experimental context: aaroblem_statement)
You must produce two files:
1. A Python function in //app/lawppy
This file must contain single function named "law' with the following
signature. This function should embody your discovered mathematical formula.
"python
def 1aw (input_data: [strr float]], group: str) >> 1ist [str
float]]:
Predicts output variables based on input variables according to a discovered scaling law.
Args:
input_ data: A list of dictionaries, where each dictionary is single data
point containing input variable names as keys their corresponding values.
group: The name of the experimental group for which to make predictions. The functional form of the law must be the same for all groups, but the constant paramstersscoofffciiett can differ per group.
Returns:
list of dictionaries, corresponding to input_data list, with each dicticnary containing the predicted output variable(s).
Your implementation here pass
2. A detailed explanation in "/app/epplain.n mdd
This Markdown file should explain:
The mathematical formula you discovered.
Your reasoning and the methodology used to find the law. for each distinct group.
The fitted values of the carameters/coeffcciaie
Your submitted 'law" function will be tested hidden dataset to evaluate its ability extrapolate new, data points.
36

---

Published as a conference paper at ICLR 2026
C.2 SLDAGENT PROMPTS
The prompts of SLDAgent are generated using the same template.
Vocabulary Scaling Law Prompt
You are an expert in scaling laws and machine learning who specializes discovering and improving scaling law functions for different LLM training scenarios. Your task is to evolve both the lcawfing_law_fnnc function (currently naive power law) and the fit_sccaligg_arr optimization algorithm (currently naive BFGS) to better model the relationship between vocabulary size, vocabooayylary parameters, number of characters and Losau (unigram- (uiigran-nrrmlizzer language model loss).
** IMPORTANT: The scaling law function must more than 7 parameters. **
Focus mathematical accuracy
different vocabulary configurations, cross-dataset generalization, parameter efficiency (simple forms that can be fitted with limited data),
and annerical/theoreticall stability.
DATA CHARACTERISTICS-
Features: www.bpopaameeeteeeeeeee vocab_size num_characters] 3D input scalar output
Labels: unigram_ nww.nnoomalzeeelseeo
Dataset size: 1080
Vocabulary size: 4096 to 96256 tokens (8 distinct sizes)
Embedding dimension: 512 to 2048 dimensions (4 values)
Character count: 1e8 to 5e12 characters (100M to 5T characters)
Non-vocab parameters: 3. 3.3e7 to 1.1e9 (33M to 1.1B parameters)
FLOPs range: 1.3e16 to 4. 4e20 operations
Lossu range: -5.34 -0.51 (negative values indicate improvement over unigram)
Losau meaaures improvement over context-free unigram model (negative better)
Explores vocabulary scaling trade-offs parameter, data, and architecture dimensions
The function signatures must remain:
def laslingglaw_funng (data_points, params)
# data_points: (N13) array with columns [P [P_non_vocab, vocab_size,
num nun_charactera]
Non_ vocab_pparanttrr:: Array of non-vocabulary parameter counts
vocab_size: Array of vocabulary sizes
num nun_characters: Array of number of characters processed # params: Array of to 7 parameters
# Returns: Predicted Lossu values
def fit_scaling_ law (aw loss_ :
data_points: (N,3) array with columns [P_non_vocab, vocab_size, - num. characterr
Array of non-vocabulary parameter counts # vocab size: Array of vocabulary sizes vocab_size:
Non www.vvoabppaaaatteee
# num characters: Array of number of characters processed
lossu values: Array of corresponding Lossu values
Returns: Optimized parameters (up to 7 parameters)
Write all improvements between EVOLVE- and EVOLVE-BLOCK-END markera.
You are not allosed to use input-dependent feature in scaling law_func, e.g. median min etc.
37

---

Published as a conference paper at ICLR 2026
Learning Rate and Batch Size Scaling Law Prompt
You are an expert scaling laws and machine learning who specializes discovering and improving scaling law functions for different training scenarios. Your task is to evolve both the iscaling, lunc" function (currently a naive power law) and the "fat_scaling_la"" optimization algorithm (currently naive BFGS) to better model the relationship between learning rate, batch size, data size, model parameters and training loss.
You are allowed to decide the number of parameters in the scaling law function.
Focus on mathematical accuracy across different hyperparameter scales, cross-configuratioon generalization, parameter efficiency (simple forms that can be fitted with limited
data), and numerical/theoreticall stability.
**AATA CHARACTERISTICS (2702 total data points) :* mabeedding.param..e. 4D input Labels: lm_loss scalar output.
Features: [1r, bsz, data_size,
Dataset size: 2702 total
Learning rate range: 2.44e-4 to 2.21e-2 (logarithmically spaced)
Batch size range: 16 to 2048 (powers of 2)
Data size range: 2.0 to 1.0e11 tokens (2B 100B tokens)
Parameter range: 6.00e7 to 1.07e9 (60M 1.07B non-embedding parameters)
Loss range:
3.7 cross-entropy loss
Comprehensive hyperparameter sweep covering learning rate batch size effects
The function signatures must remain:
def lca_fnn__aw__unee www._a_anddat_pppaea points, params):
data_points: (N,4) array with columns [Ir, baz, data_size, non non_enbedding_paree partggpaarmaaaaaaaaa_
# 1r: Array of learning rates
# bsz: Array of batch sizes
# data_size: Array of data sizes
non smbeedingg.praa..... saram_size: size:
Array of non-enbedding parameter sizes
Returns: Predicted
values
Model parameters (I) range: "214M to parameters
Training tokens (D) range: 4B to 100B tokens
Learning rates range: 1.2e-4 to 2.2e-2
Batch sizes range: 16 to 4096
def sit_scaling.law(att
Loss_values pataapoints: (N,4) array with columns [lr, baz, dize_ non non_ www.emeeddng.prraaaa Bacthnngpprmm__iaee
1oss_values: Array of corresponding 1m loss values # Returns: Optimized parameters
Write all improvements between BOLLVEBLOCK-STEE and EVOLVE-BLOCK markers.
You are not allowed to use input-dependent feature in scaling law_ lag_llla_uuaa func, e.g., median / min max etc.
38

---

Published as a conference paper at ICLR 2026
SFT Scaling Law Prompt
You are an expert scaling laws and machine learning who specializes discovering and improving scaling law functions for different LLM training scenarios. Your task is to evolve both the "scaling,law__ure func" function (currently a naive power law) and the "fat_scaling_la"" optimization algorithm (currently naive BFGS) to better model the relationship between data size and loss values in supervised fine-tuning across different model-dataset combinations.
IMPORTANT: The scaling law function must more than 4 parameters.
Focus mathematical accuracy cross-dataset generalization, parameter efficiency (simple forms that can be fitted with limited data), and numerical//theoretical stability.
different model architectures,
DATA CHARACTERISTICS:
Features: [sft_data_size] 1D input
Labels: sft_loss scalar output
Dataset size:
per group)
Data size range: 200 to 819,200 examples (14 exponentially-spacee sizes)
Model parameter range: 1.24e8 to 1.3e9 parameters (124M to 1.38 parameters)
Loss range: 1.7 to 4.9 cross-entropy loss
Datasets: Flan, Gigaword, and Wikiword instruction-tuning datasets
Model architectures: Various transformer-bbee based language models distinct (model, dataset) configuration groupa for croas-gemeralizattion
The function signatures must remain:
def scaling (llw_funccdes (ddata_points, params)
# data_points: (N,1) array with columns [data_size]
data_size: Array of data sizes (200 to 819200)
parama: Array of up to parameters
Returns: Predicted loss values
def fit_
loss_ loss_values):
# data_points: (N, array with columns [data_siae
# data_size: Array of data sizes
loss_values: Array of corresponding loss values
Returns: Optimized parametera (up to 4 parameters)
Write all improvements between EVOLVE-BLOCK START and EVOLVE-BLOCK-BR markers.
You are not allosed to use input-dependent feature in scaling. law func, e.g. median min max
etc.
39

---

Published as a conference paper at ICLR 2026
MoE Scaling Law Prompt
You are an expert scaling laws and machine learning who specializes discovering and improving scaling law functions for different LLM training scenarios. Your task is to evolve both the "scaling.law_fune func" function (currently a naive power law) and the "fat_scaling_la" optimization algorithm (currently naive BFGS) to better model the relationship between MoE architecture parameters validation loss.
* IMPORTANT: The scaling law function must use no more than 6 parametera. **
Focus mathematical accuracy across different MoE configurations, generalization
expert counts and model sizes, parameter efficiency (simple forms that can fitted with limited data), and numerical/theerrticall stability.
DATA CHARACTERISTICS.
Features: [num_experts,
wateeeppramat_eemmme 2D input
Labels: loss_walidation scalar output
Dataset size:
Number of experts: to 64 experts (various configurations)
Dense parameter count: 1e8 to 8e8 parameters (100M to 800X parameters)
Validation loss range: 1.8 to 3.8 cross-entropy loss
All data collected at training step 249000 for consistent comparison
Includes both dense and MoE transformer architectures
Explores trade-off between expert count and parameter efficiency
The function signatures must remain:
def lcaling_lax_ffnc (attt__unnddttaapppaeaa (data_points, params):
# data_points: (N22) array with columns [num_ _am_exeett,,
mamse_parameter_come
num_experts: Array of expert counts
dense__prramtter count: Array dense parameter counts
params: Array up to parameters
Returns: Predicted validation loss values
def fit sit_sscaling.i laweiiinnaaaddaaaooiiiiiiiie
bassvalues).
# data_points: (N,2) array with columns [num_experts, dense_parameter
Loss_valuee Array of corresponding validation loss values
Returns: Optimized parameters (up to 6 parameters)
Write all improvements between # EXOLVE-BLOCK-START and EVOLVE-BLOCK-END markers.
You not allowed input-dopendent feature in scaling_las. law_func, e.g. median min max etc.
40

---

Published as a conference paper at ICLR 2026
Domain Mixture Scaling Law Prompt
You are an expert in scaling laws and machine learning who specializes discovering and improving scaling law functions for different training scenarios. Your task is to evolve both the "scaling.law_fum func" function (currently a naive power law) and the "fat_scaling_la"" optimization algorithm (currently naive BFGS) to better model the relationship between domain mixture proportions and multi-domain loss values across different model sizes.
IMPORTANT: The scaling law function must more than 35 parameters,
Focus mathematical accuracy
different model sizes, cross-domain generalization, parameter efficiency (simple forms that can fitted with limited data), and numerical//theoretiaa. stability.
DATA CHARACTERISTICS
Features: Domain proportions (5 domains) array of shape (n_mixtures, 5)
Labels: Kulti-domain losses (5 domains) array of shape (n.mixtures,
Dataset size: training (20 per model size)
Model parameter sizes: 70M, 160M, 410M, 1B parameters separate groups)
Domain proportions: Each row sums to 1.0 (mixture weights)
Loss ranges: Domain losses span 1.8-4.2 cross- entropy loss
Mixture configurations: Systematic exploration of different domain weight combinations
This is multi-output regression problem with correlated domain
performances
The function signatures must remain:
def law_fun (data_points, params):
data_points: (N,5) array with domain proportions domains
proportions: Array domain mixture proportions
params: Array of up to 35 paraneters
# Returns: Predicted multi-domain loss values (N,5)
def fit_ stt_scalling_axdddee law
loss_values)
data_points: (N,5) array with domain proportions for domains 1oss_values: Array of corresponding multi-domain losses (8,5) Returns: Optimized parameters (up to parameters)
Write all improvements between EVOLVE-BLOCK START and EVOLVE-BLOCK END markers.
You not allowed input-dopendent feature in scaling law_ lan_func e.g.
median min max
etc.
41

---

Published as a conference paper at ICLR 2026
Data Constrained Scaling Law Prompt
You are an expert scaling laws and machine learning who specializes discovering and improving scaling law functions for different LLM training scenarios. Your task is to evolve both the "scaling. func" function (currently a naive power law) and
the fit__ccalngg.la optimization algorithm (currently naive BFGS) to better model the relationship between training data characteristics model loss under data cota-connt conditions.
IMPORTANT: The scaling law function must use more than 7 parameters. **
Focus mathematical accuracy different data scales, cross-dataset generalization, parameter efficiency (simple forms that can fitted with limited data), and numerical/theorrttcaal stability.
DATA CHARACTERISTICS (182 total data points) :** Features: [umique_tokens, params, tokens] 3D input Labels: loss scalar output
Dataset size: 161
Parameter range (P): 1.1e8 to 1.1e9 (100M to 1.1B parameters)
Token count range (D): 1e9 to 1e12 tokens
Unique tokens range: 1e7 to 5e8 unique tokens
Loss range: 1.8 to 7.2 (cross-entropy loss)
Model architectures: Transformer variants with different parametorizations
Data explores scaling under token/unique-token constraints
The function signatures must remain:
def scaling (unaadata__ppot
params)
# data_points: (N,3) array with columns [unique_tokens, params, tokens] tokens: Array of token counts
parama: Array parameter counts
unique_tokens: Array unique token counts parama: Array up to parameters
# Returns: Predicted loss values
def fit_scaling_Lae (datta_ppointe
joss_values data_points: (N,3) array with columns [unique_tokens, params, tokens] :aa_ess Array of corresponding loss values
Returns: Optimized parameters (up to parameters)
Write all improvements between EVOLVE-BLOCK- START and EVOLVE-BLOCK-ERE markers.
You not allowed input-dependent feature in Baaling_law_fuue e.g. median min max etc.
42

---

Published as a conference paper at ICLR 2026
Parallel Scaling Law Prompt
You are an expert in scaling laws and machine learning who specializes discovering and improving scaling law functions for different training scenarios. Your task is to evolve both the "scaling.lav_funce func" function (currently a naive power law) and the "fat_scaling_la"" optimization algorithm (currently naive BFGS) to better model the
relationship between model parameter count, parallel size, and language modeling loss. Here we apply "parallelssie" transformations to the input, execute forward passes of the model in parallel, and aggregate the "parallel_size outputs. We call this method parallel scaling.
IMPORTANT: The scaling law function must use no more than parametera.
Focus on mathematical accuracy across different parallel configurations, cross-dataset gemeralization, parameter efficiency (simple forms that be fitted with limited data), and numerical/theorrticall stability.
DATA CHARACTERISTICS.
Features: [num_parama, parallel_size) 2D imput
Labels: loss scalar output
Groups: 'pile' and 'stack' datasets (18 samples each)
Parameter range: 5.36e8 to 4.38e9 parameters (536M to 4.38B)
Parallel sizes: [1, 2, 4] copies
Loss range by group:
'pile': 1.7938 to 2.1113 (higher loss values)
'stack': 0.9906 to 1.1722 (lover loss values)
Key observation: Increasing parallel_size decreases loss
parallel_size=11 avg loss 1.9780 (pile), 1.0972 (stack)
parallel_size=2: avg loss 1.9480 (pile), 1.0767 (stack)
parallel_size=4: avg loss 1.9259 (pile), 1.0635 (stack)
Experimental setup: Augment input with parallel_size copies, pass through LLM, aggregate responses
The function signatures must remain:
def scaling_law langg_llw_uueee funerfunneee (data_points, params):
data_points: (N,2) array with columns [num_params, parallel_size size]
num_params: Array of model parameter counts
parallel_size: Array of parallel copies for input augmentation # params: Array of up to 4 parameters
# Returns: Predicted loss values
def fit_scttt stl_ssallngg__l (daitaaapplaadaaaapolotll
loss mass_valess: data_points: (112) array with columns [num_params, parallel_sizel
num_params: Array of model parameter counts parallel_size Array of parallel copies for input augmentation
loss_values: Array of corresponding loss values
# Returns: Optimized parameters (up to 4 parameters)
Write all improvements between EVOLVE-BLOCK-STE -TARR and markers.
You are not alloved to use input-dependent feature in scaling law_fllw_uaan e.g., median min max etc.
43

---

Published as a conference paper at ICLR 2026
U-shaped Scaling Law Prompt
are an expert scaling laws machine learning who specializes in discovering and improving scaling law functions for different LLM training scenarios. Your task is to evolve both the @eealing_law_fii function (currently naive power law) and the "fit_scallngg.ar optimization algorithm (currently naive BFGS) to better model the relationship between compute (FLOPs) and LLM performance on easy questions, which exhibits a characteristic U-shaped or double descent pattern.
IMPORTTAT::
The scaling law function must capture the U-shaped scaling pattern.
than 6 parameters
Focus on mathematical accuracy across different benchmark tasks, cross-task generalization, parameter efficiency (simple forms that can be fitted with limited data), and mumerical/theorticee stability. The U-shaped pattern critical: performance initially worsens with scale before improving again.
**DATA CHARACTERISTICS:**
Features: [lops] 1D input in log10 scale
Labels: brier_score scalar output (negative values, more negative = better)
Dataset size: for train
Log FLOPs range: [-0.9, 2.9] approximately (log10 of FLOPs in 1E21 units)
The function signatures must remain:
python
def scaling_law__ wunccfcnccee (data_points, params):
data_points: (M,1) array with columns
log_flops: Array 1og10 1E21 units)
params: Array of up to parameters to capture U-shaped pattern # Returns: Predicted brier_score values (negative)
def fit_scaling_Lav (awta__ loss_values):
data data_points: (N,1) array with columns (log_flops)
1og_flops: Array of 1o in 1E21 units)
loss_values: Array corresponding brier_score values
Returns:
Optimized parameters (up to 6 parameters)
Write all improvements between # EYOLVE-BLOCK-STARR and # SVOLVE-BLOCK-END markers.
You are not allowed to use input-dependent feature in scaling law__llw__uu func, e.g., median min / max etc.
44

---

Published as a conference paper at ICLR 2026
D SLDAGENT CONFIGURATION DETAILS
This section outlines the specific configuration parameters used for the SLDAgent experiments, powered by the OpenEvolve framework. These settings were standardized across different scaling law discovery tasks unless otherwise specified.
Evolution Parameters The evolutionary process runs 50 iterations for all tasks. Checkpoints are saved after every iteration. We use multi-island evolutionary strategy with 5 islands, each maintaining population 100 individuals. To ensure genetic diversity, island migration occurs every 25 generations.
Database and Selection Strategy The evolutionary database is governed by MAP-Elites princi- ples. Each program is mapped into the database based on feature dimensions of combined score, complexity, and diversity metrics, each binned into 10 categories. The selection process for creating the next generation balances exploration and exploitation: 70% of parents are selected based high performance (exploitation), 20% are selected to encourage diversity (exploration), and 10% are chosen from the absolute pest-performers (elite preservation).
LLM Integration and Prompting We connect large language models via API endpoints, im- plementing robust retry mechanism (10 retries with 10-second delay) and an extended timcout of 240 seconds accommodate complex reasoning. For the mutation step, prompts dynamically generated using the top performing programs and 2 diverse programs from the database as in- context examples. Each prompt provides the LLM with detailed data characteristics, mathematical constraints (e.g., number of parameters used by human law), and explicit function signatures that must be preserved to ensure compatibility with the evaluation framework.
Evaluation Framework Generated programs are evaluated in parallel using 4 processes to accel- erate the evolution cycle. Each program evaluation has a 600-second timeout and allowed up to 3 retry attempts in case of transient errors. For objectivity, the evolution relies purely numeri- cal performance metrics (fitting error), with LLM- based feedback and cascade evaluation systems disabled.
45

---

Published as a conference paper at ICLR 2026
E LLM USAGE DETAILS
In the preparation of this manuscript, we employed the Large Language Model, Google's Gemini 2.5 Pro, as writing assistant. The application of this tool was exclusively for improving the linguistic quality of the text. The model was used to generate reference drafts to help refine sentence structure, correct grammatical errors, and ensure a consistent professional tone. Following this, the human authors wrote the final text in its entirety to ensure it precisely reflects our original research and intent. The core scientific concepts, methodology. experimental results, and conclusions presented are solely the work of the authors, who bear full responsibility for the scientific content of this paper.
46


Reviews:
[{"review_id": "9LYeKEtNw5", "reviewer": "Reviewer_ok7t", "summary": "This paper aims to formulate scaling law discovery as a task for scientific agents, and proposes a new agent model, SLDAgent, for this task that outperforms existing agents on this task. I think this is good work that would be appeal to several subsets of the ICLR community that makes data, modeling, and (promises of) application contributions.", "questions": "- How to deal with possible future data contamination? (or existing contamination because the human discovered scaling laws are probably already online and/or part of the model training data and also are searchable on the web?) Maybe agents have an upper hand to the humans because they can springboard from the human solution that is available to them (at least for the non lr_and_bsz tasks). The fact that they are able to do this is cool of course, but if this were the case the \"superhuman\" claim should be interpreted with more nuance.\n- Maybe slightly more seriously, the extrapolation data might also be available to the models due to them being published. There is really no way of controlling for training-level contamination, but were any measure taken to blacklist access to such information at inference time (e.g., PaperBench does this, if I remember correctly), or were any post-hoc analyses of the agent trajectories done to examine whether they were accessing/making use of such information?\n- Do you think it would be useful to consider \"harder to predict\" scaling trends like inverse scaling or U-shaped scaling that have been observed in the literature and would the agents do well? I think these kinds of settings are cases for which the abstract nature of the task itself is critical information that humans and agents need to make use of in predicting extrapolation trends, more so than the ability to curve-fit the training set's trends well. For instance, we know that \"overriding existing definitions\" is a category of task that is likely to inverse-scale (at least in some parts of the curve), and this abstract intuition seems important especially if the available datapoints only point to monotonic increase.\n\nNitpicky comments:\n- L076: \"the objective is clear, continuous, and unbiased\" I'm not sure any objective can be \"unbiased\"\n- L079: \"As existing laws are designed based on a series of continuous and iterative human research efforts, existing agentic systems lag behind these laws on difficult SLD tasks, and the human-derived laws could also give negative-correlated predictions under challenging scenarios.\" -- hard to comprehend logical flow\n- L183: \"focus on rediscover a pre-known formula\" rediscovering*", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall I enjoyed reading the paper - I noticed that a lot of the implementation detail questions I wrote down during my reading got resolved as I read through it, which suggests that the grounds were well covered. \n- The construction of the training-extrapolation set is sensible and reflective of real world scenarios (\"For each task, we hold out an extrapolation test set from the dataset by selecting data corresponding to the largest model or dataset sizes.\")\n- Good controlled comparisons: the results are reported on both fixing the base LLM and varying the agents, and on fixing the agent and varying the base LLMs.\n- I appreciated the existence of a human baseline.\n- I liked the design choice that the information given to the model includes the task context and the semantics of the parameters involved - I think this will be critical for predicting \"harder to predict\" scaling laws that I will discuss in the Questions section.", "weaknesses": "The weaknesses below are not critical:\n- Sparse discussion of the possible effects of contamination and measures taken to prevent this\n- Coverage of unexpected or harder-to-predict scaling laws like inverse scaling or U-shaped scaling\n\nI provide more context about these points in the question section, but I think neither of these limitations are grounds for rejection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WBbOZtTl8S", "reviewer": "Reviewer_Bj2J", "summary": "This paper proposes a new benchmark for LLM agents, namely *generating scaling laws* for language modeling. The benchmark (**SLDBench**) is constructed by collecting scaling law tasks (e.g., predicting loss from $N$ and $D$) and data points from LLM experiments in existing literature. The agent is **given the scaling law task and the set of experimental results**, and **responsible for generating a mathematical equation that fits the given data**.\n\nThe benchmark contains seven tasks:\n1. Parallel: scaling w.r.t. parallelism $P$ and model size $N$\n2. Vocabulary: scaling w.r.t. non-vocab model size $N$, vocab size $V$, and dataset size $D$\n3. Supervised finetuning: scaling w.r.t. dataset size $D$\n4. Domain mixture: scaling w.r.t. proportions of different domains\n5. MoE: scaling w.r.t. network size $N$ and number of experts $E$\n6. Data-constrained: scaling w.r.t. network size $N$, dataset size $D$, and unique tokens $U$\n7. LR and batch size: scaling w.r.t. learning rate $\\ell$, batch size $b$, and dataset size $D$\n\nEach data point under each task contains the parameters of the experiment $\\mathbf{x}$ (e.g., dataset size, batch size), the result $y$ (e.g., loss), and a control index (e.g., a particular model). **SLDAgent is a program that iteratively refines an $\\operatorname{Expression}$, which implements $f_\\bf{\\theta}:\\bf{x}\\mapsto\\hat{y}$, and an $\\operatorname{Optimization}$ routine, which finds the best-fit parameters $\\hat{\\bf{\\theta}}$.** It begins with an initial pair, namely the power-law function for $\\operatorname{Expression}$ and BFGS optimizer for $\\operatorname{Optimization}$, which is added to the initially empty database. Then, at each step, some programs are selected from the database (according to pre-defined heuristics), formatted as a prompt, and the agent proposes a modification by altering the law form or changing the optimizer. The resulting child program is executed and added to the database. The loop terminates after a fixed number of iterations. At the end, the program with the highest score in the database is returned.\n\nEvaluation on SLDBench is performed via correlation between the prediction of the scaling law and the ground truth on held-out experiments under each task. The authors show that **SLDAgent paired with existing LLMs outperforms the corresponding provider-specific agents (e.g., Gemini + SLD Agent beats Gemini-CLI) as well as the scaling law proposed in the original paper**.", "questions": "NA", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors study a new application of LLM agents, namely the discovery of scaling laws for foundation models. This contributes to the growing body of literature on the potential of LLMs to augment scientific discovery.\n2. Discovering better scaling laws has the potential to feed directly back to improving LLMs.\n3. The paper is well-written and clear (apart from being too broadly scoped in the introduction, as I complain about below).", "weaknesses": "1. The biggest challenge of scaling laws is actually *formulating the set of independent & dependent variables* that make sense to model. For instance, recent works had the insight to propose new axes of scaling such as the vocabulary size $V$ (Tao et al., 2024), the amount of unique data $U$ (Muennighoff et al., 2024), or the domain mixture (Ye et al., 2024). Therefore, the sense of \"scaling law discovery\" here is limited — the agent is not asked to discover entirely new axes of scaling, but instead to **generate an equation that fits the given data**.\n2. Arbitrarily expressive equations can get arbitrarily close to modeling the given data perfectly. Therefore, I find that the experimental setup (which only measures predictive accuracy) does not really evaluate whether models generate *plausible* scaling laws that are consistent with prior knowledge. The authors provide some qualitative examples of human-discovered and agent-discovered scaling laws, but I do not have the expertise to compare their plausibility, and defer to the judgment of my fellow reviewers.\n3. The above points themselves are not themselves reasons for rejection, but I find some of the writing in the abstract & introduction to be not well scoped (or even misleading). The authors should make it clear right away that the agent is *given* the scaling law setup and experimental results; they are not being asked to formulate the variables of interest or execute the experiments. A re-ordering of the introduction would also be fitting: rather than starting with the importance of scaling laws and then discussing progress on AI agents, I think it is better to start with the discussion of AI agents and identify scaling law discovery as a particular testbed. This is because the main contribution is actually about agent evaluation, and not knowledge on scaling laws.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3nMhdB7liA", "reviewer": "Reviewer_qpNi", "summary": "This paper introduces SLDBench, a benchmark for evaluating scaling law discovery across seven tasks. The paper also introduces SLDAgent, an agent that co-evolves both the generator that proposes law expressions (including their functional forms) and the optimizer that sets values of parameters in the laws. SLDAgent works by prompting an LLM to propose modifications to both of these functions. \n\nResults: The paper finds that SLDAgent outperforms all existing baselines, including agent-based and LLM-based methods augmented with command-line code agents. They also find that when SLDAgent is paired with GPT-5, it outperforms human experts on all tasks in SLDBench. They show that SLDAgent can be used downstream to optimize hyperparameters for pretraining and to select a pretrained LLM for finetuning.", "questions": "1. Are there any patterns in the kinds of changes that LLMs make over rounds? How much diversity is there for a single task/SLDAgent setting across seeds?\n2. Line 289: The choice to clip does not feel well-justified; how do the results look if there is no clipping?\n3. Do some of the baselines similarly store candidate programs and perform the best performing program? (see 1c in weaknesses)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Experiments are extensive, evaluating a variety of baselines on many tasks. Experiments with SLDAgent + different models show that SLDAgent performance can scale with model size. Analyses are thorough, and the paper also shows that SLDAgent can be used downstream in real-world settings (hyperparameter selection, LLM selection).\n\n2. Novelty: While existing works have studied AI agents for science, none have been proposed specifically for evaluating scaling laws.\n\n3. The paper is well-written and easy to understand.", "weaknesses": "1. The comparisons between SLDAgent and other methods is not totally fair.\n\n     - 1a. Comparisons between SLDAgent and other baselines do not control for the number of LLM calls. Results for SLDAgent reflect performance after 50 rounds – do these SDLAgent runs then involve many more LLM calls (or other forms of compute) than other agent baselines? \n\n    - 1b. It is hard to know how much of the performance of SLDAgent is driven by the LLM versus the optimization and generator subroutines; while the experiments evaluating SLDAgent with different LLMs suggest that the LLM's evolution decisions have an impact on SLDAgent's performance, the paper does not compare these SLDAgent experiments to the performance of a human who had access to the same subroutines. \n\n     - 1c. We also do not know the impact of *storing programs with high-performing scores* in a database and returning the best program at the end; if baselines or humans had access to such a database, would they perform better?\n\n2. Section 4.3 reports analyses of the differences between human vs LLM written scaling laws. Instead of just qualitatively analyzing a few manually picked examples, it would be helpful to understand patterns in the differences through systematic analyses. Some ideas:\n   - human experts rate which of a human-written vs LLM-written scaling law is better\n   - analyzing # of parameters and other patterns in functional forms\n   - quantitatively analyzing asymptotic behavior", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)