How to Upscale Neural Networks with Scaling Law?
Authors: Ayan Sengupta
Yash Goel
Tanmoy Chakraborty
Venue: Accepted by TMLR ()

Abstract: Neural scaling laws have revolutionized the design and optimization of large-scale AI models by revealing predictable relationships between model size, dataset volume, and computational resources. Early research established power-law relationships in model performance, leading to compute-optimal scaling strategies. However, recent studies highlighted their limitations across architectures, modalities, and deployment contexts. Sparse models, mixture-of-experts, retrieval-augmented learning, and multimodal models often deviate from traditional scaling patterns. Moreover, scaling behaviors vary across domains such as vision, reinforcement learning, and fine-tuning, underscoring the need for more nuanced approaches. In this survey, we synthesize insights from current studies, examining the theoretical foundations, empirical findings, and practical implications of scaling laws. We also explore key challenges, including data efficiency, inference scaling, and architecture-specific constraints, advocating for adaptive scaling strategies tailored to real-world applications. We suggest that while scaling laws provide a useful guide, they do not always generalize across all architectures and training strategies.

Published in Transactions on Machine Learning Research (11/2025)
How to Upscale Neural Networks with Scaling Law?
Ayan Sengupta* Indian Institute of Technology Delhi
agwn.sennguplaleeee.d...cccc
Yash Goel'
ce121098, wwwp://..eeeidd...oooo ac in
Indian Institute of Technology Delhi
Tanmoy Chakraborty
tanchak@eer. sidd ac. ac.in
Indian Institute of Technology Delhi
Reviewed on OpenReview: https: // openreview. net/ forum? id= AL NNOU0fg.
Abstract
Neural scaling laws have revolutionized the design and optimization of large-scale Al mod- els by revealing predictable relationships between model size, dataset volume, and com- putational resources. Early research established power-law relationships model perfor- mance, leading to compute-optimal scaling strategies. However, recent studies highlighted their limitations across architectures, modalities, and deployment contexts. Sparse mod- cls, mixture-of-cxperts, retrieval-angmented learning, and multimodal models often deviate from traditional scaling patterns. Moreover, scaling behaviors vary across domains such as vision, reinforcement learning, and fine-tuning, underscoring the need for more nuanced approaches. In this survey, we synthesize insights from current studies, examining the the- oretical foundations, empirical findings, and practical implications of scaling laws. We also explore key challenges, including data efficiency, inference scaling, and architect urchitecttre-speciffe constraints, advocating for adaptive scaling strategies tailored real-world applications. We suggest that while scaling laws provide useful guide, they do not always generalize across all architect ures and training strategies.
1 Introduction
Scaling laws have become a fundamental aspect of modern AT development, espe- cially for large language models (LLMs). In recent years, rescarchers have iden- tified consistent relationships between model size, dataset volume, and com-- putational resources, demonstrating that increasing these factors leads to sys- tematic improvements
Transfer Leaming/Fine
Tuning
Pre-training
Theoretical
Sparsity
Inference
performance. These empirical patterns have been for- malized into mathematical principles, known scaling laws, which provide framework for understanding how the ca- pabilities of neural networks evolve as they grow. Mastering these laws is cru- cial for building more powerful AI mod- els, optimizing efficiency, reducing costs, and improving generalization.
Multimodality
Data
Mixture
Quantization Inference
Number papers
Figure 1: Papers surveyed under different categories. detailed list of papers is provided Table of Appendix A

---

Published in Transactions on Machine Learning Research (11/2025)
Category
Choshen al. (2024)
et al.
(2024b)
Ours
Covers neural scaling laws broadly Discusses fitting methodologies Analyzes architectural considerations Includes data scaling and pruning Explores inference scaling Considers domain-speedt speciffc scaling Provides practical guidelines Critiques limitations of scaling laws Proposes future research directions
Yes
No
Yes
Yes
Yes
Yes
No
Limited
Yes
No
Limited
Yes
No
Limited
Yes
No
No
Yes
Yes
Yes
Yes
Limited
Yes
Yes
Limited
Yes
Yes
Table 1: Key differences between our survey and existing surveys on neural scaling laws Choshen et al. 2024
Li et al. 2024b).
The study of neural scaling laws gained prominence with the foundational work of Kaplan et al. (2020) who demonstrated that model performance follows power-law relationship with respect to size, data, and compute. Their findings suggested that larger language models (LMs) achieve lower loss when trained on sufficiently large datasets with increased computational resources. Later, Hoofmann al.) (2022) refined these ideas, introducing the notion of compute-optimal scaling, which revealed that training moderate- sized model on larger dataset. is often more effective than scaling model size alone. However, recent stud- ies (Muennighoff et al. 2023 Caballero et al. 2023 Krajewski et al. 2024 have challenged the universality of these laws, highlighting cases where sparse models, mixtuur--ofeexeells architectures, and retrieval-augnen methods introduce deviations from traditional scaling patterns. These findings suggested that while scaling laws provide a useful guide, they do not always generalize across all architectures and training strategies.
Despite the growing importance of scaling laws, existing research remains fragmented, with limited synthesis of theoretical foundations, empirical findings, and practical implications. Given the rapid evolution of this field, there is a need for structured analysis that consolidates key insights, identifies limitations, and outlines future research directions. While theoretical studies have established the mathematical principles governing scaling, their real-world applications, such as efficient model training, optimized resource allocation, and improved inference strategies, are less explored. To address this gap, we reviewed over 50 research articles (Figure highlights papers on scaling laws on different topics) to comprehensively analyze scaling laws, examining their validity across different domains and architectures.
While prior surveys have made valuable contributions to understanding scaling laws, they have primarily focused on specific aspects of the scaling phenomenon (See Table D Choshen et al. (2024) emphasized statistical best practices for estimating and interpreting scaling laws using training data, while Li et al. (2024b) emphasized on mothodological inconsistencies and reproduction crisis in existing scaling laws. Our survey distinguishes itself by offering comprehensive coverage of architectural considerations, data scaling implications, and inference scaling areas that previous surveys either overlooked or addressed only partially,
2 Taxonomy of neural scaling laws
Understanding the scaling laws of neural models is crucial for optimizing performance across different do- mains. We predominantly explore the scaling principles for language models, extending other modalities such vision and multimodal learning. We also examine scaling behaviors in domain adaptation, inference, efficient model architectures, and data utilization. We highlight the taxonomy tree of scaling laws rescarch in Figure 2 with list of paper covered under each laxonomy branch in Table 2 As highlighted in Figure 1 neural scaling laws have been proposed predominantly for pre-training and fine-tuning scaling of large neural modols. Among the models studied, as highlighted in Figure 3a decoder-only Transformers dominate the subject, followed by vision transformers (ViT) and Mixture-of ofature-ff-Experts (MoE).
'Equal contribution
o *Corresponding author
2

---

Published in Transactions on Machine Learning Research (11/2025)
Scaling Laws
Commendation
Scaltng
Scaling
Pre-training
Post-training
Inference
Vision Models
Multimodal Models
Language learning
Finestuing
Compression
Retrieval
Inference Compute
Compression
Large LMS
Smal. LMs
Effeicet LMs
Figure 2: A taxonomy neural scaling laws.
Taxonomy
Cited Papers
Data Scaling
Muennighoff et al.
2023 ; Ye et al.
(2024]; Liu et al.
(024
Kang et al.
(022))
Que et al. (0024
Allen- Zhu & Li
2024
Tao et al.
(2024)
Pro-training Vision Models
Zhai ct al.
[022]:
Alabdulmohsin
2022
Pre-training Multimodal Models
Henighan et al.
2020
Aghajanyan al. (2023
Li
(2024a)
Pre-training + Language Models > Large
Kaplan et al.
(2020)
Hoffmann et al.
2022
Tay et al.
LMs
(2022); Caballero et al. (0023)
Pre-training + Language Models > Small
Hu et al.
2024
LMs
Pro-training > Language Models >
Effi
Clark al.
2022
Krajewski al.
2024
Yun al.
2024
cient LMs
Post-training >> Transfer learning
Hernandez et al.
2021
Post-training + Fine-tuning
Zhang et al. 202
4):
Chen et al.
(024cc
Lin et al.
2024bb
Post-training + Model Compression
Frantar et al.
(2023
Chen et al.
2024b
Inference > Retrieval
Shao et al.
2024
Inference + Inference Compute
Brown et al.
(0244)
Wu et al.
2024
Sardana et al.
2024
Inference > Compression
Dettmers & Zeltlemoyer
2023)
Cao et
(2024]
Kumar
al. (2024)
Criticisms
Sorscher al. (2023 Diaz & Madaio
2024
Table 2: Papers covered under different taxonomy.
20
30
Number of papers
of papars
20-
Number
8
10-
Number Fine-tuning Compute
Data
Model Parameters
Code Generation
Video Modeling
Image Recagintion
Language Generation
Tandberees
Tanderree
Laterts
(a) Archilecture-wiie wise statistics
(b) Variable-wise statistics
(c) Task-wise statistics
Figure 3: Number of papers studied in this survey for different model architectures (a), scaling variables (b),
and scaling tasks (c). The detailed paper list is provided in Table 13 of Appendix A
The most common neural scaling laws take the form of power laws (Equation where the model's loss (L)
or performance metric assumes to follow a predictable relationship with different scaling variables,
L(P.....
Σ
P-
(1)

---

Published in Transactions on Machine Learning Research (11/2025)
with appropriate scaling parameters B and fitting parameters a; for different scaling parameter P.. Figure 3b highlights that the number of model parameters and data size are the most common used scaling factors. Intuitively, the exponent 3, measures the rate of performance saturation: larger 3; implies faster improvement with scale, while smaller 3, indicates slower convergence. The empirical intuition of power laws in neural models stems from statistical physics and information theory principles, where performance improvements exhibit diminishing returns as scale increases. In neural networks, the marginal gain in loss reduction with respect to an increase in model parameters or data size tends to follow a sublinear pattern, indicating similar behaviour across scales. This observation aligns with complexity capatiiyy trade-offs, where each additional parameter contributes progressively less new information. As shown by Kaplan et al. (2020) and later formalized in [Hffmaan et al. (002)), the power-law exponents quantify how efficiently a model converts compute and data into predictive performance. The exact forms of all the scaling laws are highlighted in
Table 14 of Appendix A
Among all the tasks, Figure 3c suggests that language generation is the most common task used for de- veloping these scaling laws, where the training cross-entropy loss is widely used fit. the laws. Based on the values obtained empirically, the scaling laws are fitted with non-linear optimization, most commonly by running algorithms like least square and BFGS (Broyden-Fletchhe Fletcher-Gollffabbbbbbbee-.- like goodness-of of-fit metrics are used to validate the correctness of the fitted curves. We elaborate on the evaluation of neural scaling laws in Section 3 In the following sections, we review the existing literature on neural scaling across various domains.
Statistical methods
Model scaling includes both parameter and data scaling. Parameter scaling is often studied in decoder-only Transformers (Kaplan et al. 2020 Hoffmann et al. 2022), with newer works addressing small and efficient models (Hu et al, 2024 Clark et al. 2022). These studies establish power-law relationships between loss and model size or compute (Equation 1.. In parallel, power laws also implicitly encode the compute-optimal
frontier: the point where increasing parameters or data yields equivalent marginal benefits. This equilibrium condition forms the basis of the Chinchilla scaling rule, where D x N, derived by equating the partial derivatives of Equation] [ with respect to N and D.
Data scaling research has proposed laws for optimizing mixtures (Ye et al., 2024), repeated training expo- sures Muennighoff et al. 2023), vocabulary size (Tao et al. 2024), and knowledge capacity (Allen-Zhu & Li, 2024 These formulations oflen assume that information gain from additional data follows diminishing- return process similar to Zipfian or Pareto distributions observed in natural language frequency, justifying why scaling laws adopt a power-law rather than logarithmic form.
Pre-training scaling laws extend beyond language lo vision and multimodal settings. Vision models exhibit power-law scaling that saturates at large compute (Zhai et al.] 2022), while multimodal models demonstrate competition--to-yyeerg transitions as scale increases (Aghajanyan et al. 2023).
Post-training scaling captures fine-tuning and transfer learning behaviors. Transfer scaling shows larger pre-trained models yield better generalization with limited downstream data (Hernandez et al. 2021 Recent works propose scaling laws for PEFT (Zhang et al. 2024), downstream loss prediction (Chen et al. 2024c
and early stopping (Lin et al.) 2024b).
Inference scaling explores computo-cfficiet strategios during model deployment. Adaptive test-time com- pule (Chen et. al. 2024a Brown et al.] 2024) and retrieval augmentation (Shao al., 2024) allow small models to rival larger ones. Inference-specific scaling laws characterize the tradeoff between sampling cost
and performance (Wu et al. 2024).
Efficient model scaling addresses sparsity, quantization, and distillation. Sparse and MoE models provide multiplicative efficiency gains (Krajewski et al. 2024), while pruning and quantization ization laws enable compute- aware compression (Chen et al.] 2024b Cao et al. 2024).
Scaling behavior in reinforcement learning (RL) diverges from language or vision tasks, In single- agent RL, performance scales sublinearly with model size and environment interaction (Hilton et al., 2023 Horizon length, rather than task difficulty, determines scaling efficiency. In multi-agent games, predictable scaling laws govern coaputtett--erforman relationships, but generalization to complex domains like Chess
or Go remains limited (Neumann & Gros 2023). Meanwhile, graph neural networks (GNNs) lack stable
4

---

Published in Transactions on Machine Learning Research (11/2025)
Taxonomy Node
Addressed RQs
Model scaling
RQL, RQ2, RQ8
Data scaling
RQ3
Post-training scaling
RQ5
Inference scaling
RQ4
Efficient and compressed model scaling
RQ6, RQ7
Table 3: Mapping laxonomy categories to relevant research questions.
scaling laws; despite self-supervised loss improving with more data, downstream performance often fluctuates umpredictably [M et al., 2024
commendations, such as practical data laws and compression-aware training (Liu et al. 2024), and criticisms, which question the generalizability and repro- ducibility scaling laws (Sorscher et al. 2023 Diaz & Madaio 2024). Detailed discussion on these scaling law studies are provided in Appendix
Finally, the taxonomy captures two outer branches:
3 Fitting and validating scaling laws
Fitting scaling laws involves several key methodological choices that can significantly impact the final results and conclusions. The choice optimization approach, loss function, initialization strategy, and validation method all play crucial roles in determining the reliability and reproducibility of scaling law studies.
3.1 Optimization methods
The most common approaches for fitting scaling laws involve non-linear optimization algorithms like BFGS Seatn-lleecee-Gooddaee Shanno) (used by Frantar et al. (2023)). L-BFGS (used by Tao et al. 2024)
and least squares (used by Caballero et. al. (2023)). Some studies (Covert el al. 2024 Hashimoto 2021 also use optimizers like Adam or Adagrad, though these may be less suitable for scaling law optimization duc to thoir data-hungry nature and assumptions about gradient distributions. Recent works (Hoffmann) et al. 2022 Sorscher et al., 2023 Yun et al. 2024) emphasize that the choice of optimization method directly affects the stability of fitied exponents typically converge more reliably for low-dimensional, smooth objective surfaces, whereas first-order methods can oscillate under noisy residuals.
intercepts. Second-order methods such as BFGS and T-BFGS
3.2 Loss functions and objectives
Several loss functions are commonly used for fitting scaling laws:
Mean squared error (MSE): Emphasizes larger errors due to quadratic scaling (used by Ghorbani
et al. (2021)). )
Mean absolute error (MAE): Provides more robust fitting less sensitive to outliers (used by Hilton
ct al. (2023)).
Huber loss: Combines MSE's sensitivily to small errors with MAE's robustness to outliers (used by Hoffmann et al. (2022
While most studics minimize residuals between predicted and observed loss, the choice of loss function carries distinet theoretical implications. MSE magnifies the influence of outliers, which can distort estimated
power-law exponents, whereas Huber loss mitigates this by switching to linear penalty beyond a threshold. Hoffmann et al. (2022 used Huber loss down-weight outlier data points in the Chinchilla law, improving stability in under- trained regimes. In contrast, Muennighoff al. (2023) and Sorscher et (2023) employed MSE log-transforned loss, prioritizing proportional (relative) rather than absolute error.
5

---

Published in Transactions on Machine Learning Research (11/2025)
3.3 Initialization strategies
The initialization of scaling law parameters proves to be critically important for achieving good fits. Common approaches include grid search over parameter spaces (Aghajanyan et al. 2023), random sampling from parameter ranges (Frantar et al. 2023), and multiple random restarts to avoid local optima Caaallero eet al. 2023). Caballero et al. (2023); Li et al. (2024b) show that initialization sensitivity increases for broken or multi-regime scaling laws. Caballero et al. (2023) used multiple initializations per run to stabilize
piccowise fits, while Aghajanyyn et al.
(2023 proposed grid-bascd seeding for cross- scaling consistency. Proper initialization ensures convergence to the global rather than local optima, crucial for robust exponent estimation.
3.4 Validation methods
It is hugely important to understand if the scaling law fit achieved accurate and valid. Most of the papers surveyed lack in validating thoir fits. Several approaches can help validating the effectiveness scaling law fits. Statistical methods like computing confidence intervals can act as goodness-of-fit metric (Alabdul- mohsin et. al., 2022.. Furthermore, researchers can perform out-of-sample testing by extrapolation to larger scales (Hoffmann et al., 2022). Yun et al. (2024); Covert et al. (022)) additionally use RMSE-based model selection, likelihood based comparisons, and extrapolation validation. Covert et al. (2024) applied maximum likelihood estimattio to derive uncertainty intervals for fitted parameters, while Yun et al. (2024) validated on held-out extrapolated data points. Extrapolation testing remains the gold standard for evaluating scaling law robustness, whether the law generalizes beyond observed scales.
3.5 Limitations of fitting techniques
Li al. (0024b) revealed several critical methodological considerations in fitting scaling laws. Different optimizers converge to notably different solutions even with similar initializations, underscoring the need for careful justification of optimizer choice. Similarly, the analysis showed that different loss functions can produce substantially different fits when working with real-world data containing noise or outliers, suggesting that loss function selection should be guided by specific data characteristics and desired fit properties. Perhaps most importantly, the paper demonstrated that initialization can dramatically impact the final fit, with some methods exhibiting high sensitivity to initial conditions. Furthermore, many fits implicitly assume single-regime power laws, overlooking emerging evidence for broken multi-phase scaling (Caaalero et al., 2023 Systematic reporting of residual distributions, uncertainty intervals, and robustness checks is now regarded as best practice for reproducibility. Together, these findings emphasize the importance of thorough methodology documentation across all aspects of the fitting process from optimizer selection and loss function choice to initialization strategy - to ensure reproducibility and reliability in scaling law studies.
In the next section, we formulate key research questions (mapping between the taxonomy and research questions highlighted in Table 3) derived from these studies and present practical guidelines for leveraging scaling laws in real-world model development.
4 Research questions and guidelines
Grounded in the laxonomy of neural scaling laws (Figure 2 we identify key research questions spanning six dimensions: model scaling, architectural bottlenecks, inference scaling, data scaling, post-treining strategies, and efficient model design. For each, we synthesize multiple studies to extract overarching patterns, identify conflicting evidence, and propose actionable guidelines for researchers and practitioners navigating largo-scale model development.

---

Published in Transactions on Machine Learning Research (11/2025)
RQ1. Importance model and pre-training data size performance [taxonomy: model scaling pre- [ve-training]
Kaplan et al. (2020) established a power-law relationship:
ax
an
L(N,))
N N
x No.7
(2)
Hoffmann et al. (2022) refined this into a compute-optinal formulation:
B D
L(N, D) = No
+ DxN
(3)
Recent research has challenged linear extrapolations. Muennighoff et al. 2023 and Sardana et al. (2024) showed that training small models longer can outperform larger models, especially under coistrained data. Caballero et al. (2023) proposed Broken Neural Scaling Laws (BNSL):
(aN-a + -D- P LCN-" + dDD N 2 N.
No
L(N,D)
(4)
Where these laws work (and fail). As summarized in Table Kaplan-style laws are effective for large, dense autoregressive LMs trained abundant, ii.d. text with full convergence, but they break down when data is exhausted, models are under-trained, regimes depart from i.i.d. text. Chinchilla's 's compute- optimal form works well when both model and data are co-optimized on curated corpora, yet it can fail in multimodal or highly redundant-data regimes and assumes uniform data quality and linear cost. BNSL. accurately models performance when clear saturation or "phase changes" occur across scale, but it is sensitive breakpoint initialization, noise small-scale regions, and can mislead when extrapolating far beyond the fitted regimes. Figure
4 demonstrates an instance of "double-descent" booubbeedeceenttmmult--eeeeeeeed /mulli--eeimme behavior (Nakkiran
et al. 2021) where single-slope Kaplan fit systematically misses the curvature, while BNSL fit, with learned breakpoints (aunotated near 155 and 4450 training examples in the ligure), caplures the transition and yields non-negligible lower extrapolation error (RMSE of 0.02 vs. 0.14). This example highlights why piecewise formulations are needed once the learning curve crosses saturation or phase-change regions.
Syuthesis and guidelines
Model scaling success depends not only size but also training strategy, data quality, and saturation thresholds.
Use Table to match regime to law: Kaplan/Chinchilla /Chinchilla for dense, well-curated i.i.d. text scale; BNSL when empirical curves show slope changes saturation; and data-awwarerrquutiionawww formulations in data-limited settings.
Before extrapolating, test for regime breaks (as in Figure + if present, prefer piecewise fits and report breakpoint uncertainty to avoid overconfident forecasts.
Practitioners should allocate compute across parameters, data, and training duration based on observed inflection points. Use Kaplan/ /Chinchilla scaling when data is abundant; otherwise, extend training epochs adopt data-efficient curricula (see Figure 7ah.
RQ2. Scaling behaviors for different neural architectures [taxonomy: model scaling -> pre-training architecture]
>>
According to Tay et al. 2022 the vanilla Transformer consistently demonstrates superior scaling properties (P x O where P is the performance metric, C' represents compute, and a are fitting paramcters) compared to other architectures, even though alternative designs might perform betler al specific sizes. Architectural bottlenecks manifest differently across these designs. For instance, linear attention models like Performer and Lightweight Convolutions show inconsistent scaling behavior, while ALBERT demonstrates negative scaling trends. This finding helps explain why most LLMs maintain relatively standard architectures rather than adopting exotic variants. Furthermore, Zhai et al. 2022 revealed that ViT reveals that these

---

Published in Transactions on Machine Learning Research (11/2025)
Scaling Law
Primary Goal
Where It Works
Where It Doesn't Work
Kaplan
al. Predict model performance scales for model size and dataset size.
Large-scale autoregressive language models (e.g-: GPT-style) trained in pre-ssympootic regime with consistent data quality and full training to convergence.
Breaks down when data is exhausted or models under- train; fails for non-i..d. data distributions, trans- fer learning, and mixture-of- experts models.
2020
Hoffmann et al.
To establish a compute- optimal scaling law by balancing model size and dataset size for minimal loss under fixed compute.
Effective for large dense LMs trained curated text corpora where both model and data scaling optimized.
Fails under multimodal non-text regimes, fine- tuning stages, or domains with variable data redun- dancy; assumes uniform data quality and linear compute cost.
2022
Caballero
To capture multi-regime be- havior by allowing scaling exponents to change across parameter or data regimes (piecewise power laws).
Accurately models perfor- mance when clear salura- tion "phase change" oc- curs
Breaks extrapolation be- youd trained regimes; sensi- tive breakpoint initializa- tion and noise in small-scale regions.
2023
Table 4: Comparison of prominent scaling laws under RQ1 in terms of their primary goal, effective regimes, and known failure cases.
4.0
Actual Kaplan law
3.5
break
3.0
BNSL law
Test Joss
2.5
2.0
1.5
break at 450
1.0
0.5
0.0
200
400
600
800
1000
1200
Training data size
Figure 4: "Double descent" behavior is not well-captured by traditional power-law scaling laws such as Kaplan al. (2220). Piccewiso- power-law-bass parametric laws (Caballero al. 2023) (BNSL) captures the smooth transition across different loss regimes.
models exhibit double saturation, where performance platcaus at both very low and very high compute levels, suggesting architectural limitations specific to the vision domain (Equation 5). However, as shown by Li et al.] (2024a), simply scaling up vision encoders in multinodal models does not consistently improve performance, indicating that architectural scaling benefits are not uniform across modalitics.
aCC -) +c.
(5)
where E. denotes downstream error, C represents compute, and a, b. c, d are fitting parameters.
8

---

Published in Transactions on Machine Learning Research (11/2025)
Synthesis and guidelines
Architectural bottlenecks vary across domains and compute scales. Transformer inductive blases generalize best under scale.
Use architectures with proven scaling profiles (e.g,, vanilla Transformer) unless task-specific benefits out- weigh risks. For multimodal or domain-specialn setups, consult scaling behavior across compute ranges (Figure 7a
RQ3. Data strategies for performance scaling [taxonomy: data scaling]
Ye ct al.
2024)) proposed an exponential model for data mixing:
M Σ
L Li(ri...r) = cs + has exp
(6)
while
Liu et al. (2024] and Kang et al. optimize mixtures. The Domain-Continual Pretraining (D-CPT) law (Que et al. 2024 provides a theoretical
(024) developed proxy models (REGMIX, AUTOSCALE) to pre-
grounding on optimal mixture ratio between general and domain-specific data
A B. No where N represents the number of model parameters, D is the dataset size, r the mixture ratio, E, A, C, n, fitting paramcters.
C + Da (r+
D(N,
(7)
Where these laws work (and do not). Table summarizes elfective regimes for these scaling laws. The data-mixing formulation is effective for pre-training LLMs on heterogeneous corpora where the goal is compute-efficien composition, but becomes cumbersome when the number of data domains is very large due to the growth fitting parameters. D-CPT Que et al. 2024) is well-suited for continual pre-training, adapting a base model to new domains and supporting long-term updates, yet not designed to optimize single, static mixture from scratch. Figure 5 demonstrates the lack of effectiveness of traditional scaling laws under data-constrained settings, where Chinchilla law fails due to violations of data i.id condition in multi-epoch training. As epochs increase, Chinchilla law continues to predict steady loss reductions, whereas the observed test loss saturates; the data-constrained model (in the spirit of Muennighoff et al. 2023) tracks the plateau and recommends allocating budget to more epochs on smaller models rather than scaling parameters, capturing the regime dynamics.
Syuthesis and guidelines
Model performance is sensitive to data heterogeneity, mixture ratios, and interaction effects in multi-domain or continual settings.
especially
Use mixture-aware laws (e-g.: Ye et al. (0224) for heterogeneous pre-training; prefer D-CPT for continual updates; and switch to data-constrained prescriptions when repeated exposures drive saturation (as in Figure 5).
Replace manual corpus aggregation with predictive data mixing, Use D-CPT law when adapting to specific domains. Figure Ta outlines strategy paths based on data availability and domain constraints.
RQ4. Test-time scaling for better scaling efficiency [taxonomy: model scaling - inference scaling]
Recent research examining the relationship between test-time computation and model size scaling has re- vealed key insights. Brown et al. (2024) proposed that repeated sampling during inference significantly en- hances model performance, with coverage C' (fraction of problems solved) following an exponentiated power law relationship with the number of samples k; log(C) = ak-b, where are fitling parameters. Further ex- ploration by Wu et al. (2024) suggested that employing sophisticated test-time computation strategies (such as iterative refinement or tree search) with smaller models may be more cost-cffecttiv than using larger mod- els with simple inference methods. Their work establishes relationship between inference computational budget and optimal model size for compute-efficient inference, expressed log = 1.19 1.19 (N) 2.03.
9

---

Published in Transactions on Machine Learning Research (11/2025)
Scaling Law
Primary Goal
Where It Works
Where Doesn't Work
Ye et al. 2024
Predict optimal data compo- sitions for compute-efficient pre-training.
Pre-training LLMs, espe- cially for optimizing hetero- geneous datasets.
Scenarios with very large number of data domains, due to the complexity and mumber of fitting parame-
Que al. (0244)
Balance general and domain- specific data for continual pre-training.
Adapting pre-trained mod- els to new domains effi- ciently and for long-term updates.
Optimizing single, static data mixture from scratch,
as it is specifically designed for continual pre- pre-training.
Table 5: Comparison of prominent scaling laws under RQ3 in terms of their primary goal, effective regimes, and known failure cases.
4.5
Actual Chinchilla law
4.0
eata-constrained law
Test loss
3.5
2.5
2.0
5
10
25
30
35
40
Epochs
Figure 5: Importance of data-constrained scaling laws, where traditional scaling laws fail to incorporate the data-dependence factors in multi-epoch pre-training.
Synthesis and guidelines
Inference scaling offers complementary path performance, particularly where model reuse desired but compute cust must remain low.
Use adaptive compute, retrieval augmentation, or tree search for high-value queries. Integrate test-Lime scaling laws into deployment workflows (Figure 7b
RQ5. Scaling behaviors of model fine-tuning [taxonomy: model scaling + post-training scaling]
Fine-tuning scaling reflects how pre-trained models adapt across tasks and domains. Hernandez et al. 2021
introduced transfer scaling law based on effective data transferred D,:
D. = k(D))"
(8)
while Lin et al. 2024b refined this with rectified law:
L(D) = D. + D + E,
(9)
modeling diminishing returns from fine-tuning beyond a pre-learned threshold. In vision, A Abnar et al.
2021
linked downstream error to upstream error:
k(eus) +
(10)
and Mikami et al. (021) connected downstream accuracy to synthetic pretraining data size:
eDs aD +
(11)
10

---

Published in Transactions on Machine Learning Research (11/2025)
FLOPS to Loss to Performance (FLP) method (Chen et al. 2024c) predicted downstream performance from
pretraining FLOPs, and Zhang et al. 2024 showed LoRA scales nonlinearly under PRFT:
L(X, = A X x +E
(12)
D
Syuthesis and guidelines
Transferability scales with both model size and pretraining loss, but task difficulty, data availability, and
adaptation type mediate returns.
Use FLP or rectified laws to estimate post-training gains. Prefer PEFT for low-resource settings; switch full fine-tuning when compute permits. For domain adaptation, apply D-CPT strategies (Figure 7a
RQ6. Scaling efficiency and performance for sparse and efficient models [taxonomy: model scaling model compression]
As the demand for resource-efficient models grows, sparse architectares such as pruned networks and MoFs have emerged as promising alternatives to dense Transformers. These models aim to preserve the performance benefits of scale while reducing compute and memory overhead. |Faatar al. (2023) proposed a general sparse scaling law showing that sparsity acts multiplicative efficiency factor rather than changing the fundamental scaling behavior:
L(S,N, (as(1 (1cs111570 Sa cs) (1)
bx
ADD
bp
+
(13)
where S is sparsity, N is the number of non-zero parameters, and D is datasel size. In MoE models, where only a subset of parameters activated per input, Clark et al. (2022) proposed loss scaling relationship incorporating both model size and expert count:
log I, = log N + blog F. clog N log E d,
(14)
with denoting the expansion factor. This formulation was extended by Yun et al. (0024) to include dataset size:
log D,, log N E
c
+ +1)
Dr
dog N log
(15)
These results emphasize that scaling MoEs effectively requires balancing expert granularity with sufficient training data. Toward this, Krajewski et al. (2024) introduced a granularity parameter G to refine the Chinchilla-style formulation:
D(N,D,))-o - c (2 +
N + b
(16)
D
In parallel, structured pruning approaches have been formalized through the p2 law (Chen et al. 2024b); which relates post-prun cest-pruning loss to pre-pruning model size No, pruning ratio p, and post-training token count D:
L(No. Lop. = La (2)
Nc
Dc
E
(17)
No
No
DB
where Lo is the uncompressed model loss, is the pruning rate, No is the pre-pruning model size, D represents the number of post-training tokens, and Nc, Dc. E, a, B, are fitting parameters.
How the sparse/Mar MooE scaling functions evolved. Table t traces a progression from (i) sparsity- as-efficiency laws for pruned dense networks (Frantar et al., 2023 where multiplics standard size/dat terms, to (ii) MoE interaction laws that explicitly model the synergy between total parameters N and expert count. E. (including N x E. interaction terms)
YYun et. al. 2024), and finally to (iii) gramalarity-aware MoF.
11

---

Published in Transactions on Machine Learning Research (11/2025)
Scaling Law
Primary Goal
Where It Works
Where It Doesn't Work
Frantar
al.
To model sparsity (S) as multiplicative efficiency fac- tor on top of standard scaling laws.
For models with static, unstructured sparsity (e-g-- prumed networks) to find the optimal sparsity for a given compute budget.
For dynamic, structured spansity like Mixture- of of-Fxxerts (MoE), as it
2023
doesn't 't capture the complex interaction between experts.
Clark et al.
To predict MoE performance by modeling the synergis- tic interaction between model size (N) and the number of experts (E).
Specifically for MoE archi- tectures to balance total pa- rameters against the num- ber of experts for optimal performance.
For standard dense models or pruned networks where the concept of "experts" does not apply.
2022
Yun et al.
2024
Krajewski et al.
To refine MoE scaling by in- troducing a granularity pa- rameter (G), accounting for
For advanced MoE opti- mization where expert size is tunable, helping decide between many small experts fewer large ones.
When quantifiable 'granu- larity" metric unavailable, making it too specific for general MoE models.
2024
expert size and specialization.
Table 6: Comparison of prominent scaling laws under RQ6 in terms their primary goal, effective regimes, and known failure cases.
laws KKajewski et al. 2024 that introduce controllable G capturing expert specialization and yielding compute-optimal envelope across granularities.
Where these laws work (and do not). From Table the Frantar et al. 2023
formulation works well for
models with static, unstructured sparsity (e.g., pruned nets) to select optimal sparsity at a given budget, but does not capture the complex interaction patterns of MoE routing. The et al. (2022) Yun ot al. (2024) MoE laws are effective within MoE architectures to balance N against the number of experts E, yet they are not applicable standard dense/pruned models without experts. The gramularity-basee based law of Krajewski et al. (222g) is most useful for advanced MoE' optimization when expert size titzesppcialization is tunable, but it becomes less general when a mcasurable G is unavailable. Figure [ shows where the dense Transformer (dashed) underperforms family of MoE models with varying G' at the same FLOPs, and the optimal MoE envelope (solid) lies strictly below the dense curve across budgets. This exemplifies a case where dense scaling laws fail to capture attainable efficiency, while the granularity-aware MoE law correctly predicts that increasing expert gramularity (up to budget-dependent optimum) yields better compute quattty trade-offs.
budget-vs-loss curves
Synthesis and guidelines
Sparse models are
but require careful routing (MoE) and token- budget tuning (pruming)
outperform dense counterparts.
Match regime to law (Table 0 choose sparsity-efficiency laws for static pruning; interaction laws for MoE size-expert balancing; and granularity- laws when expert specialization a tunable knob and compute-optimal envelopes matter.
Use MoEs for general-purper purpose LLMs under compute limits. Apply pruning for deployment constraints.
For efficient inference, refer to Figure 7b
RQ7. Model scaling with low-precision quantization [taxonomy: model scaling - model compression quantization]
According to Dettmers & Zettlemoyer (022)) 4-bit procision appears to be the optimal sweet spot for maximizing model performance while minimizing model size. Additionally, research on scaling with mixed quantization (Cao et al. 2024), demonstrated that larger models can handle higher quantization ratios while maintaining performance, following an exponential relationship where larger models require exponentially fewer figh-precision components maintain given performance level. Kumar et al. (2024) developed a unified scaling law (Equation 18) that predicts both training and post-training quantization effects. It further
12

---

Published in Transactions on Machine Learning Research (11/2025)
MoE (G=2)
10:
MoE (G=4)
MoE (G=8)
Test loss
(G=16)
Dense Transformer
MoE (optimal scaling)
10a
1018
10
10
10
10
Training budget (FLOPs)
Figure 6: Compute optimal scaling of mixture-oofexpeets often better than dense transformers high- lighting the importance of sparser activations for achieving better performance at same compute cost. Scaling shown in log-log scale.
suggests that effects of quantizing weights, activations, and attention during training are independent and multiplicative.
L(N, D.P Pa; Prvv Poost) = AN + BD 3 + E 8PTQ.
(18)
where Pan, Pa, Pk. denote training precision of weights, activations and attentions, respectively, Prost de- note end-time weight-precision ôPTQ denotes loss due to post training quantization, and a, are fitting parameters.
Synthesis and guidelines
Scaling-aware quantization reduces memory while preserving performance. Larger models generalize better to low precision.
Apply mixed-precision for inference. Use quantization-aware training for smaller models. Refer to post-
training strategies (Figure 7b) to guide compression.
RQ8. Beyond modalities: scaling for multimodal models [taxonomy: model scaling multimodal models]
Multimodal scaling behavior builds upon, but does not replicate, unimodal trends. Henighan et al. (2020)
first proposed multimodal scaling using L(x) -r-a + B, where æ represents model size, data, compute. Alabdulmohsin et al. (2022) refined this into more flexible sigmoid-like form:
Loo (Lo Lz)a Bx
(19)
allowing transitions across saturation regimes. Aghajanyan et al. (2023) observed that smaller multimodal models exhibit competition between modalities, while larger models cross "competition barrier" and become synergistic. They proposed bimodal generalization of the Chinchilla law:
L(N, D,) = C(N, D.) + L(N, D,)
2
(20)
Aud
B
Ci, +
No.
11,
where Cij captures the degree of positive interaction between modalities i and j.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Development Objective
LLMs
Multimodal Models
Speciff Models
Multimodal scaling
Domein-contintons pre-training
Sufficient Data
(Aghajanyan et al., 2023)
(D-CPT law)
Yes....
No
Sufficient Compute
Sufficient Compute
No
Yes
Srale data and model (Kaplan/Chel CChinchilla law)
MoF scaling
Longer training (Sardana et al., 2024)
Data mixing scaling (Ye et al., 2024)
(Krajewski et al., 2024)
(a) Training strategies
Memory Constraint
Yes
No
Compression Strategy
Efficient Inference Needed
Pruning
Quantization
Yes
P2 law (Chen et al., 2024b)
Low-bit bit quantization (Cao et al., 2024)
Inference scaling (Wu et al., 2024)
(b) Inference strategies
Figure 7: Practical roadmap summarizing training and inference strategies grounded in our eight research questions and taxonomy branches. (a) Training scaling strategies can be utilized for pre-training or fine- luning unimodal and multimodal foundational and domain-adapled models. (b) Post-training inference strategies can be followed to ensure that the model is utilized efficiently for the downstream applications.
Syuthesis and guidelines
Multimodal scaling is governed by modality alignment and architectural balance more than raw model size.
Ensure models are sufficiently large to benefit from synergy across modalities. Prioritize modality balance in architecture and high-quality aligned datasets over isolated scaling. Refer Figure 7a when designing multimodal pretraining pipelines.
Cross-RQ synthesis
Data-efficient scaling (RQ1, RQ3, RQ5) consistently beats brute-force model expansion, as shown
Hu
et al. (024): Sardana et al. (2024).
Architectural innovations (RQ2, RQ6) tend scale poorly unless paired with precise training heuristics (e.g,, expert routing in MoEs).
Inference-aware scaling (RQ4, RQ7) enables small models to rival larger ones but is rarely included
current scaling laws
key research gap.
While the research questions synthesized above highlight the strengths and practical applications neu- ral scaling laws, they also expose several limitations, espocially in their generalizability, reliability under constraints and applicability modern model designs. In the next section, we critically examine these limitations and discuss the foundational assumptions that may longer hold models evolve.
14

---

Published in Transactions on Machine Learning Research (11/2025)
5 Criticisms of scaling laws
Diaz & Madaio (2024) challenged the generalizability of neural scaling laws, arguing that they fail in di- verse real-world AI applications. They argued that scaling laws do not always hold when AI models serve heterogeneous populations with conflicting criteria for model performance. Larger datasets inherently re- flect diverse commumities, making it difficult to optimize a single model for all users. Similar to issues in multilingual AI, increasing data diversity often leads to performance degradation rather than improvement. Universal evaluation metrics are inadequate for capturing these complexities, potentially reinforcing biases against underrepresented groups. The authors further argued that smaller, localized AI models may be more effective for specific communities, highlighting the need to move beyond one-size-fits-all scaling assumptions.
Beyond dataset expansion, data pruning contradicts traditional scaling laws by demonstrating that perfor- mance improvements do not always require exponentially more data. Strategic pruning achieves comparable or superior results with significantly fewer training samples (Sorscher et al. 2023) Not all data contributes equally, and selecting the most informative examples enables more efficient learning. Experimental validation on CIFAR-10, SVHN, and ImageNet shows that careful dataset curation can surpass traditional power-law improvements, questioning the necessity of brute-force scaling.
Despite their significant impact, many studies scaling laws suffer from limited reproducibility (see Ta- ble 15 in Appendix BB due to proprietary datasets, undisclosed hyperparameters, and undocumented train- ing methodologies. The inability to replicate results across different computing environments raises concerns about their robustness. Large-scale experiments conducted by industry labs often depend on private infras- tructure, making independent verification challenging. This lack of transparency undermines the reliability of scaling law claims and highlights the urgent need for open benchnarks and standardized evaluation frame- works to ensure reprodueibility. Furthermore, the field's avoidance of rigorous scaling exponent analysis constitutes critical oversight. While exponents indeed vary across models, datasets, and hyperparame- ters, this variability demands investigation rather than dismissal. This deliberate analytical gap undermines confidence in extrapolation claims and raises questions about whether observed scaling behaviors represent genuine properties or experimental artifacts.
Goel et al. (2025) further questioned the validity and practicality of neural scaling laws by emphasizing their computational inefficiency, environmental unsustainability, and deployment constraints. They argued that scaling approaches often overlook diminishing returns, particularly the disproportionate computational and carbon emission costs associated with incremental performance gains. For example, their analysis (illustrated in Figure 8] showed that modest improvements in model performance demand exponential increases in resource usage, thus challenging the assumption that scaling laws represent sustainable progress. To address these limitations, they proposed a systematic shift towards "downscaling", advocating for smaller, domain- adapted models trained with carefully curated datasets. This downscaling approach not only mitigates environmental impacts but also promotes wider accessibility by lowering computational and financial barriers. Their study highlights an urgent need for revising the prevailing scaling paradigm to ensure more efficient and responsible AT development.
6 Beyond Scale: Future Directions for Practical and Sustainable Al
While neural scaling laws have provided valuable insights, they often mis-predict in settings that deviate from the classic, dense, pee-trainin--o--iidd-to.. paradigm. In particular, real deployments involve multiple com- peting constraints (latency, energy, memory), non-i.i.d. data (duplicates, domain drift), modular inference (retrieval, routing, MoE), and objective functions beyond cross-entropy. To make scaling laws decision- useful (we highlight a guideded roadmap toward better scaling practices in Figure Dt we must move from single-metric power-laws to modulor, cost-aware, and inference-aue aware formulations that generalize across data quality, domains, and hardware.
Reframing scaling laws for rcal-world constraints. Future scaling laws must account for compute budgets, hardware latency, and energy consumption. This includes integrating training-inferet trade- offs, evaluating real-world performance under quantization or pruning, and predicting effectiveness across
15

---

Published in Transactions on Machine Learning Research (11/2025)
Environmental cost
300
Carbon emission (in tons)
Environmental cost
Carbon emis sion (in tons)
22
1.6
15000
Test loss
200
Test loss
2.0
1.4
10000
100
1.8
Performance
Performance
0
20
40
60
80
100
2000
4000
6000
8000
10000
Model size (in Billions)
Training token size (in Billions)
(a) Environment costs of increasing model size
(b) Environment costs of increasing training data
Figure 8: As synthesized by Goel al. 2025 performance scales logarithmically with model and pre- training data sizes, whereas, environmental factors like Carbon emission during pre-training increases linearly. Outgrowing environmental cost highlights the infeasibility of power-law governing scaling laws for scaling up beyond a fixed limit.
Goal: Better scaling
Safety robustness: transfer efficient, safety constraints
Real-wortd constraints: Incorporate practical constraints with Pareto front
Downscale: Small/edge models; prune; intelligent data selection
Multi-loss objective: curate, dedup and diversity
Data-quatity: curate, dedup and diversity
Current limits
Data volume quality: duplicated, noisy and low- diversity data
Single-metric target: Only optimizes loss objective
Limited practicality: Non- reproducible assumptions, limited practical applications
Figure Roadmap from "what exists" to "better scaling." The diagram contrasts current limits (left) data volume over quality, training-only laws, single-metric targets, and dense-only assumptions, with road to better scaling (right) that prioritizes: (1) Data Quality (curate diverse high-quality datasets, synthetic duplicated data); (2) Downscaling for small/edge models with quantization and pruming; (3) Incorporating Compute & Energy constraints (budget, carbon, memory) while deriving the optimal configuratio;; (4) Multi-objective optimization (Pareto trade-offs over loss loss/latenge /ss//lateecy/ee /energy//////eeeee teatenyy/nergy/roobsstee rrobustness); (5) Robustness & Safety (reliability of scaling laws in practical applications).
resource-constrained environments. Concretely, we advocate reporting Pareto fronts over (loss, latency, eneryy, memory) and learning cost-qware exponents that vary with hardware and batch/sequence shapes. Empirically, the same architecture can move across Parcto fronts as tokenization, parallelism strategy, KV-cache policy changes; future laws should parameterize these deployment knobs.
Designing for downscaling. Rather than building ever-larger models, the field should invest in scaling laws for small language models trained with optimal data, sparsity, and inference strategies. The emergence
16

---

Published in Transactions on Machine Learning Research (11/2025)
of 1-3B parameter models that rival 13B+ models (Hu et al. 2024 highlights the viability of compact yet performant systems. Building on this, Goel et al. (0025) formalize downscaling law that predicts when an ensemble of smaller, pruned models will outperform a single larger model under the same compute budget. Let Lo denote the loss of a base model and n the mumber of models in a doep ensemble; the expected ensemble loss scales as [(n) = To b + & with the p² post-pruning law (Chen et al,, 2024b) (capturing the effect of pruning ratio and post-training tokens on loss), they derive a compute-pariee condition ensuring that an n-way ensemble of pruned models attains strictly lower loss than the base model at equal cost: (+iT+) (n 1)7 6. where Ng is the pre-
with positive, lask/model-specill fit parameters a, b. Combining this
pruning parameter count and (7,6) are the pruning/fince /finntnnnntinggg.e exponents inherited from p Practically, this law trades model size, pruning/ fiinnn///ntttingggg and ensemble width to achieve the best attainable quality at edge/mobile budgets. These downscaling laws complement our emphasis on token quality, routing capacity, and quantization depth by offering a closed-form criterion for selecting the number of pruned/ /omied/qqatizd replicas that optimizes quality under tight deployment constraints.
Multi-objective scaling optimization. Current scaling laws often predict accuracy at scale but ignore trade-offs between accuracy, compute, and robustness. Future work should develop mulli-objecctive scaling fromeworks that balance these factors to guide architecture and dataset design more holistically. We recom- mend framing scaling as a multi-objective program with constraints on worst-case error error/robuse /robustness, privacy, and environmental factors. Analytically, this suggests replacing single power-laws with families of condi- tional laws L(N, D,C) indexed by constraint bundle C, and optimizing over Pareto-efficient frontiers. For instance, we can propose an augmenting loss function with compute cost, defining a multi-objective target min(L + XC) where C is compute or energy expenditure. This shifts scaling evaluation toward efficient, sustainable regimes.
Inference-aware and modular scaling laws. Traditional scaling laws assume fixed inference procedures. However, our synthesis in RQ4 and RQ7 shows that test-time compute allocation via sampling, retrieval, or routing can drastically affect performance. Future scaling formulations should modularize inference and allow flexible compute allocation per task or query. We propose inference-awart scaling that introduces explicit. knobs for retrieval volume, routing granularity (MoF experts per token), and speculative /iterativee decoding depth. The resulting laws predict quality as a joint function of training tokens, model parameters, and inference budget. This also enables per-query optimal policies (e.g., adaptively increase retrieval or expert budget for hard inputs).
Data quality over quantity. Instead of expanding datasets indiscriminately, laws like REGMIX Liu et al. 2024) and D-CPT (Que et al.] 2024) emphasize optimized data composition. Future models should prioritize informative examples and track dataset efficiency across tasks. We argue for quality-weighted token accounting: let Det = Σ; wi with weights w, reflecting information value (deduplication, diversity, noise filtering, curriculum). Scaling exponents should be learned over Den instead of raw token counts. This unifics observations that carefully curated repeated data can outperform naive laws and explains regime changes (e.g,, when data pruning "beats" the baseline power-law).
Domain-specific laws (RL, diffusion, robotics). Many domains break classic power-law assumptions (more descriptions in Appendix A In RL, non-stationarity, horizon length, and exploration alter sample complexity; scaling should condition on horizon and environment entropy. In diffussio models, quality depends on training scale and sampling heessschedulers; laws must tie generation compute to FID/IS and downstream metrics. In robotics, data is interactive, multi-modal, and safety-bound; scaling should model demonstration quality, sim2real gaps, and control latency. Dedicated, measurement seeeooedeettgrouddeedd laws will avoid the "one-size-fits-ll"" fallacy.
Transfer, safety, and robustness at scale. Power-laws fitted on pre-training losses can overestimate downstream reliability. Future laws should co-model transfer efficiency, interference between tasks (negative transfer), and safety constraints (adversarial robustness, specification gaming). This suggests hicrarchical scaling: pre-training curves that map into downstream task familics via transfer operators, with robustness penalties that widen at larger scales unless mitigated by data curation alignment.
17

---

Published in Transactions on Machine Learning Research (11/2025)
What "good" future laws look like. A practical scaling law should: (i) admit deployment knobs (la- tency, memory, precision, cache policy), (ii) be inference-auare (retrieval/routing budgets), (iii) operate on quelity-weighted tokens, and (iv) expose Pareto fronts and constraint-awary optima. Such laws turn scaling from curve-fitting into design guidance for real systems, across both upscaling and downscaling regimes.
7
Conclusion
This survey provided a comprehensive analysis of neural scaling laws, exploring their theoretical foundations, empirical findings, and practical implications. synthesizod insights across various modalities, including language, vision, multimodal, and reinforcement learning, to uncover common trends and deviations from traditional power-law scaling. While early research established predictable relationships between model size, dataset volume, and computational resources, more recent studies have shown that these relationships are not universally applicable. Sparse architectures, retrieval-augrnented models, and domain-specific adapla- tions often exhibit distinct scaling behaviors, challenging the notion uniform scalability. Furthermore, advancements in fine-tuning, data pruming, and efficient inference strategies have introduced new perspec- tives on compute-optimal scaling. Despite their signíficance, scaling laws remain an evolving area of research, requiring further refinement. to address real- world deployment, challenges and architectural innovations.
Limitations
While this survey provides a broad synthesis of neural scaling laws, it primarily focuses model size, data scaling, and compute efficiency. Other important aspects, such as hardware constraints, energy consumption, and the environmental impact of large-scale Al training, are not deeply explored. Another limitation is the reliance on prior empirical findings, which may introduce variability due to differing experimental sctups and proprietary datasets. Without access to fully reproducible scaling law experiments, some conclusions remain dependent on the methodologies employed in original studies.
Acknowledgments
T. Chakraborty acknowledges the support of the IBM-IITD Al Horizons network and Rajiv Khemani Young Faculty Chair Professorship in Artificial Intelligence. He acknowledges the support of Google GCP Grant for providing the necessary computational resources.
References
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training, October 2021. URL
02095, MxXiv:2110.00090
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models, Jamuary 2023. URL http:://axixv www.ooggg///.//ee..///...o.e.eee htrrg/abs/2301... 2301. 03728 arXiv:2301.032
Ibrahim Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision, November 2022. URL
arXiv:2209.06644
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: part 3, 3, knowledge capacity scaling laws, April 2024. URL http://aaaa...orr/aa////0...... 05405 arXiv:2404.05405.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws, February 2021. URL
06701 arXiv:2102.06701 version: 1.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws, June 2024. URL http:///www...rr///////00/0.0../
arX0v:2402.01092.
18

---

Published in Transactions on Machine Learning Research (11/2025)
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: scaling inference compute with repeated sampling, December 2024. URL
2tw.a////40022777200 21787 arXiv:2407.21787.
Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb, Distillation scaling laws, 2025. URL
08606
Ethan Caballero, Kshitij Gupla, Irina Rish, and David Krueger. Broken neural scaling laws, July 2023. URL http://arxiv.orz wttp://rrr.abb/......soo /tabs/2211.... 14891 mnXiv:2210.1489)
Zeyu Cao, Cheng Zhang, Pedro Gimenes, Jianqiao Lu, Jianyi Cheng, and Yiren Zhao. Scaling laws for mixed quantization in large language models, October 2024. URL http://arxiv. wttp:///wwrr/ba//11.o.. 06722.
arXiv:2410.06722.
Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou, Are more Ilm calls all you need? Towards scaling laws of compound inference systems, June 2024a. URL http://arxiv. org/ /tabs//43..om 02419 arXiv:2403.02419
Xiaodong Chen, Yuxuan Hu, Jing Zhang, Xiaokang Zhang, Cuiping Li, and Hong Chen. Scaling law for post-training after model pruming. arXiv preprint or N4u:2411.10277 2024b.
Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, and Heng Ji. Scaling laws for predicting downstream performance Ims,, October 2024c. URL htttp://axii.. www.eiiww...oooeeeooooeoe www.obrg/////eee/ee//.ooeeeeee htwb//410.com arXiv:2410.05527.
Leshem Choshen, Yang Zhang, and Jacob Andreas. A hitchhiker's guide to scaling law estimation, 2024. URL https://arxiv.
11840
Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew Johnson, Katie Millican, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Jack Rae, Erich Elsen, Koray Kavukcuoglu, and Karen Simonyan. Unified scaling laws for routed language models, February 2022. URL http://arxiv.. htg/abss2222..01663 arXIv:2202.01169.
Ian Covert, Wenlong Ji, Tatsunori Hashimoto, and James Zou. Scaling laws for the value of individual data points in machine learning, 2024. URL https://arxivv ortp:/i/i.rrr//a////05..../ 20456
Tim Dettmers and Luke Zeitlemoyer. The case for 4-bit. precision: k-bit Inference Scaling Laws, February 2023. URL
09720 arXiv:2212.09720.
Fernando Diaz and Michael Madaio. Scaling laws do not scale, July 2024. URL http://arxiv. org/ abs/ 2307.03200 arXiv:2307.03201.
Elias Frantar, Carlos Riquelme, Neil Houlsby, Dan Alistarh, and Utku Evei. Scaling laws for sparsely-connected foundation models, September 2023. URL http://arxii: wwwaarwwvv..ooraaaaoaaoae arg/abs/2309..o 08520 arXiv:2309.08520.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, October 2022.
URL http://arxv..rrrr abs/:/ab///// httpg//bs//21..... 2210. 10760 arX1v:2210.17760.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, June 2024. URL http://arxxxio.orr
arXiv:2406.04093
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry, Scaling laws for neural machine translation, 2021. URL URL/attps://///te/t https://arxxv..rr
abs/ 07740
19

---

Published in Transactions on Machine Learning Research (11/2025)
Yash Goel, Ayan Sengupta, and Tanmoy Chakraborty. Position: Enough of scaling llms! lets focus downscaling. aiv preprint ar arXiv:2505.0098: 2025.
Tatsunori Hashimoto. Model performance scaling with multiple data sources. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 URL https:
of Proceedings of Machine Learning Research, pp. 4107 4116. PMLR, 18 24 Jul 2021.
//proceedings... mor..mmr..mm.m.m httroeeeiigg.m....ppe//ss/////aa
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling, November 2020. URL arXiv:2010.14701.
14701
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer, February 2021. URL
01293. arXiv:2102.01298,
Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning, February 2023. URL
13442 arXiv:2301.13442.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, March 2022. URL http:://axxv..rrggeeetee hetgg/abs/22000 15556 arXiv:2203.15556
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXie preprint arXiv:2106. 09685. 2021.
Shengding HU, Yuge Tu, Xu Han, Chaoqun Hc, Ganqu Cui, Xiang Long, Zhi Zheng, Yewci Fang, Yuxi- ang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm: unveiling the potential of small language models with scalable training strategies, June 2024. URL http://arxiv. http://axvvv..r/aaaa/// arggabb//404..o 2404/003444555455
arXiv:2404.06395.
Marcus Hutter. Learning curve theory, February 2021. URL http://arxiv.org/ab//220...000 arXiv:2102.04074.
Tian Jin, Nolan Clement, Xin Dong, Vaishnavh Nagarajan, Michael Carbin, Jonathan Ragan-Kelley, and Gintare Karolina Dziugaite. The cost of down-scaling language models: fact recall deteriorates before in-context learning, October 2023. URL
(0000000000068000 arXiv:2310.046600
Andy L. Jones. Scaling scaling laws with board games, April 2021. URL http://arxiv.org/aa 03113 arXiv:2104.03113.
Feiyang Kang, Yifan Sun, Bingbing Wen, Si Chen, Dawn Song, Rafid Mahmood, and Ruoxi Jia. Autoscale: automatic prediction of compute-optimal data composition for training 1lss, December 2024. URL http: 20177 2XXiv:2407.20177
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, January 2020.
URL http:://aaxi..orrrab//000......
2tt.//001.00000 083611 arXiv:2001.0836)
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michal Krutul, Szymon Antoniak, Kamil Cicbiera, Krystian Król, Tomasz Odrzygóådz, Piotr Sankowski, Marck Cygan, and Scbastian Jaszezur. Scaling laws for fino-gra fino-grained mixture of experts, February 2024. URL lttt:://arxiv.o/ org/
abs/ abs/2402 07871 ar 0777022402.07871.
20

---

Published in Transactions on Machine Learning Research (11/2025)
Tanishq Kumar, Zachary Ankner, Benjamin F. Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision, November 2024. URL URRLhttp://// httt:://aatiiii http:://axxvv.og/////// http:/..rrg/bb///1111/1 2091//00//111133300
arXiv:2411.04300.
Bozhou Li, Hao Liang, Zimo Meng, and Wentao Zhang. Are bigger encoders always better in vision large arXiv:2408.0065
models?, August 2024a. URL http://axxiv..rgabss///////
Margaret Li, Sneha Kudugunta, and Luke Zettlemoyer. Misfitting scaling laws: survey of scaling law fitting techniques in deep learning. In The International Conference on Learning Representations (ICLR), October 2024b. URL https: http:///paaaareeeeeeeeeee//ea/aa
Zhengyang Liang, Hao He, Ceyuan Yang, and Bo Dai. Scaling laws for diffusion transformers. arXiv preprinl 1wXiv:241000818// 2024.
Fanqi Lin, Yingdong Hu, Pingyue Sheng, Chuan Wen, Jiacheng You, and Yang Gao. Data scaling laws in imitation learning for robotie manipulation. ar preprint orXiv:2410. 18.44777, 2024a.
Haowei Lin, Baizhou Huang, Haotian Ye, Qinyu Chen, Zihao Wang, Sujian Li, Jianzhu Ma, Xiaojun Wan, James Zou, and Yitao Liang. Selecting large language model to fine-tune via rectified scaling law, May 2024b. URL htttp://arxiv.orgge htwpg/abs/2402..0 2etp//2222222222422/ 02314 arXiv:2402.02314.
Licong Lin, Jingleng Wu, Sham M. Kakade, Peter L. Bartleu, and Jason D. Lee. Scaling laws in linear regression: Compute, parameters, and data, 2024c. URL https: www./oggg/////////////o.//.eee// abs/2406...../////// 0 08466
Jack Lindsey, Adly Templeton Tom Conerly, Jonathan Marcus, and Tom Henighan. Circuits up-
dates april 2024, 2024. URL http:://ttaaattooee.rr...//ii/ps
scaling- laws
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: data mixture as regression for language model pre-training, July 2024. URL
http://arxiv. www.xiv.org/abba ht///bb//2400 01492 2XXx:2407.01492
Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Inference-time scaling for diffusson models beyond scaling denoising steps. arXie preprint arXiv:2501.09733 2025.
Qian Ma, Haitao Mao, Jingzhe Liu, Zhehua Zhang, Chunlin Feng, Yu Song, Yihan Shao, and Yao Ma. Do neural scaling laws exist on graph self-supervised learning?, August 2024. URL http://arxiv. org/ /b////////// 2408. 11243 ar wrXiv:2408.1122
Hiroaki Mikami, Kenji Fukumizu, Shogo Murai, Shuji Suzuki, Yuta Kikuchi, Taiji Suzuki, Shin-ichi Maeda, and Kohei Hayashi. A scaling law for synthetic-to-real transfer: how much is your pre-training effective? October 2021. URL httt:://axxii wwwpevvvvorrgaaeeeeeee hbs///bb//21../ 11018 1111
Niklas Muennighoff, Alexander M. Rush Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling lata-constrained language models, October 2023. URL /ttt:://axxi... wwwpaiiiwrr....eeee..eeee abs/ 2305. 16264. arXiv:2305.1626 16264.
Prectum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Πlya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Esperiment, 2021 (12):144003, 2021.
Oren Neumann and Claudius Gros. Scaling laws for a multi-agent reinforcement learning model, February 2023. URL
arXiv:2210.0088
Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, Xu Tan, Jie Fu, Wenbo Su, Jiamang Wang, Lin Qu, and Bo Zheng. D- cpt law: domain-specifts continual pro-training scaling law for large language models, June 2024. URL http://arxív. http://aariv.orr//bb///aaae wttp:/ba////...111111 arXivv2406.01335.
21

---

Published in Transactions on Machine Learning Research (11/2025)
Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: account- ing for inference in language model scaling laws, July 2024. httpt:ttpp//xxi/// http://wwwwrraa////.....0 arXiv:2401.00448.
Sebastian Sartor and Neil Thompson. Neural scaling laws in robotics. arXiv preprint arXiv:2405.14000 2024.
Rulin Shao, Jacqueline He, Akari Asai, Weijia Shi, Tim Dettmers, Sewon Min, Luke Zettlemoyer, and Pang Wei Koh. Scaling retrieval-based language models with trillion-token datastore, July 2024. URL
http://arxiv.org/ /2407 12854 arXiv:2407.12854.
Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold, April 2020. URL http://arxiv.orr wttp/////////..o..////////////o. 10802 #:Xiv:2004.10800
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 1lm test-Lime compute optimally can be more effective than scaling model parameters, August 2024. URL http://arxiv. wwtp://///a./////////./// 03314 arXiv:2408.0333 03.314...
http:://araaa.orrr///////88..
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning, April 2023. URL http://arxiv. http:://aaxiv.org/abba.... arg/abs/2206. 14486, MKXiv:2206.14480
Chaolan Tao, Qian Liu, Longxu Dou, Niklas Muennigholf, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: larger models deserve larger vocabularies, November 2024. URL
http://arxiv wwwtariii.or//raaeeeae ht///bs//440 13623 arXiv:2407.13623.
Yi Tay, Mostafa. Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q. Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: how does inductive bias influence scaling?, July 2022. RRLLhhtp://aaaa http://aaxvvv..aaaaaaee arggabs/2207, 20tp//2227..0551111 arXiv:2207.1055.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Scan Welleck, and Yiming Yang. Inference scaling laws: empirical analysis of compule-optimal inference for problem-solving with language models, October 2024. URL
attp://arxivv wwrzxiv.org////
htl.:2408.0002244
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Yunhua Zhou, Jun Zhan, and Xipeng Qiu. Data mixing laws: optimizing data mixtures by predicting language modeling performance, March 2024. URL http://
arxiv. org/abs/2403 16952 arXiv:2403.16952.
Yuanyang Yin, Yaqi Zhao, Mingwu Zheng, Ke Lin, Jiarong Ou, Rui Chen, Victor Shea-Jay Huang, Jiahao Wang, Xin Tao, Pengfei Wan, et al. Towards precise scaling laws for video diffusion transformers. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18155-1188111 2025.
Longfei Yun, Yonghao Zhuang, Yao Fu, Eric P. Xing, and Hao Zhang. Toward inference-optimaal mixture-of- expert large language models, April 2024. URL http://arxiv wrgg:/// 2404, 02852 arXiv:2404.02852
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers, June 2022.
URL http:///arriv..og/aa/////// 04560 arXiv:2106.044 04660000.
Biao Zhang. Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets 1lm finetuning: the ef- fect of data, model and finetuning method, February 2024. URL
17X11:2402.17199.
A Detailed scaling laws
A.1 Scaling laws of language models
Kaplan al. (2020) suggested that larger LMs improve performance by reducing loss through power-law scaling. However, this view evolved when studies showed that many large models were undertrained, and data
22

---

Published in Transactions on Machine Learning Research (11/2025)
scaling plays equally crucial role in compute efficiency (Hoffmann et al. 2022). More recent breakthroughs challenged traditional scaling assumptions. Broken Neural Scaling Law (BNSL) introducdd non-monotonic trends, meaning that model performance can sometimes worsen before improving, depending on dataset thresholds and architectural bottlenecks (Caballero et al.) 2023 Another exciting development came from small LMs, where optimized training strategies, such as a higher aala-lo-parameele ralio and adaptiive learning schedules, enable models ranging from 1.2B to 2.4B parameters to rival significantly larger 7B-13B models (Hu al. 2024). These findings reshape the fundamental assumptions of scaling laws, proving that strategic training outperform brute-force model expansion.
Modality
Paper
Key insights
Applicability
Larger models are more sample- efficient, needing fewer training ex- amples to generalize well.
Predicts model loss decreases with increasing parameters, used in early LMs like GPT-3.
Kaplan al. 2020
The best performance comes from balancing model size and data, rather than just increasing parame- ters.
Balances compute, model size, and dataset size for optimal ef- ficiency, as seen in Chinchilla.
Hoffmann al.
2022
Language
Performance does not always im- prove smoothly; there are inflection points where scaling stops working.
Identifies plase transitions, min- imum data thresholds, and un- predictability scaling behav- ior.
Caballero et al.
2023
Smaller models with better training can rival much larger models.
Demonstrates that smaleer mod- els with optimized training can outperform larger undertrained models,
Hu et al.
2024
ViTs follow power-law scaling but plateau at extreme compute lev-
Image classification, object detection, large-scale vision dalasels.
Vision
Zhai al.
2022
els, with beuefits primarily seen in datasets > images.
Multimodal models experience petition at smaller scales but transi- tion into syuergy model and token count grow.
Multimodal learning, mixed- modal generative models, cross-domain Al.
Aghajanyan et al.
2023
Scaling vision encoders in vision- language models does not always im- prove performance, reinforcing the importance of data quality over raw scaling.
Vision-language language models, image- text aligmment, multimodal scal- ing challenges.
Multimodal
Li et al.
2024a
Table 7: Critical neural scaling laws for language, vision and multimodal models.
A.2 Scaling laws in other modalities
In computer vision, ViTs exhibit power-law scaling when model size, compute, and data grow together, but their performance plateaus at extreme compute levels, with noticeable gains only when trained on datasets exceeding 1B images (Zhai et al. 2022). Mcanwhile, studies on scaling law extrapolation revealed that while larger models generally scale better, their efficiency declines at extreme sizes, requiring new training strategies al., (2022). In multimodal lcarning, an interesting phenomenon called the "competition barrier" has been observed where smaller scales different input modalities compete for model capacity, but models grow, they shift into synergistic state, enabling accurate performance predictions based on model size and token count (Aghajanyan et al. 2023)
to maintain performance (Alabdulmohsin
However, not all scaling trends align with expectations. Contrary to the assumption that larger is always better, scaling vision encoders in vision-language models can sometimes degrade performance, highlighting the fact that data quality and modality alignment are more critical than brute-force scaling (Li et al., 2024a). These findings collectively emphasize that scaling laws are domain-dependent optimal scaling strategies require careful balance wetween compute efficioncy, dataset quality, and architecture rather than simply
23

---

Published in Transactions on Machine Learning Research (11/2025)
Paper
Key insights
Applicability
Zhai (002
ViTs follow power-law scaling but plateau at extreme compute lev- cls, with benefits primarily seen in datasets >B images.
Image classification, object detec- tion, large-scale vision datasets.
Aghajanyan
et al.
Multimodal models experience competition at smaller scales but transition into synergy as model and token count grow, following a 'competition barrier."
Multimodal learning, mixed- modal generative models, cross- domain AL.
(023)
Li ct al.
2024a
Scaling vision encoders in vision- language models (VLMs) does not always improve performance, re- inforcing the importance of data quality over raw scaling.
Vision-language models, image- text aligmment, multimodal scal- ing challenges.
Table 8: Summary of key insights found in scaling laws paper for computer vision and multimodal domains.
increasing model size. Table summarizes the scaling laws of pre-trained models for language and other
modalities.
A.3 Scaling laws for domain adaptation
Pre-training and fine-tuning techniques have accelerated the adoption of large-scale neural models, yet the extent to which these models transfer across tasks and domains remains a key research question tied to scaling principles. Studies show that transfer learning follows a power-law where pre- gre-training amplifies fine- sinetuning offoctiveness, especially in small data regimes. Even with limitod downstream data, larger models benefit significantly from pre-training, improving generalization (Hernandez et al., 2021). b. In vision, pre-training sat- uration occurs due to apstrean-downstrean interactions, rather than just task complexity. Lower network layers quickly specialize in simple tasks, while higher layers adapt to complex downstream objectives (Ab- nar et al. 2021). Similarly, in synthetic-to-real transfer, larger models consistently reduce transfer gaps, enhancing generalization across domains (Mikami et al. 2021
Fine-tuning strategies scale differently depending on dataset size. Parameter-efficient fine-tuning (PEFT) techniques like low-rank adaptation (LoRA) (Hu al. 2021] and Frompt-tuning both are well-suited for small datasets, but LoRA performs best for mid-sized datasets, and full fine-tuning most effective for large datasets. However, PEFT methods provide better generalization in large models, making them attractive alternatives to full-scale fine-tuning (Zhang et al. 2024).
Scaling laws are also being utilized to accurately predict the fine-tuning performance of models. The FLP method (Chen et al. 2024c) estimates pre-training loss from FLOPs, enabling accurate forecasts of down- stream performance, particularly in models up 13B parameters. Further refinements like FLP-M improve mixed-dataset predictions and better capture emergent abilities in large models. Finally, the Rectified scal- ing law (Lin ct al.] 2024b) introduces two-phase fine-tuning transition, where carly-stage adaptation is slow before shilting into power-law improvement phase. This discovery enables compule-elficieut model selection using the "Accept then Stop" (AtS) algorithm to terminate training at optimal points.
We summarize those findings in Table g suggesting that transfer learning is highly scalable, but effective scaling requires precise tuning strategies rather than just increasing model size.
A.4 Scaling laws for model inference
Simply scaling up models not always the best way improve model performance. Chen et al. (2024a) suggested that more efficient test-time compute strategics can dramatically reduce inference costs while maintaining or exceeding performance. Instead of blindly increasing LLM calls, they further suggested
24

---

Published in Transactions on Machine Learning Research (11/2025)
Paper
Key insights
Applicability
Pro-training amplifics finc-tuning, par- ticularly for small datasets, and bene- fits larger models even under data con- straints.
Transfer learning, pre-training op- timization, few-shot learning.
Hernandez et al.
2021
Large-scale pre-training improves
Vision transfer upstream-downstremm mance interactions.
learning, perfor-
downstrcam performance, but effective- ness depends on upstrean-dowustream interactions, not task complexity.
Abnar et al.
2021
Optimal fine-tuning strategy depends dataset size: PEFT for small, LoRA for mid-scale, and full fine-tuning for large-scale datasets.
Fine-buiing strategies, parameter- efficient tuning, LoRA, full fine- tuning.
Zhang et al.
2024
Fiñe-tuning follows two-phāse trän- sition: slow early adaptation followed by power-law improvements, guiding computo-rficient model selection.
Computte-fffiient fiñe-tuning early stopping, model selection
Lin et al. 2024b
strategies.
Table 9: Key highlights from scaling of fino-tuned and domain-adapted models.
for allocating resources based on query complexity, ensuring that harder querics receive more compute while simpler ones use fewer resources. The importance of test-time compute strategies becomes even clearer when dealing with complex reasoning tasks. While sequential modifications work well for simple queries, parallel sampling and tree search dramatically improve results harder tasks. Adaptive compute-optimal techniques have been shown to reduce computational costs by 4x without degrading performance, allowing smaller models with optimized inference strategies to surpass much larger models (Snell et. al. 2024 Brown et al., 2024]. Advanced inference approaches, such as REBASE tree search (Wu et al. 2024], fur- ther push the boundarics of efficiency, enabling small models to perform on par with significantly larger ones.
Another breakthrough came from retrieval augmented models, where increasing the datastore size consis- tently improves performance without hitting saturation (Shao et al.) 2024). This allows smaller models to outperform much larger ones on wowledge-intensiie tasks, reinforcing that external datastores provide a more efficient alternative to memorizing information in model parameters.
A.5 Scaling laws for efficient models
Scaling laws have expanded beyond simple parameter growth, introducing new methods to optimize routing, sparsity, pruning, and quantization for efficient L.M.M scaling. Routing-based models benefit from optimized expert selection, but their returns diminish extreme scales, requiring careful expert configuration (Clark) et al. 2022). In contrast, fine-grained MoE models consistently outperform dense transformers, achieving up to 40x compute efficiency gains when expert granularity is properly tuned (Krajewski et al., 2024p. However, balancing the number of experts (E) is crucial, where models with 4-8 experts offer superior inference efficiency, but require 2.5 - 3.5x more training resources, making 16-32 expert models more practical when combined with extensive training data (Yun et al.] 2024p. Sparse model scaling offers another efficiency boost. Research has demonstrated that higher sparsity enables effective model scaling, allowing 2.15x more parameters at 75% sparsity, improving training efficiency while maintaining performance (Frantar et al. 2023). Additionally, pruning laws (P² scaling laws) predict that excessive post-training data does not always improve performance, helping optimize resource allocation in pruned models (Chen et al. 2024b Dettmers & Zettlemoyer (2023) showed that 4-bit quantization provides the best trade-off between accuracy and model size, optimizing zero-shot performance while reducing storage costs. Larger models tolerate lower precision better, following an exponential scaling law where fewer high-precision components are needed to retain performance (Cao al., (2024.. Mcanwhile, training precision scales logarithmically with compute budgets, with 7-8 bits being optimal for balancing size, accuracy, and efficiency (Kumar et al. (0244). Recent reserach has expanded into dist illation as well, developing mathematical framework that predicts how well
25

---

Published in Transactions on Machine Learning Research (11/2025)
Paper
Key insights
Applicability
Adaptive test-time compute strategies reduce computational costs by while maintaining performance, enabling smaller models to compete with much larger ones.
Test-time compute efficiency, inference reduction, compute-lime limited envi- ronments.
Brown et al.
2024
Advanced inference methods like RE- Higli-efficiency inference, performance optimization for small models.
BASE tree search allow smaller models match the performance of significantly larger ones.
2024
Increasing datastore size in retrieval- augmented models consistently improves performance under the same compute budget, without evident saturation. Routing-based models show diminishing returns larger scales, requiring optimal routing strategies for efficiency. Fine-grained MoEs achieve up 40x compule efficiency gains when expert gramularity optimized,
Reirieval Metrieval-auggmentell language models, knowledge intensive tasks, compute- efficient architectures.
Shao al.
2024
Routing-based models, MoEs, trans- former scaling.
Clark et al.
2022
Mixture of Experis models, Targe-scale compute efficiency.
Krajewski et al.
2024
Sparse model scaling enables predicting optimal spansity levels for given compute budgets.
Sparse models, structured sparsily timization, parameter reduction.
Frantar 2023
Table 10: Scaling laws of efficient models.
Paper
Key insights
Applicability
Clark et al. 2022
Routing-baset models show di- minishing returns at larger scales, requiring optimal routing strate- gies for efficiency.
Routing- based models, Molis, transformer scaling.
Krajewski et al. 2024
Fine-grained MoEs achieve up to 40x compute efficiency gains when expert gramularity is opti- mized.
Mixture of Experts models, large- scale compute efficiency.
Frantar et al.
2023
Sparse model scaling enables pre- dicting optimal sparsity levels for given compute budgets.
Sparse models, structured spar- sity optimization, parameter re- duction.
Table Scaling laws for routing, sparsity, pruning, and quantization.
a student model will perform based on the student model's size, the teacher model's performance and the compute budget allocation between the teacher and the student (Busbridge et al. 2025). We summarize these practical insights Table 10 for better readability.
A.6 Data scaling laws
Scaling models involves more than just increasing parameters; optimizing data mixtures, training duration, and vocabulary size also plays crucial role in enhancing performance and efficiency. Data mixing laws allow Al practitioners to accurately predict optimal data compositions before training, leading to 27% fewer training steps without compromising accuracy (Ye et al. 2024). Techniques like REGMIX optimize data selection using proxy models and regression, reducing compute costs by 90% compared to manual data selection (Liu et al.) 2024). Meanwhile, AUTOSCALE revealed that data efficiency depends on model scale, where high-quality data like Wikipedia helps small models but loses effectiveness for larger models, which benefit from diverse datasets like CommonCraw/ (Kang et al., (024). For continual learning, the D-CPT Law provided theoretical framework for balancing general and domain-speeific data, guiding efficient domain
26

---

Published in Transactions on Machine Learning Research (11/2025)
Paper
Key insights
Applicability
Predicts optimal data composi- tions before training, reducing compute costs by up to 27% while maintaining performance.
Pro-training optimization, data effi- ciency improvements.
Ye et al. 2024
REGMIX optimizes data mix-
Cofiptee-fffiient training, automated data scloction, largo-scale models.
Liu et al. 2024
tures using proxy models, achiev- ing 90% compute savings.
Language models can store 2 bits of knowledge per parameter, with knowledge retention dependent on training exposure.
Knowledge encoding, model compres- sion, retrieval-angmente models.
Allen Zhu & Li 22024
Table 12: Critical scaling laws for data mixing and knowledge capacity.
adaptation and long-term model updates (Que et al. 2024 Additionally, Chinchilla scaling assumptions were challenged by evidence showing that training models for more epochs on limited data can outperform simply increasing model size (Muennighoff et al. 2023 Repeated data exposure remains stable up to 4 epochs, but returns diminish to zero after around 16 epochs, making longer training a more effective allocation of compute resources. Furthermore, the vocabulary scaling law suggested that as language models grow larger, their optimal vocabulary size should increase according to a power law relationship (Tao et al. 2024). Finally, knowledge capacity scaling laws established that language models store bits of knowledge per parameter, meaning 7B model can encode 14B bits of knowledge surpassing English Wikipedia and textbooks combined (Allen-Zhu & Li 2024). Table 12 summarizes the data scaling laws for developing neural models when data is not available in abundance.
A.7 Scaling laws for reinforcement learning
Scaling laws in reinforcement learning (RL) and reward model optimization reveal both similarities and differ- ences with generative modeling. Single-agent RL follows power-law scaling with model size and environment interactions, with optimal scaling exponents between 0.4-0.8 across tasks lower than the 0.5 exponent ob- served in language models (Hilton et al.) 2023). RL tasks require orders of magnitude smaller models than generative tasks, correlating with task horizon length, which dictates environment interaction scaling. Task difficulty increases compute needs but does not affect scaling exponents, highlighting horizon length as key factor in RL scaling efficiency.
In board games like Hex which involves multi-agent RL, Jones 2021 showed that AlphaZero performance follows predictable scaling trends, with compute requirements increasing 7x per board size increment for perfect play and 4x for surpassing random play (Jones) 2021h. Neumann & Gros (2023) exlended this si study to Pentago and ConnectFour, proposing scaling laws which show that player strength scales with network size as ON 0.88, performance with compute as ac Rs 0.55, and optimal network size with compute budget as Copt 0.63 (Neumann & Gros 2023 Larger multi-agent models exhibit higher sample efficiency, though these trends may not generalize to highly complex games like Chess and Go.
Reward model overoptimization in RLHF follows distinct functional forms: Best-of-n (BoN) reward opti- mization is governed by d(Oronn - d)ond), whereas RL reward optimization follows d(aRL 6RL log d), where d represents KT. divergence from the initial policy (Gao et al., 2022). RL requires higher KT. divergence than BoN for optimization, and reward model overoptimization scales logarithmically with model size, while policy size has minimal impact. These findings reinforce the importance of balancing compute allocation, environment complexity, and optimization techniques achieve scalable and efficient RL models.
A.8 Scaling laws for sparse autoencoders
Recent rescarch has established scaling laws for dictionary learning, providing insights into how latent repre- sentations and sparsity impact reconstruction error and computational efficiency. Sparse autoencoders with
27

---

Published in Transactions on Machine Learning Research (11/2025)
Colegory Fre Traning
Tonk Comrration
Architccture
Duimets
Model
Duta
Konlm
2001
Decose-enly Tranofeuur
WhlTeat
USE 1B
2257 2001
Pre- Truming
Langrage Coneration
Docobm-coly Irarsfieur
MusewTees,Gniiee C4
2104 1tB
aB- 100B
my
7007
Pre Trahning
langiage Gesreation
Satth, T5 Erode Deoder.. Funed, Met Mi.--mbere GLUU Leatr, Esdkel. Dous, Fa-
Prttaaring U1, Fire/Tuuing GLUR SigertiLU., SQ:AD
73M 3001
Thander leurning
Reaser Urivooall ALBERT
X00
Pre/Tmisitg Pre- Traning
Longing Gescmolion Domatree Insus Becogaitar and Laaqure Cencation
Doadkesolly Trandornce
Leege uixture
403 88
VIT Tranffemus, LSTM
dierratrram Badian, Caluchit), CIFAR-IOR Language Bag
Vieaa prvemand: IFT-3H0L
Bead.
leorning
Chrl
Doosve-oonl Transferuer
Wekllett Conmendraadd Finfish Githnb repir
Witpedis, Bankee Predusin JFT. IungeSa22.
Thusfa Leuning Thursfar
Reoggailou
MLP-Saaes, CarNets OnteNets
1054- 108
Recogillix
Sarceee Data
2011
Thurfer Lessing
Yachina Truudation and Lea-
Dooodm-coll Trarffeemr
WXITL Exdish Gemon (Ba-De) and WXTT Eagiis Clanse (Eo-B)). CSN/ Daals XL- SUM
1B 11B
A4B 255BB
Gezeration
Khem al
Tranafer learting
Langtage Gonrrnatin
Deoobe-onlly Transfeuure
Poe Trate: RedPagama v1. Validatton:
CBM 3B
ArXes Wilepedia CA. ReilPaumaa wal-
dalion sets. Pro.tiie
es
Tharsfy learring
largrage Geveeation
Docoive-caly Thanddetrer,
Pie Tume WMTT RrglishChhnne (Fi-Zh),, Gigg FLAN
100M
Baoode-Dreooder Trandocuur,
Mudtiingul Muß
Quallacika lufe--
Langang Geastolion
Docods-ooly Transfoenct
The Fll,, Lautooi, FIQA, HelleSnng, Wankonuusl
Iife--
Langrage Geseration
Deosbe-only Transfemur
WhrTbar,, SimiPaaaaa SDALI,
5000
Kumar
100
Qunttorkon Pre-
Langrage Genvestion
Denivescaly Traroherur
Dohon VL.7
30M 2M
18- 2601
Trainin, Quantto
lin laferene
Cley cl
Intonxs
Langang: Gcastolion
Doonks-ooly Transooem
MMLU Plesis, Trathfoga, GPQA Avcillx
Infenno
Langrage Gonornation
Desoiee-olly Transfeuure
MATH
Inferues
Language Gearmition
Desobe-coly Transfeeuur
GSMSSE, MATH. MrsP2F-MATH, CodeCantonts,
100 708
SWE-bort lice
Min
MP1
Indereeo
Langrage Geueatton
Decoiee-coly Trandfruurr Decose-cnlly Tranaforuur
MATEISOO GSMticc
110M 341
Inderroo
Langrage Gemeeation
Jemardy MMIE, BIG hrrch, Wielaats, ABC, CUPA, PIQA, Openilook Q3, AGI Eval, CS3B.
1900--68
1.22
Sparityy Spazitt
Languag Gesrmiiio
Dowstesooll Mol V-T
FFT-BB
Resocaitin
Spardty
Langtage geoveation
Denive-coll Trarofeeurr, Mol. C1
1200 309
1d 1500
Spanaty
Langrage geseration
Dacoive-coly Tearofienur, MaE.
Slmn Paguess
1003 133M
JB 2003
Specity
Long snge Geacmolion
Doondr-ooly Tranfoeuce
SlinPaaans
2003 5B
0.50
Ditilation
langiege geseration
Thalhee Stuleer Decodeoonly Trataforner
CI
30031 1391
2001
MC0
1rB3-0 21
Multinodlity
Gasmiise Eange: Modslia,
Dosuks-caly Teauoferas
PCCUIb und varionss andial dattocts
0.136-1008
100ME
Yaks: Modelas, Langune Caa- eration
Patt
Meitmnddliee Multinodllly
laage Beragattinn
ImanyNet221K
638 2B
1M
harge Reoogiltion
Mochise
Mieess Escodi- Deecderonlly
JFT-33M,, InsegeNee, Binds200, CIEARIOS, Oak Berch
10M-38
30M-494M
Thesketon
LSTM doodks
Multinnddlii
Mallinndal Trals
Dosols-onl Transoomats
OPT. Crard, LileiSproch, ConunaMiees VaaPigui Spdile Polessi, laCude, SMELES from Zucand Peoplins Speecs CCUM LATON-uoo moD
834 308
GB- 2008
Mattinedalte Muttiagree Rt. Maltiagsee ItI.
Yaltimoial taaks Hee
VIT Decodesoaly Tomnieur Alghzeen with nesral retworks Alphaeen with netral retxorbe
TB 15B
IM JMM
Bow
Perbagn, ComnettRvr
MGA
RL.
Rovard Model iiising wilh Bosi a o Rl.
Dosoks-olly Tranfooaa
Tubsa
a221
Supde-aper
Pruefes. Beachmark, Iv)
CanoNeta, LSTM
of Dota2; MNIST langerage Geamiiiou
Misture
Decube-cnly Teauffrsac
RodPajan
P008 41001
Langrage
Deonive-cnly
Pie
Langtage
Tranffenre
BodTajana
Easoks-oll Taandouuss
Que
2004
Data Misture
Geneention, Coatinal
Decdeesolly Tranfocnce
verkon misture Chdl,, Methi, Chemisttyy Mi-
0 0,524 43
0:18 2611
Prodtabiingg
Medicn.
Ti c
201
Yocabnlary
Langinge Geantution
Dasols-coly Teamffeenr Dasods-coly Teausfoenc
SlmnPajam
SDI 5G
0- 500B
Spezie soder
Autoon-
Trining, Autonacaliz
M00
Spazi coder
Arncen-
Find Intreznnable Lattens
Decobe-cnly Transfeeuse
Slao et al
Hetrioaa
Larguage Gescration
Doodenoolly Tranefenuc
langunge modelligg RedPa'emaa, SZORR) Doons stram Trivista Nq. MMBI), MecQA
Pro- Pro-Tmising
Langeage Gearmution
Docude-only tnsadumuss
1004-
5000
Kunelodge Capse-
langang Geacuotion
Dosole-only Lenadomes
Supersted
Gruph Clasehicatua
Ink:Grash, GesplCt JO4O tolde- oehemeelleooooeeedddde GmybMAA
kamiig
Meduin
HD:
Cálirie:
Iunge Becogaitoo
OutyNeta VIT
SVHK, CIPAR-1O, and Inngoee
Tsreetical
Taroetical
Pireetical
Freeettral
Tacselical
Dowssollnng
Table 13: Details on task, architecture of models and training setup for each paper surveyed.
28

---

Published in Transactions on Machine Learning Research (11/2025)
Paper
Degendent variable
Scaling variable
Functional Jorm
Kaplan 2020
Triining
Parametens, Compate,
LJN,D (%))00
Data, Training Steps
Holfmaan et
Trinnin
Parameters, Data
LIN.D: N/ +
Eay ML
Performance meltic
Compule
P
Troining
Parameters, Data
LIP.D) X E0
Cataalero el sl.
2023
Performance meltiic
Model Paramelens, Compute,
== (he 1=:"-" 1+
Data, Input Size, Training
Steps
Hermandes
al
Data Transfrred
Morill Parameters, Pine-
D,P,, N) A(()))(((
tuning
Miraur ot
2021
Downstream Brmee
Upstresm Frroe
Keurs + ons aD ,) A D7
Downstrean Error
Pro-training
Zhang
2121
Dowastreon Loss
Fine- Lane-tanngg--... Data,
Data. PET
Model Paramelers, paraoneer
Cleu el
202lc
Downstreeu perfor- Downstreeu Lose Accurnccy
Pre- Fre-Iraiiing Loss, Compute
400=(19) P(C) w1 w1-L
Lin el al.
20244b
Data, Fine-tuning Dala
LD)= D. +E
Zettlo
Total Model Bits After Quan-
Invoeer 2iy23
Miattion
3/24
Total perameters
Quantization Ratio
Ketimnz of ml.
2124
Loss
Data, Model Parameters, LNN,D, P. Pa. Pai Pao.
AND BD E E
Training Prrciston, Prot- train Precision
Chen
2024a
Optimal LLM Calls
Fraction Of Easy And Difii cult Queries Number Of Sumples
Beown
2024
Coverage
log(C)) = ak
Wru el a
Optinal Compute
Model Patunelers Model Parameters, Data Model Parunetess Number Of Experts, Data.
loKa((C) 1 (N) (N)+)))000 2.03
Surdans
2024
Pre Pre-Trainng Loss
E/N, D) d + +E
Caakk el al
Lose
log(LiN E3) = log blog clug Eog
Fountar et
2023
Loes
Sparsiyy; Model. Parameters, Data Gramaaaity, Model Parume- ters, Data
L= 8/ + ++t) (48)"
Krajowski
2024
Loes
CNN +
Yum
2024
Loss
Model Paramesers, Number Of Experts, Data
D. log
- for F) Nog
Chen
2124b
Post-Trainie Training Loes
Uncompressed Model Loss, promed ratio, Model param- before pruning, Post-
D.No, to (+) (*# #
training Data Model Parameters, Compute,, Data
Henighan
2120
Luse
((x) A
Zioni
2022
Dowusttrem Ervoe
Compute
= 1o = 332"
Ainddalanoaaa
al
Loss
Compute, Model Panumeters,
2022
Data
Aghajasyan
Al.
Loes
Model Parameteers, Data
D,N, (N..11N
20125
2024a
Loes
Model Parasneters, Data
Jomes
Compute, Bourd Size
Elo (mesnntioo bbarddiize + guhtemn) clampimgdttne boardsize Melnn lng + cantio,,)
Neamman
&
Gros
Gamee Soore
Model Paruueters, Compuiee
2023
Gaso
2022
Gold Reward model
Root OC KL Between Initial Policy And Optimized Policy (d)
Rid) Mi Blogd)
scoess
Hilton et al.
2025
Intrinsic performance
Mode] Parameters, Euvicon- Inleractious
r- (7)) a +(
Ye nl.
2124
Loss domaii
Proportion Of Training
Infr...))) -4 b, exp (MM (65))
mains
2024
Validation
Pazameters, Data,
D(N. E A Rur
Mixture Ratão
20/44
Unigramt Normalae- Joss
Non-vocabulary Parameter,, ===
Vocabulary Data
Pazamcetets.
Lindsey et al. 20124
Reconstruesin ervor
Compute, Number Of La-
tonts
Goo at al. 2024
Reconstruesion loss
Number Of Latents, Sparsity k/n,k) = expla + Su log(()) Be log(e) (og(A) log(n)) exple + log(k))
Level
Stato os al.
2024
Downstream Accuracy
Datastore Model Parame-
Data, Compute
Mosenighooff
al.
Loss
Aor Br Data. Model Panameters, LAN.D)
2013
Epochs
1/1 -aili
Buebviddge et
2026
Students
Teacher Lose, Student Pa- De lx
zuneters, Dissillation Tokens
Table 14: Scaling law forms proposed in different papers we surveyed.
29

---

Published in Transactions on Machine Learning Research (11/2025)
Paper
Training code
Analysis code
Github link
Kaplan et al.
2020
Hofmmann et al.
2022
Hottmann et 2022
N
N
et al.
2024
Link Link
Caballero et al.
2023
N
Y
Hernandez al.
N
Abnar et al.
202
N
N
Mikami a.
202
N N N N N N N Y N Y Y N N N Y
Link
Zhang et al.
2024
N
Chen al. 2024c
Lin et al.
2024b
Y
Link
Dettmers
Zettlemoyer
2023
Cao et al.
2024
N
Kumar
22024
Chen et al.
2024a
Y
Link
Snell et 2024
Brown et a. 2024
Link
Wu et a.. 2024
Link
Sardana et a.. 2024
Clark et 2222
Link
Frantar et a.. 2023
Krajewski a..
2024
Link
Yun et al. 22024
Chen al. 2024b
N
Henighan of al.
2020
Zhai el al.
2022
Y
Link
Alabdulnohsiin el
2022
Link
Aghajanyan et al.
2023
N
al. 2024a
Jones 2021
Y
Y
Link
Neumann Gros
2023
Link
Gao et al.
2022
N
Hilton
2023
Ye et al. 2024
Y
Y
Link
Liu et
202
Y
Y
Link
Kang et al.
2024
Y
Link
Que et 2024
N
Tao et al.
2024
Y
Y
Link
Lindsev
22044
N
Gao el al.
2024
Y
Y
Link
Shao et
1202
Y
Y
Link
Muennighoff et a..
2023
Y
Y
Link
Allen- Zhu & Li
2024
N
N
Ma et al. 2022
Link
Sorscher
al. 2023
N
Y
Link
Table 15: Reproducibility of different neural scaling law papers. Reproducibility status of 45 papers surveyed:
22 (48.9%) provided repositories; 29 (64.4%) did not share training code.
30

---

Published in Transactions on Machine Learning Research (11/2025)
top-K selection follow power-law scaling for reconstruction error (MSE) in terms of the mumber of latents n and sparsity k, though this relationship only holds for small & relative to model dimension (Gao et al. 2024) Larger language models require more latents to maintain the same MSE at a fixed sparsity, reinforcing that latent dimensionality must scale with model size for effective reconstruction. Additionally, MSE follows a power-law relationship with the compule used during training, suggesting that efficient scaling stralegies must balance sparsity, latent size, and training compute to minimize error effectively. This reinforced by Lindsey et al. (2024), showing that feature representations follow predictable scaling trends, where larger models develop richer, more interpretable dictionaries as the number of learned features increases.
A.9 Scaling laws for graph neural networks
Unlike in computer vision and natural language processing, where larger datasets typically improve gener- alization, graph self-supervised learning methods fail to exhibit expected scaling behavior and performance fluctuates unpredictably across different data scales (Ma et al.] 2024). However, self-supervised learning pretraining loss does scale with more training data, but this improvement does not translate to better down- stream performance. The scaling behavior is method-specifee with some approaches like InfoGraph showing more stable scaling than others like GraphCL
A.10 Scaling laws in robotics
Recent evidence suggests that robotic learning exhibits scalable regularities akin language and vision, but with embodiment-specfffe twists. A large meta-analysis of Robot Foundation Models (RFMs) and LLM- based robotics systems reports consistent. power-law improvements as model size, data, and compute scale, with scaling exponents for RFMs comparable to those in vision and in some cases exceoding those obscrved for language-only Lasks; the study further emphasizes the role of lask diversity and multimodality in realizing these gains (Sartor& & Thompson 2024). Complementing this global view, an extensive empirical investigation of imitation learning for manipulation establishes that generalization success follows an approximate power law in the diversity of training environments and object categories, while exhibiting diminishing returns in the number of demonstrations per environment once modest threshold is reached; prioritizing coverage over repetition yields substantially better zero-shot transfer to unseen scenes and objects (Lin et al. 2024a). Together, these findings point to predictive, albcit still empirically grounded laws for robotics: performance scales with broaden along the axes most relevant to embodiment (scene, object, and interaction diversity). Unlike static- language settings, robotics introduces additional constraints that shape scaling behavior: data collection is expensive and safety-criticall rewards can be sparse, sim-to-real gaps complicate extrapolation, and hardware throughput/latee /latencyy bound feasible training and deployment. Consequently, practical "robotics scaling laws" should be framed over (i) model and dataset scale, (ii) diversity rather than counts alone, and (iii) system constraints (embodiment, safety, and real-time compute), yielding design guidance that balances accuracy, robustness, and cost at scale.
/daaa/compuuue in a manner that. is measurable and forecastable, provided that dalasets
A.11 Scaling laws for diffusion-based models
Recent studies indicate that diffusson models exhibit recognizable scaling regularities, while also introducing domain-specific nuances distinct from language and standard vision settings. For text-to-image Diffusion Transformers (DiT), pre-training loss follows a power-law in compute across wide budgets, enabling fore- casts of optimal model size and dataset size under fixed compute and showing that loss trends correlate with downstream generative quality (e.g., FID) (Liang et al. 2024). In video diffusion, analogous com- pute performance regularities hold only when scale is modeled jointly with optimization hyperparameters: learning rate and batch size exert outsized influence, and an extended law that predicts their optima as func- tions of aadael/dda//cmmppp
yields tighter fits and cheaper operating points (e.g., comparable performance at markedly lower inference cost) (Yin et al., 2025). Uniquely for diffusion, a second axis of inference- time scaling is operative: boyond the diminishing returns of simply increasing denoising steps, search and verification-baste sampling converts additional test-time compute into tangible quality gains, effectively Together, these results suggest formulating
extending the scaling frontier post-training (Ma et al. 2025
31

---

Published in Transactions on Machine Learning Research (11/2025)
"diffusion scaling laws" over training compute, model/data scale, hyperparameter optima, and inference bud- gel, rather than training alone to obtain predictive, design-useful guidance for both training-timn allocation and test-time compute quality trade-offs.
B Reproducibility of scaling laws papers
The reproducibility status of neural scaling law papers presents mixed landscape in terms of rescarch transparency. We consolidate and provide the links to github code repasitories in the Table 15 Among the surveyed papers proposing scaling laws, 22 papers (48.9%) provided repository links, indicating some level of commitment to open science practices. However, more than half of the papers still lack basic reproducibility elements, with 29 papers (64.4%) not sharing training code and 27 papers (60%) withholding analysis code. This comprehensive survey suggests that while there is growing trend toward reproducibility in neural scaling law research, there remains substantial room for improvement in establishing standard practices for code sharing and result verification.
32


Reviews:
[{"review_id": "OmZSfM0PLu", "reviewer": "Reviewer_WG6m", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is a survey on existing scaling law methods in various contexts such as language or vision, and various “attributes to scale” such as compute, model size, or data. The methods are organised into a clear taxonomy, and the paper provides recommendations on which method to use for which use case. Beyond simple technical descriptions, the paper also presents advantages and criticism of these methods, as they are presented in various contributions of the literature on scaling laws.", "claims_and_evidence_yn": "No", "claims_explanation": "Strengths:\n\n- The paper provides a good and exhaustive summary of recent papers on scaling laws. \n- The methods are well organised into a tree structure separating data and model scaling, the different phases of training: pre-training, post-training and inference and the different use cases.\n- Figure 4 provides guidelines to map specific use cases to specific scaling law methods.\n- The paper acknowledges and describes criticisms of scaling laws existing in the literature.\n\nWeaknesses:\n\n- The major weakness of this paper is the contribution that is very limited for TMLR standards. There are no novel findings, no experiments, no novel insights, everything is just taken from existing paper and the paper feels more like a blog post covering existing literature on scaling laws. Interesting contributions would have been to compare some of these scaling laws in practical setups and provide actual insights based on fair comparisons, comparing different scaling laws in similar setups. The paper also points to the lack of reproducibility in most publications, doing an empirical study would have been a good opportunity to verify the reproducibility of some of the existing approaches.\n\n- I share the doubts and criticism against scaling laws and the possibility to formalize them properly. I do think that a particular scaling law is too specific to a particular setup, which makes it unusable for other setups. In practice, we often have to make our own scaling laws based on intuition and there is no single formula that can really fit our problem. If you take the example of Chinchilla’s scaling laws, they correspond to a very particular mathematical formulation, but the community realised only a few months after they were published that they were wrong and that training smaller LLMs with x10 times more data than recommended was very effective. Nowadays no one uses Chinchilla’s scaling laws in practice and this paper still recommends using them.\n\n- Even without proposing experiments or actual new insights, the paper could be improved a lot, mainly by providing more details and explanations in some of the sections. In section 3, introducing better the concept of optimization for scaling laws, formalizing losses and objectives , providing more details on initialization and validation, with examples. In section 4 providing more intuition behind the presented formulas that all seem to come from nowhere.", "audience_yn": "Yes", "audience_explanation": "Some people might be interested in a survey pointing to recent papers in scaling laws. Some practitioners might be interested in the roadmap provided by Figure 4.", "requested_changes": "- Proposing concrete experiments to validate, reproduce and compare scaling laws\n- Providing way more details, intuitions, explanation for the concepts presented in the paper", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pTZZsva46L", "reviewer": "Reviewer_MDBh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper surveys the existing landscape of neural network scaling law research. Specifically, the paper focuses on the difference of scaling laws under different architectures, modalities, training scheme, and compression methods. The strength of the paper lies in the broad range of scaling laws surveyed. The paper clearly categorizes existing scaling law research into different research questions and summarizes the conclusion from these papers.\n\nOn the weakness side, the paper does a poor job in providing useful insight. The paper overall plainly listed the scaling law equations from different work, without providing further analysis or verifications. Specifically, when different work show different scaling trends, such as the difference between Equation 6 and 7 under RQ3 or Equation 8-12 under RQ5, the paper fails to provide further analysis on how to understand such difference and whether all these scaling laws are correct or each of them only work under a certain condition. Without these anlaysis, the contribution of this survey is minimal. Furthermore, the paper only listed existing scaling laws, without proposing new scenarios that the current sclaing law may not work and motivate future research directions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "As a survey paper, this paper listed the observations from existing work and provide no new claims.", "audience_yn": "Yes", "audience_explanation": "Scaling law is a hot topic for large models, which is within the interests of TMLR audiances.", "requested_changes": "As mentioned in the summary, the paper should add more analysis on the similarity and difference between the scaling law formulations mentioned under the same research questions. Adding a few figures with the trend line of each sclaing law and real model performance will better help understanding when each scaling law is suitable. Furthermore, besides existing work, proposing new directions where the current scaling law does not work well would be a great contribution.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mynAQByIKQ", "reviewer": "Reviewer_gvz6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is a survey on the topic of scaling law. The paper proposes a taxonomy of the topic: first into data scaling and model scaling, and then further divides model scaling into pre-trianing, post-training and inference. The paper then summarizes the research questions into several categories and discusses the papers based on such categorization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper is a survey and mostly summarizes results from previous papers.", "audience_yn": "Yes", "audience_explanation": "The paper is a survey on the topic of scaling law, which is a fundamental observation in modern model design.", "requested_changes": "1. It seems to me unnecessary to mention the number of papers summarized. Table 2 is quite informative, and maybe the paper/references could also be added to the taxonomy figure (Figure 2).\n\n2. The survey does not provide any empirical figures/statistics on the scaling law from previous papers. I think this is crucial to the survey. On the one hand, we need to show the evidence (the model, the data, and the setting) that is pro-scaling-law, and on the other hand, we need to show how the scaling law could be broken under other scenarios (e.g., the MOE settings).\n\n3. I think two more fields are worth mentioning in the survey. One is about the scaling law for diffusion models, and the other is the scaling law in the field of robotics (imitation learning and etc).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)