Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
Authors: Song Bian
Tao Yu
Shivaram Venkataraman
Youngsuk Park
Venue: ICLR 2026 Poster (2026)
Keywords: Scaling Laws
Model Architecture
Inference-Efficient
Primary area: foundation or frontier models, including LLMs

Abstract: Scaling the number of parameters and the size of training data has proven to be an effective strategy for improving large language model (LLM) performance. Yet, as these models grow increasingly powerful and widely deployed, the cost of inference has become a pressing concern. Despite its importance, the trade-off between model accuracy and inference efficiency remains underexplored. In this work, we examine how key architectural factors, hidden size, the allocation of parameters between MLP and attention (mlp-to-attention ratio), and grouped-query attention (GQA), influence both inference cost and accuracy. We introduce a conditional scaling law that augments the Chinchilla framework with architectural information, along with a search framework for identifying architectures that are simultaneously inference-efficient and accurate. To validate our approach, we train more than 200 models spanning 80M to 3B parameters and 8B to 100B training tokens, and fit the proposed conditional scaling law. Our results show that the conditional scaling law reliably predicts optimal architectural choices and that the resulting models outperform existing open-source baselines. Under the same training budget, optimized architectures achieve up to 2.1\% higher accuracy and 42\% greater inference throughput compared to LLaMA-3.2.

Published as a conference paper at ICLR 2026
SCALING LAwS MEET MODEL ARCHITECTURE: To- WARD OPFRENCE-EFFICIENT LLMs
Song Bian" UW-Madison
Tao Yu
Shivaram Venkataraman UW-Madison
Youngsuk Park Amazon Web Services
Amazon Web Services
ABSTRACT
Scaling the number of parameters and the size of training data has proven to be an effective strategy for improving large language model (LLM) performance. Yet, these models grow increasingly powerful and widely deployed, the cost inference has become a pressing concern. Despite its importance, the trade- off between model accuracy and inference efficiency remains underexplored. In this work, we examine how key architectural factors, hidden size, the allocation of parameters between ML. and attention (mlp-to-attention ratio), and grouped- query attention (GQA), influence both inference cost and accuracy. We introduce a conditional scaling law that augments the Chinchilla framework with architectural information, along with search framework for identifying architectures that are simultaneously inference-efficient and accurate. To validate our approach, we train more than 200 models spanning 80M to 3B parameters and 8B to 100B training tokens, and fit the proposed conditional scaling law. Our results show that the conditional scaling law reliably predicts optimal architectural choices and that the resulting models outperform existing open-source baselines. Under the same training budget, optimized architectures achieve up to 2. 1% higher accuracy and 42% greater inference throughput compared to LLaMA-3.2.
Throughput (tokens/s)
Train Loss
3000
3250
2.736
6
3280
6
2.721
mip-to-attes attentionn ratio
5
2960
mlp-to-attent tip-to-attention ratio
5
2.655
2.706
250
2640
4
4
2320
2.676
2.661
2250
2.625
1680
2.646
2500
1360
2.631
2000
1040
2.616
3000 3500 4000 4500 5000 5500
6000
3000
3500
4000 4500 5000 5500 6000
modda
dmadel
Figure/ Scaling sweep results. (left) Inference throughput (tokens/s) and (right) Scaling-law. predicted training loss contours over hidden size dmodel and mip-to-attention ratio. Our conditional scaling law enables concurrent gains throughput and reductions in predicted training loss under fixed parameter budget. Dotted points indicate the architectures used to fit the scaling law.
"Work done during internship at Amazon Web Services. 1 "Correppondence Tao Yu (tacuu zon. com).

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Scaling law studies Kaplan et al. 2020); [Hffmann al.
(2022) Muennighof el. (2023): Krajew-
(kk et al. (0024); Abnar et al. (2025) have shown that increasing model parameters, training tokens, dataset quality, and compute budget consistently reduces pre-training loss, improves downstream task performance Hendrycks et al. (021); Austin et al. (02T), and enables the emergence of novel capabilities Wei et al. (2022). These insights have driven the development of many state-of-the-ar large language models Touvron et al. (2023) Yang et al. (025); Guo et al. (2025).
However, as the field advances, it has become increasingly clear that focusing exclusively on train- ing overlooks the practical challenges of deploying these models at scale Chien et al. (2023); ; Wu
al. (2024); Muhamed et al.] (2023). major limitation of existing scaling laws is their omission of inference costs, which constitute the dominant expense in deploying large models in real-world applications Sardann et al. (2023); Park et al. (2024). Moreover, the growing use of LLMs in reason- ing systems highlights the need for scaling laws that account for inference costs Snell et al.. (2024); Brown et al.] (2024); Luo et al. (2024); Qi et al. (2024); Guan et al. (2025). Therefore, we ask the following question:
Can we explicitly capture the trade-off between inference efficiency and accuracy of large language models?
To address this question, a recent study Sardana et al. (2023) proposed scaling laws that incorporate the total FLOPs from both training and inference. However, their formulation requires estimating the total number of tokens generated over a model's entire lifespan. Because inference is performed repeatedly during deployment, this assumption renders the proposed scaling law impractical for real-world use. Another study Bian et al. (0255) extends Chinchilla scaling laws by incorporating model architecture. However, this work has notable limitations. First, the study considers only the aspect ratio, defined as hidden size over number of layers, as the architectural factor. Yet, as shown in Figure aspect ratio alone fails capture the full range of factors that influence inference efficiency in large language models. Second, the depth of the model strongly influences accuracy: cutting layers tends to impair the model's generalization after tnnettuning Petty al. (2023). Finally, the study lacks general framework for incorporating broader architectural factors, including hidden size and GQA, into scaling laws.
In this work, we fix the number of layers and study the effect of other architectural factors, including GQA, hidden size, and the mip-to-attention ratio. This de- sign choice is motivated by recent open-weight mod- els such as LLaMA Touvron et al. (2023). Qwen Yang let al. (2025), Gemma Team et al. (2024a), and Phi din et al. (2024). which, despite having comparable number of parameters, adopt markedly different archi- tectural designs.
Throughput (tokens/s)
6000
Qwen 5--1100
Owen 2 2.5-1.5--L.. LB05
Owwnn 3-0 68-A--A10
4000
2000
0
2
2
2
2
Our primary goal is to investigate how model architec- ture influences both inference efficiency and model ac- curacy. We begin by comparing the inference efficiency of models with identical parameter counts but varying architectures. Next, we train over 200 models, ranging from 80M 297M parameters on up to 30B tokens, to systematically characterize the relationship between architectural design and accuracy. Guided by these em- pirical findings, we introduce conditional extension of the Chinchilla scaling laws that incorporates architec- tural parameters, establishing general framework for identifying model architectures that balance inference efficiency and performance.
Batch Size
Figure 2: Although larger models gener- ally achieve lower inference throughput than smaller ones, Qwen2.5-1.5B outper- forms Qwen3-0.6B. Despite having the same number of layers, Qwen2.5-1.5p benefits from higher hidden size, GQA, and mlp-to- -oap--ttttttnioon ratio.
Finally, validate this framework by fitting the proposed scaling law models between 80M and 297M parameters, and evaluating its predictions when scaling up to pretrain 3B 3B-parameter models. Our results demonstrate that, under identical training setups, the derived optimal 3B- parameter
2

---

Published as a conference paper at ICLR 2026
chitecture achieves up to 42% higher inference throughput than the LLaMA-322-3B architecture, while maintaining better accuracy.
2 BACKGROUND
Accurately predicting the performance of large language models during scaling is essential. This enables to answer key questions: (i) what is the optimal allocation of available resources between model size and training tokens, and (ii) what performance gains can be expected from additional resources? Fortunately, the model loss has been observed to follow power-law relationship with respect to the number of parameters N and training tokens D Hooffmann et al. (0022); Muennighoff al. (2023) with:
A
B
=N+
No
D3
where L. is the model loss, N is the number of total parameters and D is the number of tokens used for training and A, B, E, a, 3 are parameters to be learned.
To fit the learnable parameters in Eq. [ Chinchilla Hoffmann et al. (2022) employs two strategies: (i) training models with fixed number of parameters while varying the number of training tokens, and (ii) training models under fixed compute budge varying both parameters and tokens. The resulting data are combined to fit the learned parameters in Eq. ( With the fitted scaling laws, Chinchilla addresses the following question to determine optimal allocation:
arg min L(N, D) s.t. FLOPs(N, D) C N,D
(2)
where denotes the resource constraint, N the total number of parameters, and D the number of training tokens.
In this paper, we do not address how to optimally allocate compute between model size and training data under a fixed compute budget. Instead, our focus is on identifying model architectures that optimize inference efficiency and accuracy under fixed parameter and token budgets. For example, given model with 7B parameters trained on 14T tokens, we study how design architecture that satisfies both efficiency and accuracy requirements.
MODEL ARCHITECTURE-AWARE SCALING LAWS
3.1 MODEL ARCHITECTURE VARIATIONS
The architecture of decoder-only transformer is composed of a sequence of stacked decoder blocks, each sharing the same structure to facilitate model-parallel deployment across devices. Under this design, the overall architecture of dense LLMs is primarily determined by the hidden size and the MLP intermediate size, which together specify the attention and MLP layers structure. This work studies the optimal model architecture given fixed total number of non-embedding parameters N Nnon-embed (at different levels). Although the number of layers Flaayer also plays critical role (closely related aspect ratio (Petty et al., 2023)). varying Naaee under a fixed Naun embed substantially im- pacts both inference cost and accuracy (Tay et al. 2021 Alabdul et al., 2023). Therefore, we fix naayer and focus on the effects of hidden size dmadel and the mlp-to- mip-to-attention ratio T'lp/attn on in- ference efficiency ($3.2) and accuracy ( (33.3), noting that Mayer still varies across different N Nnon-embod levels. $3.3) we introduce a conditional scaling law to predict the performance of architectural variants, and in $3.4. present lightweight framework for identifying architectures that optimally balance inference efficiency and accuracy.
Note that the number of attention parameters is primarily determined by the hidden size dmodel and the attention projection dimension, since most open-weight models adopt non-square q, k v
projection matrices, as seen in Gemma (Team al.) 2024a) and Qwen3 (Yang al. 2025 For
consistency, we fix the pea-head dimension dhend to 64 for models with N Nnon-embod S and to 128
'The compute cost is approximated as FLOPs(N, D) ~ 6ND D in Hoffimann et al. 2022 Muennighofi al. (2023), where N denotes the number of parameters and D the number of training tokens. In this work, adopt the same settings prior studies.
3

---

Published as a conference paper at ICLR 2026
Throughput (tokens/s)
3000
Throughput (tokens/s)
1200
d=2048
0.90
r=2.40
1000
d 4009
r-1330
4.200
2000
d=8192
800
1500
600
1000
400
500
200
0
29
2
2²
24
2
2
2
Batch Size
Batch Size
Figure 3: Inference throughput. (left) hidden size d dmnddll and (right) mlp-to-attention ratio r /alp/attn on the 8B model. Under fixed parameter budget Non larger hidden sizes and higher mlp-to-attention ratios improve inference throughput for varying batch sizes.
for models with N non-embed 2 3B. Consequently, maintain constant "mip//utn we adjust the number of attention heads Rhead rather than altering the projection dimension directly. This design choice also provides flexibility to incorporate architectural variants such as grouped -uery attention.
3.2 INFERENCE EFFICIENCY
Inspired by the success and widespread adoption of open-weight dense models such as Qwen3 (Yang al., 2025), LLaMA-3.2 (Dubey et al., 2024). and the Gemma-2 (Team al.. 2024b) family, we construct architectural variants by modifying the configurations of the LLMMA-3.2 and Qwen3 dense models (Figure 2 14
Appendix F). In addition to hidden size and the mlp-to-attention ratio, we find that group-query attention has critical impact on inference efficiency, even though it only modestly reduces the number of attention parameters (by shrinking the key and value matrices). To disentangle these effects, we conduct controlled ablations of hidden size, ALP-to-attentin ratio, and GQA under the following setups:
hidden size dmodel: fix Noon-embed- 7 mmmpp/ atm and GQA- - 4, vary dmodel and number of attention heads Thead (Figure leftt.
mntp-to-attention ratio Tmip/ami fix N Nnon-embed. dmodel and GQA= vary Thead and interme- diate size (Figure (FigureJright))
GQA: fix Nuon embed, dnuodel and Tmlp/att- vary Thead and number of key-value heads (Ap-
pendix FF)
Figure 3 shows the ablation of varying hidden sizes dmodel and mip-to-attention Tmip/aun the LLaMA-3.1-8B model variants. We observe that larger hidden size (or fewer attention heads) and higher mlp-to-attention ratios improve inference throughput. Similar trends are observed in the LLaMA-3.2-IB and 3B model variants (Appendix (F). These gains arise in part because larger dmuseel and higher mmpp/att reduce the total FLOPs, as detailed in the inference FLOPs analysis (Ap- pendix KK. In addition, these architectural choices shrink the KV cache, lowering I/O cost during inference and further improving throughput Adnan et al. (2024). Figure TT in Appendix [F] presents the GQA ablation, confirming prior observations Ainslic et al. (2023) that increasing GQA consis- tently improves inference throughput. A comparable set ablation experiments on Qwen3 models, also reported in Appendix F. further corroborates these findings.
3.3 CONDITIONAL SCALING LAW
Improving inference efficiency should not come at the expense of significantly reducing model ac- curacy, making it crucial to understand how architectural choices affect accuracy and training loss. Because training large-scale language models is prohibitively expensive, a common strategy is to study smaller models and use scaling laws to extrapolate insights to larger scales. for example, the Chinchilla scaling laws (Hoffmann et al. 2022). However, incorporating multiple architectural fac- tors into such laws remains challenging. To address this, examine the effect of architectural
4

---

Published as a conference paper at ICLR 2026
3.9
3.6
3.4
r=0,36
[=2.40 T-4,20
r=0.50
r=1.35
r=0.60
r=2.10
7-1.20
r-0.84
r-2.00
r-1.10
r-3.20
3.8
Loss
Loss
Loss
3.7-
3.6
3.4
3.2
0.0
0.1
D.2
0.3
0.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.20
drodee
VN
Gmodel /
V
dyoddll / VN
Figure 4: Loss vs. hidden size. (left) 80M model variants; (center) 145M model variants; (right) 297M model variants. Across model sizes, the relationship between training loss and modee / N ex- hibits a consistent U-shaped curve when architectural factors such as GQA and the ALL--to-attentoo ratio are held fixed. The legend denotes the MLP-to-attention ratio T'mlp/atu aatn for each model.
3.5
3.30
3.15
d=512
d=1024
d=512
d=1536
d-768
d-1536
d-768
d--1336
d 1024
d=3072
3.26
3.24
3,05
3.22
mlp-to-a toeettttttteionn ratio
mlp-to-attention ratio
mil-to-atteenten ratio
Figure 5: Loss vs. MLP-to-attention ratio. (left) 80M model variants; (center) 145M model variants; (right) 297M model variants. Across model sizes, the relationship between training loss and /mlp/atn exhibits consistent U-shaped curve when architectural factors such as GQA and hidden size are held fixed. The legend denotes the hidden size d = dmodel for each model.
choices on training loss L a conditional manner, varying one factor at a time while keeping the others fixed.
hidden size dmodel- We note that dmodel generally scales linearly with v non embed- Assuming squared attention weight matrices, the number of attention parameters Naun be expressed as
4d model
Natn
Nuon embed X
where r Tmiplamn is fixed, and the constant factor 4 arises from the query, key, value, and output projection layers in each attention block. To capture this scaling behavior, we normalize dmodel by v /Nnon-embed and examine its relation to loss in Figure The resulting U-shaped curves L(d/ VN r, N, D) exhibit nearly identical optima across different model sizes. Moreover, Figure 4 confirms that excessively large hidden sizes, which reduce the number of attention heads Theaa. can degrade accuracy-a phenomenon consistently observed in prior analyses of transformer capacity and head allocation Kaplan et al. 2020 Hoffmman et al. 2022).
mlp-to-attention ratio /mlp/attn Figure illustrates how the loss varies with Tmlp/attn condi- tioned on dmodel fixed different levels, where we consistently observe U-shaped curve L(r d/vN,N, D). While the attention mechanism is central to the success of transformers (Vaswani, 2017), recent open-weight models have allocated a progressively smaller fraction of parameters to attention as overall model size increases (e.g., I.LaMA and Qwen families). Our analysis indicates that this trend is not universally optimal: there exists interior optimum in the allocation of atten- tion parameters, and deviating from in either direction degrades model performance. This suggests that careful tuning of the -tttt-ttttnnion ratio is critical for scaling transformers effectively.
As shown in Figures and 5] both hidden size and the tL--to-attennton ratio exhibit U-shaped relationships with training loss. To capture these trends, fit the function Co + C₁ log + c2/w separately for x Tmppaam and d /Nno///omeeettt This formulation effectively models the U- shaped behavior while ensuring sublinear growth &. increases. However, incorporating Tmlp/attn
5

---

Published as a conference paper at ICLR 2026
dmodel. N, and D into unified, architecture-awar scaling law remains challenging. Since fitting single all- purpose scaling law L(d/vA NN D) is unrealistic across all possible configurations, we instead propose a two-step conditional approach:
1. For given N and D. obtain the optimal loss Lug (N, D) = min LIN, min (E +
(
No + D from the Chinchilla scaling law (Eq. I as reference point.
2. Calibrate the loss of architectural variants L(d/VN,r N, D) relative to this reference.
We focus on simple and transparent calibration schemes:
(multiplicative)
VN
(bo + bn log + b Tup (3)
I(d/VN,r D) = (ao + 01 log( + 02
d
(additive) L(d/VN,r D) (ag + d1 log(
+ (b, log + (/) + Tup
VN
02
Here, a, and b, are learnable parameters that are shared across all N, D. Note that both functional forms assume the effects of /mlp/att and dmodel on loss are separable.
3.4 SEARCHING FOR ENFERENC-EEFICCEER ACCURATE MODELS
With the conditional scaling law, we can identify architectures that are both inference-efficient and accurate by solving the following optimization problem: given N, D, and set of architectural choices P,
T(P D) Ta: where (P) denotes the inference efficiency of an architecture P with total N Non--mhed parameters, and Les (2 Lopt) is the maximum allowable training loss.
argmax p P (P);
(Appendix F). GQA has a substantial impact on inference efficiency; How- ever, unlike hidden size and the mip-to-attention ratio, GQA does not exhibit consistent continuous relationship with loss (Figure 24 Appendix[1 and is highly variable, making it challenging to iden- tify settings that achieve both accuracy and efficiency. Fortunately, the search space for GQA is relatively small once N Nnon-embed- dmodel and mlmp/att are fixed, since GQA must be prime factor of the number of attention heads Thead- practice, we perform a local GQA search by enumerating feasible values and applying early stopping once performance falls below that of the GQA= 4 base- line. Algorithm| ( summarizes our overall framework for identifying inference-efficient and accurate architectures.
As shown in Figure
Algorithm 1: Searching for Inference- Enference-Efficient Accurate Model
Input:
Model parameters N, training tokens D, target loss LG; inference efficiency IN(-); optional: the optimal loss (N,, D)
Train smaller models to fit the Chinchilla scaling laws (Eq. ! if Lopr D) is unavailable
Solve the constrained optimization (Eq. 4h for dmodel, l'mip/att /att and corresponding architecture P
Perform a local search over GQA values with early stopping to maximize inference efficiency return Final model architecture (P, GQA)
EXPERIMENT SETUP
We first detail the experimental setup of training, inference, and downstream task evaluation, and then describe how we derive the conditional scaling law and scale up to larger sizes.
Training Setup. We sample the training data from Dolma-v1. 1.7 Soldaini et al. 2024)). which contains data from 15 different sources. Tokens are sampled with probability proportional to each source's 's contribution, ensuring the sampled dataset preserves a similar distribution to Dolma- v1.7. We train decoder-only LLaMA-3.2 (Dubey et al. 2024) style transformers with N Nooo-embed in (8OM, 145M, 297M, 1B, 3B), for each N Nnon-ennee we obtain model architecture candidates by vary- ing hidden size d /N /Nnon non embed and mip-to-attention ratio Tmip/attn (changing intermediate size

---

Published as a conference paper at ICLR 2026
MSE: 0.0002, Spearman: 0.8909
MSE: 0,0001, Spearman: 0.7920
MSE: 0.0001, Spearman: 0.7451
Predicted Loss
Fit
Predicted Loss
Fit
Predicted Loss
fit
3.4
3.4
3.4
Eval
3.2
3.2
3.2
3.0
3.0
3.0
2.8
2.8
2.6
2.8
3.0
3.2
3.4
3.6
2.6
2.8
3.0
3.2
3.4
3.6
2.6
2.8
3.0
3.2
3.4
3.6
Actual Loss
Actual Loss
Actual Loss
Figure 6: Predictive performances of the fitted conditional scaling law (left) Task 1: Fit 80M, evaluate on 145M; (center) Task 2: Fit on 80, 145M, evaluate on 297M; (right) Task 80, 145, 297M, evaluate 1B. Orange dots denote fitting data points, and purple crosses indicate the test data points. We compare scaling-lav predicted loss with actual pretraining loss of architectures and observed a consistently low MSE and high Spearman correlation across model scales.
on
and number of attention heads Rhead) while holding other architectural factors fixed e.g. GQA- 4. A full list of over 200 model architectures used can be found in Appendix D All models are trained on 100Nnon-mmb tokens (5x Chinchilla optimal) to ensure convergence. We tuned training hyper- parameters (mainly following prior work Chen et al. (2025) ), with a full list in Appendix
Inference Setup. We evaluate the inference efficiency using the vLLM framework Kwon et al. (023). By default, inputs consist of 4096 tokens and outputs 1024 tokens. We report the av- eraged inference throughput (tokens/second) from 5 repeated runs. Unless otherwise specified, all experiments are conducted on NVIDIA Ampere A 1100 GPUs (40GB) with vL..M.
LLM Evaluation Setup. Following prior works Biderman et al. 2023 Zhang er al. (0024)
we evaluate pretrained models in the zero-shot setting using 1m- Am-evallatioon haronsssaaassss.eee.oo. on
nine benchmarks: ARC-Easy Clark et al. (2018), ARC-Challenge Clark et al. (2018). LAM-
(2016) HellaSwag Zellers et al. 2019), QA et (2018) PIQA Bisk et al. (2020). SciQ Welbl et al. 2017
BADA Paperno
WinoGrande Sakaguchi et al. (202 and
CoQA Reddy et al. (2019)
Fitting Scaling Laws. Following Gadre et al. (002)): Bian et al. 20025) ) we use the Levenberg- Marquardt algorithm to fit the conditional scaling laws 3). The Levenberg eevenberg-Marquardl algorithm does least-squares
fitting by estimating | the solution arg min Σx= [y: f(,,, 3)]
where (Ii; yi) are the observed data pairs. Note that instead of fitting the Chinchilla scaling law, we empirically searched over architecture variants to find the optimal loss Lopr(N, D) for N Nnon-embed < IBB scale.
We scale up the scale law fitting in the following progressive manner:
(Task 1) fit on the SOM results and evaluate on 145M results; (Task 2) fit on 80, 145M results and evaluate on 297M results;
(Task 3) fit on 80, 145, 297M results and evaluate on 1B results:
This ensures a robust and consistent way of scaling up the model sizes and evaluating our conditional scaling law. Following prior work Kumar et al. (2024), we evaluate the fitted scaling law with mean squared error (MSE) metric, defined as Ei= (4, I,)² where denotes the actual loss and the predicted loss. We additionally report the Spearman's rank correlation Ceeffiiient(Searmmn (1961) to compare predicted and actual rankings. Both metrics calculated on the val data points.
EXPERIMENT RESULTS
We begin by evaluating the predictive performances of the conditional scaling laws with multiplica- tive calibration. We then conduct ablation studies to assess the impact of data selection and to

---

Published as a conference paper at ICLR 2026
Table 1: Large-Scale Model Results. We evaluate the scaling laws 1B and 3B scales by training Panda-1B, 1B, Surefire- IB, and Panda-3B, and compare them with A..MA-..--1B and LLaMA-3.2 3B, respectively. The Avg. column reports the mean accuracy across the nine downstream tasks. Panda-IB and 3B are trained using the optimal architectural configurations predicted by our scaling laws, whereas Surefire- and 3B satisfy the loss constraint Eq. (4 and achieve Pareto optimality.
Models
dmodel
fsize
Thayers
GQA
dmodel// / /V N
r
Loss (4)
Avg. (t)
LLaMA-3.2-IB
2048
8192
16
0.066
4.80
2.803
54.9
Panda-1Bh
2560
4096
16
4
0.082
1.07
2.782
57.0
Surefire-1B
2560
6144
16
0.082
3.6
2.804
55.4
LLaMA-332-3B
3072
8192
28
3
0.058
4.80
2.625
61.9
Panda-3B
4096
4096
28
0.077
2.619
62.5
Surefire-3B
4096
4096
28
7
0.077
1
2.620
62.6
2.83
Throughput (tokens/s)
5000
Throughput Ceooghhpttttooeo (tokens/s)
2000
2.82
4000
1500
2.81
2.80
d=2046
2.79
d=2560
2.78
Pandar18
dw4096
d=2816
2000
LLaMA-3.2 3.2-18
Surefire-10
500
L.2MA-3.2-30
Surefire-
2.77
2
2
2
2
I'm pratin
Batch Size
Batch Size
Figure Results for 1B and 3B models. (left) Panda- 1B closely follows the scaling law predictions for minimizing training loss. (center & right) Inference throughput comparison between LLaMA- 3.2 and Surefire models, where Surefire consistently efficient across all batch sizes.
evaluate the performance of the scaling laws under additive calibration. Finally, apply the fitted scaling laws to guide the training of large-scale models following the search framework ($5.1)
Predictive Accuracy. As Task 1-3 described in $4 we fit the conditional scaling laws on 80M, (80M, 145M), and (80M, 145M, 297M) loss-architecture data points, and subsequently evaluate on 145M, 297M, and 1B data, respectively. in tasks across different model scales validate the effectiveness and strong predictive performance of the proposed conditional scaling laws.
Figure [ the low MSE and high Spearman correlation
Ablation of Outliers. The mil-to-attennton ratio mmpp/am of weenhts models typically fall between 0.5 and 5, for example, the mlp-to-attention ratio for LLaMA-3.2-IBB LLaMA-3.2-3BB and Qwen3-8B are 4.81, 1.5, and 4.67, respectively. In Figure 6 we fit the conditional scaling law using only model architectures with m'lppatt E [0.5, 5). We ablate this choice by training model architectures with outlier /mip/aun below 0.5 and above (such as 0.1, 12.6) in Appendix D In Figure 25 (left) and Figure 25 (center) in Appendix D we show Task 3 comparison of httting the conditional scaling law without and with these outliers (with a clear Spearman correlation score degradation), which suggests to exclude extreme outliers for better predicted performances.
Ablation of Calibration. In Figure 25 (right), we ablate an alternative formulation of the scaling
laws with additive calibration, as discussed $3.3 The results on Task show that multiplicative and additive calibrations achieve similar MSE and Spearman correlations. Note that, unlike the conventional unified formulation, both calibrations assume that the effects Fmip/att and dmodel on loss separable. We further ablate more complex joint, non-separable formulations in Appendix and find that they do not provide superior predictive performance. The two-step reference-and calibration framework appears robust enough that simple calibrations perform well.
8

---

Published as a conference paper at ICLR 2026
Table 2: 3B Model Ablations. We assess the robustness of fitting-data strategy at 3B scale by training Panda-3B (using 80M, 145M, and 297M data) and Panda-3B" (using only on IB data), and compare both with LLaMA-3.2-3B. Avg. denotes mean accuracy across nine downstream tasks.
Models
mudee
ssize
Iayers
GQA
danodee/ / /VN
r
Loss (4)
Avg. (1)
LLaMA-3.2-3B
3072
8192
28
0.058
4.80
2.625
61.9
Panda-3B
4096
4096
28
3
0.077
1
2.619
62.5
Panda-3B
4096
4608
28
0.076
1.23
2.606
62.5
5.1 OPTIMAL MODEL ARCHITECTURE
Validating the conditional scaling law. We validate the conditional scaling law at the IB scale by applying multiplicative calibration on Task 3 using data from the (80M, 145M, and 297M) model variants. The learned parameters are
ag = 2.697, dj 0.0974, d2 = 0.0078, by 0.3870, 6, = 0.0063, and b2 = 0.0065.
From this, we obtain the optimal architectural configuration of / //N = 0.08, r = 1.032 for 01. = 0 and 0L. IB model by solving Demasa
ØF = O. Using this configuration, we train a LLaMA-3.2-style 1B dense model on 100B tokens, denoted as Panda-1B. Panda-1B outperforms the open-weight L.LMA-3.2-1D baseline configs by 2.1 % on average across downstream tasks (Table TT. Figure (left) further confirms the effectiveness of the conditional scaling law by showing that Panda- achieves the lowest training loss among the exhaustively trained 1B variants under the same setup.
We also scale our methodology to 3B models. Using the same approach but with data from the 80M, 145M, 297M, and 1B variants, we fit the scaling law and obtain dmodel/ /VN = 0.08 and r 1.055 for the Panda 3B model. Trained on 100B tokens, Panda-3B outperforms the open weight LLa.A-3.2-33 configuration by 0.6% on average across downstream tasks (Table [
With all components in place, we apply the search framework for inference-efficien and accurate models (Alg. D.. For the NNon-embed 1B and 3B setting trained on 100B tokens, we set the target loss L, match the training loss achieved by the LLaMA-3,2-11 and LLaMA-3.2-33 architectures, respectively.
Ablation of inference efficiency. Although inference efficiency (P) could, in principle, be expressed analytically, it depends heavily on hardware and inference configurations. Therefore, rather than solving for (P) directly, we search over feasible configurations P. that satisfy the loss constraint A100 with vLLM and select Pareto-optimal points, which denote as Surefire- 1B and Surefire-3B. Surefire- 1B and Surefire-BB outperform LLaMA-3.2- and LLaMA-3.2-30 on downstream tasks (Table with details in Appendix [ and deliver up to 42% higher inference throughput (Figure 7 center and right). We also ablate inference efficiency using both vLLM and SGLang Zheng et al. 2023) A 1100 and NVIDIA H200 GPUs (Appendix G). The results remain consistent with our vLLM-A108 1100 evaluation: Surefire- 1B IB and 3B outperform LLaMA-3.2-ID and 3B across all settings, achieving up to 47% higher throughput with SGLang on H200. This demon- strates that the efficiency gains transfer across serving stacks and hardware platforms. Detailed throughput statisties are provided Table
Ablation of fitting data strategy. While we adopt a progressive strategy for selecting fitting data across tasks ($4), results from small models (e.g., 80M) may not reliably predict behaviors at larger scales such as 3B. To assess this, we fit the conditional scaling law for the 3B model using only the variants. As shown Figure R. fitting with 1B data yields lower MSE and higher Spearman correlation when predicting 3B behavior, suggesting that the law's coefficients shift with model size. We therefore refit the law with multiplicative calibration using only the IB variants, yielding the coefficients ag 2.319, 01 0.238, 02 0.0176, bn = 0.5104, by 0.0051, and by 0.0062.
This produces an alternative optimal configuration for the 3B model, with dmnotel/ //N = 0.074 and 1.229. We train 3B model (Panda-3B") under this configuration 100B tokens and compare with both LLaMA-3.2-3B and Panda-3B (fitted from 80M, 145M, 297M, and IB data). As shown in Table Panda-3B* achieves lower training loss and comparable downstream accuracy Panda-

---

Published as a conference paper at ICLR 2026
MSE: 0.0001, Spearman: 0.5000
MSE: 0.0001, Spearman: 1.0000
Predicted Loss
3.4
Fit
Predicted Loss
3.4
Fit
Eval
Eval
3.2
3.2
3.0
3.0
2.8
2.8
2.6
2.6
2.6
2.8
3.0
3.2
3.4
2.6
2.8
3.0
3.2
3.4
Actual Loss
Actual Loss
Figure 8: Effect of the Fitting Data Strategy on Predictive Performance. (left) Fit on 80M, 145M, 297M, IB, evaluate on 3B; (right) Fit on IB, evaluate on 3B. Orange dots denote fitting data, and purple crosses indicate the test data. We compare scaling-law predicted loss with actual pretraining loss of architectures and we observe that fitting the scaling laws with only IB model data yields lower MSE and higher Spearman correlation for the 3B model loss prediction.
3B, with detailed results given in Appendix [ These findings suggest that when scaling up, is often sufficient, and sometimes preferable, to fit the law using models within closer size range to the target, such as about one third of seale.
6 RELATED WORK
Scaling laws are powerful tools to predict the performance of large language models. Existing scaling laws Hoffmann
(2022); Muennighoff al. Sardaa, et al. (2023); Kumar al. (2024); Gadre et al.J (2024): Ruan et al. (2024) characterize how model performance varies with model size, dataset size, data quality, and compute budget. With the rise of Eoxture-of-Experre (MoE) Shazeer et al.] (2017); Guo et al. (2025), , powerful architecture for large language models, recent studies Krajewski et al. (2024); Abnar et al. (2025) extend scaling laws to account for the number of experts, expert granularity, active parameters, and sparsity. Due to space constraints, defer additional related work to Appendix B
Sardana
7 LIMITATIONS AND FUTURE WORK
While our team has made notable progress, several open challenges remain that offer promising directions for future research. First, due limitations in resources and time, our evaluation does not extend to 7B models. Second, our analysis is restricted to dense models, and it remains unclear whether the results extend Mixture Experts (MoE) architectures (Shazeer al. (2017). While report inference efficiency measurements for MoE models under varying architectural choices in Appendix M, we have not yet established scaling laws for MoE architectures. Finally, our analysis limited pre-training, and it remains unclear how the results would change under post-training.
CONCLUSION
This work explores the trade-off between model accuracy and inference cost under fixed training budget. We begin by demonstrating how architectural choices influence both inference throughput and model accuracy. Building on this, we extend Chinchilla scaling laws to incorporate architectural factors and propose a two-step conditional framework for optimal architecture search: (i) train small models to fit the conditional scaling law (Eq. 3 and (ii) solve Eq. 4 for the predicted optimal architecture, followed by a local search over GQA maximize inference efficiency. Using the fitted scaling laws and our framework, we trained models up to 3B parameters, achieving up to 42% higher inference throughput and 2.1% accuracy gains across nine downstream tasks. In Table and Table of Appendix H. we compare design choices across existing open-source models at the IB and scales, further underscoring the need for inference-efficient accurate model designs.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
All experiments in this work were conducted using publicly available frameworks. Section provides details of our training, inference, and evaluation setups. In particular, we used ron LM (Shoeybi et al. 2019) for model training, vLLM (Kwon et al.] (022) and sGLang (Zheng et al.] (003) for efficient inference, and In-eval-harness (Gao al.,2202a)) for standardized evaluations.
ACKNOWLEDGEMENTS
Song Bian and Shivaram Venkataraman acknowledge the support of the NSF Diamond project OAC- 231 1767 (Democratizing Large Neural Network Model Training for Science).
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Fldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 techni- cal report. arXiv preprint arXiv:2412.08905, 2024.
Samira Abnar, Harshay Shah, Dan Busbridge, Alaaeldin Mohamed Elnouby Ali. Josh Susskind, and Vimal Thilak. Parameters flops: Scaling laws for optimal sparsity for mixture-of-experts language models. arXiv preprint arXiv:2501. /2370, 2025.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:23033 230300777, 2023.
Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J Nair, Ilya Soloveychik, and Pu- rushotham Kamath. Keyformer: Ky cache reduction through key tokens selection for efficient generative inference. Proceedings of Machine Learning and Systems, 6:114-127, 2024.
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.1 13245, 2023.
Ibrahim M Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. Advances in Neural Information Pro- cessing Systems, 16406-1666625 2023.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Temy, Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv:2108.0773:: 2021.
Song Bian. Architecture Design for Efficient LLM Training and Inference. PhD thesis, The Univer- sity of -aaoonsin-Maiioo,,, 2026.
Song Bian, Minghao Yan, and Shivaram Venkataraman. Scaling inference-efficient language mod- cls. arXiv preprint 2rX1v 2025.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Bradley Brown. Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.2 1787, 2024.
11

---

Published as a conference paper at ICLR 2026
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Keshigeyan Chandrasegaran, Michael Poli, Daniel Y Fu, Dongjun Kim, Lea M Hadzic, Manling Li, Agrim Gupta, Stefano Massaroli, Azalia Mirhoseini, Juan Carlos Niebles, et al. Exploring diffusion transformer designs via grafting. arXiv preprint arXiv:2506.00 05340, 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint (xXiv:2107.03377 2021.
Mengzhao Chen, Chaoyi Zhang, Jing Liu, Yutao Zeng, Zeyue Xue, Zhiheng Liu, Yunshui Li, Jin Ma, Jie Huang. Xun Zhou, et al. Scaling law for quantization-awae aware training. arXiv preprint arXiv:2505. 14302, 2025.
Andrew A Chien, Liuzixuan Lin, Hai Nguyen, Varsha Rao, Tristan Sharma, and Rajini Wijayawar- dana. Reducing the carbon impact of generative ai inference (today and 2035). In Proceedings of the 2nd workshop on sustainable computer systems, pp. 1-7, 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 0.5455 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 10. 2021.
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.2106) 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Worts- man, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403..m 085500 2024.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter. Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024a. URL URL /tRr://tt..//eeedd..or/////oooe
Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi, Junjie Lai, Hayden Kwok-Hay So, Ting Cao, Fan Yang, et al. Secrattention: Learning intrinsic sparse attention in your llms. arXiv preprint ar 2024b.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:23/2.0075: 2023.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small 1ms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.045 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. /2948, 2025.
Dan Hendrycks, Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv: 103.03874 2021.
12

---

Published as a conference paper at ICLR 2026
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203.15556 2022.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin- Yew Lin, et al. Minference 1.0: Accelerating pre-filling for long-context ilms via dynamic sparse attention. arXiv preprint arXiv:2407.02490 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michat Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóždż, Piotr Sankowski, et al. Scaling laws for fine-grained mixture of experts. arXiv preprini (xXXx:24022.7871,, 2024.
Tanishq Kumar, Zachary Ankner, Benjamin F Spector, Blake Bordelon, Niklas Muennighoff, Man- sheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision. arXiv preprint arXiv:241 .04300, 2024.
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Prin- ciples, pp. 61 611-626, 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang. Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024b.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint 1iXiv:2406006399. 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv: trXXiv:809.007000 2018.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-500 50376, 2023.
Aashiq Muhamed, Christian Bock, Rahul Solanki, Youngsuk Park, Yida Wang, and Jun Huan. Train- ing large-scale foundation models on emerging ai chips. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5821-5822, 2023.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint arXiv: 1:Xiv:1606.060: 2016.
Youngsuk Park, Kailash Budhathoki, Liangfu Chen, Jonas M Kübler, Jiaji Huang, Matthäus Klein- dessner, Jun Huan, Volkan Cevher, Yida Wang, and George Karypis. Inference optimization of foundation models on ai accelerators. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6605-6615, 2024.
13

---

Published as a conference paper at ICLR 2026
Jackson Petty, Sjoerd van Steenkiste, Ishita Dasgupta, Fei Sha, Dan Garrette, and Tal Linzen. The impact of depth on compositional generalization in transformer language models. arXiv preprint arXiv:2310.19956, 2023.
Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reason- ing makes smaller llms stronger problem-solvers. arXiv preprini 240i::200000699555 2024.
Siva Reddy, Danqi Chen, and Christopher Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019.
Yangjun Ruan, Chris J Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance. arXiv preprint (rXiv:2405.10938, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9);99-106, 2021.
Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint arXiv:2401.00448, 2023.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the JEEE/CVF confer- ence on computer vision and pattern recognition, pp. 4938-4947, 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gater mixture-of-experts layer. arXiv preprint 170I.v7701.066 2017.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- allelism. arXiv preprint arXiv:1909.0800 2019.
Charlic Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXivv2408.033314 2024.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arxiv:2402.00159 2024.
Charles Spearman. The proof and measurement of association between two things. 1961.
Yi Tay, Mostafa Dehghami, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint arXiv:2109. /0686, 2021.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08200 2024a.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mcsnard, Bobak Shahriari, Alcxandre Ramé, et al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv:2408. 240:22408.00/1 2024b.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint ar arXiv:2507.20534 2025.
Hugo Touvron, Thibaut Lavril Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971,
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
14

---

Published as a conference paper at ICLR 2026
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.00418, 2019.
Alex Wang. Glue: multi-task benchmark and analysis platform for natural language understand- ing. arXiv preprint arXiv:1804.07400 2018.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0766 2022.
Johannes Welbi, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. 2017.
arXiv preprint /rXivv/777.0660
Carole-Jean Wu, Bilge Acun, Ramya Raghavendra, and Kim Hazelwood. Beyond efficiency: Scal- ing ai sustainably. IEEE Micro, 44(5):37-46, 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /7453, 2023.
Guangxuan Xiao, Jiaming Tang. Jingwei Zuo, Junxian Guo, Shang Yang. Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads. arXiv preprint arXiv:
2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 3833: 2025.
Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanic Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al. Flashinfer: Efficient and customizable attention engine for IIm inference serving. arXiv preprint arXiv:2501.01000 2025.
Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung- Mong Chun. Orca: A distributed serving system for (TTransformer-aasse Based) generative models. In 16th USENIX Sympo- sium on Operating Systems Design and Implementation (OSDI 22), pp. 521-538, 2022.
Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.1 1089, 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv:1905.07830. 2019.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02338 2024.
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Efficiently programming large language models using sglang. 2023.
15

---

Published as a conference paper at ICLR 2026
A LLM USAGE
We used an LLM to improve the writing by correcting grammar in our draft. was not used to generate research ideas.
B ADDITIONAL RELATED WORK
Large Language Models. Transformers Vaswani (2017) have shown strong performance across diverse downstream tasks, such as text classification Wang 201 8); Sarlin et al. 20200, mathematical reasoning Cobbe et (2021) Hendrycks al. (2021), and code generation Chen al. (2021);
Austin et al. (2021); Jain et al. (024). The Transformer architecture serves as the foundation for many leading large language models. including GPT Brown al. (2020); Achiam al. (2023),
LLaMA Touvron et al. (202)), Gemma Team et al. (2024a), Qwen Yang et al. (2025). Kimi Team
al. 2025), and DeepSeek Liu al. (2024a): Guo al. (2025).
Serving Systems. Due to the increased inference cost, many inference systems have been de- veloped to speed up model serving Yu et al. (2022): Kwon et al. (2033; Zheng et al. (2023); Ye
et al. 2025). Specifically, M.LM/wwon al. 2023) proposes PagedAttention to manage KV cache
memory more effectively, thereby improving throughput. Similarly, SGLang /Zheng et al. (2023)
introduces RadixAttention achieve higher throughput and lower latency.
Inference-Efficien Model Design. Efforts to improve the inference efficiency of large language
models generally fall into two categories: one line of work investigates the trade-offs across different
model configurations AAladdulmohsin et al. (2023); |Bia et al. (2025): Bian (2026). while the other
focuses on designing more efficient model architectures Xiao et al. (2023); Gu Dao (2023 Gao
et al. (0024b); Jiang et al. (2024); Liu al. (002b);; Dao & Gu (2024) Xiao et al. (2024); Yuan
et al. 2025): ) Chandrasegaran et al. (2025).
C OPEN-WEIGHTED MODEL ARCHITECTURES
Table presents an overview of the open-weight model architectures utilized this paper.
Table Open- Model Architectures. We list the architectural configurations all mod- els used in this paper. Mayers is the number of layers, dmaseel is the hidden size, Thends is the number of attention heads, and Jsize is the intermediate size.
Model Name
Rlaayers
dmodel
Tiycads
fsize
GQA
Qwen2.5-1.5B
28
1536
12
8960
6
Qwen3-0.6B
28
1024
16
3072
2
D MODEL ARCHITECTURES
Table provides overview the model architectures, all configured with GQA 4 and employing I.a.MA3.2 as the tokenizer.
Table 4: Model Architectures. We list the architectural configurations of all models trained in this paper. Nuon embed is the total number of non-embedding parameters, Mayyrr the number of layers, dmodel is the hidden size, Theads is the number of attention heads, faiee the intermediate size, and T "mip/aun is the MLP-to-attention ratio.
Nnon-cmbod
Variant
Flayerss
dmodel
Theads
fsize
dmodel/ V
Tmlp/att
80M
12
768
16
2048
0.086
2.40
80M
v2
12
768
4
2688
0.086
12.6
16

---

Published as a conference paper at ICLR 2026
N non-cmbed
Variant
layers
model
Thaads
size
model V
Tmlp/attn
80M
v3
12
768
2560
0.085
6.00
80M
V4
12
768
24
1536
0.087
1.20
80M
V5
12
768
32
1152
0.086
0.68
80M
V6
12
768
40
768
0.086
0.36
80M
V7
12
768
48
256
0.087
0.10
80M
V8
12
384
32
4096
0.043
2.40
80M
12
384
5376
0.043
12.6
80M
v10
12
384
16
5120
0.042
6.00
80M
VII
12
384
48
3072
0.044
1.20
80M
v12
12
384
64
2304
0.043
0.68
80M
V13
12
384
80
1536
0.043
0.36
80M
v14
12
384
96
512
0.044
0.10
80M
V15
12
1536
1024
0.171
2,40
80M
v16
12
1536
1280
0.169
6.00
80M
V17
12
1536
12
768
0.174
1.20
80M
v18
12
1536
640
0.169
0.75
80M
V19
12
1536
20
384
0.171
0.36
80M
V20
12
1536
128
0.174
0.10
80M
V21
12
512
24
3072
0.057
2.40
80M
V22
12
512
3840
0.056
6.00
80M
V23
12
512
16
3584
0.057
4.20
80M
V24
12
512
2304
0.058
1.20
80M
V25
12
512
48
1792
0.057
0.70
80M
V26
12
512
60
1152
0.057
0.36
80M
V27
12
512
72
384
0.058
0.10
80M
V28
12
1024
12
1536
0.114
2.40
80M
V29
12
1024
1792
0.113
4.20
80M
V30
12
1024
16
1280
0.115
1.50
80M
v31
12
1024
896
0.114
0.70
80M
V32
12
1024
36
256
0.114
0.13
80M
12
2048
896
0.226
4.20
80M
V34
12
2048
640
0.231
1.50
80M
V35
12
2048
256
0.226
0.30
80M
V48
12
768
20
1792
0.086
1.68
80M
v49
12
768
28
1408
0.086
0.94
80M
v50
12
384
40
3584
0.043
1.68
80M
v51
12
384
3072
0.043
1.11
80M
v52
12
384
56
2816
0.043
0.94
80M
V53
12
384
60
2560
0.043
0.80
80M
v54
12
512
2560
0.058
1.50
80M
v55
12
512
40
2176
0.057
1.02
80M
V56
12
512
44
1920
0.058
0.82
80M
V57
12
1024
20
1152
0.113
1.08
145M
12
1024
3072
0.085
3.60
145M
V2
12
1024
3584
0.084
8.40
145M
12
1024
2560
0.086
2.00
145M
V4
12
1024
32
2304
0.084
1.35
145M
VS
12
1024
1792
0.085
0.84
145M
V6
12
1024
48
1280
0.086
0.50
145M
V7
12
1024
64
512
0.085
0.15
145M
VY
12
512
32
6144
0.043
3.60
145M
V9
12
512
16
7168
0.042
8.40
145M
V1O
12
512
48
5120
0.043
2.00
145M
VII
12
512
64
4608
0.042
1.35
145M
v12
12
512
3584
0.043
0.84
145M
v13
12
512
96
2560
0.043
0.50
145M
12
512
1024
0.043
0.15
17

---

Published as a conference paper at ICLR 2026
N non-embed
Variant
Mayers
model
heads
fsize
modet V
Tmlp/attn
145M
v15
12
2048
1536
0.170
3.60
145M
v16
12
2048
1792
0.168
8,40
145M
v17
12
2048
12
1280
0.172
2.00
145M
v18
12
2048
16
1152
0.168
1.35
145M
v19
12
2048
20
896
0.170
0.84
145M
V20
12
2048
24
640
0.172
0.50
145M
12
2048
256
0.170
0.15
145M
V22
12
768
24
3840
0.065
3.00
145M
V23
12
768
3584
0.063
2.10
145M
V24
12
768
40
3072
0.064
1.44
145M
V25
12
768
48
2560
0.065
1.00
145M
V26
12
768
56
2304
0.063
0.77
145M
V27
12
768
64
1792
0.064
0.53
145M
V28
12
1536
12
1920
0.129
3.00
145M
V29
12
1536
16
1792
0.127
2.10
145M
v30
12
1536
20
1536
0.128
1.44
145M
v31
12
1536
24
1280
0.129
1.00
145M
v32
12
1536
28
1152
0.127
0.77
145M
v33
12
1536
32
896
0.128
0.53
145M
V34
12
4096
768
0.340
3.60
145M
V35
12
4096
16
128
0.340
0.15
145M
v48
12
1024
28
2368
0.086
1.59
145M
v49
12
1024
36
2048
0.085
1.07
145M
v50
12
512
52
5120
0.042
1.85
145M
v51
12
512
60
4800
0.042
1.50
145M
v52
12
512
68
4224
0.043
1.16
145M
v53
12
512
72
3968
0.043
1.03
145M
v54
12
768
44
2944
0.063
1.25
145M
v55
12
768
52
2432
0.064
0.88
297M
VI
12
1536
24
4096
0.089
3.20
297M
12
1536
4864
0.090
11.4
297M
V3
12
1536
16
4608
0.088
5.40
297M
V4
12
1536
3584
0.090
2.10
297M
VS
12
1536
48
2816
0.089
1.10
297M
V6
12
1536
2048
0.088
0.60
297M
V7
12
1536
80
1024
0.090
0.24
297M
V8
12
768
48
8192
0.045
3.20
297M
V9
12
768
16
9728
0.045
11.4
297M
VIO
12
768
32
9216
0.044
5.40
297M
VII
12
768
64
7168
0.045
2.10
297M
V12
12
768
96
5632
0.045
1.10
297M
12
768
128
4096
0.044
0.60
297M
v14
12
768
160
2048
0.045
0.24
297M
V15
12
3072
2048
0.178
3.20
297M
v16
12
3072
4
2432
0.180
11.4
297M
VI7
12
3072
2304
0.177
5,40
297M
v18
12
3072
16
1792
0.180
2.10
297M
v19
12
3072
1408
0.178
1.10
297M
V20
12
3072
32
1024
0.177
0.60
297M
V21
12
3072
40
512
0.180
0.24
297M
V22
12
1024
36
6144
0.059
3.20
297M
V23
12
1024
12
7296
0.060
11.4
297M
V24
12
1024
24
6912
0.059
5.40
297M
V25
12
1024
48
5376
0.060
2.10
297M
v26
12
1024
72
4224
0.059
1.10
297M
V27
12
1024
96
3072
0.059
0.60
297M
12
1024
1536
0.060
0.24
18

---

Published as a conference paper at ICLR 2026
N non-embed
Variant
layers
model
Thaads
size
modee V
Tmlp/attn
297M
v29
12
2048
12
3456
0.118
5,40
297M
v30
12
2048
24
2688
0.120
2.10
297M
v31
12
2048
48
1536
0.118
0.60
297M
V32
12
2048
60
768
0.120
0.24
297M
v45
12
1536
40
3200
0.089
1.50
297M
V46
12
1536
44
3072
0.089
1.31
297M
12
1536
2688
0.088
0.97
297M
V48
12
1536
56
2432
0.089
0.81
297M
V49
12
768
6400
0.045
1.50
297M
v50
12
768
88
6016
0.045
1.28
297M
v51
12
768
104
5376
0.044
0.97
297M
v52
12
768
112
4736
0.045
0.79
297M
V53
12
3072
20
1664
0.177
1.56
297M
v54
12
3072
28
1152
0.180
0.77
297M
v55
12
1024
56
4864
0.060
1.63
297M
v56
12
1024
64
4608
0.060
1.35
297M
v57
12
1024
80
3840
0.059
0.90
297M
v58
12
1024
3328
0.060
0.71
297M
v59
12
2048
32
2432
0.117
1.43
297M
12
2048
2048
0.120
1.07
297M
v61
12
2048
40
1920
0.118
0.90
297M
v62
12
2048
1792
0.117
0.76
IB
VI
16
2048
32
8192
0.066
4.80
1B
V2
16
2048
72
5760
0.067
1.50
1B
V3
16
2816
92
2432
0.089
0.50
IB
V4
16
2816
76
3072
0.091
0.76
IB
V5
16
2816
68
3584
0.090
0.99
IB
V6
16
2816
60
4096
0.090
1.28
IB
16
2816
4480
0.089
1.50
IB
V8
16
2816
24
6144
0.089
4.80
16
2816
4736
0.090
1.85
IB
v10
16
2816
40
5120
0.090
2.40
IB
VII
16
2816
5376
0.090
2.80
IB
v12
16
2560
64
4480
0.082
1.31
IB
v13
16
2560
72
4096
0.082
1.07
IB
v14
16
2560
80
3648
0.082
0.86
IB
v15
16
2560
56
4864
0.082
1.63
IB
v16
16
2560
88
3200
0.082
0.68
IB
V17
16
2560
48
5376
0.082
2.10
19

---

Published as a conference paper at ICLR 2026
E
HYPER-PARAMETERS
Table 5
lists the detailed hyper-parameters used for training in this paper.
Table 5: Hyper-parameters. We show the hyper-parameters used for training in this paper.
Model Size
80M
145M
297M
IB
3B
Batch Size
256
256
512 8.0e-4
512
512
Max LR
1.5e-3
1.0e-3
6.0e-4
6.0e-4
Min LR
0.1x x Max LR
Optimizer
AdamW
(3)
0.9, 82 0.95)
Weight Decay
0.1
Clip Grad Norm
1.0
LR Schedule
Cosine
Warmup Steps
500
Sequence Length
2048
20

---

Published as a conference paper at ICLR 2026
F ADDITIONAL INFERENCE EVALUATION RESULTS OVER A100 GPUs
In this section, we present additional inference efficiency results NVIDIA A10 GPUs. Figure presents that, when parameter count, MLP-to-Attention ratio, and hidden size are fixed, increasing GQA leads to higher inference throughput, consistent with the findings of Ainslie al. 2023). We alter model configurations of LLaMA-33 2-0-AA-3.---- IB, 3B, and LLaMA-3.1-88 in Figure 11 Moreover, we
use the SGLang framework Zheng et al. (202)) benchmark the inference throughput of LLaMA-
3.2-1B, LLLMM-3.22 3-.2-3B, Surefir- IB, and Surefire-BB on single A100 GPU in Figure 15
Throughput (tokens/s)
Throughput (tokens/s)
Throughput (tokens/s)
8000
4000
3000
6-1024
d-1536
2-2048
2500
6000
d-2048
3000
d-3072
4-4096
4-4096
d-6144
2000
d 8199
2000
1500
1000
2000
1000
500
2
2
2
2
2 Batch Size
2
2
2
2
2
2
Batch Size
Batch Size
Figure 9: Hidden size on Inference Throughput. (left) 1B model variants; (center) 3B model
variants; (right) 8B model variants. Across varying batch sizes and model scales, larger hidden sizes yield higher inference throughput under fixed parameter budget. The legend indicates the hidden size of the models, where d dmodel.
Throughput (tokens/s)
6000
Throughput (tokens/s)
2000
Throughput (tokens/S)
1200
r=0.45
1=1.32
r=0.34
r-2.18
r=0,90
r-2
5000
1000
r-0.95
rw4.50
1500
r=3.00
r=4,20
4000
800
2000
400
500
1000
200
2
2
2
2
2
2
25
2
27
Batch Size
Batch Size
Batch Size
Figure 10: MLP-to-Attention ratio on Inference Throughput. (left) 1B model variants; (center) 3B model variants; (right) 8B model variants. Across varying batch sizes and model scales, larger -lt-tto-Attnnionn ratio increases inference throughput under a fixed parameter budget. The legend indicates the MLP-to-Auie ML--to-Attention ratio of the models, where mmip/aimn
Throughput (tokens/s)
Throughput (tokens/s)
Throughput (tokens/s)
2000
GQA-1
GQA=4
GQA-1
GOA-7
COA=1
GQA=4
6000
GOA-3
GOA-9
3000
GOA-3
GOA-15
1500
GQA-3
GOA-9
2000
2000
1000
500
2/
2
2
2 2
2
26
22
Batch Size
Batch Size
Batch Size
Figure 11: GQA on Inference Throughput. (left) model variants; (center) 3B model variants; (right) 8B model variants. This figure shows the impact of GQA on inference throughput. With the total parameter count fixed, hidden size set to 2048 (IB), 3072 (3B), and 4096 (8B), and the MI.P- to-Attention ratio is 4.0, 2.67, and 4.2, respectively. Across varying batch sizes, models with larger GQA achieve higher throughput. All evaluations are performed using the vLLM framework Kwon et al. 2023 on single NVIDIA Ampere 40GB A 100 GPU with 4096 input and 1024 output tokens.
21

---

Published as a conference paper at ICLR 2026
Furthermore, derive architectural variants by altering the configurations of Qwen3-0.6B, 1.7B, and 4B to investigate the impact of model architectural factors on inference efficiency. The results shown in Figure 12 14
Throughput (tokens/s)
5000
Throughput (tokens/s)
Throughpu (tokens/s)
d-512
d-1024
d-1280
4000
d-1024
3000
2500
d=2560
d-2048
di 4D96
2000
d=5120
3000
2000
1500
2000
1000
1000
1000
500
2
2>
2
2
2
27
2
26
Batch Size
Batch Size
Batch Size
Figure 12: Hidden size on Inference Throughput (Qwen3). (left) Qwen3-0.6B model variants; (center) Qwen3-1.7B model variants; (right) Qwen3-4B model variants. Across varying batch sizes and model scales, larger hidden sizes yield higher inference throughput under a fixed parameter budget. The legend indicates the hidden size of the models, where d dinouel-- All evaluations performed using the vLLM framework Kwon et al. (2023) on a single NVIDIA Ampere 40GB A100 GPU with 4096 input and 1024 output tokens.
Throughput (tokens/s)
8000
Throughput (tokens/s)
6000
Throughput (tokens/s)
4000
r=1.50
r=4,00
r-3.00
r=7.00
n=2285
r-
5000
6000
r=2.33
1-9.00
r-4.33
1=15.0
3000
[-4.15
r-14.4
4000
4000
3000
2000
2000
1000
1000
2>
2
27
Batch Size
Batch Size
Batch Size
Figure 13: MIP-tto-Attention ratio on Inference Throughput (Qwen3), (left) Qwen3-0.6B model variants; (center) Qwen3-1.7B model variants; (right) Qwen3-4B model variants. Across varying batch sizes and model scales, a larger MLP-to-Attention ratio increases inference throughput under fixed parameter budget. The legend indicates the MMP-to-Attention ratio of the models, where
/mmp//atn- All evaluations are performed using the vLLM framework Kwon et al. (2023) on single NVIDIA Ampere 40GB A100 GPU with 4096 input and 1024 output tokens.
Throughput (tokens/s)
5000
Through hout (tokens/s)
4000
Throughput (tokens/s)
3000
GOA=1
GOA-3
GQA-1
GQA-3
GOA-1
GOA-4
4000
GOA-2
GOA-5
3000
GQA-2
GDA-5
GQA-3
GOA-9
2000
3000
2000
2000
1000
1000
2
2
27
Batch Size
Batch Size
Batch Size
Figure 14: GQA on Inference Throughput (Qwen3). (left) Qwen3-0.6B model variants; (cen- ter) O.en3-1.7B model variants; (right) Qwen3-4B model variants. This figure shows the impact of GQA on inference throughput. With the total parameter count fixed, hidden size is set to 1024 (0.6B), 2048 (I.7B), and 2560 (4B), and the MLP-to-Attention ratio is 1.5, 3.0, and 2.85, respec- tively. Across varying batch sizes, models with larger GQA achieve higher throughput. All evalu- ations are performed using the vLLM framework Kwon et al. (2023) on single NVIDIA Ampere 40GB A100 GPU with 4096 input and 1024 output tokens.
22

---

Published as a conference paper at ICLR 2026
Throughput (tokens/s)
8000
Throughput (tokens/s)
3000
7000
2500
6000
2000
5000
1500
4000
1000
LLaMA-3.2-18
LLaMA-3.2-38
3000
500
Surefire-18
Surefire-38
2000
0
29
2
2
2
2
2
27
Batch Size
Batch Size
Figure Results for 1B and 3B models. (left) Inference throughput comparison between LLaMA-3.2-IB and Surefire-1B 1B, showing that Surefire- -B consistently achieves higher efficiency across batch sizes. (right) Inference throughput comparison between LLaMA-3,2-3B and Surefire- 3B, demonstrating that Surefire-3B consistently delivers higher efficiency across all batch sizes. The results are collected using the SGLang framework Zheng et al. (2023) on a single A100 GPU with 4096 input and 1024 output tokens.
G ADDITIONAL INFERENCE EVALUATION RESULTS OVER H200 GPUs
In this section, we present additional inference efficiency results on NVIDIA H200 GPUs. We derive architectural variants by altering the configurations of Qwen3-0.6B. 1.7B, and 4B to investigate the impact of model architectural factors on inference efficiency. The results are shown in Figure [6 I8 We also compare the inference throughput of Surefire- IB, Surefire-3B, with LLaMA-3.2. IB and LLaMA-3.2-3B over H200 GPUs in Figure T9
Throughput (tokens/s)
Throughput (tokens/s)
Throughput (tokens/s)
12000
10000
8000
d=1024
10000
1014
8000
d-2048
6000
d-2560
8000
d-2048
d-4096
d-5120
6000
6000
4000
4000
2000
2000
2000
2>
24
2
2
26
2
2
20
2"
Batch Size
Batch Size
Batch Size
Figure 16: Hidden size Inference Throughput (Qwen3). (left) Qwen3-0.6B model variants: (center) Qwen3-1.7B model variants; (right) Qwen3-4B model variants. Across varying batch sizes and model scales, larger hidden sizes yield higher inference throughput under fixed parameter budget. The legend indicates the hidden size of the models, where d = dmodel- All evaluations are performed using the vLLM framework Kwon (2023) on single NVIDIA H200 GPU with 4096 input and 1024 output tokens.
23

---

Published as a conference paper at ICLR 2026
Throughput (tokens/s)
1.5000
Throughput (tokens/s)
15000
Throughput (tokens/s
10000
12500
12500
r=2.85
rw6.68
r-2.33
r-0.00
r=4.33
r=15.0
8000
r-4.15
1-14.4
10000
10000
6000
7500
7500
5000
5000
4000
2500
2500
2000
Batch Size
Batch Size
Batch Size
Figure 17: MLP-to-Attention ratio on Inference Throughput (Qwen3). (left) Qwen3-0.6B model variants; (center) Qwen3-1.7B model variants; (right) Qwen3-4B model variants. Across varying batch sizes and model scales, a larger MLP-to-Attention ratio increases inference throughput under a fixed parameter budget. The legend indicates the MLL-to-AAttenton ratio of the models, where
/alp/attn- All evaluations are performed using the vLLM framework Kwon et al. (2023) on single NVIDIA H200 GPU with 4096 input and 1024 output tokens.
Throughput (tokens/s)
12000
Throughput (tokens/s)
10000
Throughput (takens/'s
8000
GQA=1
GQA=1
10000
GQA=2
GOA=5
8000
GQA=2
GQA=5
6000
CQA=3
GQA=9
8000
6000
6000
4000
4000
2000
2000
2000
2
2
2
2
2
2
2
2
27
Batch Size
Batch Size
Batch Size
Figure 18: GQA on Inference Throughput (Qwen3). (left) Qwen3-0.6B model variants; (cen- ter) Qwen3-1.7B model variants; (right) Qwen3-4B model variants. This figure shows the impact of GQA on inference throughput. With the total parameter count fixed, hidden size is set 1024 (0.6B), 2048 (1.7B), and 2560 (4B), and the HLLP-toAAtennii
ratio is 1.5, 3.0, and 2.85, respec- tively. Across varying batch sizes, models with larger GQA achieve higher throughput. All evalua- tions are performed using the vLLM framework Kwon et al. (2023) on a single NVIDIA H200 GPU with 4096 input and 1024 output tokens.
Throughput (tokens/s)
12000
Throughput (tokens/s)
5000
4500
10000
4000
8000
3500
3000
6000
LLaMA-3,2-1B
LLaMA-3.2-3B
2500
Surefire-18
Surefire-3B
4000
2000
24
2
2
2
24
2
2
27
Batch Size
Batch Size
Figure 19: Results for 1B and 3B models. (left) Inference throughput comparison hetween LLaMA-3.2-IB and Surefire-IB. showing that Surefire-IB IB consistently achieves higher efficiency across batch sizes. (right) Inference throughput comparison between LLaMA-3.2-- and Surefire- 3B, demonstrating that Surefire-BB consistently delivers higher efficiency across all batch sizes. The results are collected using the SGLang framework Zheng et al. (2023) on single NVIDIA H200 GPU with 4096 input and 1024 output tokens.
Furthermore, we use the SGLang framework Zheng et al. (2023) to measure the inference throughput of large language models. We construct architectural variants by modifying the configurations of
24

---

Published as a conference paper at ICLR 2026
Qwen3-0.66 1.7B, and 4B to study how different architectural factors influence inference efficiency.
The results are presented Figure 20 22 The inference throughput of Surefire- IB and Surefire-3B
compared with LLaMA-3,2-IB and LLMMA-3.2-3B on H200 GPUs is shown in Figure 23
Throughput (takens/s)
15000
Throughput (tokens/s)
Through hhrt (tokens/s)
d-512
d-1024
d-1280
10000
d=1024
d-2048
10000
6000
d-4096
6000
d=2560
d=5120
7500
4000
5000
4000
2000
2500
2000
25
2 2
2
2
2
2
2
26
Batch Size
Batch Size
Batch Size
Figure 20: Hidden size on Inference Throughput (Qwen3). (left) Qwen3-0.6B model variants; (center) Qwen3-1.7B model variants; (right) Qwen3-4B model variants. Across varying batch sizes and model scales, larger hidden sizes yield higher inference throughput under a fixed parameter budget. The legend indicates the hidden size of the models, where d dmootel- All evaluations performed using the SGLang framework Zheng et al. (2023) on a single NVIDIA H200 GPU with 4096 input and 1024 output tokens.
Throughput (tokens/s)
20000
Throughput (tokens/s)
20000
Throughput (tokens/s)
10000
r=1.50
r=4,00
n.0.00
r=7.00
r=2.85
6-6.68
15000
r-2.33
r-9.00
15000
r-4.33
1-15.0
8000
r-4.15
6000
10000
10000
4000
5000
5000
2
2
2>
Batch Size
Batch Size
Batch Size
Figure 21: MIP-tto-Attention ratio on Inference Throughput (Qwen3), (left) Qwen3-0.6B model variants; (center) Qwen3-1.7B model variants; (right) Qwen3-4B model variants. Across varying batch sizes and model scales, a larger MLP-to-Attention ratio increases inference throughput under fixed parameter budget. The legend indicates the MLL-to-Attenttio ratio of the models, where
/mmpp/atn- All evaluations are performed using the SGLang framework Zheng et al. (2023 on single NVIDIA H200 GPU with 4096 input and 1024 output tokens.
Throughput (tokens/s)
15000
Throughput (tokens/s)
12000
Throughput (tokens/s)
8000
12500
GOA=1
GQA=2
GOA=5
GOA-1
GQA-2
GOA-5
GOA-3
6000
GOA=1
GOA-3
GQA-4
GOA-9
10000
8000
7500
6000
4000
5000
4000
2500
2000
2:
26
26
Batch Size
Batch Size
Batch Size
Figure 22: GQA on Inference Throughput (Qwen3). (left) Qwen3-0.6B model variants; (cen- ter) Owen3-1.7B model variants; (right) Qwen3-4B model variants. This figure shows the impact of GQA on inference throughput. With the total parameter count fixed, hidden size is set to 1024 (0.6B), 2048 (I.7B), and 2560 (4B), and the MLP-to-Attention ratio is 1.5, 3.0, and 2.85, respec- tively. Across varying batch sizes, models with larger GQA achieve higher throughput. All evalu- ations are performed using the SGLang framework Zheng el al. (2023) on single NVIDIA H200 GPU with 4096 input and 1024 output tokens.
25

---

Published as a conference paper at ICLR 2026
Throughput (tokens/s)
14000
Throughout (tokens/s)
6000
12000
10000
4000
8000
3000
6000
LLaMA-3.2-18
2000
LLaMA-3.2-38
Surefire-18
Surefire-38
4000
24
2
2
2
1000
2
2
2
27
Batch Size
Batch Size
Figure 23: Results for 1B and 3B models. (left) Inference throughput comparison between LLaMA-3.2-IB and Surefire-1B 1B, showing that Surefire-1 1B consistently achieves higher efficiency across batch sizes. (right) Inference throughput comparison between LLaMA-3,2-3BB and Surefire- 3B, demonstrating that Surefire-3B consistently delivers higher efficiency across all batch sizes. The results are collected using the SGLang framework Zheng et al. (2023) on a single NVIDIA H200 GPU with 4096 input and 1024 output tokens.
H DETAILED THROUGHPUT STATISTICS
In this section, we present the detailed inference throughput results for the IB and 3B models in Table 6
Table 6: Summary of Results for 1B and 3B Models. We summarize the inference throughput (to- kens/s) LLaMA-3.2-1D Surefire- IB, LLaMA-3.2-3BB and Surefire-3B across vLLM and SGLang on A100 and H200 GPUs using 4096 input tokens and 1024 output tokens.
Batch Size
Hardware
Framework
Model
16
32
64
128
LLaMA-3,2-1B
1931.87
2602.72
3409.85
3825.91
Surefire-
2116.49
3290.23
4028.69
4800.05
A100
vLLM
I.LMMA-322-BB
904.83
1121.39
1136.61
1222.03
Surefire-3B
1005.44
1356.07
1613.32
1476.22
L.LMA-3.2-1B 2748.84
3643.27
4703.92
5353.29
Surefire-
3239.55
4737.63
5832.01
6962.24
A100
SGLang
LLaMA-3,2-30
1173.51
1452.97
1668.67
1762.18
Surefire-3B
1318.23
1726.20
2081.44
2251.74
LLaMA-3.2-
4311.97
6221.14
8131.65
9306.36
Surefire-111
4532.85
6992.71
9493.46
11282.56
H200
vLLM
LLaMA-3.2-3B
2269.53
3119.94
3872.14
4311.43
Surefire-3B
2309.48
3271.63
4242.33
4841.53
I.L.MA3.2-11
4812.67
6939.88
8038.34
8608.57
Surefire- 1B
5900.52
8798.68
11214.40
12645.55
H200
SGLang
LLaMA-3.2-3B
2593.04
3370.42
3868.42
4183.09
Surefire-3B
2542.21
3488.79
4446.66
4877.16
We further compare design choices across existing open-source models at the 1B and 3B scales in Table and Table For the LLaMA-3.2-11B Panda-IB, and Surefire- IB models we pretrained, we report inference throughput (tokens/s), byte-level WikiText perplexity, and full architectural config- urations in the accompanying tables. All throughput measurements are performed with vLLM on H200 GPUs using batch size 128. For the IB scale, include LLaMA-3.2. 1B B-HF and OLMo- 2-IB-HF. Because OLMo supports only a 4k context window and cannot run our standard 4k/lk setup (4096 input tokens and 1024 output tokens), additionally report results under 2k/1k k setup
26

---

Published as a conference paper at ICLR 2026
(2048 input tokens and 1024 output tokens). For the 3B scale, add LLaMA-3.2-3B-FF and Qwen2.5-3B-i all evaluated under the 4k/lk configuration.
Table 7: Comparison against open-source models at the 1B scale. We compare our pretrained L.LMA-3.2-1B. Panda-1B, and Surefire-IB IB models with LLaMA-33.2IIB-HT and OLMo-2-1B-HI in terms of inference throughput (on H200 GPUs using vLLM) and byte-level WikiText perplexity.
Model
LLaMA-3.2-IB
Panda- B
Surefire-18
LLaMA-32-IB-HH
OLMo-2-IB-HE
Wikitext PPL
1.7151
1.7016
1.7142
1.5807
1.5798
Tput (4k/Ik)
9306
6218
11283
9306
Tput (2k/1k)
11948
8961
13890
11948
7486
Model Architectural Config
Flayeess dmodel
16
16
16
16
16
2048
2560
2560
2048
2048
Tmip/ halp//ttn atm GQA
4.8
1.067
3.6
4.8
4
4
9
4
N non embed
973M
975M
965M
973M
1.074B
Table 8: Comparison against open-souree models at the 3B scale. We compare our pretrained LLaMA-3.2-3B, Panda-3B, and Surefire-BB models with LLaMA-3.2-3B-HH and Qwen2.5-3B-HHF in terms of inference throughput (on H200 GPUs using vLLM) and byte-level WikiText perplexity.
Model
LLaMA-3.2-3B
Panda-3B
Surefire-3B
LLaMA-32--B-HH
Qwen2.5-3B-HF
Wikitext PPI.
1.6489
1.6454
1.6462
1.5164
1.6185
Tput (4k/Ik)
4311
3335
4842
4311
6470
Model Architectural Config
Players
28
28
28 4096
28
36
dmodel
3072
4096
3072
2048
Tmip/amm GQA
3
1
3
7.167
3
7
3
N non--mmbed
2,82B
2.82B
2.82B
2.82B
2.77B
Our observations are as follows:
OLMM-2-1B--F is relatively close predicted optimal design, with MIP-to- attention ratio of 3 (near our predicted 3.6), but remains inference-inefficient due to its hidden dimension and GQA choices.
At the 3B scale, LLaMA-3.2-3B- achieves good accuracy but is not inference-efficient, inference-efficient but less accurate.
while Qwen2.5-3B-HH
These comparisons further underscore the necessity and relevance of our inference-efficien, high- accuracy model designs.
27

---

Published as a conference paper at ICLR 2026
I ADDITIONAL RESULTS: LOsS vs. GQA
We analyze the relationship between training loss and GQA while fixing the number of parame- ters, hidden size, and MLP-to-Attention ratio. In order keep the number of attention parameters fixed, we vary GQA by holding the total number of heads fixed (i.e., total heads = query-heads + KV-heads) and re-allocating this fixed budget between query and key-value heads, producing asym- metric changes in their effective dimensionalities.
As shown in Figure 24 unlike hidden size and MLP-to-Alee -tttett-ttnniion ratio, the relationship between loss and GQA is highly fluctuating. Varying GQA does not adjust model capacity in the coordinated way that changing dmnseee or Vmmppau does, where the dimensions of query, key, and value scale together predictably. Specifically, note the following facts when the total number of heads is fixed
Increasing the number of query-heads expands the query projection dimensionality but simultaneously reduces the number of KV-heads, increasing KV sharing and thus reducing KV expressivity.
Conversely, decreasing query-heads increases K KV-head capacity (fewer replicas) but re- duces the projection dimensionality of both query and K.V.
These opposing effects create a tradeoff, making the relationship between GQA and training loss non-smooth and often highly fluctuating.
Prior work shows only that query and KV projections can have non-interchangeable roles (e.g., head-importance heterogeneity Voita et al. (2019)). provides no monotonic or predictive theory for how reallocating capacity across query versus KV should affect loss. Consistent with this, recent open LLMs choose different GQA settings even within a single family: Qwen3 uses GQA = 2 for 0.6B/1.7B, GQA 4 for 4B/8B, GQA 5 for 14B, GQA 8 for 32B and for the 30B-A3B MoE; LLaMA-3/3.122 likewise use GQA = 4, 8, and across closely related sizes. This variation across models of similar architecture shows that GQA is treated as a discrete, model-specific hyperparam- eter, supporting our decision to tune it via local search rather than integrate into the continuous scaling law.
3.42
3.27
3.08
de768
d-1536
Log8
3.38
d67
3.26
3.07
d=512
d-1024
3.34
10
10
6
10
GQA
GQA
GQA
Figure 24: Loss vs. GQA. (left) 80M model variants; (center) 145M model variants; (right) 297M model variants. Across different model sizes, the relationship between training loss and GQA varies substantially when hidden size and the mlp-to-attention ratio are fixed. The legend denotes the hidden size of each trained model.
28

---

Published as a conference paper at ICLR 2026
J MORE ABLATION STUDY
In this section, We first evaluate the impact of outlier data on the fiting of the scaling laws in Figure 25 (left) and Figure 25 (center). Then, evaluate the fitting performance multiplicative calibrations and additive calibrations in Figure (2 (left) and Figure (right).
Finally, evaluate the performance of Joint and non-separable calibrations shown below in Fig- ure 26
dr
(ag + log(
+ a2/(
Lopt
where d = dmodel, r = / and N Nnon-embed- In Figure 26 we observe that the performance of joint and non-separable calibrations is significantly worse than that of multiplicative calibration, consistent with our discussion in $3.3
MSE: 0.0001, Spearman: 0.7451
MSE: 0.0001, Spearman: 0.3309
MSE: 0.0001, Spearman: 0.7451
3.6
3.6
3,6
Predicted Loss
Fit
Predicted Loss
Ht
Predicted Loss
Fit
3.4
Eval
3.4
3.4
3.2
3.2
3.2-
3.0
3.0
2.8
2.8
2.6.
2.8
3.0
3.4
3.6
2.8
3.0
3.2
3.4
3.6
2.6
2.8
3.2 Actual Loss
3.4
3.6
Actual Loss
Actual Loss
Figure 25: Ablation Study. (left) use multiplicative calibrations without outliers; (center) use mul- tiplicative calibrations with outliers; (right) use additive calibrations without outliers. The outlier refers to models trained with mlp-to-attention ratio below 0.5 or above 5. We observe that outlier data points harm the scaling law fit. Moreover, while multiplicative and additive calibrations differ in formulation, their MSE and Spearman values remain nearly identical. Dots denote the data points used for fitting, while crosses indicate the test data points.
0.0001, Spearman: 0.7451
0.0067, Spearman: 0.2108
Predicted Loss
Predicted Loss
3.4
Eval
3.4
Eval
3.2
3.0
3.0
2.8
2.8
2.6
2.6
2.6
2.8
3.0
3.2
3.4
3.6
2.6
2.8
3.0
3.2
3.4
3.6
Actual Loss
Actual Loss
Figure 26: Joint and non-separable calibrations. (left) use multiplicative calibrations; (right) use joint and non-separable calibrations. We observe that joint and non-separable calibrations yield higher MSE and lower Spearman scores than multiplicative calibrations, indicating inferior perfor- mance. Dots denote the data points used for fitting, while crosses indicate the test data points.
29

---

Published as a conference paper at ICLR 2026
K INFERENCE FLOPS ANALYSIS
Building on the inference FLOPs analysis from prior work Kaplan et al. 2020 we begin with the following definition:
model: hidden size
fizz: intermediate (feed-forward) size
Mayer: number layers
A: number of query heads
K: number of key/value heads
d, per-head hidden dimension (query and value)
T: per-head hidden dim the KV length prior to token generation
Based on the above definition, we have dg = Ad, and dRe = Kdh. We focus exclusively on non-embedding FLOPs, resulting in:
Attention: QKV and Project
Tlaaeers (2 + 2dmosteddsn + 2dmodeddyy + 2c
Attention: Mask
Maayers (2Tdg)
Feedforward:
Tlaayers (3 2dmodel size)
Total Inference non embedding FLOPs:
Total-FLOP Plaayrrs dy + 2dmande
+ 2dmodeadgy + 2dmnadedd
2Td, + 2d Ssize
g
up, gate,
Since Pnon-emh Dtayers (2dmateldy + 20 + f
Therefore, Total-FLOPs
2 2Pnon-emh + Mayyer Tdg
we adopt the following three approaches to accelerate inference:
Increasing the MI.P-toAAttention ratio reduces the term 27 thereby lowering the total
FLOPs.
Increasing the hidden size reduces the term 27d thereby lowering the total FLOPs.
30

---

Published as a conference paper at ICLR 2026
L MORE LARGE-SCALE TRAINING RESULTS
In this section, we first show the detailed result over downstream tasks of large-scale models in Table and Table 10
Table 9: Detailed Results on Downstream Tasks for 1B Models. In this table, we show detailed results of IB models over 9 downstream tasks.
Downstream Tasks
LLaMA-32--IB Panda-1B IB
Surefire- 1B
Arc-Easy
58.8
60.9
59.7
Are-Challenge
29.8
28.9
30.2
LAMBADA
52.8
55.1
52.0
HellaSwag
56.9
58.4
56.6
OpenBookQA
32.0
33.2
32.0
PIQA
73.6
75.2
73.0
SciQ
84.8
87.2
84.9
WinoGrande
57.1
58.6
57.5
COQA
48.7
55.3
52.7
Avg.
54.9
57.0
55.4
Table 10: Detailed Results on Downstream Tasks for 3B Models. this table, we show detailed results of 3B models over 9 downstream tasks.
Downstream Tasks
LLLMA-3.2-3B
Panda-3B
Surefire-3B
Panda-3B
Arc-Easy
66.4
65.5
67.6
66.8
Arc-Challenge
33.3
35.2
33.9
33.3
LAMBADA
60.6
61.8
61.4
61.5
HellaSwag
66.7
66.9
67.0
67.8
OpenBookQA
38.4
38.6
38.6
38,0
PIQA
76.8
76.9
77.4
76.8
SciQ
89.4
91.2
92.1
90.5
WinoGrande
62.5
63.2
60.5
62.7
COQA
63.3
63.4
65.4
64.9
Avg.
61.9
62.5
62.6
62.5
31

---

Published as a conference paper at ICLR 2026
M MOE INFERENCE
In this section, we examine how the Mixture-of-Experts (MoE) architecture affects inference eff- ciency. Figure 27 indicates that larger hidden sizes and higher Active-Expertstto-Atttnnnn ratios improve the inference throughput of MoE models, consistent with observations in dense models.
Throughput (tokens/s
10000
Throughput (tokens
5000
Throughout (okens/s)
5000
1024
9000
2048
4500
4500
8000
4096
4000
4000
3500
3500
3000
1024
3000
1024
5000
2500
4096
2500
4096
4000
2000
2000
Active- -ttttt- Ratio
Active-Experts-L t--tttn Ratio
Active-Expene Eeivve-Epeert-toAAtt Ratio
Figure 27: Active-Experts-to-AAtn on Inference Throughput. (left) 3B-A1.1B model variants; (center) 5.3B-A1.77 model variants; (right) 8.3B-AI ..B model variants, We study the effect of the Teepeeeeeettttttettttteee ratio on inference throughput by fixing the total number of active parameters, setting GQA to 4, and using batch size of 2048 to reduce MoE inference variance in this figure. All evaluations are performed using the vLLM framework Kwon et al. (2023) on a single NVIDIA Ampere 40GB A100 GPU with 1024 input and 256 output tokens.
32


Reviews:
[{"review_id": "zlIQEVYkFj", "reviewer": "Reviewer_fQwj", "summary": "This work builds on the Chinchilla scaling laws by adding architecture-based dimensions like hidden size, MLP-to-attention ratios, and grouped-query attention, to analyse the trade-off between inference efficiency and downstream accuracy. Authors fit a conditional scaling law on >200 Llama-style models (80M-30B #params), authors identify Pareto-optimal configurations and improve Panda/Surefire models that improve downstream accuracy and inference throughput over Llama-3.2 baselines.", "questions": "- What happens for larger models? Do the authors have architectural recommendations for these regimes?\n- The paper's results are based on vllm -- do results transfer also to e.g. SGLang (https://github.com/sgl-project/sglang)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Conditioning the Chinchilla scaling laws on hidden size and MLP-to-attention ratios is timely, useful, and interesting\n- Experiments over 200+ trained models provide robust empirical results\n- Architectures suggested by the new scaling laws yield sensible gains -- Panda models raise average zero-short accuracy, while Surefire significantly improves the inference throughput by up to ~25%", "weaknesses": "- Results stop at 3B parameters -- what happens at larger model scales?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pd6R2REmat", "reviewer": "Reviewer_rBcz", "summary": "This paper addresses the critical and underexplored trade-off between the accuracy and inference efficiency of Large Language Models (LLMs). While traditional scaling laws focus on training compute, parameters (N), and data (D), they largely ignore the architectural choices that significantly impact inference cost, which is a dominant expense in real-world deployment.", "questions": "The authors note the \"highly fluctuating\" relationship between GQA and loss (Figure 13)33. Do you have a hypothesis for why GQA behaves so erratically compared to the smooth curves for $d_{model}$ and $r_{mlp/attn}$?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed conditional scaling law is a novel extension of established Chinchilla scaling laws12. By formulating the loss as a separable, calibrated function of architectural ratios relative to a Chinchilla-optimal reference loss, the authors provide a practical tool for architecture design.\n\nBesides, the claims are substantiated by a large-scale empirical study involving over 200 trained models, spanning a wide range of parameter counts (80M to 3B) and token budgets (8B to 100B). This comprehensive dataset provides a strong foundation for fitting the scaling laws.", "weaknesses": "1. The experiments are validated up to 3B parameters. While this is a substantial undertaking, the authors acknowledge that the evaluation does not extend to the 7B+ scale, which is a very common size for deployed open-source models.\n2. The entire analysis is confined to dense decoder-only transformers. It is unclear if these findings, particularly the U-shaped loss curves and the separability assumption, would hold for other popular architectures like Mixture-of-Experts (MoE).\n3. The study finds that GQA significantly impacts inference efficiency (Figure 9) but has a \"highly fluctuating\" and inconsistent relationship with training loss (Figure 13). Consequently, GQA is not integrated into the conditional scaling law itself. Instead, it is handled via a separate \"local search\" after other parameters are optimized. This feels less principled and integrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b65BIr6RN9", "reviewer": "Reviewer_Bn1c", "summary": "This paper revisits scaling laws for large language models (LLMs) through the lens of model architecture and inference efficiency.\nWhile existing works (e.g., Chinchilla) focus on training efficiency by relating model parameters, data tokens, and loss, this work introduces architecture-aware scaling laws that explicitly incorporate structural hyperparameters affecting inference cost.\n\nThrough a large empirical study (>200 models from 80M–3B parameters trained on 8–100B tokens), the paper shows that performance and inference throughput follow predictable trends governed by these architecture variables. Using these fits, the authors predict “optimal” configurations for new small models (e.g., Panda-1B, Panda-3B) that achieve ~26% higher inference throughput and ~2% higher accuracy than LLaMA-3.2-1B.", "questions": "1. How sensitive are the predicted “optimal” ratios to the hardware platform (A100 vs. H100 or TPU)? Could authors provide several experiments on another hardware?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Novel perspective: Introduces an architecture-aware formulation of scaling laws, bridging the gap between training efficiency and inference efficiency.\n2. Extensive empirical coverage: The study includes over 200 model configurations across hidden sizes, attention/MLP ratios, and token budgets.\n3. Practical insight: Demonstrates that scaling law fits from small models generalize to larger ones, potentially guiding efficient architecture search.", "weaknesses": "1. Empirical “optimality” is descriptive: The proposed “conditional scaling law” is fitted post-hoc to data rather than derived from optimization principles.\n2. Hardware dependence: Reported inference efficiency improvements rely on specific implementation details (vLLM, A100), which may not generalize to other hardware or serving stacks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gXmqEHLAIy", "reviewer": "Reviewer_qzeo", "summary": "This paper addresses a challenge in the efficient deployment of large language models. Whiile existing scaling laws guidfe the trade-off between model size and training data for optimal accuracy, they largely ignore inference efficiency. This work bridges the gap by investigating how architecture hyperparameters impact inference throughput and model accuracy, including hidden size, mlp-to-attention ratio, and GQA.", "questions": "The questions are listed in the Weaknesses. I believe this direction is worth researching and this paper will be a good start if the above problems are well solved.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Novel and practical contribution**: The focus on inference scaling law is highly relevant for the practical deployment of LLMs. The proposed conditional scaling law is a clever and effective extension to the scaling law.\n2. **Rigorous and extensive experiments**: The scale of the study is impressive, with over 200 models trained across different parameter scales. This provides a solid empirical foundation for their claims.", "weaknesses": "1. **The design for the conditional scaling law is lack of theoretical support**. The paper doesn't present why the conditional scaling law is fomulated as a multiplicative or additive equation. Why these two forms act similar? Are they still similar if the candidate models are much larger (e.g., 70B+)? Which one should we take in the future research?\n2. **Lack of a clear summarization**: It is encouraged to present an summarization on the conclusions about how to design an efficient LLM architecture. It is also encouraged to present an comparison on the design choices of existing open source models about which models are closer to the optimal design.\n3. **Lack of analysis on GQA**: Why the relationship between loss and GQA is highly fluctuating? This conclusion is counterintuitive and requires a discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)