Scaling Inference-Efficient Language Models
Authors: Song Bian
Minghao Yan
Shivaram Venkataraman
Venue: ICML 2025 poster (2025)
Keywords: Scaling Laws
Inference
Language Models
Primary area: deep_learning->large_language_models
TL;DR: Applying scaling laws to train inference-efficient models

Abstract: Scaling laws are powerful tools to predict the performance of large language models. However, current scaling laws fall short of accounting for inference costs. In this work, we first show that model architecture affects inference latency, where models of the same size can have up to $3.5\times$ difference in latency. To tackle this challenge, we modify the Chinchilla scaling laws to co-optimize the model parameter count, the number of training tokens, and the model architecture. Due to the reason that models of similar training loss exhibit gaps in downstream evaluation, we also propose a novel method to train inference-efficient models based on the revised scaling laws. We perform extensive empirical studies to fit and evaluate our inference-aware scaling laws. We vary model parameters from 80M to 1B, training tokens from 1.6B to 30B, and model shapes, training 63 models. Guided by our inference-efficient scaling law and model selection method, we release the Morph-1B model, which improves inference latency by $1.8\times$ while maintaining accuracy on downstream tasks compared to open-source models, pushing the Pareto frontier of accuracy-latency tradeoff.
Notably, our experiments reveal that wider and shallower models can yield efficiency gains while preserving accuracy.

Scaling Inference-Efficiee Language Models
Song Bian Minghao Yan
Shivaram Venkataraman
Abstract
0.55
OFT-IML-1.38
Scaling laws are powerful tools to predict the performance of large language models. How- ever, current scaling laws fall short of accounting for inference costs. In this work, we first show that model architecture affects inference latency, where models of the same size can have up 3.5x difference in latency. To tackle this chal- lenge, modify the Chinchilla scaling laws co-optimize the model parameter count, the num- ber of training tokens, and the model architecture. Due to the reason that models of similar training loss exhibit gaps in downstream evaluation, we also propose novel method train inference- efficient models based the revised scaling laws. We perform extensive empirical studies to fit and evaluate our inference-awee aware scaling laws. We vary model parameters from 80M to IB, training tokens from 1.6B to 30B, and model shapes, train- ing 63 models. Guided by our inference-efficient scaling law and model selection method, we re- lease the Morph-1B model, which improves infer- ence latency by 1.8x while maintaining accuracy downstream tasks compared to open-source models, pushing the Pareto frontier of accuracy- latency tradeoff. Notably, our experiments reveal that wider and shallower models can yield effi- ciency gains while preserving accuracy.
Morph-1B-v2
Morph-1B-v1
Morph-1BX
*
Accuracy
OPT-1.3B
0.50
Open-LM-18
Pythia-1.38
Neox-1.38
0.45
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Latency (s)
Figure 1. We train Morph- 1B and its variant models 30B to- kens. The results indicate that Morph- IB maintains high accuracy downstream tasks and achieves faster inference than open- source models and their variants. OPT-IML- 1.3B achieves slightly higher performance on downstream tasks than Morph- since trained on 180B tokens (Iyer et al., 2022) and is instruction-tumed. We obtain the accuracy by evaluating models on 11 downstream tasks used by Open-LM (Gururangan ct al., 2023). The inference latency is collectod by using Hugging Face generate func- tion single NVIDIA Ampere 40GB A100 GPU with batch size input length 128, output length 256.
model design efficiency by accurately estimating an LLM's performance via the results of smaller language models, which can be developed using far less cost and fewer com- puting resources.
1. Introduction
However, the field progresses, is increasingly evident that focusing solely training does not adequately ad- dress the practical realities of deploying these models at scale (Touvron et al., 2023a). A key limitation of existing scaling laws is their disregard for inference costs, which dominate the long-term expenses of utilizing large models in real-world applications (Sardana et al., 2023). other words, while compute-optimal models minimize training loss per unit of compute, they may result in models that more expensive to serve, especially in latency-sensineg plications such as chatbots. The growing adoption of LLMSs in reasoning systems also highlights the need for scaling frameworks that explicitly account for inference costs (Snell et al., 2024; Brown et al., 2024; Luo et al., 2024; Qi et al., 2024; Guan et al., 2025).
Scaling laws have shown immense value in guiding the development of large language models (LLMs) by establish- ing predictable relationships between model size, training compute, and performance metrics, such as loss and down- stream tasks performance (Kaplan al., 2020; Hoffmann et al., 2022; Muennighoff et al., 2023; Gadre al., 2024). They reliably reduce the cost of training LLMs and improve
Equal contribution 'Department of Computer Sciences, Uni- versity of Wisconsin-Madison Madison WI, USA. Correspon- dence to: Song Bian <sbian&@wisc.edus: eisc.
ap-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Scaling Inference-Efficient Language Models
While a recent study (Sardana et al., 2023) has introduced scaling laws that consider the total number of FLOPS for training and inference, their constraint requires estimating the number of tokens inferred during the model's s lifespan. As inference is performed repeatedly throughout a model's lifecycle, their scaling law (Sardana et al., 2023) is not practical for real-world applications.
18
16-
2-98
14
Latency (5)
12-
MiniCPM-18
10.
gemmme-2226
Qwen2.5 5-14B
Liama 2-138
In addition, current scaling laws focus on balancing model size (number of parameters) and the number of training to-
8-
gemma 78
Limma Que 5.38
6-
Owen
1155
MiniCFM.28
kens within fixed compute budget' (Hoffmann et al., 2022; Muennighoff et al., 2023; Sardana et al., 2023; Gadre et al., 2024). Among these, the Chinchilla scaling law (Hoffmann et al., 2022) is the most renowned, demonstrating that the optimal training solution is D 20/N for fixed FL.OPs budget, where N is the number of parameters and D is the number of tokens for training. However, in practice, that FL.OPs are not a primary constraint. Models are trained for durations much larger than Chinchilla optimal (e.g.. IT tokens for Llama-7B and 8T tokens for Gemma-2-9B (Tou- vron et al., 2023a; Team et al., 2024b)). Additionally, practi- tioners choose the model size (number of parameters) based on the memory capabilities of the deployment device (Hu et al., 2024; Yao et al., 2024). Thus, we need scaling laws that can explicitly consider data size, device memory, and inference latency.
gemma 28
4-
Qwen2.5 2. 0.58
310
30
40
50
60
Payers
Figure 2. Open-Source LLM's Inference Latency: An overview of inference latency in open-source L.MMs. The evaluated models include LLaMA (Touvron et al., 2023b), Qwen (Yang et al., 2024), Gemma (Team al., 2024a;b), and MiniCPM (Hu et al., 2024). All evaluations were performed using the Hugging Face generate function on a single NVIDIA Ampere 40GB A100 GPU with batch size 1, input length 128, and output length 256.
see
laws, We observe that overtraining plays critical role in obtaining an accurate scaling law and that inference- efficient scaling law is more accurate and robust than the Chinchilla scaling law. Using only 6 data points and 85 A100 GPU hours for curve fitting, our inference-efficien scaling law can still accurately predict the loss of scaled-up models ($3, $4).
In this work, we aim to address the following question:
Given dataset and parameter constraints, can we train an inference-efficiett and accurate model for downstream tasks?
Lastly, we train the Morph- 1B2 model using the best model configuration predicted by our inference-efficient scaling law and ranking algorithm. Figure I summarizes our main results. Compared to other open source models of sim- ilar size, Morph-1B improves the inference latency by 1.8x while maintaining accuracy over downstream tasks. These findings underscore the effectiveness of our inference- efficient scaling law. By designing a general scaling law that focuses inference latency, our work can also capture the accuracy-efficiency trade-off for recent and future archi- tectural optimizations, such as GQA (Touvron et al., 2023b; Dubey et al., 2024) and MIA (Liu et al., 2024a).
We first show that the number of parameters is not the clusive factor affecting inference efficiency. As illustrated in Figure 2, the model architecture also plays critical role. Following this observation, we introduce inference-efficicn scaling laws, building upon the Chinchilla scaling law and incorporating model architecture considerations. Addition- ally, due to the disparity between model loss and accuracy in downstream tasks, we develop a novel method (Figure 6) that utilizes inference-efficien scaling laws to rank various model architectural choices. Our findings suggest that the relative ranking of loss predictions from scaling laws is more significant than their absolute values ($2).
2. Scaling Laws
To fit the inference-cfficient scaling laws, we train more than 60 models ranging from 80 million to 339 million parameters for up to 13 billion tokens and record the loss of models. We also train several models with more than 1 billion parameters and 20 billion tokens to evaluate the predictive power of the fitted inference-efficte scaling
In this section, we first present the formulation of existing language model scaling laws in $2.1. Next, we introduce scaling law for inference efficiency that takes into ac- count the number of parameters, training tokens, and model shape in $2.2. Finally, we present a novel method to select
"The training code is available at https: hhubthbb... com/ www.rpine/ppen-nnee Im-morph. The Morph-1B model checkpoint is available at https:s///uggingffeee Nai vaiiess/rrrorpp-ll
'Tee compute cost is approximated as FLOPs(N, D) & GND, where N is the number of parameters and D is the number of training tokens.
2

---

Scaling Inference-Efficien Language Models
4096
1B
10-
Latency (s)
5120
Latency (s)
Latency (s)
3B
6144
12
7B
6
6192
16
4-
4-
2-
2-
20
25
2000
4000
6000
8000
10000
10
10a
Mlayers
dmodal
dmdell Mayyers
(a) Vary layers (/ugen)) ). fix hidden size
(b) Vary hidden size (dmode)), fix layers
(c) Vary ratio (dmots /aaasn//aaaaada Tayerss, fix size N
Figure 3. Model Shape on End-to-End Inference Latency: (Left) We illustrate the correlation between inference latency and the number of layers, with constant hidden size. Due the sequential nature of LLM exccution, latency increases linearly with the number layers. (Center) We plot the relationship between inference latency and hidden size with the number of layers fixed. We see that model width does not affect latency for smaller models but only for larger models. (Right) show the relationship between inference latency and aspect ratio, with the number of model parameters fixed. We see downward trend in inference latency as make the model wider and shallower. evaluations were performed using the Hugging Face generate function on a single NVIDIA Ampere 40GB A A100 GPU with batch size I, input length 128, and output length 256.
inference-efficient language models for training using our scaling laws in $2.3.
2.2. Inference-Efficient Scaling Laws
Despite its popularity, the Chinchilla scaling law fails to resolve the following challenges:
2.1. Preliminaries
Scaling laws predict model's loss based on the allocated compute resource C. Following OpenAl (Kaplan et al., 2020) and Chinchilla (Hoffmann et al., 2022), the compute resource C is function dependent on the model size N and the number of training tokens D. The goal is to minimize model loss within the constraints the available compute resources:
The FLOPS constraint outlined in Eq. (1) does re- flect how model training decisions are made in practice. First, both the model size and the training corpus determined in advance to accommodate for resource constraints when deploying these models (Touvron et al., 2023a). Therefore, for each model and train- ing corpus pair, training FLOPs is essentially a fixed constant (assuming training epochs are also predeter- mined). Furthermore, while the Chinchilla scaling law suggests training 10B parameter model with 200B tokens, overtraining frequently occurs in practice. For example, the LLaMA-3-888 model uses 15 trillion to- kens for training (Touvron et al., 2023a), while the Gemma-2-9B model utilizes 8 trillion tokens (Team al., 2024b). These numbers are 44-93x larger than the Chinchilla optimal recommendation.
arg min L(N, s.t. FLOPS(N, D) C (I) NDD
Using the formulation above, several scaling laws have been established (Kaplan et al., 2020; Hoffmann al., 2022: Muennighoff et al., 2023; Sardana al., 2023) accurately model the performance of large language models from train- ing series of much smaller ones. The Chinchilla loss function L(N, 3 is widely adopted to predict a model's training loss:
Existing scaling laws focus only on how the number of parameters affeets inference latency. However, as depicted in Figure 2, smaller models can sometimes exhibit higher inference latencies than larger models. For instance, MiniCPM-IB (Hu et al., 2024) has higher latency compared to Qwen2.5-14B (Yang et al., 2024).
L(N D) = AN BD
where N is the number of parameters, is the number of tokens used for training and A, B, E, a, are parameters to be learned. Through training multiple models and curve fitting, Chinchilla (Hoffmann et al., 2022) identify D 20N as the compute-optimal solution for large language model pretraining.
In view of this, we propose rewriting Eq. (1) below to meet practical requirements:
LLike Chinchilla (Hoffmann al., 2022), smoothed training loss estimate test loss.
arg min T(N, D) s.t. N & Nc. D < De, Tin S To (3) N,D
3

---

Scaling Inference-Efficient Language Models
Throughpu (tokens/s)
Throughput (tokens/s)
4000
Throughput (tokens/s)
4000
(1792, 32)
(2304, 48)
(3072. 60]
3000
(1920, 28)
3000
(2688, 36)
5000
(3584, 40)
(2048, 24)
(3072, 28)
(4096, 32)
2000
(2304, 20)
2000
(3840, 18)
2000
(6144, 14)
(2560, 16)
(4608, 12)
(8192. 8)
1000
1000
1000
2
2'
2
2
2
2
2
2
2
Batch Size
Batch Size
Batch Size
(a) IB Model Variants
(b) 3B Model Variants
(c) 7B Model Variants
Figure 4. Model Shape on Throughput: We examine the relationship between inference throughput and model architecture by fixing the total parameter count and varying the hidden size and number of layers. Across different batch sizes, wider and shallower models consistently yield better inference throughput for large language models. Each tuple in the legend represents model configuration: the first number is the hidden size dmnddd, and the second the number of layers Wayer All evaluations Face generate function on a single NVIDIA Ampere 40GB A100 GPU with input length 128, and output length 256.
performed using the Hugging
4.5
3.5
Loss
e--mm
Loss
4.0
3.0
116M
3.5
116M
80M
3.5
164M
116M
237M
3.0
237M
2.5
164M
313M
313M
2.5
23
2
2
2
2.5
2
2
25
2
29
29
2.0
2
2
2
dmoddl
Mlayers
dodeel
Maayers
modeel/
Mlayers
(a) 20N tokens
(b) 40.V tokens
(c) 160N tokens
Figure 5. Inference-Efficceen Scaling Laws: In this plot, each data point represents a training run with the given configuration. The dashed lines represent predictions based on the inference-efffcient scaling laws outlined in Eq. (4). (Left) The number of training tokens 20N; (Center) The number of training tokens is 10N;; (Right) The number of tokens used for training is 160N, where N denotes the number of parameters. Our scaling law accurately captures the training loss different training durations.
where Nc represents constraint on model size and De denotes the constraint on the number of training tokens. To account for the inference latency budget, introduce new term To to our scaling law formulation to represent the inference latency constraint.
achieve different latency targets by changing the ratio of the number of hidden parameters in one layer (dmodel) vs.
the number of layers (Mayyn.) Moreover, in Figure 4, We study the relationship between model shape and inference throughput under fixed parameter budget. We observe that, under fixed parameter budget, wider and shallower models consistently achieve higher inference throughput. to space constraints, results on the relationship between aspect ratio and time to first token (TTFT) are provided in Appendix C.
Motivated by Figure 2, we closely examine the effect of the aspect ratio (dmodel/ /dmyeel//Haaess) on inference latency and through- put by altering the hidden size dmodel and the number of layers Mayyer as shown in Figure and Figure 4. Reasonable aspect ratios chosen based open-weight models listed in Appendix G.
Prior work (Kaplan et al., 2020) has shown the impact of the aspect ratio (dmmstel Ptayers) the performance of the model. However, it does not define the connection between model size, number of training tokens, and model shape. To establish this relationship, we trained several small models N E {80, 116, 164, 237, 313JM by varying the aspect ratio and setting D {20, 40, 160) N. Due to resource limita- tions, we only train subset of the models at D 160N.
Figure 3(a) shows that inference latency increases linearly with the number of layers when the hidden size remains constant. This occurs as the inference computation must be performed sequentially, one layer at time (Yan al., 2024). However, the matrix computations within single layer can be performed in parallel. Furthermore, Figure 3(c) indicates that for the same number of parameters, we can

---

Scaling Inference-Efficienl Language Models
fx
Downstream Tasks
Predicted Loss
Profiled Latency
GPU
Select Models for Training
Candidate Models
B Rank Loss Latency
Evaluate Models
E
Release Model
Figure 6. An Overview of Methodology: (A) The model training team first selects several candidate models with various model sizes and configurations (B) Mcasure the inference latency using open-source inference systems and predict model loss with fitted scaling laws; (C) Select top-k candidate models for training based inference latency and loss; (D) Evaluate the models over downstream tasks after training: (E) Release the best model based on inference efficiency and performance over downstream tasks.
0.7
0.5
80M
80M
80M
116M
116M
116M
Accuracy
0.7
Accuracy
0.6
Accuracy
0.4
237M
237M
237M
313M
313M
0.5
0.4
0.2
2
2
3
5
2
3
4
Loss
Loss
Loss
(a) PIQA
(b) BoolQ
(c) HellaSwag
Figure 7. Accuracy vs. Loss: (Left) We illustrate the correlation between accuracy and model loss on PIQA (Bisk et al., 2020). (Center) present the connection between accuracy and model loss on BoolQ (Clark et al., 2019). (Right) We show connection between accuracy and model loss HellaSwag (Zellers et al., 2019). These three patterns shown in the plots demonstrate the difficulty in robustly predicting individual downstream accuracies from scaling laws.
We plot the loss values against the aspect ratio in Figure 5. From the figure, we can see that the most suitable model shape adjustment the inclusion the term (1 (1+20 <R*) to the Chinchilla scaling law (Hoffmann al., 2022). There- fore, we derive the following inference-efficient scaling law formulation:
loss to errors in downstream tasks. Inherently, predicting the error in downstream tasks becomes challenging when model losses are similar, due to noise and inaccuracies in scaling laws. We observe this in Figure 7. To tackle this challenge, develop
method for training inference- efficient models, shown in Figure 6. Our key idea is that inference latency measurement has negligible overhead, and scaling laws can help us estimate the loss of scaled-up models. Thus, we propose identifying top-k candidate mod- els using inference latency and loss data, where the user can choose After training, we evaluate these models on downstream tasks and release the best-performing model to the public, taking into account both inference latency and performance on downstream tasks. Our method (Figure 6) can also be applied to different architectural optimizations, such as MLA (Liu et al., 2024a), to quantify the accuracy- efficiency tradeoff.
L(N, D. R) (E AN +BD -)- (1++e7) (4)
where N is the number of parameters, training tokens, and R demodet// /aayerss the aspect ratio. Moreover, A, B, E, a, 3, 7. are learned parameters. In Figure 5, we plot the predicted values from the scaling law against the observed values from training. More details of the experimental setup and fitting procedure can be found in $3.
the number of
2.3. Methodology
Scaling laws were first developed to predict the loss of language models. However, LLMs are evaluated on the per- formance of downstream tasks. A recent study (Gadre et al., 2024) attempts establish scaling laws that link evaluation
5

---

Scaling Inference-Efficient Language Models
3. Experiments
ting problems, where the goal is to find the parameter vector |3 of model f(x; (3) that minimizes the sum of squared deviations. Formally, the problem can be expressed as arg ming Ei=1 [3: - f 5)] where (æi, 3r) are data pairs. Following observations from Chinchilla scaling law (Hoffmann et al., 2022) and another recent work (Gadre et al., 2024), we set ax B, and ry equal to simplify the fitting procedure. To fit and evaluate the scaling law, we train 63 models using range of model sizes, shapes, and amounts of training tokens. The size of our model ranges from 80M to 339M and the number of tokens used for training ranges from 1.6B to 12,8B. Detailed model configurations can be found in Table 4 in Appendix A.
We next discuss the experiment setup we use for model training and evaluation ($3.1). Following that, in $3.2, we demonstrate how fit scaling laws using our experimental results.
3.1. Experimental Setup
Training Setup. For all experiments, we train transformer- based decoder-only language models (Vaswani, 2017). Fol- lowing (Gururangan et al., 2023; Gadre et al., 2024), the model's architecture is similar to GPT-2 (Radford et al., 2019) and LLaMA (Touvron et al., 2023a), with GPT- NeoX (Black et al., 2022) employed as the tokenizer. We train models with maximum of 1.5 billion parameters for up to 30 billion tokens, following the compute-optimal setup in (Hoffmann et al., 2022). The models are trained on uni- formly sampled subsets of DCLM-Baseling (Li et al., 2024) with one epoch, ensuring no repetition in data (other than possible data repetition in the dataset itself). More details are included in Appendix A.
4. Results
In this section, we first study the predictive power of our inference-efffcient scaling laws in $4.1. Then, in $4.2, we release an inference-efficien model that maintains accuracy on downstream tasks compared with open-sourced models by using the methodology outlined in Figure 6. We also show that our method significantly outperforms Chinchilla in predicting the best model configurations. Finally, we perform ablation studies obtaining robust scaling laws and show that our inference-efficient scaling law is more robust than Chinchilla in various scenarios in $4.3.
Evaluation Setup. We use HuggingFace (Wolf, 2019) to measure the inference efficiency of models over a sin- gle NVIDIA Ampere 40GB A100 GPU. By default, we set the number of input and output tokens to be 128 and 256, respectively, aligning with the distribution outlined in ShareGPT (Kwon et al., 2023).
Table I. Data Used to Fit Scaling Laws: In this table, we show the number of parameters and tokens used in model training to fit the scaling laws in Figure 8-10. / indicates we use all model variants with the given size and X means we do not use any model variants with the given size, & indicates that we randomly sample one model variant from the candidate set. The details of model variants are included in Appendix A.
We use LLM-foundry (lm, 2024) along with a zero-shot evaluation approach to evaluate model performance on downstream tasks. We evaluate the downstream task ac- curacy of models derived from the methodology outlined in $2.3 using the following datasets: ARC-Easy (Clark et al., 2018), ARC-Challenge (Clark et al., 2018), BoolQ (Clark et al., 2019), COPA (Roemmele et al., 2011), Hel- laSwag (Zellers et al., 2019), LAMBADA (Paperno et al., 2016), PIQA (Bisk et al., 2020), WinoGrande (Sakaguchi et al., 2021), MMLU (Hendrycks et al., 2020), Jeop- ardy (Jeo, 2022), and Winograd (Levesque al., 2012).
N
D
Figure 8
Figure 9
Figure 10
80M
1.6B
116M
2.3B
164M
3.2B
/
/
237M
4.7B
/
Furthermore, to compare the predicted loss against the ac- tual loss, we measure relative prediction error: //6 /1//), mean squared error (MSE): 1 Σ'= (t)s v. and R = 1 Σm= (ph- / (v):- - 3 where vb represents the actual loss, v the predicted loss from scaling laws, and T = Σ Vi. We also apply Spearman's 's rank correla- tion coefficient (Spearman, 1961) to evaluate how well the predicted rankings correspond to the actual rankings.
313M
6.2B
80M
12.8B
4.1. Prediction acccuracy
As shown in $3.2, we obtain the actual losses of various models by training multiple small models with different model configurations to establish the scaling law. We set N E (80, 116, 164, 237, M and D 20N to train small models and collect the data to fit the learnable param- eters in Eq. (2) and Eq. (4). Furthermore, to enhance the generality of the scaling law, we train 80M models with = 160N tokens, thereby collecting data from an over-
3.2. Fitting Scaling Laws
Following (Gadre et al., 2024), we use the Levenberg- Marquardt algorithm to fit Eq. (4). The Leven- perg-Marquardt algorithm solves least-squares curve fit-
6

---

Scaling Inference-Efficien Language Models
MSE: 0.0033, R²: 0.9895
MSE: 0.0006, R²: 0.9982
Predicted Loss
5.0
3.00
Predicted Loss
X060
4.5
2.75
4.5
2.8
3,0
28
3.0
4.0
4.0-
Chinchilia
0.82
0.87
0.82
0.62
0.62
-0.40
3.5
3.5
3.0
Inference
0.98
1.00
3.0
3.5
4.0
4.5
5.0
3.0
3.5
4.0
4.5
5.0
80M 116M
164M
237M
313M
18
Actual Loss
Actual Loss
(a) Chinchilla
(b) Inference-Efttcient Efficient
(c) Spearman Correlation
Figure Comparison: (Left) illustrate the predicted versus actual loss using Eq. (2). (Center) display comparison of predicted to actual loss based on Eq. (4). Dots represent data points used for curve-fitting, while cross marks represent test data points. (Right) We demonstrate that our inference-efficient scaling law yields significantly higher Spearman correlation, resulting in more precise predictions of the optimal model configuration.
training setting.
Table 2. Efffciece-Efficcentt Models: this table, we compare the results of Morph-IB IB variants against other open pretrained mod- els of similar size. The evaluation of large language models such Open-LM-II MM (Gururangan et al., 2023), OPT-1.3B (Zhang et al., 2022), Pythia-1.33 1.3B (Biderman al., 2023), Neox- 1.3B (Black et al., 2022) and OPT-IML-1.3 (Iyer et al., 2022) is summarized from (Gururangan et al., 2023).
Then, we train larger models more tokens to evaluate the predictive power of our inference-efficient scaling law. We present the results in Figure 8. Figure 8 demonstrates that our scaling law achieves higher accuracy than the Chin- chilla scaling law, as shown by a smaller MSE and a larger R2 (Wright, 1921) value. We reduce MSE from 0.0033 to 0.0006 while improving R² from 0.9895 to 0.9982. In addi- tion, the relative prediction error for the inference-efficien scaling law is less than 1.2%, whereas for the Chinchilla scaling laws, it ranges from 2.7% to 4.1%. This demon- strates that the inference-cfficien scaling law predicts more accurately than the Chinchilla scaling law.
Models
dmodeel
Rlayers
Avg.
Latency (s)
Open-LM-IB
2048
24
0.49
3.61
OPT-1.3B
2048
24
0.50
2.55
Pythia- 1.3B
2048
22
0.49
3.28
Neox- 1.3B
2048
24
0.49
3.99
OPT-IML-1.3B
2048
24
0.54
2.54
Furthermore, as illustrated in Figure 6, prioritizing the rank- ing of predicted loss more critical than its absolute value when employing the training methodology described in $2.3 for inference-effeee -errencceffficientt models. We calculate Spearman's 's rank correlation coefficient (Spearman, 1961) for both the Chinchilla scaling law and the inference-efffcient scaling law when predicting the loss shown Figure 8(c). The results indicate that our inference- efficient law is effective in ranking different model configurations. For example, the inference-efficient scaling law shows a Spearman correlation of 1.00 for the IB model loss prediction, in contrast to Chinchilla's s -0.40. In Ap- pendix A, we include more details on model configurations.
Morph-1B-y
2048
24
0.52
3.61
Morph-1B-v.
2560
16
0.52
2.57
Morph-1B
3072
12
0.52
1.96
models. The results are
measured inference latency and predicted loss, and train candidate models with the same training dataset. Finally, evaluate the trained models over 20 downstream tasks and we outline the results in Figure 1 and Table 2.
As baseline, the architecture of Morph-1B-y 1B-v1 is identi- cal to that of Open-LM- IB. The superior performance of Morph- 1 B-v1 over Open-LM-1B can be attributed to the higher quality DCLM-Baseline dataset (dcl, 2024). Addi- tionally, OPT-IML-1.3 outperforms Morph-1B- 1B-v1 since it undergoes pre-training on more unique tokens (180B 30B) followed by a fine-tuning stage (lyer et al., 2022). Next, we train Morph- IB and Morph-1B-v which are de- rived from Morph- IB-v1 by modifying the aspect ratio. We use the same 30B tokens to train Morph- 1B, Morph-1B-vi and Morph- Morph-1B-v2. 1B-v2. latency for Morph- BBvvI is 1.8x lower compared Morph-
4.2. Inference-Efficiee Models
Guided by the accurate inference-efficien scaling law, we employ the predict, rank, and select method outlined in Fig- ure to train inference-efffcient models. First, we generate range of variants from the Open-LM-1B model (Guru- rangan et al., 2023) by adjusting the aspect ratio. Then, we measure the inference latency of model variants on single A 1100 GPU. Next, we select 3 models based the
As illustrated in Table 2, the inference

---

Scaling Inference-Efficien Language Models
MSE: 0.9825, R²: -2.1259
MSE: 0.1165, R²: 0.6293
Predicted Loss
5
Predicted Loss
4-
4.5
4.0
3000
Chinchilia
0.95
0.87
0.94
0.90
0.94
0.40
3-
Inference
1.00
3.0
3.5
4.0
4.5
5.0
3.0
3.5
4.0
4.5
5.0
80M
116M
164M 237M
313M
18
Actual Loss
Actual Loss
(a) Chinchilla
(b) Inference-Efficient Efficient
(c) Spearman Correlation
Figure Excluding Over-training Data: We avoid using over-training data to the scaling laws. (Left) The figure is plotted by using Eq. (2). (Center) the center figure is created with Eq. (4). (Right) We plot the Spearman correlation of our scaling law versus the Chinchilla scaling law. The results indicate additional training data can enhance the precision of scaling laws.
MSE: 0.0198, R²: 0.9369
MSE: 0.0008, R²: 0.9973
3.0
Predicted LoS
Predicted Loss
X068
4.5
4.8
3.0
4.0
Chinchilla
0.82
0.87
0.82
0.62
0.81
-0.40
3.5
3.5
3.0
0.97
0.98
1.00
1.00
1.00
1.00
2.5
3.0
3.5
4.0
4.5 5.0
3.0
3.5
4.0
4.5
5.0
80M 116M 164M
237M
313M
Actual Loss
Actual Loss
Model Size
(a) Chinchilla
(b) Cesrrence-Efficiene
Spearman Correlation
Figure 10. Random Choice of Model Shape: We randomly select the model shape fit the scaling laws. (Left) The figure is plotted by using Eq. (2). (Center) The center figure is created with Eq. (4). (Right) We plot the Spearman correlation our scaling law versus the Chinchilla scaling law. The results show that inference-efficient scaling laws robust than Chinchilla scaling laws.
1B, without any loss in accuracy.
However, we note that both scaling laws' performance de- teriorates when applied to predicting losses in over-trained models. Therefore, data from over-training is essential to fit our inference-aware scaling law.
4.3. Insights from Scaling Laws Fitting
Scaling laws provide a cheap and accurate way to predict language model performance at larger scales. However, a drawback of building scaling laws is the requirement train models at various scales. In this section, study how make scaling laws robust and data-efficient.
Select Model Shape Randomly. In this ablation study, we explore the robustness of our scaling laws via fitting mod- cls with random model architecture configurations. this setting, the model architecture configuration for each size is chosen randomly. We randomly select configuration from model configuration pools (The complete list of candidate configurations can be found in Table 4 in Ap- pendix). Figure 10 shows the experiment results. Compared to Chinchilla scaling laws, our inference-efficien scaling laws exhibit greater robustness with much smaller MSE (0.0008 vs 0.0198) and higher R²2 value (0.9973 vs 0. 0.9369). We then use these two laws to predict the loss of IB mod- els. The results show that the relative prediction error for the inference-efficien scaling law is less than 0.72%, sig- nificantly lower than the Chinchilla scaling law's relative prediction error, which ranges from 11.8% to 13.4%. Fi- nally, by using only six data points to fit the two scaling
Exclude Over-training Data. In this ablation study, we fit the scaling law based entirely on the Chinchilla-optimal setup, using only data points where training tokens are set to be Chinchilla-ppimmal. We vary the model size N {80, 116, 164, 237, 313)M and set the number of train- ing tokens D= 20N, excluding data from N 80M and run
D 160N. Table 1 shows the configurations and the results are shown in Figure 9. Compared to Figure 8, we observe that the inference-efficien scaling law more robust than the Chinchilla scaling law. We achieve much lower MSE of 0. 165 compared to Chinchilla's 0.9825 and an R² score of 0,6293 compared Chinchilla's -2.1259.
8

---

Scaling Inference-Efficient Language Models
laws, we significantly reduce the training costs associated with developing these laws. The GPU hours for fitting have been reduced from 450 to 85 A100 GPU hours.
throughput and latency by using Attention A recent study introduces FlashInfer et 2025), which employs block-sparse and composable formats to tackle KV cache storage heterogeneity.
5. Related Work
Compute- Model Design. Previous research has explored the trade-off's of various model configurations in Vision Transformers (ViTs) (Alabdulmohsin al., 2023). Additionally, (Tay et al., 2021) demonstrates that training deep and narrow models can be particularly beneficial when computational resources are limited. More recently, several efficient attention mechanisms (Xiao et al., 2023; Gao et al., 2024; Jiang et al., 2024; Xiao et al., 2024; Yuan et al., 2025) have been introduced to enhance inference efficiency by modifying the attention block.
Large Language Models. Transformer (Vaswani, 2017) has been successfully applied to a variety of tasks: text classification (Wang, 2018; Sarlin et al., 2020), genera- tion (Zellers et al., 2019; Sakaguchi et al., 2021), reason- ing (Srivastava et al., 2022), and mathematics (Cobbe et al., 2021; Hendrycks et al., 2021), showcasing their broad ap- plicability and effectiveness. The development of the GPT models (Brown et al., 2020) demonstrates that increasing the scale of language models significantly enhances their performance across various downstream tasks. The success of the GPT models has inspired the subsequent develop- ment of many large language models, including but not limited to LLaMA (Touvron et al., 2023a;b), Gemma (Team et al., 2024a;b), Qwen (Bai et al., 2023; Yang et al., 2024), and DeepSeck (Liu et al., b024a;b; Guo et al., 2025), each designed to push the boundaries of language modeling.
6. Limitations and Future Work
Although there has been notable progress by our team, sev- eral unresolved challenges open up promising prospects for further study. First, due to resource limitations, we are unable to scale our training to include 7B models. Sec- ond, recently developed inference systems (Ye al., 2025) can enhance inference efficiency and create new trade- offss between inference efficiency and model performance. Furthermore, Attention modules like Multi-Query Atten- tion (MQA) (Shazeer, 2019), Grouped-Query Attention (GQA) (Ainslie et al., 2023) and Multi-Head Latent At- tention (MLA) (Liu et al., 2024a) might also influence loss and inference latency. Our work provides a flexible way to quantify and predict how these architectural optimizations affect the accuracy-efficient tradeoffs. We hope this work opens up new line of research that takes inference effi- ciency as an essential factor in designing language models.
Scaling Laws. Scaling laws are powerful predictors for how large language models behave as parameters increase (Ka- plan et al., 2020). Plenty of subsequent works have con- tributed to the development of scaling laws (Hoffmann et al., 2022; Muennighoff et al., 2023; Sardana et al., 2023; Tao et al., 2024; Kumar et al., 2024; Gadre et al., 2024; Ruan
et al., 2024; Abnar et al.. 2025; Krajewski et al., 2024). In particular, Chinchilla scaling law (Hoffmann et al., 2022) op- timizes a fixed computing budget allocation by balancing the number of model parameters against the number of training tokens to minimize the training loss. Data- Cata-Constrained scal- ing law (Muennighoff et al., 2023) extends the Chinchilla scaling laws by considering repeated data. The scaling laws presented in (Gadre et al., 2024) not only predict training loss under over-training scenarios but also connect training loss downstream error. Beyond Chinchilla- -attiiall-Opimall (Sar- dana et al., 2023) attempted to account for inference cost in their scaling law. However, unlike training tokens, the number of inference tokens cannot be measured in advance.
7. Conclusion
In this work, we perform an extensive empirical study to develop scaling laws that guide in designing inference- efficient model architecture. We first demonstrate that model architecture impacts inference efficiency and that existing scaling laws do not account for inference costs. To jointly optimize inference cost and model loss, we propose inference-efficient sealing laws. We conduct count num- ber, each point is number experiments to fit and evaluate the inference-efficien scaling laws. To tackle the dispar- ity between model loss and downstream task performance, we have developed novel methodology to train and rank inffrrnceeefficiern models using our scaling law. Finally, we design and train Morph- 1B model by leveraging inference- efficient scaling law, which enhances inference efficiency while maintaining accuracy in downstream tasks, compared to similar-sized open-sourced models.
Inference Serving Systems. Inference cost has drawn sig- nificant attention in recent years. Many inference systems and algorithms have been developed to speed up model serving (Olston et al., 2017; Gujarati et al., 2020; Gug- ger et al., 2022; Yu et al., 2022; Leviathan et al., 2023; Kwon et al., 2023; Zheng et al., 2023; Agarwal et al., 2024a;b; Ye et al., 2025; MLC team, 2023-2025). Specifi- cally, Orca (Yu et al., 2022) utilizes continuous batching to achieve higher inference throughput. vLLM (Kwon al., 2023) improves the throughput of popular LLMs by using PagedAttentton manage the KV cache memory. Further- more, SGLang (Zheng et al., 2023) improves the inference
9

---

Scaling Inference-Efficien Language Models
Acknowledgements
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y.,
Lebrón, E., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.13225, 2023.
We gratefully acknowledge the support of the NSF Diamond project OAC-2311767 (Democratizing Large Neural Net- work Model Training for Science). We thank Zhao Zhang from Rutgers University for providing access to comput- ing resources and Vaishaal Shankar for assisting in using the DCLM dataset. The authors acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing computational resources that have contributed to the research results reported within this paper. This research also used computational resources from the NSF Cloudlab (Duplyakin et al., 2019) facility.
Alabdulmohsin, I. M., Zhai, X., Kolesnikov, A., and Beyer, L. Getting vit in shape: Scaling laws for compute-optimal model design. Advances in Neural Information Process- ing Systems, 36: 16446660611455, 2023.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W. Han, Y., Huang, F,, et al. Qwen technical report. arXiv preprint arXiv:2309.
2023.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Impact Statement
This paper presents work that aims to advance the field of Machine Learning. Our work aims to train more inference- efficient language models, potentially reducing the deploy- ment cost of these models and their associated environmen- tal impacts.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
References
Jeopardy. https::/hugginggfaet http:/aawa...ccccaaaatacteeaaeee dataarddyddaaastts///eeeaaa
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L.,, He, H., Leahy, C., McDonell, K., Phang, J., et al. Gpt-neox-20hh An open-source autoregressive language model. arXiv preprint arXiv:2204.06745, 2022.
Phi-2. https: http:://hugiinffac..cooococooos
ph
dclm baseline huggingface. https://hugging
co/datasets https////foonaaonn//aiaaaaaa
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787 2024.
delm dclm-baseline 2024.
Llm foundry. https: /ttp//ggttttttt/e////e www 1lm-foundry,22 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Abdin, M., Aneja, J., Behl, H., Bubeck, S., Eldan, R., Gunasekar, S., Harrison, M., Hewett, R. J., Javaheripi, M., Kauffmaan, P., et al. Phi-4 technical report. arXiv preprint arXiv:2412.08900 2024.
Abnar, S., Shah, H., Busbridge, D., Ali, A. M. E., Susskind, J. and Thilak, V. Parameters vs flops: Scaling laws for optimal sparsity for mixture-of-experts language models. arXiv preprint arXhv:2501. 2025.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1rXhv:1955.10044 2019.
Agarwal, S., Acun, B., Hosmer, B., Elhoushi, M., Lee, Y., Venkataraman, S., Papailiopoulos, D., and Wu, C.-J. Chai: Clustered head attention for efficient Ilm inference, arXiv preprint arXiv:2403.08055, 2024a.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint /00.::/800035555 2018.
Agarwal, S., Mao, A., Akella, A., and Venkataraman, S. Symphony: Improving memory management for Ilm in- ference workloads. arXiv preprint arXiv:2412.16434, 2024b.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. /4/68, 2021.
10

---

Scaling Inference-Efficien Language Models
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783. 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint 0rXiv2200.033300 2020.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. arXiv 2021.
Duplyakin, D., Ricci, R., Maricq, A., Wong, G., Duerig, J., Eide, E., Stoller, L.,, Hibler, M., Johnson, D., Webb, K., Akella, A., Wang, K., Ricart, G., Landweber, L.,, Elliott, C., Zinkin, M., Cecchet, E.,, Kar, S., and Mishra, P. The design and operation of CloudLab. In Proceedings of the USENIX Anmual Technical Conference (ATC), pp. 1-14, July 2019. URL. www.thtpp//.o......m htwp:::/wwww..// flux, atttaheddu// paper/ http/dppplalainni atc19.
preprint arXiv:2
Hoffmann, J.. Borgeaud. S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. I.., Hendricks, I.. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 2022.
Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08550, 2024.
Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., et al. Minicpm: Unveiling the potential of small language models with scalable train- ing strategies. arXiv preprint arXiv:2404.0633 2024.
Tyer, S., Lin, X. V., Pasunuru, R., Mihaylov, T. Simig, D., Yu, P., Shuster, K., Wang. T., Liu, Q., Koura, P. S., et al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212. /2017, 2022.
Gao, Y., Zeng, Z., Du, D., Cao, S., Zhou, P., Qi, J., Lai, J., So, H. K.-H., Cao, T., Yang, F., et al. Seerattention: Learning intrinsic sparse attention in your Ilms, arXiv preprint 1xXi::2400113270 2024.
Guan, X., Zhang, L. L., Liu, Y., Shang, N., Sun, Y., Zhu, Y., Yang, F., and Yang. M. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501. N.0552501.04519, 2025.
Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., Y., et Minference 1.0: Accelerating pre-filling for long-context ilms via dynamic sparse attention. arXiv preprint arXiv:2407.0244 2024.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models.
Gugger, S., Debut, L.., Wolf, T., Schmid, P., Mueller, Z. Mangrulkar, S., Sun, M., and Bossan, B. Ac- celerate: Training and inference at scale made sim- ple, efficient and adaptable. https:/ http///gttut..t///////t///ttthht thub. com/ huggingface acceleeeaeeeeeeeeee 2022.
arXiv preprint arXiv:2001 N0NVv:2000.08361,, 2020.
Krajewski, J., Ludziejewski, J., Adamczewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., Odrzygóżdž, T., Sankowski, P., et al. Scaling laws for fine-grained mixture of experts. arXiv preprint wrXiv:2402.078711 2024.
Gujarati, A., Karimi, R., Alzayat, S., Hao, W., Kaufmann, A., Vigfusson, Y., and Mace, Serving (DNNs} like clockwork: Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 443 2020.
Kumar, T., Ankner, Z., Spector, B. F., Bordelon, B., Muen- nighoff, N., Paul, M., Pehlevan, C., Ré, C., and Raghu- nathan, A. Scaling laws for precision. arXiv preprint arXiv: 241.033,, 2024.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma. S., Wang, P.. Bi, X., al. Deepseck-rl: In- centivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint 2501::250111944 2025.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, pp. 611-626. 2023.
Gururangan, S., Wortsman, M., Gadre, S. Y., Dave, A. Kilian, M., Shi, W., Mereat, J., Smyrnis, G., Ilharco, G., Jordan, M., Heckel, R., Dimakis, A., Farhadi, A., Shankar, V., and Schmidt, L. open- Im: a minimal but performative language modeling (lm) repository, 2023. URL https: ////thutbbb/////////// com/ ml foundat ins open_1m/. 1m/ GitHub repository.
Levesque, H., Davis, E., and Morgenstern, L. The winograd schema challenge. In Thirteenth international confer- ence on the principles of knowledge representation and reasoning, 2012.

---

Scaling Inference-Efficien Language Models
Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In Inter- national Conference on Machine Learning, pp. 19274- 19286. PMLR, 2023.
Ruan, Y., Maddison, C. J., and Hashimoto, T. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405. 10938, 2024.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106, 2021.
Ji,J. Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S., Bansal, H., Guha, E., Keh, S., Arora, K., ct al. Datacomp- Im: In search of the next generation of training sets for lan- guage models. arXiv preprint arXiv:2406. 1794, 2024.
Sardana, N., Portes, J., Doubov, S., and Frankle, J. Beyond chinchilla-optinee Accounting for inference in language model scaling laws, arXiv preprint SKKi:22001.00444 2023.
Liu, A., Feng, B., Wang, B., Wang. B., Liu, B., Zhao, C.,, Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-expert. language model. arXiv preprint SEt1:2405.04434 2024a.
Sarlin, P.-E., DeTone, D., Malisiewicz, T., and Rabinovich, Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 4938 4947, 2020.
Liu, A., Feng, B., Xue, B., Wang. B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint 2.i 2024b.
Shazeer, N. Fast transformer decoding: One write-head all you need. arXiv preprint arXiv:1911.021:: 2019.
Luo, I.,, Liu, Y., Liu, R., Phatale, S., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng, L., Sun, J., et al. Improve mathemati- cal reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06554 2024.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than seal- ing model parameters. arXiv preprint arXiv:2408.03:: 033114 2024.
MLC team. MLC-LLM, 2023-2025. URL https:// github. www..////occaammm.....eeeo
Spearman, C. The proof and measurement of association between two things. 1961.
Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, A. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-50376, 2023.
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint (iXi:2200600415555 2022.
Olston, C., Fiedel, N., Gorovoy, K., Harmsen, J., Lao, L., Li, F., Rajashekhar, V., Ramesh, S., and Soyke, J. Tensorflow- serving: Flexible, perf-performance ml serving. arXiv preprint arXiv: 2017.
Tao, C.,, Liu, Q., Dou, L.., Muennighoff, N., Wan, Z., Luo, P., Lin, M., and Wong, N. Scaling laws with vocabulary: Larger models deserve larger vocabularies. arXiv preprint arXiv:2407.1360 2024.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q N., Bernrrdi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint 1rXiv:1606.06000 2016.
Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., and Metzler, D. Scale efficiently: Insights from pre- training and fine-tuning transformers. arXiv preprint arXiv:2109 10686, 2021.
Qi, Z., Ma, M., Xu, J., Zhang, L. L., Yang, F,, and Yang, M. Mutual reasoning makes smaller llms stronger problem- solvers. arXiv preprint arXiv:2408.08 06195, 2024.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint N.NN:::243330295 2024a.
Radford, A., Wu, J., Child, R. Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Team, G., Riviere M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00 000080001110 18, 2024b.
Roemmele, M., Bejan, C. A., and Gordon, A. S. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 20 AAAI spring symposium series, 2011.
12

---

Scaling Inference-Efficient Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13397 2023a.
Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z.,
Xie, Z., Wei, Y., Wang, I.., Xiao, Z., et al. Native sparse attention: Hardware-alignee and natively trainable sparse attention. arXiv preprint arXiv:2502.1 1089, 2025.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint 000202300..09288, 2023b.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? arXiv preprint
2019.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0000
Vaswani, Attention all you need. Advances in Neurul Information Processing Systems, 2017.
2022.
Wang, A. Glue: A multi-task benchmark and analysis plat- form for natural language understanding. arXiv preprint WrXiv:180.007400 2018.
Zheng, L.., Yin, L., Xie, Z., Huang, J., Sun, C., Yu, C., Cao, S., Kozyrakis, C., Stoica, Gonzalez, J. E., et al. Efficiently programming large language models using sglang. 2023.
Wolf, T. Huggingface's transformers: State-of-the- art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Wright, S. Correlation and causation. Journal of agricul- tural research, 20(7):557, 1921.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv preprint wxxv:2309.174555 2023.
Xiao, G., Tang, J., Zuo, J., Guo, J., Yang, S., Tang, H., Fu, Y., and Han, S. Duoattention: Efficient long-context Ilm inference with retrieval and streaming heads. arXiv preprint 241iv:2410. 10819, 2024.
Yan, M., Agarwal, S., and Venkataraman, S. Decoding speculative decoding, arXiv preprint arXiv:2402.01528, 2024.
Yang. A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 2024.
Yao, Y. Yu, T,, Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.01800 2024.
Ye, Z., Chen, L., Lai, R., Lin, W., Zhang, Y., Wang. S., Chen, T., Kasikci, B., Grover, V., Krishnamurthy, A., et al. Flash- infer: Efficient and customizable attention engine for Ilm inference serving. arXiv preprint arXiv:2501.0100) 2025.
Yu, G.-L,, Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.- G. Orea: A distributed serving system for (Transformer- Based} generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). pp. 521-538. 2022.
13

---

Scaling Inference-Efficienl Language Models
Table Hyperparameters: We show the hyperparameters used for training in this paper. In addition, the hatch size is the global batch size and the default sequence length is 2048.
Model Size
Warmup
Learning rate
Weight decay
7-loss
Batch size
400M
2000
3e-3
0.033
1e-4
512
IB
5000
3e-3
0.033
le-4
256
Table Model Architectures: We list the architectural configurations of all models trained in this paper. dlundel is the hidden size, Ssiee the intermediate size, Rlayers is the number of layers, and Theads is the number of attention heads.
Model Size
Variant
dmodel
fsize
Taaaee
Tlheads
80M
v1
512
1536
8
8OM
V2
576
1536
5
80M
V3
1792
3
80M
v4
448
1280
13
80M
V5
384
1024
22
86M
vI
576
1536
7
86M
V2
640
1792
4
8
116M
vI
640
1792
10
10
116M
V2
720
2048
6
116M
V3
2304
116M
V4
880
2560
3
116M
V5
560
1536
15
10
116M
V6
480
1280
24
10
126M
vI
720
2048
8
10
126M
V2
800
2304
5
10
164M
vI
768
2048
12
12
164M
V2
864
2304
8
12
164M
v3
960
2560
6
164M
V4
1056
2816
4
12
164M
V5
1152
3072
3
12
178M
v1
864
2304
10
12
178M
v2
960
2560
7
12
237M
v1
896
2560
14
14
237M
V2
1008
2816
10
237M
v3
1120
3072
8
237M
v4
1232
3328
6
14
313M
vl
1024
2816
16
16
313M
v2
1152
3072
12
16
313M
v3
1280
3584
9
16
313M
v4
1408
3840
7
16
339M
vI
1152
3072
14
Morph-Il -B
vI
2048
5632
24
Morph-IP IB
v2
2560
6912
16
Morph-1B
/
3072
8192
12
16
A. Hyperparameters and Model Architectures
We follow the hyperparameters mentioned in (Li et al., 2024; Gadre et al., 2024) with the specific details presented in Table 3. cooldown rate 3e-5 is used all experiments. All models are trained in bfloat16 precision using the AdamW optimizer. The number of parameters computed using sum (p numel () for p model parametera () ). To examine
14

---

Scaling Inference-Efficienl Language Models
how model architecture influences loss metrics and inference performance, vary the model configuraitons. Architectural details provided in Table 4.
B. Results over A30 GPUs
In this section, we first evaluate the inference efficiency of open-source large language models (LLMs), aiming to develop robust scaling law across different hardware. From Figure 11, we observe similarly that both the number of parameters and the model architecture are crucial the inference efficiency of the model. In addition, Figure 12 also demonstrates that inference latency increases linearly with the number of layers, and can reduce inference latency by adjusting model configurations, which aligns with the observations made using the A100 GPU. Furthermore, also evaluate the inference latency of models using various numbers of input and output tokens. Figure 13 demonstrates that the aforementioned conclusion remains valid when the number of input tokens is set to 1024 and the number of output tokens to 128.
18
gemma-2-98
14
MiniCPM-18
Latency (s)
12
10
gemma-2-28
gemma-78
Qwen 5-38
Lama-2-77 2-78 M
Qwenz. 5-5B
6-
gemma-28 2B
Qwen2
Qwen2.5-0.58
10
20
30
40
50
60
Mlayers
Figure 11. Open-Source LLM's Inference Latency: An overview of inference latency in open-source L.Mss. The evaluated models include LLaMA (Touvron al., 2023a), Qwen (Yang al., 2024), Gemma (Team et al., 2024a;b), and MiniCPM (Hu al., 2024). evaluations were performed using the Hugging Face genorate function on single NVIDIA A30 Tensor Core GPU. In default, the number input tokens is 128, and the number output tokens 256.
12
12
12
4096
4
18
Latency (s)
5120
Latency (s)
8
Latency (s)
3B
8
6144
12
78
6-
8192
6-
16
4-
2-
2-
15
25
2000
4000 6000
8000
10000
10a
10a
nlayers
dmode!
dmodel Naayers
(a) Vary layers (Viyen), fix hidden size
(b) Vary hidden size (dmmute), layers
(c) Vary ratio (dmoste (aayers) size
Figure Model Shape Inference Lateney over A30 GPU: (Left) illustrate the correlation between inference latency and the number of layers, with the constant hidden size. (Center) We indicate the relationship between inference latency and hidden size with the number of layers fixed. (Right) show relationship between inference latency and aspect ratio, with number parameters fixed. All results are obtained using the Hugging Face generate function, with input and output token counts set 128 and 256, respectively.
15

---

Scaling Inference-Efficient Language Models
8
4096
18
5
Latency (s)
5120
Latency (s)
Latency (s)
6
38
6144
78
8192
3
16
2
2
5
10
15
20
25
30
0
2000
4000
6000
8000
10000
102
10a
Mlayers
model
moddel Mlayers
(a) Vary layers (Mayens), fix hidden size
(b) Vary hidden size (Amudel), fix layers
(c) Vary ratio (dmote Mayers), fix size N
Figure 13. Model Shape on Inference Latency over A30 GPU with different number of input and output tokens: (Left) We illustrate the correlation between inference latency and the number of layers, with the constant hidden size. (Center) We indicate relationship hetween inference latency and hidden size with the number of layers fixed. (Right) We show the relationship between inference latency aspect ratio, with the number of parameters fixed. results are obtained using Hugging Face generate function, with input and output token counts set at 1024 and 128, respectively.
C. More Results over A100 GPUs
In this section, we evaluate the relationship between model architecture and Time To First Token (TTFT) over a single NVIDIA Ampere 40GB A100 GPU by fixing the total parameter count and varying the hidden size and number of layers. From Figure 14, observe that wider and shallower models consistently achieve lower TTFT.
0.8
0.8
0.8
(1792,
12304,
(3072,
(1920, 28)
(2688, 36)
(3584, 40)
(2048, 24)
[3072, 28)
(4096, 32)
(2304,
(3940,
ITFT
(2560, 16)
(4608, 12)
(8192,
0.0
20
2
29
24
29
29
2
2
2
23
2
2
29
2
2
Batch Size
Batch Size
Batch Size
(a) 1B Model Variants
(b) 3B Model Variants
(c) 7B Model Variants
Figure Model Shape on Time First Token (TTFT); examine the relationship between TTFT model architecture by fixing the total parameter count and varying the hidden size and number of layers. Across different batch sizes, wider and shallower models consistently achieve lower TTFT. Each tuple in the legend represents a model configuration: the first number is the hidden size dusdel. and performed using the Hugging Face generate function single NVIDIA
the second is the number of layers Players- All evaluations Ampere 40GB A100 GPU with input length 128, and output length 1.
16

---

Scaling Inference-Efficient Language Models
D. Results over vLLM
this section, we first evaluate the inference efficiency of open-source large language models over vLLM using NVIDIA Tesla A100 Ampere 40 GB GPU. From Figure 15, we find that the efficiency of model inference is influenced not only by the number of parameters but also by the model's architecture. Additionally, Figure 16 shows that inference latency increases linearly with the number of layers using vLLM framework (Kwon al., 2023). Modifying the model configurations effectively reduces inference latency, consistent with findings from the Hugging Face system (Wolf, 2019).
10
Qwen2.5-14B
Llama 2-13B
Latency (s)
gemma- 2-9B
gemma-78
Lama-2-7B
Qwen2:5-7B
gemma-2B 28
g gemma- -2-2B
Qwen2.5-3B
MiniCPM-1B
Qwen2.5-1.50
MiniCPM-2B
Qwen2.5-0.58
20
30
40
Maayers
Figure 15. Open-Source LLM'' Inference Latency vLLM (Kwon et al., 2023) using A100 GPU: An overview of inference
latency open-source I.I.Ms. The evaluated models include aM (Touvron et al., 2023a), Qwen (Yang et al., 2024), Gemma (Team al.. 2024a;b), and MiniCPM (Hu et al., 2024). All evaluations performed using the Hugging Face generate function single NVIDIA A100 Tensor Core GPU. In default, the number input tokens is 128, and the number of output tokens is 256.
4096
18
5
5120
Latency (s)
Latency (s)
38
6144
4
78
Latency
8192
2
10
15
20
0
2000
4000
6000
8000
10000
10a
10a
Mlayers
dmodel
dmodel Mlayers
(a) Vary layers (Muyess) fix hidden size
(b) Vary hidden size (dmodet), fix layers
(c) Vary ratio (duote Taayrs), fix size N
Figure 16. Model Shape on Inference Latency over vLLM (Kwon et al., 2023) using A A100 GPU: (Left) We illustrate the correlation constant hidden size. (Center) indicate relationship between inference
between inference latency the number of layers, latency and hidden size with the number of layers fixed. (Right) We show the relationship between inference latency and aspect ratio, with the number of parameters fixed. results are obtained using Hugging Face generate function, with input and output token counts set 1 28 and 256, respectively.
17

---

Scaling Inference-Efficient Language Models
E. More Scaling Laws Fits
In Section 4.3, we explore how the random selection model shapes affects Chinchilla scaling laws and inference-efficiet scaling laws. We repeat the experiments three times and we show the remaining results in Figure 17 and Figure 18. We have similar observation from Figure 17 and Figure 18 that inference-efficien scaling laws are more robust than Chinchilla scaling laws.
MSE: 0.0096, R²: 0.9694
MSE: 0.0019, R²: 0.9938
Predicted LO5S
5.0
3.00
2.75
Predicted Loss
5.0
3.0
3000
4.5
4.5
3.0
3.0
4.0
4.0
Chinchilla
0.82
0.87
0.82
0.62
0.81
-0.40
3.5
3.0
3.0
Inference
0.97
0.98
1.00
1.00
1.00
1.00
3.0
3.5
4,0
4.5
5.0
3.0
3.5
4.0
4.5
5.0
80M 116M 164M
237M
313M
1B
Actual Loss
Actual Loss
Model
(a) Chinchilla
(b) Inference- Effiicien
(c) Spearman Correlation
Figure Random Choice Model Shape- Trial randomly select model shape to fit the scaling laws. (Left) figure is plotted by using Eq. (2). (Center) The center figure is created with Eq. (4). (Right) We plot the Spearman correlation of our scaling versus Chinchilla scaling law. The models randomly selected from the fitting are 80M-v3-20N, 116M-v4-20N, 164M-v5-20N, 237M-v2-20N, 313M-x3-2ON, and 80M-v4-160N.
MSE: 0.0091, R²: 0.9711
MSE: 0.0011, R²: 0.9965
Predicted Loss
5.0
Predicted Loss
5.0
3.0
4.5
X04
4.5
2.8
4.0
4.0
Chinchilla 0.81
0.87
0.82
0.62
0.81
-0.40
3.5
3.5
3.0
3.0
Inference
3.0
4.5
5.0
3.5
4.5
80M 116M 164M
237M
313M
1B
Actual Loss
Actual Loss
Model Size
(a) Chinchilla
(b) Inference-Efiiie Efficient
(c) Spearman Correlation
Figure 18. Random Choice of Model Shape Trial 3: We randomly select the model shape to fit the scaling laws. (Left) The figure
plotted by using Eq. (2). (Center) center figure is created with Eq. (4). (Right) We plot Spearman correlation our scaling law versus the Chinchilla scaling law. The models randomly selected from the fitting are 80M-v 11-00, 1 116M-v3-200 164M-v5-200 237M-v4-20 20N, 313M-v4-2ON. and 80M-y4-160N.
18

---

Scaling Inference-Efficient Language Models
F. Evaluation Dataset Details
include the details evaluation datasets in Table and we use LLM-foundry (Ilm, 2024) to do all evaluations in this work.
Table 5. Dataset Details: We use LLM foundry (llm, 2024) to do all evaluations.
Dataset
Category
Evaluation Type
ARC-Challenge (Clark et al., 2018)
world knowledge
multiple choice multiple choice
ARC-Easy (Clark et al., 2018)
world knowledge
BoolQ (Clark et al., 2019)
reading comprehension
multiple choice
CОPA (Roemmele et al., 2011)
commonsense reasoning
multiple choice
HellaSwag (Zellers al., 2019)
language understanding
multiple choice
Jeopardy (Jeo, 2022)
world knowledge
language modeling
LAMBADA (Paperno et al., 2016)
language understanding
language modeling
MMLU (Hendrycks et al., 2020)
world knowledge
multiple choice multiple choice
PIQA (Bisk et al., 2020)
commonsense reasoning
Winograd (Levesque et al., 2012)
language understanding
schema
WinoGrande (Sakaguchi et al., 2021)
language understanding
schema
G. Open-Source Model Configurations
In this section, Table presents model configurations from Hugging Face, highlighting the vast space of architectural design choices.
Table 6. Model Configurations: We present configurations models available on Hugging Face.
Model
dmodel
Mayers
dmodel MMyyrrs
Llama-3.2-11 (Dubey et al., 2024)
2048
16
128
Llama-3.2-3B (Dubey et al., 2024)
3072
28
109.7
Qwen2.5-0.5B (Yang al., 2024)
896
24
37.3
Qwen2.5-1.5B (Yang al., 2024)
1536
28
54.9
Qwen2.5-3B (Yang al., 2024)
2048
36
56.9
Qwen2.5-7B (Yang et al., 2024)
3584
28
128
Qwen2.5-14B (Yang et al., 2024)
5120
48
106.7
gemma-2h (Team et al., 2024a)
2048
18
113.8
gemma-7b (Team al., 2024a)
3072
28
109.7
gemma-2-2b (Team et al., 2024b)
2304
26
88.6
gemma-2-9b (Team et al., 2024b)
3584
42
85.3
gemma-2-27b (Team al., 2024b)
4608
46
100.2
microsoft-phi-2 (Phi, 2023)
2560
32
80
microsoft-phi-4 (Abdin et al., 2024)
5120
40
128
19

---

Scaling Inference-Efficient Language Models
H. Parameter Fits
Table 7. Parameter Fits: Coefficients for the scaling laws presented in Figure 8.
Law
B
Cx
Chinchilla snference-Efficien
7720.62
68572.73
2.13
0.49
/
54754.14
778340.38
2.45
0.61
0.0011
Table 8.
Parameter Fits: Coefficients for the scaling laws presented in Figure 9.
Law
E
Chinchilla
-25287.67
248461 .43
2.14
0.51
/
Inference-Eftfcient
-16247.15
958437.97
2.41
0.60
0.0011
Table Parameter Fits: Coefficients scaling laws presented in Figure 10.
Law
A
B
Chinchilla
793.45
4090.85
1.09
0.35
Inference-Efficient
32515.16
408925.99
2.34
0.58
0.0016
20

---

Scaling Inference-Efficient Language Models
I. Contribution Statement
Song collaborated with Minghao to set up the experimental environment and codebase and design experiments ($3). Song collected all experimental data and developed the sifrence-efficient scaling laws based on Chinchilla scaling laws ($2.2). Song proposed the methodology for training anference-efficiene models ($2.3), conducted all experiments ($4), and prepared all figures in the paper. Ultimately, Song was responsible for writing and editing the paper.
Minghao proposed investigating how model configuration affects LLM scaling in inference efficiency and performance. He collaborated with Song to set up the experimental environment and codebase and design experiments ($3), offered constructive suggestions throughout the project, and was responsible for writing and editing the paper.
Shivaram provided assistance in shuffling DCLM datasets and recommended training 1B model that is more inference- efficient while maintaining accuracy downstream tasks compared with other models ($3). He provided numerous constructive comments throughout the project and helped us polish the entire paper.
21


Reviews:
[{"review_id": "3bW7AF3rGk", "reviewer": "Reviewer_B9RC", "summary": "Traditional scaling laws (like Chinchilla) do not account for model architectures in their modeling of the loss. This paper first highlights that the model architecture (like hidden dim, #layers) affects the downstream loss as well as the latency of the models (also studied in multiple previous works). They propose incorporating model shape (width to depth ratio) in the chinchilla scaling law. They use this scaling law to extract a couple of choices for aspect ratios that might give a good enough (top-k) loss. The inference latency of these choices are then evaluated empirically, and combined with the loss scores/ranks to guide the final choice of model to train.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The authors emphasize the importance of estimating the downstream inference latency of models...\n\n**Q2:** Moreover, inference latency depends on the number of tokens the model generates to answer a question at inference...\n\n**Q3:** I strongly encourage the authors to define precisely how they are estimating downstream latency.\n\n**A1 & A2, & A3:** We believe these three questions are related and address them together. Firstly, we agree with the reviewer that the end-to-end latency for downstream tasks depends on the number of tokens generated to answer the questions. End-to-end latency can be seen as the time per output token times the number of generated tokens. However, predicting the number of tokens generated to answer downstream tasks is challenging since we do not target any specific downstream task during pre-training. Therefore, we follow the approach used in previous work [1, 2]; we measure the latency of models by fixing the number of input and output tokens and focus on minimizing the time per output token. For example, we fix the input length as 128 and the output length as 256 in Figure 1. As a result, we don't need to model the latency in the scaling law and can instead measure a model's time per output token using an untrained model. Thus, we use empirically measured model latencies in our proposed model selection pipeline.\n\nWe would also like to stress that reducing response length while maintaining answer quality is of independent research interest and orthogonal to our work. In our paper, we follow the previous work [1, 2] and measure the latency of models while fixing the number of input and output tokens. We will make it clearer in the main text in the final version.\n\n**Q4:** The authors incorporate the d/n term in their modified scaling law....\n\n**A4:** Here, we show the d/n term for 15 open-sourced models: https://anonymous.4open.science/r/ICML25-Rebuttal-3B34/open-sourced-models-d_n.png \n\nIn Figure 4, we illustrate the variation in loss as d/n ranges from (2^4) to (2^9) rather than from (2^2) to (2^9). From the above table, we believe that our ranges are reasonable.\n\n**Q5:** This concern is further supported by Table 2. Among the top three Morph-1B model variants trained by the authors, while the aspect ratio varies significantly....\n\n**A5:** Our experimental results indicate that an aspect ratio between 64 and 256 is a practical choice for 1B models. However, as illustrated in Figure 5, the performance of two models on downstream tasks can differ even if their losses are similar. For instance, while the losses of two 164M model variants are recorded as 3.32 and 3.35, their accuracies on the BoolQ dataset differ significantly, with scores of 0.5379 and 0.5734, respectively. Therefore, while the d/n ratio may have a limited impact on model loss under a reasonable range, the performance on downstream tasks varies.\n\n**Q6:** Finally, the prediction accuracy of the trained scaling laws has only been evaluated on models 3× larger than those in the training set....\n\n**A6:** Thanks for your suggestions! We agree with your points. We plan to expand our experiments to include larger models when we obtain more computational resources.\n\nMissing References:\nThe paper overlooks some relevant prior work that studies the impact of model shape on scaling laws:\nScale Efficiently: Insights from Pre-training and Fine-tuning Transformers (Tay et al., 2022) – Discusses how model shape (depth vs. width) impacts downstream performance and questions whether existing scaling laws capture these effects.\nEffect of Model Shape in Vision Tasks (Alabdulmohsin et al., 2023) – Explores how model width and depth interact in Vision Transformers (ViTs) and whether similar trends hold across domains.\n\nThank you for the suggestions. The 'Scale Efficiently' paper explores the influence of model shape on downstream performance, but it ignores the effect on inference efficiency. Regarding the second paper, 'Effect of Model Shape in Vision Tasks,' it focuses on the impact of model shape on Vision Transformers (ViTs), whereas our study examines its impact on Large Language Models (LLMs). We will include these papers in the related work section. \n\n[1] Zhong, Yinmin, et al. \"{DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving.\" OSDI 24.\n\n[2] Yu, Gyeong-In, et al. \"Orca: A distributed serving system for {Transformer-Based} generative models.\" OSDI 22.", "claims_and_evidence": "Upto some extent (See weakness section for details)", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Upto some extent (See weakness section for details)", "relation_to_broader_scientific_literature": "Helps guide choice of model architecture to train for practioners.", "essential_references_not_discussed": "Some very relevant missing references: \nhttps://arxiv.org/pdf/2109.10686 (SCALE EFFICIENTLY: INSIGHTS FROM PRE-TRAINING AND FINE-TUNING TRANSFORMERS) that talks about impact of model shape on downstream loss\nhttps://arxiv.org/pdf/2305.13035 (effect of model shape in vision tasks)", "other_strengths_and_weaknesses": "### Strength:\n- I really liked authors using their scaling law observations to come up with the most efficient (in terms of latency) 1B model. \n\n\n### Weaknesses:\n\n- The authors emphasize the importance of estimating the downstream inference latency of models. However, they still propose to estimate model latency using an untrained model. In their final model selection algorithm, the latency scores and the predicted loss are combined to select k candidates for training. It would have been much more interesting if the scaling law could be formulated to directly predict a combined metric of inference latency and loss, rather than treating them separately. Can we not estimate inference latency based on d/n? Wondering what were the reasons behind author's choice of the specific modeling they present in the paper.\n\n- Moreover, inference latency depends on the number of tokens the model generates to answer a question at inference. More accurate models might be able to answer the same question in fewer tokens, through higher-level reasoning or by reducing unnecessary attempts. Given these factors, predicting inference latency solely based on architecture (using an untrained model), as the authors propose, seems problematic.\n\n- I strongly encourage the authors to precisely define how they are estimating downstream latency.\n\n- The authors incorporate the d/n term in their modified scaling law. While Figure 7 shows that this scaling law better estimates loss for the models they trained, my main concern is: how much does the d/n term vary in real-world models? In Figure 4, they show loss variation as d/n ranges from \\(2^2\\) to \\(2^9\\), but does it vary this much in actual models that practitioners train (e.g., Gemma, LLaMA, Qwen series)? I encourage the authors to demonstrate how much the d/n term (the ratio R) varies across existing open-source models and how accurately one could use it to estimate their loss. They could, for instance, use a best-fit approximation for a held-out dataset to estimate the loss (perplexity) of these models or, at the very least, highlight the actual range of variation of R across these models.\n\n- This concern is further supported by Table 2. Among the top three Morph-1B model variants trained by the authors, while the aspect ratio varies significantly, the downstream performance remains nearly the same. This suggests that for any **reasonable range of choice** of aspect ratio, d/n may not be that critical in modeling loss through scaling laws.\n\n- Finally, the prediction accuracy of the trained scaling laws has only been evaluated on models 3× larger than those in the training set. In general, one would want to test at least an order of magnitude larger models to validate the law’s extrapolation capabilities. However, I understand  this might stem out of computational constraints, and I will not weigh this limitation heavily in my final assessment.\n\n### Missing References:\nThe paper overlooks some relevant prior work that studies the impact of model shape on scaling laws:\n- [Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers (Tay et al., 2022)](https://arxiv.org/pdf/2109.10686) – Discusses how model shape (depth vs. width) impacts downstream performance and questions whether existing scaling laws capture these effects.\n- [Effect of Model Shape in Vision Tasks (Alabdulmohsin et al., 2023)](https://arxiv.org/pdf/2305.13035#:~:text=Unfortunately%2C%20in%20both%20,up) – Explores how model width and depth interact in Vision Transformers (ViTs) and whether similar trends hold across domains.", "other_comments_or_suggestions": "See the weakness section", "questions_for_authors": "See the weakness section", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gUHr8Bi4Zj", "reviewer": "Reviewer_zGHh", "summary": "The paper presents revised inference-time scaling laws based on model architecture choices, relying on the observation that models of the same size but different architecture choices can have up to a 3.5 times difference in inference latency. Using that, they train models of varying sizes up to 1B parameters, improving inference latency by a factor of 1.8 while keeping the same accuracy on downstream tasks as that of open-source models of the same size.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Table 1 seems to only rely on 6 data points, how statistically valid is extrapolating to larger sizes in this case?\n\n**A1:** As detailed in Table 4 of the Appendix, each model size includes several variants. We use 27 data points to fit the scaling laws in Figure 7.\n\n**Q2:** The Spearman correlation analysis (Figure 7c) strongly supports the scaling law’s ranking capability. Could you explain the negative correlation for Chinchilla’s law?\n\n**A2:** The Spearman correlation ranges from -1 to 1. In Figure 7, the actual rank of 4 models are [1, 2, 4, 3], however, the predicted rank of Chinchilla’s law is [4, 2, 3, 1]. Therefore, the value of the Spearman correlation analysis is negative. \n\n**Q3:** The modified scaling law in Eq. 4 introduces a $(1 + \\epsilon R^\\gamma)$ term to the Chinchilla loss function. While it works empirically, is there any theoretical rationale behind that?\n\n**A3:** Similar to prior work [1] on scaling laws, our work is guided by the trend of losses observed across various model variants.\n\n[1] Gadre, Samir Yitzhak, et al. \"Language models scale reliably with over-training and on downstream tasks.\" arXiv preprint arXiv:2403.08540 (2024).", "claims_and_evidence": "Claims are supported by extensive experiments and ablations.", "methods_and_evaluation_criteria": "Authors use standard evaluations, benchmark datasets and baselines which are sound.", "theoretical_claims": "Nothing to discuss in particular.", "experimental_designs_or_analyses": "Experimental settings are methodologically sound, experiments and ablations are informative and extensive. See questions for aspects to clarify.", "relation_to_broader_scientific_literature": "The paper is relevant as a training recipe for future generations of LLMs and shows a novel finding relating inference latency with architecture design choices, which motivates further work in this direction.", "essential_references_not_discussed": "None to be discussed to the best of my knowledge.", "other_strengths_and_weaknesses": "None, all submitted either as part of the paper evaluation or as questions.", "other_comments_or_suggestions": "Nothing particularly important to mention.", "questions_for_authors": "- Table 1 seems to only rely on 6 data points, how statistically valid is extrapolating to larger sizes in this case?\n- The Spearman correlation analysis (Figure 7c) strongly supports the scaling law’s ranking capability, could you explain the negative correlation for Chinchilla’s law?\n- The modified scaling law in Eq. 4 introduces a $(1 + \\epsilon R^\\gamma)$ term to the Chinchilla loss function, while it works empirically, is there any theoretical rationale behind that?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XWByKR8bdN", "reviewer": "Reviewer_4YJ4", "summary": "The paper observes that architecture modifications significantly affect inference latency whilst holding total model size fixed. The paper primarily uses model aspect ratio $r=d_{model}/n_{layers}$ as the parameterization of architecture.\n\nThe paper then introduces an inference-efficient scaling law, which is sensitive to $r$ and enables constraint optimization subject to a size, data and latency constraint.\n\nThe resulting process enables systematic improvement of the efficiency-accuracy pareto frontier.\n\n## Update after rebuttal\n\nI am maintaining my score of 5: Strong Accept. I can see this significantly deviates from some reviews presented by other reviewers.\n\nThe reasons I think this paper is well worth accepting are:\n1. The empirical procedure is sound\n2. The results are of practical value, although they are i) hardware and ii) architecture/task-constrained\n\nThe limitation of point 2i) is necessary in order to show a practical use case. A theoretical estimate of latency could also work here, but in general it's not clear how to map this estimate to scenarios of interest.\n\nThe task-constrained limitation is a consequence of essentially all empirical scaling law studies, and the authors should not be penalized for this.\n\nThe outstanding issue on the paper raised by another reviewer is why the specific form of the scaling law works from a first principles perspective. It would be great to have clarification around it, however, for practical utility all that is required for empirical studies is correct boundary conditions, asymptotic behavior, and good empirical fit, each of which the results meet.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** In line 437, you say that your approach \"avoids estimating tokens generated\". I would rephrase this, as it implies your approach is circumventing a problem in earlier work. To me it seems more like there is only a philosophical difference. The current work is concerned with model latency, which is an instantaneous measure, and useful to control. Prior work [3] is concerned with the total carbon footprint of a model during its lifetime. These constraints are both important.\n\n**A1:** Thanks for pointing it out! We agree that the total carbon footprint of a model during its lifetime is also important. We will rephrase the sentence in the final version.\n\n**Q2:** I would not call Equation 4 an \"inference-efficient\" scaling law. To me, it is a scaling law that can estimate cross-entropy and can take into account the model aspect ratio. In this sense, I would say Equation 4 is more like an \"aspect-ratio sensitive/aware\" law. Equation 4 can be used to then induce inference efficiencies, but that follows from downstream analysis and not from Equation 4 itself.\n\n**A2:** Thank you for the feedback. We will improve the name in the final version.\n\n**Q3:** Typo: there is a power of 2 missing in the MSE definition on line 271.\n\n**A3:** We will fix this typo in the final version.\n\n**Q4:** Provide your coefficient fits in the paper, preferably with bootstrap CIs.\n\n**A4:** For Figure 7, https://anonymous.4open.science/r/ICML25-Rebuttal-3B34/Figure_7_coefficient.png; For Figure 8, https://anonymous.4open.science/r/ICML25-Rebuttal-3B34/Figure_8_coefficient.png; For Figure 9, https://anonymous.4open.science/r/ICML25-Rebuttal-3B34/Figure_9_coefficient.png. \n\n**Q5:** What happens if you allow $\\gamma \\neq \\alpha$\n\n**A5:** Thank you for your suggestion! Given that \\gamma is the exponent of R, we often face overflow issues when utilizing scipy.optimize.curve_fit. To circumvent this problem, we can use the bounds parameter within the function. When removing the assumption that $\\gamma = \\alpha$ and applying necessary bounds to avoid overflow, we found that our scaling laws become more accurate compared to existing ones. In Figure 7, the R² will become 0.9985 (from 0.9982), and the MSE will become 0.0005 (from 0.0006). We will update the results in the final version. \n\n**Q6:** Is there a proxy/notion of latency we can use which is hardware agnostic?\n\n**A6:** In this study, we advocate for designing a wider and shallower Transformer model. This approach is based on the understanding that inference in models occurs sequentially, layer by layer, while GPUs can process each layer concurrently. Therefore, a wider and shallower transformer model has smaller inference latency is a hardware-agnostic notion, though the degree would differ based on the specific hardware.\n\n**Q7:** Do you observe any systematic relationship between model aspect ratio and downstream evaluation? (e.g. in [6] it is shown that denser models perform better on reasoning tasks at a fixed number of parameters that sparse MoEs).\n\n**A7:** Thanks for your question! However, we do not observe any systematic relationship between the model aspect ratio and downstream evaluation. We will include this in future work.\n\n[3] Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws https://arxiv.org/abs/2401.00448\n\n[6] Mixture of Parrots: Experts improve memorization more than reasoning https://arxiv.org/abs/2410.19034", "claims_and_evidence": "All primary claims in the paper are supported with clear, convincing explanations and evidence:\n1. Aspect ratio $r$ influences inference latency\n2. The inference-efficient scaling law (Equation 4) that incorporates aspect ratio is a good predictor of model cross-entropy\n3. Following their constrained optimization procedure pushes the inference-accuracy pareto frontier.\n\nThere is a secondary claim which is supported by evidence in the paper, but I feel more further evaluation is required: \"The inference-efficient scaling law is more robust than the Chinchilla scaling law\" (see e.g. line 358).\n\nFirst, \"robust\" is an overloaded term. It's true that the inference-efficient scaling law fits better than the Chinchilla law on the data provided (e.g. Fig 8c and 9c). However, it's unclear to me if this means the law is robust, which usually means with respect to some perturbation? This claim needs to be quantified/made explicit, e.g. using a bootstrap approach.\n\nIf instead the paper means to convey the law is more accurate, the data from the Chinchilla study is available, see [1]. As a test of the inference efficient scaling law, it would be useful to understand if it is a better fit on the Chinchilla data than the Chinchilla law (the aspect ratios of the models in Chinchilla are provided in the Chinchilla Appendix [2]). Such an observation/confirmation would be extremely useful to the community, beyond the context of targeting inference latency.\n\n[1] Chinchilla Scaling: A replication attempt https://arxiv.org/abs/2404.10102\n\n[2] Training Compute-Optimal Large Language Models https://arxiv.org/abs/2203.15556", "methods_and_evaluation_criteria": "Yes, the datasets and benchmarks chosen are all sensible and relevant.", "theoretical_claims": "No theoretical claims are made in the work.", "experimental_designs_or_analyses": "I checked the soundness and validity of all experimental designs. The paper makes sensible choices throughout.\n\nHowever I will make comments about two aspects of the investigation.\n\n### Excluding over-training data\n\nIn Figure 8 the paper shows the scaling law fits and extrapolations for training with only Chinchilla-optimal models $D=20N$. From the form of the scaling law \n$$\nL(N,D)=E+AN^{-\\alpha}+BD^{-\\beta}\n$$\nwhen we use Chinchilla optimal models, then\n$L(N)=E+AN^{-\\alpha}+B(20N)^{-\\beta}=E+AN^{-\\alpha}+B^\\prime(N)^{-\\beta}$\nFurther, as you then set $\\alpha=\\beta$ (line 248), the equation becomes\n$L(N)=E+(A+B^\\prime)N^{-\\alpha}=E+A^{\\prime\\prime}N^{-\\alpha}$.\nEven though $A^{\\prime\\prime}$ can be identified through your fitting procedure, $A$ and $B^\\prime$ cannot (infinitely many solutions of numbers summing to $A^{\\prime\\prime}$), and so the independent effects of $N$ and $D$ are lost. I.e. the data does not enable identifying of scaling coefficients (which is what you are finding in practice).\n\nTo circumvent this issue cleanly with a minimal compute budget, IsoFLOP protocols are often used (e.g. [2]).\n\n[2] Training Compute-Optimal Large Language Models https://arxiv.org/abs/2203.15556\n\n### Choice of matched coefficients\n\nIn line 249 the authors set $\\alpha=\\beta=\\gamma$. There is prior evidence $\\alpha=\\beta$, e.g. [2]. It is unclear to be why the choice to also set $\\gamma$ to these values was made. \n\n[2] Training Compute-Optimal Large Language Models https://arxiv.org/abs/2203.15556", "relation_to_broader_scientific_literature": "The work generalizes the constrained optimization training procedures of [2, 3] from compute optimal, to lifecycle compute optimal, to latency optimal within practical constraints.\n\nThe work is complementary to other investigations in architecture modification that can improve latency at a given model capability, for example Sparse MoEs [4, 5].\n\n[2] Training Compute-Optimal Large Language Models https://arxiv.org/abs/2203.15556\n\n[3] Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws https://arxiv.org/abs/2401.00448\n\n[4] Scaling Laws for Fine-Grained Mixture of Experts https://arxiv.org/abs/2402.07871\n\n[5] Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models https://arxiv.org/abs/2501.12370", "essential_references_not_discussed": "No essential references are missing.", "other_strengths_and_weaknesses": "The paper is extremely easy to read, is well-structured, and has well-made figures.", "other_comments_or_suggestions": "1. In line 437, you say that your approach \"avoids estimating tokens generated\". I would rephrase this, as it implies your approach is circumventing a problem in earlier work. To me it seems more like there is only a philosophical difference. The current work is concerned with model latency, which is an instantaneous measure, and useful to control. Prior work [3] is concerned with the total carbon footprint of a model during its lifetime. These constraints are both important.\n2. I would not call Equation 4 an \"inference-efficient\" scaling law. To me, it is a scaling law that can estimate cross-entropy, and can take into account the model aspect ratio. In this sense, I would say Equation 4 is more like an \"aspect-ratio sensitive/aware\" law. Equation 4 can be used to then induce inference efficiencies, but that follows from downstream analysis, and not from Equation 4 itself.\n3. Typo: there is a power of 2 missing in the MSE definition on line 271.\n4. Provide your coefficient fits in the paper, preferably with bootstrap CIs.\n\n[3] Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws https://arxiv.org/abs/2401.00448", "questions_for_authors": "1. What happens if you allow $\\gamma\\neq \\alpha$?\n2. Is there a proxy/notion of latency we can use which is hardware agnostic?\n3. Do you observe any systematic relationship between model aspect ratio and downstream evaluation? (e.g. in [6] it is shown that denser models perform better on reasoning tasks at a fixed number of parameters that sparse MoEs).\n\n[6] Mixture of Parrots: Experts improve memorization more than reasoning https://arxiv.org/abs/2410.19034", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NozEewuwNV", "reviewer": "Reviewer_kVaw", "summary": "This paper proposes to modify the chinchilla scaling laws to also include the model aspect ratio (embedding dim / number of layers) into the scaling law. \nThis accounts for the fact that wider and shallower models are faster in inference. \nAdditionally the paper suggests to include the latency as key metrics for inference into the model selection criteria. \nFinally, the authors train a wider and shallower 1B parameter model which has a lower latency than comparable models.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Claims**\n\n**C1:** This paper only looks...\n\n**Answer:** From Figure 3 in [2], prior work has also observed that models with smaller loss do not mean better performance over downstream tasks. See more results here: https://anonymous.4open.science/r/ICML25-Rebuttal-3B34\n\n**C2:** Just the reference to....\n\n**Answer:** Thanks for your question about the inference system and how it affects latency.  It is true that changing the inference system can change the latencies and we show results from vLLM in Figure 13, Figure 14. However, even then we see some interesting behavior where latencies for say the MiniCPM-1B is 30-40% higher than Qwen2.5-1.5B. We will clarify this point and include an explanation in the final version.\n\n**Experimental Design:**\n\n**E1:** Details on the latency...\n\n**Answer:** We used two inference frameworks in our work: hugging face (Figures 1, 2, 3, 10, 11, and 12) and vLLM (Figures 13 and 14). When using Huggingface we directly called the generate function. When using vLLM, we used torch.compile and torch CUDA graphs.\n\n**E2:** Details on the FLOP calculation...\n\n**Answer:** In this paper, we do not calculate FLOPs; instead, we directly measure the inference latency of models on GPUs. While FLOPs are a valuable metric, they do not accurately represent the real-world latency of a model [3]. Therefore, we focus on measuring the end-to-end inference latency.\n\n**Q1:** Only very small models are investigated....\n\n**A1:** Due to limited computational resources, we cannot afford to train larger models, such as a 7B model with over 140B tokens. We plan to expand our experiments to include larger models (>7B parameters) when we obtain more computational resources.\n\n**Q2:** The paper proposes to include the aspect...\n\n**A2:** We note that Distillation Scaling Laws was submitted to arXiv on Feb 12th, 12 days after the ICML deadline. We will discuss it in the final version.\n\nAs for the aspect ratio, Figure 4 illustrates that with a fixed number of parameters, the loss can increase by up to 5% by varying the aspect ratio. In general, models with higher losses are more likely to exhibit poorer performance on downstream tasks, particularly when the loss increases by 5% or more. In addition, as shown in Figure 3, the model shape plays a crucial role in inference latency. Therefore, we believe we should account for this in the scaling law to co-optimize model loss and inference latency. \n\n**Q3:** Their methodology for selecting...\n\n**A3:** Thank you for the question. We would first like to clarify our pipeline here. In our pipeline, we incorporate inference latency into scaling laws, enabling them to accurately predict the loss ranking of candidate models in comparison to the Chinchilla law. However, due to the gap between loss and accuracy, as illustrated in Figure 5, we believe the effective solution is to evaluate models based on their performance over downstream tasks. Consequently, we choose the top-k models predicted by our scaling laws, pre-train them, and then evaluate their performance on downstream tasks. We then select the best model to release based on inference latency and downstream task performance. We note that both the updated formulation of our scaling law and how to integrate the proposed scaling law into the model selection pipeline are contributions of our work.\n\n**C3 & Q4:** No other architectural changes...\n\n**A4:** We agree with your points. In Section 6, “Limitations and Future Work,” we regard architectural changes such as GQA or MQA as future work.\n\n**Q5:** Just from Equation (4)...\n\n**A5:** As shown in Figure 3, with a fixed number of parameters, the aspect ratio predominantly affects the latency. Therefore, we want to solve the following question in the paper:\n\n$\\arg\\min L(N, D, R) = (E + AN^{−α} + BD^{−β} ) · (1 + εR^{γ})$\n\ns.t. $N \\leq N_{C}$, $D \\leq D_{C}$, $T_{inf} \\leq T_{C}$\n\n**Q6:** Their Morph-1B model...\n\n**A6:** Given that we have a 24-layer base model Morph-1B-v1, we first propose several candidate models with lower inference latency. The configurations we used are (2560, 16), (3072, 12), (3200, 10), (3328, 8), (4096, 6), and (4608, 4), where the first element is the hidden size and the second element is the number of layers. Next, we use the scaling law to predict the loss of each model. Since our scaling laws can predict the rank of model loss more accurately than Chinchilla's, we selected the top 2 candidates (3072, 12) and (2560, 16) to do pre-training. After we obtain the pre-trained models, we evaluate them under downstream tasks and release the best one.\n\n**Q7:** Why did the authors not include \"standard\" scaling law plots...\n\n**A7:** https://anonymous.4open.science/r/ICML25-Rebuttal-3B34\n\n[1] Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv 2023.\n\n[2] Language models scale reliably with over-training and on downstream tasks. arXiv 2024.\n\n[3] Run, don't walk: chasing higher FLOPS for faster neural networks. CVPR 2023.", "claims_and_evidence": "Claims:\n> Models of similar training loss exhibit gaps in downstream evaluation.\n> Also L. 88: “the disparity between model loss and accuracy in downstream tasks”. \n- This statement contradicts findings in other papers ([1], [2]), which find that model loss is a good proxy for aggregate metrics on downstream tasks.  \n- This paper only looks at 3 downstream tasks and find that the have different correlations with the loss. For such general statements more downstream tasks should be considered and then very likely the observation as in [1] and [2] is confirmed.\n\n> L.130 (Figure 2): smaller models can sometimes exhibit higher inference latencies than larger models\n- Just the reference to Figure 2 is not enough. Why should this model be slower than a 14B model. I suspect problems with the Huggingface implementation. Needs more details and explanation\n\n> The authors claim that their scaling law is inference aware.\n- There is only an indirect relation to the inference efficiency: The authors find that wider, more shallow models have a smaller latency.\n\n> The paper asks the question: Given dataset and parameter constraints, can we train an inference-efficient and accurate model for downstream tasks? \n- In my view this question is too general. If I had to train an inference efficient model, I would look for more architecture interventions than just the aspect ratio, e.g. Linear Attention, Hybrids, MLA, etc.\n- Either the paper has to demonstrate that their law (or modification to the law) holds for other attention variants, like Linear Attention, MLA or GQA or limit their scope.\n\n[1] Sardana, Nikhil, et al. \"Beyond chinchilla-optimal: Accounting for inference in language model scaling laws.\" arXiv preprint arXiv:2401.00448 (2023).\n\n[2] Gadre, Samir Yitzhak, et al. \"Language models scale reliably with over-training and on downstream tasks.\" arXiv preprint arXiv:2403.08540 (2024).", "methods_and_evaluation_criteria": "The paper uses the MSE, R^2 value and the spearman correlation to compare the fits of the different scaling laws. I find these metrics suitable for comparing the curve fits. \n\nHowever, I would expect some standard scaling law plots that show the loss over number of FLOPs or number of parameters. \n\nL. 202: It is not obvious that the \"most suitable model shape adjustment is the inclusion of the term $(1 + \\epsilon R^\\gamma)$. In my opinion this needs more discussion and motivation. Are there other possibilities to include the model shape?", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- Details on the latency measurements are missing. Is torch.compile used? Are other optimizations conducted, e.g. torch CUDA graphs ?\n- Details on the FLOP calculation used in this paper are missing. Are embedding FLOPs counted? What about last linear layer (i.e. unembedding) FLOPs?\n\n- I did not find the final results of the fits in their paper, i.e. the values of the coefficients. It would be interesting how much their \"chinchilla\" fit deviates from the original \"chinchilla\" fit. This could help to assess how the papers experimental setup which contains models from 80M to 1B relates to the original \"chinchilla\" setup.\n\n- The paper includes some ablations on the scaling law fits, where the data points for fitting the scaling law are selected differently. \n\n- I would have expected additional ablations on different ways on accounting for the model aspect ratio, as this is the core contribution of this paper.", "relation_to_broader_scientific_literature": "In addition to the number of model parameters, the key contribution of the paper is a new scaling law formulation that incorporates the aspect ratio (embedding dim / num layers) of the model architecture into the scaling law. So while previous scaling law formulations (e.g. Chinchilla) depend on the number of parameters too, they remain unspecific about the allocation of the parameters in the model.\n\nConcurrent works fix this ratio for scaling law experiments [3].\n\n[3] Busbridge, Dan, et al. \"Distillation Scaling Laws.\" arXiv preprint arXiv:2502.08606 (2025).", "essential_references_not_discussed": "A recent work which studies the discrepancies between the Chinchilla Scaling law and the Hoffmann scaling law is relevant [4].\n\n\n[4] Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., & Carmon, Y. (2025). Resolving discrepancies in compute-optimal scaling of language models. Advances in Neural Information Processing Systems, 37, 100535-100570.", "other_strengths_and_weaknesses": "Strengths: \n- The paper studies the important question of how to build inference efficient language models.\n- The paper studies the impact of the aspect ratio embedding dim / num layers on scaling laws.\n- The paper tries to include key metrics for inference such as latency into scaling laws for language models.\n\nWeaknesses:\n- Only very small models are investigated.\n  - It is not clear whether wider but more shallow models also perform that well in very large models (e.g. >7B parameters).\n- The paper proposes to include the aspect ratio (embedding dim / number of layers) into the scaling law. However, from Figure 4 it seems that the loss is relative robust to the choice of this ratio, which suggests to not account for this. \n  - I suggest to include a comment on other concurrent work, that suggests to fix this ratio (See Busbridge, et al., Distillation Scaling Laws, 2025 (http://arxiv.org/abs/2502.08606) \n- Their methodology for selecting inference efficient language models seems to be motivated by the problem that predicting the downstream performance from language modeling loss is challenging. \n  - However, instead of resolving this issue, the proposed method adds another selection step, where another metric (namely the latency) is used to select model candidates. \n  - In my view the contribution of this proposed methodology is limited and not novel, as the \"novel\" method is to measure latency and select the fastest candidates, which is the setup of classical hyperparameter tuning.\n- No other architectural changes than changing the aspect ratio between embedding dim and the number of layers are investigated.\n  - For a paper that claims to scale “inference-efficient language models”, architectural changes such as GQA or MQA cannot be neglected.\n  - At least there should be an attempt to investigate how these methods could be accounted for in the law.\n  - Or it must be demonstrated that these methods have no impact on the scaling law.\n- Just from Equation (4) it seems that only the aspect ratio is accounted for, but the aspect ratio alone does not affect the latency. Therefore, it is unclear how the constraint, that is formulated in equation (3) can be fulfilled.\n- Their Morph-1B model is “just” a wider and shallower attention Transformer model.\n  - It is not clear how the scaling law “guides” this model design (e.g. in number of parameters or number of training tokens). \n  - It could have just been selected from the observation that wider and shallower models are faster on inference.", "other_comments_or_suggestions": "- L. 292, Table 2: Caption: Typo \"of of\".", "questions_for_authors": "- Why did the authors not include \"standard\" scaling law plots in there paper, where the loss is plotted over number of parameters or number of FLOPs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper modifies the training-compute optimal (Chinchilla) scaling laws to take inference costs into accounts. The paper provides extensive experiments to validate the proposed scaling laws for inference-efficient models. 

The paper got very mixed reviews:
- Reviewer B9RC (1: reject) has a number of concerns:
1. Prediction accuracy is only measured by training a 3x larger model. 
2. Has concerns about d/n term, specifically how much it matters and how different it is in practical models. The reviewer points out that the range of d/n in practical models is smaller than that considered for the search by the paper (30-200 vs 16-512), which is true, but the range considered in the paper is somewhat reasonable. 
3. The reviewer has concerns about how the inference cost is measured, and notes that inference latency depends on the number of tokens a model generates as well. The authors measure this in tokens, not taking the number of tokens generated to a query into account, and this is standard practice and reasonable.
- Reviewer (4: accept) has no concerns and raised no major point in either direction.
- Reviewer 4YJ4 (5: strong accept) emphasized that the empirical procedure is sound and that the results are of practical value. The reviewer also emphasizes that the primary claims are supported with clear explanations and evidence. 
- Reviewer kVaw (2: weak reject) notes (like reviewer B9RC) that the aspect ratio (d/n) seems to have a relatively minor effect on performance, and notes that to only characterize the model architecture by d/n is a limitation, in particular since the claim is relatively broad (i.e., the authors talk broadly about inference-efficient models). I agree with the reviewer, and recommend the authors explain early on that they focus on transformers and only vary the ratio d/n. However, taking d/n into account in scaling laws is interesting, even if the finding is that the impact is not huge.

While the paper has the weaknesses mentioned above, I agree with reviewer 4YJ4 (also based on my own reading) that the paper sound and the conclusions are interesting. I agree with some of the limitations raised by reviewer B9RC and kVaw, and think that they can be addressed in the final version by making a bit more narrow and specific claims.
Decision: Accept (poster)