Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models
Authors: Yonggan Fu
Xin Dong
Shizhe Diao
Matthijs Van keirsbilck
Hanrong Ye
Wonmin Byeon
Yashaswi Karnati
Lucas Liebenwein
Maksim Khadkevich
Alexander Keller
Jan Kautz
Yingyan Celine Lin
Pavlo Molchanov
Venue: NeurIPS 2025 poster ()
Keywords: Efficient Language Models
Hybrid Language Models
Primary area: deep_learning
TL;DR: We provide a systematic exploration and roadmap for latency-optimal small language models through optimized architectural and training strategies.

Abstract: Efficient deployment of small language models (SLMs) is essential for numerous real-world applications with stringent latency constraints.While previous work on SLM design has primarily focused on reducing the number of parameters to achieve parameter-optimal SLMs, parameter efficiency does not necessarily translate into proportional real-device speed-ups. 
This work aims to identify the key determinants of SLMs' real-device latency and offer generalizable principles and methodologies for SLM design and training when real-device latency is the primary consideration. 
Specifically, we identify two central architectural factors: depth–width ratios and operator choices. The former is crucial for small-batch-size latency, while the latter affects both latency and large-batch-size throughput. In light of this, we first study latency-optimal depth–width ratios, with the key finding that although deep–thin models generally achieve better accuracy under the same parameter budget, they may not lie on the accuracy–latency trade-off frontier. Next, we explore emerging efficient attention alternatives to evaluate their potential as candidate building operators. Using the identified promising operators, we construct an evolutionary search framework to automatically discover latency-optimal combinations of these operators within hybrid SLMs, thereby advancing the accuracy–latency frontier.
In addition to architectural improvements, we further enhance SLM training using a weight normalization technique that enables more effective weight updates and improves final convergence. This technique can serve as a generalizable component for future SLMs.
Combining these methods, we introduce a new family of hybrid SLMs, called Nemotron-Flash, which significantly advances the accuracy–efficiency frontier of state-of-the-art SLMs, e.g., achieving over +5.5\% average accuracy, 1.3$\times$/1.9$\times$ lower latency, and 18.7$\times$/45.6$\times$ higher throughput compared to Qwen3-1.7B/0.6B, respectively.

Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models
Yonggan Fu'", Xin Dongi Shizhe Diao', Matthijs Van Keirsbilek', Hanrong Ye², Wonmin Byeon', Yashaswi Karnati', Lucas Liebenwein!, Maksim Khadkevich Alexander Keller", Jan Kautzi, Yingyan (Celine) Linft, Pavlo Molchanov NNVIDIA Research, GGorggia Institute of Technology
Abstract
Efficient deployment of small language models (SLMs) essential for numerous real-world applications with stringent latency constraints. While previous work on SLM design has primarily focused on reducing the number of parameters achieve parameter-optimal SLMs, parameter efficiency does not necessarily translate into proportional real-device speed-ups. This work aims to identify the key determinants of SLMs' real-device latency and offer generalizable principles and methodologies for SLM design and training when real-device latency is the central architectural factors:
primary consideration. Specifically, identify depth-width ratios and operator choices. The former is crucial for small-batch size latency, while the latter affects both latency and large-batch-sin throughput. In light of this, we first study latency-optimal depth-width ratios, with the key finding that although deep-thin models generally achieve better accuracy under the same parameter budget, they may not lie on the accuracy-latency trade-off frontier. Next, we explore emerging efficient attention alternatives to evaluate their potential as candidate building operators. Using the identified promising operators, we construct an evolutionary search framework to automatically discover latency- optimal combinations of these operators within hybrid SLMs, thereby advancing the accuracy-lateni frontier. addition to architectural improvements, we further enhance SLM training using weight normalization technique that enables more effective weight updates and improves final convergence. This technique can serve generalizable component for future SLMs. Combining these methods, we introduce new family of hybrid SLMs, called Nemotron- Flash, which significantly frontier of state-of-the-arr SLMs, e.g., achieving
advances the eccuracy-efficiene over +5.5% average accuracy, 1.3x/1.99 lower latency, and 18.7x/45.6. higher throughput compared to Qwwe3-1.7B/0.cl respectively.
Models on Hugging Face: Nemotron-Flash-
3B In-srutt
Introduction
Recent advances in large language models (LLMs) have driven remarkable breakthroughs across a wide range of applications and industries. Despite their impressive capabilities, the substantial computational demands and high latency of LLMs present significant barriers to practical deployment, particularly latency-sensitive scenarios or on resource-constrained hardware. This challenge has intensified the demand for small language models (SLMs).
Most existing SLM designs prioritize parameter reduction achieve efficiency; however, parameter- efficient models do not necessarily yield proportional latency reductions, especially on hardware AI
"Co-first author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Nemetron-Flash-30e
Nemetron-f 1a
Qwen2.5-38
Qwen2.5-38
Average Accuracy (%)
Average Accuracy (%)
5.5
T8
55
50-
Nomotron-Flash.18
Lamm-322-39
50-
(lamm-11.-38
Nomotron-Faachh1
SmolLM2
@wen2 5-0.3
(Q0en225-0550
H20-0shubee-48
(Cwan3-0..68
H2O 120--anu003-41
4lama-3 2-18
4lama-3.2-18
40
40-
AMD-OLMc-18
15
25
30
35
40
45
50
0
1000 2000
3000
4000 5000 6000 7000
B5=1 Latency (sec)
Max Throughput (tokisec)
Figure 1: Visualizing (a) the accuracy-latency trade-off and (b) the accuracy-throughpee trade-off of our Nemotron-Flamd and SOTA SL.ss, where the average accuracy computed across tasks spanning commonsense reasoning, math, coding, and recall tasks. Latency is measured NVIDIA H100 GPU for decoding 8k tokens with batch size of using CUDA Graph. Decoding throughput is measured with 32k-token input length using the maximum batch size that does not cause out-of-memory (OOM) errors for each model. The marker size represents the model depth.
accelerators like GPUs and TPUs. Additionally, current SLM development processes rely heavily on empirical trial-and-error rather than systematic and principled methodologies. For instance, deep- thin architectures -such as those employed in MobileLLMM [ and SmolLM (2) -often result in suboptimal accuracy-latency trade-offs. Furthermore, with the rapid advent of efficient attention operators [3 4) the potential synergies of combining these operators in hybrid models have not been thoroughly explored [5] 6 &8, leading manual, heuristic-driven architecture decisions that are increasingly costly and less scalable.
To address these gaps, this paper introduces generalizable principles and automated methodologies for latency-optimal SLM design and training. We perform a comprehensive study of both architectural choices and training strategies to understand their impact efficiency and accuracy. Based these insights, we propose a series of techniques for constructing latency-optimal SLMSs and analyze their general applicability and effectiveness. These advancements are integrated into a new family of SLMs, which we refer to as Nemotron-Fiin Flash.
Specifically, our architectural exploration focuses selection, where the former is crucial for mmal--batch-size latency and the latter affects both latency and bate-bbtcch-szee throughput. Through extensive training and profiling, show that (1) deep-thin models, while parameter-efficien yield suboptimal latency-accuracy trade-offs, and (2) the optimal depth-width ratio scales with the target latency constraints. Guided by this general principle, also extend existing scaling laws (9] to relate model loss to both depth and width. This allows the sweet-spot depth-width width ratio be determined by profiling range of configurations and selecting the one that meets the latency constraint while minimizing loss, as predicted by the scaling law.
key factors: depth-widhe ratios and operator
Further, comprehensively evaluate emerging attention operators for their rccuracy-latent trade- offs and potential as SLM building blocks. With these findings, we introduce an evolutionary search framework that efficiently identifies optimal combinations of hybrid attention operators. Our search strategy exploits the early stabilization of performance rankings among LM architectures, using short training runs as reliable proxies for final performance, thereby enabling fast and reliable search.
In addition to architectural modifications, based on observations structural patterns in the weight matrices of trained LMs, we further enhance SLM training by constraining weight norms to increase the effective learning rate. This consistently improves final convergence and downstream accuracy across model families. We also employ learnable meta tokens ? for cache initialization.
By combining these architectural and training innovations, we develop the Nemotron-Flas model family, which significantly advances the accuracy latency trade-offs for SLMs. As shown in Fig. [ Nemotron-
Fanotron-FFashh Flash markedly pushes forward the accuracy- art (SOTA) SLMs. For example, with all models accelerated using TensorRT-LLM'ss AutoDeploy kernels 10 and CUDA Graph, Comotron-Flash-33 achieves +2.0% higher average accuracy, 1.7x/1.3x lower latency, and 6.4x/18.7x higher throughput compared to 2wee2.5533//went. respectively. Similarly, Nemotron- Flash- IB achieves +5.5% higher average accuracy, 1.9x lower latency, and 45.6x higher throughput than Qwen3-0.6B.
frontier compared to state-of of-te-of-the- the

---

Accuracy Parameter Trade-off
Accuracy vs, Latency Trade-off
Depch
Dapth
14
6L
24L
CR Accuracy
30L
30L
Parameters (Million)
Decoding Latency (sec.)
(a]
(b)
Figure 2: The accuracy- parioyteaaaattr/ateeeeeyy trade-offs when varying depth and width. While deeper models generally achieve a better accuracy-parameter trade-off, they may not perform as well in the
accuracy-latency trade-off and there exists an optimal depth-width ratio for a latency budget.
2 Towards Latency-Optimal SLM Design and Training
To deliver latency-optimal SLMs, both architecture design and training are critical to achieving the best accuracy-latency trade-off. This work explores both aspects. For SLM design, real-device latency is primarily determined by two key factors: the model's depth and width, and the choice of operators, which are studied in Sec and and 2.2 respectively. For SLM training, we introduce two optimization techniques in Sec 2.3 and Sec 2.4 Finally, we integrate these components construct new model family, Nemotron-Flasn Flash, in Sec and benchmark against SOTA SLMs in See 3.
2.1
SLM Design: Depth- Ratios
Previous SLMs [T find that deep-thin models generally achieve better task accuracy than wide- shallow under the same parameter budget. However, when targeting real-device latency, this may not hold, as partially noted by ([T]. Our key question is: Do deeper or wider models better optimize the accuracy-latency trade-off? To answer this, we provide systematic exploration to understand the impact of depth and width on the accuracy-latency trade-off.
Exploration settings. We train a series of Llama models with five depth settings: 6, 12, 18, 24. and 30 blocks, where each block contains one attention and one feed-forward network (FFN), on 100B tokens from the Smollm-corpus [1]. For each depth setting, we also vary the model width (i.e. hidden size) to create models with different sizes and latencies. We visualize the resulting accuracy-parane
and accuracy-lates latency trade-offss in Fig. 2 (a) and (b), respectively. Accuracy is averaged over eight commonsense reasoning (CR) tasks, and latency is measured as the decoding time for Ik token generation under a batch size of
NVIDIA A100 GPU.
Observations and analysis. We observe that: Deeper models generally achieve a better accuracy- parameter trade-off over wide depth range, although the benefit gradually saturates; @ For the vccuracy-latent trade-off, the advantage of deep-thin models may not hold, and there exists an optimal depth setting for given latency budget. For example, when the latency budget is 3 seconds, depth 12 achieves the best accuracy among the evaluated settings; The optimal depth-width ratio generally increases with the latency budget. These observations highlight the necessity of deliberate depth/widts width selection based on deployment constraints, rather than defaulting to deep-thin models.
Augmented scaling laws for determining sweet-spot depth-width ratios. Although the general trend in the above analysis holds, detailed curves may shift across devices and generation lengths, complicating the selection of model depth and width. As such, in addition to the above insights, we also explore principled methods identify the sweet-spot depth-width ratio within a model family.
In light of this, we augment existing scaling laws 13 9 by parameterizing model loss with model depth and width. Specifically, existing LM scaling laws [3 9 parameterize language modeling loss C(P, N) = Lo + C1 p-a + C2 N-7, with P and N as model size and data size, respectively, and C1, C2, ot, y as fitting parameters. We decouple the model size P into two factors, model depth D and width W, and reformulate the scaling law:
C(D, W, N) Ca aD-a + -W +
(1)
where the fitting parameters a, b. and control the contributions from each dimension, and the exponents a, 3, and govern the diminishing returns from increasing each respective dimension.
3

---

2:35
25.50
RWNKV?
2.30
25.25
GLA
Validation Loss
25.00
2.25
Wiki PPL
24.75
RWKV7
Mamba
2.20
GLA
24.50
SWA512
Mamba2
SWA512
2.15
DeltaNet
MambaZ
24.00
Deltakeet
Gated DeltaNet
Goted DeltaNet
2.10
23.75
20000
40000
60000
80000
100000
16
18
20
22
Training Step
Decoding Latency (s) at 8k (b)
(a)
Figure 4: (a) The validation loss evolution, and (b) the PPL-latency trade-off of different operators.
Since the impact of data size is additive and decoupled from depth and width, we can study the effect of depth and width on LM loss under a fixed data size, i.e., by neglecting the data size term.
As such, in practice, given a target latency budget and deployment setting, the sweet-spot depth- width ratio can be obtained by profiling range of depth-width configurations and selecting the one that meets the latency constraint while achieving the lowest loss.
Fitting and extrapolation. To validate the effective- ness this augmented scaling law, we fit using the above Llama models with varying depth D and width W. Specifically, we use perplexity (PPL) as the loss metric, fit the scaling law on a subset of depth/width set- tings, and validate on models with larger width/depth settings to assess extrapolation. As shown in Fig. [] we find that the model extrapolates reasonably well un- seen depth/width settings, staying within 5.3% of the ground-truth PPI., demonstrating that the fitted function generalizes beyond the observed training configurations.
Train Actual
Val Actual Val Predicted
Degto
Figure 3: Fitting the scaling law and vali- dating on larger depth/width settings.
Lesson learned: SLM depth-width ratios. Deep-thin models may not be latency-optimal and the optimal depth-width ratio generally increases with the target latency budget. Fitting a scaling law with depth and width provides more principled approach to identifying the sweet-spot ratios.
2.2 SLM Design: Hybrid Operators
Beyond model depth and width, the operator used in each layer is another critical dimension. We first train existing LM architectures in a fully controlled setting to identify the most promising operators on the accuracy-latency frontier. We then develop an evolutionary search pipeline to automatically and efficiently discover hybrid combinations of these operators for constructing hybrid SLMs.
Exploration settings. We train a series of 500M LMs built using emerging efficient attention alternatives, including Mamba [14] Mamba2 [15). GLA (66, DeltaNet [17], Gated DeltaNet (18].
RWKV7 19), and sliding window attention (SWA) with a window size of 512. We use the official implementation for Mamba/Mamba2, FlashAttention (20] for SWA, and Tiastiineattttentioon [21] for all other linear attention variants. All models follow the designs specified in their original papers (e.g., one FFN after each attention operator, except for Mamba/Mamba2), use the same number of blocks, and are trained on 100B tokens from the Smollm
We present the validation loss in Fig. 4 (a) and the Wikitext PPL-latency trade-off Fig. 4 (b), measured by decoding 8k tokens with a batch size of on an NVIDIA A100 GPU with CUDA Graph enabled.
In addition, inspired by recent hybrid LMs 6 Mamba/Mamba2 within the same model, we also integrate the promising operators identified in Fig. 4 with Mamba2 or SWA to construct hybrid models in layer-wise interleaved manner, aiming to gain insights into which operator combinations are well-matched and complementary, as shown in Tab. D Note that, for fair comparison, we control the number of blocks in the hybrid models be the same as in the pure models, based on the insights from Sec. 2.1 More details of the ablation settings are provided in Appendix A
22
5
23
which combine attention and

---

Observations and analysis. We observe that:
Table 1: Exploration of operator combinations. All models have 500M parameters and the same depth.
In terms language modeling, DeltaNet and Gated DeltaNet generally emerge as promising candidates, lying on the PPI.-latency Pareto fron- tier; @ When integrated into hybrid models with attention or Mamba2, pairing DeltaNet or Gated DeltaNet with Mamba2 typically results in lower PPL and higher accuracy, consistently outper- forming the corresponding pure models. In con- trast, improvements from pairing with attention are less stable. This indicates both the advan- tages of hybrid models and the importance of se- lecting complementary operator combinations; @ When used in hyhrid models, the performance gap between individual operators may narrow,
Operator1
Operator2
i Wiki PPL.
t CR Acc (%)
Mamba2
24.36
47.72
24.44
48.02
Attention
Mamba2
23.52
48.07
23.87
47.83
DeltaNet
Atie
24.32
47.48
Mamba2
48.03
23.80
47.96
Gated DeltaNet
Atnn
24.31
47.80
Mamba2
23.37
48.03
25.16
46.82
GLA
Attention
24.60
47.63
Mamba2
24.04
47.99
likely due to the complementary and diverse memory mechanisms introduced by hybrid layers. For example, although Gated DeltaNet outperforms DeltaNet in language modeling, their task perfor- mance becomes comparable when integrated with Mamba2, making DeltaNet the preferable operator in hybrid models due to its greater efficiency.
Evolutionary search for operator combinations. The emergence of various efficient attention mechanisms and their complex synergy in hybrid models motivate automated framework to identify their efficient and complementary combination in hybrid SLMs. To achieve this, we built an evolutionary search engine to efficiently navigate complex combinatorial design space.
Short-training PPL as a search proxy. A crucial observation underpinning our method is that the relative performance rankings of different LM architectures stabilize early during training, which can also be observed from the validation loss curves in Fig. (a). Using this insight, we demonstrate that short-training PPL provides reliable proxy metric predict final task performance, substantially reducing training cost for assessing each candidate. We compute the Spearman correlation [24). a measure of rank correlation that is crucial for architecture ranking in our case, between short-training PPL and full- traittrainng PPL across multiple LM architectures. We find an 88.8% Spearman correlation, which is sufficient to identify strong architectures within our search space.
Our search space. Based on the identified promising operator candidates and their synergy in hybrid models, we adopt DeltaNet, Attention, and Mamba2 as candidate operators. We search over maximum of three types of building blocks, each assigned to the early, middle, or late stages of the LM architecture. This three-stage strategy balances operator heterogeneity with architectural regularity. The search explores the ratios of each operator and the number of FFNs per block type, and the repetition count of each block type. More details are provided in Appendix C.I
Evolutionary search algorithm. We adopt the ag- ing evolution search [25] with the following steps: @ Initialization: Seed and short-train an initial population of architectures, either from known de- signs or randomly sampled ones; @ Selection: each evolutionary cycle, we use tournament se- lection (26 to identify parent architectures that are high-performing based on short-training PPL and meet a predefined target latency budget; Mutation: Selected parents undergo targeted muta- tions in one of the design factors -pprrtoor ratios, FFN ratios, or block type count; O Evaluation and replacement: Mutated offspring architectures are trained and evaluated using short-training PPL, with latency accurately estimated from precomputed look-up table (LUT). The oldest architectures replaced with new candidates, effectively balancing exploration and exploitation. A more detailed description of the algorithm is provided in Appendix C.2
Lyc
Oyte
Oyde 16
Oyte
Syoreeii frrmming PPL
Oyde 13
Cyd:
Oyte 10
Cyd:
Oyte
Cyc:
Oyts
Oydc
174 17.5 17.3 110 162 12.4
16.5 15.3 15.0
Oytc
Decoding Latancy (sec.) at
Figure 5: Visualizing the search trajectories.
Search with decoding latency. To evaluate the efficacy of our search framework, we conduct a search using decoding latency the efficiency metric (measured by generating 8k tokens with batch size of I on an NVIDIA A100 GPU). For demonstration purposes, we use a window size of 512
5

---

Table 2: Visualizing architectures searched under different efficiency metrics. m2, a, and denote Mamba2, attention, DeltaNet, and FFN, respectively. CR accuracy is averaged over eight tasks.
Search
Metric
Params (M)
Decoding Latency (8k)
PPL
Wiki
Acc (%)
7 CR
Searched Operator Combinations
Decoding Latency
17.71
20.70
51.04
[.. m2, f, f. m. a. miz d, 1m2, a. f. d, L. a. a. f. m2, a. f.a, f, f.f.f.
mZ.
Parameters
497
16.94
23,06
49.23
for attention, as this length is sufficient for general CR tasks and for the search proxy. We visualize search process in Fig. [ where 10 new architectures sampled and evaluated in each cycle. Our search process progressively improves toward better models with lower PPL. under the target latency.
The searched architecture is visualized in Tab. 2 Interestingly, we find that the latency-friendly archi- and Attention SittionnFFF-Mammaa2 FFN as basic building blocks, stacking them in an interleaved manner. This finding echoes both our earlier observations- DeltaNet and Mamba2 are strong candidates-and prior work that
tecture discovered by the search adopts
interleaves attention and state-space models 6 22
23 77.
We benchmark against baseline architec- tures that have the same model depth and a scaled hidden size to match the decoding latency of our searched architecture. All models are trained on 100B tokens from the Smollm-corpus, and we evaluate them using Wikitext PPL and CR accuracy, aver- aged over eight tasks. As shown in Tab. 3 the searched hybrid architectures outper- form their pure- mure-model counterparts in both PPL and accuracy. This improvement is attributed to: (1) efficient operator combinations that allow for larger parameter counts under the same decoding latency, and (2) the complementary roles played by the hybrid operators.
Table 3: Benchmarking the searched architecture
against baselines scaled to the same latency.
Model
Params (M)
Wiki PPL
CR Acc (%)
|Latency
SWA
18:01
GLA
862
22.67
48.43
18.19
DeltaNet
852
20.90
50.38
18.18
Gated DeltaNet
21.98
17.91
Mamba2
601
23.14
48.61
17.82
Mamba2 + FFN
889
21.43
50.04
17.73
Seurched (Ours)
20.70
51.04
17.71
Search with number of parameters. We also conduct a new round of search using the number of parameters (500M) as the efficiency metric. We find that: 1 The searched architecture achieves over 1.21% % higher CR accuracy and a reduction of more than 0.74 in PPL compared to all 500M baselines (detailed results are provided in Appendix C.3 y; and @ as shown in Tab. 2 when comparing the parameter-efficient architecture with the decoding-efficient one, the former generally includes more at- tention modules, which are purameter-efficiin modules, and has greater model depth, which is a decooing-unffriendlyy Sec. 2. This set of experiments validates our search scheme's 's efficacy in identifying operator combinations that align with the target efficiency metric.
but decoding unfriendlly, and fewer Mamba2/DeltaNNtt
design choice according to
Lesson learned: SLM operator combinations. Hybrid models show great promise, but the synergy among different operators complex, necessitating the identification and combination of complementary operators. The relatively stable ranking of architectures in the early training phases serves as a useful signal for iterating over different designs, and this process can be strategically accelerated using appropriate search algorithms.
2.3
SLM Training: Weight Normalization
The potential of SLMs can be better realized when they are properly trained. We observe that model weights trained under a standard scheme exhibit non-smoothness, with large weight magnitudes along certain dimensions, weight magnitudes can lead to smaller relative weight updates under comparable gradient magnitudes, potentially resulting in ineffective learning during later training stages when the learning rate is low.
shown in the first row of Fig. 6 As suggested by [27 28 larger
Motivated by this and following 28 we constrain weight magnitudes by projecting model weights onto unit norm sphere after each training iteration. This normalization step climinates the radial com- ponent and emphasizes angular updates, leading to larger relative weight changes under comparable gradient magnitudes. Specifically, based on the patterns shown in Fig. 6 where weight matrices ap- plied to hidden features (noted as Case-/) and those whose outputs are added back to hidden features (noted Case-2) exhibit horizontal and vertical patterns, respectively, we perform weight normaliza- tion along the corresponding dimensions. Formally, for each weight matrix W E RCou XC.,, after W.. for = 1..,,,,,, Cout for each training step. we project it onto a unit norm sphere: W,,- + TW.
W., Case-1, 1, and W .j [Welll for 1, ,,, Cin for Case-2. As shown in the second row of Fig. 6 our weight normalization results in smoother weight distributions.

---

Q projection
K projection
v projection
Out projection
momm yuBee IM
Figure 6:
The weight distributions (absolute values) of the layers in the last attention block of 1B Llama models trained without and with our weight normalization technique.
DeltaNet
DeltaNet
Mamba2
w/ winerm wnorm
w/ whamm
wnomm
whom
Vatdatton
2.4
2.4
wnom
2.4
wnorm
24
2.0
2.0
25000 75ace 25000 sab00 79630 Training Step
25000 50003
Training Step
Training Step
Vaining Step
Figure 7: Validation loss trajectories of four model families with and without weight normalization.
Evaluation across models. We apply weight normalization to different models and visual- ize the validation loss curves in Fig. 7 along with the average element-wise gradient norm and L2 norm of the weight matrices in Fig. 8 We can observe that although in the early training stage, the baseline w/o weight normal- ization has steeper convergence due to uncon- strained weights updates (with radical weight updates), the convergence speed will gradually diminish. With weight normalization, the con- vergence speed is more constant and the convergence will surpass the baseline in later training stages, leading to consistent better final convergence across model families; Weight normalization leads to much reduced L2 norm of model weights while slightly increasing the gradient norm compared to the baseline, ensuring larger relative weight changes, which is particularly helpful in late training stages. We observe consistent patterns in other models, provided in Appendix D.2
1.0
120
w/ wnorm who wnorm
0.0
w/O wnom
weight norm
300
0.4
0.2
0.00
0.25
0.50
0.75
1.00
25000
50000
75000
100000
step
165
step
Figure 8: Visualizing the gradient norm and L2 norm of DeltaNet model weights during training.
We also report the corresponding improvements in language modeling and CR accuracy in Tab. [4] We observe that weight normalization consis- tently improves CR accuracy by +1.20% and reduces PPL by 0.66 on average across model families, indicating its general effectiveness as an add-on component.
Table 4: Evaluating weight normalization across IB models trained on 100B tokens from 12].
Model
Setting
J Wikitest PPL. 1 CR Acc (%)
wha wnorm
18.67
53.81
Llama IB
wnorm
18.03
54.85
DeltaNet IB
w/o wnorm
18.
53.46
wnorm
18.19
54.39
Connections with nGPT. Our weight normal- ization technique be viewed as simplified
Mamba2
waormn
18.44
w/ wnorm
17.88
54.71
and efficient variant of nGPT (28). which enforces all computations in LMs to operate on a unit sphere by introducing multiple activation normalization layers addition weight normalization. We find that: O Weight normalization, along with the resulting more effective weight updates. is key improved convergence. When applied alone, weight normalization achieves final task performance comparable to the full nGPT solution, as demonstrated in Appendix D.I @ More importantly, the additional activation normalization layers in nGPT introduce significant training overhead-i.e. increasing SLM training time by more than 20%-which reduces the number of training tokens given a fixed training duration. Thus, our contribution lies in identifying the primary contributing component and delivering more efficient alternative.

---

2.4 SLM Training: Meta Tokens
Previous work 77] has shown that prepending a set of learnable tokens to regular tokens can alleviate attention sinks (29). which are caused by the force-to-attend phenomenon on seman- tically unimportant tokens. We find that these meta tokens can also benefit non-softmax linear attention mechanisms, as they serve learned cache initialization when reformulating linear attention in recurrent format during decoding.
Table 5: Evaluating meta tokens on two linear attention models with 48 operators and on our
searched model with 24 operators in Tab. 2
Model
Meta Taken
Wikitext CCR (%)
Momba2-481-18
weo
19.04
51.71
18,98
52.33
DetiaNa-481-188
19.60
52.12
Searched-24t
w/o
w/
20.61
20.49
50.74
51.13
As shown in Tab. 5 prepending 256 meta tokens can consistently improve both language modeling and reasoning accuracy (+0.45% on average) with negligible overhead.
3 Nemotron-Flash: The New Model Family
Combining all the above architectural improvements and training techniques, we develop and train a new hybrid SLM family called Nemotron-Flasd with two different model sizes.
Model configuration. We adopt the decoding- -rieeddy model structure searched in Sec. 2.2 which interleaves BettaNer-FFNN-amba2..
(Block-1) and Mitenion-FFFN-MMmma2--FF- (Block-2) as basic building blocks. We build two models Cemotron-Flashe IB/BB with 0.96B/2.77 7B parameters, respectively, with depth and width configured based on the scaling laws in Sec. 2.1 Specifically, Nemotron-Flash- IB has the same configuration as in Tab. 2 where the larger parameter count comes from a new tokenizer mentioned below. It has hidden size of 2048 and contains 12 blocks, each with one moken-mixing module and one FFN, or 24 operators if counting DeltaNet, Mamba2, Attention, and FFN as separate operators. Nemotron- Saaitrron-Flssh---e Flash-3B has a hidden size of 3072 and contains 36 operators, with two additional Block-1 1 and one additional Block-2.
Tokenizer. Different from previous parameter-efffiient SLMs 11 that adopt a tokenizer with small vocabulary size to save parameters, we adopt a tokenizer with a larger vocabulary size [30] We find that the latency overhead of the enlarged embedding layer/L.M head is small, while the more coarse roarse-grained token representations reduce the token count when encoding the same sentence, resulting in more significant latency reduction. Detailed analysis in Appendix B
Training settings. Both models are trained using the Adam optimizer (without weight decay, due to the use of weight normalization) and a cosine learning rate schedule with an initial learning rate of 1e-3. We first train the models on Zyda2 [11, then switch to higher-quall
datasets, including
commonsense reasoning datasets (Climb-Mix 32 and Smollm-corpus (12) ). proprietary high-
quality dataset with high proportions of math and code, and MegaMath (33 Both models are trained
for 4.5T tokens using 256 NVIDIA H100 GPUs, with batch size of 2M tokens and a context length of 4096, except for the final 25B tokens, where we extend the context length to 29000.
Deployment settings for latency measurement. To fairly benchmark against baseline models dominated by full attention layers, we adopt TensorRT-LLM's AutoDeploy kernels 10) with efficient KV cache management for full attention and use CUDA Graph for further acceleration. For other operators, we use the official implementation
Mamba2 [[5] and FlashLinearAAtentin (21) for linear attention layers such as DeltaNet, and we always wrap the entire model in CUDA Graph.
Baselines and tasks. We benchmark against SOTA SI.Ms, including Qwen3 34 Qwen2.5 355 Llama3.2 [36]. SmolL.M2 2 12o-Danube (37] and AMD-OLMM 38 series. Accuracy is evaluated using iil-evallation-haaeeee ARCC, ARCE, Hellaswag, Winogrande, OBQA), math (GSM8K, MathQA), coding (HumanEval, HumanEval-Pllus, MBPP, MBPP-Plus), and recall (FDA, SWDE, Squad). We use 5-shot evaluation for GSM8K and MMLU, 3-shot for MBPP and MBPP-Plus, and O-shot for all remaining tasks. We report the average accuracy on each domain in Tab. and provide task-wise accuracy Appendix
(39 across 16 tasks including MMLU, commonsense reasoning (PIQA,
3.1 Benchmark with SOTA Base SLMs
As shown in Tab. 6. our Nemotron-Flass family achieves the lowest decoding latency and the best accuracy among models comparable size. For example, Nemotron-Flasse delivers 5.5% higher accuracy than Qwen3-0.6B with 1.9x latency reduction and 46x higher throughput. Similarly,
8

---

Table 6: We benchmark our Nemotron-Flash- IB/3B against SOTA SLMs across 16 tasks, including MMLU, commonsense reasoning (CR), math, coding, and recall tasks. Latency is measured on an NVIDIA H100 GPU for decoding 8k tokens with a batch size of using CUDA Graph. Throughput (Thr.) is measured with a 32k-token input length using the maximum batch size w/o OOM. Nemotron- Flash-3B-7P is throughput-optimizze variant configured with the IFA+2SWA setting in Sec. 3.3
BS=1
Max BS
MMLU
CR
Math
Coding
Recall
Avg. (%)
Model
Param.
Depth
Latency
Thr. (tok/)
(%)
(%)
(%)
(%)
(%)
AMD-DLMo-IB
1B
16
24.66
142
27.11
51.21
12.66
7.35
60.32
35.63
Llamas-3 3.2 3.2-1B
1.2B
16
21.44
932
31.06
50.82
17.24
24.15
65.41
41.45
Qwen2.5-0.55
0.5B
24
22.81
2382
47.61
47.52
32.66
32.06
65.41
45.16
Qwen3-0.6B
0.6B
28
27.55
160
52.44
48.9
36.88
24.32
62,92
44.11
Nemotronn-Faas Flasb-1B IB
0.96B
12
14.45
7289
44.63
54.46
34.86
37.91
67.11
49.63
H20-Damubc3-45
4B
24
45,92
178
53.76
60.79
34,73
11.65
58.
44.37
SmolLM2-1.78
1.7B
24
31.11
151
50.21
58.61
32.88
21.11
62.42
46.21
Llama-3 1.lia-3.22. 3.2-3B
3B
28
45.47
173
56.30
58.11
30.07
34.80
70.01
50.89
Qwen2.5-1.50
1.5B
28
34.50
687
60.68
56.29
48.14
42.95
69.32
54.65
Qwen3-1.78
1.7B
28
36,20
157
62.46
57.21
53.71
43.76
66.42
55.47
Qwen2.5-3B
3B
36
49,40
439
65.56
58.88
53.83
49.46
73.03
58.96
Nemottron- 31a
2.7B
18
28.71
2939
61.19
61.02
57.62
53.33
73.25
60.98
Flasttoo-Flash-333 7P
2.7B
18
27.95
4657
61.19
60.94
57.88
52.40
73.77
60.84
Nemotron-Flash- achieves +2.0%/45.554 higher average accuracy. 1.7x/1.3x latency reduction,
and 6.4x/18.7x x higher throughput than Qwen2.5-3B/Qwen33-L. respectively. In addition. with further optimization of the attention configuration, Clemotron-Fash- 3B-h--3--PPPP achieves 10.1x and 29.7 higher throughput compared to Qwen2.5-33 and Qwen3-1.7h5 respectively.
It is worth noting that (1) in addition to achieving the most competitive latency and throughput, Remotron-Flash-Se attains the highest accuracy in commonsense reasoning. math, coding, and recall tasks among models larger than 1.5B parameters; and (2) although Nemotron-Flash 1B and Canotron-Flash--33 contain only and full-attention layers, respectively, both achieve the most competitive recall accuracy, suggesting that maintaining full K cache across all layers is unnecessary, which is consistent with observations from existing hybrid LMs [30 40).
3.2 Benchmark with SOTA Instruct SLMs
We instruction-tun the Gemotron-Flash. model using a two-stage supervised fine-tuning (SFT) strategy on two proprietary datasets. The learning rates for the first and second stages set to 8e-6 and 5e-6, respectively. Each stage is trained for one epoch using a cosine learning rate scheduler and global batch size of 384. To accelerate training, adopt the efficient packing strategy from prior with a block size of 29,000 tokens.
work [4 42 (3),
We benchmark NBmotron-FFashh-BB- (5-shot), GPQA (O-shot), GSM8K (5-shot), and IFEval. As shown in Table 7 Nemotron-Flasii 3B-Instruct demonstrates strong reasoning and instruction-followinn best average accuracy and efficiency, e.g.. over +4.7% average accuracy and 4.3x/18.77 higher throughput compared to Qwen2.5-1.5B and Qwen3-1.7B. respectively. Despite having over 1.6x more parameters, which contribute to enhanced intelligence, Nemotron-Flaash maintains superior real-device efficiency, owing its architectural improvements.
against Qwen2.5-L.5B and Qwen3- across MMLU
capabilities, achieving the
Table 7: Benchmarking Nemotron- Flann---lhh--B InIstruuct with SOTA Instruct SLMs.
Instruct Model
Param.
Lat. (s)
Thr. (tok/sec)
MMLU
GPQA
GSM8K
IFEval
Avg
SmoILMM-11.7B
1.7B
31.11
151
49.11
29.24
47.68
55,06
45.27
Qwen2.5-1.5B
1.5B
34.50
687
59.73
30.13
56.03
46.78
48.17
Qwen3-1.7B
1.7B
36.20
157
60.18
28.34
64.88
31.29
46.17
Nemotron-Flash-33
2.7B
28.71
2939
60.34
29.54
69.45
52.03
$2.84
3.3
Ablation Study on Attention Configuration
Full attention (FA) operators are essential for long-context retrieval; however, they also become the primary bottleneck for bage-batch--sze long-context throughput. To better understand this trade-off, we conduct an ablation study on the attention configurations of Nemotron-Flash--33 Starting from the pretrained Nemotron-Flash-331 base model with three FA layers, we perform continuous pretraining with 29k context length for 25B tokens under three configurations: (I) three FA layers, (2) two FA layers plus one SWA layer with an 8k window size, and (3) one FA layer and two SWA layers.

---

3FA
2FA 1 SWA
25WA
1.00
1.00
1.00
0.96
1.00
0.94
Context Length
1.00
1.00
0.99
16304
0.92
24576
1.00
0.99
1.00
0.59
32768
0.96
0.91
0.69
0.94
0.85
0.74
0.78
0.60
0.32
miah, single_3 1
niah, single_2
niah, single
mtah singiingee_1
miah, single
2
niah, single
nlah, singin
miah, single 2
miah, sin.singgee
Figure 9: Visualizing the NIAH scores of different attention configurations on Ruler 44
We report the accuracy on both general benchmarks and three needle-in-a-haystad (NIAH) tasks from Ruler 44 in Tab. B and Fig. 9 respectively. The results show that (I) replacing more FA layers with SWA substantially improves throughput, e.g., the 1FA+2SWA configuration achieves 1.6x
Table 8: Comparing Cenotron-Flash-33 variants with different attention configurations. Decoding throughput is measured with a 32k-token input length using the
maximum batch size without OOM.
Senting
Lat (5)
B8-1
Thr. (168/5)
Max BS
MMLU
CR
Math
Coding
Recall
Ang
SPA
28
2090
61.19
61.02
57.63
53.33
71:25
60:08
2FA-USWA
85
3737
61002
6193
58.56
$1.82
29.37
60:60
higher throughput than the 3FA setting; (2)
I1A+25HA
27.95
4057
61.19
00.4
37.88
52,40
73.77
00.84
general benchmark accuracy, including recall performance, remains largely unaffected when more
SWA layers with 8k window size are used; and (3) NIAH performance, however, drops significantly longer context lengths when the number of FA layers is reduced one, highlighting the importance of FA operators for long-context capability. Therefore, we recommend maintaining at least two full attention layers even in SLMs.
4 Related Work
Small language models. The large model size and computational demands of LLMs [45 46 47 48.49) 49, 50
[1 52] hinder their efficient deployment on resource-constraingg platforms. This has motivated the development of SLMs, such as MobileLLM M. MiniCPM 19, PanGu-7 Pro [I], and Timy (3). These works aim to deliver parameter-efficien SLMs within a given parameter budget. However, parameter efficiency alone often does not translate into proportional latency reductions on real devices. Our work targets real-device efficiency and aims to provide insights and methodologies for developing latency-optimal SLMs. Efficient attention alternatives. To address the quadratic computation and linearly increasing mem- ory of attention modules, efficient attention alternatives with sub-quadratic complexity in sequence length have been proposed [54] 55
17 18]. Notable examples include RWKV [54], RetNet [55], Mamba 14. Mamba2 11] GLA [6]. DeltaNet 117. Gated DeltaNet [8], and Jet-
14|
15
16
4
Block [56), each featuring different memory update rules. However, despite their potential, linear attention mechanisms have been found to exhibit limited recall capabilities (6) and to underperform
standard attention mechanisms on in-context learning tasks [57].
Hybrid language models. To combine the efficiency of linear attention with the recall capabili- ties of quadratic attention, hybrid models that incorporate both types of operators have emerged. Specifically, [57 6 5 22 23 sequentially stack Mamba and attention layers in hybrid models and show enhanced commonsense reasoning and long-contex: capabilities. [7] proposes a hybrid-heal structure that stacks attention and Mamba in parallel. However, existing hybrid models still rely on manual operator combinations, requiring tedious trial-and-erros processes. Our work aims to automate operator combination in hybrid models, enabling more scalable hybrid model development.
Conclusion
This work systematically explores key architectural and training factors essential for developing latency-optimal SLMs. By analyzing optimal depth-width withh ratios, strategically combining efficient attention operators through an evolutionary search framework, and enhancing training with weight normalization and meta tokens, we establish comprehensive framework that significantly improves both real-device latency and accuracy, and deliver the Nemotron- model family that advances the SOTA accuracy-latenc frontier. Beyond the framework, we hope the actionable insights and guidelines we provide will shed light on future research and development of low-latency high- throughput SLMs tailored to diverse, sensitvv real-world applications.
10

---

References
[1] Zechun Liu, Changsheng Zhao, Forrest landola, Chen Lai, Yuandong Tian, Igor Fedorov, Yunyang Xiong, Ernie Chang, Yangyang Shi, Raghuraman Krishnamoorthi, et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. In Forty-first International Conference on Machine Learning, 2024.
[2] Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big data-centric training of small language model, 2025.
[3] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344-16359, 2022.
[4] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnss: Fast autoregressive transformers with linear attention. In International conference on machine learming, pages 5156-5165, PMLR, 2020.
[5] Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai transformer-mante language model. arXiv preprint arXiv:2403. 19887, 2024.
et al. Jamba: hybrid
[6] Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of camba-bassed language models. arXiv preprint arXiv:2406.0788/ 2024.
[7] Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabalesh- warkar, Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, et al. Hymba: A hybrid-head architecture for small language models. arXiv preprint 1rXiv:2411.13677 2024.
[8] Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.1 19427, 2024.
[9] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, al. Miniepm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.063395 2024.
[10] NVIDIA. Tensortt- -lm autodeploy. http::///gihuu..c.//////AAL//// main/ beann/xxapples/eee hatp:////oooddpppooae 2025. Accessed: 2025-05-15.
[11] Yehui Tang. Kai Han, Fangcheng Liu, Yunsheng Ni. Yuchuan Tian, Zheyuan Bai. Yi-Qi Hu, Sichao Liu, Shangling Jui. and Yunhe Wang. Rethinking optimization and architecture for tiny language models. In Forty-first International Conference on Machine Learning. 2024.
[12] Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Smollm-corpus, July 2024.
[13] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
[14] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:23/2.00755. 2023.
[15] Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint 2rXiv:2405.21060, 2024.
11

---

[16] Songlin Yang. Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficien training, arXiv preprint arXiv:2312.06635, 2023.
[17] Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406.064484 2024.
[18] Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2//2006466, 2024.
[19] Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin. Jiaxing Liu, Janna Lu, William Merrill, et al. Rwkv-7" goose" with expressive dynamic state evolution. arXiv preprint arXiv:2503. 14456, 2025.
[20] Tri Dao. FlashAttention-22 Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.
[21] Songlin Yang and Yu Zhang. Fla: A triton-basee library for hardware-efficient implementations of linear attention mechanism, January 2024.
[22] Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. 7amba: A compact 7b ssm hybrid model. arXiv preprint arXiv:2405. 16712, 2024.
[23] Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. arXiv preprint 24iv:2406.00522,, 2024.
[24] Philip Sedgwick. Spearman's rank correlation coefficient. Bmj, 349, 2014.
[25] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. Proceedings of the AAAI Conference on Artificial Itelligence, (100)):770-4788) 2019.
[26] Brad L Miller, David Goldberg, et al. Genetic algorithms, tournament selection, and the effects of noise. Complex systems, 9(3): 1995.
[27] Tero Karras, Miia Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 24174-24184, 2024.
[28) Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, and Boris Ginsburg. ngpt: Normalized transformer with representation learning on the hypersphere. arXiv preprint arXiv:2410.01131,
2024.
[29] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /7453, 2023.
[30] Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich. Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahahaleshwarkar, al. Nemotron-h: A family of accurate and efficient hybrid mamba- transformer models. arXiv preprint arXiv:2504.03624 2025.
[31] Yury Tokpanov, Paolo Glorioso, Quentin Anthony, and Beren Millidge. Zyda-2: 5 trillion token high-quality dataset. arXiv preprint arXiv:2411.06m
2024.
[32] Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Celine Lin, Jan Kautz, and Pavlo Molchanov, Climb: Clustering-baa iterative data mixture bootstrapping for language model pre-training. arXiv preprint, 2025.
[33] Fan Zhou, Zengzhi Wang, Nikhil Ranjan, Zhoujun Cheng, Liping Tang, Guowei He, Zhengzhong Liu, and Eric P Xing. Megamath: Pushing the limits of open math corpora. arXiv preprint arXiv:2504.028877 2025.
[34] Qwen Team. Qwen3, April 2025.
12

---

[35] Qwen Team. Qwen2.5: A party of foundation models, September 2024.
[36] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint arXiv:240..21783, 2024.
[37] Philipp Singer, Pascal Pfeiffer, Yauhen Babakhin, Maximilian Jeblick, Nischay Dhankhar, Gabor Fodor, and Sri Satish Ambati. H2o-danube-1.. b technical report. arXiv preprint arXiv:2401.1683 2024.
[38] Jiang Liu, Jialian Wu, Prakamya Mishra, Zicheng Liu, Sudhanshu Ranjan, Pratik Prabhanjan Brahma, Yusheng Su, Gowtham Ramesh, Peng Sun, Zhe Li, Dong Li, Lu Tian, and Emad Barsoum. Amd-olmo: A series of 1b language models trained from scratch by amd on amd iMstinct mi250 gpus., October 2024.
[39] Leo Gao, Jonathan Tow Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurenee Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
[40] NVIDIA Nemotron Nano. Efficient hybrid mamba- tamba-trannforrer reasoning model. arXiv preprint arXiv:2508. 14444, 2025.
[41] Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310.1694 2023.
[42] Shizhe Diao, Rui Pan, Hanze Dong, Kashun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. Lmflow: An extensible toolkit for finetuning and inference of large foundation models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations), pages 116-127, 2024.
[43] Hanze Dong, Wei Xiong, Bo Pang. Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405. 07866, 2024.
[44] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654 2024.
[45] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint (rXXi:2205.01066 2022.
[46] Albert Q Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0663 2023.
[47] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint urXiv:2307.092683 2023.
[48] Meta AI. Introducing Meta Llama 3: The most capable openly available LLM to date, 2024.
[49] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.088 08295 2024.
[50] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models at practical size. arXiv preprint 2xXi022408.00118 2024.
13

---

[51] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint 25.3 2025.
[52] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[53] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint (rXiv:2401.02385 2024.
[54] Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung. Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305. 13048, 2023.
[55] Yutao Sun, Li Dong, Shaohan Huang. Shuming Ma, Yuging Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint wrXiv:2307.08621, 2023.
[56] Yuxian Gu, Qinghao Hu, Shang Yang, Haocheng Xi, Junyu Chen, Song Han, and Han Cai. Jet-nemotron: Efficient language model with post neural architecture search. arXiv preprint ar0iv:2508. 15884, 2025.
[57] Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? comparative study on in-context learning tasks. arXiv preprint arXiv:2402.04246 2024.
[58]
NVIDIA. Tensortt-Imm: High-performance inference for large language models. https //gi
2025, Accessed: 2025-05-22.
[59] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
A Detailed Experimental Settings
Depth-width scaling in Sec. 2.1 We train series of Llama models with five depth settings: 6, 12, 18, 24, and 30 blocks, where each block contains one attention and one FFN. For each depth setting, further vary the model width, i.c., hidden size, to create different models, ensuring that the resulting models across different depths have relatively comparable parameter ranges. We train each model 100B tokens from the Smollm-corpus [12] using the AdamW optimizer and cosine learning rate schedule with an initial learning rate of 5e-4.
Architecture explorations in Sec. 2.2 For vanilla Mamba and Mamba2 models, we do not insert FFNs, following their original papers. For all other pure models we explored, we add one FFN after each token mixing operator. For hybrid models, we adopt the building block of Operatorl--ppae -Opprato-- FFN. For example, when building a hybrid model with DeltaNet and Mamba2, we use the building block DeltaNet-Mambal
All models have 500M parameters and consist of 24 operators, where each token mixing operator and each FFN is counted as one operator. We train each model on 100B tokens from the Smollm-corpus 12 using the AdamW optimizer and cosine learning rate schedule with an initial learning rate 5c-4. The same training settings are used for training the searched architectures, as well as for the meta token experiments in Sec. 2.4
Weight normalization experiments in Sec. 2.3 For all models with and without weight normaliza- tion, we train them on 100B tokens from the Smollm-corpus [12] using the Adam optimizer and a cosine learning rate schedule with an initial learning rate 1e-3. This learning rate the tuned learning rate that achieves the best convergence for models without weight normalization. We then apply weight normalization on top of this setting to demonstrate that our method can further boost task performance beyond the best baseline.
14

---

Table 9: We benchmark our Nemotron-Flash-1Bie 1B/3B against SOTA SLMs across 16 tasks, including MMLU, commonsense reasoning (CR), math, coding, and recall tasks. Latency is measured on an NVIDIA H100 GPU for decoding 8k tokens with a batch size of using CUDA Graph. Throughput is measured with a 32k-token input length using the maximum batch size w/o OOM.
2ngs
Reed
61.29
03.44
21.5
1250
N:15
26-4
2141
10.65
64:16
61.06
143
31.00
0-
62
8100
1894
18:14
16.00
20-2
44:00
48:44
41mi
40/0
48.11
860
78.00
61.3
36.33
$2.00
30.00
$4.60
12.00
2018
3485
30.00
30.3
15.33
$6.31
17.06
73.3
8531
(0.11
6.18
10.0
11.5
65.11
11.00
27.44
1813
125
1701
1031
11.0
18:33
1756
18.15
0861
19
01.88
30.0
6240
06.9
0.95
68.14
3629
13.00
6800
2835
0.98
1114
1611
7305
08.3
Tasks for computing average commonsense reasoning (CR) accuracy. Unless otherwise specified, in the above exploration experiments, we adopt eight tasks to compute the average CR accuracy: Lambda, PIQA, ARC-Easy, ARC- Hellaswag, Winogrande, TruthfulQA, and SIQA, all from the Im-evaluation-hamees 39
B More Benchmark Results of Nemotron-Flash
As complement to the domain aveerged accuracies reported in Sec. 3.1 and Tab. 6 we present the detailed per-task accuracies within each domain in Tab. 9
C Evolutionary Search: More Details and Experiments
C.1 Detailed Search Space
Search template. We divide the entire architecture into three stages, with each stage repeating one type of building block. This three-stage strategy balances operator heterogeneity with architectural regularity. We search for both the number of blocks in each stage and the structure of each building block, including the involved operators, their ratios, and the ratio of FFNs.
Operators and their ratios. Based on the exploration of pure and hybrid models in Sec. 2.2 include attention, Mamba2, and DeltaNet in our search space, considering both operator effiiincy and their complementary roles. To ensure regularity and avoid overly complex blocks, allow at most two types of operators (excluding FFNs) in each building block type. We support different ratios between two hybrid operators: 0:1 (i.e., only one operator). 1:1, 1:2, and 1:3.
FFN ratios. We allow for 0, 1, or FFNs after each token mixing operator in each building block.
Number of building blocks. We allow flexible configurations of the number of building blocks in each stage, as long as the total number of operators does not exceed a predefined limit (i.e., If the total exceeds this limit, we reduce the number of building blocks in the last stage to meet the depth constraint.
After determining all the above design factors for new architecture, we select a hidden size from [1024, 1280, 1536, 1792, 2048, 2304, 2560], choosing the largest size that satisfies the target latency. This strategy allows us to use the short-training PPL solely to rank the searched architectures.
C.2 Our Evolutionary Search Algorithm
As a complement to the scarch algorithm described in Sec. 2.2 we present our adopted aging evolutionary search in Alg. [ which iteratively refines population of hybrid language model architectures by selectively sampling and mutating architectures over multiple cycles.
Search process. After initializing population of architectures from set of seeded and randomly initialized architectures, set of candidate architectures (we use 10) sampled, mutated, and evaluated concurrently in each cycle. Mutations are performed by altering one of the searchable factors: (1) changing one operator in a building block type, (2) varying the hybrid operator ratio in building block type, (3) adjusting the FFN ratio in a building block type, or (4) modifying the number of building blocks across stages. The sampled architectures are evaluated using short-training perplexity efficient proxy for final performance, and latency is quickly estimated using pre-
15

---

Algorithm 1: Aging Evolutionary Search for Hybrid Operator Combinations
Input: LUT for latency, population size P, sample size S, number of cycles C, target latency
budget Ltarget
Output: Best-performing architecture
Initialize population p + Seeded architectures;
Short-train each architecture in and compute their proxy PPL and latency;
3 for cycle - to C' do
for each architecture current cycle parallel do
Sample subset S _ P randomly, with [S] S:
Select the parent p é- arg min PPL(2) s.t. latency(x)) S Ltargeti
Create offspring by mutating one factor of (operator ratios, FFN ratios, or block type counts), adjusting to maintain depth constraint;
Evaluate by short-training to obtain proxy PPL. and querying LUT for latency;
Remove oldest architecture from P: Add offisrring to population P:
10
return architecture in P with lowest proxy PPL satisfying latency constraint Learget
measured lookup table (LUT). This parallel, proxy-driven strategy effectively balances the exploration of novel designs and the exploitation of known strong performers, facilitating rapid convergence to high-quality architectures.
Short training settings. We train each model on 10B tokens from the Smollm-corpu 12 using the AdamW optimizer and cosine learning rate schedule with an initial learning rate of 5e-4. Training and evaluating a sampled architecture takes approximately 2 hours using 32 NVIDIA A100 GPUs.
C.3 Search with Parameter Efficiency as the Objective
As complement to the searched architecture optimized for parameter efficiency, presented in Sec. 2.2 and Tab. we provide a benchmark against other baseline architectures, all with 500M parameters, in terms of Wikitext PPL and CR accuracy averaged over eight tasks. All models trained 100B tokens from the Smollm-corpus [12] using the AdamW optimizer and a cosine learning rate schedule with initial learning rate of 5e-4. As shown in Tab. TO, the searched architecture achieves 1,21 % higher CR accuracy and a reduction more than 0.74 PPL compared to all 500M baselines.
Table 10: Benchmarking searched architecture, optimized for parameter efficiency, against baseline architectures. All models have 500M parameters. CR accuracy is averaged over eight tasks.
Metric
Attention
GLA
DeltaNet
Gated DeltaNet
RWKV
Mamba
Mamba2
Searched (Ours)
Wiki PPL
24.44
25.16
23.87 47,83
23.80 47.96
25.47 46.46
24.55
24.36 47.72
23,06
CR Acc (%)
48.02
46.82
47.32
49.23
C.4
Configurations of Our Final Nemotron-Fian Models
We provide the detailed configurations of our final Nemotron-FFin -Fassh models in Tab. 11
Table 11: Architectural details of our Nemotron-Flash models. M2, A, D, and F denote Mamba2, attention, DeltaNet, and FFN, respectively. For attention layers, we use group-query attention with group size of 4, with the specific number of heads detailed in the table.
Nexetoo-Paass-l
2305
$144
156
236
ID ML-- A.F.A.A. F. D.P. MU D.P I
16

---

Table 12: Comparing the achieved task accuracy and training speed using vanilla models, weight normalization, nGPT without weight normalization, and full nGPT.
Model
Setting
Wiki PPL
CR Acc (%)
Training Time (sec.) Iter
vanilla
18.67
53.81
0.43
wnorm
18.03
54.85
0,46
LLaMA IB
nGPT w/o wnorm
19.38
51.83
0.56
nGPT
17.45
54.73
0.59
vanilla
17.96
54.36
0.78
wnorm 17.50
55.74
0.81
Hybrid IB
nGPT w/o wnorm
18.96
52.44
1.01
nGPT
17.25
55.33
1.04
D
Weight Normalization: More Analysis and Experiments
D.1
Compare Weight Normalization with nGPT
As discussed in Sec. 2.3 our weight normalization technique can be viewed as simplified and efficient variant nGPT 28) which enforces all computations LMs operate on a unit sphere by introducing multiple activation normalization layers in addition to weight normalization. To analyze the performance breakdown achieved by nGPT, we evaluate our weight normalization, nGPT without weight normalization (i.e., activation normalization only), and full nGPT on two models: Llama 1B model and 1B hybrid model with each building block structured as
As shown in Tab. 12| we observe that: (I) Using only our weight normalization achieves strong task performance, with higher CR accuracy than nGPT and slightly worse language modeling PPL: (2) Weight normalization essential to nGPT, as removing it leads to a notable drop in accuracy; (3) The additional activation normalization layers in nGPT introduce over 30% training overhead. reducing the number of training tokens under a fixed training budget. Thus, our weight normalization offers more efficient alternative nGPT while maintaining comparable task performance.
D.2 Gradient Norm Evolution of More Models
As complement to Sec. 2.3 we further visualize the gradient norm evolution during training with and without weight normalization for four models in Fig. [0] We observe that weight normalization results in slightly increased gradient norm, leading to larger relative weight updates, considering the smaller weight norm. This effect is particularly beneficial in the later stages of training and leads to consistent final accuracy improvements across model families.
Liama
DeltaNet
Gated DeltaNet
Mamba2
w/ wnom
w/ wmomm
w/ wmorr
0.8
0.6
w/o knorm
0.4
w/o wnorm
0.2
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
02
0.4
0.5
0.8
1.0
0.0
02
0.4
08
0.8
1.0
Step
Steg
165
165
step
165
Figure 10: The gradient norm evolution during training w/ and w/o weight normalization.
E
The Choice of Tokenizers
Previous parameter-efficient SLMs 11 adopt tokenizer with small vocabulary size, such as Llama2 [77, to reduce the number of parameters. In contrast, we use a tokenizer with a larger vocabulary size, i.e., Mistral-Neoo-Mnniroo [30]. The rationale is that larger vocabulary often results in more concise token representations and fewer tokens when encoding the same sentence. For example, the phrase by the way" can be tokenized into single by-the-way" token.
17

---

As shown in Tab. 13 when averaging the number of tokens per sample over two datasets, the -istral-NeMo-Minn enizer achieves 9.3% and 13.5% reduc- tion in token count. At the same time, the decoding latency for generating 8k tokens using models with embedding dimensions corresponding to the two tokenizers shows
Table 13: Tokens per sample on AG News Wiki- text datasets using LLaMA2 [47] and Mistral-NeMo- Minitron (30) tokenizers.
tok-
Dataset
LLaMA2
Mistral-NeMo--Mnntrro
AG News 63.64
55.08 (-13.5%)
Wikitext 116.86
106.00 (-9.3%)
only 5.8% latency gap (38.93 seconds 41.31 seconds). This suggests that overall efficiency could be higher when using tokenizers with larger vocabularies. Furthermore, given that larger vocabulary sizes often improve task performance, we adopt the NeMo-Minitron tokenizer [30] over smaller alternatives.
Deployment Flow
Existing deployment frameworks such TensorRT-LLM [58] and vLLM [59] support Attention and Mamba2, while linear attention variants like DeltaNet are only supported by FlashLinearAtten
tion [2I] Hybrid LMs involve diverse operators and require a deployment flow that can efficiently accelerate each operator type. To this end, we integrate TensorRT-LLM's AutoDeploy kernels [TO] with efficient KV cache management for full Attention, the official implementation
Mamba2 [5
and DeltaNet via FlashLineat inearAttention [21 to deploy our Nemotron-Fiin -lash. The entire model is wrapped with CUDA Graph mitigate kernel launch overhead, which is critical for edge applications with batch size of 1.
To demonstrate the effectiveness of deployment flow, we benchmark the decoding speed of Llama model (12 blocks, hidden size of 1536. 550M parameters) using PyTorch, vLLM, TensorRT- LLM, and our deployment flow. As shown in Tab. [4] our deployment flow outperforms vLLM in decoding speed at longer sequence lengths and achieves a latency gap within 15% of TensorRT-LLM for full-attention models. This gap could be further reduced with improved full-attention kernel optimization on top of (IO). Additionally, our deployment supports linear attention variants, which are not supported by vLLM TeM.orRT-LL.M. This of comparisons validates the effectiveness of our deployment flow.
Table 14: Decoding latency (sec.) at various decoding lengths for a 550M Llama model across different deployment flows. The batch size is set to 1, our target is edge applications.
Deployment Flow
64
256
512
1024
2048
4096
8192
16384
32768
PyTorch
0.16
0.96
3.64
7.07
14.28
28.89
56.92
113.93
222.78
457.72
vLLM
0.02
0.14
0.55
1.09
2.16
4.31
8.66
17.82
67.23
169.74
TensorRT-LLM
0.17
0.30
0.64
1.00
1.81
3.42
6.72
13.65
28.41
60.41
Ours
0.01
0.14
0.54
1.08
2.17
4.36
8.72
17.54
35.29
71.65
G Limitations and Future Work
In our architecture search for SLMs, we primarily use language modeling as the search proxy, which may not fully capture other capabilities such as long-context understanding. Additionally, our search focuses on macro-architecture, leaving room to explore more fine-grained design factors. We plan to extend our search framework to optimize for these aspects future work.
H Societal Impact
Our work aims to develop latency-optimal SLMs that lower the barrier to LM deployment and facilitate ubiquitous edge intelligence. potential negative societal impact is that the broader use of LMs may increase the risk of misuse, emphasizing the need for more advanced and efficient safety enhancement techniques.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As highlighted in the abstract and introduction, this paper targets lateney- optimal SLMs and achieves this through the proposed architectural and training strategies.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations of method in Appendix G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided the model configurations and training settings in Sec. 3 and elaborated further in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: We used open data for model training and provided the corresponding refer- ences. Model configurations and training settings are detailed Sec. B and further elaborated in Appendix A
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc,
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
httpc//ggiides/oeueubuii htt////eobuiissiiiee..ae.ee ssi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Model configurations and training settings are detailed in Sec. and further elaborated in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We performed relatively large-scale training (2 100B tokens) so that the task performance is relatively stable across runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The training resources are detailed in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpps////oourrpsccccoo////hhhtt
Answer: [Yes]
Justification:
We have read and followed this guideline.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: We used publicly available open datasets focused factual knowledge, commonsense reasoning, math, and coding, so we did not apply additional safeguards to further enhance safety.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We used open data for model training and have provided the corresponding references.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We did not release any assets the time of submission but will provide thorough documentation when the models from this paper are released in the future.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing experiments research involving human subjects were conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing experiments or research involving human subjects were conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy hatp:////baaurte.ccoooooeeeeees/ for what should or should not be described.
25


Reviews:
[{"review_id": "llJsdg7UxD", "reviewer": "Reviewer_eYGu", "summary": "This paper explores what kind of model structure can reduce efficiency while maintaining performance. This paper focuses on depth-width ratio and operator selection. It enhances the scaling law to obtain the best depth-width ratio and proposes an evolutionary search framework to determine the combination of operators. The proposed Fast-SLM achieves better accuracy-latency performance than Llama3.2-3B.", "questions": "See \"Weaknesses\"\n1. Fast-SLM failed to reach SOTA\n2. Ablation experiments of various technologies of Fast-SLM itself\n3. Experiments and analysis of operator combinations.\n4. Reasons for choosing these two key factors", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThis paper explores hybrid SLM models with optimal accuracy-latency through depth-width ratio and operators. The paper also proposes architectural improvements and training techniques, and conducts experiments to analyze their role in SLM training in turn.\nThis paper proposes a Fast-SLM that combines multiple techniques and achieves better performance than Llama3.2-3B.\nIn addition, this paper's analysis of the combination of different promising operators may help the community in subsequent model training work.\n\n**Weaknesses**\n\n1. Fast-SLM failed to reach SOTA in 7 out of 9 benchmarks, and even underperformed the 1.5B model in some benchmarks. Although this may be reasonable, as its focus is more on the accuracy-latency trade-off, you still need to explain why this happens.\n2. You may need to perform ablation experiments on Fast-SLM itself to better show the impact of each technology on the model, rather than just performing experiments when introducing the technology.\n3. More experiments and analysis on operator combinations are needed. As can be seen from Table 1, DeltaNet and Gated DeltaNet have achieved small improvements after adding Mamba2, which makes people wonder whether some of the conclusions in \"Observations and Analysis\" are statistically significant.\n4. What is the reason for the author to choose these two key factors? The latency of real devices is mainly determined by two key factors: the depth and width of the model, and the choice of operators. These two factors are intuitively very reasonable, but I still worry about whether there are omissions. Is this theory proven by previous work, or is it determined by the author based on experience?", "was_revised": true, "final_justification": "Many of my concerns were addressed by the authors, and I encourage them to include new results and interpretations in the final paper.\n\nThank you for your additional analysis, we would like to raise our rating to 5", "rebuttal": "Thank you for recognizing the insights our work offers to the community, as well as for your constructive comments! We have addressed all of your questions below:\n\n**1. Fast-SLM failed to reach SOTA**\n\nWe respectfully clarify that accuracies on certain tasks are heavily influenced by the pretraining data. For example, the Qwen series achieves notably high scores on factual knowledge tasks like MMLU due to their proprietary pretraining corpora. Therefore, in the absence of access to the pretraining data of these models, we believe the average accuracy across diverse tasks offers a more meaningful comparison than single-task performance.\n\nAdditionally, since the paper submission, we have continued pretraining our models on an additional 2T tokens (totaling 4T tokens) and benchmarked them against baselines across 14 tasks, including Commonsense Reasoning (MMLU, Lambda, PIQA, ARCC, ARCE, HellaSwag, Winogrande, OBQA), Math (GSM8k, MathQA), and Coding (HumanEval, HumanEval-Plus, MBPP, MBPP-Plus).\n\nAs shown in the table below, our Fast-SLM-2.7B achieves SOTA average accuracy among all models. Specifically, Fast-SLM-960M surpasses Qwen3-0.6B by 3.77% in average accuracy while being 1.75× faster. Fast-SLM-2.7B outperforms LLaMA3.2-3B and Qwen2.5-3B by 9.24% and 0.90% in average accuracy, respectively, with 1.46x and 1.67x speed-ups. We will include these updated results in the final version and open-source our model.\n\n\n| Model | Param. | Dec. Latency (s) | 8k Cache Size (MB) | CR (%) | Math (%) | Coding (%) | Avg (%) |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| AMD-OLMo-1B | 1B | 28.39 | 1049 | 49.21 | 12.66 | 7.35 | 32.03 |\n| Llama-3.2-1B | 1.2B | 28.37 | 262 | 49.73 | 17.24 | 24.15 | 37.78 |\n| Qwen2.5-0.5B | 0.5B | 28.19 | 98 | 48.13 | 32.66 | 32.06 | 41.33 |\n| Qwen3-0.6B | 0.6B | 33.91 | 918 | 49.92 | **36.88** | 24.32 | 40.74 |\n| Fast-SLM-960M (Ours) | 0.96B | **19.36** | **36** | **52.54** | 24.57 | **38.44** | **44.51** |\n| h2o-danube3-4b-base | 4B | 61.94 | 737 | 60.85 | 34.73 | 11.65 | 43.06 |\n| SmolLM2-1.7B | 1.7B | 41.55 | 1573 | 58.64 | 32.88 | 21.11 | 44.24 |\n| Llama-3.2-3B | 3B | 60.38 | 918 | 59.37 | 30.07 | 34.80 | 48.16 |\n| Qwen2.5-1.5B | 1.5B | 44.06 | 229 | 57.53 | 48.14 | 42.95 | 52.02 |\n| Qwen3-1.7B | 1.7B | 46.39 | 918 | 58.62 | 53.71 | 43.76 | 53.67 |\n| Qwen2.5-3B | 3B | 69.19 | 295 | 60.69 | **53.83** | 49.46 | 56.50 |\n| Fast-SLM-2.7B (Ours) | 2.7B | **41.31** | **80** | **61.62** | 48.88 | **53.22** | **57.40** |\n\n---\n\n**2. Ablation experiments on Fast-SLM’s components**\n\nThank you for the suggestion! We believe this analysis can strengthen our paper. Following your suggestion, we conducted the following ablation study: Starting from a 500M LLaMA model with 24 attention blocks, we progressively reach our final Fast-SLM through the following steps: (1) tune the depth/width ratio, (2) enhance it into Fast-SLM’s hybrid architecture, (3) add meta tokens, and (4) add weight normalization.\n\nAs shown in the table below, we observe that (1) tuning the depth/width ratio (into a shallow-wide model) reduces latency even though the parameter count is almost doubled, leading to better accuracy; (2) enhancing it into Fast-SLM’s hybrid architecture boosts task accuracy while significantly improving efficiency, e.g., a 3.2x latency reduction for (bs=32, output len=8k) generation; (3) adding meta tokens and weight normalization further improves task performance, yielding a +2.27% accuracy gain in total.\n\nWe will include this analysis in the final version of our paper.\n\n| Technique | Model Arch | Param | Latency (bs=1, output_len=8k) | Latency (bs=32, output_len=8k) | Wiki PPL | Acc (%) |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| [*Starting point*] | Llama3 24-Block | 500M | 30.41 | 110.77 | 23.74 | 48.18 |\n| + Tuned depth-width ratio | Llama3 12-Block | 970M | 24.75 | 101.11 | 20.96 | 50.11 |\n| + Hybrid operators | Fast-SLM | 965M | 19.46 | 31.87 | 20.52 | 50.77 |\n| + Meta tokens | Fast-SLM | 965M | 19.54 | 32.33 | 20.38 | 50.88 |\n| + Weight normalization | Fast-SLM | 965M | 19.54 | 32.33 | 19.42 | 53.04 |\n\n*Note*: Acc is averaged over 8 commonsense reasoning tasks.\n\n---\n\n**3. More analysis of operator combinations**\n\nAs a recap of the context, for the 500M architecture ablation in Table 1 of our submitted manuscript, we find that adding Mamba2 with other operators can result in a 0.43~1.12 PPL reduction. For example, combining DeltaNet and Mamba2 leads to a 0.5/0.99 PPL reduction and a +0.2%/+0.3% accuracy improvement over pure DeltaNet and pure Mamba2, respectively. We humbly clarify that this improvement is non-trivial, as the reported gains of emerging operators over prior baselines are often of a similar scale. For example, DeltaNet [1] improves over its Mamba baseline by -0.15 PPL / +0.3% accuracy (see their Table 1 at 340M scale).\n\nIn addition, we have extensively experimented with different operator combinations, and in general, these hybrid architectures do not consistently outperform the pure models, often resulting in comparable PPL. The benefit of adding Mamba2 is a particularly interesting observation that we have consistently seen, and we highlight this aspect to provide insights to the community. \n\nIt is also worth noting that the goal of this combination study is to identify promising operators that are efficient and complementary to one another to serve as a search space for evolutionary search, rather than to find the single best hybrid architecture. Following your suggestion, we will include the comprehensive exploration of different operator combinations we did in the appendix of the final version.\n\n\n[1] “Parallelizing Linear Transformers with the Delta Rule over Sequence Length”, S. Yang et al., NeurIPS 2024.\n\n---\n\n**4. Reasons for choosing these two key factors**\n\nThank you for the question! We also plan to emphasize this in the final version.\n\nIn general, there are two critical efficiency metrics for SLMs (corresponding to two use cases): (1) inference latency under batch size 1, and (2) generation throughput with the maximum batch size without out-of-memory.\n\nFor case (1), GPU utilization is often low with large overheads for kernel launch, making small and large kernels exhibit similar latency. In such scenarios, reducing the number of kernels is key to improving latency. Therefore, tuning the depth–width ratio, where shallow-wide models result in fewer but larger kernels, is crucial for optimizing batch size = 1 latency.\n\nFor case (2), GPUs are better saturated and become compute-bound, so adopting compute-efficient operators is critical. Building hybrid operators with efficient linear attention is an effective approach. (These hybrid operators can also help in case (1) by reducing latency per kernel.)\n\nBy optimizing these two macro-level design factors, our Fast-SLM-2.7B achieves efficiency across a wide range of deployment scenarios. For example, it is 1.67x / 3.34x faster than Qwen2.5-3B in case (1) and case (2), respectively. There may be other finer-grained or minor design factors, but we believe the above insights can also help guide those decisions. We will highlight this discussion in the final version."}, {"review_id": "UESyO5poFA", "reviewer": "Reviewer_aYwt", "summary": "This paper addresses the gap between parameter efficiency and real-device latency in small language models (SLMs), which is crucial for real-world, latency-sensitive applications. The authors systematically analyze the core determinants of SLM latency, depth-width ratios and choice of operators, and introduce design and training principles for constructing latency-optimal SLMs.", "questions": "1. To what extent do your depth-width and operator selection findings transfer to other hardware platforms, such as edge devices, mobile GPUs.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper provides an in-depth examination of the relationship between model architecture (especially depth and width) and real-device latency, an aspect that is underexplored in previous studies.\n\n2. The paper presents an evolutionary search pipeline  to automatically identify complementary combinations of efficient attention operators.\n\n3. The paper is well structured and easy to follow.\n\nWeakness:\n\n1. The developed law is validated on Llama models and a single hardware setup. It is unclear whether the fitted exponents generalise to other architectures such as  MoE architectures or to CPUs/edge NPUs.\n\n2. The latency analysis and all benchmarking are conducted on NVIDIA A100 GPUs, While this is sensible for reproducibility, the broader generalizability of the architectural/principled findings to edge-devices is also important to included considering this paper focus on SLMs.", "was_revised": true, "final_justification": "My concerns have been addressed. I will maintain my current score.", "rebuttal": "Thank you for your constructive comments! We have addressed all of your questions below:\n\n**1. The generalizability of the architectural findings on other hardware platforms**\n\nThank you for the suggestions! Following your suggestion, in addition to the data-center GPU A100 reported in our submitted manuscript, we have also verified our findings on (1) a consumer-level GPU, the RTX 3090, and (2) a workstation GPU, the RTX A5000. We will include results on other edge GPUs given more time, after obtaining access to them post-rebuttal.\n\n**a.** Depth-width trade-offs\n\nTo demonstrate the importance of depth–width trade-offs and the advantages of shallow-wide models, we fix the model size of a LLaMA model to 500M / 3B and vary its depth and width. We then report the latency on the RTX 3090 and RTX A5000 (batch size = 1, input length = 128, output length = 8000).\n\nAs shown in the table below, shallow-wide models consistently exhibit lower latency across both platforms and model sizes, aligning with the observations in Section 2.1 of our submitted manuscript.\n\n| Device | Model Size | 6L | 12L | 18L | 24L | 30L |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| RTX 3090 | 500M | 12.99 | 17.88 | 24.52 | 26.44 | 35.58 |\n|  | 3B | 62.10 | 69.34 | 85.87 | 96.05 | 102.31 |\n| RTX A5000 | 500M | 17.48 | 22.03 | 31.17 | 30.78 | 40.71 |\n|  | 3B | 83.73 | 91.02 | 108.28 | 118.80 | 126.03 |\n\n\n**b.** Efficiency of different operators\n\nWe benchmark the efficiency of different operators on the RTX 3090 and RTX A5000 (batch sizes = 1 and 16, input length = 128, output length = 8000), following the setting in Figure 4 and Section 2.2 of our submitted manuscript. All models are 500M.\n\nAs shown in the table below, Mamba2 and DeltaNet generally stand out as the most efficient operators across settings and platforms, echoing the observations in Section 2.2 of our submitted manuscript. Considering the necessity of attention for recall-intensive tasks, we adopt Attention, Mamba2, and DeltaNet as candidates in our architecture search in Section 2.2.\n\n| Device | Attn | Mamba | Mamba2 | DeltaNet | Gated DeltaNet | GLA |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| RTX 3090 (bs=1) | 18.52 | 21.17 | 14.61 | 16.03 | 17.09 | 22.23 |\n| RTX 3090 (bs=16) | 33.38 | 27.54 | 17.06 | 21.98 | 24.21 | 30.21 |\n| RTX A5000 (bs=1) | 22.02 | 25.74 | 18.96 | 20.31 | 20.98 | 26.35 |\n| RTX A5000 (bs=16) | 42.28 | 30.68 | 22.78 | 26.61 | 28.17 | 34.94 |\n\n\n**c.** Benchmark the final model\n\nWe further provide benchmarks of the final delivered models on the RTX 3090 and RTX A5000 (batch sizes = 1 and 16, input length = 128, output length = 8000). As shown in the table below, Fast-SLM-2.7B continues to stand out with the lowest latency compared to other models of similar size. Specifically, Fast-SLM-960M and Fast-SLM-2.7B achieve 3.3x and 1.9x speed-ups over Qwen3-0.6B and Qwen2.5-3B, respectively, on the RTX 3090 with batch size = 16.\n\n| Model | Param. | RTX 3090 (bs=1) | RTX 3090 (bs=16) | RTX A5000 (bs=1) | RTX A5000 (bs=16) |\n|:---:|:---:|:---:|:---:|:---:|:---:|\n| SmolLM2-360M | 360M | 32.71 | 62.70 | 34.54 | 76.61 |\n| Qwen2.5-0.5B | 0.5B | 26.68 | 52.89 | 29.86 | 59.46 |\n| Qwen3-0.6B | 0.6B | 38.15 | 131.27 | 42.84 | 164.95 |\n| AMD-OLMo-1B | 1B | 40.69 | 133.45 | 53.94 | 173.07 |\n| Llama-3.2-1B | 1.2B | 32.29 | 62.85 | 43.02 | 77.95 |\n| Fast-SLM-960M (Ours) | 0.96B | 26.61 | 40.29 | 35.39 | 49.56 |\n| Qwen2.5-1.5B | 1.5B | 51.69 | 103.63 | 63.50 | 120.49 |\n| SmolLM2-1.7B | 1.7B | 58.37 | 118.48 | 78.22 | 164.68 |\n| Qwen3-1.7B | 1.7B | 59.82 | 171.93 | 76.81 | 214.85 |\n| IBM-Granite-3.0-2b | 2b | 80.52 | 155.49 | 105.45 | 192.52 |\n| Llama-3.2-3B | 3B | 88.20 | 175.45 | 111.39 | 218.92 |\n| Qwen2.5-3B | 3B | 87.63 | 167.73 | 116.36 | 190.14 |\n| H2O-danube3-4b-base | 4B | 102.38 | 174.21 | 133.58 | 231.66 |\n| Fast-SLM-2.7B (Ours) | 2.7B | 71.93 | 88.12 | 88.81 | 118.50 |\n\nIn summary, the key conclusion is that all findings remain valid across different hardware platforms. In general, the low GPU utilization during language model decoding highlights the need for fewer kernel launches, and the quadratic cost of attention motivates the adoption of more efficient operators with linear complexity. These two factors are broadly applicable across GPU platforms, reinforcing the importance of tuning depth–width ratios (as shallow-wide models result in fewer but larger kernels) and leveraging efficient/hybrid operators.\n\n---\n\n**2. Clarifications on the depth-width scaling law**\n\nWe first respectfully clarify that the depth–width scaling law is hardware-agnostic, meaning the fitted exponents are determined solely by the model. Specifically, as shown in Equation 1 of our submitted manuscript, the scaling law predicts the model loss based on depth, width, and data. In practice, given a target latency budget and deployment setting, the optimal depth–width ratio can be identified by profiling a range of configurations and selecting the one that satisfies the latency constraint while minimizing the loss predicted by the scaling law. For models other than Llama, given more time after the rebuttal period, we can follow your suggestion to train a series of models such as MoE and fit the scaling law to observe how the exponents vary.\n\nIn addition, we would like to mention that the scaling law derived from Llama can serve as a useful guide for selecting the depth–width ratio of other models. For example, the scaling law on Llama suggests that our Fast-SLM-960M should use 12 blocks, where one block includes one (linear) attention and one FFN. We benchmark Fast-SLM-960M-12b (the adopted configuration) against its 8-block and 16-block variants under an iso-latency setting, i.e., all models are scaled to a comparable latency by adjusting the width.\n\nAs shown in the table below, Fast-SLM 12-block achieves the best PPL and accuracy across all three settings, suggesting that the depth–width ratio predicted by the LLaMA scaling law provides a strong starting point. Note that our goal with the scaling law is not to predict a single best setting, but rather to (1) emphasize the general trend of this efficiency-critical design factor, and (2) quickly obtain a near-optimal depth–width ratio under a target latency as a good starting point.\n\n| Model | Latency (sec.) | Param | Wiki PPL | CR Acc (%) |\n|:---:|:---:|:---:|:---:|:---:|\n| Fast-SLM 8-block | 19.5 | 1211M | 20.59 | 50.21 |\n| Fast-SLM12-block | 19.7 | 965M | 20.52 | 50.77 |\n| Fast-SLM 16-block | 20.4 | 533M | 23.34 | 48.40 |\n\n*Note*: Latency is measured on an A100 with an output length of 8k and batch size of 1. CR accuracy is averaged over 8 commonsense reasoning tasks."}, {"review_id": "vxS6520SUO", "reviewer": "Reviewer_X2pZ", "summary": "This paper focuses on deriving latency-optimal small language models (SLM). The authors first derive a scaling law of validation loss w.r.t. depth & width and find a sweet spot of the depth-width ratio that achieves the best result given a certain latency budget. Then the authors conduct an evolutionary search on hybrid architecture to find the best architecture that achieves the best tradeoff between short-term PPL and decoding latency. Then the authors employ a few more tricks: weight normalization, trainable meta tokens, larger vocab size, etc. and train a new hybrid model family Fast-SLM with 1B and 7B scale, with 3-phase training: phase 1 on 1.6T pre-training, phase 2 annealing on 600B high-quality data, and optional phase 3 with distillation.", "questions": "How does the LUT of the decoding latency is derived in the hybrid operator evolutionary search process? I am assuming we are not recording on all possible decoding latency and there exists some decoding latency estimated formulas for each operator. It would be helpful to include this calculation process and formula in the Appendix.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- (**major**) This is a decent attempt towards latency-optimal SLM and has profound implication for resource-constrained device settings, and even draft model for speculative decoding, etc. I perceive this paper will produce significant community impact if the model weights are released and the complete training recipe (codebase and the data) are open-sourced. \n\n- The experiments and evaluation are comprehensive and they show Fast-SLM can match frontier SLM model's performance while having shorter decoding latency.\n\n- The paper is generally clear and well-written, and provides a detailed discussion on design choices that could produce frontier SLM.\n\nWeakness:\n\nThe paper is generally clear and has profound implications, but I believe there are a few technical details to be fixed / addressed for the rigor of study.\n\n- (**critical**) The authors first explore the best sweet spot of depth-width ratio in **Llama models** (Figure 3), and then they explore the best **hybrid operators** that achieves the desired decoding latency-performance frontier (Table 2). Then, both designs are combined in Fast-SLM model family for training. In this case, we assume the best depth-width ratio derived in Llama models can be directly transferred to hybrid models. This is a rather strong assumption because different operators would have different dependencies over depth and width for both validation loss and latency. I know the search space would be too large if we search the scaling law for every hybrid operator combinations, **but at least we should search for the latency-val loss tradeoff on the final searched architectures directly instead of directly adapting the Llama model's scaling result.**\n\n- I don't know if the final decoding-latency optimized architecture in Table 2 is included in Figure 5 because points at 17.7 decoding latency in Table 2 clearly does not achieve the lowest short-training PPL in Figure 5.\n\n- **Need reconciliation**: there are some previous scaling laws (e.g. Kaplan. Chinchilla doesn't study it) suggest that the model aspect ratio's impact on the validation loss is less important than the total number of parameters (Kaplan Figure 6). **The left subfigure in Figure 2 in the paper suggests the opposite as the gap across different aspect ratio could be more significant than the gap across number of parameters.** I expect the aspect ratio to be less important for LLM but maybe more important for SLM, but we certainly need such discussion & justification in the paper.\n\nI will retain my score at this moment.", "was_revised": true, "final_justification": "I believe this paper has a clear community interest and impact so I would recommend acceptance. However, I still believe there are some technical details need to be improved, but this does not affect my overall assessment.", "rebuttal": "Thank you for describing our work as “*a decent attempt towards latency-optimal SLM*” and recognizing its potential to “*produce significant community impact*,” as well as for your constructive comments! We will open-source all of our search/training recipes and final models upon acceptance. We have addressed all of your questions below:\n\n**1. Search depth-width ratios on Llama and transfer to hybrid models**\n\n*(a) The reason for decoupling depth–width ratio search from operator search*\n\nThank you for the question! First, we agree that jointly searching both operator combinations and depth–width ratios has greater potential to yield a strong architecture, and our codebase can support this without any modification. That said, we currently decouple depth–width ratio search from operator search for two main reasons: (1) We aim to highlight the insights from each design factor independently, with the other held fixed, so that the resulting findings can better benefit the community and inspire future work; (2) As you also noted, a large design space can make the search more difficult and time-consuming to converge.\n\nAlthough this work focuses on providing insights into both design factors rather than proposing a NAS framework, we agree that a joint search has greater potential and will try it for future work given more time after the rebuttal period.\n\n*(b) The scaling law derived from Llama provides good starting points for selecting the depth–width ratio of hybrid models*\n\nAdditionally, we would like to mention that the scaling law derived from Llama serves as a useful guide for selecting the depth–width ratio of hybrid models. In practice, we first build the scaling law using Llama models and then use it to predict promising candidates for our hybrid models, selecting the best one based on profiling results.\n\nSpecifically, the Llama-derived scaling law suggests that our Fast-SLM-960M should use 12 blocks, where each block includes one (linear) attention and one FFN. We benchmark Fast-SLM-960M-12b (our final adopted configuration) against its 8-block and 16-block variants under an iso-latency setting, i.e., all models are scaled to a comparable latency by adjusting the width.\n\nAs shown in the table below, Fast-SLM 12-block achieves the best PPL and accuracy across all three settings, suggesting that the depth–width ratio predicted by the LLaMA scaling law provides a strong starting point. Note that our goal with the scaling law is not to predict a single best setting, but rather to (1) emphasize the general trend of this efficiency-critical design factor, and (2) quickly obtain a near-optimal depth–width ratio under a target latency as a good starting point.\n\n| Model | Latency (sec.) | Param | Wiki PPL | CR Acc (%) |\n|:---:|:---:|:---:|:---:|:---:|\n| Fast-SLM 8-block | 19.5 | 1211M | 20.59 | 50.21 |\n| Fast-SLM12-block | 19.7 | 965M | 20.52 | 50.77 |\n| Fast-SLM 16-block | 20.4 | 533M | 23.34 | 48.40 |\n\n*Note:* Latency is measured on an A100 with an output length of 8k and batch size of 1. CR accuracy is averaged over 8 commonsense reasoning tasks.\n\n---\n\n**2. Data points in Table 2 and Figure 5**\n\nThank you for pointing out this potential confusion! As mentioned in Lines 206–207 of our submitted manuscript, our search metric is the model with the lowest PPL under a target latency. Accordingly, the final model we selected corresponds to the one with lowest PPL under approximately 18.3s latency in Figure 5 (which is annotated by a green dot).\n\nThe latency values in Figure 5 are obtained by summing the profiled latency of each operator  from the LUT. The 17.71s shown in Table 2 refers to the same model, but with latency measured in an end-to-end manner, which results in a slightly lower absolute value. Although there is a small discrepancy between LUT-based and end-to-end latency measurements, we find that summing the profiled latencies from the LUT preserves the relative latency rankings across different models, particularly when the depth is fixed. This LUT-based approach is also widely adopted in prior NAS works such as FBNet [1]. We will clarify this point in the final version.\n\n---\n\n**3. The importance of the aspect ratio to SLMs/LLMs**\n\nThis is a great question! We have conducted additional analysis on this and plan to emphasize it in the final version. We completely agree that the impact of aspect ratio on both latency and accuracy strongly depends on model size, as elaborated below.\n\n(1) **In terms of latency**, based on our fine-grained profiling on GPUs, the reason why aspect ratios matter for SLMs is that when the model is relatively small, GPU utilization tends to be low with relatively large overheads for kernel launch, making small and large kernels exhibit similar latency. In such scenarios, reducing the number of kernels is key to improving latency, and shallow-wide models can result in fewer but larger kernels. However, for LLMs with much larger kernels, GPU utilization is higher and kernel launch overhead becomes less significant, making aspect ratios less critical.\n\nTo demonstrate this, we compare two LLaMA models with 12 and 24 blocks, respectively. We vary their widths to equalize their latency and record the number of parameters, as more parameters may indicate potentially better accuracy. As shown in the table below, with a lower target latency and smaller model sizes, shallow-wide models can yield up to 10x more parameters under the same latency. For larger models, however, the additional parameters enabled by shallower models diminish, potentially resulting in smaller accuracy gains (or even worse accuracy). This indicates that aspect ratio has a greater impact on SLMs.\n\n| Latency (sec.) | Depth=24 - Param (B) | Depth=12 - Param (B) | Increased Param Ratio |\n|:---:|:---:|:---:|:---:|\n| 2.5 | 0.08 | 0.86 | 10.62x |\n| 3.0 | 0.28 | 1.30 | 4.72x |\n| 5.2 | 1.54 | 3.18 | 2.07x |\n| 7.0 | 2.95 | 4.37 | 1.48x |\n| 8.0 | 3.82 | 5.36 | 1.40x |\n| 9.4 | 4.81 | 6.45 | 1.34x |\n| 14.0 | 8.45 | 10.32 | 1.22x |\n| 19.3 | 13.11 | 15.10 | 1.15x |\n\n(2) **In terms of accuracy**, as observed in Figure 2 of our submitted manuscript, as model size increases, the accuracy gaps among different depth–width ratios gradually narrow. This also indicates that aspect ratio matters more for SLMs than for LLMs.\n\nIn summary, the above analysis suggests that our techniques and insights regarding depth–width ratios are particularly well-suited for SLM design. We believe this analysis strengthens our paper by identifying when these design factors matter most and when our techniques should be adopted. We will follow your suggestion and add further discussion in the final version.\n\n---\n\n**4. How to construct the LUT**\n\nWe followed common practice [1] to construct the LUT by profiling the latency of each operator, where the operator type and its width (i.e., dimensionality) define an operator, under the target deployment setting. Specifically, we profile Attention, Mamba2, DeltaNet, and FFN with various widths under the target setting and record their latencies in a table. For a model architecture with an arbitrary combination of these operators, we approximate the overall model latency by summing the latency of each operator. We find that this approach preserves the relative latency rankings across different models, particularly when the depth is fixed. We will clarify this in the final version.\n\n[1] “FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search”, B. Wu et al., CVPR 2019."}, {"review_id": "LD3TijoJBd", "reviewer": "Reviewer_ictF", "summary": "This paper introduces Fast-SLM, a new series of small language models designed to run faster on real devices, not just look small in size. The authors point out that having fewer parameters doesn’t always mean the model will be faster in practice.\n\nTo solve this, they propose a full strategy that considers both how the model is built and how it's trained, aiming for the best balance between speed and accuracy.\n\nTheir key ideas include:\n\n1. Model Structure: They study how deep and wide a model should be. Instead of blindly following the trend of \"deep and narrow\" models, they use a math-based method to find the best structure for different speed limits.\n2. Operator Mixing: They test various attention mechanisms (like Mamba2, DeltaNet, etc.) and show that mixing them in one model works better than using just one type. They even use automated search to find the best mix.\n3. Training Tricks: They add two smart tricks:\n     - Weight normalization for more stable and effective training.\n     - Meta tokens to speed up generation and improve accuracy.\n4. Fast-SLM Models: They build two models (960M and 2.7B) using the above ideas. These models beat top models like Llama3 and Qwen3 in both speed and accuracy. For example, Fast-SLM-2.7B is 1.46× faster, 2.57% more accurate, and uses 10× less memory than Llama-3.2-3B.\n5. Real-world Testing: They test the models on 9 language tasks and show that Fast-SLM runs fast and performs well using real deployment tools like TensorRT-LLM and CUDA Graph.", "questions": "Questions and Suggestions for the Authors\n1. How much of the latency improvement comes from the model design vs. deployment tools (e.g., CUDA Graph, TensorRT)?\nYou report large speedups, but it’s unclear how much is due to architectural changes versus low-level software optimization.\nCould you run an ablation showing performance without CUDA Graph or AutoDeploy, to understand what is “portable” across platforms like CPU or non-NVIDIA chips? This would increase confidence in the general usefulness of Fast-SLM.\n\n2. Can your hybrid operator search framework be used under realistic training budgets (e.g., <50B tokens, <32 GPUs)?\nThe search method looks great, but you trained on 100B tokens or more. That’s hard to reproduce.\nCan you demonstrate that this search method is still effective with limited resources, like a smaller dataset or fewer GPUs? If yes, it would make your method much more appealing to both academia and small-scale industry.\n\n3. Why no evaluation on long-context or multi-turn tasks?\nYou emphasize latency, but reducing cache may hurt long-context understanding or dialogue.\nCan you report performance on long-context benchmarks (like L-Eval, LongBench, or multi-turn QA)? If results are strong, it will show Fast-SLM is not just fast, but also broadly capable.\n\n4. Have you considered risks of easy on-device misuse (e.g., spam bots, data leakage)?\nFast, on-device models are great—but also make LLM misuse cheaper and harder to monitor.\nDo you plan to release safety filters or usage restrictions? What’s your view on responsible deployment? A short note on this would help align with responsible AI practices, especially for industrial readers.\n\n5. Could your scaling law for depth-width trade-off be released as a public tool?\nYour method for selecting the best depth-width combo seems useful beyond this paper.\nWould you consider releasing a simple API or table that helps others design latency-efficient models on their own hardware? This would increase the paper’s impact and adoption in the community.", "limitations": "Limitations:\n\nA few important and practical limitations could be discussed more explicitly:\n\n1. Deployment Dependency: The reported speedups depend heavily on deployment tricks like TensorRT-LLM AutoDeploy and CUDA Graph. These are great tools—but not every user or platform (e.g., CPU-only devices, non-NVIDIA hardware) can use them. The authors could clarify how much speedup truly comes from their architecture, and how portable the results are.\n2. Training Cost and Accessibility: The paper mentions training on 1.6T–2.2T tokens using 256×H100 GPUs. This scale is out of reach for most researchers and companies. It would be helpful to discuss whether the benefits of Fast-SLM (e.g., search, hybrid design) still hold under smaller budgets.\n3. Underexplored Risks in Cache Reduction: While cache size is drastically reduced (which is good), aggressively minimizing cache might affect context consistency or in-context learning ability in real-world usage. This trade-off deserves more discussion.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths \n1. Clear Motivation: The paper clearly explains why current small models aren't always fast on real devices, and why optimizing only for parameter count isn't enough.\n2. Practical Significance: The focus on real-device latency (instead of just model size) makes this work very relevant for real-world deployment (e.g., mobile, edge, low-latency applications).\n3. Useful Insights:The depth-width trade-off and operator mixing analysis are well presented and offer general guidelines for future model design.\n4. Strong Results: The Fast-SLM models clearly outperform popular models like LLaMA and Qwen in both speed and accuracy. The 10× reduction in cache usage is especially impressive.\n5. Simple Training Tricks with Real Impact: The proposed weight normalization and meta tokens are lightweight, easy to use, and give noticeable gains.\n\nWeaknesses \n\n1. Limited Novelty in Techniques: Many of the components (e.g., scaling laws, hybrid models, weight norm, meta tokens) are based on existing ideas. The novelty comes more from combining them well rather than introducing fundamentally new methods.\n2. Lack of Statistical Significance Analysis: The paper does not provide error bars or variance across runs. That makes it harder to judge whether the improvements are robust.\n3. Search Focus Is Narrow And Searching Efficiency Is Low: The architecture search only considers macro-level blocks (e.g., which operator to use), but does not explore fine-grained design choices like number of heads, expansion ratios, etc.\nLimited Task Diversity: All benchmarks focus on commonsense reasoning and language modeling. There is no evaluation on longer-context tasks, dialog, or code, which could show broader generalization.\n4. Deployment Tuning Details Missing: While real-device latency is central to the paper, deployment tricks (e.g., CUDA Graph, TensorRT) might play a large role in the speedups. It’s not always clear how much of the gain comes from architecture vs. engineering.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the “useful insights,” “strong results,” and “practical significance” of our work, as well as for your constructive comments! We have addressed all of your questions below:\n\n**1. Novelty of our techniques**\n\nWe respectfully emphasize that the key contribution of this work lies in identifying critical factors affecting SLM real-device latency and offering generalizable insights and methodologies for latency-aware SLM design and training.\n\nSpecifically, while depth–width ratios and operator choices are known design factors, their impact on real-device latency and how to make effective trade-offs remain unclear, which are addressed by this work. Additionally, our weight normalization eliminates the training and inference overhead of nGPT (citation [28]) while achieving comparable or better accuracy, as analyzed in Section 2.3 of the main paper and Section 6.1 of the supplementary material.\n\nAs Reviewer ictF also noted, our work is “a decent attempt towards latency-optimal SLM” and has the potential to “produce significant community impact.”\n\n---\n\n**2. Variance across runs**\n\nSince we adopt ≥100B token training for all experiments that report accuracy, we find that the results are stable across runs. For example, we rerun the experiments in Table 4 of our manuscript three times and show all results below: the PPL variation across runs is within 0.1, and the accuracy difference is within 0.16%, leading to consistent conclusions. Therefore, we didn’t include error bars in the manuscript.\n\n||Run1||Run2||Run3||\n|---|:---:|:---:|:---:|:---:|:---:|:---:|\n|Model|Wiki PPL|Acc (%)|Wiki PPL|Acc (%)|Wiki PPL|Acc (%)|\n|Llama 1B|18.67|53.81|18.77|53.92|18.69|53.78|\n|Llama 1B + wnorm|18.03|54.85|18.06|54.80|18.03|54.80|\n|DeltaNet 1B|18.86|53.46|18.98|53.59|18.89|53.43|\n|DeltaNet 1B + wnorm|18.19|54.39|18.20|54.43|18.23|54.50|\n|Mamba2 1B|18.44|53.3|18.52|53.44|18.53|53.37|\n|Mamba2 1B + wnorm|17.88|54.71|17.84|54.55|17.82|54.68|\n\n*Note:* Acc is averaged over 8 commonsense reasoning tasks.\n\n---\n\n**3. Clarifications on the search cost and the search focus**\n\nFor the search cost, we clarify that each candidate architecture is trained for only **10B tokens**, which takes <4 hours using 16 A100 GPUs, as mentioned in **Section 5.2 of our supplementary material**. This follows the “realistic training budgets” you suggested. (The 100B token setting you referred to applies only to the architecture ablation in Tables 1 and 3 of our manuscript, not to the search process.)\n\nRegarding the search focus, our goal is to use evolutionary search to extend manual exploration of hybrid operators into an automated pipeline for obtaining more insights on LM operator combinations, rather than to build a comprehensive NAS framework. While our framework can support finer-grained searches with minimal changes, we deliberately avoid irregular architectures (e.g., with layerwise-varying #heads) as they may complicate deployment. Following your suggestion, we will explore larger design spaces in future work.\n\n---\n\n**4. Benchmarks on more diverse tasks: Math/Coding/Long Context**\n\nThank you for the question! We have added more benchmarks:\n\n**(a)** Math / Coding\n\nSince the paper submission, we have continued pretraining our models on an additional 2T tokens and benchmarked against baselines across 14 tasks, including Commonsense Reasoning (MMLU, Lambda, PIQA, ARCC, ARCE, HellaSwag, Winogrande, OBQA), Math (GSM8k, MathQA), and Coding (HumanEval, HumanEval-Plus, MBPP, MBPP-Plus).\n\nAs shown below, **our Fast-SLM-2.7B achieves SOTA average accuracy among all models**. Specifically, Fast-SLM-2.7B outperforms LLaMA3.2-3B/Qwen2.5-3B by 9.24%/0.90% in average accuracy with 1.46x/1.67x speed-ups, respectively. We will include these updated results in the final version.\n\n|Model|Param.|Dec. Latency (s)|Cache Size (MB)|CR (%)|Math (%)|Coding (%)|Avg (%)|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|AMD-OLMo-1B|1B|28.39|1049|49.21|12.66|7.35|32.03|\n|Llama-3.2-1B|1.2B|28.37|262|49.73|17.24|24.15|37.78|\n|Qwen2.5-0.5B|0.5B|28.19|98|48.13|32.66|32.06|41.33|\n|Qwen3-0.6B|0.6B|33.91|918|49.92|**36.88**|24.32|40.74|\n|Fast-SLM-960M (Ours)|0.96B|**19.36**|**36**|**52.54**|24.57|**38.44**|**44.51**|\n|h2o-danube3-4b-base|4B|61.94|737|60.85|34.73|11.65|43.06|\n|SmolLM2-1.7B|1.7B|41.55|1573|58.64|32.88|21.11|44.24|\n|Llama-3.2-3B|3B|60.38|918|59.37|30.07|34.80|48.16|\n|Qwen2.5-1.5B|1.5B|44.06|229|57.53|48.14|42.95|52.02|\n|Qwen3-1.7B|1.7B|46.39|918|58.62|53.71|43.76|53.67|\n|Qwen2.5-3B|3B|69.19|295|60.69|**53.83**|49.46|56.50|\n|Fast-SLM-2.7B  (Ours)|2.7B|**41.31**|**80**|**61.62**|48.88|**53.22**|**57.40**|\n\n\n**(b)** Long-context\n\nTo evaluate long-context tasks, we finetune Fast-SLM-2.7B (pretrained with a 4k sequence length) on 15B tokens using a 24k sequence length. We evaluate on the Needle-In-A-Haystack (NIAH) task from RULER, using 4k~32k input lengths. For reference, we also include results of h2o-danube3-4b (pretrained with an 8k sequence length). Both models use the NTK-based RoPE extension.\n\n|Input Length|Fast-SLM-2.7B|h2o-danube3-4b-base|\n|:---:|:---:|:---:|\n|4k|0.99|0.99|\n|8k|0.97|0.97|\n|16k|1|0.93|\n|24k|0.96|0.30|\n|32k|0.81|0.79|\n\nAs shown above, we observe that: \n\n(1) Fast-SLM-2.7B achieves strong NIAH scores, including reasonably good performance at the 32k input length; \n\n(2) Compared to the pure Transformer model h2o-danube3-4b, Fast-SLM achieves better scores on inputs ≥16k in length. These results highlight the promise of Fast-SLMs in handling long-context inputs. A fair comparison with more models requires aligning long-context training settings and we will include evaluations on more long-context benchmarks in the final version, given more time.\n\nWe also refer to recent large hybrid LMs, e.g., Nemotron-H, MiniMax-M1, and Jamba 1.6, that replace attention with efficient linear alternatives, significantly reducing cache usage while maintaining strong long-context performance. This underscores the potential of hybrid models for long-context tasks.\n\n---\n\n**5. Ablation study on deployment settings**\n\nWe first clarify that CUDA Graph has become a standard component in nearly all popular LM deployment pipelines (e.g., TensorRT-LLM, vLLM, SGLang). This is because LMs are significantly memory-bound during decoding, and without CUDA Graph, CPU overhead can dominate the latency. **CUDA Graph is indispensable for eliminating CPU overhead and ensuring reliable profiling of LM decoding.**\n\nAdditionally, **we integrated TensorRT-LLM’s attention kernel to enable CUDA Graph support for attention operators**, as the vanilla attention from HuggingFace Transformers cannot be captured by CUDA Graph. The reason we did not directly adopt TensorRT-LLM is that it does not support linear attention like DeltaNet. Therefore, we combined these components to build a unified decoding pipeline capable of running hybrid models with arbitrary operators using CUDA Graph. **This infrastructure is also a minor contribution of our work to the community.**\n\nFurthermore, following your suggestion, we have added an ablation study for each component:\n\n|Profile Setting|bs=1, out_len=8k|bs=1, out_len=8k|bs=1, out_len=8k|bs=1, out_len=32k|bs=32, out_len=32k|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|Deployment Setting|Vanilla|+ TRTLLM Kernel|+ TRTLLM Kernel + CUDA graph|+ TRTLLM Kernel + CUDA graph|+ TRTLLM Kernel + CUDA graph|\n|Qwen2.5-3B|390.32|379.08|69.19|289.70|817.46|\n|Fast-SLM-2.7B|296.32|292.63|41.31|166.82|357.19|\n|Speed-up|1.32x|1.30x|1.67x|1.74x|2.29x|\n\n\nThe table shows that:\n\n(1) The use of CUDA Graph can lead to 5x~7x gaps in decoding latency due to CPU overhead. This indicates that decoding with dominant CPU overhead, especially when it can be eliminated, is not a reliable profiling setting for GPU platforms;\n\n(2) Without CUDA Graph, the latency reduction of Fast-SLM is smaller, as linear attention incurs more extensive CPU workloads than standard attention (according to github issues in the official Mamba repo), further amplifying CPU overhead.\n\n(3) Further increasing the generation sequence length or batch size can lead to even greater speed-ups for our models.\n\nRegarding non-GPU platforms, they fall outside the scope of this work. However, readers may refer to the analysis in our work to profile their target platforms and derive device-specific SLMs accordingly.\n\n---\n\n**6. Discussions on safety issues with SLMs**\n\nThank you for the suggestion! First, prior to the public release, we will evaluate our model on safety benchmarks, including the Aegis AI Content Safety Dataset and Garak, to ensure its safety. Second, and more broadly, we agree that on-device SLMs lower the barrier to misuse and require more safety-aware tuning to ensure reliable deployment. We will include this discussion in the final version.\n\n---\n\n**7. Tool release for the depth-width scaling law**\n\nWe will release the codebase for deriving the depth–width scaling law, along with our search pipeline and deployment workflow. We hope these tools will help the community develop SLMs tailored to their specific needs.\n\n---\n\n**8. Benefits of Fast-SLM under small budgets**\n\nThe benefits of Fast-SLM hold even under smaller training budgets. As shown in Tables 3/5 of our manuscript and in the ablation study provided to Reviewer eYGu, Fast-SLM and its individual components improve accuracy or efficiency with ≤100B tokens.\n\nMoreover, Fast-SLM’s use of efficient operators enables faster training and fine-tuning, making it well-suited for dedicated downstream tasks.\n\n---\n\n**9. Other potential risks in cache reduction**\n\nAs noted in Bullet 4 of our response, Fast-SLM achieves higher accuracy across diverse tasks (e.g., commonsense reasoning/coding) compared to baselines, and maintains reasonably good long-context performance. This suggests that, for common downstream tasks, cache reduction does not hinder accuracy. Similar trends are seen in recent large hybrid LMs, e.g., Nemotron-H, MiniMax-M1, and Jamba 1.6."}]

Author rebuttal:

Meta-review: Based on the comprehensive review process and author responses, this paper presents a valuable contribution to efficient small language model design by systematically addressing the gap between parameter efficiency and real-device latency. The authors convincingly demonstrated the importance of optimizing depth-width ratios and hybrid operator selection through novel scaling laws and evolutionary search, supported by extensive experiments across multiple hardware platforms. While initial concerns regarding generalizability, task diversity, and novelty were raised, the authors addressed these through additional benchmarks on math, coding, and long-context tasks, ablation studies, and clarification of deployment dependencies. The resulting Fast-SLM models achieve state-of-the-art accuracy-latency trade-offs, with significant speed-ups and cache reductions compared to strong baselines. The commitment to open-sourcing models and methodologies further enhances the work's practical impact. The reviewer consensus supports acceptance given the paper's technical rigor, empirical validation, and potential to influence efficient language model deployment.
Decision: Accept (poster)