DND: Boosting Large Language Models with Dynamic Nested Depth
Authors: Tieyuan Chen
Xiaodong Chen
Haoxing Chen
Zhenzhong Lan
Weiyao Lin
Jianguo Li
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce Dynamic Nested Depth (DND), an efficient paradigm that adaptively identifies critical tokens and selectively deepens their computation via nested re-processing.

Abstract: We introduce Dynamic Nested Depth (DND), a novel method that improves performance for off-the-shelf LLMs by selecting critical tokens to reprocess in a nested depth manner. Specifically, at the end of the given transformer layer, DND identifies more critical tokens with a router and feeds them back for an extra round of processing, effectively "reviewing" difficult tokens while avoiding redundant computation for easier ones. The dynamic selection mechanism is tailored for precise control via two novel strategies: a router controlling loss to enhance token selection distinguishability, and a threshold control scheme to ensure selection stability. We demonstrate the effectiveness of DND by directly integrating it into pre-trained dense and MoE models during a post-training phase. On diverse benchmarks, DND boosts the performances of the dense Qwen3-1.7B, Llama3.2-1B, and Gemma3-1B by 1.88%, 2.61%, and 2.50% and the MoE Qwen3-30B-A3B by 0.87%, all with a minimal parameter and computing increase.

Published as a conference paper at ICLR 2026
DND: BOOSTING LARGE LANGUAGE MODELS WITH DYNAMIC NESTED DEPTH
Ticyuan Chen 12.3 Xiaodong Chen Haoxing Chen', Zhenzhong Lan Weiyao Lin Jianguo Li
'Shanghai Jiao Tong University, Incclsioon AI, Ant Group *ZhongguanCun Academy "Renmin University of China, *Westlake University
(t (tieyuanchen, aylin]asjtu.edu..
ABSTRACT
We introduce Dynamic Nested Depth (DND), a novel method that improves per- formance for off-the-shelf LLMs by selecting critical tokens to reprocess in a nested depth manner. Specifically, at the end the given transformer layer, DND identifies more critical tokens with router and feeds them back for an extra round of processing, effectively "reviewing" difficult tokens while avoiding redundant computation for casier ones. The dynamic selection mechanism is tailored for precise control via two novel strategies: router controlling loss to enhance token selection distinguishability. and threshold control scheme to ensure selection sta- bility. We demonstrate the effectiveness of DND by directly integrating it into pre- trained dense and MoE models during a post-training phase. On diverse bench- marks, DND boosts the performances of the dense Swen3-117BB, Llama3, 2- IB, and Gemma3-1D 1B by 1.88%, 2.61 %, and 2.50% and the MoE Qwen3-30B-A3B by 0.87%, all with minimal parameter and computing increase.
INTRODUCTION
Large Language Models (LLMs) have transformed artificial intelligence with their powerful abil- ities. The main strategy for improving them has been scaling, as empirical laws show that model performance predictably increases with more parameters, data, and computation (Achiam et al.] 2023 Team et al.] 2024 Yang al., 20255 Liu et al.) 2024). However, this scaling paradigm has significant drawbacks. The computational overhead for both training and inference grows expo- nentially with model size. This trend underscores critical need for more efficient approaches to enhance model performance beyond simple brute-force scaling.
A key insight from (Gloeckle et al.) 2024) is that prediction difficulty varies significantly across tokens; some are trivial to predict, while others demand deep computational processing. This dis- parity motivates token-level adaptive computation, where models focus resources the most critical inputs. A foundational version of this approach is token pruning, which has been shown be effective across language understanding (Bae al. 2025). model compression (Yang al.) 2022b), and vision (Hojjat et al. (025). By filtering out unimformatvv tokens, these methods reduce computational overhead and can even improve robustness by mitigating noise. This establishes bi- nary choice: a token is either discarded or processed normally. Furthermore, we propose the natural next step: instead of merely retaining challenging tokens for standard processing, we should allocate additional computation to them, ensuring these critical tokens are properly understood.
Our choice of how to allocate additional computation is inspired by latent strategies in test-time scaling (Hao et al.] 2024) (Sausshi et al.) 2025). Unlike using explicit output expansion like COT, these methods recur computation in hidden states, scaling inference without extra text generation. They observed that reasoning tokens place uneven demands computation: as illustrated in Fig. most tokens serve fluency, while a few critical ones drive complex planning or logical transitions.
Inspired by these two perspectives, we propose integrate token-level selection with latent-space deepening. Instead of uniformly applying extra recurrent depth to all tokens, we dynamically select
This work done when Tieyuan Chen research intern at Ant Group
t Corresponding authors.

---

Published as a conference paper at ICLR 2026
Q: "Two quontum states with energies El and E2 lifetime of see and sec, respectively. wont clearly distinguish these two energy levels, Which one could be their energy difference?
Shallow
Da0aa
Transformer Layer
00
Choices: 10
eV*
Share
Uiear Selection Router
"For state with finite lifetime, the energy uncertointy AE DE (28t).
Transforme Layer
Nested Depth
Figure 1: DND Motivation. The tokens highlighted in red denote critical elements in the QA pair. We propose a strategy within the transformer layers to identify and allocate additional computation to these critical tokens. L, and Le indicate the starting and end layers that adopted this strategy.
the subset of tokens that pose greater difficulty and reprocess them through transformer layers. This design not only concentrates additional computation the most critical tokens but also allows the model to refine their hidden representations through internal "review" iterations.
To achieve dynamic selection and recalculation of critical tokens, we propose novel architecture and training strategy. Specifically, as shown on the right of Fig. [ we incorporate linear layer as router at the end of the transformer block with DND strategies. To achieve more robust routing and avoid potential information leakage during the inference of auto-regressive LLMs (Raposo et al,, 2024), we adopt token-chotl routing strategy. In this approach, each token routed independently based on whether its output exceeds predefined threshold. The selected tokens are reorganized into a new sequence and are re-fed compute the dynamic nested output. Moreover, as DND is post- training method, we carefully design a normalized fusion strategy that integrates the dynamic nested output with the original forward output to preserve global pre-training knowledge.
In terms of training strategies, since our routing method treats each token independently, it lacks the precision of the top-k ratio routing when selecting tokens for recurrent computation (Raposo et al. 2024). To address this, we carefully design training strategies to control both the routers that determine token selection preferences and the thresholds that ultimately decide whether a token is selected. To enhance token selection distinguishability. we optimize the router's output distribution, encouraging the outputs across tokens to be distinguishable through a router controlling loss. To stabilize the token selection ratio during training, we adopt threshold control scheme, where the threshold is updated based on the error between the expected ratio and the actual ratio computed over sample buffer. Furthermore, to ensure more synchronized control, we update the threshold by blending it with the average top-k routing values using an exponential moving average (EMA).
Our method is a post-training approach that can be directly integrated into existing dense and (MoE) architectures. Experiments demonstrate its efficacy: by effectively se- lecting tokens, it substantially improves model performance across language, mathematics, rea- soning, and coding tasks. The effectiveness is validated on both three small-scale dense models, Qwen3-1.7B, Llama3.2-1B Gemma3-IB and a large-scale sparse MoE model, Qwen3-30B-ABB Overall, our core contributions are:
We introduce Dynamic Nested Depth (DND), efficient paradigm that adaptively identi- fies critical tokens and selectively deepens their computation via nested re- re-processing.
We design tailored training strategy with routing distribution control for token selection precision and an adaptive threshold control scheme for selection stability.
Extensive experiments show that DND can directly integrated into both dense and MoE architectures through post-training to achieve notable performance gains with minimal pa- rameter and computation increase.
2 RELATED WORKS
2.1 ADAPTIVE TOKEN SELECTION
Token-level adaptive selection is most commonly applied in model quantization and compres- sion (Yang et al. a a: Bondarenko et al.) (021). where Bayesian optimization methods are used to determine the appropriate compression ratios for tokens with varying levels of importance. This approach not only reduces computational redundancy but also mitigates the adverse effects of irrelevant tokens on the model's ' attention mechanism. Beyond the realm of model compression, to- ken selection has also been explored in the field of computer vision (Luo et al. 2025| Gadhikar al.,
2

---

Published as a conference paper at ICLR 2026
2024 Hojjat et al. 2025). Motivated by the fact that visual representations often contain significant redundancy- due to irrelevant background information and high similarity between neighboring researchrs have developed adaptive token selection strategies to address this issue. This strategy has been successfully applied to various vision tasks, including classification, detection, and retrieval. In general-purpose models, token selection is most notably employed in Mixture-of- Experts (MoE) architectures (Jiang et al.) 20244 Yang et al. 2025; Liu et al. (2024). where linear router dynamically assigns input tokens to specialized expert modules. Building on these insights, we propose the Dynamic Nested Depth (DND) approach, which adaptively selects tokens that are critical represent and dynamically extends the model's depth for their processing.
2.2 DYNAMIC REASONING DEPTH
Dynamic adjustment mechanisms for inference paths can generally be classified into two primary ap- proaches. The first branch focuses on reducing computational redundancy, such as early exit (Zhou et al., 2025; Leviathan et al. 2023) and MOD (Raposo al., 2024), which dynamically reduce the depth of computation layers to lower redundancy. The second branch investigates the test-time scaling law, which shows that repeatedly processing tokens within single layer can enhance final inference accuracy. A sophisticated variant of this approach is the Latent Strategy (Hao et al.) 2024 Saunshi et al.) (2025), where reasoning is carried out within hidden states-either by completing all steps before generating an answer or by leveraging recurrent inference to iteratively refine them.
The most closely related studies our work ITT (Chen al. 2025) and MOR (Bae al. 2025 both of which dynamically select a subset of tokens for additional computation and yield certain performance improvements. While sharing MOR (Bae et al. (025) 's goal of improving performance via dynamically increased computational depth. the two still differ fundamentally. MOR attempts to improve parameter efficiency during pretraining through a recurrent structure, which requires training a model from scratch over 200B tokens. This is extremely costly and makes it difficult to apply the approach directly to existing open-source SOTA models. In contrast, DND focuses on unlocking the potential of existing state-of-the-arl pretrained models and proposes plug-and- play post-training method. Besides, our work differs from MOR in model scale, training phase, architecture, and routing control (detailed in Appendix. Sec CCc.C.... MOR is limited to 1B (B-parameter, whereas our DND successfully scales to a 30B MoE model. We also address a key limitation in token selection control. Unlike MOR, which relies z-loss (Zoph et al., (022) for approximate load balancing, we achieve precise, stable token selection. Our method jointly enhances routing discriminability and adjusts thresholds via EMA synchrnnceo buffer errors.
METHODOL.OGY
Our method is primarily divided into two main parts: the model architecture design (Sec. 3.1 and the training strategies (Sec. 3.2). where we detail how implement dynamic nested depth (DND) and the carefully designed training strategies used to ensure the effectiveness of the architecture.
3.1 ARCHITECTURE
In the model architecture section, we will introduce how tokens are selected (Sec. 3.1.1p. how the new nested depth output is computed (Sec. 3.1.2), and how the vanilla output and dynamic nested output fused to obtain the final output (Sec. (.1.3). The whole architecture is shown in Fig. Moreover, we apply the DND strategy only to the intermediate layers of the model, keeping the initial and final layers unchanged to preserve the reasoning patterns learned during pre-training (Ma
et al. 2023 Xia et al. 2023). The layers where the DND is applied are denoted as from L, to L.
3.1.1
ROUTING DESIGN
When considering the routing paradigm, as shown in Fig. 3 (a), routing the entire sequence with ex- pert choice creates mismatch with the next-token prediction paradigm of auto-regressive models. This is because the full sequence cannot be accessed during early decoding without risking informa- tion leakage (Raposo al. 2024). To address this, we adopt token-choice selection, as illustrated in Fig. (b), which dynamically decides whether each token should undergo further processing.
Concretely, after initial forward pass through transformer layer, obtain the hidden states of the token sequence, referred to as the vanilla output X,. To determine token preferences for further computation, employ a router similar to that in MoE architectures, implemented simple
3

---

Published as a conference paper at ICLR 2026
Unselected Tokens
Vanilla
Dynamic Nested 0000
Tokens
Boster
Router
FFN
(1-
After first forward
Layer
loyer
Router
Nested Depth
(Share Weights)
X
X
Vanilla Output of Selected Tokens
Input
B
The Vanilla Pass
The Nested Pass
Normalized Fusion Strategies
Figure 2: DND Framework. The central idea of DND is dynamic nested pass of critical tokens after the vanilla forward process of the transformer layers. Whether token selected or not is determined by router. The block's final output is merged result of vanilla output and nested output, governed by normalized routing weights.
linear layer R Rd + R, where dmodal denotes the hidden size of the transformer model. For each token in the sequence, the router takes its hidden state from the vanilla pass, x R.dood as input and distributes a preference score. This score is then normalized using sigmoid function, a()). yield probability p E (0, 1):
o'=x(
The selection decision for each token is made independently by comparing its routing probability p against a pre-defined threshold т. A token i is selected for reprocessing if and only if p' т.
3.1.2 NESTED DEPTH DESIGN
Once the tokens for recurrence are identified, they undergo nested processing pass through the same transformer layer. We construct binary mask M according to the routing result, where each element mn is defined as:
1, if'' 0. ST
With the binary mask, the chosen states are assembled into a compact sequence for recurrent com- putation, refining the representations of the selected tokens. This process can be expressed
X [[, M) E.pos), M)
where X, are the output hidden states from this dynamic nested pass. The Pack X, selects tokens from the input sequence X, using mask M form compact subsequence. This subsequence is then given new positional embeddings Exps and processed by the i-th transformer layer L, Finally, the Unpack operator scatters the results back to their original positions within a zero-padded tensor, guided by the same mask M. This recurrence allows the model to perform internal "review" iterations, dedicating additional computational depth refine the representations of the critical tokens without altering the simpler ones.
M) operator
3.1.3 FUSION DESIGN
To ensure that the model effectively enhances the representations of critical tokens via the DND strategy while retaining the knowledge of global token interactions acquired during pretraining, propose a normalized fusion strategy. Specifically, we merge the outputs from the vanilla pass (X.) and the dynamic nested pass (X,) using gating mechanism. The final output X is computed
(8-p') x (1- --p') x,,
if >T
if ST
Here, x" refers to the merged hidden state of the i-th token. 3 is a learnable parameter that acts balancing factor between the original and nested paths. Similar (Raposo al. 2024). this
4

---

Published as a conference paper at ICLR 2026
Threshold
0.5
Edp
Gradient Vanish
K
(a) Expert Choice
(b) Token Choice
(c) Training strateggy towards
token choice selection
Figure 3: Routing Design and Training Strategies. Figure (a) illustrates expert-choice routing, where the top-k proportion is selected over the entire sequence. Figure (b) shows token-choice rout- ing, which selects tokens independently and suits auto-regressive models. Figure (c) summarizes our training strategy: routing outputs optimized to enhance token distinguishability by dispersing the token-level routing distribution via Ldd and preventing it from collapsing into gradient- vanishing re- gradient-vanishing
gions via Ldp- In addition, buffer proportional control (Eq. (9)) and EMA synchronization (Eq. (10)) effectively regulate the stability of the selection by computing the real-time error ratio.
fusion is modulated by the token's 's own routing score p', ensuring that tokens deemed more difficult (higher p') incorporate larger portion of their recomputed representation. This design provides smooth and adaptive integration, effectively stabilizing the learning process and allowing the model control the influence of the additional computation dynamically.
3.2 TRAINING STRATEGIES
Our training strategy is primarily designed to ensure that the model successfully learns to distinguish tokens in order perform recurrent computation. As our token-choice routing lacks the explicit ratio control of top-k mechanisms, we carefully designed strategies to control the two key factors in selection: the router's output distribution (Sec. 3.2.1 and the selection threshold (Sec. 3.2.2)
3.2.1 ROUTER CONTROLLING LOSS
A primary challenge in controlling the router is ensuring its output scores, p', are sufficiently distin- guishable. If scores cluster within a narrow range, the token selection process becomes unstable, as minor fluctuations in the threshold
cause drastic changes in the selection ratio.
To address this, as shown in Fig. 3 (c), we introduce a control strategy built upon a dual-objective loss function. The core idea is create dynamic tension between two competing goals:
1. Score Dispersion: We encourage the scores within sequence to spread out across a wide range. This makes the selection robust by creating clear distinctions between tokens.
2. Distribution Preservation:
We simultaneously constrain the scores to remain near the center of the sigmoid function's dynamic range. This ensures the router remains sensitive and responsive to its inputs, avoiding gradient vanish.
These competing objectives work in concert to produce a distribution of routing scores that is both discriminative and stable. We formulate the final routing objective as the total router loss, Crouters which is jointly optimized with the model's main cross-entropy loss. The loss is defined as:
Lrouter Cut + App
(5)
where Lad is the Score Dispersion Loss and Cap is the Distribution Preservation Loss. The hyperpa- rameters Asd and >ap balance the influence of each component.
Score Dispersion Loss. To counteract the tendency for router scores to cluster, we apply Score Dispersion Loss, Lus- This loss, based on information entropy, designed to push the score dis- tribution towards diversity at each targeted layer. For each layer in this range, we take its se- quence of N routing scores (!)) = {p-:(0) (0. N.(13 and normalize them to form a distribution: p'a-()) p)-(1)/ / The total loss is the sum of the information entropy from each layer, turning the goal of maximizing entropy into minimization problem for the optimizer:
Cat
(-H(D =
(.00 log(p" (.(1))
(6)
5

---

Published as a conference paper at ICLR 2026
This formulation incentivizes the router to produce diverse set of scores, making the routing output discriminative enough across tokens, therefore less sensitive to minor threshold adjustments.
Distribution Preservation Loss. While the router loss, Louterr promotes a dispersed distribution of routing scores, its reliance on a sigmoid activation function leads to vanishing gradients as outputs approach 0 or 1. This issue is particularly pronounced when the target selection ratio is low (e.g., 20%) or high (e.g., 80%). as the model may push many scores into the sigmoid's 's saturation regions. Consequently, the model may lose the ability to discriminate between tokens with scores near the decision threshold. To mitigate this, we introduce a Distribution Preservation Loss, Lap- which counteracts this effect by applying Mean Squared Error penalty scores that deviate from 0.5, thereby preserving gradient flow and enhancing discriminability:
Lap Σ I
(p' 1.00 0.5)²
where p²((9) is the score of the i-th token at layer 1. This objective effectively pulls the score distri- butions towards the center of the sigmoid's dynamic range, ensuring the routers remain responsive to changes in token hidden states.
Together, these two losses create balanced "push-pull" dynamic. The entropy-based dispersion loss pushes scores apart to cover a wider spectrum, while the MSE-based preservation loss pulls them collectively towards the responsive center. The result router that produces scores that are discriminative enough, facilitating more accurate and reliable token selection.
3.2.2 THRESHOLD CONTROL SCHEME
The preceding section introduced our method for enhancing the discriminability of router outputs. To further refine token selection, we propose dynamic threshold control scheme, which adap- tively regulates the threshold to achieve the desired proportion of selected tokens flexibly. Previous approaches using z-loss could only balance between selecting and not selecting tokens (Bae et al.) 2025). Inspired by the balance loss proposed DeepSeek- V3 (Liu et al. 2024), we design a method that computes the average selection ratio error in buffer to adjust the threshold accordingly. Addi- tionally, we leverage an EMA synchronization to assist in optimizing the threshold.
Buffer Proportional Control. Our primary mechanism for threshold control is loss-free method that makes real-time adjustments to the threshold т. For each mini- batch of training steps, this controller computes етгог signal, e, representing the deviation between the actual selection ratio during training and a pre-defined target ratio, kianget- é is formulated as:
EsEB No mb
6 Eves No
Atarget
(8)
where No is the number of tokens in sample b, and m² is the binary selection mask. Based on this error, the threshold is immediately updated via a simple but effective control law:
T + &
(9)
where a is a small step size (proportional gain). This mechanism provides immediate feedback to stabilize the selection ratio against short-term fluctuations. If too many tokens are selected (e > O), T increases to induce a decreasing selection trend; if too few are selected (e < 0), T decreases.
EMA Synchronization. While the buffer proportional controller excels rapid, local adjust- ments, its effectiveness can degrade when the optimization directions of routing and the threshold are misaligned. To prevent this drawback, we introduce an auxiliary mechanism that acts as a low- frequency synchronization loop. Periodically (e.g., every 50 steps), we compute a smoothed ideal threshold. Specifically, we maintain a buffer of the most recent N& samples. For each step this buffer, we calculate its corresponding Tinpk-the threshold value that would have precisely selected the target ratio. The average of these values, denoted as Tropk. serves as a more stable and robust estimate of the ideal threshold around these optimization steps. The operational threshold 7 is then gently nudged towards this averaged estimate using an exponential moving average (EMA):
= (1 - 7) T 7 Fropk
(10)
where y is a smoothing factor. This process ensures the router and threshold remain synchronized, preventing sustained periods of over- under-selection and promoting long-term training stability.

---

Published as a conference paper at ICLR 2026
Table 1: SFT Performance Comparison of Different Small-Scale Dense LLMs. Performing full- scale SFT with the DND strategy on the three widely used base models yields additional average improvements of 1.88, 2.61, and 2.50 points over full-scale SFT alone.
General Knowledge & Alignment
Math & STEM
Coding & Agent
Average
BBH
PIQA
C-Eval
MMLU
IFEval
GPQA
GSM8K
MBPP
Human BFCL.
MutuPL.F.
Qwen3-1.7B
59.53
40.82
75.38
60.00
64.11
65.47
28.54
79.38
68.38
61.59
58.73
52.45
+ ITT
59.58
41.23
75.99
59.51
64.92
65.08
27.85
80.29
68.67
60.78
58.62
52.45
+ DND
61.41
45.84
76.25
60.38
64.45
66.87
34,34
80.15
71.90
62.71
59.80
52.80
A (+-)
+1.88
+5.02
+0.87
+0.38
+0.34
+1.40
+5.80
+0.77
+3.52
+1.12
+1.07
+0.35
11ama3.2-1B
45.37
25.73
65.48
47.82
53.28
52.45
10.73
63.23
49.54
50.42
44.89
35.47
DND
47.98
29.43
66.51
49.21
55.63
55.68
14.59
66.57
52.91
52.16
47.52
37.56
(++)
+2.61
+3.70
+1.03
+1.39
+2.35
+3.23
+3.86
+3.34
+3.37
+1.74
+2.63
+2.09
Gemma3-IB
47.08
25.93
70.27
50.14
55.98
54.52
16,49
65.53
49.29
52.62
40.69
36.43
+ DND
49.58
30,62
71.33
51.00
58.04
56.91
21.79
68.68
52,96
53.76
42.88
37.41
A
+2.50
+4.69
+ +1.06
+0.86
+2.06
+2.39
+5.30
+3.15
+3.67
+ +1.14
+2.19
+0.98
EXPERIMENTS
4.1
EVALUATION BENCHMARK
We provide an extensive empirical evaluation of DND over a wide variety of benchmarks, demon-
strating its effectiveness and robustness. The evaluation suite covers three primary domains: 1.
General Knowledge & Alignment: MMLU (Hendrycks et al. 2020), CEyal (Huang et al. 2023)
CMMLU (Li et al. 2023), BBH (Srivastava et al.] 2022), DROP (Dua al., 2019), 1FEval (Zhou
et al. 2023), PIQA (Bisk et al. 2020). 2. Mathematics & STEM: Math (Hendrycks et al.
1 GSM8K (Cobbe et al. 1021). MATH-500, AIME24, GPQA-Diamond (Rein et al. [0233]
3. Coding & Agent: MBPP, MBPP+ (Austin et al.) 2021). HumanEval+ (Chen et al.
2021
C.BB-v(LiveCodeBenn)) v5) (Jain et al.] 2024). LCB-v6(Lived) CCCB-voLLiVeCodeBBenchhoo
(Jain et 2024
MultiPL-E (Cassano et al., 2022), BFCL v3 (Live).
4.2
TRAINING DETAILS
Our DND model undergoes standard full-scale supervised fine-tuning (SFT) using comprehensive and diverse dataset. with all parameters set as trainable and the same learning rate applied. Our training data incorporates significant volume of synthetic material built upon a high-quality seed set of 1-2 million instances curated from human annotations and open-source materials. The model's weights are initialized from the Qwen3-1.7B Base, Llama-3.2-18 Gemma3- 1B-pt, and Qwen3- 30B-A3B Base. Detailed hyperparameters and training settings are provided in Appendix Sec
4.3 MAIN RESULTS
Base Evaluation. As shown in Tab. [ our method achieves obvious improvements across the three widely used base models. Especially on datasets that require complex reasoning, such as BBH and GPQA, the performance boost is particularly notable, with all three models showing an additional performance improvement of around 5%. Additionally, we found that when the SFT is conducted with ITT (Chen et al. 2025) under the same computation cost, the performance improvement is not as pronounced. The limited performance gains stem from the use of Top-P- based token selection for nuto-regressive LLM, which introduces mismatch between training and inference, and may also lead to potential information leakage according to (Raposo et al.) 2024).
Scaling Evaluation. As shown in Tab. 2 our DND strategy consistently improves the performance of the Qwen3-30B-A- any performance degradation. The impact of DND is most pronounced in Coding and Agent tasks, yielding notable gains of +2.05 on BFCL v3, +1.42 on LCB-v6, and + +1.24 on LCB-v5. These results strongly support our hypothesis that DND effectively filters extraneous noise, allowing the model to focus its capacity on sparse, high-value tokens essential for complex reasoning, planning, and code generation. Importantly, the benefits are not limited to specialized domains: substantial im- provements are also observed in General and Alignment tasks (+1.83 on C-Eval), alongside robust generalization on challenging Math and STEM benchmarks. Crucially, these substantial perfor- mance gains are realized with negligible increase of only 0.03M parameters, highlighting DND as a highly parameter- effiiiee approach for unlocking significant capabilities in LLMs. FLOPs and Throughput Evaluation. As demonstrated in Appendix Sec. A reviewing 20% of the tokens adds
model, achieving an average gain of +0.87 across 17 benchmarks without
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance Comparison of Qwen3-30B-A3B with and wothout DND. The final col- umn shows the difference (A) between the SFT results of vanilla Qwen3-A3B-308 and Qwen3- A3B-30B+DND. And we list Qwen3-32B and Qwen3-30B-An Chat model for reference.
Qwenn-32B (Non-Thinking)
OwBn3-30B-A3B
Qwen3-A3B-30
Qwem3-A3BB330--DNN
A
Task
Non-Thinking)
(SFT)
(W w/o DND)
General & Aligument Tasks
MMLU
82,93
80.12
85.41
85,91
+0.50
CMMLU
84.63
83.13
84.82
85.19
+0.37
BBH
85,45
82.55
86.90
87.03
+0.13
DROP
84,02
86.38
86.21
86.48
+0.27
85.95
92
IFEval
85,27
84.55
83.09
84.31
+7.22
Mathematic Tacks
MATH
85,26
84.68
88.63
88,78
+0.15
MATH-500
87,40
88.70
92.60
92.80
+0.200
GSMRK
94.54
95.30
94.30
95.10
+0.80
27.71
GPQA-Diamond Diamond
$3.60
51.71
56.76
57.67
+0.91
Coding & Agent
HomanEval+
82,93
84.15
85.59
86.58
+0.99
MBPP+ MuliPLE EE
72.75
75.16
78.
+0.70
68.62
66.04
72.60
73.72
+7.12
LiveCodeBench
LiveCodeBencch v6
28.57
29.43
31.14
32.56
+7.42
BFCL (Live)
73.69
75.43
Average
72.81
73.44
75.70
76.57
+0.87
Table 3: Comparison of Speed across Different Input and Decode Lengths. measured using BF16 Quantization of LLM and accelerated with vLL.M kernels.
The speeds are
Speed (tokens/s)
Input Length
Decode Length
Relative Speed
Qwen3-30B-A3B
+ DND
(%)
1024
2048
148.68
136.19
91.6
6144
2048
208.60
193.51
1024
6144
76.16
70.52
92,6
6144
6144
100.89
93.93
93.1
only about 6% extra FLOPPs when applying the DND strategy to the Qwen3-30B-A3B model. To further assess the inference speed of our model practical settings relative to the baseline, follow- ing SGLang, shown in the Tab. models under four standard sequence lengths using a single H100 GPU with single batch. The measurement results in the table show that, while achieving performance improvements, our model consistently reaches 91.6-93.1 % of the speed of the vanilla model under different circumstances.
measured the throughput of both Qwen3-30B-A33 and
4.4 ABLATION STUDY
As shown in Tab. 4 we conducted ablation experiments of the DND strategy Qwen3-1.7Ht Training Strategies. We conducted ablation experiments on the proposed training strategy. We found that when using only the DND framework with a simple z-loss-like method to control token selection, performance dropped noticeably, yielding an average improvement only 1.01 points Qwen3-1.7B's SFT performance. This highlights the importance of our carefully designed training strategy. Moreover, router and threshold control function as complementary components for token selection control. While each method individually provides marginal gains, their combi- nation leads to a clear improvement of approximately one percentage point in average accuracy.
syper-parameters of Architecture. Besides, we conducted ablation experiments on several im- portant hyperparameters of the model architecture. For the expected token selection ratio, we tested 10%, 20% and 30%. We found that when only 10% of tokens were selected, the number of tokens participating in attention computation was likely too small, resulting in modest improvement of just 0.8% over the baseline. contrast, selecting approximately 20-30% tokens achieved better performance. To balance computational efficiency, we chose expected selection ratio of 20% for our DND method in the Qwen3-30B- -3BB scaling experiments. We also performed ablations on the number of shallowest and deepest layers retained in the original architecture, finding that keeping about four layers at both the beginning and the end yielded the best performance. This configuration was retained in the DND experiments Qwen3-30B-AN
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study of Qwen3-1.7B with DND under different settings. TC indicates thresh- old control (ineluding buffer proportional control and EMA synchronization), RC indicates router control (including proposed Sd and Ldp), layer (L, L.) indicates the layers with DND.
Metric/ / Setting
Qwen3-1.7B
-DND
+ND +NN
+DND
.ND
+ND
+NN
+ND
(Cuy and Cas) (BPC& EMA)
x
X
(x layer
20
20
20
20
10
30
20
20
23
Average
59.53
61.41
60.54
60.58
60.68
60.33
61.03
61.05
60.36
A (+)
0.00
1.88
1.01
1.05
1.15
0.80
1.50
1.52
0.83
BAH
40.82
45.84
44,69
43.47
44.37
44.95
45.17
44.85
44.95
C-Eval
60.00
60.38
60.21
60.38
60.17
59.60
60.32
60.44
59
MMLU
64.11
64.45
64.35
64.45
64,26
64. 38
64.66
64
64.56
GPQA-D
28.51
34.34
29.92
31.94
32.79
29.17
35.82
33.82
29.26
PIQA
BPCI-
58.73
$9.80
59.21
59.32
59.50
59 19.40
59.92
59 (9.66]
$9.40
IFEval
65.47
66.87
66.55
66.35
66.19
65.99
67.27
67.31
65.99
GSMMK
Humaneval+
61.59
62.71
61.59
61.59
61.82
62. 15
62.80
62.96
62.15
MBPP
68.38
71.90
70.73
69.84
69.79
69.32
69.09
69.56
69.32
MuliPIE
$2.45
53.05
Rebeticnship Tokon Selection Light Entropy
Relationship between Token Selection Count and Legit Ensrops Difference
Thend yoiiin::--00m
Cimernfe Decoding
Ube.y www.y- +0.25
Token Geamael during Deodling... Thred
leaad
IIADD Vonitaa
Peancety -223831
3M-
0.0
2.5
3.0
2.55
1.
12.5
17.5
Teluna Schutionn Cousst
Tokica Sdlctiion Crast
Figure 4a: Entropy of Selected Tokens. The selec- tion frequency of token correlates positively with its original logit entropy in the vanilla pass.
Figure 4b: Logits Variation after DND. For rarely selected tokens, logit entropy fluctuates evenly, but it decreases as selection frequency increases.
4.5 TOKEN SELECTION ANALYSIS
Why critical tokens are selected? As shown in Fig. 4a to examine whether the tokens selected by our DND model are indeed critical, we analyze the relation between the selection frequency of tokens routed by cessing. According to the findings in (Ma al., 2025). high entropy in the logit indicates that the model is uncertain about which vocabulary to select or is hesitating between multiple possible an- swers. We find that tokens with higher logit entropy are frequently selected by the router across multiple layers. This shows that DND preferentially selects tokens with greater uncertainty, validat- ing the motivation behind critical token selection and confirming the effectiveness of the router. Why better representations are learned? Furthermore, shown in Fig. 4b to validate that our DND strategy reduces the model's hesitation or uncertainty about critical tokens, we evaluated the variation in logit entropy for the same token after applying the DND strategy, compared to the en- tropy in the vanilla model. We found that after nested reviews, the logit entropy of the selected tokens significantly decreased, proving the effectiveness of our method.
and the vanilla model's logit entropy without any repro-
Ims EMA suachrenitation
Threshold Visualization during Training. As illustrated in Fig. 5 we analyze the threshold dynamics of the 24th layer in our DND (Qwen3-30B-A3B, model by presenting the effect of two different approaches for threshold control. Adopting buffer proportional control alone exhibits criti- cal tuning challenge: a small adjustment factor (purple line) causes the threshold adapt too slowly, consistently fail- ing to reach the target selection ratio and thereby impairing early training performance. Conversely, an excessively large factor (pink line) leads to volatile oscillations around the tar- get, which compromises stability. In contrast, incorporating
Sunchmolnsiinn ferter.
0.18
with EMA Synchronioaiin. IMA Synchmoinstionn large Dicter.
Trrsbold Volae
611
8.11
8.12
6.18
Training Step
Threshold Adjustment during Training. With EMA synchro- our EMA synchronization (blue line) enables rapid thresh- nization, the threshold can be adjusted old adjustments, maintaining synchronization between the smoothly and in real time.
Figure 5:
router and threshold, thereby ensuring stable selection.

---

Published as a conference paper at ICLR 2026
Impact of Buffer Proportional Control Selected Ratio
Impact of Router Controlling Selected Ratio
Frrir (Selected Ratio- Targe TargeRRatin Ratio)
Error (Sdlcted Ruliee Targue Ratho)
outh Butfer Proportional Control
0.20-
Withret Router Controlling Loss
winh Butter Proportioaal Control
0.15-
With Router Controling Lows
0.19
0.10-
0.05-
0.05
0.00
0.00
-0.05
-0.10-
-0.05
100
300
409
100
200
300
469
Training Step
Tralninng Step
Figure 6a: Selected Ratio Comparisons. The aver- age selection ratio is stably controlled with our pro- posed buffer proportional control strategy.
Figure 6b: Selected Ratio Comparisons. Oscilla- tions arising from insufficient discrimination are re- duced by introducing the router controlling loss.
Selected Ratio Visualization during Training, As shown in Fig. 6al introducing EMA synchro- nization alone struggles to regulate token selection, as its non-real-time adjustments often misalign with the model's ' optimization trajectory, resulting in large and persistent oscillations. By introduc- ing buffer proportional control, we provide a high-frequency corrective loop. As a result, oscillations are rapidly suppressed to within tight 5% band, successfully stabilizing the selection. Besides, as shown in Fig. 6b] introducing the router controlling loss significantly reduces the magnitude of os- cillations, demonstrating its effectiveness in enhancing the discriminability of token routing outputs.
Selected Ratio across Transformer Layers with DND
Q: "Two quantum states with unergies El and E2 have lifatime of 10*-0 sec and 10--8 sec, respectively We want to clearly distinguish these two energy levels, Which one could bo their energy difference?
Shadow Layers
Midale Layers
Deop Layors
Dataset
0.5
Drop
Choloes: A. 10 eV B. 10h-11 eV C. 10--4 eV D. 10 ev"
KBPP
0.4
FEval
A: state with fieite lifetime, the energy uncertainty approximately: h/ (200)
0.5
state 1: lifttime &r. 100 "s AE, =h/ (RMz. (6.552 10 *eV
**) 3.29
For stain 2: lifttime dd, - 10- ** AE, = h/ (20t, J- (6.562 x 10- VVia (2 10" *s) 10 TVV
0.1
0.0
10
12
14
16
24 26 Layer Index
34
36
To resoivve two energy levels, the energy difference AE between them should least comparable to larger the two uncertainties. only option *aV) large enough resclved.
Figure 7a: Selected Ratio during Evaluation. Token se- lection ratio tends to be slightly higher the middle layers as well as in the shallowest and deepest layers.
Figure 7b: Visualization Results. Highlighted tokens are frequently selected, with darker red indicating higher selection in deeper layers.
Selected Ratio Visualization during Evaluation. To validate our training methodology, we evalu- ated the DND model's inference-time behavior across diverse suite of benchmarks, including agent tasks, reasoning, and code generation. The average token selection ratios across all layers consis- tently aligned with our target, ranging from 0.178 to 0.242, which confirms the effectiveness of our whole control strategy. A more granular, layer-wise analysis in Fig. 7a reveals a nuanced pattern where selection is slightly elevated in the middle layers and the first layers from each end. Comple- menting this quantitative success, a qualitative visualization of the GPQA dataset in Fig. 7b reveals an interesting phenomenon. Tokens selected by shallower layers (lighter colors) are predominantly essential nouns, while those selected by deeper layers (darker colors) correspond to more abstract or syntactically critical components like mathematical expressions and key verbs. This suggests that the model learns a hierarchical processing strategy, using earlier layers to identify key entities and later layers to perform more complex relational and logical operations.
5 CONCLUSION
In this work, we introduce Dynamic Nested Depth (DND), a novel and efficient method for enhanc- ing Large Language Model performance. DND adaptively identifies critical tokens and selectively deepens their computation via nested re-processing. This is achieved through token-choice routing design with a normalized output fusion strategy. The precision and stability of this selection pro- cess are guaranteed by our router controlling loss and threshold control scheme. We validated DND on both dense (Qwen3-1.7B. Llama3.2-11 1BB, Gemma3-13 1B) and larger-scale sparse MoE (Qwen3- 30B-A3B) models, demonstrating substantial accuracy improvements with a negligible parameter increase (< MM) and minimal computing increase. These results affirm that targeted, dynamic nested depth computation is a powerful method for boosting L.MSs performance.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
The paper is supported in part by the National Natural Science Foundation of China (No.6232510, 62595733, 62561 160155), the Shanghai 'The Belt and Road' Young Scholar Exchange Grant (24510742000), and in part by the Ant Group Research Intern Program.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087744, 2023.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Do- han, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. ArXiv, 0tbs/2108.07732 2021. URL https://app. semanticscholar htt.:/tc.......oooo.oorraaaaa2..
Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, et al. Mixture-of-recunse sive depths for adaptive token-level computation. arXiv preprint arXiv:2507.1052. 2025.
Learning dynamic recur-
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint arXiv:2 10t.:220999/29..... 2021.
Federico Cassano, John Gouwar, Daniel Nguyen, Sy Duy Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. Multipl-e: A scalable and extensible approach to bench- marking neural code generation. 2022. URL https: app www.seeeemnttooolee.eeo org/
CorpusID:2548544
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374, 2021.
Yilong Chen, Junyuan Shang. Zhenyu Zhang, Yanxi Xie, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, and Haifeng Wang. Inner thinking transformer: Leveraging dynamic depth scaling to foster adaptive internal thinking. arXiv preprini 1rXXv:2502.13844 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 2021.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.0011 2019.
Advait Gadhikar, Souptik Kumar Majumdar, Niclas Popp. Piyapat Saranrittichai, Martin Rapp, and Lukas Schott. Attention is all you need for mixture-of-depths routing. arXiv preprint arXiv:2412.20877 2024.
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Syn- naeve. Better & faster large language models via multi-token prediction. arXiv preprint arXiv:2404. 19737. 2024.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu. Jason Weston, and Yuandong Tian. Training large language models to reason a continuous latent space. arXiv preprint arXiv:2412.06777 2024.
11

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song. and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, 0as/20009033000
2020. URL
22151 6475
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Xi- aodong Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ArXiv, abs/2103.03874, 2021. URL https: /ttp//aaii... wwwp..semattiiccccccci..i
org/
CorpustD: 233
Ali Hojjat, Janek Haberer, Soren Pirk, and Olaf Landsiedel. Thinkingvit: Matryoshka thinking vision transformer for elastic inference. arXiv preprint 1XXv:2507.10800 2025.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang. Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322 08322. 2023.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. ArXiv, abs/2403.079 079944, 2024. URL https:
//api.semantion. cittmaanticcholarr. htt.:o/la.....ooooooooooa:aa211.
Albert Q Jiang, Alexandre Sablayrolles. Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024.
Mixtral of experts. arXiv preprint arXiv:2401.0400
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMIR, 2023.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timo- thy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306. 202220922622.... 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 14X44.2412.19437, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, International Confer- ence on Learning Representations, 2017. httph//cppe/pppp..setttttttoooo. weemantiicccolla..oooom
Siqi Luo, Haoran Yang, Xin, Mingyang Yi, Guangyang Wu, Guangtao Zhai, and Xiaohong Liu. Tr-pts: Telk-relevant parameter and token selection for efficient tuning. arXiv preprint arXiv:2507.2287.. 2025.
Huan Ma, Jingdong Chen, Joey Tianyi Zhou, Guangyu Wang, and Changqing Zhang. Estimating Ilm uncertainty with evidence. arXiv preprint arXiv:2502.00290. 2025.
Xinyin Ma. Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023.
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer-based lan- guage models. arXiv preprint arXiv:24044002555 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level grogle-proof q&a benchmark. ArXiv,
2023. BtLhhtts:://ae appi. www.ntiesccoooooooooo.o hot.csccoolla.ooooooooooooo
Corpus ID: 652 95009
Nikunj Saunshi. Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank Reddi. Reasoning with latent thoughts: On the power of looped transformers. arXiv preprint arXiv:2502.17411
2025.
12

---

Published as a conference paper at ICLR 2026
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alono,, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint MxXi::2206.04115. 2022.
Gemini Team, Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. The Twelfth International Conference on Learning Representations, 2023.
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.00 0988:82505.09388 2025.
Tao Yang, Dongyue Li, Zhuoran Song. Yilong Zhao, Fangxin Liu, Zongwu Wang, Zhezhi He, and Li Jiang. Diqatten: Leveraging dynamic token-based quantization for efficient attention archi- tecture. In 2022 Design, Automation Test Europe Conference Exhibition (DATE), pp. 700-705. IEEE, 2022a.
Tao Yang, Fei Ma, Xiaoling Li, Fangxin Liu, Yilong Zhao, Zhezhi He, and Li Jiang. Dtatrans: Leveraging dynamic token-based quantization with accuracy compensation mechanism for ef- ficient transformer architecture. IEEE Transactions Computer-Aided Design of Integrated Circuits and Systems, 42(2):509-520, 2022b.
Huixue Zhou, Hengrui Gu, Xi Liu, Kaixiong Zhou, Mingfu Liang, Yongkang Xiao, Srinivas Govin- dan, Piyush Chawla, Jiyan Yang. Xiangfei Meng. al. The efficiency vs. accuracy trade-ofi Optimizing rag-enhanced Ilm recommender systems using multi-head early exit, arXiv preprint wwXiv:2501.02173. 2025.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instuuction-following following evaluation for large language models. ArXiv, abs/2311 1.0791 2023. URL/attps://apt
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, arXiv preprint arXiv:220..08906, 2022.
13

---

Published as a conference paper at ICLR 2026
A COMPUTATIONAL OVERHEAD ANALYSIS OF THE DYNAMIC TOKEN RECALCULATION STRATEGY
We conduct a formal analysis of the computational overhead introduced by the dynamic nested depth strategy within Qwen3-MoE architecture. The strategy aims to refine the representations of critical tokens by performing a secondary, partial forward pass. Our objective is to quantify the percentage increase in Floating-Point Operations (FLOPs) for scenario with sequence length of 16,384 tokens (5 = 16, 384), where 20% of tokens (r = 0.2) are selected for this recalculation. The DND mechanism is selectively applied the 40 intermediate layers of the 48-layer model, excluding the initial four and final four layers.
The FLOPs for single decoder layer are dominated by the self-attention mechanism and the fixture-of-Experr (MoE) feed-forward network. The computational complexity of self-attention is quadratic with respect sequence length (FLOPSatn d, while the MoE MLP complexity is linear (FLOPSmes Smoe ~ 6Sk mmse) The additional computation from the DND policy's 's nested pass, processing fraction of the tokens, is therefore FLOPS .FLOPSSau (S) + F-FLOPs (S).
The analysis is based on the parameters detailed in Tab. 5
Table 5: Key Model and Strategy Parameters
Parameter
Symbol
Value
Sequence Length (original)
16,384
Hidden Size
H
2,048
Number of Attention Heads
Np
32
Head Dimension
dp
128
Total Decoder Layers
L Liotal
48
Layers with DND enabled
Land
40
MoE Intermediate Size
mos
768
Activated Experts per Token
k
8
Recalculation Ratio
r
0.20
For standard layer operating on a 16k sequence, the self-attention component requires approxi- mately 4.40 X 10 FLOPs, and the MoE MLP requires 1.24 x 10a5 FLOPs, totaling 5.64 10 FLOPs layer. The additional computation from the nested pass is calculated as (0.2)² (4.40 x +
10 (0.2) (1.24 x 10 0.424 x 10 FLOPs. This constitutes per-layer overhead of 7.52%
for the OND-enabled layers. When scaled across the entire model, the total computational overhead proportional to the fraction of layers implementing the strategy.
FLOPS
Land 7.52% X
40
Overheadiotal
~ 6.27%
FLOPStayer
Liotal
48
In conclusion, the DND strategy with 20% token recalculation ratio results in modest total com- putational overhead of approximately 6.27% This increase in FLOPs should be weighed against the potential gains in model performance that the targeted recalculation may provide.
B MORE DETAILS
Hyper-parameters. For the value of L, and Lg, we determined through ablation experiments that L, and L, 43. terms of the proportion of tokens review, we balance performance and efficiency through ablation experiments on the Qwenl.78 1.B model, ultimately setting the kunget to 20%. For the initialization the parameter 3, to prevent overly gentle learning of the DND logic in the early training phase, we initialize it to 0.1. Regarding the hyperparameters defined in the training strategy loss function, due to the approximate 60x magnitude difference between the two loss extremes, are set to 5 and 5e-3 to balance stability and real-time performance, and y is set to 0.2. To ensure stability during initial training, initialize token routers with all zeros and set the threshold to 0.5.
set As to 3e-4 and Aap 0.02. The buffer size N₂ and the adjustment factor
14

---

Published as a conference paper at ICLR 2026
Table 6: Differences between our DND approach and MOR. Both methods aim improve per- formance via increased computational depth, but differ significantly in implementation and control.
Aspect
MOR
Ours (DND)
Validation & Scaling
Application Stage
Pre training (Train from scratch).
Post Post-iraining (SFT).
Model Scale
Limited to models with uround IB parameters.
Applied from IB dense to 30B MoE models.
Achieve comparable performance with fewer purameters by using dynamically cycled depth under the same FL.Pss budget.
Achieve better performance with almost unchanged purameter by adding only small amount of extra depth computation.
Purpose
Requires from-scrutch pre-truining and substantial architectural changes applying.
Strong compatibility; Can be directly applied existing models post-training.
Compatibility
Methodology
Employs normalized fusion strategy combine dynamic and original outputs.
Architecture
Dynamic depth ourput is treated final state.
Introduces a precise ratio control method vin discriminative routing output and dynamic thresholds.
Token Selection
Relies z-loss, indirect load balancing.
Flexibly regulates the token selection proportion to any specified target.
Control Precision
lacks precise control over the ratio differenn depths.
Training Details. The post-training stage uses the AdamW optimizer (Loshchilov Hutter, 2017). with 31 0.9, 32 0.95, weight decay of 0.1, and gradient clipping at 1.0. During the post-training stage, we employ a cosine learning rate scheduler with learning rate of x 10-6 that gradually decays
minimum of 1 x 10-6 All experiments Owen3-1.7B. training is conducted for two epochs across 128 GPUs, taking one day. For Qwen3- 30B-A3B, training conducted for four epochs across 256 GPUs, taking approximately three days. For Llama-3.2-11 3..-1BBBBBBBBBBBB and 1emmma-1B, training is conducted for two epochs across 64 GPUs, taking approximately 1-2 days.
H800 GPUs. For
Evaluation Metries.
The evaluation includes several specialized testing protocols:
AIME Evaluation: On AIME24, we run inference 16 times per question for each model and report the average accuracy.
Fvll Scoring:
The final score for IFEval is the average of the strict accuracies at both the prompt and instruction levels.
C DETAILED COMPARISONS WITH MOR
As shown in the Tab. 6 our DND differs significantly from MOR in terms of validated model scale, compatible training paradigms, model architecture, and token selection control strategies.
D MORE TOKEN SELECTION ANALYSIS
Beyond the entropy analysis of token logits Sec. 4.5 we further examined another selection pattern of DND across layers. Specifically, we measured the variation in each token's hidden state across transformer layers. As shown in Fig. g we found that DND tends to select tokens whose representations change more strongly after passing through certain transformer layer. This suggests that the model may be less confident about such tokens and therefore chooses to revisit them obtain more reliable representation.
E MORE VISUALIZZATIONS
In the main text, we present visualizations of token selection by DND general language tasks. In this section, we further provide visualizations for mathematics and coding tasks Fig. g and Fig. 10, As shown, DND selectively allocates additional computation to key intermediate results and calculation steps in mathematics, while focusing on critical variables and essential logical comments in code. These observations align with motivation and further validate the effectiveness of DND.
15

---

Published as a conference paper at ICLR 2026
The Relation between Representation Similarity and Selection
1,0000
0,9975
0,9950
Simularity
0.9925
0.99M0
0.9875
0,9850
Not Selected
Selected
DND Decision
Figure 8: Analysis of the Relationship between DND's Selection Preference and the Magnitude of Hidden State Changes across Transformer Layers. DND exhibits stronger preference for selecting tokens whose representations undergo larger changes after passing through given layer.
hand. seagull landed, he gobbled down half the amount of french fries that Dave ate. Than three pigeons builied him away from the food, and each pigeon three fries. How many french fries were in the pack when Dave bought it?
Question: bought pack
hungry seagull
thirds
off fry, leaving
Let's think stop Answer:"
Prediction: "Let's work through this stop by stop.
Let the total number of franch fries the pack ba
Dave ate 14 fries. remaining after
14
The seagull ate half the amount Dave ate: Half of 14 fries So, seaguil ate fries. Remaining after seagull: (x- - 21 fries: pigeons fries fries
Three pigeons each
pigeons Remaining after pigeons: -x
A raccoon stole two two-thirds of the remaining fries:
Raccoon took (2/3](x
two-thirds
thirthird remains: Remalning after raccoon: (1/3)(x 30)
Ants carried off 1 fry, leaving behind: So, before ants took 1, there were 5 fries Therefore: (1/3)/x- 6
Now salve Multiply both sides
So, the originally had 48 french fries.
Let's verify:- Start: 48 Dave eats + = 34 Seagull eats 7 34 27 Pigeons eat 9 27-9-18- Raccoon steals of 12 18 12= AnAss take 6- left All matches. The answer is 48"
Figure 9: Visualization Results of A Mathematical Example. Highlighted tokens are frequently selected, with orange indicating higher selection frequency in shallow layers and darker red repre- senting higher selection in deeper layers.
F FAILURE CASE ANALYSIS
As shown in Fig. we present a representative example in which Qwen3-30B-A3H answers cor- rectly, but 300---3--BA fails. In this example, the task is to match foods to their corre- sponding meals based on the user's description. Under the DND "reviewing" critical tokens mode, the temporal cue "Earlier" in the prompt is excessively reviewed, which leads the model to overem- phasize earlier times of day (roughly morning and noon). As result, incorrectly associates coffee with breakfast and pizza with lunch, and answers "breakfast and lunch" instead of the correct, un- known category "snack."
G THE USE OF LARGE LANGUAGE MODELS (LLMS)
Large language models (LLMs) were used as writing aid in the preparation of this manuscript. Their use was strictly limited to improving grammar, phrasing, and formatting. All scientific content, including research ideas, methodology, data analysis, and conclusions, exclusively by the authors. The authors have reviewed all Al-generated suggestions and take full responsibility for the final manuscript's accuracy and integrity.
conceived and developed
16

---

Published as a conference paper at ICLR 2026
Question: are given a sequence of N positive integers
Find the number of pairs of integers (l.r) satisfying 1\leq (\leq r\Veq N such that the subsequence
(A.loo_ forms an arithmetic progression.
arithmetic progression and only there d such that [[+1]xx_iaa j=dli_iiVVVV__ (A\ileq << [x]).
sequence
2,)doos,x_ [[x[])
In particular, a sequence of length is always an arithmetic progression.
Enclose your code within delimiters as follows.
ppthon
YOUR CODE HERE
NBM Answer: (use the provided format with backticks)"
Prediction: .". python
Read input
int(Input()
A list(map(Int,, input)).split())
count progressions
# Every single element an arithmetic progression count
Check all pairs of elements (length 2) for in range(N- 1]: count
Ali] == A(i 1]:
Check all subsequences length for in range(N):
for range() 2, NJ: #length >=
Check A[kr=1] arithmetic
d All 1] -[]]
a..rrthteeiic True
for in range(1+ + r):
Ali 1) A[i]
fs_arithmetic
break
is__ 1s_arithmetic:
count 1=
print(count)
Figure 10: Visualization Results of Code Generation Example. Highlighted tokens are fre- quently selected, with orange indicating higher selection frequency in shallow layers and darker red representing higher selection in deeper layers.
"gold"""[log_ "["lo__fodd' (food_ name': (frozen mango'). 'meal_ 'maal_type": ['snack]]], ("og_ pog_'ood': (food_ name': [cchal ten'), 'meal_type' ("snack]]), (log_ food: (food (food_names _na_an ['pepperoni pizza'). 'meal Eypel ['snack]]], (log_food) (food_ hadddd_amei [ccoffee]] 'meal, aneal_type:: ['snack]]]"'
"prediction': "loo food/food name="frazen mango', meal type="snack"), log, foodifood _na_nname"da chai tea", meal "ype="snack",, log_ fog_fooddfood name="pegperoni pepperoni pizza", meal type="lunche log food/food namer"coffer meal, meealllyeess "opp=="breaaass"]]""
"origin p
{
"content": "You given list foods someone today: apple plece) and coffee (1 cup). For each food, decide which meal lunch, dinner, snack. with Python strings, "role": "system"
type belongs
food, order.".
"content": had pleces of frozen and chai tea.inin Earller had two siices of pepperoni pizza and coffee'
Figure 11: Failure Case of DND. DND may make errors by over-interpreting certain latent weak pieces of evidence tokens.
17

---

Published as a conference paper at ICLR 2026
H LIMITATIONS AND FUTURE WORKS
Our experiments primarily focus on validating the effectiveness of the DND strategy during the post-training stage of LLM. Its impact in pre-training continual pre-training settings remains to be further explored.
While we have applied DND in large auto-regressive models, exploring its applicability to other types of LLMs, such as diffusion-based LLMs, may be promising direction for future research.
Our subjective analyses reveal layer-wise preferences for different token types and show that the final token selection ratios vary across layers. Such observations could offer valu- able insights for the design of future models.
18


Reviews:
[{"review_id": "VBvN9Z90Wj", "reviewer": "Reviewer_pkVU", "summary": "This paper proposes Dynamic Nested Depth (DND) — an efficient method to enhance pre-trained LLMs by selectively deepening computation. Instead of uniformly processing all tokens, DND identifies critical tokens that require more reasoning effort. Within each transformer layer, a lightweight router marks these important tokens after a standard forward pass. Only those tokens undergo an additional \"nested\" pass through the same layer, and their outputs are then fused with the initial results. DND can be applied in post-training (e.g., fine-tuning), making it compatible with existing models.", "questions": "Q1: All experiments are conducted on a single family of models (Qwen3). While the results are positive, this makes it impossible to know if the DND approach is a general-purpose technique or if its effectiveness relies on specific, undocumented properties of the Qwen architecture.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: While adaptive computation is an established research area, DND mechanism is a novel implementation. Instead of skipping layers or routing to different experts, the idea of re-processing a selected subset of tokens through the same transformer layer (\"nested pass\") is an elegant formulation, reminding me of recent looped Transformers. Furthermore, the decision to apply this technique during a post-training (SFT) phase is a highly original and pragmatic choice.\n\nS2: Thorough Empirical Evaluation: The experiments are comprehensive and convincing. The method is validated on two different model scales and architectures (a dense 1.7B and a sparse MoE 30B model), and tested across 17 diverse benchmarks. The performance gains, especially in complex reasoning domains like coding and mathematics, are significant and consistently positive.\n\nS3: The significance of this work is substantial, as it addresses one of the most pressing challenges in the development of LLMs: the trade-off between performance and computational cost.", "weaknesses": "W1: While this paper presents a promising direction,the paper does not quantitatively demonstrate why the selected tokens are \"critical.\" The visualizations could be the result of cherry-picking or confirmation bias. It is unclear if the router has learned a meaningful selection strategy or is simply responding to surface-level statistical patterns. To strengthen this claim, the authors should perform a quantitative analysis correlating token selection with intrinsic properties of the tokens or the model's state. \n\nFor instance: Correlation with Model Uncertainty: Are the selected tokens ones for which the model has high prediction entropy or low softmax probability in the initial pass? This would suggest DND is focusing computation on \"hard\" decisions.\n\nW2: The paper claims a \"minimal parameter and computing increase,\" basing its efficiency analysis on a theoretical calculation of FLOPs.  FLOPs are a poor proxy for actual inference speed (latency and throughput) on GPUs. The DND architecture, with its Pack and Unpack operations, breaks the massive parallelism that GPUs are optimized for, potentially leading to significant latency increases that are not captured by FLOP counts. They should report: Wall-clock latency (ms per generated token) and throughput (tokens per second) on a standard benchmark GPU (e.g., A100 or H100).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qqBIyQDVWn", "reviewer": "Reviewer_o89C", "summary": "The paper introduces a method called Dynamic Nested Depth (DND) which adaptively identifies “difficult” tokens for layer in an LLM and allocates extra computation to those tokens by re‑processing them through the same layer in a nested way, while easier tokens receive standard processing. This token‑level routing is managed by a lightweight router that assigns each token a probability of being selected using hidden state of each token in the seqeunce; if the score exceeds a threshold, the token undergoes a “nested” pass. Once processed, the outputs of the first pass and the subsequent passes are merged together before moving to the next layer.", "questions": "Based on the method description and experiments, it is unclear whether DND fine-tunes all components of the transformer blocks in the selected layers (L_s to L_e) or if it fine-tunes all model parameters like standard SFT.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The problem formulation is compelling, as allocating additional computation to difficult tokens could improve model accuracy at the cost of additional compute.\n - Experiments are adequate and includes evaluations across diverse benchmarks (knowledge, reasoning, coding).\n - Analysis is insightful, with clear plots and ablations that illustrate how token-level nested depth impacts performance.", "weaknesses": "- The reported accuracy improvement over regular SFT is only 0.87 on average in table 1, which appears minimal. The proposed method involves processing the input multiple times for some tokens in the given layers. Such process would increase the compute overhead. But this table does not include the computational overhead introduced to bring the 0.87 avg accuracy improvement. This makes the experimental results incomplete.\n -  Experiments are conducted on only a single LLM (Qwen), raising concerns about the generalizability of the method to other architectures.\n\n-  Based on the method description and experiments, it is unclear whether DND fine-tunes all components of the transformer blocks in the selected layers (L_s to L_e) or if it fine-tunes all model parameters like standard SFT. \n\n- The router-level losses and their underlying motivation are not clearly explained. Moreover, their overall contribution appears minimal, as reflected by the similar values in Table 2 (60.54, 60.58, 60.64) where these losses are not included.\n\n - Overall, the approach seems largely heuristic and does not demonstrate a substantial improvement in accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8CIRDgMWvB", "reviewer": "Reviewer_i7d3", "summary": "The paper presents a new heuristic method using token-conditional dynamic depth, applied post-training for improving accuracy at a low additional training and inference cost. The paper is aimed at LLM system engineers and inference optimization researchers, people who modify or extend pretrained transformer architectures to trade off accuracy and compute efficiency.\n\n- Introduces Dynamic Nested Depth (DND): a router selects “hard” tokens, which are reprocessed once through the same transformer block; the outputs are fused with the original states via a normalized gate.\n- Adds a router-control loss (to keep scores separable yet non-saturated) and a threshold-control loop (buffer proportional control + EMA) to stabilize the selected-token ratio.\n- Applies DND to mid-layers of Qwen3-1.7B (dense) and Qwen3-30B-A3B (MoE) models; reports modest average accuracy gains with ~6 % FLOPs overhead and negligible parameter growth (< 0.1 M).", "questions": "- Clarify whether base model weights were frozen or lightly fine-tuned during DND training.\n- Include compute-matched comparisons with other adaptive-compute methods.\n- Discuss failure modes—cases where DND reduces accuracy or increases instability.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- Simple, reproducible architecture with minimal parameter overhead.\n- Clear ablation analysis separating router and threshold controls.\n- Good methodological transparency (layer range, selection ratios, FLOPs estimate).", "weaknesses": "- Small performance improvements, both in an absolute sense and relative to known model variants. Even fine tuning or just random variation might yield performance improvements similar to those shown in the paper.\n- Weak experimental support: only within-model deltas, no compute-matched baselines, and no wall-clock profiling.\n- Incremental and heuristic: conceptually similar to existing adaptive-depth ideas (MoD, MoR, ITT, early-exit) and lacks deeper theoretical insight.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)