MoDr: Mixture-of-Depth-Recurrent Transformers for Test-Time Reasoning
Authors: Xiaojing Zhang
Haifeng Wu
Gang He
Jiyang Shen
Bochen Lyu
Zhanxing Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: large language models
mixture-of-depth-recurrent transformer
latent space
test-time reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: Mixture-of-Depth-Recurrent Transformers for Test-Time Reasoning

Abstract: Large Language Models have demonstrated superior reasoning capabilities by generating step-by-step reasoning in natural language before deriving the final answer. Recently, Geiping et al. introduced 3.5B-Huginn as an alternative to this paradigm, a depth-recurrent Transformer that increases computational depth per token by reusing a recurrent block in latent space. Despite its performance gains with increasing recurrences, this approach is inadequate for tasks demanding exploration and adaptivity, a limitation arising from its single, chain-like propagation mechanism. To address this, we propose a novel dynamic multi-branches routing approach for Huginn, termed as Mixture-of-Depth-Recurrent (MoDr) Transformer, which enables effective exploration of the solution space by shifting linear latent reasoning into a LoRA-based multi-branch dynamic relay mode with a learnable hard-gate routing. Meanwhile, we introduce an auxiliary-loss-free load balancing strategy to mitigate the potential routing collapse. Our empirical results reveal that MoDr achieves average accuracy improvements of +7.2% and +2.48% over the original Huginn model and its fine-tuned variant, respectively, across various mathematical reasoning benchmarks and improvements of +21.21% and +1.52% on commonsense reasoning benchmarks.

Published as a conference paper at ICLR 2026
MODR: MIXTURR-O--DDPPH--REERRRR TRANS- FORMERS FOR TEST-TIME REASONING
Xiaojing Zhang', Haifeng Wu', Gang He!, Jiyang Shen', Bochen Lyu'?2, Zhanxing Zhu DataCanvas University of Southampton 'Find Al
ABSTRACT
Large Language Models have demonstrated superior reasoning capabílities by generating step-by-step reasoning in natural language before deriving the final answer. Recently, [eeipigg al. (2025) introduced 3.5B-Huginn tive to this paradigm, a depth-recurrent Transformer that increases computational depth per token by reusing a recurrent block in latent space. Despite perfor- mance gains with increasing recurrences, this approach is inadequate for tasks de- manding exploration and adaptivity, limitation arising from its single, chain-like propagation mechanism. To address this, we propose novel dynamic multi- branches routing approach for Huginn, termed as Mixture-of-Dee Sere--Dpth-Reeee Recurrent (MoDr) Transformer, which enables effective exploration of the solution space by shifting linear latent reasoning into a coRA-based
alterna-
bratiibbrnchhcc dynamic relay mode with learnable hard-gate routing. Mcanwhile, introduce auxiliary- loss-free load balancing strategy to mitigate the potential routing collapse. Our empirical results reveal that MoDr achieves average accuracy improvements of +7.2% and +2.48% over the original Huginn model and its fine-tumed variant, re- spectively, across various mathematical reasoning benchmarks and improvements of +21.21% and +1. +1.52% on commonsense reasoning benchmarks.
INTRODUCTION
Transformer-base based large language models (LLMs) (Achiam et al. 2023 Team et al. 2023 Liu ct al. have achieved striking performance gains in a wide variety of reasoning tasks (Gao et al, 2023 Srivastava et al., 2023 Rein et al.] 2024). To enhance the reasoning capabilities of LLMs,
2024)
many studies have focused on scaling both model size (Wei et al.) 2022a) and test-time computation [Ji et al. 2025). However, much of this success comes the cost of significant computational and memory resources during training and deployment, and is accompanied by high latency as the model verbalizes excessively long intermediate reasoning (referred to "deep thinking") before delivering a final response.
Recent developments in latent reasoning, which leverage recurrent or looped-based methods, have shown potential to improve inference efficiency apart from scaling model size and employing ex-
plicit thinking (Dehghani al. 2019 Gao al. 2024b Chen et al., 2025 Geiping et al. (025). Among these, Geiping et al. (2025) proposed a 3.5B depth-recurrent transformer, dubbed Huginn, which explicitly disentangles the transformer into three-stage prelude/Looop/Cone structure (as shown in Figure [Ta), in order to reuse the rumination module (Loop) multiple times in the latent space to increase the computational depth per token. While the recurrent layers achieve low memory and latency alongside excellent reasoning performance, its single recurrent reasoning module limits the adaptability of the reasoning trajectory.
From the perspective of thought structure, the chain structure (Wei et al. 2022b) significantly hances the reasoning capabilities of LLMs compared to directly providing the answer. Subsequently, the structure (Yao et al. 2023) further strengthens the model's ability explore and backtrack by introducing hierarchical search. Furthermore, the graph structure (Besta et al., 2024), by in- corporating loops and N-to-1 connections, facilitates sub-problem aggregation and self-verification.
Correspondence to: {bochen. 1yu, 2. 2huhh8sstonn.accc Code is available http://gthhb.comm//hnggoodnnoo

---

Published as a conference paper at ICLR 2026
(a) Depth- Recurrent Huginn Model
(b) Mixture- Ceritur-of--epp--Reeurre Transformer (Ours)
Static Reasoning Process
Dynomic Reley Reasoning Process
Answer
Answer
C
bronch
ST
ST
Sg
R
Sg Router
Recument Module
Branch
branch
Ra
Question
Question
branch N
So - (0, of mn)
Prelude
Recurrent Block (Loop)
Coda
Input Injection
Reasoning Steps
Figure/ Comparison of (a) vanilla Huginn (Geiping et al. 2025) model and (b) MoDr (ours) reasoning patterns. Unlike Huginn, where each reasoning step relies on a single recurrent module, our proposed MoDr employs a dynamic routing mechanism to adaptively select the most suitable recurrent branch for the current context input, thereby predicting the next token. During inference, in Huginn, each token is generated by a fixed recurrent module, whereas in MoDr, tokens are produced dynamically through sequential relay of multiple recurrent branches.
Inspired by these works, we hypothesize that single recurrent reasoning module analogous the chain structure inherently limits the scope of exploration. This inspires us to explore the question: How can we construct depth- Huginn model with an adaptive exploration-ruming rumination module while avoiding extra resource burdens?
(MoDr) Transformer, which conceptualizes the reasoning process dynamic relay exploration for each token in com- binatorial solution space (see Figure [b) and 2 for illustration). Specifically, we first introduce several block-wise low-rank adapters (LoRAs (Hu et al., 2022)) versatile exploration branches. Each branch is integrated with the shared rumination recurrent module (Loop). Considering that different contextual inputs may have diverse computational demands for reasoning path exploration, a hard-gate routing mechanism is then employed to comprehensively consider the hidden state in- token. load balancing strategy to mitigate the risk of routing collapse (Shazeer al. 2017). inspired by [Wang al. (2024). Empirically, our comprehensive experiments across wide range of mathematical and commonsense reasoning benchmarks validate the effectiveness of the MoDr approach.
To this end, we propose an innovative approach, Mixture-of oecture-of-DDpth---e
formation of the context, and automatically decide which branch is for predicting the In addition, we utilize an auxiliary-loss-
Contributions.
We find that the inherent limitation of existing depth-recurrent Huginn model lies in its ru- mination module (the 'Loop'), which adopts single, chain-like propagation mechanism. This design weakens the diversity and exploration capability of the model's reasoning tra- jectories within the latent space.
We introduce Mixture-of Deee-eepthheeeeee Recurrent (MoDr) Transformer, a novel extension of the depth-recurrent Huginn architecture designed to shift latent reasoning into multi-branch dynamic relay exploration mode with negligible resource overhead.
We conduct extensive experiments on a wide variety of mathematical and common- sense reasoning benchmarks, demonstrating that MoDr achieves competitive performance. Specifically, compared to the vanilla Huginn model and its fine-tuned variant, MoDr achieves average accuracy improvements of +7.2% and +2.48% on mathematical tasks, and +21.21 21% and +1.52% on commonsense tasks, respectively.
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND
To scale test-time computation while reducing both training and deployment computational over- head, a prevailing line of research is enable depth adaptivity in Transformers through recurrent depth and looped layers. Starting from the Universal Transformer (Dehghani et al.) (219),, which pioneered dynamic recurrence over layers to iteratively refine sequence representations, this design paradigm has demonstrated that depth-adaptive reasoning is a promising substitute for the tradi- tional fixed-depth transformer architecture. Subsequently, research on the Looped Transformers has exhibited strong generalization performance across diverse tasks, including programmable comput- ing GGannoou et al., 2023). data fitting (Yang et al. 2023), and arithmetic reasoning (Saunshi et al, 2025). In addition to adopting monolithic recurrent designs, AlgoFormer (Gao al.) (2024b) and Depth-Recurrent Huginn (Geiping et (025) proposed a three-stage Loelude/Looo/Cooa structure, which can be formulated as:
Shead Seota ono... ...
prellde sembed,
T iterations
with R hidden layers involved in the Loop for times. Note that Spre and fuoda could have more than one hidden layer. The modularization of this architecture can provide efficiency and adaptability while generalizing across a broader range of applications. In addition, depth-recurrent transformers increase computational depth per token by reusing intermediate layers, which can be viewed as "deep thinking" in the continuous latent space to facilitate reasoning.
2.1 PRELIMINARY:
HUGINN
We first outline the architecture of the model that our framework built upon, termed Huginn, which is 3.5B scalable recurrent decoder-only transformer (Figure [aa). The model is primarily struc- tured around three functional modules: (1) 2 prelude blocks, which are responsible for embedding input context into latent space; (2) 4 recurrent blocks, which sequentially process the output from the prelude module; (3) 2 coda blocks, which decode from the latent space to predict the next token. Notably, all blocks follow standard transformer layer design, and each layer contains a multi-head causal self-attention mechanism.
[x1, .... "nnn where x; RIV,, n denotes the length of the input context, and [V] represents the size of the vocabulary. Unlike the forward pass of a standard Transformer, the model explicitly separates its computation flow into input encoding (prelude module), iterative implicit reasoning R (recurrent module), and output decoding C (coda module). These modules ultimately produce output probabilities p E | as follows:
Concretely, given sequence of tokens =
e P(x),
(2)
So NN00
(3)
S; R(c,5 for tE (1, T}.
(4)
-((s))
(5)
where So is a random Gaussian vector serving as the initial state of the recurrent module, o is some standard deviation, and the hidden dimension. During the T' recurrent steps, the model repeatedly applies the core unit R, which takes the latent state 8;-1 and the embedded input as input, and outputs the updated latent state s, Although this model only includes 8 trainable blocks (i.e.. 2P 4R + 2C), it allows for infinitely deep transformer in the limit as T co. Note that while the depth-recurrent Huginn gains efficiency by using recurrent states in the latent space instead of generating explicit reasoning steps, its reliance single recurrent module for all reasoning steps fundamentally constrains its reasoning capabilities and flexibility.
2.2 DYNAMIC ROUTING AND PARAMETER EERAMIETR-EEFFCIEET FINE- TUNING
Recent studies have established Mixture-of of -xperrtss (MoE) as a prominent architecture for scaling LLMs (Liu et al. 2024 Yang et al. 2025). Its key feature is that MoE models can dynamically select the most suitable expert from multiple candidates for different inputs. This approach achieves favorable trade-off between model capability and computational efficiency while maintaining
3

---

Published as a conference paper at ICLR 2026
(o)
(b)
(4)
Coda
'wight
Shored
Recurnent
R
R
R
R
Blocks
A
Nulti-Branch
Recurrent Module
hand gates
Router
+0.1
Prelude
Lass- Branch Updating
(d)
Frozen
Trained
Figure 2: The architecture of the Mixture-of-D offture-of-Dephh--- Recurrent (MoDr) Transformer. MoDr consists of N recurrent branches, which are formed by combining the original recurrent blocks of Huginn with different LoRAs, with the weights of the original recurrent blocks shared across all branches. In addition, a hard-gate routing mechanism dynamically selects the appropriate branch for the current input information. During fine-tuning, only the LoRA branches and the router are trained, and an auxiliary-loss loss-free balancing mechanism ensures even load distribution across different branches.
nearly constant number of FLOPs per token for each expert Zooph al. 2022). Instead of applying MoE directly during pre-training, methods such as MoELORA (Luo et al. 2024), MOLA al., 2024a), and MIXLORA (Li et al. 2024) enhance multi-task generalization by integrating multiple LoRA adapters as layer-wise experts within transformer sub-layers during fine-tuning. Furthermore, LoTA (Panda et al. 2024) mitigates destructive interference by optimizing a sparse sub-network of the model. As noted earlier, the inflexibility of Huginn in generating each token stems from its rigid recurrent module. To address this flaw, we propose MoDr, a novel MoE+LoRA architecture that departs from the conventional layer-wise paradigm. Particularly, MoDr integrates block-wise low-rank adapters directly into Huginn's recurrent module and couples them with a dynamic routing mechanism. This design fundamentally enhances the adaptability and richness of the model's rea- soning trajectories, establishing new direction for adaptive reasoning in recurrent-depth models. We discuss more related works Appendix A.I
METHODOLOGY
To enable adaptive latent space reasoning, this work, we propose an innovative approach, Mixture- of-Depth-Recurdee Recurrent (MoDr) Transformer, which fine-tunes the existing Huginn model to establish multi-branch, dynamic reasoning pathway. This section outlines the architecture and methodology of MoDr, as illustrated in Figure 2 Specifically, we first integrate multiple low-rank adapters (Lo- RAs) to construct distinct recurrent reasoning branches, of which share the weights of the original recurrent blocks in the Huginn model ($3.1). Then, we present hard-gate routing mechanism that takes into account the hidden state information of the current context to select which branch will predict the next token ($3.2). To alleviate the load imbalance among different branches, adopt an muxiliary-loss-fre load balancing strategy to mitigate uneven load distribution across branches during training $3.
3.1 LoRA- BASSD MULTI-BRANCE RECURRENT MODULE
As discussed above, Huginn broadly follows the design of standard transformer layers. Accordingly, the architecture Huginn's core recurrent blocks is built upon the standard transformer "sandwich" structure, which sequentially stacks a multi-head causal attention layer (Attn) and multilayer per- ceptron (MLP), with each sub-layer featuring a residual connection and layer normalization (LN). Let z' =x denote the hidden state output by the (-th recurrent block. For each recurrent step t e (1, 2, ..., T). the hidden state z is computed as:
/ LN(Att(LN(({--)) W + 2111),
(6)
z LN(MLP(LN(2)) W') 2!),
4

---

Published as a conference paper at ICLR 2026
where W denotes the parameters of the I-th recurrent block. Our objective is to construct an adaptive celloraioo-ruminnatii module. Here, a key challenge arises: how can we break free from the rigidly sequential forward propagation constraints inherent in existing recurrent architecture?
To address this, we propose to utilize different branches to dynamically take turns to predict the next token according to the current context information, until the end of reasoning. Full fine-tuning with multiple initialized instances a recurrent module in Huginn can enhance diversity and exploration within the model's latent reasoning trajectories. However, this approach incurs substantial compu- tational and memory overhead. To mitigate these costs, MoDr integrates multiple low-rank adapters (LoRAs) to create distinct recurrent reasoning branches, with the weights of the original recurrent blocks being shared across all branches, as shown in Figure 2(c). During training, only the LoRA weights of all branches are fine-tuned. This design yields two primary advantages: (1) the backbone model is frozen to preserve its world knowledge, and (2) LoRA introduces negligible computational or memory overhead owing to its minimal number of trainable parameters. Therefore, the output hidden state z of each recurrent branch is computed as follows:
$) = LN(AMt(LN()) AW, +z 2),, LN(MLP(LN( ( W AW)) + 2),,),
(8)
where (A W denotes the trained LoRA module parameters for recurrent branches. Specifi- cally, for base feature transformation Wox, our modified forward pass yields:
z Wox "AWx= Wox+
O BAx,
(10)
where Rixr and e Bgr with rank r < minnh, k) for and & being dimensions of original parameter matrix Wo. The scaling factor controls the adaptation magnitude.
3.2 HARD-GATE BRANCH ROUTING STRATEGY
Inspired by sparely-gated cexture-of-Experrs (MoE) Shazeer et al. 2017) and switch Transformer (Fedus al. 2022), we design learnable hard-gate routing network to determine which candi- date recurrent branch will predict the next token according to the hidden state information. In our method, the hidden state information h Raxh derives from two aspects: (1) the output e of the prelude blocks, and (2) the recurrent state s. We utilize an adapter matrix: 2R R to map the concatenation and into the hidden dimension
Let W router RR. x denote the trainable weight matrix of routing network, where N is the number of candidate recurrent branches. As depicted in Figure our design employs Top-K hard-gate router. For the sake of clarity, we use Top-1 single branch that achieves the highest average confidence score across the tokens, as determined by the current context information. The process is detailed below:
example to illustrate this. This router selects the
W Wrouter T u NNn
(11)
r af (u)),, r RV
(12)
argmax (ry), je (1, N),
(13)
g rj if j c
(14)
where Ç is the index of the selected recurrent branch, g is a scalar score, and a is a nonlinear activation function like sigmoid softmax. After that, all current hidden states z/, of the selected
branch are weighted 2/; 9 2/,5-
5

---

Published as a conference paper at ICLR 2026
During inference, for each newly generated token, the hard-gate router dynamically selects which recurrent branch will perform the "deep thinking" based on the contextual information from the pre- vious reasoning steps. This next-token prediction process resembles "relay race" across different branches. For an illustration, see Figure Tb).
3.3 AUXILIARY-LOSS-FRER LOAD BALANCE
To prevent imbalanced training among branches caused by routing collapse (Shazeer et al. 2017), we introduce a load balancing strategy from sequence-wise perspective. Existing studies typically employ auxiliary loss (Lepikhin al., 2020; Fedus et al. 2022) to address load imbalance in the MoF. system during training. However, evidence suggests that a large auxiliary loss could introduce significant conflicting gradients into training, thereby degrading model performance (Wang et al. 2024). To overcome this challenge, we introduce loss-free balancing approach to directly adjusts the gate scores of candidate branches based on their individual load conditions.
As illustrated in Figure 2 d), we add bias term {b,) to the original gating score [r;],-] of each candidate branch. The branch selection and the final weight score are then computed follows:
by PERN argmax,(f)) (fy), jE (1,,,.... g Ty if j j=
(15)
(16)
(17)
where ? denotes the index of the selected recurrent branch after the bias term adjustment routing strategy. Note that the corresponding weight score y does not involve the bias term bj.
To adjust the per-branch bias by (i E (1,2,..., N)) during training, each bias by is first initialized to O. For each batch, the number of assigned samples per branch and their average number e are counted. Then, the load violation error e; is computed, and the bias 6, is updated as follows:
e; ē Cj. b, b, + 17 sign(e;),
(18)
(19)
where 77 is the update rate of the bias term, and sign(.) is sign function. This load balancing strategy not only enables load balancing across branches but also prevents the direct incorporation of noisy gradients into the model.
4 EXPERIMENTS
In this section, we evaluate the effectiveness of our proposed MoDr, with focus on mathematical reasoning tasks. We present additional experiments for commonsense reasoning tasks in Appendix
A.2 All experiments are conducted on a single NVIDIA Tesla H100 GPU with 80GB of VRAM.
4.1
EXPERIMENTAL SETUP
Datasets. We conduct our experiments on six mathematical reasoning tasks: (1) GSM8K (Cobbe ct al. (021), a dataset of high quality grade school math word problems. (2) MAWPS (Koncel-
Kedziorski et al. 2016) dataset, a curated online repository of arithmetic and algebra word prob-
lems. (3) AQuA (Ling et al.) 2017) dataset, focusing on algebraic word problems. (4) Mul-
tiArith (Roy & Roth 2016), containing multi-step mathematical word problems. (5) AddSub (Hosseini et al. 2014). dataset of addition and subtraction arithmetic word problems. (6) Sin-
gleEq (Koncel-Kedziorske et al. 2015), covering grade-school algebra word problems. Regarding the above datasets, only GSM8K, MAWPS, and AQuA provide training sets, whereas MultiArith, AddSub, and SingleEq serve as three out-of-domain benchmarks to evaluate the models' robust- ness. To enhance the reasoning capabilities of the fine-tuned models, particularly their step-by-step rationales, we employed Qwen2.5-Math- Mar.----athh-B-IIsttuuee to generate chain-of of-in-of-thought reasoning steps
Instructtt

---

Published as a conference paper at ICLR 2026
for the training sets. To ensure data quality, removed samples with incorrect answers. Table summarizes the detailed statistics of the datasets. The accuracy is the evaluation metric used across all benchmarks to measure the conrectness of the predicted answers.
Table 1: Statisties of Mathematical Reasoning Datasets.
In Domain
Out of Domain
Dataset
GSM8K
MAWPS
AQuA
MultiArith
AddSub
SingleEq
Answer Type
Number 7130
Number 1826
Option 609
Number
Number
Number
Train Sample
# Test Sample
1319
238
254
600
395
508
Baselines. We primarily compare against the following baselines: First, the vanilla Huginn model (Geiping et al. 2025), which constitutes our base architecture for MoDr. Second, we introduce LoRA-based supervised fine-tuning (SFT) variant of this base model, termed Huginn-SFT, ing hyperparameters identical to those of MoDr. Furthermore, we evaluate a multi-branch Huginn model without a router (i.e., using random branch selection) to assess the router's contribution to performance.
Settings. For all experiments in this section, use the AdamW optimizer with a learning rate of 4e-5, weight decay of le-4, and betas set to (0.9, 0.95). We clip gradients with a threshold of 0.2 and employ cosine learning rate schedule with 10% warmup. Models are trained using batch size of 4 and sequence length of 512, and the number of epochs matches the number of recurrent branches. To scale the depth-recurrent architecture, we train with a mean recurrence value of 32. To reduce computational and memory costs during training, time (BPTT) to the last itcrations of the recurrent unit. For all low-rank adapters associated with each branch, both the rank and the scaling factor configured to be 16. These adapters activate the q, k. v, projections in attention layers of the recurrent blocks. Regarding the branch routing and load balancing strategies, update rate to 0.001. Among the routed branches, a single branch is activated per token.
truncate backpropagation through
sigmoid nonlinear activation function and set the bias term's
4.2
MAIN RESULTS
4.2.1
MATHEMATICAL REASONING
In this experiment, MoDr is equipped with four LoRA-based recurrent branches and hard-gate router. Its trainable parameters constitute less than 0.2% of the base Huginn model. Our baseline, Huginn-SFT, is conventional single-branch model fine-tuned on the same datasets and settings. Figure presents a comparison between the baseline models and our proposed MoDr. The results demonstrate that this multi-branch architecture significantly boosts reasoning capabilities with negligible parameter overhead. Notably, across six mathematical reasoning datasets, MoDr achieves average accuracy improvements of +7.2% and +2.48% over the vanilla Huginn model and its fine- tuned variant, respectively. This advantage stems from the fact that MoDr can dynamically route the hidden state from the prelude module to the most suitable branch for next-token prediction, thereby enhancing the adaptability of the reasoning trajectory.
To assess robustness, we split the six benchmarks into three in-domain (ID) (i.e., GSM8K, MAWPS. and AQuA) and three out-of-domain (OOD) (i.e., MultiArith, AddSub, and SingleEq) tasks. We can also observe that MoDr exhibits superior performance relative to all baseline models, irrespective of the domain setting. Specifically, MoDr excels even more on out-of-domain datasets (i.e., outper- forming the original Huginn and Huginn-SFT by +6.75% and +1.78% on the ID, and by +7.67% and +3.18% 18% on the OOD, respectively). suggesting strong generalization. Overall, these results well demonstrate the effectiveness our proposed MoDr.
4.2.2 COMMONSENSE REASONING
We also evaluated MoDr on various commonsense reasoning benchmarks. As reported in Figure 7 of Appendix A.2. A.22, our findings reveal that MoDr achieves the most significant performance gains over both the vanilla Huginn and the Huginn-SFT models. This indicates the broad applicability of MoDr for boosting L.L''s performance. See Appendix for more details.
7

---

Published as a conference paper at ICLR 2026
GSMBK (In-Domain)
MAWPS (In-Domain)
AQuA (In-Domain)
Average
50-
49.43
49.89
80.67
69.22
Accuracy (96
Accuracy (%)
Accuracy (%)
Acc Accuracy (%6
48
78-
74
47
32
66-
#6
76-
30
64
45
74-
43
43.59
72-
71.05
16
27.95
e
62.02
MultiArith (Dut-af-Domain)
AddSub (Dut-of-Domain)
SingleEq (Dut-of-Domain)
Accuracy (96
Accuracy (96)
Accuracy (96)
81.30
Methods
88
87.17
78-
Huginn
84
76-
74.68
70-
Huginn-SFT
76.97
MoDr (Ours)
78
70
Figure 3: Performance comparison of MoDr with baseline methods on in-domain and out-of-domain mathematical reasoning benchmarks. The "Average" represents the overall mean of the six mathe- matical datasets.
4.3 ABLATION STUDY
We investigated the impact of dynamic routing from two perspectives. On one hand, during inference, we deliberately disabled the router in MoDr and employed random branch selection (referred to as MoDr w/o Router). On the other hand, we directly fine-tuned model without router, the 4-branch Huginn, which also uses random branch selection in both the training and inference stages (referred to as No Router). As shown in Table 2 MoDr with dynamic routing demonstrates superior inference performance over both router-free models across all mathematical benchmarks.
Impact of Router.
In addition, two observations worth noting: (1) No Router performs best GSM8K, but only marginally. This can be attributed to the fact that GSM8K's s extensive training data (74.52%) thor- oughly trains the model branches in all setups. (2) The performance of No Router is better than that of MoDr w/o Router but is still inferior to that of the standard MoDr with its router. This further underscores the positive role dynamic routing in branch selection during inference.
Table 2: Performance comparison among three configurations: a 4-branch Huginn model without
router (i.e., using random branch selection), a MoDr model with a router, and a MoDr model with disabled router (i.e., also uses random strategy to select branch per token). The top score in each column is in bold.
In Domain
Out of Domain
Method
Average
GSM8K
MAWPS
AQuA
MultiArith
AddSub
SingleEq
No Router (random)
50.72
79.41
31.89
90.17
75.70
76.57
67.41
MoDr w/ Router (Ours)
49.89
80.67
33.07
91.17
79.24
81.30
69.22
w/o Router (random)
48.60
77.73
29.92
89.17
74.68
78.35
66.41
Impact of Single Branch. To validate the effectiveness of dynamic routing for ence, we conducted an ablation study by evaluating each individual branch of the 4-branch MoDr independently on all mathematical tasks. The results, presented in Table demonstrate that the av- erage accuracy of MoDr with dynamic routing is superior to that of any individual branch and even exceeds their collective average. Notably, MoDr with dynamic routing does not achieve the top rank on every benchmark. Specifically, it ranked first on the MAWPS, AddSub, and SingleEq datasets, second on GSM8K, and third on AQuA and MultiArith. This performance variation implies that different branches may have developed specialized capabilities for distinct scenarios during train- ing. Although dynamic routing strategy fails to select the optimal inference trajectory in certain cases, it effectively leverages the strengths of each branch across wide range of scenarios, lead-
infer-
8

---

Published as a conference paper at ICLR 2026
Table Performance comparison between 4-branch MoDr with dynamic routing and its four indi- vidual branches (Branch-(b), b {1,2, 3, 4/). wyg.Br-(~-4) denotes the average score of branches to 4. The top score in each column bold, and the second-highest underlined.
In Domain
Out of Domain
Method
Average
GSM8K
MAWPS
AQuA
MultiArith
AddSub
SingleEq
MoDr (Ours)
49.89
80.67
33.07
91.17
79.24
$1.30
69.22
L, Branch-(1)
50.19
75.63
35,43
91.67
71.39
75.98
66.72
4 Branch-(2)
48.07
79.37
34.65
89.17
75.19
80.71
67.86
B3)
47.84
80.25
28.35
92.50
77.97
80.71
67.94
Branch-(4)
49.66
74.37
26.77
86.83
70.13
78.15
64.32
Avg.Br-(I~-4
48.94
77.41
31.30
90.04
73.67
78.89
66.71
ing to superior overall performance compared to any individual branch or their simple average.
This confirms the critical role of multi-branch dynamic routing framework in developing effective depth-recurrent reasoning models.
Impact of Load Balance. An unbalanced branch load can lead to routing collapse
JAvB-ACC-69955
Balance Entropy
1.75
1 0.001 (Ang.ALL-69.22
(Shazeer al., 2017), causing an imbalance in branch utilization and diminishing computa- tional efficiency. To mitigate this, employ a load balancing strategy governed by the up- date rate in Eq. [9] which controls the con- vergence rate of the branch biases ( to suitable bias. To evaluate this strategy, we con- ducted an ablation study by comparing model with the strategy enabled (7=000001) against baseline without it (1=0), We introduce met- ric called balance entropy to quantify the degree of load balance as follows:
1.50
1:25
www.
1.00
0.75
0.50
0.25
0.60
0
1000
2000
3000
4000
5000
6000
7030
8000
Step
Figure 4: The impact of update rate on training load balance. Lower entropy makes it casier for the router to choose the same branch.
Count(br)
Count( (r)
Halance
Σ
loga
(20)
[2
Ω
brEUnique
where S2 is the set of branches selected by the router within a batch, and br an element this set. higher entropy indicates a more balanced load distribution. As shown in Figure the model with 7=0 (without load balance) converges to a limited subset of branches, resulting in their dispropor- tionate over-utilization. In contrast, the model with 700.001 ensures more even distribution of the training load across all branches, which ultimately leads to superior generalization performance.
4.4 SENSITIVITY ANALYSIS OF RECURRENT BRANCH NUMBERS
The number of recurrent branches a critical hyper- parameter, influencing both the search space size and computational resource allocation. To investigate its impact, we conducted series of experiments where we varied the number of branches while keeping all other settings constant. To guarantee a fair compar- ison, the total number of training epochs was scaled proportionally with the number of branches, ensur- ing that each branch received an equivalent amount of computation. The results, as shown Figure 5 reveal a clear trend: average performance across all benchmarks consistently improves as more branches added. This positive correlation between branch count and performance validates the effectiveness of dynamic multi-branch architecture. However,
69.5
Average Accuracy
60.0
68.0
67.5
67.0
Number of Branches
Figure 5: The changes of average performance under different recurrent branch numbers.
also observe that the performance gains be-

---

Published as a conference paper at ICLR 2026
gin to diminish beyond four branches, suggesting that this configuration offers the best trade-ofi off between performance and computational cost.
4.5 QUANTITATIVE ANALYSIS OF BRANCH ROUTING
To gain insights into the routing selection dis- tribution across different tasks, we conducted frequency analysis on six mathematical reason- ing benchmarks, as shown in Figure The re- sults reveal two key observations. First, the rout- ing distribution is highly sensitive to the specific dataset. On complex, multi-step reasoning tasks like MultiArith, Branch-3 is activated at signif- icantly higher frequency (46.72%) than other tasks. contrast, for the multiple-choice ques- tion set AQuA, Branch-4 emerges the second most utilized branch (33.16%). This indicates that the router dynamically allocates computa- tional resources in response to the unique de- mands each task. Second, the branches exhibit clear division of labor. Branch-2 serves as a "generalist," consistently handling the majority of computations across all benchmarks. The re- maining branches function as "specialists." For instance, Branch-3 is predominantly activated for complex arithmetic tasks (MAWPS, AddSub), whereas Branch-4 is more engaged in tasks requiring distinct reasoning patterns, such as those in AQuA. This non-uniform distribution provides strong quantitative evidence that the branches have learned distinct, non-redundant functions.
0.277
01:18 5.000
Branch Index
0.97
6:444
8666
0.977
5.220
2241
GSMBK
MAWPS
AQUA
MultiArith
AddSub
SingleEq
Tasks
Figure 6: The changes of average performance under different recurrent branch numbers.
4.6
CASE STUDY
We further check more cases of the reasoning processes of MoDr. We find that different reasoning steps during inference. MoDr dynamically invokes diverse branches to predict the next token based on the available context state, which is consistent with our hypothesis. Due to space limitations, more details are shown in Appendix A.3
CONCLUSION
In this paper, we introduce the Mixture-of-Deptte Recurrent (MoDr) Transformer, a novel dynamic routing framework that advances the depth-recurrent Huginn model. The vanilla Huginn model's reasoning flexibility is constrained by its reliance on a single, chain-like propagation mechanism within the rumination recurrent module. MoDr addresses this limitation by incorporating multi- ple LoRA branches and employing a hard-gate router to dynamically select the most appropriate branch for next-token prediction. Extensive experiments across diverse set of mathematical and commonsense reasoning benchmarks demonstrate that MoDr can significantly improves upon the performance of the existing Huginn model while incurring negligible computational overhead.
5.1 LIMITATIONS & FUTURE WORK
MoDr offers a dynamic multi-branch framework for the depth-recurrent Huginn model (Geiping et al. (025). designed to enhance the exploration capability and adaptivity of its rumination recurrent module (Loop) within the latent space. By leveraging LoRAs as distinct branches, our approach avoids significant computational overhead. However, for practical deployment, MoDr necessitates an efficient KV cache strategy, which remains a key challenge and a primary direction for future work. Inspired by (Geiping et al. 2025 Bae et al., 2025), identify two promising solutions: (1) caching KV pairs from the most recent recurrent iterations under a fixed budget, or (2) caching the initial KV pairs and sharing them across all recurrent branches for subsequent reasoning steps.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We thank anonymous reviewers for their valuable and insightful feedback. The computational re- sources for this research were supported by the DataCanvas Alaya NeW platform.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 088774 2023.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoc Lc, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.0777 2021.
Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, et al. cexture-of-recuriion Learning dynamic recur- sive depths for adaptive token-level computation. arXiv preprint axiv:2557.10522 2025.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gian- inazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 77682-17690 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on arlificial intelligence, volume 34, pp. 7432-7439, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.0333 137.0334, 2021.
Yilong Chen, Junyuan Shang, Zhenyu Zhang, Yanxi Xie, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, and Haifeng Wang. Inner thinking transformer: Leveraging dynamic depth scaling to foster adaptive internal thinking. arXiv preprint arXiv:2502.138422 2025.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: /003.0v400050555 2018
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv: 200.:2110.11 14168. 2021.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference Learning Representations, 2019.
Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, al. Supergpqa: Scaling 1lm evaluation across 285 graduate disciplines. arXiv preprint arXiv:2502. 14739, 2025.
David Eigen, Mare' Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint arXiv: 1312.4314 2013.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39 2022.
Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. Higher layers need more lora experts. arXiv preprint arXiv:2402. 08562, 2024a.
11

---

Published as a conference paper at ICLR 2026
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aice -iddd language models. In International Conference on Machine Learning. pp. 10764-10799, PMLR, 2023.
Yihang Gao, Chuanyang Zheng, Enze Xie, Han Shi, Tianyang Hu, Yu Li, Michael K Ng. Zhen- guo Li, and Zhaoqiang Liu. Algoformer: An efficient transformer framework with algorithmic structures. arXiv preprint arXiv:2402. 13572, 2024h.
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05177, 2025.
Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers. In International Conference on Machine Learning. pp. 11398-111 11442. PMLR, 2023.
Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit- Yan Yeung, James T Kwok, and Yu Zhang. Mixture of cluster-conditiona lora experts for vision-language instruction tuning. arXiv preprint arXiv:2312. 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in continuous latent space. arXiv preprint arXiv:2412.06769, 2024.
Mohammad Javad Hosseini. Hannanch Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 conference on empirical methods natural language processing (EMNLP), pp. 523-533, 2014.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficcent transfer learning for nlp. In International conference machine learning, pp. 2790-27999 PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. LIm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.0193: 2023.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720 2024.
Gagan Jain, Nidhi Hegde, Aditya Kusupati, Arsha Nagrani, Shyamal Buch, Prateek Jain, Anurag Arna,, and Sujoy Paul. Mixture of nested experts: Adaptive processing of visual tokens. Ad- vances Neural Information Processing Systems, 37:58480-58497, 2024.
Yixin Ji, Juntao Li, Hai Ye, Kaixin Wu, Kai Yao, Jia Xu, Linjian Mo, and Min Zhang. Test-time compute: from system-1 thinking to system-2 thinking. arXiv preprint Siti::2501.04499 2025.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-22213, 2022.
Rik Koncel-Kedzziorkii Hannanch Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. Parsing algebraic word problems into equations. Transactions of the Association for Com- putational Linguistics, 3:585-597, 2015.
12

---

Published as a conference paper at ICLR 2026
Rik Koncel-Kedziorske Subhro Roy, Aida Amini, Nate Kushman, and Hannanch Hajishirzi. Mawps: math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pp. 1152-1157, 2016.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Dengchun Li, Yingzi Ma. Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, et al. Mixlora: Enhancing large language models fine- tuning with lora-based mixture of experts. arXiv preprint arXiv:2404. 1519,, 2024.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint ar arXiv:2101. 10t.0009000000 2021.
Ziyue Li, Yang Li, and Tianyi Zhou. Skip layer or loop it? test-time depth adaptation of pretrained llms. arXiv preprint 2rXiv:2507.07996 2025.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale gener- ation: Learning to solve and explain algebraic word problems. arviv preprint 1705.:: 2017.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseck-v2: A strong, economical, and efficient mixture- of-experts language model. arXiv preprint arXiv:2405.044434 2024.
Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts on parameter-effiiine fine-tuming fnn-tuning for large lan- guage models. arXiv preprint arXiv:2402. /2851, 2024.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relation- ships in multi-task learning with multi-gate -fxture-of-exxprts. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1930-1939, 2018.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint trXiv:1809.0220 2018.
Ashwince Panda, Berivan Isik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, and Prateek Mit- tal. Lottery ticket adaptation: Mitigating destructive interference in 1lms. arXiv preprint arXiv:2406. 16797, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In First Conference on Language Modeling, 2024.
Subhro Roy and Dan Roth. Solving general arithmetic word problems. arXiv:1608.014 2016.
arXiv preprint
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers. arXiv preprint arXiv:2502.17416, 2025.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gates mixture-of mixture-of-experts layer. arXiv preprint arXiv: 1701006538, 2017.
Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts. arXiv preprint arXiv:2303.07226, 2023.
13

---

Published as a conference paper at ICLR 2026
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu. Yali Du, and Yulan He. Codi: Compressing saain-of-thouglle into continuous space via self-distillation. arXiv preprint arXiv:2502.21074, 2025.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri Garriga-Alonso, et al. Beyond the imita- tion game: Quantifying and extrapolating the capabilities of language models. Transactions on machine learning research, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11005, 2023.
Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-loss-fee load balancing strategy for mixture-of-experts. arXiv preprint arXiv:2408, 15664, 2024.
Xiaoqiang Wang, Suyuchen Wang, Yun Zhu, and Bang Liu. System- 1.5 reasoning: Traversal in language and latent spaces with dynamic shortcuts. arXiv preprint arXiv:2505.18902 2025.
Jason Wei, Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo-
gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint arXiv:2206. 07682, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837 2022b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv,, et al. Qwen3 technical report. arXiv preprint arXiv:2505.009388 2025.
Liu Yang, Kangwook Lee, Robert Nowak, and Dimitris Papailiopoulos. Looped transformers better at learning learning algorithms. arXiv preprint arXiv:2311.1242,,22 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36: 11809-11822) 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv:1905.07830. 2019.
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xucling Liu, Bill Yuchen Lin, Jic Fu, Wenhu Chen, and Xiang Yue. Opencodeinterrpeeee:: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402. /4658, 2024.
Barret Zoph, Irwan Bello, Sameer Kumar. Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, arXiv preprint arXiv:2202.08900 08906, 2022.
14

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 RELATED WORK
A.1.1 TEST-TIME REASONING
Recent advances have demonstrated that large language models emerge with remarkable reasoning capabilities. In particular, Chain-of-Thought (CoT) reasoning (Wei al.) 2022b Kojima et al. 2022) with few-shot or even zero-shot examples further achieves superior performance. These prompting techniques explicitly elicit intermediate reasoning steps before the model generates its final answers. Furthermore, several studies have further unlocked the potential of CoT when inte- grated with reinforcement learning-based fine-tuning (Jaech al. 2024, Guo et al. 2025), uncov- ering significant "aha- momemmmmmtt" in model behavior. Alternatively, compelling counterpart is to prompt LLMs that perform reasoning latent space without verbal narrative. A strategy of this line is to explore auto-regressive latent reasoning by progressively replacing CoT tokens with continuous representations through multi-stage training (Hao al. 2024). or compressing CoT tokens into con- tinuous space via self-distillation in single stage (Shen et al. 2025). Another line of work focuses on loop-based architectures, which iteratively refine hidden states in single forward pass by recur- rently propagating information across layers (Dehghami et al. 2019; Chen et al. 20255 Geiping et al. 2025). In addition, to achieve test-time depth adaptation, recent studies have explored methods for dynamically allocating computation in latent space. These methods learn adapt inputs of vary- ing complexity by adjusting recursive depths (Bae et al. [022 Chen et al. 2025). creating shortcuts (Wang et al., 2025), or combining both approaches (Li et al. 2025). Orthogonal these works, we focus on developing versatile recurrent branches that can handle diverse contextual inputs, thereby
improving the udaptability of the reasoning trajectory.
A.1.2 PARAMETER- FINE-TUNING
Supervised fine-tuning of large language models is a common practice for adaptation various specific downstream tasks. However, as model parameters scale up, the computational cost of full fine-tuning becomes a significant challenge. To mitigate this, parameter-efficient fine fine-tuning (PEFT) methods (Houlsby et al., [019 Li & Liang 2021 Hu et al., 2022) have emerged over time. Low- rank adaptation (LoRA) (Hu et al. (022) enables plug-and. and-play adaptation of pretrained LLMs by freezing initial model weights and fine-tuning a small set of low-rank matrices. For instance, (Gou et al. (023)) proposed a mixture of cluster-conditional LoRA experts to activate tasc-specific adapters based on instruction clusters. (Li al. (024) enhanced model performance by utilizing in- dependent uttention-laye LoRA adapters. Compared to full fine-tuning, this paradigm offers a more flexible and efficient solution with constrained computational resources. As for our approach, integrate LoRAs as diverse exploration branches instead of simply copying the backbone recurrent module, thereby expanding model capacity and avoiding additional overhead.
A.1.3 MULTI-BRANCH NETWORKS AND DYNAMIC ROUTING
Given that diverse inputs could have different computational demands, it is intuitive perform in- ference with dynamic pathways tailored to each sample. Recent studies, such as MoCLE (Gou et al. 2023) and MoNE (Jain et al. 2024), aim enhance model performance sub-networks and a learnable routing mechanism. The paradigm of Mixture-of-Experts (MoE) not only provides a promising solution for managing computational costs but also results in remark- able performance improvements. In conventional soft MoE (Eigen et al.) 2013 Ma et al., 2018). the weights predicted by the router are adopted to dynamically aggregate the representations of all branches ("experts")). In contrast, hard MoE models (Shen et al. 2023 Guo et al.) 2025) dynami- cally activate the branches with the highest or top-k confidence. While effective, these methods often encounter load imbalance, potentially leading routing collapse (Shazeer al. 2017). Therefore,
maintaining multiple
introducing auxiliary balancing strategies (Fedus et al. 2022 Wang et al. (024)) is essential during the training phase. Inspired by these works, MoDr combines hard-gate routing mechanism and an auxiliary- loss- loss-free free load balancing strategy to adaptively select the most suitable recurrent branch for the next reasoning step, thereby improving robustness while retaining performance.
15

---

Published as a conference paper at ICLR 2026
A.2 EXPERIMENTS FOR COMMONSENSE REASONING
A.2.1 EXPERIMENTAL SETUP
Datasets. We conduct our experiments on six commonsense reasoning tasks: (1) PIQA Bisk et al,
2020), dataset focused physical commonsense question answering. (2) HellaSwag (Zellers
et al. 2019). commonsense natural language interface (NLI) dataset. (3) WinoGrande (Sak-
aguchi et al. 202I). dataset for commonsense reasoning involving pronoun disambiguation and sentence completion. (4) ARC-E and (5) ARC-C (Clark et al. 2018), the Easy and Challenge sets of the ARC dataset, which contains genuine grade-schoo level, multiple-cooee science questions. (6) OBQA (Mihaylov et al. 2018), dataset requiring multi-step reasoning, the use of additional common knowledge, and rich text comprehension. The detailed statistics of the datasets are outlined in Table D To facilitate fine-tuning in the domain of commonsense reasoning, we adopt the Com- monsense 70K dataset constructed by Hu al. (2023). and conduct evaluations on the individual testing dataset for each task.
Other Protocols. For the evaluation metric, comparison methods, and settings, we maintain consis- tency with the experiments mathematical reasoning tasks (See $4.1)
Table 4: Statistics of Commonsense Reasoning Datasets.
Dataset
PIQA
HellaSwag
WinoGrande
ARC-E
ARC-C
OBQA
Answer Type
Option
Option
Option
Option
Option
Option
# Train Sample
16.1K
39.9K
63.2K
1.1K
2.3K
5.0K
# Test Sample
1830
10042
1267
2376
1172
500
A.2.2 RESULTS AND ANALYSIS
Figure illustrates a comparison of our proposed MoDr against the baseline models (i.e., vanilla Huginn and Huginn-SFTI performance gains over its competitors across most commonsense tasks, with the exception of HellaSwag, where it underperforms Hug by 0.83%. In terms of average accuracy, MoDr achieves improvements of +21.21 % and +1.52% over vanilla Huginn and Huginn-SFT, respectively. Overall, these results further verify MoDr's effectiveness on commonsense reasoning tasks.
six commonsense reasoning tasks. MoDr demonstrates consistent
PIQA
HellaSwag
WineGrande
Average
82.5
77.5 5
Accuracy (96)
75
74.35
75
79.31
77.5
70.0
Accuracy
70
78
72.5
Accura
65
67.5
67.5
64.67
66.0
57.77
57.17
57.5
ARC-E
ARC-C
OBQA
87.5
70
68 26
70.56
74.40
85.0
84.05
Accuracy (96)
70-
Methods
Accuracy KY
82.5
Huginn
77.5
Huginn-SFT
75.0
So
MoDr (Ours)
69.49
37.71
37.60
67.5
Figure 7: Performance comparison of MoDr with baseline methods on commonsense reasoning benchmarks. The Huginn results are reported by (Geiping al. 2025). The "Average" represents the overall mean of the six commonsense datasets.
http:///wgginniacccooooooe///see 170k
16

---

Published as a conference paper at ICLR 2026
A.3 CASE STUDY
We present three cases as follows.
Question: James decides run. 3 sprints 3 limes a week. He runs 60 meters each sprint How many total meters does he run week?
Answer:
meters James runs week,
steps:
Calculate the number of aprints James runs in week
the distance sprint
Multiply number sprints the total distamee
Fine
total distamoe
weeks
number sprints
week runs
sprints times week.
Therefore, number of sprints
Himes
need determine
one sprini
speint
meters.
run one
9 Himes 60 540
meters
!hoxed $40
Figure 8: Case study: The dynamic trajectories of text tokens across various reasoning branches in MoDr, color-coded as
2 and for a test instance from GSM8K.
Question: Lana was unboxing some of her old winter clothes, She found boxes of clothing and inside each box there were scarves and mittens. How many pieces of winter clothing did Lana have total!?
Answer: To determine the number
of winter clothing I Land to follow these
Calculate the number @il scarves the boxes.
2. Calculate the number of mittens the boxes 3] Add the mumber of searves and mittons togethel
First, let's culculate number
boxes Euch contains scarves there
buxes. Therefore (be total number
scarves ist
scarves pet
sccrrves
Next, let's calculate the mumber of mittens in the boxes, Fach box contains mittens, and there are boxes. Therefors, the total numbar
text
boxes
kimes 4 text mittens per hox)
ttx miltens
the number acarves mittens logether
pieces clothing!
[ 9 text scarves
12 hexti mittens] 21 Mext pieces winter clothing?
Thus, total mumber
pieces
winter clothing has
/boxed
Figure 9: Case study: The dynamic trajectories of text tokens across various reasoning branches in MoDr, color-coded as
and for a test instance from MultiArith.
From these cases, we have two interesting findings:
During each inference process, not all branches participate throughout the entire lifecycle. This is shown in the first two cases.
Branch handles most of the upfront analysis and planning, while Branches 3, and 4 focus detailed reasoning and reaching conclusions. This is demonstrated across all cases.
This in turn, corroborates hypothesis: distinct contextual inputs necessitate different computa- tional demands. a dynamic mechanism for invoking specialized branches should be employed for next-token prediction.
17

---

Published as a conference paper at ICLR 2026
Question: Mika had 20 stickers. She bought 26 stickers from store in the mall and got 20 stickers for her birthday. Then Mika gave 6 the stickers to her sister und used 58 to decorate a greeting card. How many stickers does Mika have left?
Answer: To determine how many stickers Mika has left, to follow these steps
1. Start with the initial number of stickers Mika had.
2. the number stickers she bought.
3. Add the number of stickerss she received for her birthday,
4 Subtruct the number of stickers she gave la her sister.
Subtract the number stickers sbe used decorato greeting card.
Let's go through each stop in detaill
Mika initially had 20 stickens
She bought stickers. So, total mumber or stroker she after buying
[
She received 20 stickers for her birtbday So thee total number of stickers abs has aho receiving the birthday stickers
She gave stickens
total number sstickers she hus left after giving stickers her
66 - 60
stickens to decorated greeting sard. So, total number stickers has after decorating thepprecting
Therefore, the numbor
stokkrs
Mika has
lett lboxod
Figure 10: Case study: The dynamic trajectories of text tokens across various reasoning branches in MoDr, color-coded
and for test instance from SingleEq.
A.4 ADDITIONAL EXPERIMENTS FOR CODE GENERATION
A.4.1 EXPERIMENTAL SETUP
To evaluate the model's code capabilities, we fine-tuned it on the CodeTeedback-FFile Instruction Zheng al. 2024) dataset. After cleaning the data remove invalid entries. obtained a final traming set of 79K samples. For evaluation, we used the HumanEval (Chen et al.) 202T) and MBPP (Austin et al. 202T) test sets, adopting Pass @ as the evaluation metric. The base- line methods and experimental settings were kept consistent with those used for the mathematical
reasoning tasks (see Section 4.1
A.4.2 RESULTS AND ANALYSIS
Figure presents the @ I scores the HumanEval and MBPP benchmarks. As shown, proposed MoDr achieves substantial and consistent improvements over all baselines average per- formance, suggesting its potential for scalability. These results demonstrate that MoDr effectively extends the reasoning capabilities of the depth-recurrent Huginn model across wide range of tasks, achieved by constructing dynamic-routing. multi-branch mechanism within its recurrent rumina- tion module.
Humaneval
MBPP
Average
29
33
28.05
34.00
34
32
31.23
28
Methods
23
30.11
Pass@1
27-
26.22
28
30
29
Huginn-SFT
Huginn
27
25
25.00
28
27.60
25
MoDr (Ours)
Figure 11: Performance (Pass@) 1) comparison of MoDr with baseline methods on code generation. The "Average" represents the overall mean of the HumanEval and MBPP datasets.
18

---

Published as a conference paper at ICLR 2026
A.5 COMPARISON TO STANDARD TRANSFORMER MODELS OF SIMILAR SIZE
As depicted in Table 5 we compare our proposed MoDr with several open-weight standard Trans- former models of similar scale. is worth noting that some of the open-weight baselines are trained on more computational resources (10-15x). The open-weight models results are reported by (Geip- ing al. 2025). These results further validate the advantages of our proposed multi-branch dynamic routing mechanism within the depth-recurrent transformer architecture.
Table 5: Performance comparison of MoDr with standard transformer models of similar size on commonsense reasoning benchmarks. The top score in each column is in bold.
Model
Param
Tokens
PIQA
HellaSwag
WinoGrande
ARC-E
ARC-C
OBQA
Average
Pythia-2.88
2.8B
0.3T
73.29
59.17
57.85
58.00
32.51
35.40
52.70
OLMo-7B
7B
2.75T
80.69
77.76
67.17
74.28
43.43
41.60
64.16
Qwen2.5-3B
18T
78.18
55.02
67.80
77.23
44.54
29.40
58.70
Llama3 2-3B
3B
9T
76.77
55,20
69.22
74.58
42.58
31.00
58.30
Llama-3.1
8B
15T
80.20
59.99
73.72
81.73
51.02
33.00
63.28
Huginn-3.5B
3.5B
0.8T
75.79
64.67
57.77
69.49
37.71
37.60
57.17
MoDr (Ours)
3,5B
0.8T
79.71
80.15
75.93
85.52
70.56
78.40
78.38
A.6 COMPARISON TO COT, MAJORITY VOTING, AND FULL FINE-TUNING
For comprehensive performance evaluation, we also compared our proposed MoDr against sev- eral widely-used baselines, including few-shot Chain-of-Thhouge (CoT) prompting, Majority Vot- ing (MV), and full fine-tuning (Full-FT). For the CoT and MV baselines, we adopted an eight-shot prompting configuration. As shown in Table the results indicate that MoDr demonstrates signifi- cant performance advantages over all the basclines.
Furthermore, we made two key findings. First, compared to zero-shot prompting with the "reason step by step" instruction, the few-shot prompting does not improve the performance of the Huginn model. This may be attributed to the model's 's latent reasoning mechanism, which appears to respond better to implicit guidance. Second, found that the performance of full fine-tuning remains suboptimal despite the convergence of the training loss curve. This suggests that LoRA fine-tuning more stable and effective fine-tuning method for the Huginn model.
Table 6: Performance MoDr against baselines on mathematical reasoning benchmarks. We com- pare against eight-shot Chain-of- -hoought (CoT), CoT with Majority Voting (CoT-MV), and Full Fine- Fine-nuning (Full-FT). The best score each column is bold.
In Domain
Out of Domain
Method
Average
GSMSK
MAWPS
AQuA
MultiArith
AddSub
SingleFq
CoT
37.23
72.69
27.95
76.33
74.18
73.03
60.24
COT-MV
37.76
79.41
29.53
81.17
79.49
74.02
63.56
Full-FT
6.67
50.00
6.69
78.83
65.32
47.83
42.56
MoDr (Ours)
49.89
80.67
33.07
91.17
79.24
81.30
69.22
A.7 EXPANDED RESULTTS OF CHALLENGING REASONING BENCHMARK
We conducted additional evaluation on the Super- Super-GPQA Du et al. 2025) benchmark, composite dataset curated be challenging across approximately 285 graduate disciplines. shown in Table 7 our MoDr achieves the best performance and significantly outperforms the base Huginn model, which performs nearly at a random level. This demonstrates the effectiveness our proposed MoDr even this out-of-domain challenging benchmark. However, we also acknowledge that the absolute performance of both Huginn-SFF and MoDr on this highly challenging dataset is not yet satisfactory. We attribute this two main factors: first, the relatively small scale of the base Huginn- 3.5B model; second, the limited pre-traine training corpus of 0.8T tokens compared to the 10T+ tokens typically used for modern LLMs. These limitations likely constrain the performance of MoDr
19

---

Published as a conference paper at ICLR 2026
highly challenging reasoning tasks. Therefore, exploring the mixture-deplee recurrent architecture more advanced Recurrent Transformer models to achieve better generalization remains a promising direction for future research.
Table 7: Performance on Super-GPQA benchmark. The best score in each column is bold.
Benchmark
Random
Huginn
Huginn-SFT
MoDr (Ours)
Supe
10.59
10.63
14.07
14.40
A.8 ANALYSIS TOP-K ROUTER
MoDr is flexible enough to support Top-K routing, in addition to its default Top-1 hard-gate router. To validate the impact the number routing branches (Top-K) on MoDr 's performance, we conducted ablation study the Top-1 and Top-2 routing configurations. The results, shown in Table g indicate that Top-2 routing improves performance on most datasets, suggesting that aggregating information from multiple experts is beneficial. However, we observed significant performance drop on the AQuA dataset. This highlights trade-off: while Top-K routing enhances performance across diverse tasks, may introduce instability or sub-optimal routing on certain benchmarks.
Table 8: Impact of the number routing branches (Top-K) on the performance of MoDr. The top score in each column is in bold.
Method
Top-K
GSM8K
MAWPS
AQuA
MultiArith
AddSub
SingleEq
Average
49.89
80.67 84.45
33.07 28.35
91.17 96.67
79.24 85.82
81.30 84.06
69.22
MoDr
50.27
71.60
A.9 PERFORMANCE COMPARISON WITH SAME TRAINING DURATION
To further validate the advantages of MoRA+MoE, we conducted additional experiments to evaluate the contribution of the proposed design. Specifically, fine-tuned Huginn-SFT, MoDr (Top-1), and MoDr (Top-2) under the same training budget. The results, detailed in Table demonstrate that MoDr with Top-1 and Top-2 routing outperforms the Huginn-SFTT baseline by +0.25% and +3.03%, respectively. Notably, the Top-2 variant exhibits superior multi-branch dynamic reasoning capabil- ities. These findings confirm that MoDr achieves substantial improvements with the same training duration, reinforcing our central claim: the performance gains are attributed to the LoRA+MoE design rather than increased training time.
Table 9: Performance comparison with Huginn-SFT on mathematical reasoning benchmarks. The top score in each column in bold.
Method
Top-K
MAWPS
AQuA
MultiArith
AddSub
SingleEi
Average
Huginn-SFT
80.25
31.50
93.17
77.97
81.30
72.84
1
80.67
33.07 28.35
91.17
79.24
81.30
73.09
MoDr
2
84.45
96.67
85.82
84.06
75.87
A.10 DETAILS FOR EXPERIMENTS
In Table 10 we report the computational efficiency and resource requirements of different methods. From this table, it is evident that our MoDr introduces only modest number of additional train- able parameters compared to Huginn-SFT, resulting a comparable FLOPs count. Importantly, its deployment memory and inference speed remain par with the baseline methods. This demon- strates that our proposed dynamic routing and multi-branch architecture improves performance at additional cost deployment memory inference speed.
20

---

Published as a conference paper at ICLR 2026
Table 10: Details for Experiments.
Metric
Huginn
Huginn-SFT
MoDr (Ours)
Trainable Parameters
3.56B
2.03M
8.13M
TFLOPs
35.00
35.55
35.60
Training Memory
N/A
54GB
54GB
Deployment Memory
13GB
13GB
13GB
Training Duration
N/A
3h 58min
16h 26min
Inference Speed
15 token/s
14 token/s
13 token/s
A.11 EVALUATION DETAILS
We conducted evaluation experiments on models using the following system prompts. all experiments, the temperature of the models was set to 0.0001, and the max_tokens parameter was set to 1024. The number of steps for the recurrent module was set to 16. In particular, the AQuA dataset sometimes fails follow the specified instructions for the final answer (e.g., placing a numerical value in a box rather than the corresponding option), we also manually double-checked the gold answers.
Evaluation Prompt for Mathematical Reasoning Tasks
System: Please reason step by step, and put your final answer within [i User: [Question]
Assistant:
Evaluation Prompt Commonsense Reasoning Tasks
Below instruction that describes task. Write response that appropriately completes the request.
System:
User:
### Instruction: {Instruction]
##0 Response:
Assistant:
Evaluation Prompt for Code Generation Tasks
System: You an exceptionally intelligent coding assistant that consistently delivers accurate able responses to user instructions.
User:
// For Humaneval
#### Instructien: Read the following function signature and doestring, and fully implement the func- tion described. Your response should only contain the code this function.
Function signature: (Example)
#*# Response:
For MBPP
### Instruct eeserrction in:: {Test}
Example Test Cases: Listt
#iil Response:
Assistant:
A. 12 FINE-TUNING EXAMPLES
The following are example questions and solutions from the mathematical reasoning, commonsense reasoning, and code generation datasets used for fine-tuning in our experiments.
21

---

Published as a conference paper at ICLR 2026
A.12.1
MATHEMATICAL REASONING TASKS
GSM8K
System:
Please reason step by step, and put your final answer within (boxee Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether
User:
April and May?
Assistant: determine the total number clips Natalia sold April May, we start by finding out how many clips she sold in May. According to the problem, Natalia sold half as many clips in May as she did in April. Since she sold 48 clips in April, we calculate the number of clips sold in May as follows:
Clips sold in May
24
Next, we add the number of clips sold in April to the number of clips sold in May to find the total number of clips sold over the two months:
Total clips sold = Clips sold April Clips sold in May = 48 24 72
Therefore, the total number of clips Natalia sold in April and May is 72
AQuA
System:
Please reason step by step, put your final answer within ()
Carrie to buy t-shirts the local clothing store. They cost $9.95 each. One day, she bought 20 t-shirts. How much money did she spend? The options are: AJ$150 BY$248.75 C)$199 D)$171.6 F)$190
User:
Assistant: To determine how much money Carrie spent t-shirts, need multiply the cost of t-shirt by number t-shirts she bought. The cost of one t-shirt $9.95, and she bought t-shirts. Let's perform the multiplication step-by-step 1. Multiply the cost of one t-shirt by the number of t-shirts:
9.95 20
To make the multiplication easier, we can break it down:
9.95 20 (10 0.05) x 20
3. Use the distrihutive property to multiply:
(10 0.05) 20 10 20 0.05 20
Calculate each part separately:
10 x 20 = 200
0.05 20
5. Subtract the second result from the first:
199
Therefore, Carrie spent dollars.
MAWPS
System: Please reason step by step, and put your final answer within $iboxed()$. For the fifth grade play, the chairs have been put into 27 rows with 16 chairs in each row, How many chairs have been put out for the play ?
User:
Assistant: To determine the total number of chairs put out for the play, need to multiply the number of rows by the number of chairs in each
Here the steps:
Identify the number rows: 27 Identify the number of chairs in each row: Multiply
number of rows by the number of chairs in each row: 27 x 16 Now, let's perform the multiplication:
27 16 27 (10 6) (27 10) (27 6)
22

---

Published as a conference paper at ICLR 2026
First, calculate 27 x 10:
27 270
Next, calculate 27 x 6:
27 X 162
Now, the two results together:
270 + 162 = 432
Therefore, the total number chairs put out play 432
A. A.12.2 COMMONSENSE REASONING TASKS
PIQA
System: Below is an instruction that describes task. Write response that appropriately completes
the request.
User:
#### Instruction Please choose the correct solution to question: When boiling butter, when it's ready, you can
Solution1: Pour it onto plate
Solution2: Pour into a jar Answer format: solutionl/solutiin ### Response:
Assistant: the correct answer solution2
HellaSwag
System: Below instruction that describes task. Write response that appropriately completes the request.
User:
### Instruction: Please choose the correct ending to complete the given sentence: Removing ice from Then, the writes the covering the window of car, and woman wearing winter clothes smiles. then Endingl: the man adds wax to the windshield and cuts Ending2: person board ski lift, while two men supporting the head of the person wearing winter clothes snow as the we girls sled. Ending3: the man puts christmas his car.
knitted with netting. Ending4: the continues removing the
Answer format: enddingg/oouiinn222ddin///nennn
### Response:
Assistant: the correct answer is ending4
WinoGrande
System:
Below an instruction that describes task. Write a response that appropriately completes
the request.
User:
#### Instruction: Please choose the correct fill in the blank complete the given Tan volunteered to Dennis's menudo after already having bowl hecause despised eating intestine.
sentence:
Option1: Tan Option2: Dennis Answer format: option /option2
#ill Response:
Assistant: the correct answer is option2
ARC-E
System:
Below an instruction that describes a task. Write a response that appropriately completes
the request.
User:
23

---

Published as a conference paper at ICLR 2026
### Instruction: Please choose the correct answer the question: Which factor will most likely cause person develop fever?
Answerl: a leg musele relaxing after exereise Answer2: a bacterial population in the bloodstream
Answer3: several viral particles on the skin Answer4: carbohydrates being digested in the stomach
Answer format: answer wwwpe/////wwee//////w/aa/asvee
##ll Response:
Assistant: the correct answer is answerr
ARC-C
System: Below an instruction that descrihes task. Write a response that appropriately completes the request.
User:
#ill Instruction: Please choose the correct answer the question: George wants his hands quickly by rubbing them. Which skin surface will produce most heat? Answerl: dry palms Answer2: wet palms Answer3: palms covered with oil Answer4: palms covered with lotion
Answer format: anw.et//asseere/sswerraaaaeeet
### Response:
Assistant: the correct answer is answerl
OBQA
System: Below is instruction that describes task. Write response that appropriately completes the request.
User:
#illi Instructions Please choose the correct answer to the question: The sun is responsible for Answerl: puppies learning new tricks Answer2: children growing up and getting old Answer3: flowers wilting vase Answer4: plants sprouting, blooming and wilting
Answer format: wwwwer///awwwer/aaneessssvvvvee
#### Response:
Assistant: the
answerd
A. CODE GENERATION TASKS
CodeFeedback
System:
You exceptionally intelligent coding assistant that consistently delivers accurate and reliable responses instructions.
User:
#illi Instruct Write a recursive function that calculates the factorial of a given number in Python. The function should handle negative numbers well. Additionally, the function should only accept non-negative integers input and raise ValueError for any other input. The function should also check if given number prime number, and if so, return the factorial of the next prime number greater than the given number. ### Response:
Assistant: Here's recursive function in Python that calculates the factorial of given number, handles negative numbers, and raises ValueError for any non-negative integers:
'python
def factorial (n)
if not isinstance (n, int) raise ValueError ("Input must a non-nogative integer")
0:
if <= 1:
return
else:
return factorial (n-1)
24


Reviews:
[{"review_id": "E98E1oBRoK", "reviewer": "Reviewer_eRKJ", "summary": "This paper studies a modification to the recurrent-depth model Huginn aimed at improving performance on reasoning tasks. Specifically, it introduces MoDr: a set of multiple LoRA parameter groups and a routing mechanism that determines which LoRA parameter set will be active for a given token. After supervised finetuning on reasoning traces, MoDr outperforms the baseline model and the model with a single LoRA parameter set on mathematical and commonsense reasoning tasks.", "questions": "- Please find and discuss related works to address the contextualization weakness above. Some examples might include StructMoE or Lottery Ticket Adaptation.\n\n- Figure 1: Could this be modified to illustrate the proposed approach more clearly? In contrast, Figure 2 is very helpful and clear. \n\n- Line 246: The “relay race” analogy is unclear. Could you please clarify what is meant? Modifying Figure 1 could help address this point, too.\n\n- How is the introduced load balancing different from the approach proposed in Wang et al. (2024)?\n\n- Line 316: If the number of training epochs is greater for models with recurrent branches than it is for baseline models, should you have an ablation study that trains baseline models for more epochs? Relatedly, Figure 5 shows that performance improves with the number of branches, but the number of epochs is also changing, so it's unclear how much of the benefit is from branch count (as opposed to epoch count).", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "**Originality**  The submission shows that the multiple branches of a MoDr model specialize in different areas, and the routing mechanism appropriately sends data to the right specialist to obtain high performance (Table 3). This analysis of MoDr's LoRA+routing approach supports this general direction's potential for impact. Moreover, the performance benefits on reasoning tasks are notable. \n\n**Quality**  The baselines facilitate testing of the proposed approach’s components (e.g., the contribution of the router). The method's generalization to OOD tasks is well established.\n\n**Clarity**  The paper is mostly well written and clearly communicates key points. \n\n**Significance**  The overall idea to equip latent reasoning with different modes that a router can send tokens through is very interesting. As the authors point out, the potential for more efficient reasoning models makes this area of study significant.", "weaknesses": "I would be happy to raise my score if the following can be addressed:\n- There might be a weakness regarding the baselines' training durations. Please see \"Questions\" for details.\n\n- Important related work is not discussed in the main text and appears to be missing in the appendix. Clear discussion of prior work that uses MoE/routing + LoRA would help clarify the methodological novelty of the submission. Applying such an approach to Huginn-like models seems new, and the submission should clarify other novel aspects through such discussion.\n\nA minor weakness is that some points are made multiple times (e.g. chain-like reasoning and loss-free balancing are mentioned >5 times each in the first few pages), and more efficient writing could make space for clearer discussion of related work in the main text. Currently, related work is primarily discussed in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AODJbMg4N5", "reviewer": "Reviewer_o7Ko", "summary": "The paper proposes MoDr, a Mixture-of-Depth-Recurrent Transformer that improves test-time reasoning by replacing Huginn's single recurrent block with multiple LoRA-based branches and a hard-gate router.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed idea is both novel and compelling. The authors introduce an innovative multi-branch dynamic routing mechanism for recurrent transformers, leveraging LoRA adapters to create lightweight and diverse reasoning branches, all while sharing the backbone weights for efficiency.\n\n2. The method demonstrates strong performance, outperforming baseline models not only on in-domain tasks but also showing robust generalization to out-of-domain benchmarks.\n\n3. The paper presents a comprehensive analysis, including detailed ablation studies, router effectiveness evaluations, branch specialization insights, and load balancing assessments. These provide robust empirical evidence supporting the model’s effectiveness, adaptability, and practical value.", "weaknesses": "1. It would be helpful to include more details about the trained model, such as training duration, detailed parameter count, etc.\n\n2. A comparison with a standard Transformer model of similar size would strengthen the evaluation and better highlight the effectiveness of the proposed approach.\n\n3. Including memory usage and FLOPs would provide a clearer picture of the model’s efficiency and practical deployment cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q7t3AAbVkP", "reviewer": "Reviewer_kgDC", "summary": "This paper proposes Mixture-of-Depth-Recurrent (MoDr) Transformer, a novel extension of the depth-recurrent Huginn model. The key idea is to enhance the adaptability and exploration capability of latent-space reasoning by transforming Huginn’s single recurrent reasoning path into a multi-branch, dynamically routed system. Specifically, the authors:\n\nIntroduce multiple LoRA-based recurrent branches that share the Huginn loop weights, providing lightweight diversity in reasoning behavior.\n\nEmploy a hard-gate dynamic router to select the most suitable branch per token based on hidden-state context.\n\nDesign an auxiliary-loss-free load balancing strategy to prevent routing collapse during training.", "questions": "None", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Novel extension of depth-recurrent architectures — The work builds directly on Huginn and introduces a meaningful architectural innovation: turning the single-chain recurrent loop into a mixture-of-recurrent-paths framework.\n\nDynamic routing in latent reasoning — The proposed hard-gate router for token-level branch selection is conceptually elegant and empirically validated, pushing the field toward adaptive latent reasoning rather than verbal CoT.\n\nParameter-efficient design — The use of LoRA branches preserves the efficiency of the original model while expanding reasoning capacity.\n\nComprehensive experiments — Results on both mathematical and commonsense reasoning tasks are convincing, and ablations (router removal, branch count, load balancing) are thorough and insightful.\n\nStrong empirical gains with negligible overhead, making the approach practical for future LLM test-time reasoning research.", "weaknesses": "I am not very familiar with the previous work (i.e., Huginn), but I find this paper to be quite intuitive and well-motivated. The Huginn model repeatedly reuses a single core module for latent reasoning, which inherently limits the model’s exploration depth. In contrast, this paper introduces a gated routing mechanism combined with LoRA adapters to merge multiple parallel recurrent branches, thereby enhancing the reasoning capability of LLMs. Overall, this is an elegant and valuable piece of work.\n\nHowever, I still have some critical questions:\n\nThe paper only reports results on Qwen/Qwen2.5-Math-7B-Instruct, which is a relatively small model. Have the authors attempted to test larger LLMs (e.g., Qwen3, Qwen2.5-72B)? Since the proposed approach seems computationally lightweight, restricting the experiments to a single small model weakens the empirical support for the paper’s claims.\n\nThe evaluation benchmark is not sufficiently comprehensive. Some challenging reasoning benchmarks, especially in mathematics and science (e.g., AIME, GPQA, Super-GPQA), are missing, which limits the generality of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KwTPPFpbiS", "reviewer": "Reviewer_WUhc", "summary": "In this work, authors propose the Mixture-of-Depth-Recurrent Transformer (MoDr), which replaces the single recurrent loop in the Huginn model with multiple LoRA-based branches. To switch between branches authors propose a hard-gated dynamic routing mechanism that adaptively selects the best branch for each reasoning step. This allows the model to explore diverse reasoning paths in latent space efficiently. A loss-free load balancing mechanism ensures that routing remains stable and evenly distributed across branches.\nAuthors test their model on several math and commonsense reasoning tasks and show improvements over vanilla model and LoRA finetuning.", "questions": "See \"Weaknesses\" section", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper in well organized and easy to follow\n2. Authors propose a novel architecture, combining MoE with dynamic routing, LoRA-based adapters, and recurrent nature of the Huginn model. \n3. Authors explore an interesting paradigm for “dynamic latent exploration” during test-time reasoning, bridging the gap between latent reasoning and mixture-of-experts routing.\n4. Authors show improvements on six mathematical and six commonsense reasoning datasets, covering both in-domain and out-of-domain settings.", "weaknesses": "1. While overall work is novel, all its components - Depth recurrence, MoE, and auxiliary-loss-free load balancing, - are assembled from existing paradigms rather than introducing a fundamentally new principle\n2. The comparison set is limited mostly to Huginn and LoRA-Huginn models:\n2a. For fair comparison, authors should specify additional compute budget, specifically number of additional parameters, total number of trainable parameters, spent FLOPS, and change in inference time, if any. Since MoDr introduces routing and multiple LoRA branches, empirical runtime and memory cost comparisons with Huginn are essential.\n2b. Simple Chain-of-Thought prompting, majority voting, and full-SFT baseline are missing, while all being popular and cheap techniques widely used in literature and on practice.\n3. The paper only reports Top-1 routing; Top-k routing or soft gating could reveal trade-offs between performance and stability.\n4. The paper’s claim that routing improves reasoning exploration is plausible but largely untested. While the authors include a qualitative case study, no quantitative evidence shows that different branches develop distinct reasoning skills or that routing adapts to task complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)