MeSH: Memory-as-State-Highways for Recursive Transformers
Authors: Chengting Yu
Xiaobo Shu
Yadao Wang
Yizhen Zhang
Haoyi Wu
Jiaang Li
Rujiao Long
Ziheng Chen
Yuchi Xu
Wenbo Su
Bo Zheng
Venue: ICLR 2026 Poster (2026)
Keywords: Recursive Transformer
Language Model
Parameter Sharing
Parameter Efficiency
Primary area: foundation or frontier models, including LLMs
TL;DR: We diagnose why recursive transformers underperform and propose a targeted solution for building stronger recursive backbones.

Abstract: Recursive transformers reuse parameters and iterate over hidden states multiple times, decoupling compute depth from parameter depth.
However, under matched compute, recursive models with fewer parameters often lag behind non-recursive counterparts.
By probing hidden states, we trace this performance gap to two primary bottlenecks: __undifferentiated computation__, where the core is forced to adopt a similar computational pattern at every iteration, and __information overload__, where long-lived and transient information must coexist in a single hidden state.
To address the issues, we introduce a **Me**mory-as-**S**tate-**H**ighways **(MeSH)** scheme, which externalizes state management into an explicit memory buffer and employs lightweight routers to dynamically diversify computation across iterations.
Probing visualizations confirm that MeSH successfully resolves the pathologies by inducing functional specialization across iterations. On the Pythia suite (160M–6.9B), MeSH-enhanced recursive transformers consistently improve over recursive baselines and outperforms its larger non-recursive counterpart at the 1.4B scale, improving average downstream accuracy by +1.06\% with 33\% fewer non-embedding parameters. Our analysis establishes MeSH as a scalable and principled architecture for building stronger recursive models.

Published as a conference paper at ICLR 2026
MESH: MEMORY-AS-SSATT-HHGGHHE FOR RECUR- SIVE TRANSFORMERS
Chengting Yu'co, Xiaobo Shu" Yadao Wang", Yizhen Zhang Haoyi Wu ,Jianng Li", Rujiao Long", Ziheng Chen", Yuchi Xu", Wenbo Su", Bo Zheng Alibaba Group < Zhejiang University
ShanghaiTech University
ABSTRACT
Recursive transformers reuse parameters and iterate over hidden states multiple times, decoupling compute depth from parameter depth. However, under matched compute, recursive models with fewer parameters often lag behind non-recursive counterparts. By probing hidden states, trace this performance gap two primary bottlenecks: undifferentiated computation, where the core is forced to adopt a similar computational pattern every iteration, and information over- load, where long-lived and transient information must coexist in single hidden state. To address the issues, we introduce a Memory-as-Staiee Stats---------aaay scheme, which externalizes state management into an explicit memory buffer and employs lightweight routers to dynamically diversify computation across it- erations. Probing visualizations confirm that MeSH successfully resolves the pathologies by inducing functional specialization across iterations. On the Pythia suite (160M-6.9B), MeSH-enhanced recursive transformers consistently improve over recursive baselines and outperforms its larger non-recursive counterpart at the 1.4B scale, improving average downstream accuracy by +1.06% with 33% fewer non eendeddddding parameters. Our analysis establishes MeSH as scalable and principled architecture for building stronger recursive models. Our code available at wwtp:/ihhhh.ccm.iiiiiinuuuuuel
Highways (MeSH)
INTRODUCTION
Scaling up model parameters and data has been primary driver of improvements in the general capabilities large language models (LLMs) (Kaplan al. 2020 Hoffmann et al. 2022; Brown
let al.
2020:
Wei et al.
2022
Chowdhery et al.
2023
Grattation al.) 2024 OpenAl 2023 Snell
jet al. 2024 Liu et al.
2024:
Comanici et al.] 2025). However, further gains along this axis face
headwinds: the supply of high-quality text is nearing exhaustion (Villalobos et al. 2022 Muen-
nighoff et al., 2023) empirical scaling curves show signs of saturation (Hackenburg et al. 2025
Hoffmann al. 2022), and distributed pre-training incurs substantial, often super- communi-
cation overheads as models grow (Narayanan et al. 2021 Pati et al. 2023
Li al. 2024
Patterson
al. 202T Momeni al. 2025).
As parfieter-efffciien architectural response to the scaling bottlenecks of large models, recursive
transformers have recently attracted growing interest (Geiping al. 2025 Bae et al.) 2024 2025
Zeng et al.] 22255) Saunshi et al.) (2025). The core idea behind is to decouple computational depth from parameter depth by repeatedly applying compact, weight-shared core block loop. By breaking the tight coupling between these two depths, recursive transformers natively enable dy-
namic computation: they can, in principle, allocate computational budgets adaptively based on task difficulty to reduce inductive bias (Bac al. 2025), and open up new scaling axis of computational
depth, complementing model size and data volume (Zhu et al. 2025b Geiping et al. 2025 Saunshi
al. 2025
However, a critical challenge remains: under matched compute, recursive models with fewer pa- rameters often lag behind their non-recursive counterparts (i.c., they exhibit higher perplexity or lower accuracy compared to standard Transformers with equivalent FLOPS but unique parameters
Equal contribution
Corresponding author

---

Published as a conference paper at ICLR 2026
high simlenty
1.
Relativg Computations I Leforr
1.00
Normalizer Singular Value
Pairwise OKA Siillarit
8000
1,00
0.75
0:59
0.64
0.35
0.5
0.73
1.00
0.90
0.39
bra
6.67
0.90
0.47
0.4
hra
6.61
0.64
0.74
0.7
1.00
0.00
02
0:33
0.35
0.7
0.
Speciate
feate
20
40
Singular Value Index
(a] Skewed computational pattern
[b] Representational stagnation
(c) Loop representational collapse
Figure 1: Diagnostic visualizations of the recursive transformer. Analyses are performed on Pythia-410M- model with the Prelude-Reccuren--Coda architecture (3 core loops). Hidden state matrices (h Repxdimm are extracted from 500 samples from the Pile dataset. The states hemb, h has refer to the initial token embeddings, the states to each block, and the final output Section[4.1 (a) Skewed computational pattern. Plots the relative magnitude of the state update, calculated for each computational block (/) (Il F ||b|l>) F). where |1-1| is the Frobenius norm, which serves proxy for the computational effort of each block. The x-axis tracks the flow from Prelude through con- secutive Core iterations to Coda within the common
state. We leave further experimental details and analysis
as (h) h||
structure (Geiping et al. 2025). Bars show the mean and standard deviation across 500 samples. (h) Representational stagnation. Displays the pairwise Centered Kernel Alignment (CKA) (Kornblith et al., 2019) simi- larity with an RBF kernel between the hidden state matrices. High CKA values between consecutive
loop states (1(*), (((++)) indicate that the representation has ceased to evolve, trapping the model in fixed point. (c) Loop representational collapse. Shows the top 50 normalized singular values (as//o) for each hidden state matrix on logarithmic Y-axis. The decay rate of the spectrum indi- cates the effective rank or intrinsic dimensionality of each state matrix.
per layer). this work, we provide measurable evidence that the performance gap stems from fundamental bottlenecks: undifferentiated computation and information overload, quantified by three observables as skewed computation, representational stagnation, and dimensional collapse. To address the pathologies, we propose the Memory-as Stss-Sttte-iile Highhays (MeSH) scheme, a principled architectural modification that replaces the overloaded hidden state with an explicit memory buffer governed by lightweight, step-wise routers. The proposed design separates persistent memory from transient computation, effectively converting the implicit challenge of state management into a clear, learnable routing problem for recursive transformers.
WHY NAIVE RECURSION FAILS: A DIAGNOSTIC ANALYSIS
The core premise of recursive transformer is to weight-shared computational block, yet the design introduces a fundamental limitation: the block lacks any explicit information about its progress within the iterative sequence, which prevents effective functional specialization and leads to inefficient computation. This also forces a single hidden state to handle multiple conflicting informa- tion. We define these primary bottlenecks undifferentiated computation and information overload.
Undifferentiated computation. The inability to differentiate between loop steps prevents the model from assigning specialized roles to each iteration. This leads to two distinct failure modes. First, the model exhibits a skewed computational pattern, as shown in Figure Dp. The first core loop performs the vast majority of the computational work, while the update magnitudes of subsequent iterations drop to near zero. This sharp decay suggests that the features stabilize prematurely, indi- cating that the model fails to effectively utilize the computational depth of later loops or distribute its processing logic over multiple steps. Second, consecutive loop states exhibit high representa- tional similarity, indicating representational stagnation (Figure [p). We employ Centered Kernel Alignment (CKA) (Korbblih et al.) (219)) to measure this, as it provides a robust similarity metric invariant to orthogonal transformations, High CKA similarity (Kornblith et al. 2019) between
2

---

Published as a conference paper at ICLR 2026
secutive loop states (i.e., h()) = (1+ reveals that the model becomes trapped in fixed-point attractor, repeatedly applying a mear-identical transformation instead of progressively refining its representation.
Information overload. In principle, coordinating multiple functional roles across recursive steps benefits from nigher-dimensional representations, under naive recursion, however, the model is con- strained a single hidden state, making such separation difficult in practice. Concurrently, the single hidden state vector is forced to be the sole carrier for all information, creating severe bottleneck, where the single state could be forced to manage multiple, often conflicting, roles simultaneously:
Preserving key information from the initial input to prevent catastrophic forgetting and maintaining stability across repeated iterations.
Long-term Memory:
Working Memory: Preparing intermediate features for the subsequent iteration and supporting transient computations within each step.
This single-state constraint induces trade-off between stability and plasticity. Empirically, naive recursive models tend to prioritize long-term stability to avoid forgetting, leading the representation to converge toward stable, shared "common ground" that favors persistence over flexible pro- cessing. The information overload on the hidden state forces the model to find a low-dimensional "common ground" representation that can safely survive multiple transformations, which directly causes loop representational collapse. We quantify this by analyzing the normalized singular value spectrum of the hidden state matrices, a proxy for their effective rank (see Figure (Te). The singular value spectrum of the loop states decays much more rapidly than that of the initial state, indicating a collapse into lower-dimensional subspace and significant loss of expressive capacity. Comple- menting this, a subspace analysis across iterations shows that the variance of hidden states concen- trates within a shared, low-dimensional (Appendix E.8). Taken together, these observations are consistent with the hypothesis that, under naive recursion, information overload biases the system toward a stable, low-rank manifold that pre- serves global context at the expense of the nigh-dimensional capacity required for transient, stepwise processing.
consistent with dominant long-tern component
The diagnosis of undifferentiated computation and information overload directly motivates lution, MeSH, which is specifically designed to address these identified problems.
3. METHODOLOGYY: ALLEVIATING INFORMATION OVERLOAD AND ENABLING FUNCTIONAL SPECIALIZATION OPCCTIONALSPEIIIAIIZAIIIOOO
The pathologies diagnosed in Section - undifferentiated computation and information over- load-stem from the architectural limitations of naive recursion. this section, we develop a methodology aimed at alleviating these core issues. We first review common heuristic-based re- currence schemes, which use fixed, additive connections to supplement the context at each step, that can be seen attempts to mitigate information overload but do little to address the prob- lem of undifferentiated computation. We then introduce our proposed solution, MeSH, a general framework designed to systemically alleviate both information overload and the lack of functional specialization.
3.1 PRELIMINARIES: ARCHITECTURE OF RECURSIVE TRANSFORMERS
Overall Architectural Structure. Recursive transformers achieve computational depth by repeat- edly applying a shared, weight-tied core block, Jewe(.). The central idea is to refine a hidden state (0 E LxD (where is the sequence length and 1 is the hidden dimension) over a sequence of K iterations. Starting from an initial state h!),, the simplest form of recurrence updates the state as (1+1) = Sawe (h))). The core recurrence loop could be embedded within a broader network topol- ogy that defines how the initial state (0) is produced and how the final state (K) consumed. We adopt the Recuree-Reerree--oodd structure (Geiping et al. (025) (also called Siddle-Cycle (Bae jet al. 2025)), which augments the core recursive block with specialized, non-tied prelude and coda networks. The framework first uses prelude block, fpr. process the initial token embeddings
(hemb) and prepare the first state for the loop: h (0) fpre (hemb). The recursive loop then runs for K
3

---

Published as a conference paper at ICLR 2026
steps, after which its final output state, h(*), is passed to coda block, foods. produce the model's final representation: hanall fooda (h ())
Core Recurrence Variants. While the base recurrence, h((+1) = fcore (h))), represents straight- forward cascade of computations, it may struggle with information retention, as each iteration can overwrite or forget crucial aspects of its input. To alleviate this representational burden on the core, the state update can be augmented with historical information. We summarize several common vari- ants, which represent different strategies for information propagation (illustrated in Figure 2 The general update rule with such a context supplement is:
h (++1) Score (0)
sup
(0)
(1)
(f) where haap is supplementary context. Common choices for this context include:
Setting haup = h introduces a standard skip connection between adjacent iterations, which allows the core to learn residual update, enabling the model to incrementally refine the
Residual:
representation and aggregate information from all preceding steps (Yu et al. 2025 Zeng et al.
2025
Bae et al. 2025).
Anchor: Setting haup h)) explicitly tethers each iteration to the initial state that entered the
loop. The intuition to prevent the iterative process from drifting too far from the initial seman-
tics by continuously reinforcing the starting context (Yang et al., 2023 Mohtashami et al. 2023
Geiping et al. 2025).
Anchor#: An alternative, haup hemb. anchors each iteration the raw token embeddings.
The heuristic connectivity schemes can be seen as attempts to mitigate information overload. By providing a direct, additive path for historical information (like the initial state (0) or the previ- ous state h()), they partially offload the burden of memory from the main hidden state pathway. Note that these refer strictly to state-passing mechanisms improve information flow, distinct from parameter-addition methods like that specialize weights (Bae et al. 2024).. This allows the core to focus more on transformation rather than just preservation. However, these are rigid, non-adaptive solutions. The choice among them is often a heuristic design deci- sion that requires careful empirical validation. Crucially, they do little to address the problem of undifferentiated computation, as the core block remains blind to its position in the loop.
important note that
3.2 MESH: MEMORY-AS-STATEE
FOR RECURSIVE TRANSFORMERS
We move beyond fixed recurrence rules by introducing the Memory-as Atemory-as-Statt State- Hoite-Hggwwass (MeSH), a mechanism that replaces the simple state-passing scheme. As shown in Figure MeSH externalizes state management into an explicit state buffer controlled by learnable, step-wise read-write routers. This design decouples transient computation within the recursive from persistent memory, al- lowing the model to dynamically manage iteration sppooonccppicccccc information flow. The MeSH-augmented loop consists of several lightweight components:
1. State Buffer and Initialization. MeSH maintains a state buffer M with B memory slots, M (ma; m mB-1), 1), where each slot m, E RL*D shares the same dimensions as hidden states. Before the loop begins, the buffer is initialized by placing the raw token embeddings, homb into the first slot. This designated slot, mo, serves as a initial anchor to the original input. All other slots initialized to zero:
hamb and
m 8=0
0
(2)
Core Computation and Dynamic Routers. The core block feure remains the central compu- tational unit. The interface to the buffer is managed by step-wise Write and Read Routers (R
and (2(t) which have unique parameters for each iteration = O,... - 1. each step, they compute routing weights based on the current hidden state (2) E RRLD
w (0) write Softmax([inear write (h (h)))),
(t)
Softmax([.nnerr (h))))
read
4

---

Published as a conference paper at ICLR 2026
f coda
h
hy
Deup
state fusion (optional)
Rymbb
103
anchor" Jandhor rosidual
fcore
(b) Heuristic Recurrence
Recursion
Maop
H
read
hm
Memory
m
...
..
1
Sprclude
write
Numb
(a) General Recursive Architecture
(c) MeSH Recurrence
Figure 2: Comparison of recurrence schemes. (a) The general architecture of a recursive trans- former involves the general dataflow passing a state (t) through a core computational block foor to produce the next state h (f++1, (b) Common heuristic variants employ a fixed, additive state update to optimize the information flow, where the core output is supplemented by historical states hap (e.g., initial state (0) for anchor previous state (tt) for resi residual).
(c) Our proposed MeSH replaces this rigid addition with a dynamic memory mechanism, which explicitly manages historical states via learnable write and read operations, allowing the model to flexibly retrieve and combine information to form the next state h (t+1),
Each (!)ear function is one-layer projection that maps the D-dimensional hidden state of each token to vector of B logits, corresponding to the number of buffer slots. A softmax function is then applied along the slot dimension for each token to normalize these logits, producing the final (t) weight matrices w wrile and w (t)
real both of shape RLxB
MeSH-Augmented Recurrence and Integration. The fixed context supplementation
placed by memory update logic, as illustrated in Figure 2p. At each step f, the core first computes (4) its output h from the current state (c).
m Soore (*)
The buffer is then updated via a distributed write operation for a soft insertion of the state, where the output (t) is scaled by the computed write weights before being added the memory slot:
(++1) m b m
(*)
(c) m
for 0, ......
write,b>
where 3 denotes element- wise multiplication with broadcasting. Subsequently, the state for the next iteration, (t++1), is synthesized via read operation from the updated buffer:
h (++1)
Σ 6=0
m (t+1)
w
Wead,
In the prelude-reeurrent--ooa setting, a dedicated transitional write-read cycle first processes the prelude's output fpre (hemb) to synthesize the initial state (o), After the main loop, a final read operation computes the output (4) from the memory buffer before passed to the coda. The full computational process is detailed in the pseudocode in Appendix

---

Published as a conference paper at ICLR 2026
3.3 How MESH ADDRESSES THE DIAGNOSED PATHOLOGIES
The architectural design of MeSH, centered on state externalization and dynamic routing, directly counteracts the core pathologies diagnosed in Section 2
Enabling Functional Specialization via Dynamic State Composition. MeSH explicitly breaks the cycle of undifferentiated computation by replacing the rigid, additive update rule of heuris- tic methods with dynamic read-write cycle controlled by step-wise routers. Since each router (R Riad has its own unique set of learnable parameters for each iteration the model is no longer forced to apply a single, universal transformation. Instead, at each step, it learns to dy- namically synthesize the next state by retrieving context-specific mixture of information from the memory buffer, which contains all relevant historical states. This flexibility allows McSH to learn and dynamically switch between complex recurrence behaviors. The ability to adapt the recurrence rule on the fly is the implicit mechanism to enables functional specialization.
Alleviating Information Overload via State Externalization. McSH directly alleviates infor- mation overload by decoupling persistent memory from transient computation. The external state buffer M serves dedicated, multi-slot highway for long-lived information. This design relieves the primary hidden state (O) from the burden of simultaneously storing historical context and serv- ing as the workspace for the core block. The hidden state can now utilize its full dimensionality for complex, transient computations, knowing that essential long-term information is safely preserved in the buffer and can be retrieved on demand by the read router. This allows the model to maintain nigh-dimensional, expressive representations throughout the entire iterative process.
In essence, M&SH replaces the single, overloaded information channel of standard recurrence with multi-slot memory buffer and dynamic, state-aware routers. The principled design provides systemic and highly expressive solution to the core problems in recursive transformers, subsuming prior heuristic approaches into more general framework (see Appendix D for more discussion).
4 EXPERIMENTS
We pretrain models from scratch, closely following the methodology of the Pythia suite (Bi- derman et al. 2023). We employ the same GPT-NeoX-based architecture and train on a dedu- plicated subset of the Pile dataset (Gao al. 2020). curated by EleutherAI. For evaluation, assess two primary aspects of model performance. We report perplexity scores on the validation sets of the Pile (Gao al. (2020), Wikitext, and the Lambada (both OpenAl and Standard versions) datasets (Paperno et al. 2016) to measure language modeling capabilities. We also evaluate down- stream performance on a suite of 9 few-shot benchmarks using the LM Evaluation Harness frame- work (Gao et al. 2024). Detailed training configurations and evaluation procedures are described in Appendix B
4.1 A COMPARATIVE DIAGNOSTIC ANALYSIS OF RECURRENCE SCHEMES
In Section 2, we identified three critical symptoms arising from naive recursive transformers: skewed computational pattern, representational stagnation, and loop representational collapse. We conduct a detailed analysis of the internal dynamics of four model variants: base recursive model, two common heuristic variants (+res i duel and +anchor), and our proposed +mesh architec- ture. The analysis is performed results over 500 samples from the Pile dataset.
Pythia-410M model with configuration of 3 3+633+3, averaging
MeSH mitigates the skewed computational pattern. Figure visualizess the computational effort of each block, confirming the pathology of naive recursion. The base model exhibits an extreme computational imbalance: the first core loop (Ixf accounts for the vast majority of the work, while subsequent loops contribute negligibly, demonstrating a classic case of diminishing returns. While the +residiu dual and +anchor heuristics offer partial relief, the computational effort still decays sharply. In stark contrast, the +mesh sh model achieves a remarkably balanced computational distribution, with all three core loops contributing significantly. This demonstrates that MeSH's

---

Published as a conference paper at ICLR 2026
Relattvv Computational Effort
base
+residual
+anchor
+mesh
2.0
2.0
13
1.3
13
0.5
0.
051
00
Figure 3: Skewed Computational Pattern. Plots the relative magnitude of the state update, caleu- lated for each computational block (f) as 2h) - ( + [)bl()), where || ||e is the
Frobenius norm, which serves proxy for the computational effort of each block. Bars show the mean and standard deviation across 500 samples.
dynamic read-write mechanism endows the model with a sense of iterative progress, allowing it to assign distinct and meaningful computational roles to each step.
base
*residual
+anchor
mmesh
1.0
Pairwise CKA Similantty
Barb
1:0
8::00
0:0
1.00
11:42
0.8
M/
1.00
1.00
0.72
3.39
0.23
22
0.6
M/D
22
0.20
1.00
22
2/2
0:
177
N
2.22
4:35
0.39
1.00
0.32
2:22
6:3
0140
0.
2.00
Pg
0:33
0.32
0.00
3.28
6:32
0,43
0:22
3,00
0.2
19
1/2
100
1/2
102
1/
129h
Bure
Figure 4: Representational Stagnation. Displays the pairwise Centered Kernel Alignment (CKA) similarity with an RBF kernel between hidden state matrices (h E Resexiimm at different stages of the model. The matrix shows the mean similarity across 500 samples. High similarity (values near 1.0) between consecutive loop states indicates that representations have stopped evolving.
MeSH breaks representational stagnation. Figure displays the CKA similarity (Kornblith et al. 2019) between hidden states. High similarity between consecutive loop states (h!2), 1224, 3) signals that the model is trapped in a fixed-point attractor. The base model's loop states ex- hibit very high CKA similarity, confirming severe representational stagnation. The +mesh model reduces the similarity between consecutive loop states, proving it has broken free from stagnation while its memory buffer allows maintain a strong connection the initial context.
base
+residual
+anchor
+mesh
Hormd Sndrgaa (Mear
Moreel and Siearee Value (Ntonn
12
and Srgpar
10
callopsed states
10
10
Singiar Volue Indio
Singular Value Indox
Singiar Vilue Indox
Sing/lr Volue Inzlc
Figure 5: Loop Representational Collapse. Shows the top 50 normalized singular values (a,/ov) for key hidden state matrices logarithmic Y-axis. The decay rate of the spectrum indicates the effective rank or intrinsic dimensionality of each state matrix. faster decay signifies a collapse into a lower-dimensional representation. Lines and shaded areas represent the mean and standard deviation across 500 samples.
MeSH prevents loop representational collapse. Figure 5 plots the singular value spectrum for hidden states. In the base model, the loop states (1) (2) (3)) show a much faster spectral decay than the input state ())), confirming loop representational collapse into a low-dimensional sub- space result of forced "representational compromise". The heuristic fixes offer only marginal
7

---

Published as a conference paper at ICLR 2026
Table Comparison of MeSH, Recursive and Vanilla Transformers. Performance is measured by perplexity (PPL.) on four datasets and average accuracy (Avg. accf) on a suite of downstream tasks. The percentage reduction in non-embedding parameters for recur- sive models is shown in parentheses. The Layers for recursive models follow the format [ J Nop ( indicating the number of layers in the prelude, core, coda. Aacc shows the absolute accuracy change relative to the Vanilla ala non-recursive baseline. LD-O and LD- refer Lambada OpenAl and Standard. The best results within each recursive block are bolded, and second best results underlined.
Structure
Perplexity
Task Avg. acct Aace (%)
Scheme
Layers
Variant
Pile
Wiki
LD-0
LD-S
0-shot
5-shot
Vanilla
12
11.31
30.32
42.86
129.89
39.88
40.54
160M
base
11.79
32,32
53.06
217.87
38.90/
-0.98
39.29 / -1.25
Recursive
2+4R2+2
+anchor
11.63
31.69
50.38
195.11 11
38.81 / /1.07
40.15 / -0.39
(-33.3%)
+mesh
11.37
30.43
46.60
178.77
39.41
-0.47
40.60 +0.06
Vanilla 24
9.07
21.79
19.48
65.86
43.87
45.31
base
9.31
22.74
22.57
53.76
43.26
-0.61
45,03 / -0.28
Pythia-4
Recursive (-33.3%
4+8R2+4
+anchor
9.19
22.12
20.37
52.55
-0.17 43.70
45.68 / +0.37
+mesh
9.09
21.84
19.63
42.51
44.12 +0.25
45.56 +0.25
base
9.65
23.88
26.76
81.75
41.94 11.94/-1.93
44.01 / -1.30
Recursive (-50.0%)
+residual
9.69
24.05
26.31
76.76
42.16 -1.71
44.24 -1.07
3+6R3+3
+anchor
9.49
23.31
24.49
72.30
42.85 -1.02
44.90 / -0.41
+mesh
9.35
22.80
20.72
52.07
43.53
-0.34
46.04 40.73
Vanilla
7.96
17.66
13.53
33.65
46.95
49.07
Pythia-1B 1B
8.20
18.64
14.44
36.39
4
47.75 -1.32
+residual
8.19
18.46
14.18
35.54
-0.76 46.85
46.19
47.85 / -1.22
Recursive
3+5R2+3
+anchor*
8.07
18.06
12.90
30.56
-0.10
49.18
+0.11
(-31.3%)
+anchor
8.10
18.15
13.32
32.34
46.73 -0.22
48.83 / -0.24
+mesh
7.90
17.54
12.19
26.71
47.53 +0.58
49.51 +0.44
Vanilla
24
7.44
15.97
10.51
22.81
49.50
51.93
Pythia- 1,4
base
7.63
16.64
11.38
23.69
50.99 / -0.94
+residual
7.58
16.44
10.91
20.44
49.50
+0.00
51.18 -0.75
Recursive
4+8R2+4
*anchor*
7.51
16.27
10.81
19.14
49.29
-0.21
51.83 -0.10
(-33.3%%
+anchor
7.51
16.25
10.71
19.37
49.39
-0.11
51.27 -0.66
+mesh
7.39
15.84
9.72
19.39
50.56 / +1.06
52.79/ 40.86
gains. The +mesh model, however, demonstrates the ability to preserve representational rich- ness, allowing the hidden state to maintain high-dimensional, expressive structure throughout the iterative process.
4.2
MAIN RESULTS
We conducted experiments on Pythia models ranging from 160M to 1.4B parameters, creating recur- sive variants with approximately 33% fewer non-embedding parameters to compare against standard Sanilla basclines (non-recursive models with unique parameters) and simpler recursive schemes (Table [.. While naive recursion (base) degrades performance and fixed schemes like +anchor offer only partial recovery, our MeSH-enhanced models (+mesh) consistently outperform all other variants. The +mesh models can even surpass their larger, more parameter-heavy Vanilla coun- terparts. For instance, the Pythia-1.44 +mesh model, despite its smaller footprint, improves 0-shot and 5-shot average accuracy by +1.06% and +0.86% respectively over the Vanilla version, while also achieving ofte-of-the-an perplexity scores across all datasets. Furthermore, the performance advantage scales favorably with model size, confirming that McSH's dynamic state management is not only effective but also highly efficient and scalable architectural principle.
8

---

Published as a conference paper at ICLR 2026
2.4
base ranchor +mesh
Downstream Avg.
0-shot Accuracy (%6
50
Training Loss
2.0
4.9
base
44
rrssdual
ranchor
1.8
10k
20k
40k
60k
80k 100k 100k 120k
120k
Training Steps (log scale)
Training Steps
Figure Training Dynamics of Recursive Variants. Comparison of training loss and downstream O-shot accuracy for the 1.4B-Pythia-bas recursive models. (Left) Training loss curve over 120k steps on a logarithmic x-axis. (Right) Downstream average O-shot accuracy evaluated at checkpoints along a linear x-axis.
Downstream Avg. Acc. (%6
X63 29.285
2.20
Eval. Loss
96
Non-embedding parameters (million)
500
Non-embedding parameters (million)
Figure 7: Scaling Analysis of MeSH vs. Baselines. Performance of Vani lla (non-recursive), naive Recurs ve, and sm models plotted against non-embedding parameter counts. (Left) Average downstream accuracy (0-shot and 5-shot). (Right) Evaluation loss.
4.3 FURTHER ANALYSIS AND ABLATION STUDIES
Analysis of Training Dynamies. To understand not just the final performance but also the learning process itself, we visualize the training dynamics of the 1.4B 1.B--paraetter recursive variants in Fig- 6 By juxtaposing pre-training loss with downstream accuracy evaluated at various checkpoints, we can assess both the learning efficiency and the rate at which models acquire useful capabilities. The training loss curves (Figure left panel) reveal that the +mesh model consistently achieves a lower loss throughout pre-training. This indicates superior learning efficiency, as MeSH is able and
to the training data more effectively at every stage compared to the base, tanchor variants. The training advantage translates directly into stronger downstream perfor- mance. The right panel of Figure 66 shows that the +mesh model not only starts from a stronger initial checkpoint but also exhibits a steeper and more consistent improvement in 0-shot accuracy. The superiority provides compelling evidence that MeSH's architectural modifications fundamen- tally enhance the model's ability to acquire and retain useful knowledge throughout the entire pre- training process, rather than being just final-step improvement.
Scaling Properties and Parameter Efficiency. We provide scaling results in Figure revealing the parameter efficiency of the MeSH architecture. While naive recursive models (blue lines) con- sistently underperform their standard vanilla counterparts (green lines) despite saving about 33% of parameters, our +mesh models (purple lines) not only decrease the performance degradation but could even outperform the Vani lla baselines at large scales. For example, our 805M -arrmmetee +mesh model achieves 50.6% (0-shot) and 52.8% (5-shot) accuracy, surpassing the parameter Vani model's 49.5% (0-shot) and 51.99 .%% (5-shot), which translates to 1.46x im- provement in parameter efliciency, allowing McSH-enhance model to achieve the same level of evaluation loss as a Vanilla model with almost a third fewer parameters.
Impact of Layer Distribution and Parameter Scaling, To further dissect the architectural bene- fits of MeSH, we conduct a control study on the distribution of layers within the framework. Using the Pythia-410M architecture
testbed, train several recursive models with varying configurations while keeping the total compute equivalent to the 24-layer non-recursive Vani 1 la model. We plot the validation perplexity against the percentage of non-embedding pa-

---

Published as a conference paper at ICLR 2026
2+584+2
10.0
vanilla
#loop=2
Eval. Perplexity
recursive
#loop=3
9.8
+mesh
#loop=4
2+5R4+2
3+6R3+3
9.6
4+4R4+4
2
6+3R4+6
9.4
3+6R3+3
2+10R242
4+822+4
4+4R4+4
6+4R3+6
9.2
6+3R4+6
4+8R2+4
6+4R3+6
40%
50%
60%
70%
Relative Non-Embedding Parameters (%)
Figure 8: PPL vs. Parameter Efficiency for Pythia-410M. The plot shows the Pile perplexity as a function of non- coneeddddiing parameters, shown as a percentage relative to the Vanilla baseline. Each point represents different distribution of layers (prelude, core, coda). The total computational depth for models is aligned with the 24-layer non-recursive Vanilla.
rameters relative to the Vanill 1la baseline, with the results shown in Figure The +mesh archi- tecture (purple line) consistently achieves lower perplexity than the baseline recursive model (blue line) across all parameter allocations, demonstrating its robust performance advantage. MeSH also shows remarkable parameter efficiency against the non-recursive baseline. As trend, the performance of +mesh (purple line) approaches that of the full 24-layer Vanilla model (green dashed line) while using approximately 30% fewer non-embedding parameters. The study shows that MeSH is not just additive improvement but powerful architectural principle that enhances the parameter efficiency and scaling properties of recursive transformers.
CONCLUSION
In this work, we diagnose the underperformance recursive transformers, tracing it, through the lens of quantified observables, to the systemic pathologics of undifferentiated computation and infor- mation overload. We further propose MeSH as principled architectural solution that externalizes state management into explicit memory buffer controlled by dynamic, step-wise routers. Our experiments validate that McSH successfully addresses the diagnosed pathologies while also deliv- ering substantial performance gains recursive backbones. We conclude that this work establishes explicit, routed state management as a scalable and effective principle for building stronger recursive models, offering promising architectural path forward as the field seeks more sustainable scaling paradigms.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Preslav Aleksandrov, Meghdad Kurmanji, Fernando Garcia Redondo, David O'Shea, William Shen, Alex Iacob, Lorenzo Sani, Xinchi Qiu, Nicola Cancedda, and Nicholas D Lane. Abbie: Autoregressive block-based iterative encoder for efficient sequence modeling. arXiv preprint arXiv:2507.08567, 2025.
Sangmin Bae, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Seungyeon Kim, and Tal Schuster. Re- laxed recursive transformers: Effective parameter sharing with layer-wise lora. arXiv preprint arXiv:2410.20m6 2024.
Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, et al. sive depths for adaptive token-level computation. arXiv preprint arXiv:2507. 10524, 2025.
Learning dynamic recur-
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pp. 2397-2430. PMLR, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamerr. Advances in neural information processing systems, 33:1877-1900 2020. 2020.
Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. In Advances in Neural Information Processing Systems, volume 35, pp. 11009-11099 2022.
Xinghao Chen, Anhao Zhao, Heming Xia, Xuan Lu, Hanlin Wang, Yanjun Chen, Wei Zhang, Jian Wang, Wenjie Li, and Xiaoyu Shen. Reasoning beyond language: A comprehensive survey on latent chain-of-thought reasoning. arXiv preprint arXiv:2505. /6782, 2025a.
Yilong Chen, Junyuan Shang, Zhenyu Zhang, Yanxi Xie, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, and Haifeng Wang. Inner thinking transformer: Leveraging dynamic depth scaling to foster adaptive internal thinking. arXiv preprini arXiv:2502. 13842, 2025b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.85433.77 2018.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626/1 2025.
Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber, Christopher Potts, and Christopher D Manning. Moeut: -fxtere-of-eeperts universal transformers. Advances in Neural Information Processing Systems, 37:28589-28611 2024.
Damai Dai, Cheng Deng, Chen Zhao, Renze Xu, Hongkai Gao, Di Chen, Ju Li, Wei Zeng, Xin Yu, Y. Wu, et al. DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experrs language models, 2024.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe V Le, and Ruslan Salakhutdinov. Transformer-xl xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th annual meeting of the Association for Computational Linguistics, pp. 2978-2988, 2019.
11

---

Published as a conference paper at ICLR 2026
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. arXiv preprint
2018.
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. arXiv preprint arXiv:/910 10073, 2019.
Ying Fan, Yilun Du, Kannan Ramchandran, and Kangwook Lee. Looped transformers for length generalization. arXiv preprint arXiv:2409. 2024.
Evelina Fedorenko, Steven T Piantadosi, and Edward AF Gibson. Language primarily a tool for communication rather than thought. Nature, 630(8017):575-586, 2024.
William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. The Journal of Machine Learning Research, 23(1): 5232-5270, 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2 101. 101.00027. 2020.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. Utt.ttps: httttttss://eeoo......to zenodo .racctoorrg org/ rec. rss 122208622222222
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint 250.02250..0517,, 2025.
Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D Lee, and Dimitris Papailiopoulos. Looped transformers as programmable computers. International Conference Machine Learning, pp. 11398-1 11442. PMLR, 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:240..21783, 2024.
Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint
arXiv:1603.0888 0803.08983, 2016.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. 1rXiv:1410.540) 2014.
arXiv preprint
Kobi Hackenburg, Ben M Tappin, Paul Röttger, Scott A Hale. Jonathan Bright, and Helen Margetts. Scaling language model size yields diminishing returns for single-message political persuasion. Proceedings of the National Academy of Sciences, 122(10)ee211344312) 2025.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in continuous latent space. arXiv preprint arXiv:2412.06769, 2024. 2024.
Tamir David Hay and Lior Wolf. Dynamic layer tying for parameter-efficient transformers. arXiv preprint arXiv:2401.12811 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 033000 2020.
12

---

Published as a conference paper at ICLR 2026
Jaemu Heo, Eldor Fozilov, Hyummin Song, and Tachwan Kim. Ringformer: Rethinking recurrent transformer with adaptive level signals. arXiv preprint arXIv22500.131812202.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Jordan Hoffmann, Sehastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203.15553 2022.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 4700-4708, 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMIR, 2019.
Guanghao Li, Wenhao Jiang, Li Shen, Ming Tang, and Chun Yuan. Zero token-driven deep thinking in lims: Unlocking the full potential of existing parameters via cyclic refinement. arXiv preprint 1rXiv:2502.12214, 2025a.
GuoLiang Li and Yiyang Li. Recurrent multiple shared layers in depth for neural machine transla- tion. arXiv preprint arXiv:2108. 10417, 2021.
Jindong Li, Yali Fu, Li Fan, Jiahong Liu, Yao Shu, Chengwei Qin, Menglin Yang, Irwin King, and Rex Ying. Implicit reasoning in large language models: A comprehensive survey. arXiv preprint 0tt.:25509.03550. 2025b.
Wenxue Li, Xiangzhou Liu, Yuxuan Li, Yilun Jin, Han Tian, Zhizhen Zhong, Guyue Liu, Ying Zhang, and Kai Chen. Understanding communication characteristics of distributed training. In Proceedings of the 8th Asia--aciiia Workshop on Networking, pp. 1-8, 2024.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.11447, 2024.
Llama Team.
The Llama 4 Herd: The Beginning of New Era of Natively Innovation. 2025.
Multimodal
AI
URL https://ai meta. otlp:///o.mea.cccooooo/oooo//
Ilama mu t
Amirkeivan Mohtashami, Matteo Pagliardini, and Martin Jaggi. Cotformer: A chain-of- thought driven architecture with budget-adaptive computation cost at inference. arXiv preprint arXiv:2310. 10845, 2023.
Ali Momeni, Babak Rahmani, Benjamin Scellier, Logan G Wright, Peter I. McMahon, Clara C Wanjura, Yuhang Li, Anas Skalli, Natalia G Berloff, Tatsuhiro Onodera, et al. Training of physical neural networks. Nature, 64568019) 2025.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-50376, 2023.
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Wei- jia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.0206) 2024.
13

---

Published as a conference paper at ICLR 2026
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Effi cient large-scale language model training gpu clusters using megatron-Im. In Proceedings of the international conference for high performance computing, networking, storage and analysis, pp. 1-15, 2021.
Kei-Sing Ng and Qingchen Wang. Loop neural networks for parameter sharing. arXiv preprint 149999 2024.
Anthony Nguyen and Wenjun Lin. Intra-layer recurrence in transformers for language modeling. arXiv preprint arXiv:2505.01855 2025.
Parsa Omidi, Xingshuai Huang, Axel Laborieux, Bahareh Nikpour, Tianyu Shi, and Armaghan Eshaghi. Memory-augmented transformers: A systematic review from neuroscience principles to enhanced model architectures. arXiv preprinl arXiv:2508. 10824, 2025.
OpenAl. Gpt-4 technical report: Tech. rep. 2023.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint arXiv:1606.0600 2016.
Suchita Pati, Shaizeen Aga, Mahzabeen Islam, Nuwan Jayasena, and Matthew D Sinclair. Com- putation vs. communication scaling for future transformers on future hardware. arXiv preprint arXiv:2300.002855, 2023.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis Luquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2 104. 10350, 2021.
Jacob Pfau, William Merrill, and Samuel R Bowman. Let's think dot by dot: Hidden computation in transformer language models. arXiv preprint arXiv:2404.15758, 2024.
Jack W Rae, Jonathan J Hunt, Tim Harley, Ivo Danihelka, Andrew Senior, Greg Wayne, Alex Graves, and Timothy Lillicrap. Scaling anemory-augmente neural networks with sparse reads and writes. In Advances in neural information processing systems, volume 29, 2016.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. In International Conference on Learning Rep- resentations, 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 106, 2021.
Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank Reddi. Reasoning with latent thoughts: On the power of looped transformers. arXiv preprint 2rXiv:2502.17416 2025.
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing chain-of-thought into continuous space via self-distillation. arXiv preprint arXi::2502.21774, 2025.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 2024.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387 2015.
Sho Takase and Shun Kiyono. Lessons on parameter sharing across layers in transformers, arXiv preprint wxXv:2204.06002. 2021.
Shawn Tan, Yikang Shen. Zhenfang Chen, Aaron Courville, and Chuang Gan. Sparse universal transformer. arXiv preprint 10.00:23 2023.
14

---

Published as a conference paper at ICLR 2026
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? limits of Ilm scaling based on human-generated data. arXiv preprint arXiv:2211.04322 040325, 2022.
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Aug- menting language models with long-term memory. In Advances in Neural Information Processing Systems, volume 36, pp. 74530-74555 2023.
Jason Wei, Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. Xrxiv preprint arXiv:2206 07682, 2022.
Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv: 1rXiv:1707.06200 2017.
Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, and Zhou Yu. Memformer: A memory-augmented transformer for sequence modeling. arXiv preprint arXiv:2010.06891, 2020.
Yuhuai Wu, Markus N Rabe, DeLesley S Hutchins, and Christian Szegedy. Memorizing transform- ers. In International Conference on Learning Representations, 2022.
Da Xiao, Qingye Meng, Shengping Li, and Xingyuan Yuan. Muddformer: Breaking resid- ual bottlenecks in transformers via multiway dynamic dense connections. arXiv preprint arXiv:2502.12170, 2025.
Kevin Xu and Issei Sato. To cot or to loop? formal comparison between chain-of-thought and looped transformers. arXiv preprint arXiv:2505.1922 2025.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.093888 2025.
Liu Yang, Kangwook Lee, Robert Nowak, and Dimitris Papailiopoulos. Looped transformers are better at learning learning algorithms. arXiv preprint urXiv:2311.12424 2023.
Qifan Yu, Zhenyu He, Sijie Li, Xun Zhou, Jun Zhang, Jingjing Xu, and Di He. Enhancing auto- regressive 2025.
through loop-aligned reasoning. arXiv preprint arXiv:2502.08482,
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv::905.0700 2019.
Boyi Zeng, Shixiang Song, Siyuan Huang, Yixuan Wang, He Li, Ziwei He, Xinbing Wang, Zhiyu Li, and Zhouhan Lin. Pretraining language models to ponder in continuous space. arXiv preprint arXiv:2505.20674, 2025.
Xiang Zhang, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. Autoregressive+ chain of thought= recurrent: Recurrence's 's role in language models' computability and revisit of recurrent transformer. arXiv preprint arXiv:2409 240.099999999999999..99 2024.
Defa Zhu, Hongzhi Huang, Zihao Huang, Yutao Zeng, Yunyao Mao, Banggu Wu, Qiyang Min, and Xun Zhou. coperr-cnnections. arXiv preprint 1.Ki::249..16066 2024.
Rui-Jie Zhu, Tianhao Peng, Tianhao Cheng, Xingwei Qu, Jinfa Huang, Dawei Zhu, Hao Wang, Kaiwen Xue, Xuanliang Zhang, Yong Shan, et al. A survey on latent reasoning. arXiv preprint arXiv:2507.06203, 2025a.
Ruike Zhu, Hanwen Zhang, Tianyu Shi, Chi Wang, Tianyi Zhou, and Zengyi Qin. The 4th dimension for scaling model size. arXiv preprint arXiv:2506, 18233, 2025b.
15

---

Published as a conference paper at ICLR 2026
A
RELATED WORK
Recursive Transformers and Loop-based LLMs. The idea of iterating a Transformer layer in a loop originates from the Universal Transformer (UT) (Dehghani al. 2018), which showed that peatedly applying a single, weight-shared layer can achieve the expressive power of a much deeper Transformer while allowing variable computation per input. UT also introduced an Adaptive Com- putation Time mechanism (Graves) 2016) that dynamically adjusts how many iterations to run for each token, together with a trainable positional signal that distinguishes time steps. Since UT, a series of works have extended the concept of looped Transformers (Tan et al. [2033] Giannou et al.
2023
Li Li, 202T Takase & Kiyono| 202T Elbayad al. 2019] | Yang et al.] [203]] ZZhang al.
2024
Fan et al. [224]] Hay & Wolf |024|Cheen et al. 2025b Nguyen & Lin |025 [i et al. 2025a
Aleksandrov et al. 20.5 BBae et al. 2025). ). Recent studies (Saunshi et al. [025] Zhu et al. 2025b
Geiping et al.) (2225) demonstrated both empirically and theoretically- that increasing depth by
looping a small Transformer can match or surpass a far deeper fixed-depth model on challenging
reasoning tasks. Later, Zeng al. (2025) reinforced the view that iterative "pondering" could be critical for test-time scaling and linked the behavior of looped Transformers to an implicit chain- of-thought process. Collectively, these efforts underscore the promise of recursive transformers for adaptive depth and latent reasoning.
Sharing and Iteration Differentiation. A central challenge for recursive transformers is to preserve expressiveness even though all iterations reuse the same parameters. An empirical study of parameter sharing in Transformers showed that naïvely sharing every layer-as in the orig- inal UT (Dehghani et al., 2018) -often degrades performance on language tasks, implying that additional mechanisms are required alleviate representational bottlenecks (Ng & Wang. 2024).
Parameter
Several strategies have been explored: Learned loop-index embeddings. By injecting a small train- able vector or matrix that encodes the iteration number, models can behave slightly differently at each step while still sharing the main weights (Dehghani et al., 2018 Mohtashami et al., 20233) However, element- -iise addition of such embeddings practically produces limited gains (Geiping ett al. 2025| Zhu et al. (222a).. LoRA per iteration. In similar spirit, recent works (Heo et al. Bae et al. 2024) attach a separate low-rank adaptation (LoRA) module to each repetition of pre-trained model, granting every loop its own lightweight set of parameters and mitigating the drawbacks of strict sharing. Mixture-of-Exper
2025
in a loop. MoEUT (Csordás et al., 2024) combines weight sharing with a Mixture-of-Ease of-Exxerts (MoE) every layer: the base layer is reused across it- erations, while expert gating adds conditional capacity. MoEUT slightly outperforms a non-loop Transformer of equal compute, underscoring the value of learnable gating and expert routes within a loop architecture. Our work proposes a different paradigm. Instead of adding unique parame- ters to the loop core to differentiate iterations, we focus on dynamically managing the information flow itself. We propose MeSH to externalize state management into a memory buffer and employ lightweight, step-wise routers to control what is read from and written to it. This externalizes the functional specialization into a routing problem, keeping the core block purely weight-shared and non-invasive.
Skip Connections and Dense Connectivity. In deep networks, skip or shortcut connections have long been essential for training very deep architectures effectively (He et al. 2016). Residual Net- works (ResNets) and Highway Networks showed that adding identity skip paths improves gradient flow and allows each layer to learn a simpler "update function" on top of an identity mapping He et al. (2016); Srivastava et al. (2015). DenseNet (Huang al.) (017)) further generalized this idea by connecting every layer to all previous layers, so that each layer receives the feature maps of all preceding layers input; this dense feed-forward architecture promotes feature reuse, mitigates vanishing gradients, and even reduces parameter count. More recently, Hyper-zonnections (Zhu ct al., 2024) widen the hidden state into multiple parallel streams and use learnable coefficients to mix these streams, effectively replacing the standard residual path with more complex, multi- lane data highway. MUDDformer (Xiao et al., 2025) introduced dense connections into standard decoder-only Transformers, generalizing residuals by adding multiple learnable skip paths between layers; dynamic dense skips allowed a 2.8 model to match the perplexity of a 6.9 B model with only minimal overhead (Xiao et al.) 2025). While conceptually related to dense connectivity, our work focuses on block-to-block (loop- to- loop) connectivity in the recursive setting rather than layer- to-layer wiring. We diverge from direct connections by introducing an external memory buffer and lightweight, step-wise routers. The system facilitates a flexible read-write cycle for managing infor- mation flow across iterations, rather than simply gating feed-forward paths. The mechanism is both
16

---

Published as a conference paper at ICLR 2026
principled and more native to the recursive design, it is explicitly engineered to enable functional specialization between iterations.
Latent Reasoning and Chain-of-Thouun latent (hidden) bhain-of-thougee (Hao et al. 2024 Shen et al. 2025). Instead of explicitly outputting intermediate reasoning steps in natural language, a loop-based model processes those steps internally in vector form (Zhu et al. 2025a Xu & Sato 2025). Recent research has examined the differences
Loop-based LLMs closely related to the idea of
between prompting model with an explicit chain-of-thought versus giving it the capacity to "think" silently via latent reasoning (Chen et al. 2025a Fedorenko et al. 2024] Hao et al. 2024] Pfau et al.) 2024]. In general, loop offers promising way achieve the benefits of multi-step reasoning without incurring the cost of longer outputs or the need for supervised intermediate steps (Li et al. 2025b). Our work contributes to this area by improving the recursive architectural foundation on which latent reasoning unfolds. By enhancing how information is preserved and combined across iterative steps, we aim to make iterative execution more effective. While approaches like CoTForerr inject special tokens to mimic multi-step reasoning inside the model (Mohtashami et al., 2023) most of recursive transformers focus on intrinsic connectivity of loop iterations (Zeng et al. 2025 Geiping et al. 2025| Saunshi al.) 2025). These two methodologies can be seen complementary, as we can imagine a loop-based LLM that also uses latent CoT training signals. Indeed, analysis of hidden state evolution under different connection schemes can be viewed as an interpretability study of latent reasoning shedding light on whether the model is gradually refining a solution or oscillating, and how much it relies on initial information versus newly computed results at each step.
Augmented Transformers. Our work is situated within the broad field of memory- augmented Transformers (Omidi et al., 2025), including diverse approaches to enhance neural net- works with memory modules. A historic line of memory-augmented neural networks use memory for long-term knowledge storage and algorithmic reasoning, evolving from the cell state in LSTMs (Hochreiter & Schmidhuber, 1997) to architectures with explicit external memory matrices, such as the Neural Turing Machine (Graves et al., 2014) and its sparse-access variants (Rae et al. 2016). In the Transformer era, a line of work uses memory to address the fixed context-length limitation of Transformers. This began with Transformer-Xe
(Dai et al. 2019) caching hidden states from previ- segments, was improved by Compressive Transformer (Rac et al.) (2000) which compresses older states, and was further developed by models like RMT (Bulatov et al. (022)) and Memformer (Wu et al. 2020) that introduce dedicated memory tokens slots. More recent models, such as LONG- MEM (Wang et al., 2023) and the Memorizing Transformer (Wu et al.] 2022). employ retrieval- based augmentation, where key-value pairs from past segments are stored in an external bank and a retrieval mechanism is used to pull relevant information into the current context. The shared ob- jective of these varied approaches is enable information flow across long temporal sequences or distinct, sequential input segments for knowledge storage or context extension. In contrast, our pro- posed MeSH mechanism operates on a single, fixed-length input, where the memory buffer manages the intermediate hidden states generated during successive computational iterations over that same input. The function of McSH is to structure the information flow within a token-wise recursive computation, a target distinct from long-term knowledge storage or long-context extension.
B EXPERIMENTAL DETAILS
Pre-training. All models are pretrained from scratch, closely following the methodology of the Pythia suite (Biderman et al. 2023). Our training is conducted on a 250B-token deduplicated subset of the Pile dataset (Gao et al. 2020). using the original GPT-NeeX tokenizer with a vocabulary size of 50,257. All models are trained for one epoch.
Model Architecture. Our implementations are based on the GPT-NeoX architecture provided by the Pythia suite (Biderman et al.] (003). For recursive models, we adopt the prelude-loop-coda. structure. We denote the layer distribution using the notation Lpre +Icooe +Lzods- For ex- ample, a ++8R2++ 4 configuration corresponds to a model with a 4-layer prelude (Ipre), ) an 8-layer shared core (Loore) that is looped twice (N (Nooe and a 4-layer coda (Looda). Our +mesh variant is implemented by inserting a state buffer and step-wise routers at the boundaries of these conceptual blocks. Each router consists of a single linear layer followed by softmax function to generate dy- namic routing weights. The buffer size B is set following the empirical B = Nuwp + 3 derived from our ablation study (see Appendix R.2 Following standard Transformer practice (Vaswani et al.
17

---

Published as a conference paper at ICLR 2026
2017). scale the input embeddings by factor of dimodel before they enter the first layer. To ensure training stability in these deep computational graphs, we employ a depth-awar aware weight ini- tialization, scaling the standard deviation of output projection weights by 1/ 1/v2 N computee where N Ncompute is the total number of layers in the unrolldd computation graph.
Training Hyperparameters. We use the AdamW optimizer with 3 = 0.9, 82 - 0.95, a weight decay of 0.01. The learning rate follows a cosine decay schedule with a 1% warmup, decay- ing to 10% the peak value. The peak learning rate is scaled according to model size, ranging from 6.0 10-4 for the 160M model to 2.0 x 10 for the 1.4B model. All models consistent global batch size of 512 and a sequence length of 4096 tokens. To improve training eff- ciency, we utilize BF16
trained with
and FlashAttention-22 (Dao 2023). Our distributed training
setup is managed by DeepSpeed with ZeRO Stage O.
Downstream Task Evaluation. assess model performance, we evaluated few-shot accuracy
on 9 benchmarks using the Language Model Evaluation Harness framework (Gao al. 2024). The evaluation suite includes: Lambada (Paperno al. 2016) in both its OpenAI (LD-O) and Standard
O
(LD-S) versions, PIQA (PQ) (Bisk et (020)), HellaSwag (HS) (Zellers et al. 2019), WinoGrande
(WG) (Sakaguchi et al. 2021). 1. Easy (ARC-E) and ARC-Challenge (ARC-C) (Clark et al.
2018), SciQ (Welbl
2017), and continuation-MMLU (cMMLU) (Hendrycks al.) (2020). We
report accuracy normalized by the byte length of the target string for PIQA, HellaSwag, ARC-E, ARC-C, and SciQ and standard accuracy for Lambada, WinoGrande, and cMMLU. All evaluations are conducted in both 0-shot and 5-shor settings. All measurements were performed on a single
NVIDIA H20 GPU. Detailed results are shown in Table
C
PSEUDOCODE
We provide detailed pseudocode for the recursive architectures discussed the main paper. Algo- rithm [ outlines the implementation of common recursive variants, which rely fixed, heuristic- based state-pase state-passing schemes. In contrast, Algorithm details our proposed MeSH-augmented re- currence, which replaces the rigid logie with dynamic, memory-based system.
Algorithm 1 Recursive Transformers with Common Variants
1: Input: Token embeddings homb, Prelude fpre. Core feore: Coda feods
2: Hyperparameters:: Loop iterations K, Variant type E (base, residual, anchor)
3: # Prelude
4: (0) + Spre (hamb) (Compute initial state for the loop}
5: # 2. Main Recursive Loop
6: for t
do do
7:
-Select supplementary state based on variant
8:
h 0
9:
if Variant type is residual then
h (1)
10:
11:
else if Variant type is anchor then
12:
h (0)
13:
else Variant type is anchor then (t)
14:
hamb
15:
end
16:
(t-1)) Soure (h( haup (Apply core and add supplement]
17:
end for
18: # 3. Final Coda Processing
19: himal & Fouta(h (h(K) (K))
20: return hanal
18

---

Published as a conference paper at ICLR 2026
Table 2: Detailed downstream evaluation results (stacked). For each model variant, performance is shown for both O-shot and 5-shot settings. We report accuracy values for all tasks. The average ac- curacy ("Avg.") ") is computed over the preceding tasks. Dataset abbreviations correspond to: LD- (Lambada OpenAI), LD-S (Lambada Standard), HS (HellaSwag), PQ (PIQA), WG (WinoGrande)) ARC-E (ARC-easy), ARC-C (ARC- C Scig (SciQ), and CMMLU M.MLL--cotinuation)
Structure
Downstream Task Performance
Scheme
Layers
Variant
LD-0
LD-S
HS
PQ
WG
ARC-E
ARC-C
SeiQ
cMMLU
Avg.
Pylhiu 160M
Vanilla
12
0-shor
5-shot
27.11
32.31
23.64
24.22
31.38
31.14
62.46
62.95
50.67
50.59
42.21
39.56
22.53
23.21
78.2
70.3
25.69
25.55
40.54
39.88
0-shot
29.30
20.18
30.85
60.72
49.57
40.03
23.21
70.9
25.30
38.90
5-shor
24.32
19.43
30.76
61.43
51.14
41.75
22.53
76.5
25.75
39.29
Recursive
2+4R2+2
0.chot
30.04
21.11
30.93
60.39
51.14
38.13
23.81
68.3
25.40
38.81
+anchor
s.ohot
26.16
21.23
31.44
61.15
50.75
41.71
23.04
80.2
25.70
40.15
31.32
21,48
31.02
60..6
53.43
39.06
22:27
69.7
25.73
39.41
+mesh
5-shot
26.43
21.00
31.48
60.72
51.93
42.93
23.04
81.9
26.00
40.60
Vanilla
0.chot
41.74
29.65
37.65
64.80
51,93
43.60
25.68
73.1
26.68
43.87
5-shot
35.59
28.92
38.01
67.19
50.36
50.08
25.43
85.2
27.03
45.31
O-shot
39.47
30.43
36.71
63.71
53.59
42.59
24.57
71.8
26.47
43.26
5-shot
35.22
28.26
36.71
64.91
52.88
48.86
25.77
85.7
26.95
45,03
Fy thin 410M
Recursive
4+8R2+4
+anchor
0-shot
5-shot
41.45
36.56
31.24
30.06
36.82
37.13
64.09
65.62
52.80
51.30
43.14
49.24
23.72
25.43
73.6
88.8
26.39
26.94
43,70
45.68
O-shot
41.92
32.33
37.27
64,20
53.83
42.30
25.09
73.5
26.66
44.12
+mesh
5-shot
36.25
31.71
38.00
65.40
51.22
49.37
24.49
87.0
26.93
45.56
0-shot
37.32
25.93
35.43
63.44
50.67
41.79
23.38
72.8
26.68
41.94
5-shot
30.97
25.64
35.95
64.96
51.93
47.73
87.7
26.77
44.01
Recursive (-9015))
3+6R3+3
+residual
O-shot
5-shot
37.80
33.13
27.98
28.45
35.65
35.60
64.64
65.29
50.20
52.01
42.30
47.39
25.00
23.98
68.7
86.3
26.42
26.78
44.24
42.16
+anchor
0-shot
38.33
29.11
36.01
65.45
51,46
43,18
22.78
72.8
26.56
42.85
5.chot
33.92
36.61
65.56
53.04
47.22
23.89
87.8
26.65
44.90
+mesh
41.88
31.87
36.86
65.51
52.17
42,26
24.15
70.9
26.15 15
43.53
shhhtt
37.84
37.08
65.34
87.8
46.04
Vanilla
16
O-shot
46.73
34:02
43.61
66,87
52.01
48.53
26.28
76.6
27.86
46.95
5.
40.60
43.98
52.33
54.46
89.9
49.07
O-
45.76
33.84
41.57
66.87
52.25
45.83
25.77
72.0
27.55
45.72
hase
5-shot
42.54
53.41
89.1
28.33
47.75
Pythia BB
+residual
O-shot
45.70
34.06
41.85
66,44
52.49
47.10
26.02
74.4
27.61
46.19
5.
38.15
32.78
42.50
67.52
52.78
26.11
89.0
28.35
47.85
Recursive
(-31.399
3+5R2+3
+anchor
O-
47.62
35.15
43.06
67.25
53.35
46.51
25.68
75.1
27.97
46.85
5-shot
42.46
34.58
43.24
68.55
52.01
55.22
27.82
90.3
28.42
49.18
O-shot
46.17
34.68
42.62
67.68
53.51
46.80
25.26
75.9
27.99
46.73
+anchor
3-shot
39.92
32.89
43.26
69.15
53.12
55.05
27.22
90.0
28.83
48.83
+mnehh
O-
48.40
36.95
44.36
67.03
52.01
46.93
26.54
77.6
27.91
47.53
5-shot
42.62
34.87
44.68
67.95
55.14
27.22
91.4
49.51
0-shot
$1.08
39.82
47.74
68.83
55.41
50.04
26.11
77.3
29.18
49.30
Vanilla
24
5-shot
46.17
39.69
48.01
64
54.22
59.22
29:27
91.2
29.95
$1.93
base
O-shot
49.56
39.32
46.50
69.37
53.67
49.79
27.
75.9
28.34
48.89
Pytin-11 4B
5-shot
44.95
38.04
46.73
69.55
56.82
90.4
29.52
50.99
rrssduul
0-shot
$1.08
41.10
47.
04
53.12
49.03
26.79
79.6
28.66
49.50
5-shot
47.20
38.81
47.06
69. 26
54,30
56.23
27.65
90.9
29.18
51.18
Recursive
(-3
4+8R2+4
+anchor*
0-shot
51.35
41.28
47.28
67.90 .0
$5.09
49.16
27.47
75.3
28.76
49.29
shot
45.88
40.17
47.72
69.31
53.75
58.25
28.75
93.0
29.61
51.83
0. thot
50.75
40.93
47.65
69.75
53.75
48.40
26.62
78.0
28.62
49.39
+anchor
5-shot
45,99
40.95
47.85
69.37
52.96
56.82
26.79
91.0
29.65
51.27
53.46
41.84
48.58
69.53
54.85
49.75
27.82
80.3
28.89
50.56
+mesh
shot
49.14
42.69
49.21
69.70
54.78
57.79
29.35
92.7
29.76
$2.79
D DISCUSSION: EXPRESSIVE POWER OF MESH AS A GENERAL RECURRENCE
The baseline recurrences described in Section 3.1 employ a fixed, non-adaptive state update rule: the output of the core block, h favee (9), is always supplemented by a predetermined state (e.g., zero, the previous state h(t), or the initial state (1)), We propose that McSH offers a more general and powerful alternative by replacing this rigid addition with a learnable, dynamic state composition mechanism.
19

---

Published as a conference paper at ICLR 2026
Algorithm 2 MeSHI-Augmented Recurrence within Prelude-Recurrent-Cood Structure
1: Input: Token embeddings hemb, Prelude fpr.. Core fcore: Coda feoda
2: Parameters: MeSH buffer M. Routers {R Reedd R
]K-1
3: Hyperparameters: Loop iterations K, Buffer slots B
4: # Initialize MeSH Buffer
5: ( f- zeros (Initialize buffer with zeros)
6: m < hemb (Place embeddings in the first slot}
7: # 2. Prelude
8: b f Spre (hemb) {Compute prelude output}
9: (-1) W (-1)
Routers((=- (h!! {Use transitional routers)
write
10: for b= 0 to B 1 do do
11: m
m
hm
w (-1) write,
(Write prelude output to buffer)
12: end for
13: (0))
b=
B-1
m
w Weald (Synthesize first loop state}
#. Main Recursive Loop
15:
for to K - do do
h + Soore (t) {Core computation]
16:
(t)
w (8)
read +- (*) {Compute step-wise weights}
w
((++1)
(
19:
for b = to
do do
20:
m
(++1)
t- m
(++1)
hm
(t)
w
(t) write,b {Update buffer with distributed write)
21: end for
(t-1)) Eb= m
8-1 (++1)
{Synthesize next state]
read,
end for
Final Coda Processing
25: hgnal + Seoda (ooda(((k) (h(k)) {Use the state after the last read]
26: return hanal
Proposition 2.1. The MeSH recurrence, defined by the compute-write-ren cycle, generalizes the concept of additive state updates (as in residual and anchor variants) by learning dynamically retrieve and combine historical states from memory to form the state for the next iteration.
To reveal the underlying mechanics, we can unroll the MeSH update equations. The state for the next iteration, (t+1), formed by reading from the just-updated memory M(++));
Demonstration.
h
(++1)
m
(++1)
(t) read,
b=0
B
(t)
(()
(c())
wr())
m
write,b)
read,
B
B-1
read,
+
Σ
W
b
w (t) read,
Oh
(6)
b=0
Retrieved Historical Summary
Gating Factor
Let us analyze the two resulting components. The second term is the core's output, h.( scaled by a learned gating factor. The first term is a dynamic retrieval of information from the memory state
m! before the current write operation. Note that this term is distinct from the previous state (t),
which formed using the read weights from the prior step, w Read
(t-1)
20

---

Published as a conference paper at ICLR 2026
The formulation reveals that the next state h(++) is generalized residual update composed of:
1. A refrieved historical summary that dynamically combines states presented in the buffer. The read router learns what historical information is most relevant at this step.
2. A gated output of the current core block, (t) scaled by a learned gating factor.
The dynamic process generalizes the fixed baseline recurrences, as we can conceptually unroll the retrieved historical summary even further as weighted combination of the initial memory state
and all previous core outputs (hm (0) ..... hm 3 that have been written to the buffer. Therefore, the next state h (++1) can be viewed as a comprehensive, dynamic aggregation of all computations performed so far:
h ((++) = & h
Clemb hemb
where all coefficients implicit coupled with write and read weights during previous iterations. The perspective makes the generalization self-evident:
Simulating anchor (hm + hemb): This is achieved by learning a routing scheme where the coefficients in Eq. arre set follows: the weight for the current computation, ag, approaches 1; the weight for the initial embedding, Oemb. approaches I; and all other historical weights, 0gers are driven to zero. MeSH can learn to adopt this specific weighting only when needed, rather than being hard- wore-wiredie to it.
Simulating residual (hm + (+):: ) To approximate this, MeSH needs to reconstruct the previous state, (t), as its historical summary. This is naturally achievable. Since (4) is itself a weighted sum of (hemh; (0) hn=")), the routers step
learn to compute the appropriate coef- ficients (Cremb Cg..... ax-1) to reconstruct closely approximate h(*), More powerfully, MeSH can choose to form a "better" historical summary by up- ue-weighting more relevant past states (e.g.,
hm ((-5)) and down weightiig irrelevant ones (e.g., hm ( thus forming more effective long-range dependencies.
not fixed. They are functions of the current state h(), allowing the model to change its recurrence rule on the fly. It can leam to behave like an Anchor in early steps, transition Residual-like update, and synthesize complex summary from multiple past states for the final output, all within a single forward pass.
Adaptive Combination: The core advantage is that the coefficients
In conclusion, McSH does not just replicate the fixed recurrences; subsumes the underlying prin- ciple of combining past and present information into flexible, learnable framework. It replaces the hard-coded "what to add" (e.g., (()) or (()) with a learned "what to retrieve and combine," offering a substantially more expressive mechanism for managing state in recursive transformers.
E
MORE RESULTS
SCALING TO LARGER RECURSIVE MODELS
To further investigate the scalability of MeSH. we conducted additional experiments on larger Pythia models, specifically at the 2.8B and 6.9B scales. For these models, we employed recursive config- uration of 6+1 0R2+6, 6, which achieves non-embedding parameter reduction to 68. 7,5%%% compared to their non-recursive counterparts. We compared the performance of the base recursive model (rec) against the MeSH-enhanced version (tmesh) in Table 3 The results show that the benefits of MeSH scale effectively to larger models. At the 2.8B and 6.9B scales, MeSH still delivers sub- stantial improvements across the board, significantly lowering perplexity on all four datasets and boosting O-shot and 5-shot average accuracy. This scaling experiment provides strong evidence that McSH is a robust and effective method for enhancing the performance of recursive language models parameter scales.
21

---

Published as a conference paper at ICLR 2026
Table 3:
Performance of MeSH on larger-scale models (Pythia-2.8B and 6.9B). Both models use a ++1002+6 recursive configuration. Since no vanilla counterparts were trained, Aace for the +mesh variant shows the absolute accuracy change relative its base recursive counterpart. Best results within each model block are bolded.
Structure
Perplexity
Task Avg. acc!
Scheme
Layers
Variant
Pile
Wiki
LD-O
LD-S
0-shot
5-shot
Pythia-2.8B
Recursive
6+10R2+6
base
6.90
14.18
8.41
16.87
52.49
54.92
(-31.25%)
+mesh
6.70
13.60
7.30
11.36
54.71
56.85
Recursive (-31.25%)
base
6.29
12.14
6.34
10.29
56.67
59.43
Pythia-6.9B
6+10R2+6
+mesh
6.09
11.64
5.48
8.66
58.83
60.49
Table 4: Ablation on McSH buffer length across multiple datasets for the Pythia-410M recursive model (4+8R2-4) Performance is measured in perplexity (1).
Perplexity.
Scratchpad Slots (k)
Buffer Length
[ (Nocp + 1) + k)
Pile
Wiki
LD-O
LD-S
0
(2+1) + =3
9.1231
21.9348
20.8286
55.9474
12+1)+12
9.1003
21.8439
20.6861
48.9034
2
(2+1) 1-11+2==3
9.0944
21.8351
19.6316
42.5129
(2+1)+33 +3=6
9.1088
21.9120
19.7172
56.5541
E.2 ABLATION STUDY: MESH BUFFER LENGTH
To establish principled heuristic for setting the MeSH buffer length (B), we hypothesize that its capacity should scale with the number of major computational states generated during the recursive process. For model with Noop iterations, this includes the initial state from the prelude network plus the output from each of the Noop core blocks, totaling Niup + 1 essential states. We therefore model the buffer size B (N + 1) + k where is the number of auxiliary "scratchpad" slots available for flexible composition.
We conduct ablation study to find the optimal i using the Pythia-410M model with 4-8R2+4
configuration, where Noop = 2, evaluating the performance across four the evaluation datasets. The add scratchpad slots, peaking k 2. This configuration, corresponding to total buffer length of B = 5, achieves the lowest perplexity on 3, suggesting a point diminishing returns. This indicates a sweet spot where the buffer has dedicated slots for each major computational state, plus two auxiliary slots for managing intermediate representations, without making the routing task overly complex. Based on empirical results, we adopt the general rule B = Nioop + 3 for all MeSH models in our main experiments.
results shown in Table 4 Performance improves
all four datasets. Performance slightly degrades
E.3 ABLATION STUDY: HEURISTIC PASSING SCHEMES
While individual heuristic schemes like +anchor and gresidual improve over the base recur- sive model, natural question arises: can we achieve further gains by combining them, and can the model learn the optimal combination? To investigate this, we conducted an ablation study on Pythia-410M model (4+8R2+4 configuration) exploring both fixed additive combinations and learnable linear combinations of supplementary states (h For the learnable schemes, we de- fine the supplementary state as combination of the base, anchor, anchor* and residual states:
(() a + agh + a-hbemb We test two variants for the coefficients a.:
Static Combination: The coefficients are trainable scalar parameters that fixed after training
(Ng & Wang 2024).
Dynamic Combination: The coefficients are dynamically computed at each iteration based on the previous state (t-1).
22

---

Published as a conference paper at ICLR 2026
The results, summarized in Table 5 reveal the brittleness of heuristic design. Simply adding degrades performance, yielding ity (9.24) than using +anchor alone (9.19). This confirms that naive "more is better" approach is not a reliable strategy. While a carefully hand-picked combination paancorrtncchree the best result among all explicit schemes (9.17 17 PPL.), this requires manual tuning. The learnable statie ic and dynamic combinations effectively avoid the worst-case performance degradation but fail to match the best-performing heuristic.
all states (+resi waw.uatttachhotaacchhaeee
yielding a higher perplex-
achieves
Table 5: Ablation on combinations of supplementary context schemes for Pythia-410M (4+8R2+4). We report evaluation loss and perplexity on the Pile dataset. While hand-picked combination (tanchor aacchor)) works best among heuristics, McSH surpasses all explicit schemes.
Scheme
Loss :
PPL L
vanilla
2.2047
9.0675
recursive-base
2.2358
9.3542
Single Heuristic Baselines
+anchor
2.2178
9.1867
Fixed Additive Combinations
+residual+anchor
2.2251
9,2541
#residual+-nchoo-annhho
2.2237
9.2415
+anchor-anchor*
2.21559
9.1694
Learnable Combinations
+static comb.
2.2163
9.1731
+dynamic comb.
2.2176
9.1851
+mesh (ours)
2.2077
9.0944
This suggests that while explicit, learnable weighting can provide a "safe" baseline by ignoring detri- mental combinations, lacks the expressive capacity discover optimal synergistic interactions. In sharp contrast, our +mesh model (9.09 PPL) significantly outperforms all heuristic- based ap- proaches. Instead of being constrained to an explicit, oow-dimesiional linear combination of prede- fined states, McSH learns a complex, non-lineat function for retrieving and composing information from its memory buffer. This allows to discover implicit, high-dimensional combinations, effec- tively breaking through the performance ceiling imposed by simpler, explicit state-passing schemes.
E.4 COMPLEXITY ANALYSIS
Table 6: Parameter counts for Pythia-1.4HP .44B variants. Percentages show the change relative to the Manilla baseline for total and non-embedding parameters.
Hon-Embedding Params
Router Weights
Model
Variant
Config
Total Params
vanilla
24
1,423,036,416
10208.602,624
170,2,4
805,736,448
Pythia-1.4B
recursive
4+8R2+4
(-28.3109)
(-33.333%)
4+8R2+4
1,020,231,710
805,797,911
61,470 (+0.005%)
+mesh
(B 5)
(-28.3069)
(-33.328%)
Parameter Overhead. MeSH introduces set of lightweight, step-wise routers for its read and
write operations. The total number of additional parameters determined by (Nisop + 1) x Dridden X B x 2, where Nooop is the number of loop iterations, Driäden is the hidden size, B is the buffer length, and the factor of accounts for both read and write routers. This overhead is negligible compared to the significant parameter savings achieved through recursion. As detailed in Table 6 for our Pythia-1.44 1.4B model in a 4+ BR2+4 4 configuration, the recursive structure reduces the non- embedding parameters by 33.33% compared its vanilla counterpart. The MeSH routers add
23

---

Published as a conference paper at ICLR 2026
mere 61,470 parameters (0.005% relative to the non-embedding part), which shows that MeSH achieves its substantial performance gains with virtually no cost to parameter efficiency, making an architecturally lightweight yet powerful enhancement.
Table 7: FLOPs overhead analysis for Pythia-1.4B recursive variants. The total GFL.OPs
measured for single forward pass with an input of size [1. 4096]. The overhead is calculated relative to the base recursive model.
Total GFLOPs
Extra GFLOPs
Model
Variant
Config
(1e9)
(Ic9)
recursive (base)
4+8R2+4
5373.792
0,0168
Pythia-1.4B
+residual
4+8R2+4
5373.809
(+0.000312%)
0.0168
+anchor
4+8R2+4
5373.809
(+0.0003 (20)0
4+8R2+4
0.7551
+mesh
5374.547
(B 5)
(+0.014051
Computational Overhead. To quantify the computational cost, we measured the FLOPs for a single forward pass using the fvcore library our Pythia-1.4B model with a 4+8R2+4 4 recursive configuration and a standard input tensor of shape [1, 4096]. We report the computational overhead of the tresidual, +anchor and +mesh variants relative the base recursive model in Ta- ble 7 The analysis reveals that the overhead from MeSH is negligible, adding only approximately +0.014% for single forward pass. This efficiency theoretically grounded. Let bs be the batch size. Sten the sequence length, Dhidden the hidden dimension, Nooop the number of loops, and B the buffer length. The total extra FLOPs for McSH are given by:
AFLOPSmesh (Ntoop + 1) (6. (6-bs- Sten Dridden B)
For configuration (Ntoop 2, bs 1, Sten 4096, Dridden 2048, B 5), this formula yields x 0.755 GFL.Os,, matching the empirically measured overhead. The analysis confirms that the significant performance and stability gains provided by McSH are achieved with minimal and practically insignificant increase in computational requirements, highlighting its architectural efficiency.
E.5 DETAILED TRAINING DYNAMICS ON DOWNSTREAM TASKS
To provide a more granular view of the training dynamics presented in Section 4.3 Figure (isure/sshows shows the performance of the 1. I.B-parraeterr models on 9 individual downstream tasks and their average accuracy, evaluated various checkpoints throughout the pre-training process.
E.6 APPLYING MESH TO CON-RECURSIVE ARCIITECCUUESS
In our main experiments, the recurs ve+mesh model for Pythia- 1.4B surpasses its larger Vanilla counterpart, even with 33.3% fewer parameters. The result suggests that the MeSH mechanism might offer architectural benefits beyond the the recursive setting. We hypothesize that the performance bottleneck of parameter sharing were removed, the benefits of MeSH could be even more pronounced. We conduct an experiment applying McSH-like structure to a standard, non-recursive Vanilla transformer. We conceptually partition the 24 layers of the Pythia-1.4B Vanilla model into blocks that mirror our 4+8R2+4 recursive design: a 4-layer prelude, two distinct 8-layer core blocks (core_1 and core_2), and 4-layer coda. Crucially, unlike in the re- cursive setup, core_l and core_2 do not share weights. The MeSH mechanism, with its memory buffer and routers, is then inserted at the boundaries between these conceptual blocks to manage information flow. Results are shown Table g The vanillatmesh model achieves a lower perplexity (7.26) than the standard Vanilla baseline (7.44), confirming that MeSH provides direct performance uplift even without the constraint of parameter sharing. This finding provides compelling explanation for the strong performance of our main recurs ve tmesh model: the
24

---

Published as a conference paper at ICLR 2026
Lambada OpenAl
Lambada Standard
O-shot
5-shot
O-shot
5-shot
54
51
48
4.5
42
333
30
-3
33
27
30
20k 40K 60K BOK 100K120K
20k 40k 60K 100K120 Training Stape
20k 40k 60K BOk 100K120K
20k 40k 60x BOK 100K120K Taining Steps
Trairing Stops
Taining Stops
HellaSwag
PIQA
O-shot
5-shot
Q-shot
5-shot
46
444
42
-40
66
38
65
36
64
63
20k 40k 60k 80k 100K120k
0
20k 40k 6ök BOK 100k120k Training Stope
0
20k 40k 60k 80% 100K120K
20k 40k 60c BOK 100c1 20k
Training Shace
Thiiing Steps
Trshing Steps
ARC-Challenge
ARC-Esay
0-shot
5-shot
O-shot
5-shot
60
30
228
54
27
26
48
25
24
42
23
Trairing
WinoGrande
SciQ
0-shot
5-shot
O-shot
5-shot
56
90
55
554
84
633
01
78
51
72
50
69-
100K120k
20k 40k sök 8ök
100k120k
20k 40k 60k B0K 100K120K
20k
60c BOC 100KI
MMLU-Continuation
Avg. Accuracy
O-shot
5-shot
O-shot
5-shot
30
52
29
50
Acturacy 26
48
2.8
46
44
26
20k 40k 60k 80K 100K120K Trariig Shepe
20k 4ök 6ök BöK 100K120k Thinnin Shepe
20k 40k 60k 8ök 100k120k Traiing Steps
20k 40k, 66x 80k 100x120k Traiinng Stepe
vanilla
recursive (base)
recursive (+anchor)
recursive (+mesh)
recursive (+residual)
recursive (+anchor*)
Figure 9: Detailed Training Dynamics of 1.4B Recursive Variants on Downstream Tasks. Each panel displays the 0-shot and 5-shot accuracy for one of the individual downstream tasks or their overall average ("Avg- Accuracy"), evaluated at different checkpoints throughout the 120,818-step pre-training process.
25

---

Published as a conference paper at ICLR 2026
architectural benefits of MeSH potent enough to not only compensate for the performance loss typically incurred by weight sharing but to exceed the original baseline.
Table Performance comparison of applying MeSH to recursive and non-recursive (Vanilla) back- bones the Pythia-I.4B scale. All metries are evaluated the Pile dataset.
Non-Emb Params
Variant
Config
Loss
PPL
vanilla
24 layers
100%
2.0070
7.4406
vanilla+mesh
4+84844
100%
1.9818
7.2559
recursive (hase)
448R2+4
66.7%
2.0317
7.6267
recursive+mesh
4+8R2+4
66.7%
1.9996
7.3865
While McSH framework considered in recursive transformers, the result indicates that its core principle of explicit, routed state management has broader applicability. Exploring MeSH general architectural primitive for enhancing deep. non-recursive transformers is promising direction for our future research.
E.7 APPLYING MESH TO MAEEBASED ARCHITECTURES
Table Performance of MeSH on a MoE-based backbone. Results for -aaaaaaeeeeeer OL- MoE model with 512M activated parameters. The recursive variants 4 ++822+4 +4 configuration. PPL. reported on Pile, Wikitext (Wiki), Lambada-OpenAl (LD-O) and Lambada-Standant (LD-
S). Task Avg. accf reported for 0-shot and 5-shot settings.
Structure
Perplexity
Task Avg, accf (%)
Scheme
Layers
Variant
Pile
Wiki
LD-O
LD-S
0-shot
5-shot
Vanilla
24
7.31
14.93
11.29
22.29
49.83
51.87
Recursive
base
7.60
15.97
11.99
22.52
48.96
50.83
4+8R2+4
(-33.3%)
+mesh
7.46
15.72
11.61
22.40
49.51
51.53
Dai et al. 2024), ) as a common technique in large language models (Liu et al. [024 Llama Team 2025 Yang al. 2025). ) also leverage dynamic routing. However, the objective of this routing differs significantly: while MoE aims for computational sparsity by selecting subset of experts within feed-forward layer, McSH is specif- ically designed to manage information flow across iterations recursive architectures. This makes them orthogonal approaches that be complementary in principle. To empirically validate that MeSH remains effective on a MoE-based recursive backbone, we adapted the open-source OLMoE architecture (Muennighoff et al.) 2024)). Our non-recursive vanilla baseline is an OLMoE model with 2.6B total and 5 12M activated parameters, configured with hidden size of 1024 and 24 layers. For the recursive variants, we adapted this non-recursive backbone into 4+8R2+4 configuration, aligning with our main experimental setup. The MoE-specific settings, such as using dropless token- choice routing select consistent. All models were pre-trained on the Pile dataset with learning rate of 3e-4. As shown Table 9 the results confirm that while naive recursion (base) leads to performance degradation, the +mesh variant successfully mitigates this drop. It recovers performance to level comparable to the vanilla counterpart under matched compute, demonstrating McSH's effectiveness on MoE-based backbones.
Lixture-of-Experr (MoE) models (Fedus et al. 2022
of 16 experts (with an expert hidden dimension of 2048), were kept
E.8 SUPPLEMENTAL ANALYSIS: SHARED SUBSPACE COLLAPSE
To provide more direct and intuitive visualization of the loop representational collapse identified in our main analysis (Figure ), we conduct a supplemental subspace analysis based on Singular Value Decomposition (SVD). Our central hypothesis is that the pathology term information overload that leads to representational conffict, forcing the naive recursive model to make a dire trade-off. To ensure stability, the model sacrifices its role as "Information Processor" and degenerates into
26

---

Published as a conference paper at ICLR 2026
simple "Information Preserver", forcing all iterative states to converge simple, low-dimensional
"common ground" representation.
(a) Naive Recursive Model (base)
(b) Mesh-enhanced Model (+mesh)
Cumulative Explained Variance (36)
100%
100%
80%
801
b
20%
20%
h
10
50
10
50
Principal Component Index (SVD base)
Principal Component Index (SVD base)
Figure 10: Subspace Analysis. Visualization results of how information is structured within the recursive loop for (a) naive recursive model and (b) our MeSH-enhanced model. Both models based on the Pythia-410M backbone with 3+ 6R3+3 recursive configuration. We first define a "Shared Subspace" by performing SVD on the concatenated hidden states of all loop iterations (h("), h()) h2),, (()) and using the top singular directions as the shared basis. The plots then show the cumulative variance of each individual state when projected onto this shared subspace. In (a), the states of the naive model dramatically collapse onto the shared subspace, where variance is almost entirely explained by the first few singular directions, indicating severe structural redundancy and failure to generate new information. In (b), the states of the McSH model maintain distinct, high- dimensional structures, where variance is distributed across many components, demonstrating that each iteration preserves unique information and avoids collapse. Lines and shaded areas represent the mean and standard deviation across 500 samples.
As shown in Figure TO we construct the shared subspace that captures the most dominant and consistent feature directions across all loop iterations. We then project each individual state back onto this shared basis and measure how much of its variance is explained. The results provide a stark contrast. In the naive recursive model (Figure 10p).. the representations of all loop states (h(1) through h()) collapse onto the shared subspace. Their variance curves are extremely steep and nearly identical, indicating that their entire structure is almost completely contained within the first few dimensions of the shared subspace. This provides compelling evidence of profound structural redundancy. The model is not generating new, diverse information at each step; instead, it is trapped in a low-dimensional attractor, merely preserving a static set of features. This is the hallmark of a system that has abandoned its "Processor" role. Conversely, the MeSH-enhanced model (Fig- ure Iop) completely averts this pathology. Each state exhibits a distinct and gracefully rising curve, signifying that each state preserves significant amount of unique, high-dimensional information not captured by the others. By offloading the duty information preservation to its external mem- ory, McSH liberates the hidden states to engage in meaningful, dighedimensional computation each step. The result provides visual evidence that MeSH resolves the representational bottleneck caused by the representational conffict, enabling progressive refinement of information across the recursive loop.
F CASE STUDY: UNPACKING THE INTERNAL DYNAMICS OF MESH
To provide more granular, qualitative view of MeSH's s internal mechanisms, we provide a case study on single input sequence. Figure 11 unpacks the model's dynamics, showing how MeSH organizes information flow across recursive iterations. Panel (a) visualizes the router weights, reveal- ing how the model learns to route information with both iteration-level and token-level specificity. The 'Write" router learns a clear policy: on average, it directs the output of each iteration to distinct memory slot (e.g., Iteration targets Slot Iteration 2 targets Slot 4), where the memory organizes information from different computational stage. Furthermore, the stark difference between the aver- age weights and those for the specific token highlights that routing is a highly dynamic, token-wins wise
27

---

Published as a conference paper at ICLR 2026
(a) Dynamic Router Weights
(b) Causal State Composition
Average ippoloas
5.00
1.09
3.2
0.01
0.00
0.16
1110
1.00
1:41
KCB
-00
Stat
Write
Road
Token "ssu" (idx
55
137
Reeaton
Ruub
11
Ng
N2
Average
Token "gu" (idx
0.00
1.2
-20
Memory Stot
Memory Skat
Visualization of Internal Dynamics. The analysis is performed on MeSH-enhanced Pythia-410M model with a 3+6R3+3 configuration. Case study: "The failure of his handcrafted novel to become bestseller was the ultimate wake-up call, forcing him second-guess every- thing." The visualizations show: (a) Write (red) and Read (green) router weights, shown for the sequence average and for an intermediate token "gu" (idx 27). (b) State composition analysis. The coefficients shown quantify the causal contribution of each source state (hemh. hm to the formation of a subsequent input state (h()). These are derived from the router weights and reflect the unrolled
Figure 11:
recurrence relation as Eq. 7 Here, hm (-1) is the prelude output, and ha is the core output loop iteration I.
decision. The state composition analysis in panel (b) shows how information is integrated across iterations. For instance, the state h()), which serves as the input to the final loop, is composed of information from the prelude output h as well as the core outputs from previous loops and h This illustrates a sophisticated recombination strategy, where the model maintains strong con- nection to the initial "anchor-like" state while flexibly incorporating information from intermediate steps, thereby effectively managing the information flow throughout the recursive process.
G LIMITATIONS AND FUTURE WORK
While this work establishes McSH as promising architectural principle for recursive transformers, we recognize several limitations that open up avenues for future research. First, our experiments have validated the effectiveness of McSH models up to the Pythia-6.9B scale, trained the deduplicated Pile dataset. It remains unclear whether the parameter-efficiengg gains persist at larger scales and under different training regimes. A natural and important direction for future work is to apply and evaluate the MeSH architecture on state-of-the-art foundation models at much larger scales. Second, although our ablation study reveals that MeSH can also benefit non-recursive trans- formers. a comprehensive investigation beyond recursive backbones falls outside the scope of this paper. Exploring McSH as a general-purpose architectural primitive for improving information flow remains a promising direction for our future work.
H STATEMENT ON LARGE LANGUAGE MODELS USAGE
We used large language models only for language editing to improve the clarity and readability of the paper. The LLMs did not contribute the research ideation, methodology and the experimental implementation. All substantive content is original and was verified by the authors.
28


Reviews:
[{"review_id": "QZzI3mN4uC", "reviewer": "Reviewer_4cbP", "summary": "This paper attempts to improve upon recently introduced recursive transformer models.  Such networks replace a long sequence of layers in transformer models, with a shorter sequence.  The paper claims recursive networks fail due to not encoding where they are in the looping sequence and sets out to fix this problem.  The paper does some analysis and suggests that the failure is due to two factors it calls \"undifferentiated computation\" and \"information overload\", respectively.  The solution, MeSH, uses what it calls \"lightweight routers\" to specialize computations across iterations.", "questions": "The description of \"+residual\" on lines 153-156 does not seem to capture the LoRA technique described in \"Bae et al., Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA\" nor does the \"+anchor\" approach described on lines 157-160.  Conceptually, how does the specialization used in \"+mesh\" differ from the use of LoRA to specialize weights across iterations in that work?\n\nI'm confused about the premise of this paper.  Section 2 heading \"Why Naive Recursion Fails\" presumes that recursive transformers are doing poorly but a cursory review of related works on these networks show the opposite -- that they work very well indeed.\n\nWhat is meant by directed computation? \n\nWhat is representational stagnation and why is CKA similarity a good way to measure it?\n\nIt is unclear to me what Figure 1a is plotting.  What is meant by 'computational effort'?  What are prelude, 1st f_core, etc. of the x-axis labels referring to?   Why is the Frobenius norm of hidden states a meaningful measure of effort?", "limitations": "", "rating": null, "confidence": 2, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "Brings attention to recursive transformers, which look like a very promising direction for improving efficiency of LLMs.", "weaknesses": "Very poorly written: For example, the paper frequently introduces terminology without defining it.  Similarly it employs metrics without clear explanations of why they are the right thing to measure or even what they are attempting to measure.   \n\nUnclear novelty:  The paper does not sufficiently differentiate the proposed solution from similar strategies in the works it cites.   For example, on lines 086-087 the paper says \"the block lacks any explicit information about its progress within the iterative sequence\" but the related work explicitly tackles this problem and the reader must actually read the related works to discover this discrepancy.\n\nThe abstract and introduction say that recursive models \"lag behind\" non-recursive models, but it is unclear (a) what \"lag behind\" means and (b) which specific recursive models (e.g., which papers) this assessment applies to.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qCguX575bS", "reviewer": "Reviewer_nSZz", "summary": "In the present paper, the authors introduce MeSH, a recursive Transformer backbone which utilizes a dynamic routing mechanism with read-, and write-routers which manage an external memory buffer. The proposed backbone is deeply motivated through failure analysis on existing recursive backbones. Performance is evaluated across a wide scale of trained recurrent models for perplexity across 4 datasets, and average accuracy on 10 downstream tasks.", "questions": "* What is the author's intuition on the behavior of the routing mechanism through training, but equally as important, at inference time. The routing mechanism evokes analogies to mixture-of-expert model routing mechanisms, which pose a number of intricate concerns when seeking to deploy these models. How do you intuit these deployment concerns to play out with a MeSH-based model?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper has an expansive number of strengths, beginning with its analysis of failure modes of current recurrent mechanisms as the foundation the paper is built from. This clarity persists throughout the paper. Most notable are the following strengths of the paper:\n* Clarity of the problem formulation, starting with the problem analysis and the subsequent presentation of the core MeSH architecture block\n* Highly expansive evaluation with a very wide set of models trained from scratch to enable the performance evaluation of the architecture at varying sizes. In addition, the authors perform evaluation on a good number of downstream evaluations. A vast effort not to be underestimated.\n* The provided codebase permits for reproducibility, but furthermore the easy ability to build on the results\n* The authors go to great lengths to embed this work into the existing literature with references going back to the RNN era, where the first memory augmentations appeared. \n\nWhile the paper leaves a number of questions open, see the weaknesses for example, its intellectual clarity, the expansive evaluation, reproducible experiments, and good embedding into existing literature make this a very good paper which should stay significant as time progresses.", "weaknesses": "The paper as is suffers from two core weaknesses in the eye of the reviewer:\n* While the routing mechanism is briefly introduced, there exists no ablation or design exploration amongst routing mechanisms in the paper. Further design comparisons & ablations would aid greatly here to understand inherent tradeoffs better, especially considering the orthogonality of said routing mechanism to similar mechanisms in mixture-of-expert models.\n* While the reviewer appreciates the great cost of the many experiments and trainings run by the authors, the paper would yet benefit from comparisons to other contemporary recursive models. A simple fix here could potentially be to add evaluations of recurrent Gemma, and potentially other publicly available open-weight recurrent models, to the evaluation to provide the much needed ability for comparison.\n* Comparison to the historic memory-augmented neural network of Rae et al. [1] is missing. While these are RNNs, they nevertheless display many of the same traits of the models presented here.\n\nReferences:\n1. Rae, J.W., Hunt, J.J., Danihelka, I., Harley, T., Senior, A.W., Wayne, G., Graves, A., & Lillicrap, T.P. (2016). Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes. ArXiv, abs/1610.09027.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mXcIOJq5TB", "reviewer": "Reviewer_hA3W", "summary": "This manuscript describes MeSH (Memory as State Highway), a novel approach to enhancing the representational power of recursive transformers, i.e., models with repeated Transformer layer blocks for better parameter efficiency. MeSH is based on computing and updating a bank of memory buffers during the looping over the recursive blocks. The results show that augmenting the recursive transformer alleviates the undifferentiated computation seen in unaugmented (base) recursive transformers (as demonstrated by the increase in the changes in the magnitude of the state update and reduced CKA metric in non-initial iterations of the recursive block). Experiments on the Pythia suite show that the MeSH augmentation significantly improves the learning and the eval performance (lower perplexity and loss and higher task accuracy) compared to both the vanilla transformer baseline and the base recursive transformer baseline in the majority of the cases. Interestingly, at the larger model size tested (e.g., 1.4B), the MeSH architecture is shown to outperform the vanilla transformer in the Pythia benchmarks, despite having roughly 30%-50% fewer parameters.", "questions": "Q1. The use of CKA (Centered Kernel Alignment) as a metric for showing the similarity of computation over iterations of the recursive transformer needs to be better motivated.\nQ2. Table 1 and/or Section E.3 should additionally report the computational overhead (FLOPs) caused by the augmentation types (residual, anchor, and MeSH). \nQ3. The paper mentions the open source code for the implementation of MeSH, but the files in the anonymized repo appear to be unavailable.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "S1. Well motivated study on improving the computational efficiency and representational richness of recursive transformers. Includes thorough literature review that sets the stage nicely and makes the paper easy to follow for readers.\nS2. Includes a set of baselines including the vanilla transformer, base recursive transformer, and recursive transformer with anchor and residual augmentation. This makes the MeSH results meaningful and strong.\nS3. Analyzes the model's internal representation and how it is altered by the MeSH augmentation, which forms a nice mechanistic interpretation of the cause of MeSH's benefits.\nS4. The finding of superior performance of the smaller, MeSH-augmented recursive transformer over that of the vanilla transformer indicates a future direction of investigating the performance of MeSH for larger model sizes.", "weaknesses": "W1. As the authors pointed out, the experiment results are limited to relative small transformers (up to 1.4B). It is not entirely clear whether the MeSH augmentation of recursive transformers can extend to larger, SOTA LLMs. The Pythia suite actually supports up to the model size of 12B, the authors did not explain why they stopped at 1.4B.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nSCBHniNwc", "reviewer": "Reviewer_LF4x", "summary": "This paper investigates performance bottleneks in recursive transformers used in LLMs, and identifies two issues: \"undifferentiated computing\" where the recursive computational function cannot specialize its function to different iterations, and \"information overload\" where the single hidden state needs to perform various roles. To address these issues, the paper prooses a new network design, Memory-as-State-Highways (MeSH). MeSH improves over common recurrent schemes that re-include long-term context (e.g. residual connections, or re-adding the original input) by introducing a multi-slot memory module. For this a simple Read and Write routing blocks are introduced, simple linear layers+softmax that map each D-dimensional hidden state element to one of B memory slots, such that the network can differentiate different long/short term memory needs inside the hidden state. Experiments on four Pythia LLM benchmark models show that MeSH outperforms common recursive baseline schemes, and sometimes even non-recursive baselines with significantly more parameters. The paper also presents analysis of the state per iteration, which supports that MeSH is better able to distribute its information updates over the iterations.", "questions": "* How does the work relate to other works that have integrated memory in recurrent networks, and transformer networks specifically? What are the main differences and similarities to the proposed approach, and what does that mean for the contributions?\n* See my issues on the argumentation in Weaknesses, can the authors comment on these?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The proposed MeSH architecture is simple and intuitive, both conceptually and in terms of implementation.\n* The proposed MeSH architecture is tested on four models from the Pythia LLM benchmark, and is shown to outperform non-memory baseline approaches, and sometimes even non-recursive vanilla networks of similar depth but therefore more parameters.\n* The paper's story is very well-structured. The paper first presents an analysis of using various metrics and visualizations of a problem in regular recursive transformers. This analysis is used to motivate the proposed novel MeSH design, and finally the experiments refer back to the analysis and show how the new design improves the analysed network behavior. This makes the storyline and core argument of the paper very clear (although I find some arguments still doubtful, see below)\n* There are various supporting experiments, such as performance as a function of # of parameters, and learning curves. Even more experiments can be found in the Appendix. Overall, experimental validation appears thorough.", "weaknesses": "## [W1] Relevant related work missing\n\nThe introduction of explicit memory modules in recurrent networks is not novel per se, but the paper's related work review, comparison, and discussion does not seem to make any connections to prior work in this area. One obvious parallel older than transformers are LSTMs, which also provides a memory mechanism for recursive networks to selectively update the latent state and differentiate short-term and long-term information. Various more recent works have also proposed integrated memory banks into transformer architectures, e.g. [Bulatov,NeurIPS'22][Wang,NeurIPS'23], among others. Perhaps the review in [Omidi,25] could help. In any case, there is no mention of or comparison to such works, which makes it difficult to assess this paper's contributions.\n\nI see this as the main weakness of the submission.\n\n## [W2] The paper's arguments are not clear at various points (or perhaps I do understand and simply disagree):\n* line 108: \"The information overload on the hidden state forces the model to find a low-dimensional 'common ground' representation that can safely survive multiple transformations, which directly causes loop representational collapse.\"\n\t* -> while I do not doubt the representational collapse, based on the presented diagnostics in the paper, the paper presents as cause the \"information overload\", but I don't find this a logical explanation by itself. Intuitively a low-dimensional subspace would indicate that the feature distribution is simpler than what a model can express. So, if the network has to manage \"multiple, often conflicting, roles simultaneously\" (line 103), wouldn't that mean we should expect the network to require a higher-dimensional subspace, rather than less? I don't have hard evidence for this either, but my point is that it doesn't seem so self-evident to me what causes representational collapse as is posited in the paper.\n* line 252: \"Since each router (R(t)_write,R(t)_read) has its own unique set of learnable parameters for each iteration t, the model is no longer forced to apply a single, universal transformation.\"\n\t* But also in MeSH, the f_core function remains fixed. Yes, MeSH allows to swap input and output, but the core operation is still not aware of the processing step (the \"undifferentiated computation\" of line 092). The proposed memory block operations themselves don't appear to impute information that could inform the core compute unit of the iteration, and it is unexplained how adding memory enables the model to assign specalized roles to each iteration. Of course, the model could learn to encode some step counter in the input/output statespace to inform it of its processing stage, but this was already true for the memory-less baseline.\n\n## [W3] Missing analysis of how proposed memory banks operate in practice\n* The paper lacks experiments focused on inspecting the behavior of the proposed new module, which are critical to assess if the method works because of the claimed reasons, and to validate the various claims and hypotheses made throughout the paper:\n\t* I would have liked to see some investigation if the memory model works \"as advertised\": can we see that the different iterations indeed assign different weights in the linear layer to different memory banks, and that memory written in earlier iterations is used in later iterations? Some sort of analysis on how the memory banks are read/write in real-world scenarios could help.\n\t* The effect of the number of memory banks B also belongs in the main paper, in my view. I did find the ablation study on Pythia-410M in Appendix E.1, but I was somewhat surprised a general rule of B = N_loop + 3 was propose on this single result. Investigating the effect of B on all four datasets would have made the analysis more thorough and convincing.\n\n## Minor\n* Line 133, symbols $L$ and $D$ are not explained.\n* It is not completely clear from the text what the \"Vanilla\" baseline refers to in Section 4.2. Only later in the paper did I understand that it is a non-recursive transformer, which has different parameters for each iteration.\n\n\n## References\n[Bulatov,NeurIPS'22]: Bulatov, Aydar, Yury Kuratov, and Mikhail Burtsev. \"Recurrent memory transformer.\" Advances in Neural Information Processing Systems 35 (2022): 11079-11091.\n\n[Wang,NeurIPS'23]: Wang, Weizhi, et al. \"Augmenting language models with long-term memory.\" Advances in Neural Information Processing Systems 36 (2023): 74530-74543.\n\n[Omidi,25]: Omidi, Parsa, et al. \"Memory-augmented transformers: A systematic review from neuroscience principles to enhanced model architectures.\" arXiv preprint arXiv:2508.10824 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)