ChainGPT: Dual-Reasoning Model with Recurrent Depth and Multi-Rank State Updates
Authors: Yunao Zheng
Xiaojie Wang
Lei Ren
Chen Wei
Venue: ICLR 2026 Poster (2026)
Keywords: Latent Reasoning; Recurrent Depth; RWKV-Product; State-Guided Sparse Attention
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models, constrained by the fixed-depth Transformer architecture, struggle to solve complex reasoning tasks in an end-to-end manner. Existing approaches, such as Chain of Thought, improve reasoning depth to some extent but rely heavily on natural language generation, with computational costs increasing rapidly as the length of the generated sequence grows. To address these limitations, we propose ChainGPT, a dual-reasoning model that shifts reasoning into latent computational space. Within each layer, ChainGPT employs multi-substep state updates combined with state-guided sparse attention, enabling deep local computation and efficient long-range modeling without quadratic costs. Across layers, recurrent depth approach iteratively refine latent states, supported by adaptive training and stopping strategies that balance reasoning depth against computational budget. Theoretically, we show that ChainGPT can, in principle, simulate general computation, and empirically it delivers consistent improvements over comparable models, including on reasoning tasks that remain challenging for existing systems. By unifying efficiency and reasoning ability, ChainGPT provides a principled foundation for next-generation language models.

Published as a conference paper at ICLR 2026
CHAINGPT: DUAL-REASONING MODEL WITH RE- CURRENT DEPTH AND MULTI-RANK STATE UPDATES
Yunao Zheng Xiaojie Wang"
Beijing University of Posts and Telecommunications (BUPT), Beijing, China
Lei Ren Wei Chen
Li Auto Inc., Beijing, China
ABSTRACT
Large language models, constrained by the fixed-depth Transformer architecture, struggle solve complex reasoning tasks in an end-to-end manner. Existing ap- proaches, such as Chain of Thought, improve reasoning depth to some extent but rely heavily on natural language generation, with computational costs increasing rapidly the length of the generated sequence grows. To address these limita- tions, we propose ChainGPT, a dual-reasoning model that shifts reasoning into latent computational space. Within each layer, ChainGPT employs multi-substep state updates combined with sate-ggided sparse attention, enabling deep local computation and efficient long-range modeling without quadratic costs. Across layers, recurrent depth approach iteratively refine latent states, supported by adap- tive training and stopping strategies that balance reasoning depth against compu- tational budget. Theoretically, we show that ChainGPT can, in principle, simulate general computation, and empirically it delivers consistent improvements over comparable models, including reasoning tasks that remain challenging for ex- isting systems. By unifying efficiency and reasoning ability, ChainGPT provides principled foundation for next- gexe-eenerttion language models.
INTRODUCTION
Large Language Models (LLMs) have demonstrated strong capabilities in natural language inter- action, but their architectures remain fundamentally constrained in expressive power. From the perspective of computational complexity theory, the fixed depth of standard Transformers (Vaswani et al. 2017) places them within circuit classes such as AC or TC (Arora & Barak 2009 Hahn 2020 which limits their ability to reliably perform reasoning tasks that scale with problem size, including multi-step planning, symbolic manipulation, and combinatorial problem-solving requir- ing deep recursion. In other words, under finite precision, LLMs are not Turing complete and thus cannot solve tasks involving deep planning in purely end-to-end manner.
To address these limitations, Chain of Thought (CoT) (Wei al. (022) has emerged as a mainstream approach, guiding models toward solutions through intermediate natural language steps. However, since reasoning is represented as discrete token sequences, is inherently restricted by the expres- siveness of language, making it difficult to capture nonlinear or parallel reasoning modes (Yao et al.) 2023]. Furthermore, as task complexity increases, the length of CoT outputs grows rapidly, intro- ducing significant computational overhead.
Researchers have explored two major directions to address these challenges. The first focuses on improving reasoning strategies, such as Tree of Thoughts (ToT) (Yao al. 2023), which employs tree search to explore more complex reasoning paths. The second focuses on improving underlying architectural efficiency. For example, RWKV-7 (Peng et al. 2025) leverages faster Recurrent Neural
Network (RNN) designs to reduce computational cost, while Jamba (Lieber et al. (024] and Samba (Ren al. 2024) adopt hybrid architectures balance reasoning quality with efficiency.
"Corresponding author: xjwang@bupt.. .du..c

---

Published as a conference paper at ICLR 2026
Despite these advances, fundamental limitations remain. On the hand, improved reasoning strategies can capture richer semantic structures but still rely on natural language, and their compu- tational cost grows exponentially with the number of branches or samples. On the other hand, RNNs are more efficient but their fnite-dimensional state matrices cannot faithfully retain detailed context, which hinders performance on tasks requiring precise memory (Jelassi al., 2024). Current hybrid architectures are mostly sequential combinations that lack effective coordination: preserving global attention faces scalability bottlenecks, while restricting attention to local windows fails to resolve
the problem of limited state matrix capacity.
These observations indicate that an effective solution must address two fundamental challenges at once: enabling deep reasoning beyond simple token generation, and sustaining computational efficiency while preserving long-range dependencies. To meet these challenges, we introduce ChainGPT,, novel architecture that shifts reasoning from explicit token-level processes into a latent computational space. By integrating multi-substep computation within layers and iterative refine- ment across layers, ChainGPT achieves both reasoning depth and efficiency, maintaining near-linear complexity.
Intra-layer reasoning is realized through the Chain-Block, which integrates state layer and an attention layer. In the state layer, we employ RWKV-Product, a LoRA-based multi-substep update mechanism that enables the model to perform multiple state updates within single forward pass. Since each substep is low-rank, the computation remains highly efficient, while their cumulative effect can approximate high-rank transformations, allowing this layer significantly enhance the expressiveness of state updates without incurring substantial computational overhead. To ensure the model can retain comprehensive contextual information, the attention layer introduces State-Guided Sparse Attention (SGSA). Unlike other RNNs that compress the entire history into a finite state, SGSA only attends to recent tokens within sliding windows and a set of periodic anchors. As the sequence grows, contextual information is progressively written into the state, enabling the state space to scale naturally with length and distributing memory across block-level fragments. Through this approach, the model maintains near-linear complexity while still guaranteeing effective recall of long-range information.
Inter-layer reasoning employs recurrent depth approach (Geiping al. 2025). dividing the model into a bottom-level feature extractor, a recursive reasoning core, and top-level output layer. The recursive core iteratively refines latent states until convergence, while training stability is ensured through a two-phase "gradient-free warmup + truncated backpropagation" strategy with entropy- based early stopping, offering controllable trade-offs between reasoning depth and compute.
Theoretically, ChainGPT is Turing complete and can model any computable function. Empirically, solves problems that remain challenging for state-of-the-art LLMs, even without pretraining CoT supervision. Across benchmarks, ChainGPT consistently outperforms comparably sized baselines on diverse tasks.
The main contributions of this work are summarized as follows:
1. We propose the ChainGPT architecture, introducing dual-reasoning mechanism for lan- guage model reasoning. ChainGPT combines intra-layer deep computation with inter-layer iterative refinement, while maintaining near-linear complexity.
2. We introduce RWKV-Product, LoRA-based multi-substep state update mechanism that significantly enhances intra-layer reasoning depth and the effective rank of state updates with minimal parameter overhead. And State-Guided Sparse Attention (SGSA), which builds long-range dependency paths with near-linear complexity and effectively resolves the state bottleneck problem of RNN.
3. By combining recurrent depth approach with intra-layer sub-step reasoning, ChainGPT shifts iterative reasoning from token generation to multi-round latent optimization, enabling dual recursive reasoning within the model. We prove theoretically that ChainGPT is Turing complete under ideal conditions, and experiments show that it surpasses existing methods on some challenging reasoning tasks and consistently outperforms baseline models of com- parable scale on benchmark evaluations.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
2.1 EXPRESSIVE BOTTLENECKS OF RNN MODELS
Although RNNs offer efficiency advantages, theoretical studies reveal a fundamental trade-off be- tween expressiveness and computational feasibility. Merrill al. (2024) showed that while linear RNNs with full-rank transition matrices can, in theory, recognize any regular language, their training costs prohibitively high, limiting practical applicability. Grazzi et al. (225) further demonstrated that under finite numerical precision, RNNs with diagonal transition matrices (e.g.. Mamba (Gu & Dao 2023 fail perform even basic state-tracking tasks. This observation empirically vali- dated by Jelassi et al. (2024), who found that RNNs exhibit significant performance bottlenecks in tasks such as multi-hop reasoning.
To balance efficiency and expressiveness, RWKV-7 introduced a "diagonal + rank-1" structural re- finement, which achieved modest performance gains without substantial computational overhead. However, since each update is restricted to rank-1, its expressive power remains limited. DeltaProd- (Siems et al. 2025) attempted to address this issue through multi-step Householder updates, but the additional parameter overhead makes it impractical. The AWKY-Produs proposed in this work introduces a LoR b-ase multi-substep decoupled state update mechanism. Within single forward pass, it executes multiple sub-steps, enabling "multiple rounds of reasoning within one layer." With only a small increase in parameters, the effective rank of updates can dynamically extended to M (a tunable hyperparameter), resulting in faster convergence and stronger representational power.
2.2 DIFFERENT DESIGN PRINCIPLES OF HYBRID ARCHITECTURES
Given the respective limitations of pure RNN and pure Transformer architectures, recent years have seen the emergence of various hybrid models that attempt to combine the strengths of both.
Jamba adopts 1:7 alternating stack of Mamba and Transformer layers, retaining full self-attention mechanisms. While this approach excels in long-sequence tasks, the embedded O(N2) self- attention modules become the primary scalability bottleneck. For instance, at a 32k context length, the computation of a single attention layer already exceeds that seven Mamba layers combined. In contrast, ChainGPT employs sparse attention design, reducing complexity O(N(W N/G)), where W denotes window size and the global anchor interval, thereby enabling near-linear scala- bility.
Samba alternates between Mamba and sliding-window attention (SWA) to achieve hybrid modeling. However, pure window-based mechanisms still face state bottlenecks. Our experiments confirm that for long-context tasks (>8k), pure SWA architectures exhibit degraded performance, whereas the global anchor mechanism introduced in ChainGPT completely resolves this issue.
2.3 REASONING METHODS BEYOND FIXED DEPTH
Research aimed enhancing the reasoning ability of fixed-depth Transformers can be broadly di- vided into two categories; generation- based reasoning extensions and internal state-based reasoning enhancements.
The former is exemplified by COT. CoT and its variants extend effective computational depth by generating structured reasoning steps. ToT generalizes this idea into tree search, enabling the ex- ploration of multiple reasoning paths. Self-Consistency (Wang et al. (022) improves reliability by ensembling multiple reasoning chains through voting. However, these methods face inherent limi- tations: computational cost is tightly coupled with generation length, and the reasoning process is constrained by the expressive power of natural language.
The latter category focuses on iterative computation within the model's latent state space. Universal Transformer (Dehghani et al. 2019) introduced depth-adaptive mechanisms that allow different po- sitions to perform varying numbers of computational steps. Looped Transformer (Liu et al.) 2023) placed the entire model loop. effectively increasing depth via multiple forward passes. More recent work explores recursion in Transformers, such the Hierarchical Reasoning Model (HRM) (Wang et al. 2025). which performs iterative computation across hierarchical levels, and the Depth- Recurrent Model, which refines the Looped Transformer by proposing recurrent depth approach.
3

---

Published as a conference paper at ICLR 2026
ChainGPT belongs to the category of internal-state reasoning but innovatively introduces a dual- reasoning mechanism: WWWKV-Produc performs multi-substep state updates within single forward pass, explicitly enhancing intra-layer reasoning depth; recurrent depth approach, combined with truncated backpropagation and adaptive early stopping, achieve a stable trade-off between reasoning depth and computational budget.
CHAIN-BLOCK
3.1
OVERALL DESIGN
ChainGPT is composed of multiple stacked Chain-Blocks. Each block consists of two core sub- structures: a state layer and an attention layer, as illustrated in Figure
Nett
Attention Layer
Crusel Globai Anchors
Query Position
Glkbbll
State Layer
Propagation
Picket
Figure 1: Chain-Block
The core the state layer is the proposed RWKV-Product mechanism. employs a "multi-substep reasoning" approach, in which multiple substeps are executed within single forward pass. Each substep maintains its own key-value representations and gating strategy. By adjusting the number of substeps, the model can explicitly increase its intra-layer reasoning depth without adding additional layers.
State Layer:
Attention Layer: Based the output of the state layer, SGSA attends only tokens within local window and anchor tokens sampled at fixed intervals. By leveraging context summaries gen- erated through multi-substep reasoning, SGSA constructs sparse yet globally covering dependency connections.
This design of "multi-step reasoning sparse aggregation" enables ChainGPT to achieve deep soning within single layer at near-linear computational cost, while efficiently modeling long-range semantic dependeneies across layers.
3.2 RWKV-PRODUCT
Let x, denote the input step f, and E RxD the state matrix (D is the input dimension):
St A(x)ss B(x)).
Here A(x,) E RDD is the transition matrix, and B(x;) the injection matrix. Each sub-step j has its own Aj (xt) and (x;). so
M-1 A(x,)
II (diag(ar.) - Bog K2 1=0
where (iag(a,.)) applies correction. With independent update vectors, A(x;) exhibits "diagonal rank-M" structure.
applics thannel-wise decay, and Bo.j controls the strength of the subsequent rank-

---

Published as a conference paper at ICLR 2026
Query
Other Taiees
Local Window (W-2)
Global Anchers (Gm3)
Query
Figure 2: State-Guided Sparse Attention mechanism
The injection matrix aggregates sub-step contributions:
M-1
M-1
B(x:) = Σ
I
(Bos (Boss vT)).
(3)
j=0
k=j+1
where k.) and Vi.j denote c-path key and value vectors, and Bag determines the weight of the sub-step contribution.
To balance expressiveness and parameter efficiency, all keys/values adopt a shared baseline plus LoRA increments:
k1 k base
+ x. W
(4)
(b)
(5)
[ ll2
Ki. kt kaas x W (o ((.k2))
(0)
Utj u/ base x.e Wylcol m)(c22
Step sizes are specific to each sub-step, with shared bias an,, between the and c-paths. symbol denotes the sigmoid activation function:
The
Ab.j = d(o.. (x, W 1. 2.))
(8)
Beg = of ao.5 (x, W (
This multi-substep design strictly enhances representational capacity with increasing M. Proofs and details appear in Appendix implementation in Appendix D.I.
3.3 STATE-GUIDED SPARSE ATTENTION (SGSA)
To capture long-range dependencies, we introduce SGSA, which operates on state outputs. For query position, the accessible key consists of:
Local Window: W neighbors around the query
Global Anchors: positions sampled every stride G
Thus, the Chain-Blode decomposes retrieval into "write and pointer-read": RWKV-Product aggre- gates local fragments and propagates them to anchors, while SGSA retrieves content via sparse addressing (Figure
Appendix B proves that with proper hyperparameters, Chain-Block solves the Multi-Query Asso- ciative Recall (MQAR) task (Arora et al., 2024) for arbitrarily long sequences: for any (k,, 02), dj kü the model retrieves 24.
Compared with dense O(72) attention, our method requires only O(T(W + +T/G)) sparse links, cnabling efficient global modeling. Implementation is provided in Appendix D.2.
RECURRENT DEPTH APPROACH
4.1
OVERALL DESIGN
The recurrent depth approach decomposes the model into three modules: a bottom-level feature ex- tractor, recursive reasoning core, and top-level output module. Each is constructed from stacked

---

Published as a conference paper at ICLR 2026
Chain-Blocks. As shown in Figure the bottom-level module generates stable contextual represen- tations, the recursive core conducts multi- reasoning over them, and the top-level module maps the final states into the target output distribution. Typically, the bottom-level and top-level modules each contain a single block. A full algorithmic description is provided in Appendix D.3.
This design enables ChainGPT achieve computational universality with sufficient memory and time, allowing it to simulate any Turing machine and thus model arbitrary computable functions. A formal proof provided in Appendix C.
Bottom
Reasoning Core
Top
Figure 3: Recurrent depth approach
4.2 ADAPTIVE REASONING DEPTH
To reduce redundancy, an entropy-based early stopping mechanism monitors output stability, At iteration i, the output is decoded:
logits, Decode(X.), 4 logits,, -1, Pt softmax(4s).
(10)
Stopping Criterion: Entropy for sample b is
H.(b) Σ
(b) log (br,;(b) =);
(11)
i=1
with difference
(H,(b) = (b) - H,(b).
(12)
(b) S T. recursion halts at step t. Threshold t fixed constant.
EXPERIMENTS
This section validates the effectiveness of the ChainGPT architecture through comprehensive bench- marks, systematic ablations, and scalability analyses. Unless otherwise stated, all experiments are conducted on NVIDIA L20 GPUs.
5.1 COMPREHENSIVE PERFORMANCE EVALUATION
We train ChainGPT models with 0.5B and 1.5B parameters, as well as baselines (Qwen2.5 (Qwen Team 2024)), on 20B and 40B tokens from the Wib dataset (Penedo et al.) (024). Evaluation follows the lin-eval-har rness framework (Biderman et al.] 2024) under a zero-shot setting, covering six representative tasks on language understanding and commonsense reasoning (Clark) 2018 Zellers et al., 2019 Bisk et al. 22200; Welbl et al. [0177] Wang et 2018). As shown in Table I ChainGPT significantly outperforms baselines with the same parameter counts, with notable gains on ARC-Challenge and HellaSwag- waskk tasks that require more complex reasoning-indse stronger general-purpose reasoning ability.
al. 2018
Table 1: Comparison of comprehensive performance results
Model
arc_challenge
arc_easy
hellaswag
piga
sciq
glue
Avg.
Qwen2.5-0.5p
0.2218
0,4082
0.3224
0.6425
0.5290
0.4664
0.4317
ChainGPT-0.5B
0.2389
0.4773
0.3644
0.6632
0.5330
0.4679
0.4575
Owen2.5-1.5B
0.2696
0.5488
0.4091
0.6915
0.6380
0.4783
0.5059
ChainGPT-1.58
0.2986
0.5779
0.4269
0.7018
0.6860
0.4836
0.5291

---

Published as a conference paper at ICLR 2026
5.2 VALIDATION OF ARCHITECTURAL EPFECTIVENESS
Effectiveness of the LoRA Sub-Step Mechanism
Building on (oo.kt./mmmddddnannoooo
(BlinDDL, 2024). we extend RWKV-7 with
single LoRA sub-step construct RWKV-Produce As shown in Table with only 0. increase in parameters, RWKV-Prodiee reaches lower validation loss with fewer training steps, indicating that intra-layer multi substes updates improve both training efficiency and representational capacity.
Table 2: Validation of the Effectiveness of the LoRA Sub-step Mechanism
Model
Training Steps
Validation Loss
GPT2
19560
R:3.28
RWKV-7
3200
3.2715
RWKV V-Prductt
2500
3,2684
RWKV-Product
3200
3.1901
Impact of Sub-Step Count on Performance
To study how the number of sub-steps affects performance, we evaluate models (180M parameters) trained with different sub-step counts M. Training uses 8B tokens from FineWeb and evaluation is performed on 1,000 validation samples. As shown in Figure[4 validation loss consistently decreases M increases, with M offering a favorable trade-off. These results suggest that sub-steps effectively deepen intra-layer computation, thereby improving representational capacity.
1.45
2.55
335
2:50
0.43
335
135
Than kg 2t2p3
Figure 4: Effect of sub-step count on model performance
Figure 5: Comparison of coupled and
decoupled state propagation paths
Necessity of Decoupling State Propagation Paths
We compare decoupled and coupled designs for state propagation. With approximately matched parameter counts, the decoupled variant consistently outperforms the coupled version throughout training. as shown in Figure
Hybrid Architecture Resolves Associative Recall Bottlenecks
On the MQAR task, with head dimension 64 and number of heads 2, the pure State Space Model architecture (RWKV-7) exhibits noticeable accuracy degradation as sequence length grows, whereas ChainGPT with SGSA maintains accuracy above 99% across settings (Table (3). This in- dicates that SGSA's "anchor writing sparse pointer retrieval" mechanism effectively mitigates the capacity bottleneck caused by finite state dimensionality.
Table 3: Performance on the MQAR task
Model
(128,8)
(256,16)
(512,64)
(1024,128)
(2048,256)
RWKV-7 > 99%
99% 99%
98.43% 99%
95,01 %
72.93%
ChainGPT
99%
999
>> 99%
Effectiveness of the Global Anchor Strategy
7

---

Published as a conference paper at ICLR 2026
We evaluate different global anchor intervals G PG-19 (Rae al. 2019) As shown in Table pure sliding-window window attention (Samba approach) degrades beyond 8K context. In contrast, introduc- ing only sparse periodic global anchors yields a perplexity trajectory that closely matches the much more expensive global attention (Jamba approach), with no notable degradation. This suggests mechanism maintains long-range dependency modeling
that SGSA's near- near-linear complexity.
Table 4: Perplexity comparison for global anchor strategies
Global Token Interval (G)
2k
4k
6k
8k
10k
12k
14k
16k
Global Attention
17.78
16.82
16.46
16.20
16.11
16.07
16.07
16.08
G=32
17.78
16.82
16.46
16.20
16.11
16.07
16.07
16.09
G=64
17.78
16.82
16.46
16.20
16.11
16.07
16.07
16.08
G=128
17.77
16.81
16.46
16.20
16.11
16.07
16.07
16.08
Pure Window Attention
17.78
16.83
16.47
16.21
16.12
16.09
16.10
16.12
Impact of Recurrence Depth on Performance
Keeping other conditions fixed, we increase the number of recursive iterations from 16. As
shown in Figure 6 validation perplexity decreases steadily with more iterations and reaches its best 12.
3.4
3.00
Thirdig stops
Figure 6: Effect of recurrence depth on model performance
Comparison with Other Recurrent Models and CoT Methods
To directly evaluate the performance of ChainGPT against other recurrent models and CoT meth- ods, we conducted controlled arithmetic reasoning experiments on the public GOAT arithmetic dataset (Liu & Low (203) in Table [ All models were trained from scratch under identical train- seg-testing splits, tokenization schemes, and optimization settings to ensure fair comparison. The evaluated models include the baseline architectures Qwen3 and RWKV-7, their recurrent variants (Qwen3+Loop. RoKV)7+Loop) HRM, our proposed ChainGPT, as well as their CoT-augmented counterparts (Qwen3+CoT, ChainGPT+CoT)).
Table 5: Accuracy the GOAT dataset
Model
Accuracy
Qwen3 RWKV-1 V-7
33.82%
23.69%
Qwen3 Loop
50.54%
RWKV-7 Loop
24.81%
HRM
54.00%
ChainGPT
57.53%
Qwen3 + CoT
88.43%
ChainGPT + CoT
99.98%
Without CoT supervision. ChainGPT outperforms existing recurrent reasoning architectures, in- dicating that its intrinsic latent recursion mechanism enables more effective multi-step arithmetic
8

---

Published as a conference paper at ICLR 2026
reasoning. When combined with explicit CoT guidance, ChainGPT+CoT significantly surpasses Qwen3+CoT, achieving nearly perfect accuracy. This result suggests that ChainGPT's implicit re- cursive states and the external CoT trajectory are complementary, jointly enhancing the model's reasoning capacity.
Analysis Challenging Reasoning Tasks
To assess ChainGPT in complex reasoning scenarios, we conduct systematic evaluations on three representative hard tasks: ARC-AGI-1 (Chollet) 2019). Sudoku-Extreme, and Maze-Hard (Wang et al. 2025). Following HRM, we train 30M-parameter ChainGPT from scratch on 1000 samples, and train Direct Pred as Transformer baseline under identical settings. Results shown in Fig- ure 7 For ARC-AGI-I, the scores are taken from the official leaderboard, while for and Maze-Hard, the scores are obtained through the corresponding APIs.
ARC-AGI-1
Sudoku-Extrems
Maze- Hard
100
Accuracy (%)
80
Accuracy (%)
Accuracy (%)
40
M.1%
20
Direct pred
03 03-mini-high
Claude 3.7 sk
Deepseek R1
ChanGGP
Direct pred
03-minihhigh
Claude 3.7 8K
Deepseek R1
ChainGPT
Direct pred
03-mini-high
Claude 3.7 BK
Deepseek R1
ChainGPT
Figure 7: Benchmark accuracy comparison across different tasks
ARC-AGI-1 evaluates abstract rule induction under extreme few-shot conditions, where each prob- lem provides only a handful of inpul-output examples. The central challenge lies in learning reusable compositional rules rather than memorizing patterns. Conventional deep learning and CoT- based approaches often fail to achieve stable generalization in such settings.
Sudoku-Extrems aggregates acknowledged hard Sudoku instances ranging from Kaggle datasets to Magictour- 1465 and difficulty measure. The difficulty stems from the need for precise constraint propagation and back- tracking. Fixed-depth Transformers
with tdoku backtracking steps used as an objective
well-suited for long-trajectory backtracking, and parallel CoT sampling cannot reliably ensure deterministic correctness.
Maze- requires finding shortest paths on 30 30 grids with enforced path lengths 110. The challenge is to construct optimal solutions-mes reachability is insufficient. Large Transformers and CoT reasoning often generate suboptimal invalid paths large-scale mazes, showing their limitations in stable graph search.
ChainGPT excels in these scenarios due its dual-reasoning mechanism, which enables multi- round, revisitable reasoning beyond conventional one-pass prediction or shallow CoT sampling. This unified design supports iterative rule induction and constraint propagation with contradiction checking, allowing the model to emulate algorithmic processes more faithfully across tasks. As shown in Figure 7 ChainGPT achieves 38.6% accuracy on ARC-AGG-I-suppasing strong base- lines such as 03-mini-high and 1ouude-37-8K-54.444 accuracy on Sudoku-Extreme,, and 77.4% accuracy on Maze-Hard with valid optimal solutions. These results suggest that ChainGPT ef- fectively addresses the limitations of existing approaches and demonstrates robust performance challenging reasoning tasks.
5.3 SPEED BENCHMARKING
We evaluate the computational efficiency of different modules sequence lengths from 4K to 16K using a single NVIDIA A800 GPU, with total model dimension = 1024 and head dimension = 64, and attention implemented via Fsscaled.dot pwwpppuctaattetttiotto... Results in Figure & show that standard attention suffers from quadratic scalability bottleneck as sequence length N increases, whereas RWKV-Product achieves approximately linear runtime growth with controllable overhead across sub-step configurations, enabling an rerciincy--efffrmmnnee trade-off. Furthermore, the hy- brid architecture, which integrates state layers with sparse attention, demonstrates near-linear scaling

---

Published as a conference paper at ICLR 2026
through "state compression sparse aggregation" strategy, offering clear efficiency advantages for long-range modeling under constrained computational cost.
RATE Pratec (h-7)
Compuestio 10r 1710
Product
Figure 8: Computation efficiency comparison (ms)
6 CONCLUSION
In summary, ChainGPT introduces a principled dual-reasoning architecture that directly addresses the expressiveness bottlenecks of existing LLMs by shifting reasoning from token-level generation into latent computational space. Through the combination of RWKV-Produch, multi-substep state updates and SGSA's near-linear global modeling, ChainGPT achieves intra-layer depth and efficient long-range reasoning simultaneously, while recurrent depth approach further extend its computa- tional universality. Theoretical analysis establishes Turing completeness under proper configura- tions, and extensive experiments confirm that ChainGPT not only surpasses state-of-the-art base- lines of comparable scale but also solves reasoning tasks that remain unsolved by current L.MMs. These results highlight ChainGPT's unique advantage in unifying efficiency, scalability, and reason- ing power, opening new directions for the design next-generation language models.
ACKNOWLEDGMENTS
The work is supported by Beijing Natural Science Foundation (L224701) and National Key RD Program of China (No. 2024YFF0907003). We are grateful to the anonymous reviewers for their valuable and encouraging feedback and suggestions this study.
REFERENCES
Sanjeev Arora and Boaz Barak. Computational Complexity: A Modern Approach. Cambridge
University Press, 2009.
Simran Arora, Sabri Eyuboglu, Aman Timalsina. Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language mod- cls. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL httpps://openreview net. wtt.ee/ew.eeeeoooeoooooo........
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, and et al. Lessons from the trenches on reproducible evaluation of language models. 2024, arXiv preprint.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference Artificial Intelligence (AAAT), 2020.
BlinkDL. moddedd-nanogpt--wk: Nanogpt speedrun + rwky integration. htttps:/
rwkv, 2024. Achieves 3.278 validation loss in 7 min on 8xH100; integrates Rotary, QK-Norm, ReLU?, Muon optimizer, RWKV-7 attention free model. Accessed June 2025.
com/
François Chollet. On the measure of intelligence: The abstraction and reasoning corpus (are-agi). 2019. Introduces the ARC-AGI benchmark focused fluid intelligence.
10

---

Published as a conference paper at ICLR 2026
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018. arXiv preprint.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In International Conference on Learning Representations (ICLR), 2019. Introduces per-position adaptive halting (ACT) for variable computation depth.
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv22502.05171, 2025. URL
Bttpss//araa..oorr//////2222 05171
Riccardo Grazzi, Julien Siems, Arher 7ela, Jörg K.H. Franke, Frank Hutter, and Massimiliano Pon- til. Unlocking state-tracking in linear rnns through negative eigenvalues. In Proceedings of the International Conference Learning Representations (ICLR), 2025.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023.
arXiv preprint.
Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics (TACL), 8:156-171, 2020.
Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat after me: Trans- formers are better than state space models copying, 2024. arXiv preprint.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman, and Yoav Shoham. Jamba: A hybrid transformer-mamby language model, 2024. arXiv preprint.
Tiedong Liu and Bryan Kian Hsiang Low. Goat: Fine-tuned Ilama outperforms gpt-4 on arithmetic tasks. arXiv preprint arXiv:2305. 14201, 2023. doi: (0.48550/rxxiv22355142000
Yang Liu, Kangwook Lee, Robert Nowak, and Dimitris Papailiopoulos. Looped transformers are better at learning learning algorithms. arXiv preprint arXiv:2311.12422 2023.
William Merrill, Jackson Petty, and Ashish Sabharwal. The Illusion of State in State-Space Models, 2024. arXiv preprint.
Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. 2024. arXiv preprint.
Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng. RWKV-7 "goose" with expressive dynamic state evolution, 2025. arXiv preprint.
Qwen Team. Qwen2.5 technical report. 2024. arXiv preprint.
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv: 2019. doi: 00448500/a2xiv.1911..00
Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling, 2024. arXiv preprint.
Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riccardo Grazzi. Deltaproduct: Improving state- trac-itacking in linear mns via houscholder products, 2025. arXiv preprint.

---

Published as a conference paper at ICLR 2026
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems, volume 30, pp. 5998-59989999 60088 2017.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. 2018. arXiv preprint.
Guan Wang, Jin Li, Yuhao Sun, Xing Chen, Changling Liu, Yue Wu, Meng Lu, Sen Song, and Yasin Abbasi-Yadkori. Hierarchical reasoning model. arXiv preprint arXiv:2506.21734 2025. Two-level recurrent modules for multi-timescale reasoning in a single forward pass.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Denny Zhou, Sharan Narang, and Aakanksha Chowdhery. Self-consistency improves chain of thought reasoning language models. arXiv preprint arXiv:2203.11177 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. 2022. arXiv preprint.
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. 2017. arXiv preprint.
Shunyu Yao, Dian Yu, Jeffrey Zhao, and et al. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:230S. /00001,,,,00,,,,, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
A THEORETICAL ADVANTAGES OF RWKV-PRODUCT'S EXPRESSIVE POWER
This appendix presents a rigorous theoretical analysis of the multi-substep update mechanism used in RWKV-Product. Our goal is to show, mathematically, that as the number of sub-steps M increases, both the family of realizable linear transformations and the capacity of information injection strictly expand. This analysis provides a solid mathematical foundation for the advantages of multi-substep updates over traditional single-step updates and supports the design's rationale.
To keep notation consistent, we first define the update operator for each sub-step. Let Dt,j == diag(ar.) E DxD be a diagonal matrix. For simplicity, we absorb the step-size factors into the vectors by defining Ut.j = //B.,kk
(b) and Vrj == ko. Hence, at time I, the transformation matrix of the j-th sub-step has the following "diagonal minus rank-1" form:
A Dt.j V
(13)
Basic Assumption: To ensure the validity of subsequent derivations, assume all diagonal matri- ces Dr.j are invertible; i.e., no diagonal entry zero.
A.I LOW-RANK STRUCTURE OF PRODUCTS: COMPACT WY REPRESENTATION
Theoretical significance: Equivalently viewing the composite effect of AM successive rank-1 1 up- dates as a single "diagonal + rank-M" transformation A(x,) = D ÜTVT clarifies the algebraic structure underlying enhanced expressiveness: the upper bound attainable rank increases from to M.
Lemma 1: Given vector pairs {uu, v let À 11 -1-,(,
u,vJ). Then there exist matrices U, V E DxM and an upper-triangular matrix T E RMMM such that
and (uTV & M.
(14)
12

---

Published as a conference paper at ICLR 2026
Proof:
By induction. Let Am =I[===-- u, The base 1 is immediate. Assume
Am - Um Tm Vm with Tm E mm upper triangular. Consider Am+1 An (I
Um V m.):
Am+1 = (T Um T' VIT)(( (1 1m Vm)
(15)
I - -- ViT
Um Um + (Um Tm V. Um)
(16)
I'm
-Tm (Vi Um
V
1 [l Um
(17)
1
Define Um+1 1Um; Un]] Vm+1 = [V,, Vml, and the corresponding (m + 1) x (m + 1) upper-
triangular' T m+1- The inductive step holds.
Lemma 2: Suppose each sub-step transformation A, D2 u,v,. Then their product
[ Ag can be written
A D- UTVV
(18)
where D = =[I--=--1 D3 and rank M.
Proof: Again by induction. Let Pm = I[==m- Ay. The base case m = is trivial. Assume
Pm Dm - Lms where Dm = [1==m-1 Dj and rank(Lm) I m. Consider Pm+1 = AmPm =
(Dm um v (Dm - Lm):
Pm11 Dm D + (um W (um V M
(19)
Lm+1
Let Dm+1= = Dm. The column space of the low-rank part Lm-+1 is contained in col( Lm) + span(um). Since (ank(L.)) S m. we have dim(col( Dm I.m)) m, hence dim(col (Lm+1)) S m + 1, ie., rank([m+1) Σ m + 1. The induction closes.
Conclusion 1: A(x:) D UTV with rank (UTTV S M.
A.2 STRICT INCLUSION: Fi FM FOR M 2
We prove, in constructive manner, that multi-substep updates strictly enlarge the family of realizable linear transformations. Concretely, there exist matrices representable by product of two "diagonal minus rank- 1 factors (M=2), but not represetaabeee by any single-step mechanism (M=1). This establishes set-theoretic notion of strictly stronger expressive power.
Theoretical significance:
Define the matrix families (diagonal factors are assumed invertible, consistent with the basic as- sumption in Appendix A):
FM = II (D, (Dz-uyv)): : Dy invertible diagonal J=M-1
(20)
F {D-uu uv invertible diagonal]
(21)
Theorem 1: For dimension D 2 3 and sub-steps M 2 2, we have the strict inclusion
F FM-
(22)
Proof: We show (i) Fi FM, and (ii) there exists matrix in Fg but not in Fi.
(i) Inclusion Fi S Fas. Take any A E Fi, so A = uv for some invertible diagonal D. Construct element of FM by setting
Dp m==D, D, Ug = u, vo = v, where / is the identity matrix (invertible diagonal). Then
D, = 1, u = (j=1.... M 1),
II j=M-1
(D,-
uv A,
13

---

Published as a conference paper at ICLR 2026
hence Fi S F.M
(ii) Strictess via a constructive separation. Proposition 1 below constructs an explicit matrix A e F2 such that A & Fi. Since F2 S FM for all M 2 2, this implics F1 Ç FM.
Proposition 1 (Constructive separation): Let 2 3 and {ei) be the standard basis IP. Define
:= (I- (I - e = - e102 6203.
(23)
Then A E F2 but A $ Fi.
Proof: The membership A" F2 is immediate: both factors are of the form
I T D uv with D I, Cas Eb.
where D is invertible diagonal.
To show A & F1, assume for contradiction that
A' D uv
for some invertible diagonal D and vectors E IRD. For any i * j. the diagonal matrix con- tributes nothing to off-diagonal entries, hence
Aij (uv Dis U4y
i + j.
(24)
From the explicit form A = - Ele] e2e3. we have the off off-niagonal pattern
A = =11 Ags -1,
Ais = 0.
The equalities Ai2 --1-111- -1 and A23 U2l -1 imply
01 + 0, U2 * 0, and U2 + 0, 03 + 0.
In particular, u, * 0 and 03 + together imply i 03 * 0, hence
Ais = u1 033 + 0,
which contradicts Ais 0. Therefore A $ Fi
Remark ofeereo heuristic). A single-step form D- T has roughly 2D 3D 1 effective degrees of freedom (accounting for the scale redundancy (u, v) (-u,a- vv)) A two-step product introduces (at least) twice many parameters, suggesting a strictly larger family, However, since the parameterization is not injective, we rely on the explicit separation in Proposi- tion 1 (rather than on dimensional counting) to establish strict inclusion.
o
Conclusion 2: There exists A E FM Fi for any M 2 2 and D 2 3 (e.g., A above), hence Fi S FM.
A.3 RANK UPPER BOUND AND EXPANSION FOR THE INJECTION TERM Bxx;)
Theoretical significance: Not only does the expressiveness of the state transition A increase, but the capacity the injection channel B expands tandem, jointly enhancing affine transformation capability.
By definition, the injection matrix B(x,) EnHo' Ptς Re where rank(Re,j) = 1.
Proposition 2: For fixed input x,, the rank of Bxx;) satisfies
rank(B(xx)) S M.
(25)
Equality holds when the image spaces of (Pr., Rr, are linearly independent.
Proof: By the rank inequality for matrix products, min{rrak((PE (P.s), ;)) = Using subadditivity of rank.
rank(Pr,,R.,
M-1
M-1
rank(B(x;)) = rank
Σ 1=0
Pa. Rej
Σ 2=0
rank (ank Re.)) S M.
(26)
Conclusion 3: rank(B(xx)) M. and the upper bound is tight.
14

---

Published as a conference paper at ICLR 2026
A.4 SPECTRAL PROPERTIES AND STABILITY
Theoretical significance: Enhanced expressiveness does not the expense of numerical stability. The diagonal factors control spectral contraction, while rank-1 terms act as bounded per- turbations that contribute expressiveness.
Lemma 3: Let D - Dr.j- Owing commutativity of diagonal matrices, the spectral radius pl-) satisfies
M-1
((D) S II (DD.s) ==
(27)
Proof: Write Dt.5 = diag(d
( Then D diag(Il
II, 4. "i.,d,.....I, 1.) Hence
p(D) = max II
max
( 2.j
S III
max d
III ((D.))
(28)
Lemma 4: The norm of the difference between A(x,) and its diagonal-dominne dominant part D is bounded. Using the telescoping identity:
M-1
M-1
M-1
m--1
M II j=m+1
II j=0
X,
II i=0
Σ
II Xj
(Xm - Ym)
Y
(29)
m=0
j=0
and setting X, At, and norm inequalities yields:
Y = Dr.. we have Xm Ym = -m, V i,m- Substituting and applying
M-1
A(x:) - Dr.
Drs
|Arill
llll
II j=m+1
(Deiille)
(30)
so the bound depends solely on the norms of sub-step parameters.
Conclusion 4: The spectral radius controlled stepwise by the diagonal factors, while the rank-1 terms serve as bounded perturbations.
A.5 STRICT EXPANSION OF THE AFFINE TRANSFORMATION FAMILY
Theoretical significance: Collecting the above results, we provide unified functional functional-family state- ment of the expressiveness gains afforded by multi-substep updates.
Define the family of affine maps:
GM == {s + A(x)s + B(x) A(x) E FM, B(x) satisfies Proposition 2}.
(31)
Theorem 2: For M 2 2 and dimension D 2 the single-substep affine family 91 is strict subset of the multi-substep family Gas:
G, SM
(32)
Proof: This follows directly from the preceding results. By Theorem 1, the linear part expands strictly, Fi & FM. By Proposition 2, the rank upper bound of the affine injection increases from to M. The simultaneous enhancement of both parts ensures a strict expansion the overall function family.
Conclusion 5: 91 S. 9M the intra-layer expressive power of RWKV-Product increases strictly with the number of sub-steps M.
15

---

Published as a conference paper at ICLR 2026
B CONSTRUCTIVE PROOF OF CHAIN-BLOCK SOLVING MQAR FOR ARBITRARY SEQUENCE LENGTHS
This appendix proves that single Chain-Blocn (RWKV-Product + SGSA) can solve the Multi- Query Associative Recall (MQAR) task (Arora et al.) 2024) for arbitrarily long sequences under idealized assumptions. The purpose to justify the main-text intuition: RWKY-Product performs lossless "write/overwrite" in latent state, while SGSA performs sparse "pointer-read" over local windows and periodic anchors.
B. PROBLEM SETUP AND OBJECTIVE
We observe a length-T sequence where some positions are writes of key-value pairs (Rg, 24) and some positions queries with qi K. For each query time t, the correct answer is the value associated with the most recent write of the same key:
MQAR Task (Latest-Write
Ans(t) = Uz*,
=
max S t:: 9t)-
This is the Latest- Write- Wins (LWW) semantics.
Chain-Block. A Chain-Block consists of: (1) RWKV-Product state layer that updates latent state and emits per-token representations: and (2) a Stat-GGided Sparse Attention (SGSA) layer that performs causal sparse attention (i) local window of size W and (ii) set of periodic anchor positions with stride G.
Goal. We show there exists parameter setting such that single Chain-Block outputs the correct (ns(() at every query position I, for any sequence length T, while respecting LWW.
B.2 IDEALIZED ASSUMPTIONS (EXISTENCE PROOF)
We assume:
1. Finite key universe and sufficient dimension. The key set K. is finite, and the model has enough hidden dimension to allocate a dedicated "associative memory" subspace.
2. Orthogonal key encoding. There exists an encoding map Φ - IRdx such that
(k)),, ((K)) ōg
3. Injective value encoding. There exists an encoding v V - Rdd that is injective (e.g., one-hot or any separable embedding).
4. Sufficient numerical precision. We can make attention "hard" by scaling logits and can treat linear updates as exact up to an arbitrarily small error.
5. RWKY-Product expressivity for low-rank affine state updates. RWKV-Produet can realize (up small error) the rank-1 projector-and-ingee updates described below on designated memory subspace, by choosing appropriate diagonal decay and rank-1 terms (consistent with its "diagonal low-rank" structure).
B.3 MEMORY ENCODING FOR MQAR
We use an explicit associative-memory construction.
Outer-product notation.
We use ab for the outer product, producing matrix in Rdxxd.
Periodic anchors. Define the anchor set
A (l (I,..., T} mod G G -
SGSA will allow each query to attend to positions in
D()) (t' max W) s1' (aEA:as))
16

---

Published as a conference paper at ICLR 2026
Two-level memory: segment buffer + global memory. We maintain two associative memories:
Segment (recent) memory (Pite, _M/")) that stores writes since the most recent anchor.
Global memory (Pelub, M,lloh that stores all writes up time t.
Each memory consists presence vector P E Rdd and a value matrix M Rddedd;
P.(-)
Σ
[ has appeared) ((k),
Me
Σ ((k) (()))((((() kEK
REK
where (t) denotes the latest written value of key &o up to time
Correct readout (no vec/Kronecker required). If the above invariant holds, then for any query key E IC,
M, ((4) = (t))
(33)
ie., multiplying by o(q) selects the corresponding value embedding (because pf-) are orthonormal).
B.4 RWKV-PRODUCT LAYER: LWW WRITE/OVERWRITE AND PERSISTENCE
We now specify update rules that enforce LWW exactly, and argue RWKV- can implement them on its memory subspace.
Rank-1 overwrite for LWW. Let := o(ks) Rd and == vb(x)) E R& for a write token at time t. Define the project-ands sogject-and-injert update:
M, (I cee, Mi-1 + cuu.
(34)
This overwrites the row corresponding to k, with de while leaving all other keys unchanged:
Mf (bk;) = Ut,
M, o(k) = M2_1((K) for k' + k..
Likewise, presence can maintained by
P. -1- ( ( Pt-1 + Ct.
(35)
which sets the coefficient of key hy in P. to (in the orthonormal basis).
Global and segment updates. For write at time l: Mglob = ee, M + GuT,
pelob = (I - G,e, ) + +
and similarly for the segment memory:
M (I - cte, M + cut, pank (I ete, ) Pray cz.
For non-write token, keep both memories unchanged (identity transition).
At an anchor position E A, after producing the token representation (see below), we reser the segment buffer:
M O, peck 0.
(36)
This ensures the segment buffer always represents writes after the last anchor.
RWKV-Produs updates its state via an affine map with a "diagonal + low-rank" transition and low- rank injection. On a designated memory subspace, one can choose: (i) diagonal factors close (to preserve memory), (ii) rank-1 terms proportional to eye, (to realize (/ e,e," )), and (iii) injection terms proportional to ezu, (to realize the overwrite). Anchor resets in (36) can be implemented by conditioning binary anchor-indicator feature 1 (rE.A) and applying diagonal contraction the segment channels. Since RWKV W-roruuud supports multi-substep low towkkank updates, these operations can be composed within single time step disjoint channel blocks.
17

---

Published as a conference paper at ICLR 2026
B.5
WHAT RWKV-PRODUCT EMITS TO SGSA
Let denote the WWKV-Productoutppu at position / (the input to SGSA). We construct to expose:
Segment summary (always visible locally): a linearized payload of (P. M
A. additionally expose a payload of (Pglob, .Mglob); ); otherwise this part is zeroed.
Global checkpoint (only at anchors):
Anchor indicator: bit 1 (1EA))
Concretely, we may write (schematically)
y/
po ① veo( M
((E))
(PRlob @ )
(other channels)
segment payload
anchor payload
This matches the intended "recent write in local window far history sparse anchors" decompo- sition.
B.6 SGSA LAYER: SPARSE POINTER-REAT READ RETRIEVES THE CORRECT VALUE
We now show SGSA can recover the correct value for each query using only the sparse domain D(t).
Attention domain. For query position t, SGSA attends over
D(1) = {1 () Age.
Keys: presence-based addressability.
Define the query vector (in the key-embedding space) as
Q: a ((q)),
0.
For each visible position pos D(t,, define its addressing key as
pos $ A, pos E A.
K Kpos
pos pgtoh
Then, by orthogonality,
(Q+, Kpms) = a (d())), Kpax) E (0, indicating whether the queried key g: is present in that memory snapshot. Recency bias enforces LWW selection. Let the attention score be
score(t, pos) (Qt, Kpos) p(pos- - t),
(37)
where p(A) is any strictly increasing function of A S (i.c., more recent positions get larger bias). Under the causal mask, pos < t always holds.
In the limit a co, the softmax concentrates on the most recent visible position where the key is present:
pos pos*(t) (t) = arg max score(t, pos) = max(pos E D(t) (d(4)), Kpos) = 1}. D(t)
Values:
return the selected memory payload. Let SGSA return (approximately) the value pay-
load of the selected position:
vec(M, Msoo pos poss(()) (()): pos* (t) AA (anchor case).
pos*(t) = (segment case),
my
(Using self-attention, pos" (1) is allowed.)
18

---

Published as a conference paper at ICLR 2026
Final readout. From the selected matrix Mael (segment or global), obtain the value embedding by
(t) MM
which equals the correct vb( (A))(()) by the LWW invariants of 34 This multiplication is standard senciative-memory read; it can be implemented by fixed readout head that consumes both the query representation (which contains ((q:)) and the retrieved payload (which contains Mael), e.g.,
via a bilinear form on the designated subspace.
B.7 CORRECTNESS SKETCH
We summarize why the above construction solves MQAR for any T.
Case 1: the latest write of qe occurs after the last anchor. Then is present in the segment buffer time t, i.e., (d(4:), ), pr 1, so position t itself is valid match. By the strictly increasing recency bias pos" (t) t. SGSA retrieves M and by (33) outputs the value of the latest write within the segment, which equals the latest write overall.
Case 2: the latest write of qi occurs at or before the last anchor. Then (o(x). P: = 0 (it absent from the segment buffer), but dt is present in the global memory checkpoint at the most recent anchor S t. Thus (t) a, and SGSA retrieves Mglob which encodes the latest value up to a. Since there no later write after a (by assumption of this case), this equals the correct LWW answer.
Therefore, each query retrieves (AAn(tt) exactly (under the idealized assumptions). Since anchors exist periodically, the argument does not depend on T.
C
TURING COMPLETENESS OF CHAINGPT
This appendix proves that the proposed recurrent depth formulation of ChainGPT is, in principle, computationally universal.
C.I SETUP AND MAIN RESULT
Recall the recurrent depth formulation in the main text. Let the input token sequence be embedded and processed by a bottom module to obtain a static basis (read-only context)
Ho Bottom
(ootom((mmee((1)) (Embed(X)) E Rxd,
(38)
which remains fixed during recursion. Let X; E Sxd denote the writable latent state at recurrent iteration i, updated by fixed- parameeaeeter map
X = (X, Ho),
(39)
where 0 shared across all iterations (a single "interpreter step" repeatedly applied).
Idealized conditions. We make the following standard assumptions, which are common in univer- sality proofs for neural architectures:
1. Unbounded workspace and time: the sequence length S (workspace size) and the number of recurrent iterations can be arbitrarily large.
2. Arbitrary-preeision real computation: internal activations support arbitrarily precise real values. Equivalently, we may assume that sufficiently sharp winner-take-all selection (hard- max limit) and saturated gates can be implemented to any desired tolerance.
3. Structured primitives available in a fixed-depth block: the architecture provides (i)
residual connections, (ii) elementwise gating/multipliatiie interactions (e.g., gated-MLP/ RWKV-style gates) enabling masked overwrite, and (iii) position-adjacent access operators (e.g., time shift or an equivalent fixed Toeplitz/conv operator).
4. Sparse attention compatibility: for the theoretical construction, we store the program on global anchor positions that it is always visible under SGSA (or, degenerate special case, we set the anchor stride = so that all program slots are anchors). This ensures random access to program instructions within the sparse retrieval domain.
19

---

1). Published as a conference paper at ICLR 2026
Under these assumptions, we prove computational universality by simulation of a two-counter Min- sky machine (2CM), which known to be equivalent in power to a Turing machine.
Theorem C.1 (Turing Completeness). Under the above idealized conditions, ChainGPT with current depth updates 39 can simulate an arbitrary Turing machine; hence is Turing complete
in principle.
We give a constructive simulation of one step of a 2CM by a single application of Fg. Since any Turing machine can be compiled into 2CM program, repeated application of Fo reproduces the computation trace.
C.2 STATE LAYOUT AND ENCODING
We interpret the sequence axis ID workspace of length (a "tape"), where each position stores a d-diimensional vector. We reserve contiguous segments of positions for different roles:
8) S =
R1 U R2 U U
(40)
where:
P a program segment containing instruction slots (read-only, stored in Ho);
R1, R2 register segments (writable, stored in X₂) for counters R1, R2;
L/ is unused padding (all zeros), allowing the workspace to expand.
Program storage on anchors (SGSA compatibility). Let the SGSA anchor set be S} {I.....
{i = For the proof, we place each instruction slot at an anchor position: P S A (or simply take G Thus, any causal SGSA query position occurring after P can attend to all program slots through the anchor links.
mod G 1).
(41)
Instruction representation. A 2CM program is finite list of instructions indexed by Mprog ) each the form
(op, r, larget, next), op E (INC, DECJZ), E (1,2).
We store each instruction in Ho at one program position p(m) using fixed channels:
an op code indicator (INC vs. DFCIZ),,
register selector indicator (r (r=11 r)
two one-hot masks over P: TargetMask(mm) E {0, I| and Mask(a))(m) E 10
embedded channels aligned with program positions.
This "mask-in-prooe -ons-innprogram" representation avoids explicit arithmetic on program indices and makes jumps purely mask-based.
Program counter (PC) encoding. We represent the currett instruction pointer one-hot mask over program slots, stored in a dedicated channel of the writable state:
PC,(1) E (0. i E
Σ PC)(1) =
(42)
SEP
Intuitively, exactly one program slot is "active" at each step.
Register encoding (unary with delimiter). For each register R2 (r E (1,2)), we use unary encoding within segment R,:
R, =n
1.
0, O.O....]
(43)
where the delimiter "I" is represented by a one-hot delimiter mask channel
Delim! (7) (i) E {0, 1}. i R,,
Σ (elimm('())
(44)
KER,
The tape also contains a data channel holding the cell value (approximately 0 or 1). The delimiter position marks the boundary between ones and zeros.
20

---

Published as a conference paper at ICLR 2026
C.3 PRIMITIVE OPERATIONS REALIZABLE BY ONE FIXED-DEPTE STEP
We now outline how one application of Fe can implement the required discrete operations using a fixed-depth composition of Chain-Blocks (state update + SGSA gated-MLPP, under the idealized assumptions.
C.3.1
POINTER-DRIVEN INSTRUCTION FETCH UNDER SGSA
Because program slots lie on anchors (P S A), SGSA can retrieve from all instruction positions via sparse anchor links.
Let u(i) E I denote the instruction embedding stored at program slot E (as part of Ho). We define attention head (or an equivalent linear aggregation) that computes instruction summary
Instr,
Σ (i) uti),
(45)
KEP
which equals the active instruction embedding since PC, is one-hot.
Implementation note. Equation (45) can be realized by SGSA by setting keys include the PC,(1) channel (or using it to bias logits) and using sufficiently sharp selection temperature. Under arbi- trary precision, the softmax can approximate hard selection to any desired tolerance; we additionally apply saturated gates keep PC, binary-stable (see
C.3.2 ADIACENCY ACCESS VIA SHIFT OPERATORS
We assume access to fixed shift operators along the sequence axis for designated mask channels: (i))t))gg = z(i - 1), Shi)t = =(i + 1).
(46)
Such operators can implemented by RWKV-style time-shife or an equivalent fixed convolu- tion/Toeplitz linear map.
These shifts allow each cell to locally detect whether it is the delimiter position or a neighbor of the delimiter, enabling local rule-based updates without global write operations.
C.3.3 MASKED OVERWRITE (CONSTANT-SUPET) SUPPORT WRITEBACK)
A key requirement is to modify only a constant number of tape locations per instruction (in the sense of ner change), while leaving the remaining cells unchanged.
Given (possibly multi-hot hotti-hot) mask m(i) E {0. 1} supported on a small index set I, a gated-MLP can implement masked overwrite:
X'ii) Xi) + m(i) (V X(i).
(47)
where (i) is the desired writeback value at masked positions and @ denotes elementwise multipli- cation provided by gating. When m is supported I, only positions in / change.
In our construction, m mi) will be formed from delimiter masks and their shifted versions, so | is constant (e.g., {b} or (b, b 1)).
C.3.4 ZERO TEST FOR UNARY REGISTER
Let R, be the register-r segment. Let b be the delimiter position, i.e., Delim, (*) = 1. The register is zero iff the delimiter is the left boundary of the segment, equivalently there no "1" immediately before it inside the segment.
Using shifts, can form a pre-demimiter mask:
PreDelim,") ShiftLeft (Dellmm()
(48)
and then read the data channel at that location. A piecewise-linear threshold (hard-sigmoid in the idealized setting) yields binary flag:
Zon (r) 10,
FLAG, 1 NonZero,
(49)
For DECJZ, FLAG, 1 indicates the jump-on-zero branch.
21

---

Published as a conference paper at ICLR 2026
C.4 SIMULATING ONE STEP OF 2CM
We now describe how Fg implements the two instruction types.
C.4.1 INC
Suppose the fetched instruction (INC, r, target, next). Let Delim!" ( denote the delimiter mask for register r. Define
= Delim +
(50)
which is supported on {6, + 1}. Using (47 we overwrite:
at write valu-value and clear delimiter;
at 6 + 1: set delimiter and keep data-value O (or leave as 0).
All other positions remain unchanged.
The program counter advances deterministically:
PC2+1 NextMask((nstr.))
(51)
where (extMask((nstr.) is decoded from the fetched instruction embedding (a one-hot mask over
P).
C.4.2 DECJZ
Suppose the fetched instruction is (DECJZ, v, larget, next).
First compute the zero-test flag FLAG, {0, 1 as in $C..44
* FLAG 1 (zero case), we leave the register unchanged and jump:
PC TargetMask(Instr))
(52)
If FLAG, 0 (nonzero case), decrement by moving the delimiter left by one: define
Maeec
Delim,") (DeeliL
(53)
supported on {b- 1, b). and overwrite:
at b 1: set delimiter;
b: clear delimiter and write data-value
Then advance:
PC = NextMask((nstr.))
(54)
C.5 BINARY STABILITY AND ERROR CONTROL.
Because the construction relies on near-discrete masks (PC and delimiters), we include a stability mechanism prevent small numerical leakage from accumulating over arbitrarily many recurrent iterations.
After each step, we apply bistable contraction (e.g., hard-sigmoid with thresholds) on the dedicated mask channels:
0,
S
1
3
I1(=) = 3z
1,
3 <
(55)
1
>> 2.
ensuring that PC and delimiter masks remain within e-neighborhood of {0, 1}, and the one-hot constraints remain satisfied up to an arbitrarily small tolerance (guarantced by the cebttary-precistie assumption).
22

---

Published as a conference paper at ICLR 2026
C.6 CORRECTNESS OF THE SIMULATION
Let Step2aM denote the true one-step transition function of the 2CM on configurations encoded as (PC, Delim Delim data).
Lemma C.1 (Single-step correctness). There exist fixed parameters 0 such that for any valid en- coded 2CM configuration Xx, one application of Fg satisfies
Fo (X,, Ho) = Xx+1 = Encode (Step2cm (Decode(Xx))
(56)
and the net modification of the tape is supported on a constant-size set of positions (those adjacent to the active delimiter), while all other positions are preserved by residual identity paths.
Instruction fetch (45) selects exactly the active instruction. The INC/DECIZ updates are imple-
mented by masked overwrite (47) with masks supported on {b, + 1}. PC updates ( (54)
mask-based, thus remain one-hot after applying II(-). Residual connections preserve unmodified positions.
Lemma C.2 (Multi-step correctness). By induction on t, repeated application of Fo produces sequence {X1) that matches the 2CM execution trace step-by-step.
C.7 FROM 2CM TO TURING MACHINE
It is a classical result that a two-counter Minsky machine can simulate an arbitrary Turing machine (and vice versa). Therefore, for any Turing machine M, there exists 2CM program P that simulates M. We encode P into the read-only basis Hu on the program segment P, and encode the initial tape/state of M into the writable register segments R1, R2 (and auxiliary padding L().
If M halts, the corresponding 2CM reaches halting configuration. We can represent halting by dedicated HALT instruction that forces PC into an absorbing slot and/or sets halting flag channel to 1, after which Fe becomes the identity map on the writable state. The output module then read out the final configuration (e.g., via the logits at designated readout position).
Combining Lemma C.2 with the 2CM-TM equivalence yields Theorem C.I.
D ALGORITHMIC IMPLEMENTATION DETAILS
For clarity, this section provides the core algorithmic implementations of ChainGPT.
D.1 RWKV-PRODUCT
Given an input sequence H = [h, E BxTxC and the number of sub-steps M. Let the state matrix sequence be S {81, 82, 82,
Initialization:
80
For / = 1 to T:
0, t=l 1 h
14 14 1> 1
Ah h4 h4
2 h + Ale @ x
2 h + Ahe ④ 10
E h + Are O t
x h4 + Ahe O x
It h4 + Aht ④ S
J h Alt ④ 2
23

---

Published as a conference paper at ICLR 2026
R₂ W
waanhx/(ww)uru W3 = sofpplusssss-u + tanh((2 (ew )12)))))))))))))))))))))))))() 0.5
Kē Wa
V.- Wo
For j 0 to M - 1:
Re:
j M otherwise
Ttj =
(0
We,
j ==0 otherwise
wrd
00,
A AK = xx B)
AKī x ASA;
KhJ K. AKt.
Ki = K. + AKi
Bi daa. + (zf )
Bi oaa + (zf )
Ki, Kig 0 [1+ (Bi 1) kag
k2
|K
KY
2 Bh.
|
Ve, = V, + + A 1 1
Call the CUDA implementation:
X inter
CWW.CUD....KV/.. w. he v. b= . Hbesa)
Xa1 reshape (eshape(Xmeer, B, M.M. 1,
Xg GroupNorm(X.n)
term2, Σ (r (r K T
X' = Xmm + term2,
9: g(
Wo(X's ④ (g.))
24

---

Published as a conference paper at ICLR 2026
D.2 GUIDED SPARSE ATTENTION
Given a state sequence X E RBxxxx window size W, and global interval G.
Q XW K = V= XW,
= = 1:
0.01 t/10000
J = =i/(d/2)
B(f) ston + (Brast Bstow)
n 1+ B(J)
0 = 0/.0 /
Apply 0 to rotate-encode 42i. 92i+1 and k2 k2i+1.
Q' = RoPE(Q), K' RoPE(K)
Construct sparse mask Mij for i,j to S:
0,
i-j W and j S
Mú
0,
j mod G and j S
OC,
otherwise
If nvm_kv
heads hheads
H, expand K', V' to match the head dimension.
(Q'KTT
Attention(Q' K" M) = softmax
+ MM
V
Vd
Out concat heads W.
D.3 RECURRENT DEPTH APPROACH
Let the input sequence be e BxS and the hidden dimension be d. Embedding and Bottom-Level: H = Embed X) Bx5xd
Ho Bottom(H) BxSxd
Looped Iterations:
Z = X2; Hy] Bx8x2d
X, Z,Wa + by E BxSxx
X R(X,).
Two-Phase (no-gradient no steps + backprop ny steps) with truncation:
(No-Grad Phase)
X,+1 R( R([;; Ho Wa + ba), X,;
0,...,no no
(Truncation Boundary)
Xo = stopgrad (Xno))
Ne+1 cl (R(( NA: Ho + bal).
(Backprop Phase)
h: = 0,.., 1i 1,
(Output)
X* = -Nn-
25

---

Published as a conference paper at ICLR 2026
Top Layers and Normalization:
_ TTo((T))
Y Norm(X
Adaptive Inner-Loop Reasoning: After the t-th iteration, obtain the hidden state
Tt R Bx8xd
Decode logits by running only the top blocks:
logits, Decode(x;) E BXXXVV
where V the vocabulary size.
Extract the distribution at the last position:
& logits,/: -1, ER
BBV
Convert to probabilities:
Pt = softmax(41) E BBV
Compute the entropy for each batch element
H.(b) Σ (b) log (pr,(0) + +.),
10-10
Define the entropy difference:
AHl(b) Hb)+(()) H,(b),
where l is the checking interval.
Early-Stopping Criterion (entropy-diff):
(H) S
terminate early at step t,
where T is a threshold, either a constant or an adaptive function (e.g., =(1) = c/ VO).
E COMPARISON OF RECENT RNN MODULES FOR LANGUAGE MODELING (TABLE 6
F EXPERIMENTAL REPRODUCTION GUIDE
The code provided in the supplementary materials. For convenience, Table lists the main hyperparameters for ChainGPT models at different scales.
For MQAR task reproduction, wide-range time convolution needs be applied the RWKV- Product layer to ensure stability. This an optimization for training stability and does not affect related conclusions. The experimental setup follows Zoology (Arora al. (024).
You can calculate the total number of model parameters using the following method. Let V be the vocabulary size, D the hidden dimension, L. the number of decoder layers, with L., [L/2] even-indexed RWKV-v7 layers and L, = [L/2] odd-indexed GQA layers. Let Dm, be the MLP intermediate size, M the micro-step count, r the LoRA rank, DDEC, DGATE, DAAA the low-rank bottlenecks, and H₂, Hgv, dy, the query heads, key/value heads, and per-head dimension, with K Hg.dy, Non-layer terms are:
Pembed VD, Pim DV, Padapeer 2D= +D, Peore-nom 2D,
and per-layer shared components:
PaLN 4D,
PMPP = 3DDm.
26

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of recent RNN modules used for language modeling
Architecture
Stute Evolution
FP
HRR
GE
RWKY4
ReNN
8,
XWKVS
(--idigf of
XWKV-6 & GLA
S, S--1 expi expi-(mr (w)"1) 1 expia)) (an Oa)",,
HGRN-2
8. (iidiigfon)) of
St. (diggtoy) of
TTT
8. 5 4
Goaed DeizaNer
8. SL-1 a kf) (wa)
M. (1 a.
5. MiSt-a oz VVV
RWKV-7
8, 8y (digggl)) 48 (a Ri), of
DeltaProduct
H. [IT) (1 A.AA) Mi-1 Ej (TR Asalaadf)) Aughuols
x
x
8. = Ata)) B(rr), where
RWKV-Product (ours)
Ma) - 11.-0 (diaglora) - Ai
B(a) Egsr (Mazz Aea) (B
FP (Full-dim Params): all parameters match the model dimensionality, enabling adaptive channel-wise modulation.
DP (Decoupled Path): parameters for state transition and information injection are lincarly independent, improving expressiveness and convergence.
HRU (High-Rank Update):: state updates possess multi-substep high-rank structure, allowing single-step
update capture complex, diverse state changes.
(Generalized Figenvalue): state update matrix admits generalized eigenvalue factorization, enabling efficient diagonal-foor foom updates.
Table Main hyperparameters for different model scales
Parameter
180M
0.5B
1.5B
RWKV head dim
64
64
64
GQA head dim
128
128
128
Hidden dim
1024
2048
2560
KV heads
2
2
8
Layers
8
8
12
.loo_rank
64
96
128
Decay low rank dim
64
96
128
Gote_low_rg rank dim
128
256
320
M rank dim
96
128
256
Sub-steps
2
2
2
Recursive iters
12
16
Grad-enabled iters
8
8
8
Layer division strategies
C121
C121
C141
Window size
512
512
512
Anchor interval
64
64
64
RWKV-v7 attention parameters per layer are
PRWKY = 4D² + 2D(DDEC + DGATE) + DM + +2M + 11)D,
while GQA attention parameters per layer
POoa = 2D(H,dh) + 2DK + 2d,, (if Hydn = D, POos 2DD + 2DK +2d)).
The total trainable parameter count is therefore
Pioaa 2VD (2D2 + + L(3DD,, 4D) + L. PRWKV + Lo POOA
With weight tying, replace 2V by VD and adjust if biases added elsewhere.
27

---

Published as a conference paper at ICLR 2026
G LLM USAGE STATEMENT
In this work, LLMs were utilized exclusively for improving the clarity of the manuscript and en- hancing the code, and were applied to any other critical components.
28


Reviews:
[{"review_id": "AR3LHhvK4X", "reviewer": "Reviewer_mHtM", "summary": "This paper introduces ChainGPT, a dual-reasoning architecuture. In one block of ChainGPT, RWKV-Product achieves intra-layer communication, and SGSA achieves efficient long-range attention. The block design is aimed for \"internal multi-step reasoning\" (RWKV-Product) and \"sparse aggregation\" (SGSA). Across blocks, ChainGPT uses a recurrent paradigm to interatively refine internal states, with dynamic early stopping using entropy. Theoretical discussion and empirical experiments are conducted.", "questions": "1. **Hyperparameter Sensitivity (Weakness 3 Related)**: The SGSA module relies on window size $W$ and anchor interval $G$. Can you provide guidance on selecting these hyperparameters in practice?\n\n2. **Empirical Comparison with Closest Works (Weakness 4 Related)**: Could the authors provide experimental results or more detailed discussion comparing ChainGPT’s dual-reasoning approach to releated methods? What are the measured gains or trade-offs in similar settings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. **Clear Positioning**: The introduction provides a clear case for why fixed-depth Transformers and current architectural hybrids suffer from xpressive and computational limitations for deep reasoning.\n\n2. **Architecture Innovation with Theoretical Support**: This paper proposes a grounded achitecture innovation, including (1) intra-layer multi-substep reasoning using RWKV-Product, theoretically proven to expand representational power; (2) inter-layer recurrent refinement, with a theoretically motivated early stopping mechanism based on entropy.\n\n3. **Extensive Details**: Many ablations are given showing the effectiveness of each component, and details provided in the Appendix further improve the relia", "weaknesses": "1. **Concern on Component Integration**: ChainGPT appears to be an ensemble of several independent components, each building upon or modifying existing prior work. This approach could be argued to undermine the central academic contribution by suggesting the performance gains are primarily due to a complex engineering aggregation rather than a singular, fundamental architectural breakthrough.\n\n2. **Missing Discussion about Soft Thoughts**: The paper lacks a critical discussion comparing ChainGPT's approach to existing literature [1-3] that utilizes dense gist tokens (termed \"soft thoughts\"). These works also achieve reduced computational cost and perform reasoning via implicit states, making a comparison essential to fully delineate and underline the unique significance of ChainGPT's methodology.\n\n3. **SGSA Issue**: I'm a little concerned about the SGSA experiments. In Table 4, it seems that even a fully localized sliding window attention can achieve comparable PPL with long contexts, which raise doubts about the evaluation reliability. In practice, the selection of window size $W$ and anchor interval $G$ for all models are all set to 512 and 64, lacking empirical justification.\n\n4. **Experimental Issues**: Few related methods are compared with ChainGPT. Moreover, Since ChainGPT is framed as a \"dual-reasoning\" model, it is crucial that it be compared against SOTA CoT reasoning models. For example, Qwen3-1.7B can achieve an accuracy of $>0.8$ on the ARC-Challenge task, which significantly outperforms ChainGPT's $0.3$ accuracy, severely compromises the contribution of the proposed reasoning capabilities.\n\n> [1] Training Large Language Models to Reason in a Continuous Latent Space.\n> [2] CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation.\n> [3] LightThinker: Thinking Step-by-Step Compression.\n\n### Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l72ODe2ID0", "reviewer": "Reviewer_4WdW", "summary": "This paper\n- Proposes the ChainGPT architecture and introduces RWKV-Product. The architecture proposed can achieve good performance on reasoning tasks compared to Qwen.\n- Mathematically proved that the proposed architecture has superior expressivity and is Truing complete under ideal conditions.", "questions": "Q1: In Chapter 3, I do not find the definition of \"multiple sub-steps reasoning\".\n\nQ2: Can you calculate the number of params explicitly so that I can compare to the normal models?\n\nQ3: There is a reference error in line 252.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think the paper is sound provided the combination of math proof and supportive experiments.", "weaknesses": "W1: Though you mentioned Geiping's work in intro, I don't find any comparison between existing recurrent models and yours, which is hard for me to judge your contribution to recurrent models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pRYSAmnGig", "reviewer": "Reviewer_9THH", "summary": "The paper proposes ChainGPT, a dual-reasoning architecture that pushes reasoning from token generation into latent space via (i) RWKV-Product multi-substep state updates inside each layer and (ii) State-Guided Sparse Attention (SGSA) with sliding windows plus periodic anchor. It uses a recurrent-depth core with entropy-based early stopping across layers.", "questions": "* **SGSA complexity & memory**: Can you quantify the runtime and activation memory of SGSA vs dense attention across $T∈[4k,32k]$ and report end-to-end wall-clock with/without anchors, beyond the single-GPU microbenchmarks? Also include KV-cache implications.\n* **Efficiency Analysis**: You report a comparison of computation time. Could you provide a more comprehensive efficiency analysis that also covers training GPU-hours and inference latency?\n* **Early-stopping robustness**: The entropy-diff rule uses k and threshold τ. How sensitive are quality and compute to these hyperparameters across tasks?\n* **Ablation on RWKV-Product substeps and rank**: Can you include an ablation removing either the multi-substep (K) and the rank-M updates in RWKV-Product?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* **Clear architectural idea with theory hooks**: The dual mechanism (multi-substep “diagonal + rank-M” updates + sparse global anchors) is well motivated and analyzed (rank expansion, MQAR solvability, Turing-completeness under idealizations).\n\n* **Efficiency claims backed by complexity and ablations**: SGSA reduces attention cost to $O(T(W+T/G))$ and tracks full-global attention perplexity on PG-19 when using periodic anchors.\n\n* **Empirical improvements at matched scale**: On LM-Eval tasks, ChainGPT-0.5B/1.5B outperform Qwen2.5 models of the same size, respectively. The sub-step and recurrence ablations are thorough.", "weaknesses": "* **Novelty vs prior recurrent/hybrid work is somewhat incremental**: RWKV-Product extends RWKV-7 with LoRA-style multi-substeps. Many components (looped/recurrent depth, window+anchors) resemble existing hybrid archs. Thus, positioning versus models like DeltaProduct, Jamba/Samba, Mamba-2, and HRM could be sharper.\n\n* **Claims on “hard tasks” need stronger rigor**: The ARC-AGI-1 (38.6%), Sudoku-Extreme (54.4%), and Maze-Hard (77.4%) numbers are promising but depend on small (30M) models and bespoke setups; fairness vs large LMs and exact eval pipelines deserve more detail.\n\n* **Theoretical claims hinge on idealized assumptions**: Turing-completeness requires unbounded memory/steps and arbitrary precision; practical implications for finite precision and training stability remain unclear.\n\n* **Paper structure can be polished**: From my reading perspective, the organization of Sections 3 and 4 could be improved to better align with the proposed pipeline and enhance overall clarity. For example, these two sections could be merged into a single, cohesive section. In that structure, the current Section 4.1 could serve as an overarching overview, explaining where the chain-block fits within the entire pipeline and its overall function. Subsequent subsections could then provide a more detailed introduction and analysis of the chain-block itself.\n\n* **Illustrations can also be improved**: The current Figures 1, 2, and 3 each contain limited information, and presenting them separately leads to fragmented understanding. It would be more effective to integrate all three into a single, comprehensive pipeline diagram. Additionally, the current figures appear blurry.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)