Smooth Reading: Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Understanding
Authors: Kai Liu
Zhan Su
Peijie Dong
Fengran Mo
Jianfei Gao
Shaoting Zhang
Kai Chen
Venue: ICLR 2026 Poster (2026)
Keywords: RNN
Recurent LLM
Long Context Modeling
Large Language Model
Primary area: foundation or frontier models, including LLMs

Abstract: Recurrent large language models (Recurrent LLMs) offer linear computational complexity as efficient alternatives to quadratic self-attention-based LLMs (Self-Attention LLMs). However, Recurrent LLMs underperform on long-context tasks due to limited fixed-size memory. Previous research focused on architectural innovations to enhance memory capacity, but failed to match Self-Attention LLM performance. We argue this limitation stems from processing entire contexts at once being ill-suited for Recurrent LLMs. We propose Smooth Reading, a co-design of recurrent architecture and inference method. It introduces a end-to-end multi-round inference method that processes context incrementally and iteratively summarizes information, reducing memory demands.
Methodologically, we reveal architecture-inference interactions play an important role for performance, efficiency and scalability, shedding light on future Recurrent LLM design.
Besides, our method substantially bridges the performance gap between Recurrent and Self-Attention LLMs on long-context tasks while preserving efficiency advantages.
Smooth Reading boosts SWA-3B-4k from 5.68% lower to 3.61% higher performance than Self-Attention LLMs on LongBench, while maintaining 2.5× faster training and 2× faster inference at 64k context.

Published as a conference paper at ICLR 2026
Smooth Reading: BRIDGING THE GAP OF RECURRENT LLM TO SELF-ATTENTION LLM ON LONG-CONTEXT
UNDERSTANDING
Kai Liu
Zhan Su
Tongji University" & Shanghai Al Laboratory
Halmstad University zhan su@hh. se
liuka weckaii9980toooo 9988tongj edu.cn .n
Peijie Dong
Fengran Mo University of Montreal fengran. mo & umontreal.co ca
Hong Kong University of Science and Technology (GZ)
pdong212@connectt www.oonnnncccuuttttteooee hkust-gz cewnattgz.euuc.o. cn
Jianfei Gao & Shaoting Zhang & Kai Chen
Shanghai AI Laboratory
{gaojianfei, zhangshacting, chenkai][ppiab, org.
ABSTRACT
Recurrent large language models (Recurrent LLMs) offer linear computational complexity efficient alternatives quadratic self-attention-bart LLMs (Self- Attention LLMs). However, Recurrent LLMs underperform on long-context tasks due to limited fixed-size memory. Previous research focused on architectural in- novations to enhance memory capacity, but failed to match Self- Self-Attention LLM performance. We argue this limitation stems from processing entire contexts once being ill-suited for Recurrent L.MMs. We propose Smooth Reading, co- design of recurrent architecture and inference method. introduces end-to-end multi-round inference method that processes context incrementally and iteratively summarizes information, reducing memory demands. Methodologically, we reveal prchitecture-inferenne interactions play important role for performance, effi- ciency and scalability, shedding light on future Recurrent LLM design. Besides, our method substantially bridges the performance gap between Recurrent and Self-Attention LLMs on long-context tasks while preserving efficiency advantages. Smooth Reading boosts SWA-3B-4k from 5.68% lower 3.61% % higher perfor- mance than Self-Attention LLMs on LongBench, while maintaining 2.5x faster training and 2x faster inference 64k context.
INTRODUCTION
-tttettttionbased large language models (Self-Attention LLMs) have demonstrated strong long- context understanding capabilities (Bai et 2024 Hsieh et 2024). However, applications
increasingly require longer contexts-such as complex reasoning (DeepSeek-Al & et al. 2025). embodied agents (Zhang et al. 2024a). and research (Zheng et 2025 the quadratic computational cost of self-attention becomes a major barrier to scaling, In response, Recurrent LLMs have re-
emerged promising alternative (Peng et al.; Gu & Dao; Yang al. 2025 2024 et al.), offering linear-time computation and constant memory.
Sun et al.) Yang
We categorize mature LLM use cases by context length into: (1) general chat (short context), (2)
long chain-of-though (CoT) reasoning (long output), and (3) long-context understanding (long
input). Recurrent LLMs match Self-Attention LLMs on short-context tasks (Yang et al. 2025 2024 Yang et al.; Dao & Gu) and long CoT reasoning (Zhao et al. 2025), but lag substantially behind on
Kai Liu is with the Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University.

---

Published as a conference paper at ICLR 2026
User
User
Text
User
Memory
Pasa Text
Chunk 0
<heado Chunk 0
Summar Updan
Re fed
Load Menrryy
<Peads Chunki
<Rads Churk
Summary
Burrmary and Updans
Summery
Load Merrary
alleads Chunk
Summary
Bummary and Uodater
Answer and <Stop
Answer and <Stop..
(a)
(b)
(c)
Figure 2: Comparison of inference methods. (a) One-Round inference (OR): process the entire
context in a single pass; (b) Non-End-ae End-to- End Multi- inference (NMR): chunked processing with summaries re-inserted into the prompt; (c) End-to-End nouti-Round inference (EMR): iterative chunk reading with hidden memory updates preserved across rounds.
long-context understanding Yang et al., 2024 Waleffe et al., 2024 Liu et al. 2025). This gap is primary obstacle to the wider adoption of Recurrent LLMs.
Most prior work attempts to close this gap through architectural changes that increase memory capacity (Peng et al.) 2024 Yang et al. 2025: Peng et al., 2025 Dao & Gu,
Qin et al. (unn et al.; Liu al. 2025), such more expressive update rules Yang et al. 2025) larger state sizes (Liu et al.,
2025). Despite progress, these approaches alone have not matched the performance of Self-Attentio
LLMs. A key reason is the misalignment between architecture and inference: traditional
(OR) inference (Figure 2 processing the entire context in a single mass-demands much larger
memory than Recurrent LLMS can provide.
We address this limitation with Smooth Reading, a co-design method to optimize both architecture and inference method jointly for Recurrent LLMs. Our approach consists of three main components: (1) A tailored EMR procedure for Recurrent LLMs that processes long inputs chunk- blunk-by-chunk. After reading each chunk, the model produces a short con- textual summary and updates its hidden state, pre- serving and refining hidden memory across rounds. This compresses information into a local working window and avoids overloading fixed-size memory. Thanks to the linear efficiency of Recurrent LLMs, the Multi-Roun overhead remains manageable and still more efficient than Self-Attention LLMs. (2) An architectural perspective showing that different inference strategies favor different properties: while OR prefers maximal single-pas, paggle-pas memory capacity, EMR benefits more from strong length extrapolation (length generalization). It demonstrates the impor- tance of inference method in architecture design. (3) joint analysis of architecture and inference that yields practical guidelines for balancing performance and efficiency, enabling optimal co-design.
100
8 SWA-3B 4K-EMR
Qwenn-3--RR
Accuracy (%)
50
100% gap reduce
=20% overhead
750
1000
1250
1500
Inference Time (s)
Figure 1:
Inference Time vs. Accuracy on
NIAH at 64k. Applying EMR to Recurrent LLM (SWA-3B-4K) significantly closes the gap to Self-Attention LLMs (Qwen-2.5-3B- OR) while incurring only a 20% overhead over OR inference, and remains about 2x faster than Self-Attention L.Mss.
We evaluate architecture-infent inference combinations on LongBench (Bai et al. 2024). Needle-in-a-
Haystack (Hsieh et al. 2024). RULER (Hsieh et al. 2024), and HELMET (Yen et al. 2025
Smooth Reading provides a favorable accuracy-efficienn trade-off. As shown in Figure I Smooth Reading significantly narrows the performance gap between Recurrent LLMs (SWA-3B- -WWA-3--4-EBB 4K-EMR) and Alteetttenion LLMs ( (Qwen-2.5 2.5-BBB5-B-O on long-context tasks while incurring only a 20% over- head over inference (SWA-3h-4k-00) and remains about 2x faster than Atte-Atttntoon
2

---

Published as a conference paper at ICLR 2026
counterparts. On real-workd long-context tasks, Smooth Reading boosts SWA-3B-4K from 5.68% lower to 3.61% % higher performance than Self-Attention LLMs on LongBench. These results indicate the significance of architecture inneurencee co-design for Recurrent LLMs.
In summary, our contributions are:
While Recurrent LLMs suffer long-context understanding tasks, our Smooth Reading method substantially closes the performance gap with Self-Attention LLMs while preserving efficiency advantages. significantly pushes the boundary of Recurrent LLMs and makes them more practical for real-world applications.
We demonstrate the importance of grchitectur-iinfeeee co-design, showing how their interaction governs performance, efficiency, and scalability, and provide actionable guidelines beyond architecture--nnl improvements.
RELATED WORK
Most prior work improves either model architectures or inference strategics isolation. Inference is typically designed around Self- Belf-Attention LLMs, whereas new Recurrent LLMs are often evaluated under traditional
inference, leading to a mismatch between architecture and inference.
2.1 SELF-ATTENTION LLMS AND INFERENCE STRATEGIES
Self-Attention LLMs. Self-attention Vaswani 2017 underpins most modern LLMs (DeepSeek-A]
& ct al. 2025; Owen et al. 2025 Achiam et al. 2023 By attending over all preceding tokens, these models can, in principle, use arbitrarily long context. However, the cost scales quadratically in computation and linearly space with input length, making very long contexts inefficient.
One-Round inference. The standard approach for Self-Attention LLMs is single forward pass over the entire input (Figure ().. This aligns with their ability to attend to all prior tokens, but incurs O(L²) computation and ( space, where L is the input length, limiting scalability.
End-to-End Multi-Round inference. End-to-End Multi-Round inference methods split the input into chunks and iteratively process them, often appending intermediate summaries back into the prompt (Figure 2.). This increases the effective context to XL tokens (A > 1), yielding (((L)) computation and O(L)) space. For Self-Attention LLMs, this compounds the already high complex- ity.
Von-End-to-Eme Multi-Round inference. Methods such prompt compression chain-of-thought prompting (Zhang et al. [2224b] Yoon et al., 2024 Qian et al., 2024 Lee et al. 2024 Chen et al.
2023) discard hidden states and re-feed compressed context at each step to cap the per-step length (Figure 2p). This yields (AL) computation and ((1) memory. However, discarding hidden states causes information loss and can degrade performance.
Table summarizzs these strategies for Self-Attention LLMs. Despite its quadratic cost, One-Round inference remains prevalent due to its simplicity and strong performance. End-to-Ent Multi-Rouu inference is rarely used with Self-Attention LLMSs long-context tasks due to its high complexity.
2.2 RECURRENT LLMS
Recurrent L.Ms use fixed-size memory, enabling linear-time and constant-space inference regardless of input length. This efficiency arises from architectural choices such as linear-attention variants or gliding-window mechanisms (Beltagy et al. 2020). However, most recurrent models are still used with standard One-Round inference, which can overwhelm their fixed memory on long-context tasks and limit performance. Recent efforts target these limitations by (1) increasing memory capacity (Qin al.; Gu & Dao Dao & Gul Du et al. 2025) or (2) improving memory efficiency (Yang et al. 2025 Sun et al.j] Yang et al. 2024]. While helpful, Recurrent LLMs still trail Self-A Atff-Attetionn LLMs on tasks
requiring long-context understanding.
3

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of architecture--infreni combinations. "C" and "S" denote computational and space complexity. "L" is the input length. "X" (A > 1) is scaling factor for the effective context length in Multi-Round settings. "OM" indicates whether the inference strategy overwhelms the model's memory. "HM" indicates whether hidden memory is preserved during inference. Both memory overwhelm and discarding hidden memory degrade performance. The main drawback of each combination is underlined. Our Smooth Reading (End-to-Hee to--ndd Multi-Round inference with Recurrent LLMs) achieves the best balance between performance and efficiency.
Architecture
One-Round
Non-End-te End Multi Nuuti-Round
End-to-Enn Multi-Round
C: C(O()); S: O(L)
C: O(L); S: O(1)
C: O(((L))));; S: O(A))
Self-Attention
OM: No; II::
OM: No; HM: No
OM: No; HM: Yes
C: O(L): S: O(1)
C: O(L);; S: O(1)
C: O(L);; S: O(1)
Recurrent
OM: Yes; HM:
OM: No; HM: No
OM: No; HM: Yes
2.3 ARCIITECTURE-INTEEENG CO-DESIGN
The interaction between architecture and inference is seldom examined, yet is crucial for Recurrent LLMs with fixed memory. We analyze this interplay and propose an inference method tailored Recurrent LLMs, leveraging their strengths while addressing their constraints. We demonstrate the interaction of architecture and inference significantly influences performance. efficiency, and scalability, A concurrent line of work, OPRM (Ben-Kiss et al., 2025). adapts aerieval-augmentee generation to mitigate the memory overflow problem in lincar-attention architectures. This highlights the importance of inference design for Recurrent L.LMs. However, OPRM does not systematically study architecture-inferene co-design, which can limit performance and generalization.
METHODOL.OGY
The interaction between model architecture and inference strategy is critical for long-contex, per- formance. We first review the characteristics of Recurrent LLMs, then introduce Smooth Reading, which comprises:
1. An End-to-End Multi-Round (EMR) inference procedure tailored to Recurrent LLMs;
2. Guidance for selecting recurrent architectures compatible with EMR;
3. A co-design approach that jointly optimizes efficiency via architectural and inference choices.
3.1 CHARACTERISTICS OF RECURRENT LLMs AND INFERENCE STRATEGIES
strengths and limitations. Recurrent LLMs process each token with constant computation and memory, enabling efficient long-context handling. However, their fixed-size memory limits the amount of information retained at any moment. The central challenge is the trade-off between efficiency and performance on long-context tasks.
Recurrent LLMs:
Drawbacks of One- RounR inference. With One-Round inference, Recurrent LLMs enjoy O(L) compute and O(1) memory complexity, but processing the entire context in a single pass can overwhelm their limited memory, causing notable performance degradation.
Advantages of End-to-End Multi-Round inference. EMR enables incremental, selective processing.
By reading in steps, the model can retain only the most relevant information, avoiding memory overload. Although EMR introduces token-generation overhead, yielding O(AI)) compute and O(1) memory complexity, it remains far more efficient than the quadratic complexity of Self-Attention LLMs with Round inference. Crucially, Recurrent LLMs' constant-size hidden memory can be preserved and updated across rounds at extra memory cost, which leads to superior performance compared to Self-Atte Attentionn LLMs using Non-End-to End-to-Ent End Multi- -outd inference (which discards hidden states between rounds).
In summary, Smooth Reading combines Recurrent LLMs with EMR to preserve efficiency while strengthening long-context understanding. As summarized in Table One-Round inference best
4

---

Published as a conference paper at ICLR 2026
Context 1/3: Answer the question based on the given passages. Only give me the answer and do not ourput other words.
? Question: What was population of the city where the Dutch retired footballer Fooke Booy bom?
The following are given passages:
Leeuwarden Stadsfries: "Liwwadden" city and municipality with population 108, in Friesland the Netherlands. It is the provincial capital of the States of Friesland...
Target: What was population of the city where the Dutch retired footballer Focke Booy bom??
Reason: need find a city where the Dutch retired foothaller Foeke Booy horn
population of that city. The given passage mentioned that Lecuwarden city has population 108,249,
This information may be useful later.
Clues: 1) eeuwardenn city has population of 108,249.
Action: Analysis complete, but no answer found yet. <READ>
Context 2/3: ...He is currently a youth coach, managing the D2 squad (under-13 3) at Almere City FC. Foeke Booy (born 25 April 1962 in Lecuwarden) is a Dutch retired footballer and football manager...
Target: What was population of the city where the Dutch retired footballer Foeke Booy born?
Reason: This passage mentioned that Focke Booy was born Leeuwarden, and we have known the population Leuwarden is 108,249. Therefore, the population of the city where the Dutch retired foothaller Foeke Booy born is 108,249.
Clues: 1) Leeuwarden city has population of 108,249. Forke Booy was born in Leeuwarden.
I
Action: The final answer is 108,249, <STOP>
Figure 3: Inference Example of Smooth Reading. White boxes indicate the information gathered by the model and the decision points for whether to continue reading. Blue boxes represent chunks of the query context.
suits Self-Attention LLMs, whereas EMR preferable for recurrent architectures, underscoring the importance of architecture-inference co-design. Recurrent LLMs with EMR also offer the best balance between performance and efficiency among these strategies.
3.2 END-TO-END MULTI-ROUND INFERENCE FOR RECURRENT LLMS
End-to-End Multi-Round agentic pipeline. Our EMR procedure proceeds in rounds: at each step. the model takes an action (including <READ> and <STOP>) and receives observation, iterating until it decides that enough information has been gathered. This stepwise process avoids memory overload and ensures that relevant context remains locally available (sce Figure (2b).
Chunked reading to avoid memory bottlenecks. To respect recurrent memory limits, we partition the input into manageable, semantically coherent chunks processed sequentially. Each step introduces only a small amount of new information, reducing overflow risk and encouraging focus on salient content. Models advance by emitting <READ> action. Specifically, hierarchical chunker driven by a prioritized delimiter list. At each level, the context is split (e.g.. first by paragraphs using " "\n"".. then adjacent units are merged until maximum chunk size is reached. If a unit exceeds the limit, it is recursively split using finer- mner-grained delimiters (c.g., sentences via "."), and merging repeated. In practice, we apply the following delimiters in order: "nn'nnin "" ")n and (space), yielding size-bounded chunks with
a lightweight, rule-based
"nn", " " an "! un reasonable semantic boundaries. To avoid tokenization overhead, we estimate chunk length using Tokeen ~ Int(1.5 x ), based on the empirical average of ~ 1.5 tokens per word.
Contextual summary compression. After reading each chunk, the model produces a contextual summary that distills essential information. These summaries (1) represent key content with fewer tokens where possible and (2) update the hidden state so that recent, relevant information prioritized in recurrent memory. As illustrated in Figure 3 each contextual summary contains: 1) Target: the task objective, to prevent distraction from irrelevant context; 2) Clues: salient information relevant to solving the task, such accumulated summaries (for summarization) query-related facts (for QA); 3) Reason: brief rationale for updating clues at the current reading step. These elements ensure that the model can answer the query using focused, relevant context.
Preserving hidden memory throughout inference. Unlike Self-Attentn LLMs, Recurrent LLMs maintain fixed-size hidden memory regardless of context length. EMR retains and incrementally
5

---

Published as a conference paper at ICLR 2026
updates this memory across rounds, accumulating both recent and long-rarn -aaggraage information without additional memory cost.
Early stopping mechanism. After each chunk, the model decides whether it has sufficient infor- mation to answer the query. If so, it stops reading and produces the answer with a <STOP> action; otherwise, it continues. This adaptive early stopping conserves computation.
An algorithmic summary of our EMR is provided in Algorithm for reference.
3.3 ARCHITECTURAL CONSIDERATIONS FOR RECURRENT LLMS
Different inference strategies favor different architectural properties: One-Round inference benefits from architectures with maximal effective memory, since all information must be retained simul- taneously. End-to-End Multi-Round inference (Smooth Reading) relies more on robust length generalization and dynamic compression than large single-pass memory. The emphasis shifts from raw memory capacity to length extrapolation beyond the training regime.
Two principal recurrent families are: (1) sliding-window window attention models (Beltagy et 2020), and (2) linear-attention models (Katharopoulos et al., (2020). While linear-attention approaches can offer larger effective memory under One-Round inference (Liu al. 2025 Yang et al. 2025 Peng et al.) 2025), sliding-window window attention exhibits stronger length extrapolation under EMR. Our experiments indicate that sliding-window window architectures are better suited for Smooth Reading at extreme sequence lengths.
3.4 CO-DESIGN: OPTIMIZING EFFICIENCY
Efficiency emerges from the joint choice of architecture and inference. We analyze how EMR maintains linear complexity and how to optimize wall-clock time via co-design. Let the per-token wall-clock times for prefilling and decoding he functions of model complexity s: p(s) and d(s), respectively. These scale with s and are independent of context length. For inference, define chunk size number of chunks n, and total context length L decoded tokens per round, assumed independent of L. The total inference time is
n. Let g be the average number of
T Recurrent = n1 p(s) + n d(s) = L (p(s) + d(+))
Linear complexity: For fixed 8, and g. Recarrentt EMR scales linearly with L. Efficiency opti- mization: Efficiency depends jointly on model complexity and chunk size e wall time increases with s but decreases as grows (larger chunks reduce the number of rounds and, hence, decoding steps). However, larger requires greater effective memory. Optimal efficiency therefore requires co-designing architecture and inference method. In practice, we jointly tune the window size (control- ling memory span and effective complexity) and the chunk size to optimize sliding- window models under EMR
3.5
DATASET CONSTRUCTION
To train models for Smooth Reading, we construct supervised fine-tuning dataset covering several long-context tasks (e.g., QA). For each task, we:
1. Collect raw data: use standard training splits containing query, answer. and context;
2. Generate summaries: We generate stepwise contextual summaries and final answers using either a rule-based method or a state-of-the-at LLM (DeepSeek-V?) V3 (et al.) 2024)). selected based on task complexity. Rule-based generation is applied to simpler tasks (e.g.. retrieval), whereas the LLM is used for more complex tasks (e.g., summarization and question answering). For LLM-based generation, we adopt a Non-End-to-Ent Multi-Round pipeline for efficiency and use one-shot prompting with manually crafted examples to guide contextual summary generation. The prompt template is shown in Appendix A. After each contextual summary, we append <READ> token if reading continues; otherwise, append a <STOP> token. Maximum chunk sizes are varied during summary generation to enhance robustness, ranging from 128 to 4096 tokens.

---

Published as a conference paper at ICLR 2026
Table 2: Results on LongBench. "Infer" indicates the inference method: "OR" (One-Round), "EMR" (End-to-End Multi-Round), and "NMR" (Non-End-to- -nd Mulni-Round). Best results in bold.
Architecture
Infer
Model
SQA
MQA
Summary
FowShot
Synthetic
Codc
Avg
Qwen-2.5-3B-OR
24.20
41.25
30.22
65.89
56.75
66.00
47.38
Self-Attention
NMR
Qwcn-2.5-3B-NMM
31.54
42.17
21.91
68.96
61.50
64.13
48.37
RWKV-7-3B-OR
16.96
11.39
29.16
67.72
60.50
62.84
41.43
OR
SWA-3B-4K-
16.43
26.14
26.96
66.43
48.00
66.22
41.70
Recurrent
RWKV-7-3B-EMP
28.87
40.02
28.23
65.90
65,25
59,92
48.03
EMR
SWA-3B-4K-FMR
30.46
47.67
26.27
69.60
66.75
65.18
50.99
Table Results on the Needle-in-al -aa-a--ayytack (NIAH) benchmark. All models are trained with a 32k context length and evaluated at context lengths ranging from 8k to 256k.
Architecture
Infer
Model
8k
16k
32k
Avg
64k
128k
256k
Avg
Owen-2.5-3B-00
98.
98.60
97.00
98.13
98 08.00
26.00
0.00
41.33
Sclf-Attention
NMR
Qwen-2.5-3B-NM
87.80
87.00
92.60
89.13
93.80
95.00
67.20
85.33
RWKV-3B-OR
98.40
95.80
86.60
93.60
39.00
8.60
0.00
15.87
OR
SWA-3B-4k
53.60
22.40
29.20
6.80
1.60
Recurrent
RWKV-3B-ER
99.40
98.80
97.20
98.47
75.20
20.60
1.94
32.58
EMR
SWA-3B-4k
100,00
99.93
100.00
99.80
99.60
99.80
3.
Enable early stopping: for tasks amenable to early stopping (e.g., QA), allow the generator to halt reading once sufficient information has been found;
4. Clean the data: apply rule-based filters remove low-quality outputs.
In total, the dataset comprises 48,856 items. We use this dataset to train models to produce contextual summaries, embedding Smooth Reading behavior directly into the model. The configuration of our dataset construction is summarized in Table 5
4
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We systematically compare model architectures and inference methods to assess the effectiveness of Smooth Reading. We evaluate four configurations:
Self-Attention LLM with One-Round inference (OR).
2. Self-Attention LLM with Non-End-to-Enn Multi-Round inference (NMR).
Recurrent LLM with One-Round inference (OR).
Smooth Reading (ours): Recurrent LLM with End-to-FE to-End
inference (EMR).
Our experimental pipeline comprises two steps:
Base Model Preparation:
For Self-Attention LLMs, use Qwen-2.5 (Qwn al. 2025)
Window LLMss: Based on Qwen-2.5. modified to use sliding- window attention mechanism (denoted "SWA-rk," where is the window size; 4k tokens unless otherwise specified). The conversion process is detailed in Appendix A.1.2 2) RWKY-7 (Peng al., 2025): strong Recurrent L.LM employing variant of linear attention.
representative model. For Recurrent LLMs, we consider: 1) Sliding-Window
Model Training and Evaluation: We derive three dataset variants tailored to each inference method (OR, NMR, EMR), and use them to train all models indell--iclluiig Self-Attention LLM (Qwen-2.5-3BB Sliding-Wii Window LLM (SWA-3B-44 and RWKV-7 (RWKV-7-3B) under identical training settings for fair comparison. The inference method is appended to each model name (e.g.. Qwen-2.5-33 2.5-3B-OR OR, SWA-3B-44 4k-EMR). Further training and evaluation details are provided in Appendix A.1.3 and Appendix A.1.4
7

---

Published as conference paper at ICLR 2026
1500
Qwen-3B-OR
Qwen-3B-OR
10
Training Time (s)
Inference Time (s)
SWA-3B-4k-EMB
1000
SWA-3B-4K-EMRR
500
0
4 8
32 Context Length (k)
64
4
8
16
32
64
Context Length (k)
(a) Training efficiency
(b) Inference efficiency
Figure 4: Efficiency vs. Context Length. Recurrent LLMs with Smooth Reading scale linearly; Self-Attention LLMs scale quadratically. "+ES" indicates early stopping. The training speed of WWA-3B-4k-EMMA is about 2.5x that of Qwen-2.5-3B- 64k. At 64k, SWA-3B-4k-ENa is about 2x faster than Qwen-2.5-3B-OO during inference, improving to to 4x with early stopping.
4.2 LONG-CONTEXT PERFORMANCE COMPARISON
We evaluate LongBench (Bai et al., 2024 and NIAH (Hsieh al. 2024) (Tables and For NIAH, we report both in-distribution lengths (8k-32k, matching the 32k training maximum) and extrapolated lengths (64k-256k).
One-Round Inference: Recurrent LLMs underperform relative to Self-Attention LLMs. Qwen- 2.5-3B-OR averages 47.38% LongBench: RWKV-7-3B-OR and SWA-3B-4K-OR are below 42%. On NIAH within the 32k training limit, Qwen-2.5-33 2.5-------B-O reaches 98. 13%, outperforming recurrent baselines and highlighting the mismatch between One-Round inference and fixed recurrent memory.
End-to-End Multi-Round inference (Smooth Reading): EMR substantially improves Recurrent LLMs. On LongBench, RWKY-7-3B-EMMS and SWA-3B-4K-EMME reach 48. 48.03% and 50.99%, surpassing Qwen-2.5-3HB and SWA-3B-4K EMR match or exceed Qwen-2.5-3B-OR; notably, SWA-3B-44 4M--MMRK---R------ maintains near-perfect accuracy up to 256k tokens.
by 0.65 and 3.61 points, respectively. On NIAH, RWKV-3B-EMR
End-to-End vs. Kon-End-to-Eii Multi-Round: Recurrent LLMs with EMR outperform Self- Attention LLMs with NMR by preserving hidden states across rounds. On LongBench, SWA-3B- 4k-EMR exceeds Dwwe-22.53B--NM. by 2.62 points. On NIAH, NMR is less stable and generally worse than Recurrent LLMs with EMR.
We conduct additional studies in Appendix A.2
Scaling to 7B models (Appendix A.2. yields consistent gains with Smooth Reading.
Comparisons with additional Self-Attentinn L.LM baselines (e.g., Llama3.1 (Dubey al. 2024): Appendix A.2.2) further validate the effectiveness of our method.
Evaluation on out-of-distribution tasks from RULER (Hsieh et al. 2024) and HELMET (Yen) jtt al.] 2025) (Appendix A.2.3). including tasks unseen during training, demonstrates strong generalization.
Comparisons with additional inference methods RAG, CompACT (Appendix A.2.5) and OPRM (Appendix A.2.4 show consistent advantages.
4.3 EFFICIENCY COMPARISON
key advantage of Recurrent LLMs is linear computational complexity in context length, compared with the quadratic cost of Self-Attention LLMs. We report training the Sliding- Window LLM; RWKV-7 omitted from the efficiency plots due to the lack of a highly- optimized inference engine for fair comparison.
and inference efficiency using
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study on window size (W) and chunk size (C) for accuracy (%) and inference time.
(a) Accuracy
(b) Inference Time (Seconds)
C
512
1024
2048
4096
512
1024
2048
4096
W
512
97.0
77.8
0.0
0.0
512
528
457
327
100
1024
99.2
89.0
26.4
12.6
1024
537
444
374
364
2048
100.0
100.0
95.4
31.2
2048
577
471
387
343
4096
99.8
99.8
100.0
83.4
4096
646
505
423
366
Training efficiency: Sliding-Wined Window LLMs train substantially faster than Self-Autention L.Mss. As shown Figure 4a 64k context length, SWA-3B-4k SEAA-B--kkEERE 10--EKMMMRR is 2.5x faster than Qwen-2.5-3B-OK
Inference efficiency: We evaluate inference speed on NIAH. Sliding-Window Window LLMs with Smooth Reading slightly slower than Self-Attention LLMs at short lengths due to overhead, but scale linearly and become advantageous as the length grows. At 64k, 11MA-3B-4k-EMA is 2x faster than Qwen-2.5-3B (Figure (b).
Early stopping: Our method supports halting once sufficient information is found, further reducing latency. At 64k, SWA-3B-4k --- improves from 2x faster to up to faster than Qwen-2.5- 3B-OR. An ablation in Appendix 4.2.66showe that early stopping does not compromise performance.
Inference time vs. accuracy: We compare inference time versus accuracy across architecture- inference combinations in Figure [ SWA-3B-4k-Ei closes the performance gap Qwen-2.5-3B- OR with only a 20% increase inference time over SWA-3B-4K-OR, while remaining faster than Qwen-2.5-3B-OR
ANALYSIS AND ABLATION STUDY
5.1 LENGTH EXTRAPOLATION AND SCALABILITY
We evaluate length extrapolation up to 256k tokens on the NIAH (Hsieh et al. 2024) benchmark, as shown in Table 3)
All models trained with 32k context length and tested substantially longer
sequences.
Under One-Round (OR) inference, models such Qwen-2.5-3B-OR perform well within the training length but collapse at long contexts (0% at 256k). The Non-End-to-Ean Multi-Round (NMR) strategy extrapolates better than One-Round (OR), yet still degrades at 256k and is unstable because it does not preserve hidden memory.
Self-Attention LLMs:
Sliding- Window LLMs:
With One-Round (OR) inference, performance declines as context length grows due to limited memory capacity (c.g., SWA-3B-4K, 3B-4k-OR reaches 53.6% at 8k). In contrast, with EMS, scalability is excellent, achieving 99.6% 256k.
RWKV: With One-Round (OR) inference, RWKY is strong within the training range (86.6% at 32k) but degrades rapidly beyond With EMS, RWKV-3B-ENT reaches 75.2% 64k, then declines on longer sequences.
Figure compares length extrapolation for SWA-3B-4k-00 and RWKV-3B-OR while keeping the needle's 's relative position fixed. SWA-3B-4K-OR has limited local memory (it can only retain content within its 4k window) but exhibits stable length extrapolation, maintaining performance beyond the training length. RWKV-3B-OR can remember much longer spans within the training range, yet its performance drops sharply beyond that. Thus, One-Round (OR) inference favors RWKV's longer memory, whereas Smooth Reading benefits more from SWA's stronger length extrapolation. These results underscore that different inference methods call for different architectural properties.
Comparison of SWA and RWKV:

---

Published as a conference paper at ICLR 2026
100
550
Inference Time
Accuracy (%)
75
SWA-4K
500
SWA-4K 5k
50-
RWKV7
RWKV7
450
25
0.5k/Ik 1k/2k 2k 4k/ 8k 8k/ 16k
8 633
64
128
256
Chunk Size vs. Window Size
Context Length (k)
(a) Inference time with fixed chunk-to-window ratio
(b) Length extrapolation comparison on NIAH
Figure 5: (a) Inference time with a fixed chunk-to-window ratio of 1:2. (b) Length extrapola- tion comparison of Sliding-Window LLM (SWA-4k) and RWKV-7 NIAH. Curves labeled "[ "[arch)-[neede position)" indicate different models and the relative position of the searched token. SWA-4k exhibits stronger length extrapolation, while RWKV-7 provides longer memory within the training context but limited extrapolation.
5.2 IMPACT OF CO-DESIGN ON PERFORMANCE AND EFFICIENCY
We ablate the window size (model memory span) and chunk size (processing granularity) for Sliding- Window LLMs NIAH. Results summarized in Table []
Larger chunks reduce the number of processing rounds and speed up inference, but the chunk size exceeds the window size, accuracy collapses (e.g., W=512, C=4096 yields 0%). With a sufficient window, increasing the chunk size from 512 to 4096 markedly reduces time (e.g., 646s to 366s W-4096).
Chunk size:
Window size: Larger windows retain more information and permit larger chunks, improving accuracy (e-g., at C=2048, accuracy rises from 0% at W=512 100% at W=4096), but they increase per-step computation and thus inference time for fixed chunk size.
Interaction between chunk and window sizes: Joint tuning is essential: 1) For high accuracy, the chunk size should not exceed the model's window (hidden memory) size. 2) For efficiency, moderate ratio (c.g., chunk:window 1:2) balances fewer rounds against manageable per-step complexity.
Fixing the chunk-to-winden towwwodowwwwwooooooooooooo ratio 1:2 and scaling both, inference time follows a U-shaped trend (Figure 5a): very small or very large sizes are inefficient, whereas moderate settings (c.g., W=4096 or W=8192) are fastest. These findings highlight the importance of co-designing architecture and inference: balancing model complexity (window size) and processing steps (chunk size) is critical for both efficiency and performance of Recurrent LLMs on long-context tasks.
CONCLUSION
Recurrent L.Ms have underperformed on long-contess tasks primarily because One-Round inference is misaligned with their fixed-size memory. We make two contributions: (i) an End-to-End Multi- Round (EMR) inference strategy that processes long inputs incrementally and substantially improves Recurrent LLMs, demonstrating that better inference can overcome apparent architectural limits; and (ii) evidence that inntteccttre-ffeeeneee co-design, rather than architecture alone, is critical for accuracy, scalability, and efficiency. Together, these findings reveal the untapped potential of recurrent architectures and establish co-designed paradigm for long-coo cong-context language modeling. Last, discuss limitations and future directions in Appendix A.3
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research project was supported by Shanghai Artificial Intelligence Laboratory.
ETHICS STATEMENT
We declare that there are no conflicts of interest that could inappropriately influence this work. Our study does involve human subjects, the collection of personal data, experiments involving protected groups All datasets used in this work are publicly available and widely adopted in the research community, such the LongBench and Needle beedle-in-a-Haystae benchmarks. These datasets have been carefully curated to minimize biases and ethical risks.
REPRODUCIBILITY STATEMENT
We provide comprehensive implementation details in Section 3 Section 4.1 and Appendix | including our method, dataset construction, training. evaluation protocols, and all hyperparameters. to ensure full reproducibility. In addition, we will release our code and datasets upon publication to facilitate independent verification and further research.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 1-Xix:2303.08771 2023.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reftection. In The Twelfth International Conference Learning Representatiinns 2023.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: bilingual, multitask benchmark for long context understanding, 2024.
Iz. Beltagy, Matthew F. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. URL BtLttttt://arxx org http//abb/20..0000 05150
Assaf Ben-Kish, Itamar Zimerman, M. Jehanzeb Mirza, James Glass, Leonid Karlinsky, and Raja Giryes. Overflow prevention enhances long-context recurrent Ilms, 2025. URL https://
www.orrggabb//5555.0
Howard Chen, Ramakanth Pasunuru, Jason Weston, and Asli Celikyilmaz. Walking down the memory maze: Beyond context limit through interactive reading, 2023. URL hattps://arxiv..og/ abs/ 05029
LMDeploy Contributors. Lmdeploy: A toolkit for compressing, deploying, and serving 1lm.
https
//github.com/ www..ooommnneeeii////. Imdeploy. 2023a.
XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning 1lm. https://github.c com/ InternLM/xtuner 2023b.
Tri Dao and Albert Gu. Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of informnation-eekkin questions and answers anchored research papers. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4599-4610, 2021.
DeepSeck-Al and Daya Guo et al. Deepseck-rl: Incentivizing reasoning capability in 1lms via reinforcement learning, 2025. URL /tl/:tttss.//rrxxv.ooo//////////
11

---

Published as a conference paper at ICLR 2026
Jusen Du, Weigao Sun, Disen Lan, Jiaxi Hu, and Yu Cheng. Mom: Linear sequence modeling with nixture-of-memories, 2025. URL hottts://axxiv.cee abt.g//aab//////// 13685
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprini arXiv:2407.21783, 2024.
DeepSeck-AI et al. Deepseck-v3 technical report, 2024.
Alexander Richard Fabbri, Irene Li, Tianwei She. Suyi Li, and Dragomir Radev. Multi-news: A large- scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1074-1084, 2019.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human- annotated dialogue dataset for abstractive summarization. Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp. 70-79, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10.18653/v 1/D1--5409
URL http::://wwww.....o www.wwwww..bbccm...wwwwwww...... org/
Ontholooy/DD199400
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang. Jiadai Sun, Jiajie Zhang. Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang. Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang. Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang. Yueyan Li, Yushi Bai, Yuxiao Dong. Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm- 130b to glm-4 tools, 2024.
Albert Gu and Tri Dao. Mamba: Linear- Linear-Time Sequence Modeling with Selective State Spaces.
Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. Longcoder: long-range pre- trained language model for code completion, 2023. URL https:::/aaxii..rgg////////
14893
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. Proceedings of the 28th International Conference on Computational Linguistics, pp. 6609-6625, 2020.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models?, 2024.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 1419-1436, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2022 www.cl-aaaaaa 112. URL httpss/:: a/cc anthology org 2021 nwacl-ma 112
Albert Q. Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: /ttttp:://rxii
org/abs/ /t//aabb//11... 06825
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL jttpss:///// net Www.Eooruuuiuieo. 8yNQM66
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URL https: //arxiv.ogg/ abs /705. 03551
12

---

Published as a conference paper at ICLR 2026
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns:: Fast autoregressive transformers with linear attention, 2020.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis. and Edward Grefenstette, The narrativeqa reading comprehension challenge, 2017. URL https 07040
//arxiv.
Kuang- Huei Lee, Xinyun Chen, Hiroki Furuta, John Canny, and Ian Fischer. A human-inspired reading agent with gist memory of very long contexts, 2024. URL
09727
Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can open-source Ilms truly promise context length?, June 2023, URL http::///aas...rr///ll/////////p
22000000-299990 29-1ongchat
Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th International Confer- ence on Computational Linguistics, 2002. WRLLhttps: peettps://www.c.. http://wwwwwlllllccccooooo org/ anghooogh
C02-1150
Kai Liu, Jianfei Gao, and Kai Chen. Scaling up the state size of RNN LLMs for long-context scenarios. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 11516-115222 1529, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 977-8-89176-2511111 //aclanthology.o .rg. 2ww.0///2222..22 acl 1ong.
doi:
URL. https
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and
Noam Shazeer. Generating wikipedia by summarizing long sequences, 2018. URL https:
10198
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint (XXi:2336603091 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https //rrxiv.org/abs/////
05101
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the Transformer Era.
Bo Peng. Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemyslaw Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartlomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr., Jiaju Lin, Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Cahya Wirawan, Stanistaw Wožniak, Ruichong Zhang, Bingchen Zhao, Qihang Zhao, Peng Zhou, Jian Zhu, and Rui-Jie Zhu. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence, 2024.
Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou -Zheng. Rwkv-7 "goose" with expressive dynamic state evolution, 2025. URL https: wxw.//rrxiiiaa org/ abs 2503. 14456
Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Yujia Zhou, Xu Chen, and Zhicheng Dou. Are long-llms necessity for long-conter tasks? ArXiv, a5s/2405.15318, 2024. URL https 7/api. seg.i.ssmanttccccho.... wtt.://ar.....roooou..u....d:.
Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. HGRN2: Gated Linear RNNs with State Expansion.
13

---

Published as a conference paper at ICLR 2026
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei 7hang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025, URL https:/ arxiv. org/ abs 2412.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: /0084, 2019.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang. Yu Liu, and Hongsheng Li. Visual cot: Advancing multi-modal language models with comprehensive dataset and benchmark for chain-of-thought reasoning, 2024. URL
abs 0tss/200.16999
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (Learn at Test Time): RNNs with Expressive Hidden States. doi: 10.48550/ARXIV.2407.. 006600.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-5545 2022.
A Vaswani. Attention all you need. Advances in Neural Information Processing Systems, 2017.
Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, and Bryan Catanzaro. An Empirical Study of Mamba-based Language Models. 2024. doi: 10-4555/ARXXI.2240607755 Publisher: arXiv Version Number: 1.
Songlin Yang, Bailin Wang. Yikang Shen, Rameswar Panda, and Yoon Kim. Gated Linear Attention Transformers with Hardware-Efficient Training. doi: 10.4850/ARXXI.2312..666
Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06466. 2024.
Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length, 2025. URL
06484
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: dataset for diverse, explainable multi-hop question answering, 2018. URL https: /tttps://arxiie wttpaaiiiivvooo www.vvorrgoooo abs 110/9/00990 09600
Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. Helmet: How to evaluate long-context language models effectively and thor- oughly, 2025. (tRL:ttts::///
wwtp://.rggabbb//bboooooo htsp://bb//241000 02694
Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, and Jaewoo Kang. Compact: Compressing retrieved documents actively for question answering. ArXiv, 0atp//407.000114 2024.
URL httphhtpp/ppppp htw.ppp.eeemmmicccccoooor...o.m
2111611422
Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei- Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, and Hao Zhou. Memagent: Reshaping long-context Ilm with multi-cony rl-based memory agent, 2025. WtRl::htss:////
02259
Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models, 2024a.
14

---

> Partition context into chunks of size C b Initialize hidden state tensor Initialize answer Published as a conference paper at ICLR 2026
Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Ô. Arik. Chain of agents: Large language models collaborating on long-context tasks, 2024b. URL https
w/taxxiv.rr/////.a.laa..o 22066 02818
Xueliang Zhao, Wei Wu, and Lingpeng Kong. Scaling reasoning without attention, 2025. URL
https://arxxt org/ abs 2505. 22425
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. 2025. URLhttps://agt
semant: cschola ar. wrgg/oorpssD:::1110
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization. In North American Association for Computational Linguistics (NAACL), 2021.
A APPENDIX
Algorithm Smooth Reading: End-to-End Multi-Round Inference
Require: Context D, chunk size C, recurrent model M, initial hidden state hn R
Ensure: Answer
X' CHUNK(D,
2: h + ho
3: < NULL
4: for i = 1 to X do
5: response, < (X h)
> Process chunk, get text response and updated state 6: action, summary PARSERiSPONSE(reSponss)) >> Extract action and summary from text
7: if action "READ" then
8: continue
b Process next chunk
9: else action = "STOP" then
PARSEREsUULr(summar) 10: break
11:
Early termination with answer
12: end if
13:
end for
return
Table Configuration for Dataset Construction. "Num" indicates the number of samples, and "Avg. Length" indicates the average length of the samples. The raw datasets include the training sets of
HotpotQA (Yang al. 2018), NarrativeQA (Kočiský et al.) (017), GovReport (Huang et al. [002],
QMSum (Zhong al. 2021 TREC (Li & Roth 2002). TriviaQA (Joshi et a 2017). SAMSum Gliwa et al. 2019), WikiSum (Lu et al. 2018), and LCC (Guo et al. 2023). Note, we only use the training sets of these datasets to construct our dataset, avoid data leakage.
Task
Raw Dataset
Summary Generator
Early Stop
Clean Metric
Num
Avg. Length
HotpotQA NarrativeQA
LLM
Yes Yes
Fl
5.652
4,249
Question Answering
LIM
4,348
10,
GovReport
LLM
No
Rouge-L
8.114
12,411
Summarization
QMSum
LLM
No
Rouge-L.
742
15,245
TREC
LLM
No
Exact Match
3,259
7.759
Few-shot
TriviaQA
LLM
No
Fl
3,370
13,942
SAMSum
LLM
No
Rouge-L
3,371
12,100
Passage Count
WikiSum
Rule
No
Exact Match
3,333
15,777
WikiSum
LLM
Yes
Exact Match
3,333
8,254
Retrieval
HotpoiQA
Rule
Yes / No
Fxact Match
1,667
20,
Code Generation
LCC
LLM
No
Edit Similarity
10,000
3,331
15

---

Published as a conference paper at ICLR 2026
A.1
IMPLEMENTATION DETAILS
A.1.1
MORE DETAILS ABOUT DATASET CONSTRUCTION
Table 5
summarizes the configuration for our dataset construction, and below is the prompt template we use to generate stepwise contextual summaries with LLMs.
{example)
Above is an example.
Now, you need to address problem with long context, chunk by chunk. You can access one chunk at time and the saved information. You should update the information to solve the problem. The saved and updated information are in JSON format. They have four keys.
"target": The target of the request. If there is no target, please leave it empty.
The output format requirement should be included in the target.
"reason": The reason for updating the information. You should explain why you update the information according to the given passage.
"clues": The updated information. You should update the information according to the given passage. The format is list of strings.
"result": The result of the request. You should give the answer to the target if the information is enough. If the information not enough, please leave it empty.
Respond in the format of ISON.
Summary of the previous chunks:
(Summary of Previous Chunks} Current chunk:
{Chunk Content)
A.1.2 HOW TO CONVERT SELF-ATTENTIO LLM TO SLIDING WINDOW L.M
It is simple to convert Self-Attention LLMs to Sliding-Windon Window LLMs. As Sliding- Window LLMs do not introduce any parameters, we directly inherit all parameters from the Self-Attention LLM, such as MLP layers and QK of attention layers. Then we modify the self-attention mechanism to attend only to the tokens within the sliding window.
A.1.3 TRAINING DETAILS
We use AdamW [oshchilov & Hutter 2019) with a learning rate of 4e-5 and a weight decay of 0.01. The batch size is set to 2, and the context length is set to 32k. We use a cosine learning rate decay schedule with warmup. As an exception, we use learning rate of le-5 and weight decay of 0.1 for models with 7B parameters, as we find this setting to be more effective. We train our models with Xtuner (Contributors) 2023b). For models with parameters, we train them for epoch on one H800 GPU, with about 12 hours of training time.
A. 1.4 EVALUATION DETAILS
We evaluate our models using the LongBench (Bai et al. 2024 and NIAH (Hsieh et al. 2024) benchmarks.
LongBench: LongBench comprises five main task categories, each with several sub-tasks:
Single-document question answering (SQA):
NarrativeQA
(Kočiský et al. 201 7
Qasper (Dasigi al. 202I), MultiFiclQQA (Bai et
2024)
Multi-document question answering (MQA): HotpotQA 2022)
(Yang al. 2018). 2WikiMulti-
hopQA (Ho et al. 2020). MuSiQue (Trivedi et al.
Summarization: GovReport (Huang et al. 2021
QMSum (Zhong et al. 202 MultiNews (Fab
bri et al. 2019)
16

---

Published as a conference paper at ICLR 2026
Table 6: Results on LongBench with 7B-parameter Models.
Architecture
Infer
Model
SQA
MQA
Summary
FewShot
Synthetic
Code
Avg
OR
Qwen-2.5-7B-OR
34,90
56.72
32.07
72.77
57.75
73.41
54.60
Self-Attention
NMR
Qwcn-2,5-7B-NMM.
34.37
45.37
25.20
71.27
66.50
65.26
51.33
OR
SWA-7B-4K-OR
26.10
37.04
28.12
71.50
42.38
71.88
46.17
Recurrent
EMR
SWA-7B-4K-FMR
37.80
54.67
27.39
69.86
65.75
67.71
$3.86
Table 7: Results on LongBench with More Models
Architecture
Infor
Modal
SQA
MQA
Summary
FewShot
Synthetic
Code
Avg
LongChat-vi.. 5-77 Li 2023
28.7
20.6
26.7
15.8
54.1
34.3
CharGl M2-6B (Gl Meaii 21223
32.9
33.7
27.6
59.1
39.2
52.7
40.9
Self-Amention
OR
Mistral-7B.fnssructe-e (Ting er
LJama-3. 3,1 8B InstracT (Dabey et 2024
25.3
23.7
28.4
27.2
69.4
53.0
56.6
46.5
42.7
ChatGL.M3-6D (GLM et il. 2074
40.3
46.6
29.5
68.1
50.5
56.2
48.5
Qwen2.5-7B Instrac Vwen
2125
50.0
Self-Attentinn
OR
Qwen-2255778 2.00-25---B-RR (ours)
34.90
56.72
32.07
72.77
57.75
73.41
54.60
Recurrent
EMR
SWA-7B-Ik- (ouns)
27.4
53.9
Few-shot learning: TREC (Li & Roth
2002), TriviaQA (Joshi al.
2017). SAMSum (Gliwa
ct al.
2019)
Synthetic tasks: PassageCount (Bai et 2024), PassageRetrieval (Bai et al. 2024
Code generation: LCC (Guo al. 2023). RepoBench-P (Liu al. 2023)
We use essays as haystacks, words as keys, and UUIDs as values. We use only one needle in our experiments by default, but four needles in Table 3 as one needle is insufficient to distinguish between models.
Needle-in-a- Nedlle-n-a--Hayttackk (NIAH):
Chunk Size during Evaluation: During inference with Smooth Reading, we set different chunk sizes according to the model type and task. For Sliding-Window Window LLMs, we use a chunk size of 1024 for LongBench and 2048 for NIAH. For RWKV-7, 256 for NIAH.
chunk size of 512 for LongBench and
Inference Engine: We use LMDeploy (Contributors 2023a which provides high performance for Self-Attention LLMs and supports Sliding-Window LLMs in interactive mode, maintaining hidden memory throughout inference. It provides strong support for Smooth Reading inference. To support the inference of RWKV-7, we implement a custom inference engine from scratch, as no optimized engine currently available.
A.2
MORE EXPERIMENTS
A.2.1 ADDITIONAL EXPERIMENTS ON 7B MODELS
To further assess the scalability of our method, conduct experiments using 7B 1B-parameter models, presented Table WWA-7B-4k-EMME achieves performance comparable to Qwen-2.5-7BB with less than 1 % difference and significantly outperforms SWA-7B-4k-00 and Qwen-2.5-7B -.--7B--MR.. The results confirm that Smooth Reading consistently enhances performance across model sizes.
A.2.2 COMPARISON WITH MORE MODELS
We further compare our models with more commercial and open-source Self-Attention LLMs with
6B-8B parameters, including LongChat-v 1.5-7B (Li et al. 2023), CharGLMM26B (GLM et al. 2024
Llama-3, 1-B- Instruct (Dubey et al. 2024), Mistral-7B - (Jiang et al. 2023). CLarGLM3-
6B (GLM et al. 2024). and Qwen2.5-7B B-Bnttuuttt Qwen et al. (2025). As shown in Table 7 our Qwen-2.5-7B 5--RRR is strong baseline among Self Self-Attention LLMs, and
WWA-7B-4k-ER achieves performance comparable to Qwen-2.5-7B-00 outperforming all other Self-Attention LLMs.
Notably, this indicates the solid performance of our SWA-7B-4K-FMR.
17

---

Published as a conference paper at ICLR 2026
Table 8: Evaluation of Generalization Ability on the entire RULER Benchmark. "VT" stands for Variable Tracking, "CWE" for Common Words Extraction, "FWE" for Frequent Words Extraction, and "QA" for Question Answering.
Model
VT
CWE
FWE
QA
Avg
Dwen-2.553B-OFF
0.00
0.30
62.80
41.90
26.25
Dwen-2.5-3B-NME
3.76
1.34
5,40
14.80
6,33
SWA-3B-4K-OR
0.00
1.48
9.87
0.10
2.86
SWA-3B-4K-EMR
8.40
0.32
50.07
53.60
28.10
Table 9: Evaluation of Generalization Ability on HELMET. "RAG" for Gerrival-Auumentte Gen- eration, "PRR" for Passage re-ranking. "ICL" for Many-shot in-context learning, "LQA" for Long- document QA, and "SR" for Synthetic recall.
Model
RAG
PRR
ICL
LQA
SR
Avg
Qwen-2.5-3B-OR
47.67
0.36
34.80
14.14
31.25
25.64
Qwen-2.5-3B-NMR
21.50
0.00
2.96
12.70
35.06
14.44
SWA-3B-4K-OR
10.83
0.50
39.36
15.96
7.69
14.87
SWA-3B-44 4K-EMR
54.44
0.47
16.68
19.51
33.94
25.01
Table 10: Comparison with OPRM on LongBench.
Infer
Model
SQA
MQA
Summary
FewShot
Synthetic
Code
Avg
Eecurrent-Gemmat -eemma--TT-9B
29.83
29.60
26.45
34.23
3.75
43,41
27.88
OPRM
Falcon-Mamba- Ma----Maaaaann--- Mamba-Instruut--777
32.38
26.58
27.60
52.32
8.75
36.76
30.73
Falcon3-Mamba- Mamma--nsttuutt--
26.91
33.86
25.57
53.05
8.75
39.93
31.35
RWKV 7-3B (MR(ours)
28.87
40.02
28.23
65.90
65.25
59.92
48.03
EMR
SWA-3B-4K-EMR(ousy)
30.46
47.67
26.27
69.60
66.75
65.18
50.99
A.2.3 MORE EXPERIMENTS ON OUT-OFF-IISTRBBTIIO BENCHMARKS
To assess the generalization ability of our method, we conduct additional experiments on out-of- distribution (OOD) benchmarks with question formats and prompts unseen during training. We evaluate our model on RULER (Hsieh et al. 2024) and HELMET (Yen et al. 2025) with 32k context length, and the results are presented in Table and Table respectively. As shown in the tables, our SWA-3B-4K-EMM Variable Tracking(VTT in RULER and Passage re- re-raning(PRRR) in HELMET are the tasks that differ most from training distribution, and both models perform worse these tasks than others. Nevertheless, SWA-3B-4k-B B---4--MMR surpasses Qwen-2.5-3B-OOr on these two tasks, indicating stronger OOD generalization. Additionally, both Qwen-2.5-3B-NMB and SWA-3B-4K-OR perform poorly these benchmarks, further underscoring the importance of co-designing architecture and inference methods.
achieves performance comparable to 2w5en-2.5-BB--
on both benchmarks.
A.2.4 COMPARISON WITH OVERFLOW PREVENTION FOR RECURRENT MODELS (OPRM)
Overflow Prevention for Recurrent Models (OPRM; Ben-Kish et 2025 proposes a RAG-style inference method to address overflow in Recurrent LLMss. Our approach differs in two key ways: (1) OPRM's RAG-style inference constrains generalization and hampers tasks that require reasoning over the entire context; and (2) OPRM does not fully exploit the recurrent architecture and therefore shows no clear advantage over Self- Attention LLMs. We compare our method with OPRM on LongBench (see Table IO), where our model outperforms all OPRM variants by substantial margin, demonstrating the effectiveness of our approach.
A.2.5 COMPARISON WITH OTHER MULTI- STEP INFERENCE METHODS
To assess the effectiveness of our approach, we compare Smooth Reading with several other LLM inference methods. For fair evaluation, implement each method as follows:
18

---

Published as a conference paper at ICLR 2026
Table 11: Comparison with More Multi-Step Inference Methods on Question Answering Tasks.
Architecture
Infer
Model
HotpotQA MuSiQue
2WikiMQA
TriviaQA
RAG
34.59
13.44
27.67
81.16
Self-Attention
RAG+
36.91
19.92
35.47
82.15
CompACT
Owen-2.5-3B-Imma
36.57
18.35
39.95
79.69
Recurrent
EMR
54.25
31.77
56.99
85.61
Table 12: Comparison of Performance with and without Early Stopping on NIAH.
Early Stop
8k
16k
32k
64k
128k
256k
Avg
without 99.80
100.00
100.00
99.80
99.60
100.00
99.87
with 99.60
99.60
99.80
99.80
99.60
99.80
99.70
We utilize standard RAG models introduced by (Asai et al. 2023). For each query, Contriever-Mr MARCO retrieves the top five documents from Wikipedia, using the official embeddings from the 2018 English Wikipedia. These retrieved passages form the long context. Note that this method incorporates external knowledge.
RAG:
To avoid using external knowledge, we split the context into multiple passages for each question. Both the query and each passage are encoded using sentence embedding mod- els (Reimers & Gurevych 2019). We compute the cosine similarity between the query and the passages, select the top three most relevant passages, and provide them as the context for answer generation.
RAG+:
CompACT Yoon et al. 2024): We also compare with CompACT, strong baseline. This method iteratively compresses the context based on the question, and the answer is generated from this compressed context. In our experiments, adopt the off-the-shelf CompACT compressor
As shown Table II our Smooth Reading approach consistently achieves the highest F1- Fi-scores across all evaluated datasets, demonstrating its superior capability in handling long-context passages. Compared to the compressor-basee method CompACT, Smooth Reading attains better results, which suggests that the compression process in CompACT may discard information important for accurate reasoning.
A.2.6 INFLUENCE OF EARLY STOPPING ON PERFORMANCE
We further compare the performance of our method with and without early stopping by evaluating on NIAH, as presented Table T2 The results show that early stopping has minimal impact on performance, with average accuracy exceeding 99% both scenarios. Moreover, by default, we enable early stopping for question answering on LongBench, and the high performance of our LongBench results (Table 2 also indicates that early stopping does not significantly affect performance.
A.3 LIMITATION AND FUTURE WORK
There several limitations to work.
No Completely New Architecture or Inference Method We focus on the co-design of architecture and inference rather than introducing entirely new architectures inference algorithms. Although the individual components we build upon already exist, our contributions are twofold: (1) to the best of our knowledge, inference method, demonstrating that judicious co-design can substantially improve performance; and (2) we significantly improve the long-context performance of Recurrent LLMs, bringing them on par with Self-Atee Attention LLMs a notable milestone for Recurrent LLMs. Designing entirely new architectures or inference methods is outside the scope of this paper and is left to future work.
the first systematically study the interaction between architecture and
19

---

Published as a conference paper at ICLR 2026
Architectural Limitations We study sliding-window
window LLMs and RWKV as representative recurrent LLM architectures. Other recurrent architectures exist (e.g.. DeltaNet Yang et al. 2025)). ). but we did not include them due to two practical barriers: (1) the absence of open-source well well-pretrained base models, and (2) the lack of optimized inference engines that support high-throughput, multi-round processing. Extending our approach to these architectures is important direction for future work.
Need for SFT Training Our method requires supervised fine-tuning (SFT) to adapt models to the Smooth Reading inference paradigm. This reflects a broader limitation of multi-round inference: current LLMs are mainly trained under a one-round SFT protocol, making their out-of-the the-box multi-round performance suboptimal. Prior work on multi-round inference (Yoon al. 2024] Yu et al. [2025] Shao et al., (2024) similarly relies on additional training. As future work, we plan to explore training regimes that produce models capable of both one-round and multi-round inference without requiring separate SFT datasets.
Need for Data Construction Because SFT is required, we must construct training data tailored to multi-round inference. Advances in reinforcement learning (RL) may reduce this requirement by leveraging simpler annotations, thereby lowering data construction costs. Given the complexity RL and the current lack of mature RL infrastructure for Recurrent LLMs, we leave this to future work.
Order of Query and Context
We use the default order of query followed by context (i.e., " "[QUERY] [CONTEXT]] ") and do not support the reversed order (" [CONTEXT] [QUERY] "). This limitation common among Multi- Round Inference methods, most prior work (Yoon et al. 2024 Yu et al. 2025) also adopts this order. Nevertheless, addressable: since our agentic multi-step pipeline has an expandable action space, future extensions could allow the model to read chunks in any order and learn to select the reading sequence autonomously. This would enable the model to prioritize relevant chunks regardless of query position. Achieving this, however, would require more sophisticated data construction and training strategies, such as reinforcement learning, which we leave for future work.
Limited Tasks We evaluate primarily on long-context tasks, as these pose central challenge for Recurrent LLMs. While our method is, in principle, applicable to other domains such as deep research (Zheng al. 2025) and software development (Jimenez et al., 2024). broader evaluation is left to future work. We expect our approach to improve both accuracy and efficiency in these settings.
Despite these limitations, we believe our work takes meaningful step toward unlocking the potential of Recurrent LLMs and advancing the frontier of LLM research.
20


Reviews:
[{"review_id": "5OiuyWvFtp", "reviewer": "Reviewer_aS6X", "summary": "The paper proposes Smooth Reading: an End-to-End Multi-Round (EMR) inference procedure co-designed with recurrent LLMs. Instead of one pass over the full context, the model reads chunks, emits a contextual summary, and updates hidden memory across rounds; this preserves recurrent efficiency while avoiding fixed-memory overload. On LongBench and NIAH, EMR closes or beats (but lacks SSM/RetNet baselines) a self-attention baseline (Qwen-2.5-3B-OR) while keeping linear scaling; at 64k tokens it reports ~2.5× faster training and ~2× faster inference than self-attention, with ~20% overhead vs the recurrent one-round baseline. Methods, datasets, and training/eval protocols are described, with Algorithm 1 outlining EMR.", "questions": "1. SOTA recurrent/SSM baselines: Can you add Mamba-7B and a RetNet-style model on LongBench/NIAH/RULER, with matched training protocol and report accuracy, tokens/s, and peak memory?\n\n2. Seed & CIs: For Table 2–3 and Fig. 3, can you re-run with ≥5 seeds and include 95% CIs to calibrate the extrapolation/stability claims?\n\n3. Wall-clock decomposition: Please provide a table with FLOPs saved vs. added (per-round summaries/decoding), and map it to the 2× inference speed at 64k (Fig. 3b).\n\n4. EMR hyperparameters: What default chunk:window ratios do you recommend (Table 4 suggests ~1:2), and how sensitive is accuracy/speed to this ratio across tasks?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "### Originality\n\n* Frames architecture–inference co-design for recurrent LLMs, advocating EMR over one-round or non-end-to-end multi-round strategies; analyzes complexity/performance trade-offs (Table 1, Eq. (1)).\n\n### Quality\n\n* Provides a concrete algorithm (Algorithm 1), chunking/summary scheme, and datasets (A.1), plus ablations on window/chunk sizes showing co-design effects.\n\n### Clarity\n\n* Clear figures/tables separating OR / NMR / EMR and self-attention / recurrent; centralized setup in §4.1.\n\n### Significance\n\n* Demonstrates gap-closing on LongBench (SWA-3B-4k-EMR avg 50.99 vs Qwen-2.5-3B-OR 47.38) and near-perfect NIAH at up to 256k, while maintaining recurrent efficiency.", "weaknesses": "1. Long-context SOTA families like Mamba/SSMs and RetNet are not benchmarked in Tables 2–3, limiting external validity of the “closes the gap” claim. Add head-to-head results (accuracy and throughput/memory).\n\n2. Seed counts and uncertainty are not prominent in the main text. For stability/extrapolation claims, report ≥5 seeds on key curves with CIs. (If already in appendix, surface in main text.)\n\n3. Realized wall-clock gains can be non-monotonic with skip/round settings; include a FLOPs vs. wall-clock breakdown and gate/round overheads alongside Figure 3, and consider a kernel-optimized path for recurrent EMR.\n\n4. EMR details are spread across §3 and A.1; a compact main-text box with the chunking rules, summary template, and early-stop criteria would improve reproducibility at a glance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rjKlpJKiqB", "reviewer": "Reviewer_DGWQ", "summary": "This paper presents an approach to enhance recurrent LLMs ability to preform long-context tasks. Instead of processing the entire input at once, the paper proposes a method called \"smooth reading\" -- essentially chunking the input and outputting a segment of summaries before outputting the next chunk of text. Experiments shows that two recurrent LLMs of 3B sizes trained in this manner achieve better performance by standard LLMs on long context tasks (LongBench and NIAH).\n\nOverall I find the paper to present an interesting approach to improve recurrent LLMs with comprehensive experiments and analysis.", "questions": "* Out-of-domain evaluation: the experiment for out-of-domain evaluation on HELMET (Table 9) is a bit incomplete -- what's the performance for QWEN-2.5-3B-NMR and SWA-3B-4k-OR?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper presents an innovative method to enhance recurrent LLMs ability to process long-context. The method is intuitive and relatively straightforward. \n* Experiment results demonstrate the effectiveness of the method.", "weaknesses": "* Missing reference: There has been some chunk reading work proposed before for standard LLMs, such as: [MemWalker](https://arxiv.org/pdf/2310.05029), [ReadAgent(ICML 2024)](https://arxiv.org/pdf/2402.09727).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2gT2MeuLtx", "reviewer": "Reviewer_8gR7", "summary": "This paper systematically studies the persistent performance gap between Recurrent LLMs and Self-Attention LLMs in long-context understanding. While Recurrent LLMs offer linear-time and constant-memory efficiency, they struggle with processing long inputs due to fixed-size memory constraints. To tackle this issue, the authors propose Smooth Reading, a co-design of architecture and inference method that uses End-to-End Multi-Round (EMR) inference. Instead of processing all tokens in one pass (“One-Round”), the model reads the input in chunks, produces contextual summaries, and updates its hidden state across rounds—effectively address the memory overwhelming issue of Recurrent models while not sacrificing the efficiency.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a well-organized and systematic study to demonstrate that R-LLMs' performance depends critically on inference design—a neglected dimension in prior works.\n\n2. The End-to-End Multi-Round inference approach is conceptually simple yet powerful, avoiding common pitfalls of non-end-to-end chunking.\n\n3. Evaluation spans multiple long-context benchmarks (LongBench, NIAH, RULER, HELMET), multiple architectures, and comparisons with both Self-Attention and RAG methods.", "weaknesses": "The comparison has only been conducted using standard transformer models, so it would be valuable to evaluate whether the proposed method can also outperform more efficient transformer variants such as Mamba, Jamba, or Hymba.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dgBJDENGof", "reviewer": "Reviewer_AYW3", "summary": "This paper proposes smooth reading, a method that combines architecture design and inference optimization, which aims to minimize the memory demands during inference time.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow. Readers can easily get the design philosophy behind the proposed methodology.\n\n2. The research question is practical and meaningful. The degradation of LLMs' capabilities under long context remains an unsolved problem, and the multi-turn chat is one scenario that such situation could usually happen.\n\n3. The author conducts extensive experiments on many benchmarks, which show significant improvements and competatives to current self-attention LLMs.", "weaknesses": "1. The idea of context compression or LLM with memory has been extensively explored in prior works (e.g., [1][2][3]). \nAlthough the authors claim that EMR models are primarily designed for multi-turn chat scenarios, the proposed approach does not exhibit substantial conceptual or methodological differences from existing studies. [1] compresses context into special compression tokens in both training and inference time, [2][3] preserve the most significant information at head level.\nAs a result, the overall novelty of this work appears limited. The author should compare and explain the differences between these works and the proposed method in detail.\n\n2. The paper sets its motivation in a multi-turn conversational setting; however, this aspect is not clearly reflected in either the formulation or the experimental design. The evaluations and problem setups mainly focus on single-turn instruction-following input–output pairs, while an EMR model is expected to demonstrate its advantage in multi-turn observation and planning. The authors should show their model's performance on such tasks to substantiate their claims.\n\n[1] Zhang, Peitian, et al. \"Soaring from 4k to 400k: Extending llm’s context with activation beacon.\" arXiv preprint arXiv:2401.03462 2.3 (2024): 5.\n\n[2] Li, Yuhong, et al. \"Snapkv: Llm knows what you are looking for before generation.\" Advances in Neural Information Processing Systems 37 (2024): 22947-22970.\n\n[3] Zhang, Zhenyu, et al. \"H2o: Heavy-hitter oracle for efficient generative inference of large language models.\" Advances in Neural Information Processing Systems 36 (2023): 34661-34710.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)