Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agents
Authors: Yaorui Shi
Yuxin Chen
Siyuan Wang
Sihang Li
Hengxing Cai
Qi GU
Xiang Wang
An Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Agent
Reinforcement Learning
Long-Context LLM
Primary area: reinforcement learning

Abstract: Large language models face challenges in long-context question answering, where key evidence of a query may be dispersed across millions of tokens.
Existing works equip large language models with a memory corpus that is dynamically updated during a single-pass document scan, also known as the "memorize while reading" methods.
While this approach scales efficiently, it suffers from irreversible forward-only processing, information loss through overwriting, and sparse reinforcement learning signals.
To tackle these challenges, we present ReMemR1, a memory-augmented agent with callback-enhanced memory that allows selective retrieval from the entire memory history and allows non-linear reasoning and revisiting of early evidence.
To further strengthen training, we propose Reinforcement Learning with Multi-Level Rewards (RLMLR), which combines final-answer rewards with dense, step-level signals that guide effective memory use.
Together, these contributions mitigate information degradation, improve supervision, and support multi-hop memory utilizing.
Experiments on long-document QA show significant gains over existing memory-based approaches, which validates ReMemR1 as an effective solution for long-context reasoning agents.

Published as a conference paper at ICLR 2026
LOOK BACK TO REASON FORWARD: REVISITABLE MEMORY FOR LONG-CONTEXT LLM AGENTS
Yaorui Shi" Yuxin Chen Siyuan Wang'. Sihang Li", Hengxing Cai
Qi Gu Xiang Wang", An Zhang!:
1 University of Science and Technology of China, National University of Singapore 3 Shanghai Jiao Tong University, DP Technology, 5 Meituan (yaoruishi, 2iaggaagg22238@pg com, an_ mw._zhnn@@ssc.edu... edu.co Equal Contribution. Corresponding author.
ABSTRACT
Large language models face challenges in long-context question answering, where key evidence of a query may be dispersed across millions of tokens. Existing works equip large language models with a memory buffer that is dynamically up- dated via linear document scan, also known the "memorize while reading" methods. While this approach scales efficiently. suffers from pruning of latent evidence, information loss through overwriting, and sparse reinforcement learn- ing signals. To tackle these challenges, we present ReMemR1, which integrates the mechanism memory retrieval into the memory update process, enabling the agent to selectively callback historical memories for non-linear reasoning. To further strengthen training, we propose multi-level reward design, which com- bines final-answer rewards with dense, step-level signals that guide effective mem- ory use. Together, these contributions mitigate information degradation, improve supervision, and support complex multi-hop reasoning. Extensive experiments demonstrate that ReMemR1 significantly outperforms state-of-the-as baselines on long-context question answering while incurring negligible computational over- head, validating its ability to trade marginal cost for robust long-contex reasoning. Our code is available at https:/ //gi thub. wwt/ https/yy-ccneeeeeee
INTRODUCTION
Reasoning vast, els (LLMs) (Hsich et al., 2024; Team et al., 2024; Beltagy et al., 2020; Ding et al., 2023; Child et al., 2019). This capability is crucial for real-world applications, such synthesizing legal precedents or reviewing scientific literature, where critical evidence for a single query can be scattered across millions of tokens. However, the quadratic complexity of attention mechanisms makes difficult for LLMS to track long-range dependencies and faithfully synthesize disparate information into coherent answer.
contexts remains critical bottleneck for large language mod-
To mitigate this, two primary paradigms have emerged. The first is Full-Text Context Retrieval (Figure (a)), where retriever fetches relevant chunks from a corpus form prompt (Jin al., 2025: Song et al., 2025; Shi et al., 2025; Du et al., 2026; 2025a). While widely used, this approach presents the LLM with fragmented, partial information and suffers from a heavy storage burden for the vector index. Alternatively, recent research explores the "memorize while reading" paradigm (Yu et al., 2025a; Li al., 2025; Wang et al., 2025b; Shi al., 2026) to handle infinite contexts linearly. As shown in Figure 1(b), this framework employs a memory agent that digests documents sequentially. At each step, the agent consumes a document chunk C together with its previous memory my and compresses them into new memory m441. After single linear pass through the entire document, the agent uses this final memory my buffer to generate an answer for the given question. This reduces the complexity of long-context question answering to linear time.

---

Published as a conference paper at ICLR 2026
(1) Full-Text Context Retrieval
(b) Memorize while Reading
(c) Memorize with Callback
Document Corpus (Full-Texi)
Resilever
L.M
L.M
LIM
X Heavy Storage Burden
X Partial Info
Mansry
Mamary
LLM Summa lood Key Menoey mil
For Veerry
Partial Decument Retrieved
Pruning Latent Rvidence
X Progressive information Loos
Non-Linear Memory Visiting
LLM
Answer
Light-Wiighted
Queston Q
Queatio Q
Anuwer
Imput:
<nstrieved Infor answer squestio (*..
Input: <m;-3: chunk,-1?
Inpul: son..,.ffunk, y (m,))
Based lotest memory.
retroved memory.
Output: Answer
Output: my
Output: me
Comparison of memory paradigms. (a) Full-Text Retrieval separates retrieval from reasoning and incurs heavy storage burden. (b) "Memorize while Reading" paradigm suffers from progressive information loss and important information neglection due linear memory overwrit- ing. (c) This work introduces callback mechanism, enabling non-linear memory visiting over past details and liggt-we storage.
Figure 1:
Despite its efficiency, we identify the following intrinsic limitations in the existing "memorize while reading" paradigm:
Premature Pruning of Latent Evidence. Standard memory agents evaluate the importance of the current document chunk er based solely on the current memory state my However, complex multi-hop reasoning could require integrating evidence found at different positions in text. For instance, an agent might encounter a piece of evidence early on (Step t) whose significance only becomes apparent after reading later section (Step + k). Crucially, such limitation cannot be solved solely by improving the memory update policy, as the relevance of specific information may only become apparent with certain prior knowledge, which can be embedded in future context.
Progressive Information Loss in Memory Overwriting. The paradigm's 's reliance on a fixed- length memory buffer necessitates constant information compression. This progressive degrada- tion of memory makes it difficult to maintain the full context and impedes the ability to resolve complex queries that require synthesizing evidence spread across distant sections of the document.
Sparse and Delayed Supervision. Training these agents using reinforcement learning typically relies on a single reward signal, such as the correctness of the final answer. This sparse reward the end of the reasoning process offers limited guidance for the long sequence of intermediate memory updates, leading to inefficient optimization and suboptimal memory management strate- gies, particularly complex tasks where producing correct final answers is especially challenging.
To address these challenges, we introduce ReMemRI, memory-augmennted LLM agent that can callback historical memories when navigating long documents. Conceptually, we introduce the mechanism of explicit memory retrieval into the "memorize while reading" paradigm, thus move beyond the restrictive state of the conventional MDP. Instead of passing only the memory m, during iteration, we augment the state 8; (m, q:), where qe a callback query that enables retrieval over the agent's 'sss entire memory history. At each step, the agent not only updates its memory my based on the new chunk 6, but also generates a callback query 9++1 to reach its past memories (m;)is (Figure 3). The retrieved information is then integrated into the context for the next state update. As depicted in Figure I(c), this mechanism empowers the agent to construct non-linear reasoning paths, and selectively revisit critical facts from early stages connect with new evidence. This directly counters the progressive information loss and breaks the irreversible forward-only constraint.
To robustly optimize this architecture, implement multi-level design tailored for the multistep memory updating of ReMemRL. Unlike general RL environments where agent actions alter future observations, the sequence of document chunks in task remains identical across all trajecto-
2

---

Published as a conference paper at ICLR 2026
de-s
Question g
Question
GY
Miemayy
Latest Memory my
Momory
LLM Ageut
m,
Callback Query
Momorics
Callbock Mensory
Rulo-hanc
1.M Agent
mf
Memosies
Answer
(a) Memory Update with Callback
(b) Final Answer Generation
Framework of ReMemR1. (a) Memory Update with Callback: At each time step, the agent updates the current memory my and generates a callback query St to retrieve relevant history memories. The state update integrates the previous memory my-1, the current chunk, and the retrieved history. (b) Final Answer Generation: The final answer is synthesized using the latest memory state and final query over the accumulated memory history.
Figure 2:
ries at any given step This isolation allows us to pinpoint the specific contribution of memory updates and callback actions without environmental noise. Leveraging this, training objective combines trajectory-level outcome rewards (answer correctness) with fine-grained, step-level sig- nals that strictly evaluate the information gain of each memory transformation, thereby solving the sparse supervision bottleneck inherent in long-context reasoning.
Extensive experiments on both in-distribution and out-of-distribution benchmarks demonstrate that ReMemR1 consistently surpasses general-purpose purpose LLMs and specialized memory agents. Beyond overall performance, we further conduct systematic analyses of memory callback strategies and multi-level reward designs, confirming the superiority of our RL-driven framework. Furthermore, we provide detailed analysis of computational overhead, which reveals that while ReMemRI ex- plicitly stores intermediate memories, the retrieval latency is negligible (< 0.2% time overhead). This confirms that our approach successfully trades marginal increase computational cost for significant gains (over 20% error rate reduction) in reasoning accuracy, effectively addressing the limitations progressive information loss without incurring prohibitive scalability issues.
METHOD
In this section, we present ReMemR1, a memory-augmented agent that incorporates history-aware retrieval and reinforcement learning with multi-level rewards to enhance long-context reasoning. We first review the formulation and limitations conventional "memorize while reading" paradigm, where memory agents solve long-context QA through a single-pass Markov decision process ($2.1). We then introduce our history- augmentd state mechanism, which enriches the memory update process with query component that enables retrieval past memory pieces and supports non-linear reasoning paths ($2.2). Finally, we describe the proposed multi-level reward structure, which combines trajectory-level outcome rewards with step-level state rewards to provide more effective training supervision ($2.3). Related work is discussed in Appendix A.
that be formulated as
2.1 PRELIMINARIES: MDP MEMORY AGENT FOR LONG-CONTEXT QA
We consider the task of long-context question answering (QA), where each dataset sample is given as (Q, Y). Here, Q denotes question and Y is the set of all acceptable correct answers to that question (i.e., candidate answer list, and answering with any element in Y is regarded correct). Each sample further associated with a long document C, which is divided into small chunks C0,0,, .
- Cr- and sequentially provided the model.

---

Published as a conference paper at ICLR 2026
New Memory
Documents
Callback Query
M++1 = To cr m,)
(m++1 9c+1) = = (Q ct, m, E(m 9i))
Retrieve Historical Memories
Question
Last Memory
Figure 3: The comparison state transition functions between "memorize while reading" and our method. (left) Conventional memory agents restrictive state 8t my. where the next memory m4+1 only depends on the current context C and memory m,. (right) Our method presents states 8; ()) where the agent generates callback query dr to retrieve relevant information from its entire memory history Emgi, enabling non-linear reasoning paths.
Standard memory-augmented agents process long documents a "memorize while reading" paradigm: the agent reads chunks one by one and continuously updates its memory to preserve important information. This sequential procedure can be naturally cast as a Markov Decision Pro- cess (MDP), written (s,U, P, R). At each step t:
The state S is defined by the agent's memory my (i.e., St my), which serves as the sufficient statistic summarizing the past trajectory. The agent also receives external inputs from the environment, consisting of the question Q and the document chunk ct.
The action 4N E U represents an update to the memory, which is determined by the policy given the current state and imputs.
The transition P(s++1 St, uz) specifies how the next state produced. In particular, the memory is updated as
9t+1 M To To ce, m²), for [0
The reward R is defined based on the quality of the final answer after the entire document has been processed ($2.3).
The model begins with empty memory, i.e., mg Ø. After all T document chunks processed, the agent produces a terminal output state by updating:
ST+1 = 0 = (Q, Ø, mr),
where the empty input indicates that no document chunk is provided this final step.
In this formulation, the memory my is assumed to be a sufficient statistic of the entire history of previously processed chunks {crice. However, this formulation is inherently restrictive. First, in multi-hop reasoning, the agent may scan over evidence that is crucial for later hops but fail to recognize its importance at the time, since the preceding hop has not yet been resolved. As the memory is updated, such overlooked evidence can be overwritten and thus lost for subsequent reasoning. Second, because the memory typically constrained to a fixed length to guarantee linear-time time complexity, early evidence progressively compressed and discarded more chunks are processed. Finally, the MDP structure itself prohibits the agent from revisiting past inputs once they overwritten, further limiting its ahility to integrate evidence scattered across distant parts of the document.
2.2 MEMORY AGENT WITH HISTORY-AUGMENTER STATE
To address these limitations, we extend the agent's reasoning capability beyond strictly forward trajectory by enabling it revisit and incorporate past evidence demand. Specifically, the agent not only maintains the current memory m, but also generates callback query qe to search over its history of memories (malist- The retrieved content is then integrated into the state representation, yielding & = (m,, 9.) This design allows the agent to selectively recall overlooked information and construct non-linear reasoning paths, rather than being confined to irreversible memory updates.
To realize this mechanism, at each step the agent receives the fixed question Q, the current doc- ument chunk et and the current state 8t. It is further equipped with retrieval function E, which
4

---

Published as a conference paper at ICLR 2026
selects relevant content from the previous memories {ms)ice on the overlap of words with the query de- The state transition is then defined as
= (1++1, ,, It+1) = Th (Q, ct, m,, ((mss)ss qc)),
(3)
where b(X,b) arg m recall(b, with recall(a,b) b) denoting the proportion of words in that also appear in b.
The query component 9t+1 evolves alongside the memory, enabling the agent iteratively refine its retrieval strategy over time. This design frees the agent from a strictly linear trajectory through the document, allowing it to form non-linear reasoning paths by recalling earlier evidence and thereby mitigating the information loss inherent to fixed-length memory.
2.3 REINFORCEMENT LEARNING WITH MULTI-LEVEL REWARD SHAPPNG
primary challenge in training nemory-augmente agents the sparse and delayed nature of su- pervision. For instance, a reward signal based solely on the final answer's correctness provides weak guidance for the many intermediate steps leading it. To address this, we analyzed the agent's rea- soning process and made the key observations: (1) In GRPO optimization, there are multiple rollouts for a single query Q and document set {ct) yet they explore different reasoning paths leading to different answers. (2) At each given step 6, the agent across different trajectories sees the same external context (Q,ca) cz) but maintains different internal state In this situation, the agent's ss task to integrate the current context with its evolving state to approach the correct answer.
Based on this insight, implement multi-level reward formulation tailored for the robust op- timization of memory agents. As illustrated in Figure 4(b), this algorithm comprises two main components: trajectory-lews reward that evaluates the final outcome, and dense, step-level state reward designed to shape the agent's intermediate behaviors by measuring relative information gain. These rewards are normalized across the corresponding trajectories and steps acquire the overall advantage for group relative policy optimization (GRPO) (Shao et al., 2024) optimization.
2.3.1 TRAIECTORY-LEVEL OUTCOME REWARDS FOR FINAL CORRECTNESS
The ultimate measure of an agent's success its ability to answer the given question correctly. We capture this with trajectory-level outcome reward, which is calculated based on the terminal state of each trajectory. Specifically, we first extract the predicted answer glm), enclosed in a \box(}, from the state 87+1 The outcome reward is then computed using an exact match metric against the set of ground -oound-truth answers Y:
R
max I(ffa) v),
WEY
where I(+) the indicator function that returns if the condition is true and otherwise.
2.3.2 STEP-LEVEL ACTION REWARDS FOR BEHAVIOR SHAPING
To provide the dense, fine-grained supervision that outcome rewards lack, we introduce step-level state rewards. These rewards evaluate the quality of intermediate state updates within a trajectory, directly shaping the agent's behavior toward greater efficiency and effectiveness.
Information Gain in Memory Updates: To combat the progressive information loss discussed in the introduction, we use rubric-based reward measure the information gain in the agent's memory. After each update from m4-1 to m,, we assess the presence of crucial entities from the ground-truth answer. If m, contains more information that are directly relevant to the ground truth Y than m,-1, we believe there's positive information gain achieved at time step t. Building on such rationale, we use the change in recall as a reward:
(g) memory,4
max recall(m (9) .y) max recall BEY
(5)
WEY
Bonus for Callback Retrievals: When the query component ge triggers retrieval through E((m(") (a) Jis dt (0)) the agent supplements its current memory with recalled information. To en- courage meaningful retrieval, design reward that measures the additional recall of critical
5

---

Published as a conference paper at ICLR 2026
(a)
State Rewords
Group Norm
All States
(x3)
A
Actor
y(3)
Group-Norm
Refenence Model
Rewords
Trajectory
A
States
Trajectories
Advantages
Normalize Rewards States at same Time Step
700
Go?
Intermediate State
Normalize Outcome Rewards across
Terminal States
%
Trajectories
Figure 4: Overview of the Multi-Level Reward Design. (a) From the trajectories generated by the actor model, we compute outcome rewards terminal states and state rewards all states. (b) Each reward type is normalized at the corresponding level: state rewards across the states at the same step, and outcome rewards across all trajectories in the group.
information provided by the retrieved content beyond what is already available in the current memory m and the immediate context ct. Formally:
(g) callbuck.t max recall E(mm (g) Jise, ((
(0) U Um!
(9) Uct)
max maxallllyyy m m Ue..
(9)
(6)
vEY
Format Reward: To ensure that the agent's outputs can be reliably parsed, introduce format (9)
reward 7 format, for all steps. For intermediate states, this reward checks for the correct usage of <callback> and <memory> tags. For the final step, verifies the presence of the \box() tag for the predicted answer.
The total step-level state reward at time for trajectory is the sum of these components:
(s) staaee memory.. caallback, format,
(g)
(9)
2.3.3 TRAINING OBJECTIVE
Given actor model and reference model Trefs sample group of G' trajectories {r(s) 1==1 (0) where each trajectory ((a) = (s(") (1) s ..... ST+1) ) is generated according to the state-transition dynamics in $2.2. The optimization objective is a variant of GRPO (Shao al., 2024) algorithm. Refer to Appendix C.I for the full form of our training objective.
The normalized group advantage A(9) is a composite of our multi-level rewards, with components calculated at different scales to reflect their distinct roles. For the outcome reward, we compute
trajectory-level advantage A(9) by comparing trajectory's outcome the group average. For the state rewards, we compute a step-level advantage Afmi by comparing a state's 's reward to the average reward of states at the same step / in the group. Following (Liu et al., 2025b;c), we omit the

---

Published as a conference paper at ICLR 2026
Table I: Long-connext QA results on HotpotQA (Yang al., 2018) and 2WikiMultiHopAA (Ho et al., 2020). Values are accuracy (%), rounded to 1 decimal. Bold denotes the best performances.
Accuracy HotpotQA (In-Distribution)
Number of Context Documents
Scale
Method
50
100
200
400
800
1600
3200
6400
Qwen2.5 (Yang et al., 2024) MemAgent (Yu et al., 2025a)
59.4
57.0
3B
70.3
69.4
60.9
68.8
60.9
60.2
59,4
58.8
ReMemR1 (Ours)
70.9
71.7
63.8
74.0
65.4
65.0
65.4
66.1
Qwen2.5 (Yang et al., 2024)
70.3
75.0
RI-Distill (DeepSeek-AI ct al., 2025)
40.6
25.8
10.2
0.8
1.6
2.3
1.5
3.1
7B
Qwen2.5-1M (Yang al., 2025b)
75.8
71.9
68.0
67.2
69.5
54.7
22.7
0.0
MemAgent (Yu al., 2025a)
81.8
78.9
78.9
77.0
79.7
72.1
74.0
75.8
ReMemRI (Ours)
82.3
82.8
81.1
78.9
82.0
79.7
80.0
80.8
(b) Accuracy on 2WikiMultiHlpQQ
Number of Context Documents
Scale
Method
50
100
200
400
800
1600
3200
6400
Qwen2.5 (Yang et al., 2024) MemAgent (Yu et al., 2025a)
39.8
39.1
39.0
3B
41.4
45.3
40.2
39.4
36.3
28.9
26.7
25.9
ReMemR1 (Ours)
53.5
50.4
42.5
41.7
37.0
36.2
35.4
37.8
Qwen2.5 (Yang al., 2024)
53.9
49.2
61.1
R1-Distill-Qwen -hsnn (DeepSeek-AI et al., 2025)
36.7
29.7
25.8
0.0
0.8
2.3
2.3
0.8
7B
Qwen2.5-1M (Yang et ul.. 2025b)
62.5
59.4
57.8
47.7
46.1
45.3
25.8
0.0
MemAgent (Yu et al, 2025a)
61.7
57.8
50.8
47.6
50.7
44.5
46,9
44.7
ReMemR1 (Ours)
63.9
63.1
55.6
54.5
54.7
45.4
48.9
50.3
standard deviation term during normalization to avoid introducing difficulty bias:
A
1
R!
Attate
R(s)
1
Σ
R
(k)
(8)
G
G
Finally, the overall advantage A( in Eq. 10 is combination of these two components:
A
(0) + (1 a:) A
(9)
where a is the hyperparameter that controls the importance of each term.
3 EXPERIMENTS
In this paper, we conduct experiments to answer the following research questions (RQs):
RQ1: Does ReMemRI outperform other memory agents general-purpose LLMs on long-context tasks, and can it alleviate the progressive information loss?
RQ2: Does ReMemR1 achieve nonlinear document utilization through the callback mechanism?
RQ3: Is ReMemRI computationally efficient, and how does the extra time and memory cost scale?
RQ4: Does our proposed multi-level rewards help the memory agent converge into better solution?
RQ5: What's the benefits of the RL-driven memory callback, comparing with rule-based design?
3.1 EXPERIMENTAL SETUP
Datasets. Our training data is sourced from HotpotQA (Yang et al., 2018). We pad the context of each training sample with random documents to 200 (about 30K tokens) per sample. For evaluation, the in-distribution (ID) HotpotQA and the out-of distriibittiiioonn (OOD) 2WikiMultiHopQA (Ho et al., 2020) datasets. The context documents of test data are also padded, ranging from 50 to 6400 documents per sample. For more implementation and dataset details, refer Appendix C.

---

Published as a conference paper at ICLR 2026
Baselines. In our experiments, compare method against three categories of baselines: (1) general LLMs, including Qwen2.5 models (Yang et al., 2024) and Qwen models distilled from DeepSeek-RI (DeepSeck-Al et al., 2025). (2) Long-context LLMs, including Qwen2.5-IM (Yang et al., 2025b); (3) tailored memory agents, such as MemAgent (Yu et al., 2025a). By default, we use the instruct version for all models. For comparison with more baselines, refer to Appendix B.1.
3.2 MAIN RESULTS (RQ1)
As shown in Table 1, our method consistently achieves the best accuracy across all model scales, datasets, and context lengths, surpassing both general-purpose LLMs and specialized memory agents. Compared with MemAgent, it achieves up to 7.3% higher accuracy on 3B model and 7.6% on 7B model, underscoring the effectiveness adaptive memory recall. We further observe that the number of context documents increases, the role of memory becomes increasingly critical. Pure reasoning models and long-come long-context models exhibit sharp performance degradation when facing very long contexts, while MemAgent mitigates this issue by adopting "memorize while reading" strategy that stores salient information a memory buffer. Building upon this, our method equips the agent with an RL-driven memory callback mechanism that adaptively selects what and when to retrieve, thereby enhancing the quality of the maintained memory. This advantage becomes increas- ingly evident as the document length grows, since in longer contexts important evidence more likely to be overwritten or overlooked, amplifying the need for precise recall to preserve reasoning accuracy. Notably, the gains are even more pronounced on the OOD 2WikiMullti indicating that our approach goes beyond memorizing dataset-specific patterns and instead acquires a genuine retrieval and reasoning ability, leading to stronger generalization across domains.
dataset,
3.3 DISTANT EVIDENCE CHALLENGE (RQ2)
To rigorously test the effectiveness and ac- curacy of the proposed memory callback mechanism, construct more challeng- ing evaluation setting. Specifically, for each question, the supporting evidences are arranged in the reverse order of their required reasoning sequence, and the dis- tance between successive evidence is en- forced to exceed half of the total number of context documents. This setup makes infeasible for the model to rely on lo- cal context alone; instead, it requires the model to identify and utilize interdepen- dent evidences across long spans.
MemAgent
ReMemm11
50
Accuracy (96)
30
10
200
400
800
Number of Context Documents
Figure 5: Accuracy 2Wiki with distant evidences.
As shown in Figure 5, our method surpasses MemAgent by large margin under this setting. MemA- gent suffers pronounced accuracy degradation due its inherent inability look back and reliably recall distant, scattered evidences. In contrast, our RL-driven callback mechanism adaptively re- trieves and maintains critical information, achieving far superior performance. These results demon- strate that the proposed callback design is both effective and robust, particularly when reasoning requires nontrivial coordination of evidences over long contexts.
3.4 COMPUTATIONAL EFFICIENCY AND SCALABILITY (RQ3)
3.4.1
INFERENCE-TINE EFFICIENCY
To evaluate the computational viability of our recurrent-memott design, we compare ReMemRI with the seemorize-hhilerreddee
baseline MemAgent under varying numbers of context documents. Figure 6(b) reports the overall accuracy and total memory usage both methods, while Figure 6 (a) presents the time and memory overhead introduced by the memory retrieval module.
We find that the retrieval process itself is highly efficient. Although ReMemRI stores all intermedi- memory states, the callback operations require less than seconds of latency and under 1MB of
8

---

Published as a conference paper at ICLR 2026
Comparison Accuracy
Computational Overhead Callback Module
3500
Labency (ReMemR1)
3000
Callback Nemory Usage (NaMemM1)
(BM) adesn Mmowio
Accuracy (56
75-
2500
Callback Latancy
2.0
(BN) 440000 1000000
70-
65-
60-
Accuracy (MamAgent)
-1000
Accursiyy (RcMem831)
55-
Nurney Usage (Merisg wnl)
500
Memory Usage (RattammL))
50
1000
2000
3000
4000 Documeitts
5000
6.00
1000
2000
3000
4000
5000
5000
Documents
Figure 6: Computational performance under different context lengths. (a) Comparison of accu- racy and total memory usage between ReMemRI and MemAgent, (b) Time and memory overhead introduced by the retrieval module. ReMemR1 consistently achieves higher accuracy with only modest additional computation (< 2s latency and < 1MB memory).
Table 2: Training-time computational comparison between MemAgent and ReMemR1.
Method
Avg Time Step
Step
Step 10
Step 100
Peak Memory Usage (GB)
MemAgent
1247.17 1467.72
1278.8 85
1366.91
1377.29
124.97
ReMemRI
1463.25
1518.99
1456.69
131.15
additional memory even at the 6400-document setting. This efficiency stems from the fact that the retrieved states are compact, model-gennrrted summaries rather than full external documents.
Importantly, this small computational overhead translates into substantial performance gains: Re- MemRI achieves up to 5% absolute accuracy improvement over the baseline, corresponding to 20% reduction in errоr rate. These results illustrate that ReMemRI offers favorable accu- racy-efficiency tradeoff, and provides stronger long-context reasoning while maintaining practical computational cost. Refer Appendix B.4 and Appendix D for additional empirical results and theoretical analysis over computational overhead ReMemR1.
3.4.2 TRAINING- TIME EFFICIENCY
We additionally measure training-timg time computation, including average per-step time, early/late step latency, and peak GPU memory usage. Results presented in Table 2.
Training overhead is moderate. ReMemR1 exhibits higher per-step latency than MemAgent due to callback-query generation, but the difference remains within a practical range.
GPU memory usage remains similar. Peak memory is dominated by the L.M backbone, and storing intermediate memories adds only small constant overhead.
3.5
ABLATION STUDIES
3.5.1 EFFECTIVENESS OF MULTI-LEVEL REWARD DESIGN (RQ4)
In ReMemR1, we propose a multi-level rewarding method to alleviate the sparse supervision prob- lem by combining trajectory-level outcome rewards with step-level state rewards. The balance be- tween these two rewards is controlled by a hyperparameter a, which determines how much weight placed on final-answer correctness versus intermediate behavior shaping (Eq. 9). We evaluate {1.0, 0.8, 0.5, 0.2) on Qwen2.5-3B Instruct examine its impact.
Results in Table demonstrate that a = 0.8 consistently delivers the best accuracy across different context lengths. A larger (e.g., 1.0) corresponds to using only outcome rewards, which neglects the benefits of dense step-level guidance and leads to weaker optimization. Conversely, smaller values (e.g., 0.2) overly emphasize seep-level shaping, which distracts the model from optimizing for final correctness. Based on these findings, we adopt a = 0.8 by default all the other experiments, as provides the best trade-off between global outcome rewards and local step-level supervision.

---

Published as a conference paper at ICLR 2026
Table 3: Accuracy on HotpotQA with different a values.
Number of Context Documents
Method
50
100
200
400
800
1600
3200
6400
1.0
70.3
73.4
61.5
59.6
60.9
64.1
62.5
63.3
0.8
70.9
71.7
63.8
74.0
65.4
65.0
65.4
66.1
ReMemRI
0.5
71.7
68.5
62.2
66.1
63.0
58.3
59.6
65.4
0.2
68.8
68.5
55.9
62.5
53.5
45.7
49.6
52.0
Table 4: Comparison of accuracy (%) on HotpotQA and 2WikiMultiHopQA across different call- back implementations. Bold denotes the best performance.
Number of Context Documents
Benchmark
Method
50
100
200
400
800
1600
3200
6400
MemAgent
70.3
60.9
68.8
60.9
60.2
58.8
HoiporQA
MemAgent rule-based callback
69.5
66.4
57.0
60.9
61.4
53.9
61.7
60.9
ReMemR1 (Ours)
70.9
71.7
63.8
74.0
65.4
65.0
65,4
66.1
MemAgent
41.4
45.3
42.2
41.4
38.3
28.9
26.7
25.9
2WikiMuliHHoPOA
MemAgent + rule-based callback
49.2
43.0
35.9
35.2
33.4
33.6
30.5
27.3
ReMemRI (Ours)
53.5
50.4
42.5
41.7
37.0
36.2
35,4
37.8
3.5.2 RL-DDRVEN V.S. RULE-BASED MEMORY CALLBACK (RQ5)
A key component of ReMemR1 is the RL-driven memory callback, where the agent learns through reinforcement learning generate informative queries that retrieve past evidence most relevant to the current step. This mechanism allows the agent to dynamically determine when and what to recall during reasoning. As intuitive yet strong baseline, we design rule-based memory callback, where the agent uses the question Q itself as fixed query for retrieval at every step. This design is motivated by the fact that the question contains rich information about the target answer, and thus provides a natural heuristic for guiding memory recall without requiring additional training.
Table 4 reports the results on HotpotQA and 2WikiMultiHopQA, with Qwen2.5-3B Instruct as the base model. We observe that RL-driven memory callback consistently outperforms both the vanilla MemAgent and the rule-based callback on both datasets across all context lengths. Notably, the rule-based callback does not always yield improvements and can even cause performance drops of up to 7.9%, highlighting that determining when and what to recall is non-trivial. We also observe that the advantage of our method increases the document length grows, indicating that effective memory recall becomes increasingly crucial in longer contexts. These results confirm that learning adaptive recall strategies via RL is essential for robust and generalizable long-context reasoning. Refer to Appendix B.2 for extended discussion about the impact of RL training.
4 CONCLUSION
This work examined the inherent limitations of the prevailing "memorize while reading" paradigm for long-context question answering, including irreversible forward-only processing, progressive in- formation loss from memory overwriting, and the sparsity of supervision signals. To address these challenges, we proposed ReMemRI, a memory-augmented agent that enhances the state represen- tation with callback queries, enabling retrieval from historical memories and facilitating non-linear reasoning paths. To further improve training efficacy, we developed RLMLR, reinforcement learn- ing framework with multi-level rewards that combines trajectory-leve outcome supervision with step-level state rewards. Experiments across both in nn-distribution and diitriibttiitnoi benchmarks show that ReMemR1 consistently surpasses general LLMSs and prior memory agents, and remains robust under the challenging distant-evidence setting. Ablation studies further confirm the necessity of the RLMLR training scheme and the RL-driven memory callback for enabling effective and gen- cralizable long-context reasoning. Looking ahead, we believe this work opens up new potential for future research on robust long-context understanding agents across diverse real-world domains.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our research is confined to computational experiments on publicly available benchmarks, specifi- cally HotpoiQA and 2WikiuultHHooQA. These datasets consist of publicly sourced text and do not contain personal information or other forms of sensitive data (Yang et al., 2018; Ho et al., 2020). No human subjects were involved in any stage of our work, including data collection or model evalua- tion. The focus of this paper is on foundational research for long context reasoning, and we do not develop or evaluate applications in high-stakes domains such as medicine, law, or finance.
We acknowledge the broader ethical challenges inherent in LLM-based systems, including the risk of perpetuating societal biases present their training data. While our methodological focus is on reasoning capabilities, the introduction of memory mechanism raises specific considerations regarding privacy and security. A system with the ability to store and recall information over long contexts could pose risks if deployed with private proprietary data without robust safeguards. Any downstream application of this work should undergo evaluation for fairness, transparency, and potential discriminatory impacts.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results, provide anonymous downloadable source code package in our abstract, as recommended by the conference guidelines. This package includes:
Complete code for generating our evaluation datasets from publicly available benchmarks (Hot- potQA and 2WikiMultiHopOO) using fixed random seeds.
Configuration files and instructions for setting up the experimental enviromment.
The training procedure of ReMemR1, including the implementation of the callback mechanism, RLMLR, and runnable training scripts based verl.
Evaluation scripts for both baseline models and our proposed method.
In addition, detailed descriptions the experimental setup and hyperparameters are reported in $3.1 and Appendix C. We hope that these materials will enable researchers to fully replicate and further extend our work.
ACKNOWLEDGEMENT
This research is supported by National Natural Science Foundation of China (U25A20445), The authors also acknowledge Xunliang Cai and the Longcat team for their valuable support this study.

---

Published as a conference paper at ICLR 2026
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learn- ing from human feedback in lms.. arXiv preprint 24X1::2402.147400, 2024.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint 20Xx::2004005150, 2020.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wangxiang Che. Towards reasoning era: A survey of long chain-of- thought for reasoning large language models. arXiv preprint arXiv:2503.0956) 2025a.
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan RGuan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wangxiang Che. Towards reasoning era: A survey of long chain-of- thought for reasoning large language models. arXiv preprint arXiv:2503. 09567, 2025b.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint (xXi::230611599 2023.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L.. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in ilms via reinforcement learning. CoRR, 1it//2501.12448 2025. doi: 10. 08550/ARXIV.2501.22
Jayy Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang. Wenhui Wang, Nanning Zheng, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.022 02466, 2023.
Mingxuan Du Benfeng Xu, Chiwei Zhu, Xiaorui Wang. and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506. 11763, 2025a.
Mingxuan Du, Benfeng Xu, Chiwei Zhu, Shaohan Wang, Pengyu Wang, Xiaorui Wang, and Zhen- dong Mao. A-rag: Scaling agentic serrivall-uggeente generation via hierarchical retrieval inter- faces. arXiv preprint arXiv:2602.034422 2026.
Yiming Du, Wenyu Huang. Danna Zheng. Zhaowei Wang, Sebastien Montella, Mirella Lapata, Kam-Fai Wong, and Jeff Z Pan. Rethinking memory in ai: Taxonomy, operations, topics, and future directions. arXiv preprint arXiv:2505.00675. 2025b.
Romain Duverger, Alexis Bonnin, Romain Granier, Quentin Marolleau, Cédric Blanchard, Nassim Zahzam, Yannick Bidel, Malo Cadoret, Alexandre Bresson, and Sylvain Schwartz. Metrology of microwave fields based on trap-loss spectroscopy with cold rydberg atoms. Physical Review Applied, 22(4):04403, 2024.
Albert Gu and Tri Dao. Mamba: Linear- sequence modeling with selective state spaces. arXiv preprint arXiv:23/2.0075:: 2023.
12

---

Published as a conference paper at ICLR 2026
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021.
Xanh Ho, Anh- Khha Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th Interna- lional Conference on Compulational Linguisticy, pp. 6609-6625, 2020.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arX1v:2404.06654, 2024.
Jian Hu, Reinforce++: A simple and efficient approach for aligning large language models, arXiv preprint arXiv:2501.03266 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, An- drey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton. Brandon McKinzie, Brydon Eastman, Camillo Lu- garesi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras. Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Ed- mund Wong, Elizabeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni. Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo. Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hes- sam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, lan Kivlichan, lan O' O'Connell, Ian Osband, Ignasi Clavera Gilaberte, and Ilge Akkaya. Openai ol system card. CoRR, abs/2412.116700, 2024. doi: 10.4550/ARXIV2412..770
Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang. Hamed Zamani, and Jiawei Han. Search-rl: Training Ilms to reason and leverage search engines with reinforcement learning, arXiv preprint arXiv:2503.095/6, 2025.
Leslie Pack Kaelbling. Michael L. Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
Angelos Katharopoulos, Apoory Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. 5156-5165, PMLR, 2020.
Yukun Li and Liping Liu. Enhancing diffusion baasusdio---sedd point cloud generation with smoothness con- straint. arXiv preprint arXiv:2404.02336, 2024.
Zhiyu Li, Shichao Song, Chenyang Xi, Hanyu Wang, Chen Tang, Simin Niu, Ding Chen, Jiawei Yang, Chunyu Li, Qingchen Yu, et al. Memos: A memory os for ai system. arXiv preprint arXiv:2507.03724 2025.
Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, et al. Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems. arXiv preprint 0.Xi::2504.01990. 2025a.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context. arXiv preprint arXiv:2310.01880 2023.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025b.
13

---

Published as a conference paper at ICLR 2026
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025c.
Adyasha Maharana, Dong- Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of Ilm agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 13851-13870, 2024.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27744,27744, 2022.
Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir Patil, lon Stoica, and Joseph Gonzalez. Memgpt: Towards Ilms operating systems. arXiv preprint arXiv:3310.0856/ 2023.
Bo Peng. Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung. Matteo Grella, et al. Rwkv: Reinventing mns for the transformer era. arXiv preprint arXiv:2305.13044 2023a.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarı: Efficient context win- dow extension of large language models. In The Twelfth International Conference on Learning Representations, 2023b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint mxXiv:4402.03300, 2024.
Yaorui Shi, Shihan Li, Chang Wu, Zhiyuan Liu, Junfeng Fang, Hengxing Cai, An Zhang, and Xiang Wang. Search and refine during think: Autonomous retrieval-augmented reasoning of llms. arXiv e-prints, pp. arXiv-2505, 2025.
Yaorui Shi, Shugui Liu, Yu Yang, Wenyu Mao, Yuxin Chen, Qi GU, Hui Su, Xunliang Cai, Xiang Wang, and An Zhang. Memocr; Layout-aware visual memory for efficient long-horizon reason- ing. arXiv preprint arXiv:2601.21468 2026.
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in Ulms via reinforcement learning. arXiv preprint arXiv:2503.05592. 2025.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- 2024.
standing across millions of tokens of context. arXiv preprint
Fanqi Wan, Weizhou Shen, Shengyi Liao, Yingcheng Shi, Chenliang Li, Ziyi Yang, Ji Zhang, Fei Huang. Jingren Zhou, and Ming Yan. Qwenlong-11: Towards long-context large reasoning models with reinforcement learning. arXiv preprint arXiv:2505. 17667, 2025.
Kun Wang, Guibin Zhang, Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, et al. A comprehensive survey in Ilm (-agent) full stack safety: Data, training and deployment. arXiv preprint arXiv:2504. 2025a.
Yu Wang, Ryuichi Takanobu, Zhiqi Liang, Yuzhen Mao, Yuanzhe Hu, Julian McAuley, and Xiaojian Wu. Mem- I alpha): Learning memory construction via reinforcement learning. arXiv preprint 250.592509. 2025b.
14

---

Published as a conference paper at ICLR 2026
Wujiang Xu, Kai Mei, Hang Gao, Juntao Tan, Zujie Liang, and Yongfeng Zhang. A-mem: Agentic memory for Ilm agents. arXiv preprint 1xXiv22502.12110, 2025.
Sikuan Yan, Xiufeng Yang. Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Hinrich Schütze, Volker Tresp, and Yunpu Ma. Memory-rl: Enhancing large language model agents to manage and utilize memories via reinforcement learning. arXiv preprint arXiv:2508. 19828. 2025.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1rX1v:2412.15115, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025a.
An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, et al. Qwen2. 5-1m technical report. arXiv preprint arXiv:2501.15383 2025b.
Zhilin Yang. Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2369-2380 2018.
Hongli Yu. Tinghong Chen, Jiangtao Feng, Jiangjic Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, al. Memagent: Reshaping long-context Ilm with multi-conv rl-based memory agent. arXiv preprint arXiv:2507.0223 2025a.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:25033 14476, 2025b.
15

---

Published as a conference paper at ICLR 2026
A
RELATED WORK
We review three areas of prior research relevant to our anisms for LLM-based agents, approaches for extending context length in language models, and reinforcement learning techniques for improving L.LM reasoning abilities.
LLM agent: memory mech-
Memory Augmented LLM Agents. The reasoning and planning capabilittess of LLM agents are fundamentally limited by the fixed size of their context window (Hsieh et al., 2024; Maharana al., 2024: Liu et al., 2025a). To overcome this, researchers have built external memory systems to retain information across long interactions, enabling agents to recall past experiences and adapt their behavior (OpenAI, 2023; Wang et al., 2025a; Du et al., 2025b). Early memory systems primarily focused on simple short-term memory (e.g.. prepending a conversation history to the prompt) and long-term memory (e.g., storing information in vector database for retrieval) (Li & Liu, 2024; Duverger et al., 2024; Packer et al., 2023; Yan et al., 2025). More recent approaches explore a "memorizing while reading" paradigm, where the LLM autonomously organizes its memory corpus during single-pase paggse-pss scan through the documents (Xu et al., 2025; Li et al., 2025; Yu et al., 2025a; Shi et al., 2026).
Long-Context LLMs. This long-context challenge in LLM has driven a variety of solutions, which can be broadly categorized into architectural modifications and context window extension techniques. Novel architectures, such as state space models (Gu et al., 2021; Gu & Dao, 2023; Peng et al., 2023a), achieve linear-time complexity and are highly efficient for long sequences. Other ef- forts focus on extending the context windows of iitention-based LLMs. One approach involves de- veloping more efficient attention mechanisms to reduce computational burden (Beltagy et al., 2020; Ding et al., 2023; Child et al., 2019; Katharopoulos et al., 2020; Liu et al., 2023). A complemen- tary technical route modifies Rotary Position Embedding enable models to extrapolate effectively beyond their original training length (Su et al., 2024; Chen et al., 2023; Peng et al., 2023b).
Reinforcement Learning (RL) (Kaelbling et al., 1996) has emerged powerful paradigm for post-training LLMs recently (Chen et al., 2025a;b; Jaech al., 2024; DeepSeek-AI et al., 2025). Early efforts focus on Reinforcement Learning from Human Feed- back (RLHF) (Ouyang et al., 2022) using algorithms like Proximal Policy Optimization (PPO) to align the L.M with human preferences (Schulman et al., 2017). More recent work has explored scal- ing this process by using outcome-bac based rewards. These Techniques such as Group Relative Policy Optimization (GRPO) (Shao al., 2024) and Reinforee (Hu, 2025) are central to this trend, which offer alternatives to traditional PPO that reduce the need for a separate value model or extensive data (Ahmadian et al., 2024; Yu et al., 2025b).
Reinforcement Learning in LLMs.
B ADDITIONAL RESULTS
B. COMPARISON AGAINST MORE BASELINES
We also conduct comparisons with a broader set of long-context models beyond 7B level. The baselines include recent Qwen3 models (Yang et al., 2025a), 14B variant of Qwen2.5-IM (Yang et al., 2025b) and RI RI-Distill-Qwen (DeepSeek-AI et al., 2025), and the 32B long-context LLM QwenLong-Li
(Wan al., 2025).
Table reports the extended comparison on both ID and OOD settings. In the table, we observe: (1) At high context lengths, ReMemR1 outperforms long-context LLMSs that are four times larger. On HotpotQA, ReMemR1 achieves 80.8% accuracy at 6400 documents, substantially higher than QwenLong-LI-32B (38.3%) and 14B-level RI-Distill-Qwen (31.3%). Similarly, on 2WikiMulti- HopQA, ReMemRI reaches 50.3% accuracy at 6400 documents, outperforming QwenLong-Ll- 32B (29.9%) and R RI-Distill-Qwenn 14B (32%). This highlights ReMemRI robustness under ex- treme context scaling. (2) At mid-range context lengths (200-800 documents), ReMemR1 remains highly competitive. For example, on HotpotQA at 400 documents, ReMemR1 (78.9%) surpasses QwenLLong-LL1-328 (73.4%) and all other baselines.
16

---

Published as a conference paper at ICLR 2026
Table Extended long-context QA results on HotpoiQA (Yang al., 2018) and 2WikiMullHHopOA (Ho et al., 2020). Values are accuracy (%), rounded to decimal.
(a) Accuracy HotpotQA (In-Distribution)
Number of Context Documents
Scale
Method
50
100
200
400
800
1600
3200
6400
Qwen3-4B (Yang al., 2025a)
75.0
75.8
63.3
60.2
21.9
18.8
CB
ReMemR1 (Qwen2.5-3B)
70.9
71.7
63.8
74.0
65.4
65.0
65.4
Qwen3-8B (Yang et al., 2025a)
81.3
78.9
71.9
70.3
74.2
33.6
23.4
19.5
RI-Distill- Distill-Qwen-T7 Qwen-7B (DeepSeek-Al at., 2025)
40.6
25.8
1.6
2.3
RI-Distill-Qwen--4B (DeepScck-Al at., 2025)
79.7
76.6
64.1
57.8
40.6
33.6
20.3
31.3
>7B
Qwen2.5- 1een2.5-1M-70- Yang et (2025b)
75.8
71.9
68.0
67.2
69.5
54.7
22.7
0.0
Qwen2.5-1 14B Yang et al. (2025b)
78.1
83.6
76.6
73.4
70.3
60.9
42.2
0.0
QwenLoog-LL LL-32B Wan et al. (2025)
83.6
85.2
74.2
73.4
57.8
45.3
38.9
38.3
ReMemRI (Qwen2.5-7B)
82.3
82.8
81.1
78.9
82.0
79.7
80.0
80.8
(b) Accuracy on 2WikiMultHHopQA Dout-Of-Distriuttionn
Number of Context Documents
Scale
Method
50
100
200
400
800
1600
3200
6400
Qwen3-4B (Yang 2025a)
60.9
43,0
32.0
25.0
21.1
<7B
ReMemR1 (Qwen2.5-3B)
53.5
50.4
42.5
41.7
37.0
36.2
35.4
37.8
Qwen3-8B (Yang et al., 2025a)
67.2
60.9
57.0
51.6
49.2
25.8
26.6
31.3
RI-Distill- Qwen-7B (DeepScek-AA al., 2025)
36.7
29.7
25.8
0.8
2.3
2.3
E1-Distill-Qwenn-ee (DeepSeck-Al al., 2025)
71.9
57.8
52.3
42.2
28.1
29.7
28.1
32,0
37B
Dwen2.5-1M-77 (Yang al., 2025b)
62.5
59.4
57.8
47.7
46.1
45.3
25.8
0.0
Qwen2.5-1M-144 M--14BBB (Yang et al., 2025b)
58.6
56.3
56.3
49.2
47.7
45.3
34.4
0.0
Qwenl Lon-T.L--g (Wan et al., 2025)
74,2
69.5
65.6
58.6
38.3
28.1
24.6
29.9
ReMemR1 (Qwen2.5-7B)
63.9
63.1
55.6
54.5
54.7
45.4
48.9
50.3
Table 6: Ablation RL training. We report accuracy (%) HotpotQA and 2WikiMultiHHpQA with and without RI. The based models are Qwen2.5-3B Instruct.
Number of Context Documents
Benchmark
Method
Setting
50
400
800
1600
3200
6400
MemAgent
w/o RL
60.2
47.7
35.9
28.9
24.2
23.4
14.8
14.1
ReMemR1
w/o RL
35.4
40.9
31.5
25.2
26.0
24.4
16.5
20.5
HotpotQA
MemAgent
70.3
69.4
68.8
60.9
60.2
59.4
58.8
ReMemR1
70.9
74.0
65.4
65.0
65.4
66.1
MemAgent
w/o RL
37.5
30.5
32.0
22.7
16.4
16.4
16.4
15.6
ReMemR1
w/o RL
26.0
25.2
26.8
18.9
16.5
17.3
22.8
22.0
2WikiMultHHoPAA
MemAgent
41.4
42.2
38.3
26.7
25.9
ReMemR1
53.5
41.7
37.0
36.2
35.4
37.8
B.2 IMPACT OF RL TRAINING
We further examine the impact of reinforcement learning on long-context reasoning. Table 6 com- pares model performance with (w/) and without (w/o) RL across different numbers of context doc- uments, where all methods use Qwen2.5-3B Instruct (Yang et al., 2024) the foundational model. Without RL, both our method and MemAgent suffer from sharp performance drops as con- text length grows, indicating difficulties in optimizing with only supervised signals. Introducing RL substantially improves accuracy on both HotpotQA and 2WikiMultHHppA.. In particular, our method with RL consistently achieves the highest scores across most context lengths, outperforming MemAgent by a clear margin.
We also observe that without RL training, the two paradigms (MemAgent and ReMemR1) shows different behavior at different context length levels:
800 Documents. When the context length relatively small, directly applying Qwen-3B on ReMemR1 without RL shows lower accuracies than MemAgent. We find out this phenomenon
17

---

Published as a conference paper at ICLR 2026
(a) Outcome Reward
Action Reward
(c) Reward
1.0
0.6
0.5
0.5
0.9
0.0
0.3
0.3
0.7
0.2
0.6
0.1
0
160
150
200
0
50
100
150
200
50
100
150
200
Epochs
Epochs
Epochs
Figure 7: Training dynamics of our method. ReMemRI enables the LLM to generate both inner memory and callback queries, introducing additional formatting requirements. These constraints initially lead a lower success rate due to frequent parsing errors, but performance rapidly improves during the first 20 steps as the model quickly learns to follow the required format.
caused by the imperfect Iestruction-follwwee in the untrained model. As the callback mech- anism provides opportunity to include more information. also introduces additional format requirements. According to Figure 7, the 3B-level LLM begins with around 0.6 average format reward, which means the LLM fail extract the updated memory for 40% steps. As the training processes and the format reward grows, ReMemRI quickly learns the format requirements under the guidance of action-level rewards, resulting in quickly increasing early-stage rewards.
800 Documents. As the context length raises to more than 800 documents, ReMemRI shows slower accuracy drop. resulting in about 6% improvements on both benchmarks. This observation concurs with the findings in Section 3.5.1, where rule-based callback yields better long-horizon performances, which validates the benefits of callback mechanism in preventing long-term in- formation losses. These results highlight the importance of reinforcement learning in stabilizing training and enabling effective reasoning under long-context settings.
B.3 DETAILED INFLUENCE OF DIFFERENT ALPIIA VALUES
The influence of different values during RL training is shown in Figure 8. Overall, all three settings 1.0, 0.8, 0.5) follow a similar early-stage learning trajectory: the outcome reward rises rapidly during the first 100 steps the model acquires basic formatting ability and coarse coarse-grained reasoning skills.
As training progresses, however, the curves begin to diverge. The model trained with a 0.8 con- sistently achieves the highest outcome reward after convergence. This suggests that incorporating a moderate amount of step-level reward helps address the sparse and noisy credit assignment problem inherent in purely outcome-basee basedd RL. The intermediate signal guides the model toward identifying and reinforcing the steps that meaningfully contribute to producing useful memories.
The = 1.0 setting, which relies solely on outcome reward, converges more slowly and ultimately lower plateau. Without step-level feedback, the model struggles attribute credit individual memory updates, especially when multiple reasoning steps interact. Conversely, tracks the other curves but collapses mid-training due to instability introduced by overly dominant step-level signals its reward becomes overly sensitive noisy intermediate states, leading di- vergence.
0.5 initially
Taken together, these results demonstrate that balanced combination of final-outcome and inter- mediate rewards (c.g., a = 0.8) provides the most stable and effective training dynamics. It offers sufficient step-level guidance stabilize credit assignment, while still grounding optimization in the final-answer correctness that the evaluation metric ultimately cares about.
B.4 ADDITIONAL RESULTS ON COMPUTATIONAL OVERHEAD
This section provides a detailed examination of the computational overhead ReMemR1 during both inference and training.
18

---

Published as a conference paper at ICLR 2026
Parameter Setting
Outcome Reward (S-step Smoothed)
atpra0.
alphw0.5
0.1
500
800
Training Step
Figure 8: Trianing Curve at different values.
Table 7: Full inference- time performance comparison.
Number of Coniest Documents
Canegory
Method
Metric
50
100
200
400
800
1600
3200
64000
MemAgent ReMemRI
Accurney
81.8
78.9
78.9
77.0
79.7
72.1
74,0
75,8
Accuracy
Accuracy
MemAgeut
Time /Sample (s)
14.51
22:41
38 16
69.89
152.62
356.82
676.80
1422. 17
ReMemRI
Time Sample (s)
16.70
26.02
46,90
90.53
211.33
527.85
1004.29
1935.84
RoMemRI
Time (s)
0.27
0.93
ReMemRI
Callback / Toral
0.07%
0.10%
0.14%
0.14%
0.13%
0.08%
0.09%
0.09%
MemAgent
Total Memory (MB)
811.95
818.32
833.18
862.68
924.18
1050 88
1358
1989.44
Memory
ReMemR1
RoMemR]
Callback Memory (MB)
Total Memory (MB)
808.90
0.01
824.80
0.02
0.03
893.44
0.05
0.10
1117.41
0.16
1418.86
0.23
20055,30
0.34
RoMemK1
Callback / Total
40.001% 16
40.00%
40.01%
40001%
<0.001%
<0.001%
40.011%
40001%
B.4.1 INFERENCE- TIME PERFORMANCE
We evaluate the inference-time computational characteristics of ReMemR1 HotpotQA context lengths ranging from 50 to 6400 documents. We report three groups of metrics:
Accuracy
Latency: total inference time per sample, callback time per sample, and the ratio of callback time over total
GPU Memory Usage: total memory consumption, callback memory consumption, and corre- sponding ratios
The full results are shown in Table 7. Several observations emerge:
Retrieval overhead negligible. Although ReMemR1 stores all intermediate memory states, each entry is short model-generated summary. As result, the callback operation contributes fewer than 0.2% of total inference-time latency and less than 0.001 of total GPU memory across all scales.
Accuracy benefits outweigh the cost growth. ReMemR improves accuracy by up to 5% compared with MemAgent, corresponding to 20% reduction in error rate, while introducing only modest computational overhead.
Primary overhead stems from callback-query generation. The additional latency comes primarily from autoregressively generating the <recall> callback query each step, rather than from the retrieval itself.
B.5
CASE STUDY
To qualitatively evaluate the impact of the proposed <recall> mechanism, we conduct a compar- ative study between ReMemR1 and the "memorize while reading" baseline MemAgent. We
19

---

Published as a conference paper at ICLR 2026
analyze challenging multi-hop reasoning sample that requires identifying attributes (death dates) of two distinct entities found in separate document chunks from 2WikiMultHHoQA. Consider the query: "Which film has the director died first, Is There Justice? The Barrier Of Flames?", two-hop question which requires three steps to answer:
identify the directors of both films ( "Stuart Paton and Jack Harvey, respectively"),
retrieve their respective death dates (= "Stuart Paton died on December 16 1944, and Paul
Landres on November 9, 1954"), and
perform a temporal comparison "Stuart Paton died first").
In the following visualizations, we highlight three types of evidence:
Correct Evidence: evidences that are relevant to the question, and successfully recog- nized by the LLM.
Erroncous Evidence: evidences that are not relevant the question over- emphasized by the LLM, incorrect inferences drawn by the LLM.
Ignored Evidence: evidences that are relevant to the question, but overlooked by the LLM.
Additionally, we use bold text to indicate the information that the model adds to its memory at each step.
B.5.1
CASE STUDY OF BASELINE METHOD (MEMAGENT)
MemAgent: Step
Chunk (Document 10): ...Stuart Paton (23 July 1883 16 December 1944) was British director, screenwriter and actor of the silent Hra.He directed 67 films between 1915 and 1938...
Memory 1: .h.The section does provide the directors' names their death dates for There Justice? or Barrier Of Flames. We would need to look additional information to answer this question.
MemAgent: Step
(nothing important in the chunk) ...
Memory ...The section does provide the directors' names or their death dates for There Justice? or The Barrier Of Flames. We would need to look for additional information to answer this question.
MemAgent: Step 3
Chunk (Document 83): ...John JosephHarvey, also known Jack Harvey( September 16, 1881 in Cleveland, Ohio- November 9, 1954 in Las Angeles, California) was American film actor... Chunk (Document 109): ...Is There Justice? 1931 American crime film directed by Stuart Paton... is now considered lost film...
Memory 3: The section provided mentions that Is There Justice? is 1931 American crime film directed by Stuart Paton... However, the section does not provide the directors' names or their death dates... We would need to look for additional information...
20

---

Published as a conference paper at ICLR 2026
MemAgent: Step
Chunk (Document 154): ...The Barrier of Flames is 1914 American short silent drama film, directed by Jack Harvey. It stars Shep the Dog, Helen Badgley... Chunk (Document 158): ...Chris Marker( 29 July 1921 29 July 2012) was a French writer, pho- lographer, documentary film director...
The section provided mentions that Is There Justice? a 1931 American crime film directed by Stuart Paton and starring Rex Lease, Henry B. Walthall and Blanche Mchaffey. It is now considered lost film. The section also mentions that The Barrier of Flames is 1914 American short silent drama film, directed by Jack Harvey. The section does not provide any new information However, we now know The Barrier of Flames a 1914 film, which earlier than There Justice? (1931), and Chris Marker, director who died in 2012, is mentioned. Therefore, we can infer that the director of The Barrier of Flames, Jack Harvey, died before the director of Is There Justice?, Stuart Paton.
Memory 4:
MemAgent: Step
(nothing important in the chunk)
...The section does not provide any new information about the directors of Is There Jus- fice? or The Barrier Of Flames. We would need to look for additional information to answer this question. However, There Justice? (1931), and Chris Marker, a director who died in 2012, mentioned. Therefore, we can infer that the director of The Barrier of Flames, Jack Harvey, died before the director of Is There Justice?, Stuart Paton.
Memory 5:
now know that The Barrier Flames is a 1914 film, which is earlier than Is
MemAgent: Final Step
Which film has the director died first, There Justice? or The Barrier Of Flames? ...The section does not provide any new information about the directors of Is There Justice? or The Barrier Of Flames. We would need to look for additional information to answer this question. However, we now know that The Barrier Flames is 1914 film, which is earlier than Is There Justice? (1931), and Chris Marker, a director who died 2012, mentioned. Therefore, we can infer that the director of The Barrier of Flames, Jack Harvey, died before the director of Is There Justice?, Stuart Paton.
Question:
Latest Memory:
Answer: Therefore, the answer is The Barrier Flames.
As illustrated in the illustration boxes below, MemAgent processes the context purely sequentially. In Step 2, it correctly identifies that Is There Justice? was directed by Stuart Paton. However, because the immediate context chunk does not contain Paton's date of death, MemAgent updates its memory with a passive note: "The section does not provide... death dates." Crucially, as the model proceeds to Step
read about the second film (The Barrier of Flames, directed by Jack Harvey), it suffers from forward-on onny processing constraint. Without mechanism to look back search for the missing data regarding the first director, attempts infer the answer from irrelevant entities present in the current chunk (e.g., confusing the target with a different director mentioned in the text, Chris Marker). Consequently, MemAgent relies on hallucinated reasoning to force a conclusion, and yields incorrect prediction.
The Constraint of Forward-Only Processing The "Forward-Only" limitation prevents the agent from retrospectively attending to past information once the relevance of that information becomes clear in later time step. This is explicitly demonstrated in the disconnect between Step and Step
In Chunk 10 (Step 1), the text explicitly provides the death date of Stuart Paton (16 December 1944). However, the agent fails recognize its relevance at this stage, as it has not yet identified Paton as the director of Is There Justice?.
The Missed Evidence (Step 1):
The Delayed Context (Step 3): It is not until Step 3 (Chunk 109) that the agent learns Is There Justice? was directed by Stuart Paton.
21

---

Published as a conference paper at ICLR 2026
The Inability to Revisit: Because the architecture processes the stream unidirectionally (t + 1), the agent cannot revisit Chunk 10 to extract Paton's ' death date after recog- nizing his relevance in Step 3. This temporal misalignment of evidence leads to critical information gap.
Information Loss through Overwriting The case further demonstrates how the finite memory bottleneck leads to the overwriting of potential knowledge with erroneous inferences. As the agent struggles to maintain coherent narrative across long contexts, valid query states are overwritten by noisy, local correlations.
Accumulated Error (Step 4): the model have already lost the grounded truth about Stuart Paton (due to the forward-only failure), the agent forced to rely on immediate, local context.
By Step the agent's memory
bottleneck. Since
The Overwriting Happens: In Chunk 158, the agent encounters irrelevant information about Chris Marker. Desperate to resolve the query, the agent constructs spurious logical connection, inferring that because Chris Marker died in 2012 and the films have different release dates, comparison can be made.
The Result: This hallucination overwrites the agent's previous state of "searching for in- formation." " The memory transitions from neutral state of missing data confident, false state (Memory 4: w...we can infer that... Jack Harvey, died before... Stuart Paton"). This false memory effectively increases the difficulty of correctly identifying the directors' lifespans, and ultimately leads to the incorrect answer in the Final Step.
B.5.2 CASE STUDY OF REMEMR1
ReMemR1: Step
Chunk (Document 10):
....tuart Paton (23 July 1883 16 December 1944) was British director, screenwriter actor the silent era.He directed films between 1915 and 1938...
Recalled Memory: No memory was recalled.
Memory St.art Paton (23 July 1883 16 December 1944) was a British director, screen- writer and actor of the silent era. He directed 67 films between 1915 and 1938, He also wrote for 24 films between 1914 and 1927...
Callback 1: who's 's the director of Is There Justice? and The Barrier Of Flames?
ReMemR Step 2
... (nothing important in the chunk)
Recalled Memory (who's the director of There Justice? and The Barrier Of Flames?): ....ttrrt December 1944) was a British director, screenwriter and actor silent
Paton (23 July 1883
era...
Memory 2: St.art Paton (23 July 1883 16 December 1944) was a British director, screenwriter and actor of the silent era. He directed 67 films between 1915 and 1938. He also wrote for 24 films hetween 1914 and 1927... Callback who's the director There Justice? and The Barrier Of Flames?
22

---

Published as conference paper at ICLR 2026
ReMemR1: Step
Recalled Memory (who's the director of There Justice? and The Barrier Of Flames?): ...Stuart Paton (23 July 1883 December 1944) was a British director, screenwriter and actor silent
era...
Chunk (Document 83): John Josephllarvey, also known Jack Harvey( September 16, 1881 in Cleveland, Ohio- November 9, 1954 in Los Angeles, California) Chunk (Document 109): ...Is There Justice? 1931 American crime film directed by Stuart Paton... is now considered lost film...
American film actor...
Memory 3: Stuart Paton (23 July 1883 16 December 1944) was a British director, screenwriter and actor of the silent era. He directed 67 films between 1915 and 1938. He also wrote for 24 films between 1914 and 1927...ls There Justice? is 1931 American crime film directed Stuart Paton and starring Rex Lease, Henry B. Walthall and Blanche Mchaffcy...Jackk Harvey( September 16, 1881 Cleveland, Ohio- November 9, 1954 in Los Angeles, California) was an American film actor Callback 3: who's s the director of The Barrier Of Flames?
ReMemR1: 1: Step 4
Recalled Memory (who's the director of The Barrier Of Flames?): ...Stuart Paton (23 July 1883 16 December 1944) was British director, screenwriter and actor the silent era...
Chunk (Document 154): ...The Barrier of Flames is a 1914 American short silent drama film, directed by Jack Harvey. It stars Shep the Dog, Hclen Badgley...
Chunk (Document 158): ...Crris Marken 29 July 1921 29 July 2012) a French writer, photog rapher, documentary film director...
Memory 4: Stuart Paton (23 July 1883 16 December 1944) was British director, screenwriter and actor of the silent He directed 67 films between 1915 and 1938. He also wrote for 24 films between 1914 and 1927...ls There Justice? a and starring Rex Lease, Henry B. Walthall and Blanche Mehaffey...Tle Barrier of Flames a 1914 American short silent drama film, directed by Jack Harvey... Callback 4: who's i the director of The Barrier Of Flames?
American crime film directed by Stuart Paton
ReMemRI: Step 5
(nothing important in the chunk)
Recalled Memory (who's 's the director of The Barrier Of Flames?): ...The Barrier of Flames is 1914 American short silent drama film, directed by Jack Harvey...
December 1944) was British director, screenwriter and actor the silent era. He directed 67 films between 1915 and 1938. He also wrote for 24 films between 1914 and 1927...ls There Justice? is a 1931 American crime film directed by Stuart Paton and starring Rex Lease, Henry B. Walthall and Blanche Mchaffey...The Barrier of Flames 1914 American short silent drama film, directed by Jack Harvey...Jack Harvey (born 1883 died 1954) was an American director, screenwriter, and actor...
Memory Stuart Paton (23 July 1883
Callback 5: who's 's the director of The Barrier Of Flames?
ReMemR Final Step
Question: Which film has the director died first, There Justice? The Barrier Of Flames? Recalled Memory (who's the director of The Barrier Of Flames? ...The Barrier of Flames is 1914 American short silent drama film, directed by Jack Harvey..
Latest Memory: Stuart Paton (23 July 1883 - 16 December 1944) was a British director, screenwriter and actor of the silent era. He directed 67 films between 1915 and 1938. Нe also wrote for 24 films between 1914 and 1927...lv There Justice?
1931 American crime film directed Stuart Paton starring Rex Lease, Henry Walthall Blanche Mehaffey... Barrier of Flames a American short silent drama film, directed by Jack Harvey...Jack Harvey (bom 1883 died 1954) was an American director, screenwriter, and actor...
Answer: From memory, we can see Stuart Paton, director There Justice?, died 1944. Jack Harvey, director of The Barrier Of Flames, in 1954, Therefore, Stuart Paton died first. Is There Justice?
23

---

Published as a conference paper at ICLR 2026
As demonstrated in the inference case, ReMemRI successfully identifies the correct answer (Is There Justice?) by employing callback-enhanced memory and non-linear reasoning. These mecha- nisms directly address the structural failures observed in the baseline model.
Overcoming "Forward-Only" Constraints via Callbacks Unlike the baseline, which discarded Stuart Paton's s death date in Step I because its relevance was not yet known, ReMemRI preserves this critical evidence through its active Callback Mechanism.
Proactive Retention (Step I): In Step 1, even though the connection to the film Is There Justice? has not yet been established, ReMemR1 records Stuart Paton's biographical data in Memory 1. Crucially, it generates Callback query: "who's s the director of Is There Justice?... This signals to the model that information related to this director may become relevant later.
Non-Linear Bridging (Step 3): When the agent finally encounters the film title in Step 3, it does not need to backtrack to the raw text of Step 1. Instead, the relevant information (Paton's death date) bridged naturally from the memory history. The agent instantly links the new evidence (Film A = Paton) with the retained evidence (Paton died 1944), effectively bypassing the limitations of forward-only processing.
Mitigating Information Loss via Selective Retrieval The baseline model suffer from "memory overwriting." where early facts are overwritten by later, irrelevant noise (e.g., the Chris Marker hallucination). ReMemR1 prevents this through Selective Retrieval.
Robust State Maintenance (Step 4 & 5): Instead of relying on single, degradable mem- ory state, ReMemR1 I utilizes a retrieval mechanism. In Step 4, the Recalled Memory field explicitly retrieves the previously stored facts about Stuart Paton while simultaneously pro- cessing the new facts about Jack Harvey.
Noise Filtering: By selectively recalling only the data relevant to the active Callbacks, ReMemR1 filters out the noise that confused the baseline. It ignores the irrelevant "Chris Marker" segment once the segment is passed, and focuses on the verified facts about both directors.
In the end of the inference (e.g., s. final step), ReMemR1 successfully synthesizes evidence across dis- tant timesteps (Step and Step 3) in the reversed order, and ultimately reaches the correct inference: Stuart Paton (died 1944) died before Jack Harvey.
B.5.3 FAILURE ANALYSIS OF REMEMRI
In this section, conduct analysis two error cases ReMemR1 have made on HotpotQA to reveal specific vulnerabilities its recall query generation and memory update policies.
Failure Pattern 1: Recall Mechanism Collapse In this failure mode, the agent fails to generate contextually relevant queries when faced with information gaps. Instead of formulating targeted question to retrieve missing information, the model falls back to irrelevant queries (c.g., asking about the US President, which is the example used in the system prompt).
Consequently, the socualld__eemryy field is not populated with relevant historical context that could link "Liberal Conservative" (the ground truth style) with the specific party names found earlier. The model ultimately reaches final answer that focuses the name ("People's Party") rather than the ideology. This could be partly because the recall mechanism failed to retrieve the specific semantic constraints requested by the problem.
24

---

Published as a conference paper at ICLR 2026
Failure Case 1: Recall Degeneration
Problem:
What kind political party that combined conservative policies with liberal stances Pio Cabanillas Gallas and Jose Maria Areilza the leaders of?
Step Recall: who's Pio Cabanillas Gallas and what political party did he lead? (Relevvntt)
Chunk 2: ...Spanish Social Reform... dissolved in 1977... Manuel Cantarero would join the Liberal
Citizens Action...
Memory 2: ...Long paste- of document text regarding Spanish Social Reform and United National
Party...
Callback <recall>who's 's the president the United States?</recall> (Failure)
Chunk 3 (Doc 91): ...People's Party (Spanish: "Partido Popular"; PP) was Spanish liberal conservative political party... The leaders of the PP were Pío Cabanillas Gallas and José María de Areilza.
Callback <recallswho's the president of the United States?</recal>> (Failure)
Final Answer: People's Party
Ground Truth: Liberal Conservative
Failure Pattern Memory Polution The second pattern illustrates a first impression bias, which results in "Memory Polution Mechanism". If ReMemR1 hallucinates misinterprets information in an early step and writes it into memory, this "poluted" memory could appear over and over again in the later steps, thus overrides correct evidence encountered. The agent fails to the memory overwrite nor memory callback mechanism to correct previous falsehoods.
In this case, the model misinterprets list of cars used in race list of cars manufactured by Holden. This hallucination was immediately written into the new memory update. In the following step the model explicitly read Document 92 which lists the correct models (sedan, station wagon). However, instead of performing a corrective update to remove the false Ford/Toyote claim, it simply appends the new text. When generating the final answer in the end, the model prioritizes the incorrect "manufactures Ford and Toyota" statement it generated earlier over the implicit correct evidence found later.
Failure Case Memory Polution
This Australian racing driver currently drives the No. 22 Holden VF Commodore produced by Holden Commodore which manufactures what other models? (Ground Truth: sedan and station wagon)
Problem:
Chunk 1: .heThe Virgin Australia Supercars Championship is motorsport series in Australia that uses Holden VF Commodore, Ford Falcon, and Toyota Supra models... Memory 1: .heThe Holden VF Commodore
model produced by Holden Commodore, which also manufactures the Ford Falcon and Toyota Supra models.
Chunk 3: The Holden Commodore (VF)... Its range includes the sedan and station wagon vari- ants...
Memory 3: The Holden VF Commodore is model produced by Holden Commodore, which also manufactures the Ford Falcon and Toyota Supra models... Its range includes the sedan and station wagon variants that sell under the luxury Holden Calais (VF) nameplatc Callback who's the president of the United States? (Irrelevant)
Final Answer: Ford Falcon, Toyota Supra Ground Truth: Sedan and Station Wagon
25

---

Published as a conference paper at ICLR 2026
C IMPLEMENTATION DETAILS
C.I FULL EXPRESSION OF TRAINING OBJECTIVE
Our model is optimized with a variant of GRPO objective. The full expression of our training objective can be written as:
argmax (f) = (100)))
min
Pi,
(n)
A
G(T + 1] g=1
(10)
clip Pi, e, + A(5)
Dx (OO Tredl
(9) where Pt.e is the importance sampling ratio:
TH
S1, c,
(g) (9) (a)
321
Q, ct-1)
Pt,
(g)
T8
(9)
$22 S,
(0)
Q, Ct-1
(11)
Here, S,, denotes the i-th token the 1-th state trajectory g, e clipping ratio, 3 the KL coefficient, and A(s) is the normalized advantage. We assume eT Ø for notational convenience.
(9)
C.2 TRAINING HYPERPARAMETERS
The training of ReMemR1 I was built upon the verl' framework, with efficient trajectory generation powered by the sglang* engine. We employed Fully Sharded Data Parallelism (FSDP) for distributed training, and used bfloat precision for both training and evaluation. Table 8 summarizes the primary hyperparameters used in our method.
Although we evaluated the model with varying numbers of context documents during testing, the training setup consistently used 200 documents per sample, resulting in approximately 30K input tokens. Each document chunk was limited to maximum length of 5000 tokens, yielding T ~ 6 during training. At each timestep and at the final state, the model generated rollouts with a temper- ature of 1, up maximum of 2048 tokens.
The 3B version ReMemR1 and its variants are trained on 16 H800 GPUs and converge after 100 hours. The 7B model is trained on 32 H800 GPUs, reaching convergence after 80 hours.
Table 8: Primary hyperparameters used in training.
Hyper-parameter
Value
Training Batch Size
128
Micro Training Batch Size
8
Total Converge Steps
200~300
Actor Model Learning Rate
1 10-6
Actor Model Warmup Steps
20
Rollout Temperature
Max Chunk Length
5000
Training Chunk Number
Max Response Length
2048
KL Coefficient
0.001
Clip Ratio é
0.2
Group Size G
phtttps://gitie thub. com/ com/voolceniine/verl Phttps: s/ttps:://iiii thub. cwm/ http:////loooooc////ggeeeoooe
26

---

Published as a conference paper at ICLR 2026
C.3 EVALUATION SETTINGS
To ensure the challenging nature of the samples, we only use samples from the hard difficulty level for training. Questions in these datasets typically require at least two pieces of evidence to answer, and there exist dependencies between the evidence. Due to the extraordinary computational cost of long-context QA, we subsample 128 samples from each benchmarks with random seed of 4, following Yu et al. (2025a).
D COMPLEXITY ANALYSIS
this section, we analyze the computational complexity of ReMemRI and show that preserves the linear complexity of conventional memory-agent approaches.
D.1 BASELINE COMPLEXITY
In the "memorize while reading" paradigm, the agent processes sequence of document chunks {cr -2, .r) in order. At each step it updates the memory via:
m4+1 xQ2, ca m²).
(12)
Each update requires ((1) memory operations and a constant number of forward passes through the policy network. Thus, the overall time complexity O(T). The space requirement is summation of the document chunks and the memory at each step, which is O(T + 1) = O(T) in total.
D.2 COMPLEXITY OF REMEMRI
ReMemR1 augments the state by including a query component dr and retrieval function past memories:
St+1 (m--1. = π (Q,, et, m,, E(mm)ic 9:)).
(13)
This paradigm also performs the same number of state transition, which is ((n) times of LLM generation. Compared to Eq. 12, our method includes two sources of computational overhead:
Storage of previous memories. Although the state transition references {milign. each my is itself fixed-length vector (e.g., the hidden state of the model). Maintaining this list across T steps requires (TT) additional space. This is the same order storing the original text chunks, but with smaller constant term.
Retrieval operation The retrieval function computes similarity between qe and past memory states. If implemented with exact maximum similarity search over {m, [m, the cost per step could be Ott). However, in practice, we use lightweight recall-based heuristics or an index that supports sublinear approximate nearest neighbor search. This operation is negligible compared against the consumption of the state transition model иTs which often a 7B level L.M.M Thus, the total cost across 7' steps remains O(T) in expectation.
Therefore, ReMemR11 preserves the same asymptotic OTT) time and O space complexity as the conventional memory-agen paradigm, while substantially enhancing the agent's 's ability to perform non-linear reasoning through retrieval.
E PROMPT TEMPLATE
We use separate prompt templates for the generation of intermediate states $ sigiET and the final states 87+1. The prompts are listed below:
27

---

Published as a conference paper at ICLR 2026
Prompt Template for Intermediate States.
You presented with problem, section of article that may contain the the problem, a previous memory. You should generate a response the following format: Output your thinking process in <thinking2your. htrp@_youu_tttkiinngge thinkiing_process//// provided section carefully and update the memory with the new information that helps to answer the problem in only one <update>the, sepuddtttee_uuddeeeded www.tpddpp....... memory</update: action. Be sure to retain all relevant details from the previous memory while adding any new, useful information.
wtt.sppoossssthteeeeeeeeeee siccsss/thinking>..
Read the
you notice partial key evidence that not enough to answer the problem, also output only one peeccal>query/// recall> (e.g. "<recall>who's 's the president of the United States<<rreaall>)) to retrieve information in previous memories.
<problem> QUESTION </problem>
<recalled memory>> RECALLED MEMORY </recalled memory>
<memory> MEMORY </memory> <section> DOCUMENT CHUNK </section>
Updated memory:
Prompt Template for Final States.
You are presented with problem a previous memory. Please answer the problem based on previous memory and put the answer in (boxed[).
<problem> QUESTION </problem>
<recalled memory> RECALLED MEMORY </recalled memory> <memory> MEMORY </memory>
Your answer:
F THE USE OF LARGE LANGUAGE MODELS
In the preparation of this manuscript, utilized LLM as writing assistance. The of the LLM was limited to proofreading for grammatical errors, checking for typos, and improving the clarity and readability of existing text. The LLM not used for any core intellectual contributions, including but not limited to research ideation, formulation of the methodology, analysis of results, drafting of the original manuscript. All scientific claims, arguments, and the final text the sole work of the human authors, who pay full responsibility for all content.
28


Reviews:
[{"review_id": "7sYy06B83V", "reviewer": "Reviewer_1HMb", "summary": "This paper addresses the challenge of question-answering over long contexts, in which large language models (LLMs) typically utilize a memory corpus to summarize a single pass over the context in order to answer the question. This approach, which uses only forward processing, is storage-efficient but often suboptimal, as the importance of some parts of the context may only be apparent after scanning the initial context. This paper proposes to mitigate this challenge by appending a query and retrieval function to the agent memory for each document chunk. This query can retrieve information from prior chunks, allowing them to be “revisited.” The paper further introduces chunk-level rewards that measure the information gain in memory updates to encourage retention of new information relevant to the ground truth, using reinforcement learning to learn the memories and retrievals for each document chunk. Experiments on multiple short and long context datasets show that the proposed ReMemR1 technique outperforms MemAgent as well as existing standard and long-context LLMs.", "questions": "Please see also Weaknesses above.\n\n1) What is the tradeoff introduced by varying alpha? I understand that a smaller value of alpha corresponds to more emphasis on step-based rewards, but conceptually, since the evaluation target is the final accuracy, the optimal value of alpha should be alpha = 1, which emphasizes only the final reward. Thus, the paper should better justify the need for step-based rewards. Given enough training iterations, will alpha=1 eventually converge to a better (i.e., higher accuracy) solution?\n\n2) Why does ReMemR1 use the relatively simple word similarity as a reward in the step-based rewards? Would a more sophisticated information gain measure (e.g., one that evaluates the semantics) work better?\n\n3) The experiment section also includes an ablation study of sorts that compares ReMemR1 to that of MemAgent with a rule-based callback, showing that ReMemR1’s learned callbacks outperform a rule-based one. It is not clear, however, whether this is a true ablation study on ReMemR1; is MemAgent equivalent to ReMemR1 without revisitable memory? It’s also not clear why MemAgent + rule-based callback often performs worse than MemAgent itself in Table 3.\n\n4) In Figure 5 of Appendix B, it’s not clear why the paper states that performance rapidly improves around epoch 20. There seems instead to be a smoothly decreasing increase in performance across the training duration.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The experiment results show a clear gain in performance relative to the baselines for both in-distribution and out-of-distribution test datasets, for most document context lengths. The accuracy of ReMemR1 is also fairly consistent as the number of context documents increases.\n\n+ The proposed idea of using learned queries to revisit past context information, while simple to conceptualize, is intuitively likely to work. It also appears to use about the same amount of memory as the usual single-pass approaches (though I’d appreciate if the authors could confirm this explicitly).", "weaknesses": "--It seems like the presence of learned queries in ReMemR1 would increase the LLM inference time, as the model would need to query past context while making an inference. How much more time would this take in practice (and if I am incorrect and the inference time is comparable to that of single-pass memory agents, can the authors explain why?)\n\n-- The experiments pad the contexts of both HotPotQA and 2WikiMultiHopQA with random documents. This would seem to give an unrealistic picture of true long-context documents, which would very likely have different patterns from random. Indeed, it seems that the random padding would simply lead to the model learning not to query these random documents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yy6tNUHLNK", "reviewer": "Reviewer_K85P", "summary": "This paper introduces ReMemR1, an LLM training paradigm designed to overcome memory-related challenges in long-context QA. Existing long-context methods process documents in a single forward pass, while memory agents may result in irreversible information loss due to the linear processing of memories. In contrast, ReMemR1 addresses this by using callback-enhanced memory, in which the agent generates a callback query each step to selectively retrieve and revisit information from its previous memories and thereby enabling non-linear reasoning. To train this agent effectively, the authors propose RLMLR and combines sparse outcome-based rewards with dense, step-level rewards based on the \"information gain\" in memory updates and callback retrievals. Experiments show that the proposed ReMemR1 can outperforms existing memory-based approaches and long-context alternatives.", "questions": "1. On the RL training level, is each state treated as one training instance in RLMLR?\n\n2. What is the precise format of the memory? Can you provide more qualitative examples of the memories and the generated queries?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method allows memory agents to selectively retrieve information from its previous memories. This counters the potential information loss in standard memory agent methods and enables non-linear reasoning paths.\n\n2. The propsoed RLMLR introduces a RL-learning framework with multi-level reward design. This provides more dense and step-level rewards for each states in the reasoning trajectory, which offers better guidance than RL methods that solely rely on the final outcome.\n\n3. The authors perform extensive experiments to show that ReMemR1 can outperfor memory agents and lon-context LLMs in open-domain QA benchmarks. They also highlight the performance gains on out-of-distribution datasets.", "weaknesses": "1. Limited datasets / baselines. For open-domain QA benchmarks, the authors should also consider one-hop datasets like triviaQA, natural questions and further multi-hop questions like bamboogle, musique. Further long-context baselines like IterDRAG and search agents like Search-R1 should also be considered as baselines.\n\n2. The method requires storing the entire history of all previous memory states over many retrieved document chunks. For each step, the LLM then perofrms a retrieval search across this history and generate new states / memory or final answer. This adds additional compute, memory overhead and latency compared to standard single-pass methods or search agents.\n\n3. The authors adopt a very simple retrieval function based on word overlapping, other alternative methods like embedding-based retrieval is not discussed. Furthermore, the proposed method is designed to return one single past memory with the maximum word overlapping. This could be a limitation preventing the agent from synthesizing information across multiple documents / reasoning steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FNGvA12qi1", "reviewer": "Reviewer_EGXY", "summary": "The paper presents a novel approach for long-text QA comprised of two components. The first component is a memory-augmented agent that stores multiple historical memories while processing the long text rather than rewriting a single memory state. It allows the agent to revisit past memories via query-based retrieval, enabling non-linear reasoning paths.  The second component is a modification of GRPO with a detailed reward function that combines traditional outcome rewards and step-level state rewards to improve the relevance of the generated reasoning sequences. The proposed method is evaluated on two MHQA datasets and shows improvement over vanilla model and a MemAgent baseline.", "questions": "1. How does the memory overhead scale with task complexity and document length compared to the \"memorize while reading\" paradigm?\n2. What are the typical failure modes of the proposed retrieval mechanism?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper provides a clear motivation for addressing the limitations of \"memorize while reading\" paradigm. \n2. The experimental results demonstrate consistent improvements across multiple model sizes and varying context lengths. \n3. The proposed RLMLR approach is clearly described.", "weaknesses": "1. Poor figure presentation. Throughout the paper, figures are consistently misaligned with their textual references, which significantly disrupts the reading flow and makes it difficult to follow the arguments.\n2. The memory efficiency of the proposed method compared to the \"memorize while reading\" baseline is not addressed. The proposed approach requires storing multiple historical memory states, and it is not discussed how such large memory storage affects inference time and cost.\n3. The statement about per-step reward innovation should be clarified. Per-step rewards are a common practice in RL. \n4. The paper does not provide the callback mechanism failures analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HDi2i2aTNO", "reviewer": "Reviewer_VkNG", "summary": "This paper presents ReMemR1, a memory-augmented large language model (LLM) agent for long-context question answering (QA). The key innovation is a callback-enhanced memory mechanism that enables the model to revisit and retrieve earlier memory states instead of relying solely on a forward-only “memorize while reading” paradigm. To address the sparse reward issue commonly faced in reinforcement learning (RL)-based memory agents, the authors propose Reinforcement Learning with Multi-Level Rewards (RLMLR), which combines trajectory-level (final-answer) and step-level (intermediate) rewards. Experiments on two datasets demonstrate consistent improvements over strong baselines such as MemAgent and Qwen2.5, particularly under long-context and distant-evidence settings.", "questions": "1. How does ReMemR1 perform when the memory corpus exceeds the retrieval limit or available storage capacity? Or how does other methods enlarge their memory storage?\n\n2. Does the system employ any pruning or compression strategy to prevent unbounded memory growth?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written, well-organized, and easy to follow, which facilitates understanding of the technical contributions.\n    \n- The proposed method achieves consistent gains on both in-distribution and out-of-distribution datasets, with notable advantages in challenging long-context scenarios.\n    \n- The idea of non-linear reasoning via callback-based memory is conceptually interesting and potentially useful for improving long-context reasoning.", "weaknesses": "- While the callback mechanism is intuitively appealing, it can be viewed as an incremental extension of existing memory-retrieval or recurrent-state models. The distinction from prior works (e.g., MemAgent) is not sufficiently emphasized.\n    \n- The paper provides limited formal or analytical insight into *why* the proposed retrieval and multi-level reward structure lead to improved reasoning or generalization.\n    \n- The authors argue that existing memory-based agents suffer from irreversible forward-only processing and information loss because they cannot revisit previous states. However, this argument is not entirely convincing. When the policy updates its memory at each step, it already considers both the previous memory and the current input (query or document chunk), which inherently allows recursive integration of prior information. It is unclear why an additional “memory recall” mechanism is strictly necessary, rather than improving the existing update function to retain or reweight key information.\n    \n- The proposed design retrieves relevant content from all previous memory states, which implies that the model must maintain access to the entire memory history. This could be computationally expensive and may not scale to extremely long contexts. The paper should clarify whether ReMemR1 explicitly stores all intermediate memories or uses a compressed or differentiable representation to manage memory growth.\n    \n- Important implementation aspects, such as retrieval efficiency, computational overhead, and inference latency, are under-discussed. It remains unclear how the proposed approach scales in both training and inference compared to other memory-based methods.\n\n- Additional baselines, especially recent long-context or memory-augmented LLMs, would make the evaluation more comprehensive and convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)