MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents
Authors: Zijian Zhou
Ao Qu
Zhaoxuan Wu
Sunghwan Kim
Alok Prakash
Daniela Rus
Bryan Kian Hsiang Low
Paul Pu Liang
Venue: ICLR 2026 Poster (2026)
Keywords: deep research
reasoning
context compression
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a RL-based learning framework that enables interactive agents to solve long-horizon tasks with constant context

Abstract: Modern language agents often need to solve long-horizon tasks requiring multiple turns of interactions with the environment, where they retrieve external information, adapt to observations, and answer interdependent queries. Yet, most LLM systems rely on full-context prompting, appending all past turns regardless of their relevance. This leads to un-bounded memory growth, increased computational costs, and degraded reasoning performance on out-of-distribution input lengths due to LLM forgetting the context. We introduce MEM1, an end-to-end reinforcement learning framework that enables agents to operate with near constant context size when solving long-horizon tasks. At each turn, MEM1 updates a compact shared internal state that jointly supports memory consolidation and reasoning. Leveraging reinforcement learning (RL) and rollout trajectory truncation, we train a MEM1 agent to develop internal states that integrate prior memory with new observations from the environment while strategically discarding irrelevant or redundant information. Experiments across three domains, including internal retrieval QA, open-domain web QA, and multi-turn web shopping, show that MEM1-7B improves performance by 3.5$\times$ while reducing memory usage by 3.7$\times$ compared to Qwen2.5-14B-Instruct on an augmented multi-hop QA dataset with 16 objectives in each task, and generalizes beyond the training horizon. Our results demonstrate the promise of reasoning-driven memory consolidation as a scalable alternative to existing solutions for training long-horizon task-solving agents that involve multiple interactions, where both efficiency and performance are optimized. Code can be found at https://github.com/MIT-MI/MEM1.

Published as a conference paper at ICLR 2026
MEM1: LEARNING TO SYNERGIZE MEMORY AND REASONING FOR EFFICIENT LONG-HORIZON AGENTS
Zijian Zhou Ao Qu Zhaoxuan Wu', Sunghwan Kim*, Alok Prakash' Daniela Rus Bryan Kian Hsiang Low Paul Pu Liang
SSingapore-Ma MI' Alliance for Research and Technology Centre, Singapore
Department of Computer Science, National University of Singapore, Singapore 3 husssa Institute of Technology, USA
4 Yonsei University, South Korea
(zhou_zi pi j jinn, zhaoxuan| ) @u. mus edu, lowkh@comp. nus. edu.ss
quua, polaapppinng @mit edu, alok. pwakasrrsaaaaaeaa.aa mit edu, rus@csail mit edu k kimsh8564@yonset ac.
ABSTRACT
Modern language agents often need to solve long-horizon tasks requiring multiple turns of interactions with the environment, where they retrieve external information, adapt to observations, and answer interdependent queries. Yet, most LLM systems rely on full-context prompting, appending all past turns regardless of their relevance. This leads to unbounded memory growth, increased computational costs, and degraded reasoning performance out-of-distribution input lengths due to LLM forgetting the context. We introduce MEM1, an end-to-end reinforcement learning framework that enables agents operate with nearly near constant context size when solving Long-horizons tasks. At each turn, MEM1 1 updates compact shared internal state that jointly supports memory consolidation and reasoning. Leveraging reinforcement learning (RL) and rollout trajectory truncation, we train MEM1 agent to develop internal states that integrate prior memory with new observations from the environment while strategically discarding irrelevant or redundant information. Experiments across three domains, including internal retrieval QA, open-domain web QA, and multi-turn web shopping, show that MEM1 -BB improves performance by 3.5x while reducing memory usage by 3.7 x compared to Swen22.511BB Instructt objectives in each task, and generalizes beyond the training horizon. Our results demonstrate the promise of reasoning-drived memory consolidation alternative to existing solutions for training long- long-horizon task-solving agents with multiple interactions, where both efficiency and performance are optimized. Code can be found at https: /github. com/ com/MIT /MIT SELTTMMI/MEEEE /MMM..
augmented multi-hop QA dataset with 16
scalable
INTRODUCTION
Large language models (LLMs) have shown remarkable performance in single-turn tasks such as question answering, summarization, and code generation (Brown et al., 2020; Touvron et al., 2023). However, emerging real-world applications increasingly operate over multiple turns-searching documents, interacting with environments (Zhou al., 2024), and making decisions based evolving external information (Wang et al., 2024). Examples include AI search agents such as Perplexity AI (Perplexity AI, 2024) that automate complex tasks by iteratively gathering information, and web-navigatton agents such as OpenManus (OpenManus, 2025) and BrowserUse (Müller & Zunič, 2024), which must complete goals across dozens of interactive turns.
Unlike traditional tasks where the input is static self-contained, long-horizon settings often involve answering sequence of related questions, requiring the agent to continuously retrieve new
* Equal contribution. Correspondence: zhou_zi jia_iiiianu mus. edu, qua@mi e.d

---

Published as a conference paper at ICLR 2026
information, revise beliefs, and adapt to evolving contexts over time. For instance, consider research assistant tasked with "What's the evidence for X?". Subsequent queries like "Who published it?" require further information retrieval, while "Is the source credible?" calls for self-reflection and assessment. Fach query builds on the previously collected and accumulated information. Similarly, a shopping assistant may be first asked "Which product is cheapest?", then "What are its reviews?", and "Is it compatible with my device?". These interactions span multiple turns, featuring evolving contexts and compound reasoning.
In long-horizon systems, a common strategy is to append all past observations, actions, and thoughts to the context at each step (Wei al., 2022; Yao al., 2023). This creates three challenges. (1) Growing inference cost and memory usage. Transformer- based LLMs scale with O(N2) compute (or O(N) with KV caching) and O(N) memory as context length N grows (Vaswani et al., 2017),
forcing deployments to reserve large GPU memory and often wasting resources (Kwon et al., 2023;
Zheng et al., 2024). (2) Generalization limits beyond the training horizons. Contexts longer than those seen during training push the model nut-of-distribution, reducing its ability to reason reliably (Yoon et al., 2024). (3) Overloaded context and forgetting. Redundant or irrelevant content that grows with agentic interaction dilutes attention and makes the model prone to forgetting important details, even when they remain technically available in the context (An et al., 2025; Liu et al., 2024; Wu et al., 2025).
Recent progress in long-context modeling largely targets static inputs (e.g. long documents) and does not address multi-turn interaction with external environments (Beltagy et al., 2020; Gu et al., 2025). Some other approaches introduce external memory modules (e.g., summarizers or retrievers) (Yoon al., 2024; Li et al., 2023; Chhikara al., 2025; Xu et al., 2025), but these are typically trained separately and cannot be optimized end-to-end with the agent's 's policy. This also introduces additional engineering overhead, as engineers must manage and integrate two separate models. Meanwhile, existing works on tool-using agent systems trained with reinforcement learning leave memory management unsolved, letting the prompt length grow unboundedly (Jin et al., 2025; Zheng et al., 2025). A natural question is raised: Can language model learn to consolidate its memory as part of its reasoning process so that it retains only what is essential for solving the task?
Motivated by this question, we present MEM1: Memory-Efficient Mechanism via learning 1- step integrated reasoning and consolidation-a method for training LLM agents that maintain nearly constant memory usage across arbitrarily long horizons. At each turn, the model updates consolidated state composed of prior memory and newly obtained information. This consolidated state becomes the agent's 's only retained memory, allowing all observations obtained via external tool use to be discarded after use, which prevents prompt expansion altogether (illustrated later in Sec. 4.2). A key insight of our method is that inference-time reasoning (Wei et al., 2022; DeepSeek-Al et al., 2025; Muennighoff et al., 2025; Yue et al., 2025) serves two purposes: While reasoning about the current query, the model also extracts and stores the essential information it needs for the future. By unifying reasoning and memory consolidation, MEM1 enables the agent to both reason and remember within shared representational space, without requiring extra modules or architectural changes. We train this behavior end-to-end with reinforcement learning (RL) (Sutton & Barto, 2018; Zhu et al., 2023), optimizing for task success via verifiable rewards (Shao al., 2024). Although not explicitly optimized for memory efficiency through reward signals, the agent learns to manage memory as part of its policy, resulting in near-constant memory usage across long horizons.
We empirically evaluate MEMI on (1) multi-turn information retrieval task, where we augment standard single-objective QA datasets into multi-objective settings by composing N multi-hop questions, and (ii) the WebShop environment (Yao et al., 2022), which requires the agent to perform multiple steps of interaction with a textual web environment. Across these diverse settings, MEM1 consistently matches or exceeds the performance of leading baselines while achieving efficiency gains of up to 3.5x in memory usage. Moreover, agents trained on our 2-objective augmented tasks generalize robustly to much harder cases with up to 16 sequential objectives. At this extreme, MEMI not only outperforms all baselines in accuracy but also reduces peak memory usage by 1.27x x and accelerates inference by 1.78x relative to the strongest uncollapsed baseline.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
LLM agents in multi-turn environment. LL-M-based agents have evolved from handling single- turn queries to serving autonomous agents capable of multi-turn interactions such as web naviga- tion (Yao et al., 2022; Zhou et al., 2024) and complex research (Zheng et al., 2025). To enable such capabilities, Yao al. (Yao et al., 2023) introduced the ReAct (i.e., Reason + Act) framework, which enhances LLMs' ability to interact with external environments by interleaving reasoning and action. Building this reasoning-aa acting prompting paradigm, subsequent works have explored ways to improve agent performance through natural language feedback, enabling iterative refinement (Shinn et al., 2023; Madaan et al., 2023). Recently, inference-time scaling has emerged as promising direction for enabling complex reasoning, with prior research incorporating evaluators (e.g. verifier, reward model) (Snell et al., 2024; Liu et al., 2025) world models (Chae et al., 2024). In addition, there are two major lines of training approaches: (1) behavior cloning (BC), which involves imitating expert trajectories to guide agent behavior by supervised fine- mue-ttnin (SFT) (Yin et al., 2023; Deng et al., 2023; Cheng et al., 2024), and (2) reinforcement learning (RL), which optimizes agent policies by incentivizing desirable outcomes through rewards (Song et al., 2024; Bai et al., 2024: Qi et al., 2024). These methods aim to align the agents' behaviors with task objectives, enabling more robust and generalizable performance.
Context management for LLM agents. A widely adopted approach to context management in LLM-based agent systems involves appending all prior information, such as observations, interme- diate thoughts, and actions, into the prompt at each interaction turn (Yao et al., 2023). While this method is straightforward and effective when the number of interactions required is small, it results in unbounded context growth, leading to linearly scaled inference memory. Moreover, long contexts often contain irrelevant or redundant information, which impairs the model's 's reasoning capabili- ties (An et al., 2025; Liu et al., 2024; Wu al., 2025). To mitigate these issues, recent studies have proposed external memory frameworks, including vetrieval-augmente modules (Yoon et al., 2024; Li et al., 2023; Chhikara et al., 2025; Xu et al., 2025) and hierarchical working memories (Hu et al., 2025b; Rasmussen et al., 2025; Sun & Zeng, 2025). However, these methods are typically applied independently of the agent's 's policy, creating a disconnect between memory and the reasoning process. In addition, managing and integrating such modules often incurs extra computational overhead and system complexity. Despite these advancements, many RL ap- proaches for training LLM agents still rely on accumulating the full interaction history memory (Jin et al., 2025; Zheng et al., 2025; Qi et al., 2024), leaving memory management during training an underexplored area. In this work, we seek to bridge this gap by tightly integrating memory with the agent's 's reasoning process, thereby cnabling more efficient and context-aware decision Docision-making.
generation and summarization
MEM1
Complex reasoning tasks often require iterative process of information gathering and synthesis, as seen in applications such as deep search (Perplexity AI, 2024) and web-based agents (Nakano et al., 2022; Gur al., 2024). We consider interactive agent operating in multi-turn environment with vocabulary space V. The generation process of an agent is defined as a Markov Decision Process (MDP) parameterized by (S, A,, म. r). where A represents the action space, S represents the state space, π S x A x S - [0, 1] is the transition distribution (i.e., the policy), and r S + I the reward function. A trajectory generated by policy π is denoted with 7 = ((o,,, (1, At each step t, the agent receives an observation O, E AA (from external tools, APIs, or the environment), maintains an internal state S, E A (reasoning history and memory). and produces an action A, E A (e.g., answering question or issuing a query), where A+ represents the set of sequences of a E A. The agent's goal is to maximize task success across long-horizon trajectories while keeping the retained context bounded. Formally, let r V - IR be the reward function. The learning problem is a maximization over the model parameters 0
e
argmax, Eqeg Eiars)eer r(s))
where 7 - A,, 0 n (S,,AA) is a trajectory sampled from policy Te.Q with n turns and Q refers to the set of questions. In this work, primarily consider tasks with verifiable rewards (i.e., is a rule-based mapping). A long, multi-turn reasoning task is characterized large n, requiring the agent iteratively perform long series of searches and reasoning derive the answer An,
3

---

Published as a conference paper at ICLR 2026
Rollout
Reward Assignment
Objective Computation
Policy Update
Environment
Exact Match
Ale. Aburigge
DKc. El Penalit
V(&) Usnae Fancler
Tokan Lavel Loss
Task
Query (Adtory
Siservation (Irff)
Actor Model
F1 Score
Actor Moder
Model
Critic Model
Traj. Tokens
Internal State (8)
Oues
Herm Store (t-1]
Query (1-1) Info (-11 Insor Shw (t)
Query (1)
Does
Fcarne Booe [1-1)
Query (2-1)
Info (c-1)
Inaww Suz (3)
Query (t)
Provoa centaet lanewad
Ques
Toana Stow
Query (t)
Info
anvirssmmnt
wel 31
Stots (1)
Query
Stow (+=1) Query
:onttxx
Generated Tokens
2D Attention Mask
Info Mask
For 1501-2 only Ques, HSMr Query[trr) Infuttet) are NOT mosked
Figure (Top): the RL pipeline used to train MEMI. (Bottom left): The evolution of context in MEM1-old internal states (S), query/answer (A), and external information (O) are cleared as new states enter the context during rollout. (Bottom right): the 2D attention mask used during the objective computation stage. The mask is applied during the forward pass to compute action log bog-probabilities for the actor model and state value estimates for the critic model. During policy update, an information mask is then applied to mask out O tokens not generated by the model itself.
3.1 MEMORY AS PART OF REASONING
To achieve a constant memory, MEMI is particularly trained to iteratively refine its understanding by processing new information in conjunction with consolidation of its prior state. At each turn i, the agent produces a new S,, which summarizes past information and reasons about subsequent actions. Following this, the agent generates an action A,-a subsequent query or the answer if a direct response is warranted. If the agent issues query, the corresponding feedback from the environment O, is appended to the trajectory. At the next turn, i + 1, the agent consolidates the tuple (5,, A,, O,) into new Si+1, which serves as the basis for further interactions. After each turn, (S A, O,) pruned from the context, effectively compressing memory and preventing prompt bloat. Fig. (bottom left) illustrates the evolution of the model's context over time. At each turn, the agent retains at most two S's, two A's, and one O ensuring bounded and efficient memory usage. The detailed rollout algorithm is in Alg. 1 of App. B.5.
RL offers a powerful mechanism for shaping agent behavior through reward signals (Sutton et al., 2000). In MEMI, we leverage this framework to incentivize effective state consolidation by designing environments in which the agent is rewarded only when it strategically retains and integrates useful information. Specifically,
construct tasks that require numerous interactions with the environment to arrive at a correct answer (see Sec. 3.3). Success depends on the agent's ability to rely on information collected along the inference path. At each turn, we prune the agent's 's context to retain only the most recent internal state S, forcing the agent to perform memory consolidation as part of its reasoning process. Without access full historical context, the agent must learn to preserve and update relevant knowledge internally in order to reap the reward. This learning procedure mirrors how humans cultivate memorization skills through structured tasks such as Sudoku or crosswords (A Cognitive Connection, 2024), where success hinges on selectively attending to key information and building upon it. Over time, such tasks help individuals develop cognitive strategies that jointly support efficient memorization and reasoning, similar to our RL method for training MEMI.
3.2 MASKED TRAJECTORY FOR POLICY OPTIMIZATION
Popular RL algorithms (Schulman et al., 2017; Hu al., 2025a) update the policy with policy gradient, which requires the calculation of VoTo.o(a,, st). For LLM, (a. is viewed as the logit of the output a of the model, where the input is 8t. Existing RI. frameworks typically
4

---

Published as a conference paper at ICLR 2026
compute the VoTo, st)'s for all pairs (1) E by passing the entire rollout trajectory through the LLM once (i.e., prefilling). However, as MEMI dynamically consolidates its context, the tokens a, do not belong to one single trajectory т. A naive solution is to break each turn into a sub-trajectory TÜ = (St, A,, O;), where i represents the ith interaction turn. However, this in computing the temporal difference d; = r(st) + ((+++1) (st) for the last token in the current sub-trajectory Tr. as ((4+1) is calculated in separate sub-trajectory T # i. Here V S + I is the value function.
approach introduces difficulties (at least implementation-wise
To overcome this challenge, we introduce a masked trajectory that compresses (2,..,.... Jn for a task with n turns into a consolidated full trajectory T/ull The full trajectory encodes all information needed for ac- curate policy learning while respecting MEMI's memory consolidation at each turn. Note that Tful is a "stitched" trajectory where the S/'s and A,'s is do not belong to the same roll-out. As such, to ensure that policy gradients VgsTa 8,) are correctly computed under this consol- idated memory regime, we apply a two-dimensional attention mask (S., 2024) across Tfull- This mask restricts each token's attention to only the tokens retained in memory at the time that token was generated. Specifically, let the attention mask for the tth token in the ith turn be Attn, = (8; 1.4, 1:0, A.,) X lag (ag (kE We can compute the policy for the ith turn Te,Q.r. (az, St) = TT. T (at, St X Attn;) and VaTT accordingly. Algorithmically, this can be achieved by first constructing the attention mask Mask = (Attnı..... Attrr) and masking the attention matrix during the transformer forward. Fig. I (bottom right) shows the masking mechanism that enables stable and accurate policy optimization under MEM I's memory-constrainee execution.
(TI. T2....: Tn)
(S, 01, S2, Ag, O2....,,, Sn. Aa).
3.3 MUL.TI-ORJECTIVE TASK DESIGN
Although MEMI is designed to address the tasks involving multi-turn interaction with the external world, there are limited publicly available datasets that support training for such long-horizon interactive processes. Existing benchmarks, such as HotpotQA ( (Yang et al., 2018), Bamboogle (Press et al., 2023), and 2wiki (Ho et al., 2020), are often cited as multi-turn benchmarks, yet they typically involve only two information-seekken steps. Moreover, these datasets are not explicitly structured to support long-horizon interactions that necessitate the agent to manage the memory state.
To bridge this gap, we introduce novel task musk-uuti-ojjecttive question answering (QA) extends the number of reasoning steps required to solve a problem. Building on existing multi-turn datasets such as HotpotQA and Natural Questions (Yang et al., 2018; Kwiatkowski et al., 2019), we interleave multiple questions from the original QA corpus and construct a single composite query that requires answering all constituent sub- quessiiosstnn. shown in Prompt 1 of App. B.3. Unlike standard multi-turn QA, this formulation compels the agent to (i) issue multiple search queries, each targeting distinct sub-question, and (ii) organize the sub-answers into a coherent final response. The augmented dataset inherits the multi-turn retrieval tasks presented in the original HotpotQA. Additionally, to test the agent's long-horizon processing capability, our multi-objective QA combines multiple multi-turn questions into a grand objective and tasks the agent to answer all the questions combined. For instance, the original HotpotQA dataset contains the following two questions: "Which magazine was started first Arthur's 's Magazine or First for Women?" and "The Oberoi family is part of a hotel company that has a head office in what city?". An augmented task then: "Answer each of the following questions: "Which magazine was started first Arthur's Magazine or First for Women?" and "The Oberoi family is part of a hotel company that has a head office in what city?" Organize your final answer in <answer> and </answer> and separate the answer to each question with semicolon.
EXPERIMENTS & RESULTS
We empirically demonstrate the effectiveness of our approach in training the MEM1 agent to perform multi-turn tasks while preserving a near-constant-size against several baselines using a comprehensive set of metrics categorized into accuracy (e.g. Exact Match, Fl score, Environment Reward) and efficiency (e.g.. Peak Token Usage, Dependency Length, Inference Time). All MEMI variants fine-tuned from the Qwen2.5-7B Base model (Yang et al., 2024). We use PPO (Schulman et al., 2017) as the RL algorithm as it computes leven-level advantages, bringing stability to the training process. While we also experimented with instruction- -uuned and
memory state, We evaluate MEMI
5

---

Published as a conference paper at ICLR 2026
supervised fine-tuned models using curated high-quality trajectories, reinforcement learning from the base model consistently yielded the best performance and generalization.
Our experiments are conducted in two standard environments, each reflecting real-world scenarios that require multi-turn agent interactions. The first environment question answering with retrieval- augmented generation (RAG) (K (Wwiakowski al., 2019; Yang al., 2018), where the agent must answer queries by retrieving relevant information from an external knowledge store (either a database or an online search engine). We trained on RAG with a local database (i.e., Wikipedia Corpus) and evaluated on tasks involving open web browsing. For QA, following Sec. 3.3, we construct multi- oojtti-oeecctive tasks and tested the model performance on tasks with more questions than seen in the training. The second environment is WebShop navigation (Yao et al., 2022), where the agent assists users in online shopping by browsing website and selecting items based on natural language descriptions. This task requires the agent to iteratively read page content and make navigation decisions, following protocols similar to those in WebGPT (Nakano et al., 2022).
4.1 IMPLEMENTATION DETAILS
Datasets and evaluation metrics. We train two versions of MEMI agent for both long- -oorion QA and web navigation. For long-horizon QA, we augment the multi-turn QA dataset from (Jin et al., 2025) that mixes data from both HotpotQA (Yang et al., 2018) and Natural Question (Kwiatkowski et al., 2019) to form multi-objective composite tasks. During training, we use 2-objective task only and test the agent's performance on tasks with more objectives.
For the web agent, we use the WebShop environment (Yao et al., 2022), which also produces reward during training (Yuan et al., 2025). For all datasets, the train-test split follows the original papers. During RL training, employ the exact match (EM) metric for QA tasks (details in App. B.4. and the environment reward for WebShop (Yao et al., 2022: Yuan et al., 2025). To evaluate the effectiveness of various approaches, measure the EM and F1 score for QA tasks and final reward for the WebShop environment (Yao et al., 2022; Yuan et al., 2025). To evaluate efficiency, we consider the peak token usage, average dependency, and average inference time. The test datasets are obtained data. The former two metrics measure
from the original papers which consist of the memory efficiency, while the latter measures the time efficiency. The detailed definitions of the metrics are in App. B.4.1. The prompt and format can be found in App. B.3.
To evaluate the accuracy and efficiency of MEM1, we compare it against two groups of baselines: (i) prior published methods, and (ii) ablations of our approach.
Baselines.
Prior published baselines.
QA environment: Rearch-RI (Jin et al., 2025), DeepResearcher (Zheng et al., 2025), and the larger-
scale model Qwen2.5-1 1..55-11BB-ia 14B-Instruct (Yang et al., 2024). Details of Search-RI and DeepResearcher provided in App. B.4.2.
WebShop environment: Agent-FLAN (Chen et al., 2024), Agent-R (Yuan et al., 2025), and
AgentLM (Zeng et al., 2023).
Context compression baseline: A-MEM (Xu et al., 2025), which augments an Instruct model with a vector database for memory retrieval.
Ablations of our method.
Truncation baseline (prompt only): We apply MEM1's 's agentic truncation prompt template and rollout to an instruct model without RI., isolating the effect of the prompt and rollout design alone. We find that training provides significant performance gains, even though prompt-only rollout already offers some efficiency benefits. Detailed results are presented in Tab. 1, Tab. 2, and Tab. 3. SFT baseline: Train supervised fine-tuned model on trajectories curated from GPT-4o (OpenAI, 2024) based on MEM1 's rollout, enabling comparison with the RL-trained agent. We find that although supervised fine-tuning improves performance, reinforcement learning is much more effective at enabling generalizability. Detailed results are presented in App. F.1.
Memory-Reasoning Coupling: Explicitly separate memory and coupling in MEM1's internal state investigate the effect of integrating the two in terms of performance and efficiency. We find that integrated memory and reasoning can benefit both performance and efficiency in our evaluated tasks. Detailed results are presented in App. F33

---

Published as a conference paper at ICLR 2026
2.00
5000
4000
150
Paak PaakToken Token
3000
100
0.5
0.00
Number of Otjectives
Seargn-11 (Eaec Match!
Doven2.1. ratruct (Peak "Mkar)
HEM1-78 (Peak: Tokeen!
Figure 2: Performance and efficiency scaling of MEMI (trained on 2-objective QA) with the number of objectives multi-ohjective tasks. MEM1 outperforms the other models and baselines while having an almost constant scaling in memory usage. Note that at 16-objective, the context of baseline models does not increase anymore since their model performance has degraded (some collapsed).
Table Comparison models on multi-objective multi-turn QA tasks. Arrows indicate the desired directions. Numbers in red indicate collapsed model behavior (extremely low performance). (truncate) means using MEM I 's prompt and rollout pipeline. (A-MEM) means using MEMI's prompt and rollout pipeline with A-Mem's 's external memory module (Xu et al., 2025). MEM1 1-QA means MEM trained on 2-objective QA task. Dependency scores are presented due to limited space.
Model
2-Obgethvv
5-Otgetive
16-Ohjective
EM T
F1.
Peak (XFF11
Time 53
BM
F1.1
Peak (X10P)
Timee (1) 1
EM
F11
Peak (x107))
Time (el
Qwes2.5 7B
0.366
15.
4,606118
0.57
09.5:040
15995115
0.213
43.3:1162
15:5=0.23
Qwex2.5-22 7B (A-MMM)
0.285
0.371
14:1:0.10
24.5:0.1
1.13
1.43
186:6.10
537221.26
0.730
0.961
18:5:0.14
01:2-2.44
Qwer2.5 5-7 (lunsalc)
0.362
0.336
8:28:006
3.99.0.16
0.07
1.23
118:0.00
11.210.00
0.395
0.497
13:3:00.16
22.1:0000
Seach-Rt congiral)
0:851
0.311
130-008
0-35
0.64
0.08
4.25-0.16
01993
0.011
20:0-1115
475--61K
teansd 2-bbj
12.52:0.22
4.05:8.02
21.5:0.15
0.107
24,6:0.22
RespRestaccerr
0.536
0.650
21.0:043
4.01:007
0.73
0.90
5188:035
11.326114
0.073
0.106
45.9:20.66
158:0019
MEMI-QA
0.709
0.83%
6.40-8822
0:09
1.87
2.31
8.01::006
9.68-1112
1.97
2.39
184:111
8.20-8112
4.2 MEMI ON MULTI-OBJECTIVE MULTI-TURN TASKS
One key advantage of MEM1 agents lies in their efficient management of long-horizon horizon interactions with the environment. To demonstrate this, train our MEM agent with a 2-objective augmentation of the QA dataset, and subsequently test against other models on held-out multi-objective test sets similarly constructed from the original QA corpus. As elaborated in Sec. 3.3, these multi-objective tasks require substantially more reasoning turns to complete, thus serving as a more demanding benchmark for memory management. As shown in Tab. MEM1 consistently outperforms other 7B counterparts: when tested on 16-objective task, it achieves over 10x higher EM score, while reducing peak context length by more than 70% and cutting inference latency by about one-half. This demonstrates that MEM1 not only improves task success on more complex multi-objective settings, but also does with markedly better efficiency both memory and runtime. A visualization of the trend in agent performance as the complexity of tasks increases is shown in Fig. 2. As the number of objectives increases, the Peak Token Usage of all other methods and models scales nearly linearly. In contrast, MEM1 maintains an almost constant peak token count with only slight increase.
Notably, while MEM1 initially underperforms Qwen2.5- In-trutt its performance gradually catches up as the number of objectives increases, eventually surpassing the 14B model, which has double the parameter count. As shown in Fig. 2, at 1, 2, 3, 4, objectives, MEMI has close EM scores compared to the 14B model. As the number of objectives continues increase, MEM achieves significantly higher EM scores. In the 16-objective task, MEMI achieves over 3x the EM score, while requiring only 27.1% of the peak tokens and 29.3% of the total inference time compared to Qwen2.5- Ins-tust This efficiency translates to significantly reduced GPU memory requirements and overall computing resource demands.
7

---

Published as a conference paper at ICLR 2026
Table 2: The experimental results for WebShop. For fair comparison, we do not report GPT's inference time. For Agent-R, scores are taken from the original paper, as the model is closed source. MEMI- W means MEM1 trained on WebShop environment.
Model
Avg Final Reward
Peak Token (x10))
Dependency (x10%)
Inference Time Per Traj (8)
GPT-4o
25.48
5.30 +: 1.23
3,99 + 1.16
N/A N/A
GPT-4o (truncate)
13.82
0.99 + 0.99
0.81 + 0.23
GPT-4o (A-MEM)
24.50
1.84 +: 0.06
0.31 1: 0.11
N/A
Qwen2.5-7B- 1--0l.5-7--nnstruee
18.42
5.64 + 1.34
3.38 = 0.89
12.31 1.82
Owen2.5-114BInn Ins-uutt
12.34
5.44 + 0.92
3.30 = 0.61
18.17 2.32
Agent-FLANN-T5
40.35
3.37 + 1.12 N/A
2.18 + 1.62
9.95 + 6.19 N/A
Agent-Hg R-8B
63.91
N/A 0.28 = 0.07
AgentLM-7B
63.60
2.24 = 0.40
3.91 = L.07
AgentL.M- 13B
70.80
2.36 + 0.46
0.30 0.08
5.23 1.59
MEMI WebShop
70.87
0.81 0.10
0.15 0.16
2.61 0.48
4.3 MEMI ON SINGLE-OBJECTIV MULTI-TURN TASKS
While MEM1 is designed to train agents for very long-horizon tasks, our training method also delivers improved capability with existing multi-turn tasks while achieving much greater efficiency at the same time, all without being explicitly trained on the single-ohjective versions of these tasks. Note that single-objective tasks also require multiple turns of interaction to produce the desired output.
Long-horizon web navigation in WebShop. Beyond QA tasks, we further evaluate the effective- of MEM1 in managing long-horizon interactions in the form of web navigation. We show the experimental results in Tab. 2. Trained in the WebShop environment (see App. B.6), MEM1 outperforms other agent training baselines, including Agent-Flan, Agent-R, and AgentLM when utilizing models of similar size. Furthermore, MEMI achieves remarkable efficiency improvements compared to the best baseline method, AgentLM,, featuring 2.8x improvement in Peak Token Usage, a 1.9x improvement in Dependency. and a 1.5x improvement in Inference Time. MEM1 even surpasses AgentI.V -M- a model with twice the parameter count of our trained model. Additionally, our results indicate that using MEM1 is significantly better than OpenAl's GPT-4o on the WebShop tasks, even when the truncation prompt templates or A-MEM techniques are applied to GPT-4o.
Single-objective QA in Wikipedia. Tab. 3 presents the accuracy and efficiency metrics for evalua- tions on single-objective QA tasks on Wikipedia (Jin et al., 2025), where the agent make retrieval requests from the Wikipedia datastore via RAG. The MEM1 used this evaluation the same as the one detailed in Sec. 4.2, which is trained solely on 2-objective task. Overall, MEM1 demonstrates superior efficiency across all three evaluated efficiency metrics, while simultancously achieving the highest EM score and an Fl score comparable that of Qwen2.5-14BB 144.5---1e---strucc.. This improvement is attributed to the MEM1 1 agent's ability to consolidate memory from previous interactions into compact internal state, which reduces the number of tokens used in the context. We also observe that SFT significantly underperforms RI., highlighting the necessity for RL-based training.
Transfer performance to online Web-QA. To validate the transferability and generalizability of the trained MEMI agent, we test MEMI trained on RAG-QA in an online web-QA environment (Zheng al., 2025). which is unseen by the agent. This environment is similar the Wiki-QA task in Sec. 4.3. However, instead of retrieving information from a local Wiki document store, the agent needs to conduct web searches through API service (e.g., Google Search API) that returns results including titles, snippets, and URLs to answer the QA problems As shown in Tab. 3, MEM1 consistently exhibited improved efficiency alongside comparable effectiveness in this unseen setting. The result demonstrates that MEMI I has learned actual reasoning and memory management capability rather than overfitting to the RAG dataset.
4.4 ANALYSIS ON EMERGENT AGENT BEHAVIORS AND FAILURE CASES
Through analyzing MEMI's multi-turn interaction traces trained on 2-objective QA, we observe range of emergent behaviors that critical for handling long-horizon, multi-objective tasks,
8

---

Published as a conference paper at ICLR 2026
Table 3: Transfer performance comparison across environments for out-of-distribution single- objective tasks. Arrows indicate the desired direction. (SFT) means training with SFT and applying MEMI's prompt and rollout. Note that DeepResearcher is specifically trained on the single-objective Online Web-QA task with Fl score the optimization objective, and Search-RI is specifically trained on the single-objective Wiki-RAG task with EM as the objective.
Euvironment
System
EM 1
F1
Peak Token (x10P)
Dependency (x10")
Inference Time
Qwen2.5 Bree2.5-7B-hhe Inst (truncate)
0.287
0.382
6.28 + 0.05
1.65 = 0.04
2.26 = 0.01
Qwen2.5-7B- S-n---tt (A-MEM)
0.246
8.47 0.12
0.92 0.03
11.2 0.40
pwen2.5-7B-Ion
0.269
0.390
9.32 + 0.19
1.17 = 0.04
2.31 = 0.04
Wiki RAG
Qwen2.5. Inst
0.422
0.534
8.89 + 0.21
2.22 + 0.10
6.73 + 0:24
Search-RI (original)
0.445
0.516
11.0 0.25
1.50 0.05
2.23 0.14
DeepResearcher
0.419
0.503
13.3 + 0.34
7.04 + 0.33
3.86 + 0.09
MEMI-QA (SFT)
0,302
0.358
6.54 + 0.05
3.30 0.13
4.84 0.21
MEMI-QA
0.405
0.471
5.63 1: 0.03
0.76 + 0.02
3.79 0.07
Qwen2.5-7B-Be 7-T-Intt
0,334
0.451
8.37 + 0.18
1.39 + 0.06
2.20 + 0.04
Online Web-QA
DeepResearcher
0.372
0.492
10.27 = 0.19
2.86 0.14
2.87 = 0.06
MEMI-QA
0.397
0.485
5.79 0.06
0.44 0.02
1.84 0.03
(A) Storing Information
(B) lodging & Moving On
(C) Integrated Ressoning and Memory
(D) Updlate Memory As New Information Streams In
Separately
MEMI:
MEM:: Unfortunanely, based on the intormation poodddd the nine Veldan Simpean that waea the langead the world upon opening, Moving on the second question...
[omited History! Erw: [Sea ch Results)
[onitted History! Erw: (Eeorch Resu to:
According to the provided Information:
They had World Series
dbcurients, ents, am unazis
Dec Prabiestati Buse sxlected be the first president of the Manila Reshtryy Sangha, However, we need to confimm
After
have had 11 World Seriee
infermaton, lound oul that Bemanino DI Stacomo was born in Porto Recanot off Now we can add this informatio our cumulaaive
appearances in Los Angeles,
Therefort, ... Now. to anowor the second quaci on....
From the provided information:
perform web search find the information.
born Perto Recanati arswer second q metion, what city was Tim Alexnnde born?" we need to perform web search..
* The show originally ran on ABC from 2011 to 2017 for six sessons...
enearcho Annie Becant Indian National
Congress-/search
a MEMT stones key extracted Info about each question soparately
OL MEMT leams to skip one questioo after several searches without fledin useful information.
I'l MEMT Integrates summary of key facts Into is, facilaacing reasoning
l MEMT explicity updates its memory bafore moving on to the next question
(E) Swif- Verllication and Correcting
Making Plan
(G) Iterative Search
(H) Adjusting Search Query
MEMI: Heaiing searchedt and finding tional infomaatoon Shirley MacLaine asssumpt ont jsince ch sractera mon wa do not desczibe the thameetves this way! is wrongly idemtilidd research nnceesary, the Information provided
[A compie rigive ieeoifcctiin avestion]
MEMI (ctop 1e: To answer the first question
MEM1 (step 11:
capital nated?" need to find out the country where
<pcarote largest city
original <isearcto-
MEMT To answer the first question, will froe need identilly the professional foorteller who plays for
yoge origi neted and then find itx
Erw: [Sewchh Beautss]
capital.
MEMT (step 21: The provided
Kerala assers, deternine his sohlovement Buigorion football makes iilm oordeduned the time beadis, Laatly. reeed to deliemene the
<search orignn yoga country of
informaton mentions
and
misleadingg Hercce direct search "al catiest procucatera sed set-- would help get the right mattch.
Fist, we need lcuntily the fourth bergest city in Dermanry and then mine its original home. burgest city crearcto- </search>
Enw: Bose ch Besutis] MEMT (step 2): From proviced information, yoga originated in India.
fourdaation used fire "ame, and the scholore was And--MS Yarak
Now, India...
find
% MEMI reasons through collected correct betlef conducts search validate
MEMT tends develop search plans complex questions
MEM1 learns decompose question and make separate
MEM1 learns to edjust the query when retrieved information insufficion!
Figure 3: Snippets of internal states and actions showing MEMI's Emergent Behaviors in 2-objective QA tasks. Light Blue denotes behaviors related to multi-objective tasks. Beige denotes behaviors related to memory in internal state. Pastel Green denotes behaviors related to general search strategies.
demonstrating capabilities well beyond simple retrieval. Additionally, we provide detailed qualitative and quantitative analysis of cases where MEM1 fails in App. G First, MEMI learns to manage multiple questions concurrently by maintaining a structured internal state. As shown in Fig. 3(a), when faced with two multi-tun multi-turr questions, the agent stores and updates memory for each question separately, guiding subsequent searches based on the identified information gaps. In (b). MEMI exhibits the ability to shift focus when progress on one question stalls, recognizing difficulty and prioritizing the more tractable objective. Meanwhile, MEM1 1 learns to interleave reasoning and memory in its internal state S's, weaving important information into its decision-making process to support both information retention and action selection. In Fig. (c), MEMI explicitly extracts important information from previous search results and leverages it to formulate the next query that best addresses the current information gap. In addition, (d) shows that when new, relevant information is retrieved, MEMI explicitly reasons about its significance and selectively updates its memory. We believe that learning these interleaved behaviors is key to achieving efficiency gains

---

Published as a conference paper at ICLR 2026
in memory without degrading performance. Beyond behaviors unique multi-objective setup and memory architecture, MEMI also exhibits several general-purpose search strategies. In (e), the agent performs self- serffiiriii,tiinn. correcting earlier misconception and issuing new query for confirmation. In (f), complex queries are decomposed into manageable subgoals before initiating the search. In (g), for questions requiring multi-turn information gathering, MEM1 extracts key information from search results and uses it to inform the next search. In (h), when overly specific queries fail, MEMI re-scopes its query to improve retrieval. Notably, many such behaviors, like verification, making plan, and iterative search, are also reported in recent studies on deep research agents (Jin al., 2025; Zheng al., 2025).
CONCLUSION, LIMITATIONS, AND FUTURE WORK
We introduced MEM1, I, reinforcement learning framework that enables language agents to perform long-horizon reasoning with consolidated memory. By integrating inference-time reasoning and memory consolidation into unified internal state, MEM1 addresses the scalability challenges of prompt growth and achieves competitive performance across QA and web navigation benchmarks, with substantially reduced memory usage and inference latency. Despite these advantages, MEMI assumes access environments with well-defined and verifiable rewards. However, many open-ended tasks present ambiguous or noisy reward structures. Fully realizing the potential of MEM1 therefore requires advances in modeling such tasks and designing suitable reward mechanisms challenges that lie beyond the scope of this paper. A promising future direction to explore methods for training MEM1 agents in these open-ended settings where reward signals sparse, delayed, implicit.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This research is supported by the National Research Foundation (NRF), Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme. The Mens, Manus, and Machina (M3S) is an interdisciplinary research group (IRG) of the Singapore MIT Alliance for Research and Technology (SMART) centre.
REFERENCES
Cognitive Connection. Surprising exercises that will sharpen your short-term memory, January 2024.
URL http:ss//cconniivvecomm. surprising exercises that sharrenaapnn--eeee your short -eemm memory.
com/
Accessed: 2025-05-10.
Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, and Lingpeng Kong. Why does the effective context length of Ilms fall short? Proceedings of the International Conference on Learning Representations (ICLR), 2025.
Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wil device-control agents with autonomous reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 37: 37:12461-12495, 2024.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.055 1500 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 4165, 2020.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 5209-5235, Vienna, Austria, 21-27 Jul 2024. PMLR. doi: 14485500arXi..2401107774 URL httpp://///........ //tp///eeoecec.eeeeee/ee//es.eee mlr, press / v255//2255///// cai24b. html.
Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. Web agents with world models: Learning and leveraging environment dynamics in web navigation. arXiv preprint 2rX 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:23322.0131.. February 2023. doi: 10.48550/arXi.22202.333 URL https: 01318.
Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-flan: Designing data and methods of effective agent tuning for large language models. In Findings of the Association for Computational Linguistics (ACL.), pp. 9354-9366, 2024.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 9313-9333 2024.
Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory. arXiv preprint 1911:32504.19413.
2025.
ModelScope Community, SWIFT: scalable lightweight infrastructure for fine-tuning. https:
//gi thub. www/cooo//a/// wcw.p swi ft, 2024. Accessed: 2025-05-15.

---

Published as a conference paper at ICLR 2026
DeepSeek-AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems (NeurIPS), (166:8899-281111 2023.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanue Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint wrXiv:2401.08281, 2024.
Xiangming Gu, Tianyu Pang. Chao Du, Qian Liu, Fengzhuo Zhang. Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. In Proceedings of the International Conference on Learning Representations (ICI.R), 2025.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust, real-world webagent with planning, long context understanding, and program synthesis. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Xanh Ho, Anh-Khos Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference Computational Linguistics (COLING), pp.
2020.
Jian Hu, Jason Klein Liu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness both prompt and reward models. arXiv preprint urXiv:2501.03262. 2025a. Version 3, revised 6 Apr 2025.
Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. HiAgent: Hierarchical working memory management for solving Long-horizon agent tasks with large language model. In Proceedings of the 63rd Annual Meeting of the Association for Com- putational Linguistics (ACL. 2025), pp. 32779-32798, Bangkok, Thailand, July 2025b. Asso- ciation for Computational Linguisties. doi: 10. 10.88633///20025.a--oooon 1575. URL https:
//aclanthology wwt.oogg////// 2tg/2025.22 acl 1onlllnnn..... 1575/.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Ilms to reason and leverage search engines with reinforcement learning. arXiv preprint 100iv:2503309116 2025.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, 2020.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 466, 2019.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 19274-19286, Honolulu, Hawaii, USA, 23- 29 Jul 2023. PMLR. URL https:/ html.
BttLaatttpprcceeeiiiiii /ptpp/eeeengg...e//ee/pp/./...se . mlr www.mmwww.p...eeeemmmmmmmmee press/v202 wttp//2/w22ewaa22..aa222
12

---

Published as a conference paper at ICLR 2026
Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6921-6935, 2023.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
2024.
Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504. 250,:::500. 02495, 2025.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances Neural Information Processing Systems (NeurIPS), 36:46534 2023.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19392 2025.
Magnus Müller and Gregor Zunië. Browser Enable ai to control your browser. https: //github. caclub.com/// www.m//oowwerruue wwwtosee-uueeeeooseeeeee-----eee bww.//r-ssseeeeeeeeeeeeeeeeeeeee 2024.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vincet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question question-answering answering with human feedback. arXiv preprint 12.0:221 2022.
OpenAl. Gpt-4o system card. http:///pppaaaa.coo////////////- 2024. Accessed: 2025-05-15.
OpenManus. Openmanus: Open-source ai agent framework. https://github..
marnaandgoem// wwtp://oeemm/eeeansas...mmm 2025.
Perplexity AI. Perplexity. https: http:/pppppppppeee pewplexity. ai, 2024. Accessed: 2025-09-23.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics (EMNLP), pp. 5687-5711, 2023.
Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. Webrl: Training Ilm web agents via self-evolving online curriculum reinforcement learning. arXiv preprint arXiv:2411 1.02337, 2024.
Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. Zep A temporal knowledge graph architecture for agent memory. arXiv preprint arXiv:2501. 2025. doi: 10.488500/2Xiv.2551113333
URL https: org/ abs///bss//21 13956.
Ruslan S. 4d masks support in transformers. https: fid-masks, 2024. Hugging Face Community Blog.
co wwtpb/////p//a.....rrr.oo//o
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707:0077444 2017.
Serper. Serper api: Fast and affordable google search api. https://seiper. Accessed: 2025-05-15.
2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. arXiv preprint arXiv:2409. 19256, 2024.
13

---

Published as a conference paper at ICLR 2026
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 36:8634-86522 2023.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling I1m test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 24X1::2408.03314, 2024.
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-based trajectory optimization of Ilm agents. Proceedings of the 62nd Annual Meeting of the Association Computational Linguistics (ACL), pp. 7584-7600, 2024.
Haoran Sun and Shaoning Zeng. H-mem: Hierarchical memory for high-efficiency long-term
reasoning in Ilm agents. arXiv preprint arXiv:2507.22995, 2025. doi: 10.48550/arxiv22557222955
URL pestps://azxivie org/ abs/ 200/2507... 22925.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press,
Cambridge, MA, second edition, 2018. URL
the-book-2nd. html.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NeurIPS), volume 12, pp. 1057-1063, 2000.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint (xXix:2302.19777 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, tukasz. Kaiser, and Illia Polosukhin. Attention all you need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, pp. 5998 6008, 2017.
Lei Wang, Chen Ma, Xueyang Feng. Zeyu Zhang. Hao Yang. Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(3):1-25, 2024.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly contrastive pre-training. arXiv preprint arXiv:2212.0355 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, H. Chi, Quoe V. Le, and Denny Zhou. Chain-of of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 24824-24837 2022.
Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai- Wei Chang, and Dong Yu. Longmemeval: Benchmarking chat assistants on long-term interactive memory. In International Conference on Learning Representations (ICLR), 2025.
Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for Ilm agents. arXiv preprint arXiv:2502.1 12010, 2025.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan. Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 1111v22412.15115 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi- hop question answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language 2018
Processing (EMNLP), pp.
14

---

Published as a conference paper at ICLR 2026
Shunyu Yao, Howard Chen, John Yang. and Karthik Narasimhan. Webshop: real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Towards scalable
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai- Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. Lumos: Learning agents with unified data, modular design, and open-source Ilms. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2023.
Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, and Jaewoo Kang. Compact: Com- pressing retrieved documents actively for question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 21424-21439, 2024.
Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen. Agent-r: Training language model agents reflect via iterative self-training. arXiv preprint a:Xiv 2025.
Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuanhui Wang, and Michael Bendersky. Inference scaling for long-context retrieval augmented generation. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agentunnng: Enabling generalized agent abilities for llms. arXiv preprint 0rXiv:2310.12822 2023.
Jintian Zhang, Yuqi Zhu, Mengshu Sun, Yujie Luo, Shuofei Qiao, Lun Du, Da Zheng, Huajum Chen, and Ningyu Zhang. Lightthinker: Thinking step-by-step compression. arXiv preprint arXiv:2502. 15589, 2025.
Lianmin Zheng. Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai. Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint 2504.003 2025.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tranyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise k-wise comparisons. In Proceedings of the International Conference on Machine Learning (ICML), pp. 43037-43067,,025
15

---

Published as a conference paper at ICLR 2026
A LARGE LANGUAGE MODEL USAGE
We use GPT-5 for both paper writing and literature research. Specifically, we use GPT-5 to help us with proofreading the paper, correcting grammar mistakes, and polishing the writing throughout the paper. We also use GPT-5 to conduct literature research in our related works section.
B DETAILS OF MEMI
B. COMPUTING RESOURCES AND TRAINING DETAILS
All trainings of MEM1 are conducted on H100 or H200 GPUs. We use the veRL framework (Sheng et al., 2024) for RI. and Swift (Community, 2024) for SFT. For RI., both the data batch size and mini batch size are set to 64. Learning rate set to 10-6 for the actor model and 10-5 for the critie model with linear warmup of 50 steps. Temperature is set to 1 during training and 0.01 during inference.
All evaluations are conducted on a single H200 GPU, which serves the respective models as an API service using the vLLM framework (Kwon et al., 2023) with automatic prefix caching enabled.
B.2 RAG CONFIGURATION
For RAG on local Wiki corpus, we use Faiss-GPU (Douze et al., 2024) serving an E5 Base model (Wang al., 2022). The Wiki corpus taken from Wikipedia 2018 dump (Karpukhin et al., 2020). The number of passages for each retrieval set to 3 for fair comparison with other methods.
For online web search queries, including titles, snippets, and URLs. For each search, we return the top 10 results to the agent as external information. We do not ask the agent retrieve the content of specific webpages.
Serper API (Serper, 2025), which offers Google search results
16

---

Published as a conference paper at ICLR 2026
B.3 PROMPTS
Prompt 1: Multi-Objective Task (QA)
You will answer multiple complex questions using iterative reasoning, summarization, and web search.
At each step, you will see the questions, cumulative summary of relevant information, the current search query, and search results (except in the first step, where only the questions provided). Your task is to:
1. Perform reasoning and update a cumulative, concise summary within <think> persistent memory and must include all essential information from previous <think> and <information> tags.
</think>. This acts as
2. Then choose one of the following actions:
If any question remains unanswered, issue single query for one question inside <search> should consist of keywords or shert phrase. Only search one question at time.
</search>. The query
If all questions are answered, provide the final answers--separated by semicolons-wwithn <answer> answerl; answer2; </answer>. The answers must be concise, contain only essential words, and avoid any explanations.
Important:
Always follow this structure after <information> or the initial questions: <think> </taihk>>ssaach>>>> </search> or <think> Do not search multiple queries or questions simultaneously.
</think>><nsser>> </answer>,
Answer the follewing questions: (QUESTIONS)
17

---

Published as a conference paper at ICLR 2026
Prompt 2: Single-Objective Task (QA)
You will answer a complex question through iterative reasoning, summarization, and web searches.
At each step, you can see the question, previous summary </think>, search query in <search> </search>, and the returned information in <information> </information> (except the first step where you will be given only the question). Then, you should:
<think>
1. Conduct reasoning, and then update concise, cumulative summary with essential information inside <think> </think>. This is your persistent memory and should include all important information from previous <think> </think> and <information> </information> (i.e. information and answers already found for questions).
2. Then choose one:
Tssue a query (i.e., key words / phrases for search) inside <search> </search> (you may search repeatedly until the answer is clear). This query will be used to conduct search and return the results in <information> results </information>
Provide the final concise answer (no explanations) if additional information is needed inside <answer> </answer>. The answer should be concise and only contain the words necessary to answer the question.
After <information> </information> (or question at the beginning), you should always follew the order: <think> </think><search>>>>>
</search> or <think> </answer>.
</Ehank>><asswe>>>>>>>
Question:
[QUESTION]
18

---

Published as a conference paper at ICLR 2026
Prompt 3: Single-Objective Task (WehShop)
You are browsing an online shop. Your goal is to find product that matches the given description. You will interact with the site step. Each step gives you <state>... .<//tttte>>tt>>> representing the current webpage. You must decide what action to take next until you identify the correct product.
Available actions (shown in the <state> tag) depend the page:
On the search page: search [<kewwords>]
on search result pages: click [<item url>] to view a product, click[next On product pages: click click (color), clicx,size), click (buy now] To return to search: click [back to search]
go to the next results page
click [features],
Example goal: "Find gingko light and 20x20 pillow cover that is hand painted." " Example first action: <answer>search [gingko light 20x20 pillow cover hand [ainted]</answent Only respond with valid actions formatted as: search click[...], etc.
After you navigate and find the product that best fits the user goal, you should click [buy now] to buy the product at the product page when the buy now button is available.
Product Description:
[PRODUCT DESCRIPTION)
B.4 IMPLEMENTATION DETAILS OF METRICS AND BASELINES
B.4.1 METRICS
Exact match. In QA tasks, we use exact match (EM) as both the verifiable reward for the RL pipeline and the evaluation metric for the final output. The final response is extracted from between <answer> and < answer>> In multi-objective settings, the response should contain answers to each question separated by semicolons. the XML tags are mismatched, or the number of provided answers does not correspond to the number of questions, score of is assigned. Otherwise, 1 point credited for each correct answer. During RL training, we do not provide any other intermediate rewards format penalties, learning process (see more in Sec. 4.4).
find that such manual interventions interfere with the agent's
Fl score. The Fl score computes the harmonic mean between the precision p and recall In the case of string matching, we split both the predicted answer and the ground truth. For example, if the ground truth is "United States America", it is split into list with lower-case words: "united". "states". "of", "america". The same works for the predicted answer. Then, denote the number of common words as c. Further denote the number of words the predicted answer as and the number of words in the ground truth as Then, precision is calculated as c/l and recall is calculated as r == c/g. The Fl score is finally computed as
Fl = 2x pxr
p+r
multiple ground truths are present, the maximum of scores is chosen. For multi-objective tasks, the final Fl is the sum of the Fl scores for each sub-question.
Peak token usage. Peak token usage is calculated as the maximum number of tokens (using GPT--4o-mini tokenizer) in any single sequence throughout the agent's entire trajectory. For fair
19

---

Published as a conference paper at ICLR 2026
comparison in experiments, the system prompt excluded when computing this sequence length. The peak token usage serves as a proxy for the inference- memory requirement.
Dependency length, Following (Zhang et al., 2025), the dependency metric is defined as the total number of historical tokens on which each generated token effectively depends. Let ''' denote the total number of interaction steps. For each step i [7]. let ni the number of prefix tokens and no be the number of output tokens generated. The dependency metric is then calculated as
(4)
(1) (2no + np
(9)
Dependency
2
6E|2]
At high level, this metric quantifies the cumulative computational cost associated with the generation of an output trajectory. It is important to note that in MEMI, prefix tokens from previous steps are consolidated into new internal state, rather than being continuously accumulated. In our experiments, we ignore the tokens in the system prompt when calculating the dependency metric.
Inference time. Inference time for each trajectory is recorded as the total elapsed time required to generate the complete output trajectory. For all experiments, these measurements conducted single H200 GPU, operating with 10 concurrent threads. The vL.LM inference framework utilized, with its automatic prefix eaching feature enabled.
B.4.2 BASELINES
Search-R1. As detailed in (Jin et al., 2025), the model is trained the 1-objective task with the same dataset as MEM1. Search-R1 also uses exact match as reward function. In comparison, MEM1 is trained exclusively on 2-objective tasks.
Deep Researcher. As detailed in (Zheng al., 2025), the model trained 1-objective task with a curated set from various QA datasets including HotPotQA and Natural Questions. Deep Researcher adopts the Fl score the reward function.
B.5 ALGORITHM
We provide an outline of the rollout of MEMI, which actively manages its context in Alg, 1. Parts of
the pseudo-code follow (Jin et al., 2025). We follow (Wei et al., 2022; DeepSeek-AI et al., 2025) and annotate each component using XML-style tags: <IS> for internal state (reasoning S,), <query> for environment queries A, t < T, <answer> for the agent's 's responses Ar, and <info> for external observations or tool outputs O,.
20

---

Published as a conference paper at ICLR 2026
Algorithm 1 MEMI Rollout
Require:
Task prompt policy model no. world model W, maximum turn T
Ensure: Final response y
1: Initialize rollout sequence y Ø
2: Initialize turn count <- 0
3: while t T do
4: Initialize current policy rollout sequence y/1 + Ø
5: while True do
6:
Generate response token yr ~ To(- | x; + y/)
7:
Append y, rollout sequence Y/ + 3/ + 1/-
8:
(f - T 1) and yr E (</answer>, (ennssrr>,eess) then
9:
break
// prevent the agent from searching further then
10: else if E ht////ury.//nnsssssssc.. <eos>]
break 12: end if
end while
14:
y Bh
I| all previous context removed.
15:
if <'quryy> </query> detected in ye then
16:
Extract search query (arse <query>, </quer>>>
17:
Retrieve enviromment feedback d t W(q) from local storage, Search engine, HTML, HINT You have (T t turns left.
18:
19:
Insert into rollout y + + <info>HINT + d</info>
20: else if <answer> </
detected in yı then
21:
return final generated response
else
23:
Mark the sample as invalid end if
25:
Increment turn count L
26:
end while
27:
return final generated response y
B.6 MEMI ON WEBSHOP TRAINING DETAILS
We the same rollout pipeline and policy update mechanism for training MEM1 WebShop. Compared to the QA tasks, we use a tailored prompt that retains the gist of memory consolidation with instructions specific to the WebShop environment, as shown in Prompt. 3. Another distinction is that the WebShop environment comes with its own reward function corresponding to each state. Therefore, we do not use exact match but the built-in reward function as the reward signal when training in WebShop environment. The training and test splits also follow the original paper (Yao al., 2022), with the first 1000 samples as the test set, the 1000th 1500th as the val set, and the remaining the train set.
B.7 ADDITIONAL DISCUSSION THE ATTENTION MATRIX DESIGN.
We wish to note that our modification to the attention matrix does not fully recover the attention of the original trajectories because of the change in position ids. Specifically, prior works (Leviathan et al., 2023; Chen et al., 2023; Cai et al., 2024) that utilized the attention matrix to compress multiple trajectories mainly targeted eiceexploration, i.e., generating multiple sequences with the same prefix. For these works, on top of the attention matrix, they adjusted the position ids as well, so each trajectory follows consecutive increasing position ids. However, in MEM1, the prefix does not remain the same because of memory consolidation. This results in each <IS> having two possible position ids, one for the previous turn and one for the next turn. To completely recover the original attention, need to duplicate each <IS> and assign different position ids to the two copies. However, such duplication can significantly slow down training because the training trajectories are now much longer.
As such, for training efficiency, we do not duplicate the <IS> and assign the position ids for the previous trajectory each <TS>. While this modification slightly deviates from the "ideal"
21

---

Published as a conference paper at ICLR 2026
implementation, effectively, it can be viewed simply adding white spaces in the training trajectories and has no significant impact on the experimental results.
C BROADER IMPACTS
MEMI opens up the potential to enable more scalable, efficient, and intelligent Al agents capable of sustaining long, goal-directed interactions in dynamic environments. As Al systems are increasingly deployed in complex real-world tasks-such as scientific research, legal analysis, personalized education, and digital customer service models must go beyond single-turn capabilities and manage evolving contexts models to maintain high performance without the growing computational and environmental costs typically associated with long-context processing. By reducing inference-time memory and compute demands, MEMI paves the way for more sustainable and scalable AI deployment, making advanced reasoning agents accessible wider range users and institutions, including those with limited resources. Moreover, MEMI's unified framework of reasoning and context consolidation sets precedent for future research on intelligence that can learn to adapt, reflect, and summarize information autonomously, inspiring more trustworthy, interpretable, and numan-alignen Al systems.
many steps. MEMI'S memory- conooryconsslliattion mechanism allows language
D TRAINING TRAJECTORY ANALYSIS OF MEM1
surer serthe
aeltunng Ago
Searches
Rg0 Valid Actions
50
100
150
200
250
300
50
100
150
200
250
300
Steps
Steps
1.2
0.8
1.0
0.6
EAN00 1055
0.4
0.6
Tran
0.4
0
50
100
150
200
250
300
50
100
150
200
250
330
Steps
Stops
Figure 4: Metrics of training progresses for MEMI with RL.
We present the training dynamics of the 2-objective QA-trained MEMI in Fig. 4, where several distinct phases emerge during the learning process. In the initial exploration phase (first 50 steps), the agent demonstrates little task proficiency. The reward remains consistently low, while the entropy loss is high, suggesting random or undirected behavior. The ratio of valid actions hovers around 0.55, indicating that the agent frequently fails follow the expected output format. During this period, MEM1 has not yet learned to reliably use the required structure involving <query> and <answer> tags.
Shortly after, we observe the onset of format acquisition. The agent gradually improves its structural consistency, reflected in the rising ratio valid actions. This improved adherence to format correlates with an increase in reward, suggesting that proper formatting directly contributes the agent's 's task success. By around step 150, a notable behavioral shift occurs. The number of valid searches begins to drop sharply, while the reward continues to increase. This implies that the agent has discovered shortcut: by reducing the number searches pertapper to avoid format violations it can maintain high format fidelity and improve its reward without fully solving the task. This short-horizon
22

---

Published as a conference paper at ICLR 2026
optimization suggests the agent is exploiting the reward structure, favoring formatting compliance over content completeness.
Between steps 150 and 200, the agent enters phase of refined format mastery. The ratio of valid actions steadily climbs, but the number searches remains low. During this phase, reward growth slows, and entropy begins flatten. The plateau in entropy indicates that the agent is looking for new policies to boost the reward. At this stage, the agent has reached a local optimum: it's producing valid but under-informed answers.
After step 200, a second behavioral shift occurs. The number of valid searches begins to rise again, suggesting that the agent is learning extend its interaction horizon to gather more information. The agent learns to balance formatting constraints with information acquisition. As result, the reward increases more sharply. Finally, after step 250, the agent enters phase of policy consolidation. The entropy loss drops sharply sharply-sinnliing signaling transition from exploration to exploitation as the agent settles into a more deterministic, high-reward policy. By this stage, the agent effectively combines format compliance, sufficient searching, and high-quality answer generation.
E ADDITIONAL RESULTS
E. RESULTS ON THE DEPENDENCY METRIC
Due the limited page width, we present the full dependency metric, as defined in App. B.4.1, results for Tab. 1 in Tab. 4. From these results, we observe that MEM1 consistently achieves the lowest dependency across all task settings. Although Search-RI exhibits dependency close those of MEMI, its performance degrades substantially on more complex tasks, making its efficiency metric less informative.
Table Additional dependency score results for MEM1 1 (L lower is better). Dependency values (mean + standard deviation) are reported in units of x 106. The lowest and second lowest values in each column are shown in bold.
Model
2-Objective
8-Objective
16-Objective
Swen2.5--B-IInsteee (Amem)
0.401 + 0.280
1.440 + 0.959
1.697 + 1.819
1ren2.5-7B-Instruee (truncate)
0.573 0.540
1.535 1.134
2.060 + 2.509
Swen2.5-7B-Insste
0,877 = 1,028
4.135
4,411 #.020
Qwen2.5-14B Ins-tructttt
0.297 = 0.269
1.462 + 1.021
4.676
Search-R1
0.173 = 0.154
0.601 + 0.490
0.784 + 0.902
Search-R1 (2-objective trained)
1.180 0.919
5.800 2.316
6.450 + 2.909
Search-R1 (2-objective + truncate)
1.240 + 0.918
5.950 + 2.318
6.700 + 2.910
DeepResearcher
0.799 0.919
3.891 2.316
4.355 + 2.909
MEM1-QA
0.177 0.104
0.486 + 0.332
0.724 + 0.405
E.2 ANALYSIS OF THE LENGTH OF THE INTERNAL STATE WITH RESPECT TO THE NUMBER OF INTERACTION TURNS
To examine how the agent's internal state evolves through reasoning, we conduct a subsample analysis on MEM1-QA using 2-objective tasks with up to 6 turns and 8-objective tasks with up to 15 turns. As shown in Fig. 5, although MEMI does not explicitly regulate internal-state length, we observe no substantial growth across turns: the agent consistently discards unhelpful information and retains only what necessary. Because the final answers in these QA tasks are short, the agent does not require continually expanding internal state to converge correct solutions. While 8-objective tasks naturally produce somewhat longer internal states, the overall size remains stable across interaction turns.
23

---

Published as a conference paper at ICLR 2026
Objective Tasks
2-Objective Tasks
Internal State Length rokens)
Internal State Lenqth (tokens)
120
250
100
200
150
60
100
50
0
Interaction Turn (Q-114)
Interaction Tum (ID-15)
Figure 5: Average internal state length (with standard deviation) versus interaction for MEM1- QA across 2-objective (right) and 8-objective (left) tasks, shown using the combined figure.
F ANALYSIS ON IMPLEMENTATION DETAILS
F.1 RL GENERALIZES BETTER THAN SFT
natural question arises: can Supervised Fine-Tuning (SFT) with high-quality trajectories match the performance of reinforcement learning (RL)? To investigate this, we compare MEMI-QA trained via RL against MEMI-QA (SFT), where both models are trained on the 2-objective QA task. Additionally, the SFT model is further trained on I-objective and 3-objective QA tasks to enhance its generalization ability. As shown in Tab. 5, the SFT model consistently underperforms compared to its RL counterpart across tasks with varying numbers of questions (objectives). Notably, when the number of objectives exceeds six, the performance the SFT model collapses, whereas the RL-traineed model continues demonstrate strong robustness and scalability.
Table 5: Comparison RL and SFT on increas- ing number of multi-turn questions. Exact match scores f is better. Gap shows absolute difference. Red numbers show collapsed SFT behavior.
Without
0.
#Q
RL
SFT
Gap t
RL Gain (%) 1
Tee N6
1
0.410
0.300
0.110
+36.7%
02
2
0.709
0.433
0.276
+63.7%
3
0.976
0.648
0.328
+50.6%
0.0
4
1.120
0.626
0.494
+78.9%
300
130
200
250
300
Step
6
1.630
0.088
1.542
+1752%
8 1.870
0.027
1.843
+6826%
Figure 6: Training curves comparing MEMI trained with and without format reward.
16
1.900
0.000
1.900
F.2
FORMAT REWARD ACCELERATES CONVERGENCE BUT DEGRADES FINAL PERFORMANCE
It is common to incorporate format reward when training reasoning models and multi-turn reasoning agents DeepSeek-A AI (2025); Zheng et al. (2025); Jin et al. (2025). In our study, we experimented with format reward that enforces the agent to produce outputs using specific structural tags: <TS>, <query>, and <answer>. If the agent fails to use the expected tags correctly, the turn is terminated and penalty of 1 is applied.
As shown in Fig. 6, using the format reward leads to faster convergence during training but results in worse final performance. The format-constrained agent achieves exact match score of 0.466, compared to 0.709 for MEMI trained with only outcome-based reward on the same testing set for the 2-objective QA task. Additionally, the format-constrained agent generates fewer tokens, with an average peak of 514.9 tokens, whereas the outcome-reweee -rrinned MEMI reaches an average peak of 640 tokens.
24

---

Published as a conference paper at ICLR 2026
We hypothesize that the format reward accelerates structural learning but constrains exploration of effective reasoning strategies. As a result, the agent learns to produce shorter responses with valid syntax but develops less effective internal state representations, leading to degraded task performance.
F.3 COUPLING MEMORY AND REASONING BENEFITS BOTH PERFORMANCE AND EFFICIENCY
We explored whether separating memory and reasoning could simplify the agent design. In this ablation setup, we replaced the integrated internal state with explicit <sunma <sunmary></ summary> and <reason peaasonnng>//// reasen. i ng> tags and trained the model using the same rollout procedure and reinforcement learning algorithm. As shown in Tab. 6, coupling memory and reasoning in a unified internal state consistently yields higher Exact Match scores and substantially lower peak token usage across 2-objective, 8-objective, and 16-objective tasks. These results suggest that maintaining an integrated internal state enables more efficient reasoning and better information retention, ultimately benefiting both performance and computation.
Table 6: Ablation study comparing integrated versus separated memory and reasoning. Peak token counts are reported in units of x x102, with standard deviations in parentheses.
MEMI-QA (integrated internal state)
2-Objective
8-Objective
16-Objective
Exact Match (EM)
0.709
1.87
1.97
Peak Tokens (x 109)
6.4 (0.02)
8.01 (0.06)
10.4 (0.09)
MEMI-QA (separate memory reasoning)
2-Objective
8-Objective
16-Objective
Exact Match (EM)
0.686
1.78
1.92
Peak Tokens (x 109)
11.8 (0.05)
9.63 (0.07)
13.37 (0.
G FAILURE CASE ANALYSIS
We analyze the failed trajectories from the two-question setting to understand the limitations of the model and identify recurrent patterns that hinder reliable multi-step reasoning. After careful examination, we identify five representative ways in which the model fails to answer all questions correctly. Overall, three of these five failure modes are related to general limitations of the search agent, while the remaining two arise directly from our iterative compression rollout and compositional tasks. We name them entity ambiguity, hallucination instead of search, logical fallacies, insufficient query refinement, and overwriting correct answers due to loss of the original reasoning path. In Tab. we provide full explanations for these failure types, an illustrative example containing original evidence from the trajectory, and the corresponding frequency.
There are also some non-typical failures that do not fall under these five categories, and we group them as "others." Some failures are difficult to avoid because the Wikipedia corpus we use as the information source may not contain answers for all questions, and some failures stem from formatting issues. We further compute the percentage of each failure type by labeling the error category of every trajectory. Among the 1000 trajectories we examined, where the model answered correctly for 397 tasks and both questions correctly for 156 tasks, the percentages and counts for each failure mode are also reported in Tab. 7. Note that single failed trajectory may contain multiple failure types, so the total count does not necessarily match the number of failed trajectories exactly, but it provides useful reference for understanding the distribution of failure modes.
question
25

---

Published as a conference paper at ICLR 2026
Failure Mode
Explanation
Example
Statistics
Entity Ambiguity
The agent chooses the wrong real-world entity among several with the same similar name and builds
The query asks for the birthplace the performer of "Step Into The Light (Myra Song)". Scarch re- sults contain
164 (20.1%)
mistaken identity.
"Myra" entitie: the Mexican. A American pop singer who performs the song, and Myra Taylor, unrelated jazz singer whose biography lists birthplace. The model lects Myra Taylor and outputs "Ronner Springs, Kansas" which irrelvannt.
Hallucination stead of Search
agent instead of searching knowledging uncertainty.
Shapiro, agent ricates occupations ("interior designer", "lawyer") claims already "rocalled" them. The true occupations politician and screenwriter.
(47.9%)
Logical Fallacies
The agent invalid imelevant criteria
Asked which director is older, compares film lease years (2005 vs 2008) rather than birth dates, similar directing ages, earlier film's director older. snippet: "Assum- ing both directors were similar ages... the recior of Fllynng Boys older since released earlier."
31 (3.8%)
pare cutitics and draw conclusions.
Insufficient
The agent repeats low-yield queries without adapt- ing afler irrelevant results..
For "In what country is Awe!", it repeats the exact and keeps retrieving pages
35 (4.3%)
Query
Refine-
samn query three about the emation "awe" or River/Bridge Awe in Scotland. For "Mräkotin", repeatedly gets unre- lated resalts Mrkonjić Grad) without refining the search strategy.
ment
Overwriling Cor-
The ageni remembers the answer but not the reu
For the question puiring King Midas and Prince's 's song, agent
700.9%
Answers Loss of Reason-
soning steps that justified
Purple hut later, lacking the full reasoning chain, focuses on gense metadata and incorrectly claims "be spocific type of ruin" mentioned, replacing the correct answer with "power ballad."
ing Path
Others
Miscellaneous failures not captured by the five
188 (23.4%)
Table 7: Summary of failure modes, explanations, examples, and statistics from the two-question setting.
26


Reviews:
[{"review_id": "xTscvBQOct", "reviewer": "Reviewer_zPnP", "summary": "This paper introduces MEM1, a reinforcement learning (RL) framework that trains LLM agents to handle long-horizon tasks by managing its own contextual memory. Instead of suffering from context exploding in the conventional approaches that append all history in the context, MEM1 trains, via RL, an agent to autonomously and iteratively consolidate its existing context, and learn to synergise its reasoning and information compression. The paper conducts extensive and delivers promising experimental results, where a 7B model trained with MEM1 outperforms a much larger 14B model as well as existing approaches like Search-R1 with significantly improved performance and memory usage on 16-objective tasks. The paper also demonstrates generalizability of the method on WebShop task.", "questions": "- What are some of the failure cases when MEM1’s context consolidation fails to be effective?\n- On Figure 2, why is the peak token for 8x objectives more than 16x objectives for some models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The problem is well motivated, targeting a critical bottleneck of using LLM agents for real-world problems. Curbing the unbounded context growth finds many applications in AI agent applications, including deep research, web agents, game playing agents, etc..\n- The proposed method is simple but effective. MEM1 encourages the AI agent to synergise reasoning and memory consolidation by designing a clever rollout mechanism in an RL pipeline. The simplicity and end-to-end nature also implies the potential scalability to solving highly complicated long-horizon tasks.\n- The paper demonstrates impressive empirical results, where MEM1 reliably outperforms baseline methods as the tasks become more long-horizon by increasing the number of objectives.", "weaknesses": "- MEM1’s reliance on verifiable and dense reward signals limits its applicability to open-ended or subjective tasks. Many realistic LLM-agent settings (e.g., creative reasoning and open-ended QA) lack such clear supervision. It is interesting how it can be extended to cases where the rewards are more implicit.\n- Some presentation issues: (1)The naming is a bit messy. The paper has used “long-turn”, “long-horizon”, and “multi-turn” throughout. Do they mean the same thing? If so, the authors should standardize the term. (2) The paper repeatedly sells “constant context size,” but the evidence is proxy metrics (peak token count excluding system prompt) that still increase with objectives (e.g., 6.40 -> 10.4×100 tokens from 2->16 objectives). There is no formal upper bound on internal state (IS) length or a proof that memory is constant w.r.t. horizon. I suggest the authors restate claims as “near-constant peak tokens under our rollout policy” to make it more rigorous.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "udQ4iW00Qn", "reviewer": "Reviewer_Dxgb", "summary": "The paper introduces MEM1, an end-to-end RL framework for language agents that maintains constant memory over arbitrarily long multi-turn tasks by updating a compact shared internal state instead of appending full histories. Training integrates memory management with reasoning, and a scalable task-augmentation scheme builds multi-objective, multi-hop environments. On internal retrieval QA, open-domain web QA, and web shopping, MEM1 delivers strong—often SOTA—accuracy with markedly improved memory and inference efficiency, especially at long horizons.", "questions": "- What are the expected limitations or bottlenecks when scaling MEM1 to hundreds or thousands of objectives/turns? Are there inherent trade-offs between memory consolidation, retrieval accuracy, and forgetting?\n\n- To what extent can MEM1’s method be positioned as an advance in lifelong or continual learning memory management (rather than purely RL agent efficiency) ?\n\n- What specific performance gains (if any) are attributable to tightly integrating reasoning/working memory within a single update, versus staged or modular designs where reasoning and memory summarization are decoupled? A targeted ablation would aid understanding.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- **Clear, Well-Structured Presentation**: The technical motivation, algorithm, and evaluation methodology are clearly described. Figures such as Figure 1 (“RL pipeline“) and Figure 2 (\"performance and efficiency scaling\") directly help crystallize the approach and the empirical insights.\n\n- **Sound, End-to-End RL Optimization**: The use of reinforcement learning to train both reasoning and memory management jointly is well argued and empirically shown to benefit generalization to longer, more complex tasks. The masking-based trajectory construction and objective computation address the non-trivial technical challenge of dynamically changing agent context during rollouts.", "weaknesses": "- **Evaluation Reflects Synthetic Compositions, Not Open-Ended Dialogue.** Benchmarks are mainly constructed by composing QA subsets, which exercises multi-turn reasoning but biases toward compositional templates. Even WebShop, though interactive, is governed by predefined tasks, constrained action spaces, and scripted reward assumptions, so the current setup under-represents genuinely open-ended interactions with ambiguous goals or shifting task boundaries.\n\n\n- **Insufficient Direct Comparison to Hierarchical and Structured Memory Architectures**: While the related work section covers many approaches, no empirical or conceptual comparison is offered with recent hierarchical/structured working memory proposals such as HiAgent [1], Zep[2] or [3]. These works explicitly address long-horizon memory management and could reveal functional or performance trade-offs.\n\n\t\n - **Lack of Ablations on Memory-Reasoning Coupling**: The core contribution is the tight integration of memory and reasoning, but no detailed ablation quantifies the benefit of this coupling versus traditional staged (e.g., memory-summarize-then-reason) approaches. Including such an ablation would clarify to what extent performance gains are due to this integration versus reinforcement learning or prompt design alone.\n\n[1] Hu, Mengkang, et al. \"Hiagent: Hierarchical working memory management for solving long-horizon agent tasks with large language model.\" _arXiv preprint arXiv:2408.09559_ (2024).\n\n[2] Rasmussen, Preston, et al. \"Zep: a temporal knowledge graph architecture for agent memory.\" _arXiv preprint arXiv:2501.13956_ (2025).\n\n[3] Sun, Haoran, and Shaoning Zeng. \"Hierarchical memory for high-efficiency long-term reasoning in llm agents.\" _arXiv preprint arXiv:2507.22925_ (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2kCdXrLHyC", "reviewer": "Reviewer_pi4p", "summary": "This paper proposes MEM1, a reinforcement learning framework that jointly learns memory consolidation and reasoning for long-horizon LLM agents. Instead of storing full interaction histories or relying on external memory, MEM1 maintains a compact internal state updated at each step and discards the previous context. The framework is trained end2end via PPO using masked trajectories that enables stable and accurate policy optimization under MEM1’s memory-constrained execution. Experiments on multi-objective question answering and WebShop navigation demonstrate that MEM1 improves long-horizon task performance while significantly reducing inference memory and latency.", "questions": "1. Can the MEM1 framework be adapted to offline datasets (e.g., human or expert trajectories) and combined with RL algorithms like GRPO? What are the key challenges in doing so? For instance, would the unobserved internal state in offline data pose a fundamental identifiability issue?\n\n2. The system prompt is excluded from token counting，but it’s part of the deployed model’s memory. Isn’t this misleading for real-world efficiency claims?\n\n3. Training only on 2-objective tasks but testing up to 16, does performance gain come from generalization or simply from avoiding context collapse? Baseline models (e.g., Qwen2.5-14B) collapse on 16-objective tasks (Table 1: EM drops to near zero), likely because their context exceeds practical limits or attention dilution occurs. MEM1 avoids this by design. But is MEM1 truly reasoning better, or just not failing catastrophically?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes to treat memory as part of the policy and learn it jointly with reasoning, instead of relying on an external memory module or retrieval system. This is a refreshing take on long-horizon agents and feels conceptually meaningful: the model actually learns what to remember rather than being manually engineered to store past information.\n\n2. Existing LLM agents don’t scale well over long contexts because their memory grows linearly (or worse) with interaction steps. MEM1 tackles this by enforcing a constant memory budget using an internal state. This is a practical and relevant contribution for real world agent deployment where inference cost matters.\n\n3. The experimental analysis is relatively thorough and covers both quantitative and qualitative dimensions. On the quantitative side, the paper reports multiple evaluation metrics, including task accuracy, long-horizon generalization performance, peak memory usage, and inference efficiency. In addition, the paper goes beyond standard benchmark reporting by including behavioral analysis of the learned policy. The qualitative inspection of internal states and step-by-step trajectories provides useful insight into how MEM1 operates.", "weaknesses": "1. EM reward design lacks ablation study and maybe limits real world applicability . Although the paper adopts EM as the sole reward signal during RL training for QA tasks, it does not use ablation study to analyze or justify this specific reward choice. For example, they do not compare EM with other potential reward signals such as token-level F1, partial matching, or step-wise retrieval rewards that might better capture intermediate reasoning quality.  Also, the assumption of verifiable rewards such as EM restricts the scope of applicability. In most real world long-horizon tasks e.g., scientific literature review, legal case analysis, or customer support, ground-truth answers are ambiguous, subjective, thus EM-style rewards are unavailable.\n\n2. Task Design lacks interdependent objectives. The multi-objective QA benchmark introduced in Section 3.3 constructs tasks by simply concatenating independent questions from HotpotQA and Natural Questions (e.g., “Which magazine was started first…?” and “The Oberoi family is part of a hotel company…?”). Crucially, these sub-questions share no entities, context, or logical dependencies, meaning the agent can solve them in isolation without cross-question reasoning or memory integration. As a result, the task primarily measures multi-task efficiency rather than true synergistic reasoning across interdependent objectives. One possible design can be referenced in M3Agent [1].\n\n3. The evaluation lacks comparison to strong, lightweight memory compression baselines that do not require external modules or reinforcement learning. Notably absent is a simple LLM-based summarization baseline, where the same Qwen2.5-7B model summarizes the full history into a fixed-length context at each turn—a strategy that could achieve similar efficiency gains without RL. The “Truncation (prompt only)” ablation (Table 1) already achieves 0.396 EM on 16-objective QA—more than double the full-history Qwen2.5-7B-Instruct—suggesting that much of MEM1’s gain may stem from its prompt and rollout design rather than the RL policy itself. \n\n[1] Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "moJ9NypwVy", "reviewer": "Reviewer_CDp4", "summary": "The authors introduce MEM1, an end-to-end reinforcement learning (RL) framework that enables agents to update a compact, shared internal state at each turn. This state jointly supports memory consolidation and reasoning by integrating essential prior information with new observations while strategically discarding irrelevant data. The framework is trained using RL with a 2D attention mask to manage the dynamic context during policy optimization. Experiments were conducted in three domains: internal retrieval QA, open-domain web QA, and WebShop navigation. The authors introduced a \"multi-objective QA\" task, augmenting datasets like HotpotQA by composing multiple questions. Results shows that Mem1 achieves competitive performance across QA and web navigation benchmarks with substantially reduced memory usage and inference latency.", "questions": "1. The tasks used in this paper are based on a final outcome reward setting. It would be interesting to explore how the model’s performance changes when applying RLVR algorithm, such as GRPO. \n2. If a model incorrectly summarizes or omits a key fact, that fact is permanently lost. How to solve this problem?\n3. It could guide the model to use a summarization tool to condense all historical information into a fixed context window. What advantages does MEM1 offer compared to this method?\n4. Do different search providers (such as Google Seach and Bing Search) have an impact on performance? What is the search parameter?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written, well-structured, and strongly motivated.\n2. On extra-long horizon tasks (16 objectives), MEM1 demonstrates both efficiency and strong performance, indicating that it effectively mitigates the “forgetting” problem in long-context scenarios. Its ability to generalize from training on 2-objective tasks to 16-objective tasks is particularly impressive, suggesting that the model has acquired a universal memory management capability rather than task-specific heuristics.\n3. A key novelty of this paper lies in integrating memory management directly into the model’s reasoning process, instead of relying on external memory modules. This design allows the agent to learn what to remember as part of its inheent decision-making strategy, which represents an elegant and conceptually unified approach to handling memory within large reasoning models.", "weaknesses": "1. The paper lacks evaluation on several representative multi-hop reasoning benchmarks, such as BrowseComp, GAIA, and WebWalker QA. Including these datasets would provide a more comprehensive assessment of MEM1’s capabilities, particularly in extra-long-horizon tasks that require complex, multi-step reasoning.\n\n2. Incomplete analysis of the Internal State.\n\n(1)\tThe paper does not provide sufficient analysis of the Internal State mechanism. It would be helpful for the authors to report statistics such as the average length of the Internal State during inference, and to clarify whether its size can be explicitly controlled, or whether it grows dynamically with the number of interaction turns. \n\n(2)\tFurthermore, the authors should discuss how the model mitigates hallucinations or inconsistencies in the Internal State representation. Presenting a few failure cases or qualitative examples where the Internal State leads to incorrect reasoning would provide valuable insight into the model’s limitations and potential areas for improvement.\n\n3. The authors claim O(1) context efficiency, but this only accounts for the attention cost. Internal State at step t must include all historical information, so its length and generation cost increase with the task complexity at each step. The authors should report the total FLOPS of MEM1 and compare with other methods (DeepResearcher, SearchR1).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)