ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
Authors: Siru Ouyang
Jun Yan
I-Hung Hsu
Yanfei Chen
Ke Jiang
Zifeng Wang
Rujun Han
Long Le
Samira Daruki
Xiangru Tang
Vishy Tirumalashetty
George Lee
Mahsan Rofouei
Hangfei Lin
Jiawei Han
Chen-Yu Lee
Tomas Pfister
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Agents
Memory Mechanism
Reasoning
Test-Time Scaling
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: With the growing adoption of large language model agents in persistent real-world roles, they naturally encounter continuous streams of tasks. A key limitation, however, is their failure to learn from the accumulated interaction history, forcing them to discard valuable insights and repeat past errors. We propose ReasoningBank, a novel memory framework that distills generalizable reasoning strategies from an agent's self-judged successful and failed experiences. At test time, an agent retrieves relevant memories from ReasoningBank to inform its interaction and then integrates new learnings back, enabling it to become more capable over time. Building on this powerful experience learner, we further introduce memory-aware test-time scaling (MaTTS), which accelerates and diversifies this learning process by scaling up the agent's interaction experience. By allocating more compute to each task, the agent generates abundant, diverse experiences that provide rich contrastive signals for synthesizing higher-quality memory. The better memory in turn guides more effective scaling, establishing a powerful synergy between memory and test-time scaling. Across web browsing and software engineering benchmarks, ReasoningBank consistently outperforms existing memory mechanisms that store raw trajectories or only successful task routines, improving both effectiveness and efficiency; MaTTS further amplifies these gains. These findings establish memory-driven experience scaling as a new scaling dimension, enabling agents to self-evolve with emergent behaviors naturally arise. Our code can be found at https://github.com/google-research/reasoning-bank.

Published as a conference paper at ICLR 2026
REASONINGBANK: SCALING AGENT SELF-EVOLVING WITH REASONING MEMORY
Siru Ouyang' Jun Yan I-Hung Hsu?, Yanfei Chen', Ke Jiang", , Zifeng Wang,, Rujun Han', Long T. Le", Samira Daruki², Xiangru Tang', Vishy Tirumalashetty? George Lee", Mahsan Rofouei", Hangfei Lin", Jiawei Han', Chen-Xo Yu Leea, Tomas Pfister"
UICC "Google Cloud AI Research Yale University 'Google Cloud Al
siruo2@illinois.edu {junyann, jennyulle)@googge.com
ABSTRACT
With the growing adoption of large language model agents in persistent real- world roles, they naturally encounter continuous streams of tasks. A key limitation, however, is their failure to leam from the accumulated interaction history, forcing them to discard valuable insights and repeat past errors. We propose REASONINGBANK, a novel memory framework that distills generalizable reasoning strategies from an agent's ' self-judged successful and failed experiences. At test time, an agent retrieves relevant memories from REASONINGBANK to inform its interaction and then integrates new learnings back, enabling it to become more capable over time. Building on this powerful experience learner, we further introduce memory-aware test-time scaling (MATTS), which accelerates and diversifies this learning process by scaling up the agent's 's interaction experience. By allocating more compute each task, the agent generates abundant, diverse experiences that provide rich contrastive signals for synthesizing higher-quality memory. The better memory in turn guides more effective scaling, establishing powerful synergy between memory and test-time scaling. Across web browsing and software engineering benchmarks, REASONINGBANK consistently outperforms existing memory mechanisms that store raw trajectories only successful task routines, improving both effectiveness and efficiency; MATTS further amplifies these gains. These findings establish memory-driven experience scaling as new scaling dimension, enabling agents to self-evolve with emergent behaviors naturally arise. Our code be found at https: /ttt::://ittee http//ggtttttttte com/ resegooolllesseerch/reeeooiooo
INTRODUCTION
The rapid advancement of large language models (LLMs) has significantly accelerated the development of interactive LLM agents (Wang et al., 2024; Liu et al., 2025a), which are crucial in tackling complex real-world tasks that require multi-turn interactions with environments. These agents have demonstrated great potential across diverse scenarios, including web browsing (Gur et al., 2024), computer use (Yang et al., 2024; Xie et al., 2024), and scientific discovery (Ghafarollahi & Buehler, 2025). As these agents are increasingly deployed in persistent, long-running roles, they naturally encounter continuous stream of tasks and interactions. However, critical limitation persists: they largely fail to learn from this accumulated experience. By approaching each new task in isolation, they are doomed to (i) repeat similar errors observed in the past (Yin al., 2025), (ii) discard valuable insights gained from related problems, and, most importantly, (iii) lack self- evolving capabilities that make the agent system more capable over time (Gao al., 2026). This phenomenon highlights the necessity of building memory-awart agent systems that could learn from their past experiences (Zhang et al., 2025b).
"This work was done while Siru interned at Google Cloud Al Research. Cooresponning authors.

---

Published as a conference paper at ICLR 2026
Trejectory Memory
Workflow Memory
18
ReasoningBank
coctioro click 1881 circtions
Rutrieve Information from Specille Page fectior
Number of Successful Taske
Marshi counts
othnica have succosefully naaggated the 'My Account pago. The user wards unow the date
spss do This workllow designod to accees speetic section <ithets
15
No Memory
Grand total Invoice
purchase. "Fos"" purchses, need sine the mpplete
MWCCO JN format
pagh.
cuction- didd'ectionn or bao,, atticno cactions
12
send meg. udonooxtraceed
order History, just the rosent enea.
Top-2 searches
These a. "Vaoy All likk next to
B ReasoningBank Memory
Nur review menticring "decent"
Recort Orciess lics should take me order humory <ttanks
cactora 1500 stactiono othiccl the
page
#4 Title fusiigation Strategy
#1 Duscription suarching
with
produatt 2003
lop- best. selling
Best so ling product type Jan 2023
Earieat customer
The
inf Content Detect onion made and
thee purdhess. The orders
hemm melecett
y displayed from most recencc to eldest. M.hiii caations 16117 chactions
orders, 2. Awold infiiit scrolls use la loades if primary made fall, 3. Cross reference with common.
20 Number of Tested Tasks
30
40
Figure I: REASONINGBANK induces reusable reasoning strategies, making memory items more transferrable for future use. This enables agents to continuously evolve and achieve higher accumulative success rates than the "No Memory" baseline on the WebArena-Admin subset.
Recent efforts on agent memory (Huang et al., 2026) have primarily focused on storing past interactions for reuse (Zhao et al., 2024; Tang et al., 2025b; Chen et al., 2025; Sun et al., 2025). However, these approaches are often limited to leveraging raw trajectories (Zheng et al., 2024; Kagaya et al., 2024; Kong et al., 2025) or successful routines (i.c., workflows, procedures) (Wang et al.. 2025d; Fang et al., 2025). They suffer from two fundamental drawbacks. First, they lack the ability to distill higher-level, transferable reasoning patterns. Second, by over-emphasizing successful experiences, they leave the valuable lessons from an agent's own failures largely underexplored (Zhang et al., 2024). Consequently, existing memory designs often remain limited to passive record-keeping rather than providing actionable, generalizable guidance for future decisions.
To bridge this gap, we propose REASONING REASONINGBANK distills and organizes memory items from both successful failed experiences judged by the agent itself without ground-truth labels. As shown in Figure 1, it captures not only effective strategies from successes but also crucial preventative lessons from failures, abstracting them into collection of actionable principles. This process operates in a closed loop: when facing a new task, the agent retrieves relevant memories from REASONINGBANK to guide its actions. Afterward, the new experience analyzed, distilled, and consolidated back into the REASONINGBANK, allowing the agent continuously evolve and improve its strategic capabilities.
novel memory framework for agent systems.
With REASONINGBANK as a strong experience learner, we study experience scaling to establish a powerful synergy between memory and test-time scaling. Instead of scaling experience through breadth by adding more tasks, we focus on scaling experience through depth by tackling each single task with more exploration. We introduce memory-aware test-time scaling (MATTS) in both parallel and sequential settings, which generates diverse exploration to provide contrastive signals, enabling REASONINGBANE to synthesize better memories. It creates synergy between memory and test-time scaling: high-quality memory steers the scaled exploration toward more promising paths, while the rich experiences generated forge even stronger memories. This positive feedback loop positions memory-driven experience scaling as new scaling dimension for agents.
We conduct extensive experiments on challenging benchmarks for web browsing (WebArena, We demonstrate that REASONING-
Mind2Web) and software engineering (SWE-Bench-Veris BANK outperforms baselines in both effectiveness (up 20% relative improvement, Table and efficiency (up to 16% fewer interaction steps, Table 1). Additionally, REASONINGBANK synergizes best with MATTS, making it an essential component for memory-driven experience scaling.
Our contributions are threefold: (1) We propose REASONINGBANK, a novel memory framework that distills generalizable reasoning strategies from both successful and failed experiences, beyond prior work that primarily stores raw trajectories or success-only routines. (2) We introduce MATTS that establishes a powerful, bidirectional synergy between memory and test-time scaling, with memory-driven experience web browsing (WebArena, Mind2Web) and software engineering (SWE-Bench Verified) tasks. We demonstrate that our approaches not only outperform baselines in effectiveness (up to 20% relative improvement) and efficiency (up to 16% fewer interactions). but also uniquely learn from failures and enable agents develop increasingly complex, emergent reasoning strategies over time.
new scaling dimension. (3) We conduct extensive experiments on
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Memory for LLM Agents. Memory has emerged an essential module in modern agent systems (Zhang et al., 2025b; Hu et al., 2025c; He et al., 2026) to enhance their performance by utilizing past information. Existing memory systems organize and store information in various forms, including plain text (Packer et al., 2023), latent embeddings ( (Wang et al., 2025b), and structured graphs (Xu al., 2025; Chhikara et al., 2025; Li al., 2025b). Beyond memory content, those methods usually involve retrieval mechanisms (c.g., semantic search) with memory management strategies (e.g., updating) (Tan et al., 2025; Hu et al.. 2025a). More recently, with the growing development of reinforcement learning (RL) in LLM agents, RL has also been leveraged for memory management in agent systems (Yu et al., 2025a; Zhou et al., 2025). While most efforts primarily emphasize personalization (Zhang et al., 2025a; Zhong et al., 2024) and long-context management (Maharana et al., 2024; Wu et al., 2025; Hu et al., 2025b), this paper falls in the research line of learning from past experiences as memory (Zhao et al., 2024; Su et al., 2025), which is a critical aspect for developing self--vollvngg agent systems (Gao et al., 2026; Liang et al., 2024). Different from previous works that emphasize reusing successful trajectories (Zheng et al., 2024; Tang et al., 2025a), procedural workflows (Wang et al., 2025d; Qian et al., 2024; Fang et al., al., 2024), or instance-level concepts (Liu et al., 2025c; Suzgun et al., 2025; Zhao et al., 2024), REASONING
2025; Liu et al., 2025b; Nottingham
stores high-level strategies and reasoning hints. By abstracting experiences into reusable reasoning units, REASONINGBANK enables agents to generalize not only from successful cases (Zhao et al., 2024; Alazraki et al., 2025; Fu et al., 2024) but also by learning from failures, providing richer guidance for test-time learning.
Agent Test-Time Scaling, Test-time scaling (TTS) (Snell et al., 2025) has demonstrated strong effectiveness and has become a widely adopted practice in end-to-end problem-solving such as coding (Li et al., 2025a; Yu et al., 2025c) and math reasoning (Muennighoff et al., 2025), where methods including best-of-N (Chow et al., 2025), beam search (Wu et al., 2024b), and leveraging verifiers (Setlur et al., 2025) are commonly employed. However, its application to multi-turn interactive scenarios, particularly agentic tasks, remains underexplored. Existing works mainly adapt the lesson learned from reasoning tasks (Zhu al., 2025) and scale different dimensions of agentic systems, including the search space for each action (Yu et al., 2025b), the number of agents in multi-agen systems (Jin et al., 2025), and the number of interactions with the environment (Shen et al., 2025). We found that none of these efforts considers the role of agent memory in scaling, where an agent can learn from past experiences guide future decisions. Our work extends this line of research by introducing memory-aware test-time scaling (MATTS), As we will show in our empirical results ($4.3 and $4.4), memory offers benefits beyond mere computational scaling, where memory and scaling synergistically work towards better performance.
METHODOLOGY
In this section, we introduce the problem setup ($3.1), and present our proposed REASONINGBANK ($3.2), based on which we further develop memory-aware test-time scaling (MATTS) (53.3).
3.1 PROBLEM FORMULATION
Agent Configuration. The scope of this work focuses on language model (LM)-based agents. The agent policy T (,(, is parameterized by the backbone LLM L, conditioned on a memory module M, and the action space A, denoted as Te. for short. The agent needs to perform task via interacting with the environment, which can be viewed as sequential decision-making process. Formally, the transition function of the environment is defined as T (s (tt, az) where s, is the state and a, is the action selected by Fg at time t. We focus on web browsing and software engineering (SWE) tasks. A is set of web navigation operations for web browsing and bash commands for SWE, M is REASONINGBANK and initialized as empty. For each given task, the agent generates a trajectory of (oo.t, (g:t)) for i steps, where observation o, is from the current state 8,. Observations are text-based accessibility tree of web pages' for web browsing tasks and code snippets for SWE. The
We use the thinking process of me the approximation of oo due to lengthy observation representations following Wang et al. (2025d).
3

---

Published as a conference paper at ICLR 2026
ReasoningBank: a collection of memory items
Title: Prioritize account sections for personal data
FV
Memory::
Description When a query requesta user-specific.
(i) Mernory Retrieval
(ii) Mernory Consolidation
item
Content: Systemetically look and click links
Memory Extractor
(ii Memorr Extraction
3f: experienca/rajectory
Suciom Iniighs
SnterrianowTrrjactton
<thinko.... nevigete to.... Chhinko
<actiono cickk" 189 slactiono
Memory Items
aJJudge
Agent
Env
Redection
Task g): me the status my order and when it will arrive
Task
Tesk 4:
Task
Task
Time
Figure 2:
Overview of REASONINGBANK. Experiences are distilled into structured memory items with title, description, and content. For each new task, the agent retrieves relevant items to interact with the environment, and constructs new ones from both successful and failed trajectories. These items are then consolidated into ReAsONINGBANE forming closed-loop memory process.
agent needs to generate an action at-1 E A via Tz (0u.; ao:4: ao:; AM. - a4+1. For implementation, the memory module M contributes relevant memories as additional system instruction for TT.
Test-Time Learning. We focus on the test-time learning paradigm (Wu et al., 2024a; Wang et al., 2025c) where a sequence of task queries Q {qi 92.-.. 9N) arrives in a streaming fashion, i.e., each query is revealed and must be completed sequentially without access to future ones. In this setting, no ground truth is available during test-time, the agent must continually evolve by only leveraging its own past trajectories and any self-verification without relying on external labels. This streaming setting highlights two key challenges: (i) how to extract and preserve useful memory from past trajectories, and (ii) how to effectively leverage such memory for future queries to avoid redundantly re-discovering already successful strategies or repeating past mistakes.
3.2 REASONINGBANK
Past raw trajectories (or experiences), while being comprehensive and original, often too lengthy and noisy to be directly applied to the current user query. As illustrated in Figure 2, REASONINGBANK distills useful strategies and reasoning hints from past experiences into structured memory items, which are then stored in the agent's 's memory for future reuse.
Memory Schema. Memory items in REASONINGBANK are designed and induced from past experiences as structured knowledge units that abstract away low-level execution details while preserving transferrable reasoning patterns and strategies. Each memory item specifies three components: (i) title, which serves as concise identifier summarizing the core strategy or reasoning patter;; (ii) description, which provides a brief one-sentence summary of the memory item; and (iii) the content, which records the distilled reasoning steps, decision rationales, or operational insights extracted from past experiences. Together, memory items extracted are both human- iannan-interpretable and machine-usable, facilitating efficient usage and integration with agents.
Integrating REASONINGBANK with Agents. An agent me equipped with REASONING can draw upon curated pool of transferable strategies to guide decision-making, This enables the agent to recall effective insights, avoid previously observed pitfalls, and adapt more robustly to unseen queries. The integration proceeds in three steps: (i) memory retrieval, (ii) memory extraction, and (iii) memory consolidation, as shown in Figure 2. During memory retrieval, the agent queries REASONINGBANN with the current query context identify the top-k relevant experiences and similarity search. Retrieved items are
their corresponding memory items using embedding- injected into the agent's system instruction, ensuring that action prediction from A is grounded with useful past experiences. When the current query task is completed, we will perform memory extraction to extract new memory items. The first step to obtain proxy correctness signals for trajectories: we adopt an LLM-as-a-judge (Gu et al., 2024) to label outcomes success or failure given the query and trajectory without any ground-trutt reference. Based on these signals, we apply different extraction strategies: successful experiences contribute validated strategies, while failed ones supply counterfactual signals and pitfalls that help sharpen guardrails. In practice, we extract multiple memory items for each trajectorylexperience consolidation incorporates these items into REASONINGBANK with a simple addition operation, maintaining evolving repository of memory items. Details in Appendix A.2. Together, these
detailed in Appendix A. Finally, memory
4

---

Published as a conference paper at ICLR 2026
Men
Traj
New
Trajectory
Self Refine
Mem 2
Current memory
Tajectory 2
New momory
Current memory
New
Mem
Self- Contrand
Trai
Traj"
Memai
Trjectory !
Task
Task
Task 4:
Task 4/+1
(a) (MaTTS aggragation)
(b) MaTTS Parallel
(c) Soquential
Figure 3: Comparison of (a) vanilla T7S that independently run REASONINGBANK for multiple times without aggregating on trajectories, MATTS with (b) parallel scaling, where self-contrast across multiple trajectories curates reliable memory, and (c) sequential scaling, where self- refinement enriches memory with intermediate reasoning signals.
steps form a closed-loop process: the agent leverages past experiences, constructs new memory from current tasks, and continually updates its memory, enabling sustained evolvement in test-time learning scenarios.
3.3 MATTS: MEMORY-AWARE TEST-TIME SCALING
REASONINGBANK enables learning from experiences to translate more experiences into greater improvements. As test-time scaling (TTS) (Snell al., 2025) recently emerged powerful strategy for boosting the performance of LLM agents (Zhu et al., 2025), it shows strong potential by allocating additional inference-tima computation generate abundant exploration histories. A vanilla combination is depicted in Figure 3a), where more trajectories are independently converted more memory items. However, this is suboptimal because does not leverage inherent contrastive signal that arises from abundant explorations resulting performance advantage brought by TTS. To address this, we propose Memory-aware Test-Time Scaling (MATTS), novel integration of test-time scaling with our memory framework, REASONINGBANK. Unlike the vanilla approach, MATTS learns by deliberately aggregating from the abundant successful and failure trajectories generated during scaling for more effective memory curation. We design two complementary instantiations for MATTS, parallel scaling and sequential scaling, illustrated in Figure 3(b) and 3(c) with detailed implementation in Appendix A.3.
the same problem, which limits the
Parallel Scaling, In the parallel setting, we generate multiple trajectories for the same query under the guidance of retrieved memory items. By comparing and contrasting (self-contrast (Chen et al., 2020)) across different trajectories, the agent can identify consistent reasoning patterns while filtering out spurious solutions. This process enables more reliable memory curation from multiple trials of a single query and promotes diverse yet grounded exploration.
Sequential Scaling. We iteratively refine its reasoning within a single trajectory after the initial completion, following the principle of self-refinement (Madaan et al., 2023). In this process, the intermediate notes generated in self-refinement are also used as valuable signals for memory, since they capture reasoning attempts, corrections, and insights that may not appear in the final solution.
We define the scaling factor denoting the number of trajectories for parallel scaling and refinement steps for sequential scaling. Equipped with REASONINGBANK, both parallel and sequential strategies become memory-aware, ensuring that the additional computation allocated at test time translates into more transferable and higher-quality memory for future tasks.
EXPERIMENTS
4.1
SETUP
Following existing work (Wang al., 2025d), we conduct experiments WebArena (Zhou et al., 2024) which features general web navigation across diverse domains?, and Mind2Web (Deng
2 We deliberately keep the memory usage pipeline simple, avoiding additional complexity in retrieval consolidation so as to highlight the contribution ReasoniNG itself. These components, however, can he further enhanced with more sophisticated techniques, which could provide additional henefits.
We exclude the domain of Map due website issues following Miyai (2025) for fair comparison.
5

---

Published as a conference paper at ICLR 2026
Table 1: Experiment results of REASONINGBANK and MATTS (parallel scaling, k 5, pass @ 1) on WebArena benchmark. Success rate (SR 1) and the number of steps (Step L) are reported on subsets for 3 different backbone LLMs.
Shopping (187)
Admin (182)
Gitlab (180)
Reddit (106)
Multi (29)
Overall (684)
Models
SR
Step
SR
Step
SR
Step
SR
Step
SR
Step
SR
Step
Bemini-2.5-flaxs
No Memory
39.0
8.2
44.5
9.5
33.9
13.3
55.7
6.7
10.3
10.0
40.5
9.7
Synapse
40.6
7.0
9.1
35.6
13.0
59.4
6.5
10.3
10.5
42.1
9.2
AWM
44.4
7.0
46.7
8.8
37.2
13.2
62.3
6.1
3.4
7.7
44.1
9.0
REASONINGBANK
49.7
6.1
51.1
8.2
40.6
12.3
67.0
5.6
13.8
8.8
48.8
8.3
+MATTS
53.0
6.3
53.8
7.6
42.8
11.9
70.8
5.4
17.2
8.0
51.8
7.9
Gemini-2.5-pm
No Memory
45.5
7.6
51.1
8.7
35.0
11.6
71.7
6.0
6.9
8.8
46.7
8.8
Synapse
46.5
52.2
8.9
11.3
68.9
5.9
9.0
47.7
8.5
AWM
48.1
6.4
49.3
9.8
40.0
11.2
68.9
6.4
3.4
9.3
47.6
8.7
REASONINGBANK
51.9
6.0
56.6
7.7
44.4
9.8
80.2
5.1
13.8
8.2
53.9
7.4
+MATTS
54.0
5.9
58.2
7.4
46.7
9.1
83.0
5.3
20.7
7.2
56.3
7.1
-laude-3.7-sonnce
No Memory
38.5
6.1
49.5
8.4
36.7
10.6
53.8
5.5
11.6
41.7
8.0
Synapse
39.6
50.5
8.5
38.0
10.0
53.8
6.1
11.8
42.6
7.9
AWM
39.6
7.2
47.8
9.3
34.6
10.9
52.8
7.0
0.0
12.4
40.8
8.9
REASONINGBANK
44.9
5.6
53.5
7.6
41.1
9.5
55..
5.2
3.4
10.5
46.3
7.3
*MATTS
47.1
55.5
7.4
9.4
5.0
9.1
48.8
7.2
al., 2023) that tests generalization of agents versatile operations and environments. We also conduct experiment on SWE-Bench-Veriffee for repository-level issue-resolving. For comparison, consider baselines ranging from memory-free agents (No Memory) to trajectory-based memory (Synapse) (Zheng et al., 2024) and workflow-based memory (AWM) (Wang et al., 2025d). Our agents are built on Gemini-2.5 (Comanici al., 2025) and Claude-3.7 (Anthropic, 2025) models using BrowserGym (de Chezelles et al., 2025) for web browsing and bash-only for SWE. following ReAct (Yao et al., 2023) style with default decoding configurations. We evaluate effectiveness (success rate, SR) and efficiency (average steps, AS), with specific metrics varying for each dataset. Full descriptions for datasets, basclines, implementations, and evaluation are in Appendix B.
4.2 RESULTS OF REASONINGBANK
Tables 1, 2, 3 summarize the main evaluation results of REASONINGBANK on WebArena, Mind2Web, and Se--Benhh-eerfiee accordingly. We have the following observations.
REASONINGBANE consistently outperforms baselines across LLM backbones on all datasets. Specifically, REASONINGBANE improves the overall success rate WebArena (Table 1) by +8.3, +7.2, and +4.66 with three different backbone LLMS compared to memory-free agents. A similar pattern holds on Mind2Web (Table 2), where REASONINGBANK delivers clear gains across cross- task, wross-website, and cross-domain settings, underscoring both the consistency and scalability of its benefits across datasets and model sizes. Results SWE-Bench-Veriee confirm its robustness. Crucially, unlike baselines such as Synapse and AWM that rely on narrow, homogeneous memory source derived exclusively from successful trajectories, REASONINGBANK employs superior extraction strategy that is key its consistent out-performance.
(Table 3) further
REASONINGBANK enhances generalization with better transferrable memory across tasks. We also evaluate in challenging generalization settings. On WebArena (Table 1), the Multi subset requires transferring memory across multiple websites, where REASONINGBANI achieves a notable gain of +4.6 averaged SR over the strongest baseline. In contrast, strong baselines such as AWM fail to provide gains and even degrade in this setting. On Mind2Web (Table 2), which includes cross-task, cross- website,, and cross-domain evaluations that impose progressively higher demands, REASONINGBANK consistently improves success rates. The gains are especially pronounced the cross-domain setting, which requires the highest level of generalization. These results demonstrate

---

Published as a conference paper at ICLR 2026
Table 2: Results Mind2Web benchmark for cross-task, cross- and cross-domain generalization test. EA (1) is short for element accuracy, AF1 (f) is short for action F1, and SSR (1) short for step success rate. SR (1) the task-level success rate measuring if all steps are correct.
Cross-Tass
Cross- Website
Models
EA
AF1
SSR
SR
EA
AF
SSR
SR
EA
AF1
SSR
SR
Gemini-2.5
2.5-slahh
No Memory
46.0
59.1
40.3
3.3
39.8
45.1
31.7
1.7
35.8
37.9
31.9
1.0
Synapse
47.0
59.5
41.2
3.5
40.3
46.0
32.1
1.9
36.3
38.5
32.4
1.1
AWM
46.3
56.1
41.0
3.5
39.1
42.2
31.7
2.1
33.3
36.5
0.7
REASONINGBANK
52.1
60.4
44.9
4.8
44.3
52.6
33.9
2.3
40.6
41.3
36.6
1.6
Gemini-2.5-pry
No Memory
49.3
60.2
44.4
3.5
41.2
49.8
34.8
3.4
37.9
37.7
35.0
1.4
Synapse
50.1
61.0
44.7
3.6
41.8
51.2
35.0
3.2
38.5
39.8
35.6
1.5
AWM
48,6
61.2
44.4
3.7
41.9
47.9
34.8
2.3
37.3
38.1
34.4
1.2
REASONINGBANK
53.6
62.7
45.6
5.1
46.1
54.8
36,9
3.8
42.8
45.2
38.1
1.7
that memory curated by REASONINGBANK is more robust and transferable, enabling agents to generalize effectively across diverse scenarios.
BAAK achieves superior efficiency by leveraging past experiences memory. In addition to higher success rates, REASONINGBANE also reduces the number of interaction steps needed to complete tasks, as shown in the Step metric of Table and 3. On WebArena, across almost all subsets and backbones, REASONINGBANK lowers the average step count by up to 1.4 compared with "No Memory", and 1.6 compared with other memory baselines. The average step on SWE-Bench- -ee--ennc--eriiiied is also smaller by saving 2.8 and 1.3 steps respectively. This indicates that RFASONINGBANK enables agents to solve tasks more efficiently by reusing and refining reasoning knowledge, thus avoiding unnecessary or redundant exploration.
Table 3: Experiment results of REA- SONINGBANK on SWE-Bench-Veriiee dataset for issue-resolving a given repository.
Methods
Resolve Rate
AS
Gemini-2.5-flassh
No Memory
34.2
30.3
Synapse
35.4
30.7
REASONINGBANK
38.8
27.5
Gemini-2.5
Memory
54.0
21.1
Synapse
53.4
21.0
REASONINGBANK
57.4
19.8
4.3 RESULTS OF MATTS
We first present the overall results of MATTS on WebArena in Table I, which demonstrate additional strong performance improvement and efficiency gains with REASONINGBANK: To study the scaling effect in-depth with both parallel and sequential variants, we experimented MATTS with Gemini-2.5-flash on Sebarena-Shopping subset. To investigate the overall scaling effect, we benchmark with (i) MATTS w/o memory, which represents the scaling setting without memory mechanism, (ii) MATTS w/o aggregation, which equal to Vanilla TTS in Figure 3(a) and (iii) MATTS to demonstrate the effect with respect to scaling factor k. Notably, k 1 is the setting without scaling. For parallel scaling, we compute Best-of-N (BoN) as the final metric detailed in Appendix A.3. Results shown in Figure 4.
Both parallel scaling and sequential scaling boost performance. Increasing k generally improves success rate, confirming the benefit of allocating more inference-tine computation. With MATTS, parallel scaling grows from 49.7 (k = 1) to 55.1 (k = 5), while sequential scaling rises from 49.7 to 54.5. For the baseline of MATTS w/o memory, the gains are smaller and less consistent (e.g.. parallel scaling fluctuates between 39.0 and 42.2 and sequential scaling fluctuates between 37.4 44 and 40.6). In contrast, MATTS enables stronger and more stable improvements across both scaling strategies, highlighting its role in making scaling more effective.
MATTS is consistently better than vanilla TTS. With REASONINGBANK, MATTS consistently surpasses MATTS w/o aggregation (vanilla TTS), showing that memory aware coordination and aggregation is important. Specifically, l MATTS achieves 55.1 in parallel scaling compared
By default, MATTS integrates REASONINGBANS but it could also use other memory mechanisms.

---

Published as a conference paper at ICLR 2026
with 52.4 for vanilla TTS, and 54.5 versus 51.9 in sequential scaling. These improvements highlight that meao scaling effectively directs the agent toward more promising solutions by synthesizing insights from multiple trajectories or interaction steps to leverage contrastive signals.
Sequential scaling shows short- term advantage, but parallel domi- nates at larger scales for REASON- INGBANK. With stronger memory mechanisms such as REASONING- BANK, sequential refinement brings higher gains at small k, but its benefit quickly saturates-once the model either succeeds or fails decisively, further refinements add little new insight. In contrast, parallel scaling continues to provide diverse rollouts that allow the model to critique and improve upon its own generations, leading it to surpass sequential at larger & (e.g., 55.1 vs. 54.5 k 5). In contrast, for vanilla TTS which is not equipped with memory module, sequential scaling yields little or even no benefit as scaling goes on, and parallel scaling consistently dominates.
MoTTS w/o memory
MaTTS w/o aggregation
Matrs
56
56
92.4
$6.0
52.5
Success Rata (SR)
52
50.9
52
$2.9
51.9
48
48
422
29.0
39.4
40.1
37.4
36
2
4
5
(a) Parallel Scaling
Sequential Scaling
Figure 4: Effect of scaling factor is for MATTS under with REASONINGBANK on WebArena WebArena-Shopping subset. We compare (a) parallel and (b) sequential test-time scaling.
4.4 SYNERGY OF MEMORY AND TEST-TIME SCALING
While the previous section establishes the overall effectiveness of MATTS, we highlight the synergy between memory and TTS in this section. Figure 5 presents snapshot of MATTS on the WebArena- Shopping subset with parallel scaling factor = 5, where we report both Pass (randomly selected trajectory) and Best-of-5 (BoN). This setting allows us to examine the bidirectional interaction between memory quality and scaling effectiveness.
Better memory enables stronger test-time scaling performance. To see how memory improves the effectiveness of scaling, we focus on the BoN results, which directly measure an agent's ability to surface the best outcome among multiple rollouts. As shown by blue
60
No Scaling
Pass@1
Success Rate (SR)
55
Best-of-5
53.0
50
47.6
49.7
44.4
42.2
41.2
44.4
bars in Figure 5, the benefit of scaling depends critically on the underlying memory. Without memory, scaling yields slight improvement, with BoN rising only from 39.0 to 42.2. Weaker memory mechanisms such as Synapse and AWM provide moderate gains, reaching 44.4 and 47.6, respectively. In contrast, MATTS with REASONINGBANK delivers the strongest benefit, with BoN climbing from 49.7 to 55.1. These results show that high-quality memory directs scaling toward more promising rollouts, ensuring that the additional trajectories are not wasted but converted into higher success rates.
40
39.0
39.0
35
Memory
Synapse
AWM ReasoningBank
Figure Snapshot of MATTS on WebArena- Shopping subset with different memory mecha- nisms with 5. We compute BoN for all 5 trajectories and Pass@1 with one randomly selected trajectory.
Scaling yields better memory curation. To fairly evaluate how scaling feeds back into memory, we report Pass @ 1, which measures the average quality of trajectories after memory curation and allows direct comparison with the no-scaling case. The trend is depicted in pick bars and is striking: scaling actually reduces performance for weaker memories, where Synapse slightly increases from 40.6 to 41.2, and AWM from 44.4 to 45.5. These phenomenon suggest that without strong guidance, the extra rollouts generated by scaling offer diminishing returns. In contrast, REASONINGBAN provides far more benefits: Pass@1 rises from 49.7 to 53.0, showing that high-quality memory can harness the diversity of scaling extract constructive contrastive signals. This asymmetry highlights that scaling alone is insufficient; only when paired with good memory mechanism does it contribute curation of more effective memory, closing the virtuous cycle.
8

---

Published as a conference paper at ICLR 2026
5 ANALYSIS
We analyze REASONINGBANK through several aspects: incorporating failure trajectories, examining emergent strategies, and evaluating efficiency across both successful and failed cases. Additional analyses are presented in Appendix C, including but not limited number of retrieved experiences, additional results on smaller open-source model, and inference cost study.
Emergent behaviors with REASONINGBANK. We find that the strategies in REASONINGBANK are not flat or monolithic, but instead evolve over time, exhibiting emergent behaviors that resemble the learning dynamics of RL (Wang et al., 2025a). As illustrated in a human case study in Figure 8, memory items describing a specific strategy "User-Specific Information Navigation" in REAS could gradually evolve during test-time learning process. It starts from dnccution-orientee or procedural strategies (e.g., find navigation links), where the agent follows straightforward action rules. It then progresses to adaptive self-reflections such re-verifying identifiers to reduce simple mistakes. With more experiences, the same memory item evolves into adaptive checks, where the agent systematically leverages available search or filters ensure completeness before results. Finally, it eventually matures into compositional strategies such as crose-eefn task requirements and reassessing options. This evolution highlights how REASONINGBANK enables agents to refine strategies from low-level actions to high-level reasoning during test-time learning.
REASKNINGBANK makes good use of failure trajec- tories. Figure 6 compares different memory designs on WebAreea-SSoppping with Gemini-2.5-flash under two settings: using only successful trajectories versus leveraging both successes and failures. Baseline methods such Synapse and AWM build memory solely from successful trajectories, and thus are not equipped to benefit from failures. As a result, when failures are added, their performance is limited or even degraded: Synapse increases only from 40.6 (success only) to 41.7 (with failures), while AWM drops from 44.4 to 42.2. In contrast, the design of REASONINGBANK enables distillation of reasoning patterns from both successes and failures, achieving 46.5 on success-only traces and further improving to 49.7 when failures are included. This highlights that, unlike baselines, REASONINGBANK can transform failures into constructive signals rather than noise, enabling more robust generalization.
50
Suncess only
89.7
47
Success Rate
w/ Failuro
44
44.4
41
41.7
38
Nomem: 39.0
35
Synapse
AWM
ReasoningBank
Figure 6: Ablation results of incorpo- rating failure trajectories for memory induction.
REASONINGBANK exhibits robustness to LLM-as-a- Judge calibration. A critical step in our method sourcing proxy correctness signals for agent trajectories via an LLM-as-a-Judge. Here we quantitatively calibrate this judge and analyze its robustness to verification noise. We conduct our analysis on the Ser-rren--Shppping subset, the judge. We first establish the baseline accuracy by comparing the judge's predictions against ground-truth labels, which find to be 72.7%. To systematically study the robustness of REASONINGBANK the judge quality, we simulate different levels of verification accuracy. This is achieved by probabilistically correlating the judge's labels with the ground-truth. For example, 100% accurate verifier uses the ground-truth labels directly. A 90% accurate verifier is simulated by using the correct (ground-truth) label 90% of the time and the incorrect (flipped) label 10% of the time. We extend this simulation down to 50% accuracy, which represents random guess baseline for this binary (success/failure) classification task. The results presented in Figure 7. We observe that the judge's 's accuracy does not significantly impact the performance of REAsONINGBANE as all variants achieve similar success rates within reasonable accuracy range (70%-90%). Intuitively, the 100% (ground-truth)
52.4
51.8
Success Rate
50.6
49.4
49.7
using Demini-2.5-fflat
48.2
48.7
47.6
47.6
100
90
80
60
50
Simulated LLM-es-a-Judge Accuracy
Figure 7: Results for SR w.r.t simulated LLM-as-a-judge accuracy.

---

Published as a conference paper at ICLR 2026
cnucial re.sheck element's current dentifier
Atomic selt-rellection
Generalized complex strategy
Regularly cross referennenni the current view with the task requirements helps pravent arror.... doesn't align with expactations (a.g. contents incurrect irre frrelevan), reassess available options search filters alternative sections.
Procedural/axecction
Before scanning. always leverage any or functionalities, enzure completeness before reporting
currnnt
stratggy
actively look for and click on "Next Page."
"Page X,"
"Load More" links.
Evolved adaptive check
Learning
Figure 8: A case study illustrating emergent behaviors in REASONINGBANK through memory items.
Table 4: Average number of steps on successful and failed test instances across four WebArena domains. REASONINGBANK consistently reduces the number of steps compared to the vanilla baseline, with notably larger reductions on successful instances.
Shopping
Admin
Gitlab
Reddit
Models
Successful
Failed
Successful
Failed
Successful
Failed
Successful
Failed
No Memory REASONINGBANK 4.7
6.8
8.7
8.4
10.4
8.6
15.7
6.1
7.6
4.3
7.041.4
9.5
7.6
15.5
5.0.1.1
6.8
accuracy setting yields the best performance. These findings confirm that REASONINGBANK is robust noise in the verification step.
REASONINGBANK delivers targeted efficient gains. While the overall number of steps in Table provides a general view of model efficiency, does not distinguish whether reductions come from successful or failed trajectories. To gain deeper insight, further separate the analysis into successful and failed test cases, which allows us to understand the source of step reduction: desirable system should reduce unnecessary exploration when it is the right track, rather than merely cutting short failed attempts. The results are shown in Table 4. We find that REASONINGBANK consistently reduces the number of steps across all domains compared to the baseline. More importantly, the reduction is particularly pronounced on successful cases, reaching up 2.1 fewer steps (a 26.9% relative reduction) than on failed ones. This indicates that REASONINGBANE primarily helps the agent reach solutions with fewer interactions by strengthening its ability to follow effective reasoning paths rather than simply truncating failed trajectories, which highlight the role of memory in guiding purposeful decision-making and improving efficiency in practice.
6 CONCLUSION
We introduce REASONINGBANK, memory framework that distills strategy-level reasoning signals from both successes and failures and integrates them into test-time scaling (MATTS). Extensive experiments show that REASONINGBANK consistently improves performance while reducing redundant exploration. Further results reveal a strong synergy between memory and scaling: REASONINGBANE guides scaling toward more promising rollouts, while diverse rollouts enrich memory with valuable contrastive signals. We also provide analyses of individual components and emergent behaviors. Our findings suggest a practical pathway toward building adaptive and lifelong- learning agents, with additional future directions and limitations in Appendix D and E.
ACKNOWLEDGEMENT
We thank Jiao Sun, Jing Nathan Yan, Chi Wang, and members from Google Cloud AI Research for their valuable feedback during the preparation of the paper. Siru supported by the Molecule Maker Lab Institute: An Al Research Institutes program supported by NSF under Award No. 2019897.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work does not involve human subjects, sensitive personal data, or any tasks requiring Institutional Review Board (IRB) approval. All datasets used in our experiments (WebArena, Mind2Web, SWE-Bench- -eEEBencc-Veriiied are publicly available and widely used in the community; detailed descriptions and preprocessing steps are provided in Appendix B to ensure transparency and compliance. Our methods, REASONING efficiency and generalization of LLM-based agents in web navigation and software engineering tasks. While these advances aim to benefit research adaptive and lifelong-learning agents, we are mindful of potential concerns regarding misuse (c.g., automation of harmful tasks). We therefore restrict our experiments to established academic benchmarks and commit to open-soureing code under research-only license to encourage responsible use. We have carefully followed the ICLR Code of Ethics throughout this research and submission process.
and MATTS, focus improving the effectiveness,
REPRODUCIBILITY STATEMENT
We have made extensive efforts ensure the reproducibility of our work. The full methodology of REASONINGBANE and Memory-Awar Test- Test-Time Scaling (MATTS) is detailed in Section 3, with additional implementation details provided in Appendix A. Experimental setups, including datasets (WehArena, Mind2Web, SWE-Bench- (WF--eench-Verifeed), baselines, evaluation protocols, and environment configurations, are described in Section 4.1 and Appendix For clarity, we provide ablation studies (Section 5), efficiency analyses (Section 5), and emergent behavior case studies (Section 5) to validate the robustness descriptions of the memory schema, retrieval, construction, and consolidation pipeline in Section 3.2 and Appendix A.1-A.2, and we outline the scaling procedures for both parallel and sequential variants in Section 3.3 and Appendix A.3. We have released our full codebase the open-source community to foster transparency, reproducibility, and future research.
our findings. To further facilitate replication, we include precise
REFERENCES
Lisa Alazraki, Maximilian Mozes, Jon Ander Campos, Tan Yi-Chern, Marek Rei, and Max Bartolo. No need for explanations: LLMSs can implicitly learn from mistakes in-context. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.). Proceedings of the 2025 Conference on Empirical Methods Natural Language Processing, pp. 33191-33215, Suzhou, China, November 2025. Association for Computational Linguussic. ISBN 979-8-89176 332-6. doi: (0.18653/v1/2022 wtt.2222..emmpmmmme mamulp-main.10nn 1686. URL https://acllatthol.gg http:eg...rr//////////// emnlp-main. 1686/
Anthropic. Claude 3.7 sonnet and claude code, 2025. URL claude-3-7-sonnet
Silin Chen, Shaoxin Lin, Xiaodong Gu, Yuling Shi, Heng Lian, Longfei Yun, Dong Chen, Weiguo Sun, Lin Cao, and Qianxiang Wang. Swe-exp: Experience- software issue resolution. ArXiv preprint, abs/2507.23361, 2025. URL
2533.33311.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020. URL URLhtttp:/// wwwphhtpp/ppooo/ http::/pproceedinggs mlr. press/v119/chen2gg html.
Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory. ArXiv preprint, abs/2504.19413, 2025. URL https://arxivv www..orgg/bb////0/// 19413.
Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Aviral Kumar, Rishabh Agarwal, Sridhar Thiagarajan, Craig Boutilier, and Aleksandra Faust. Inference- fine-tuning for best-of-n sampling in large language models. The Thirteenth International Conference on Learning Representations, 2025. http://////peeeeeeeeeeeeeeeeoon/

---

Published as a conference paper at ICLR 2026
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. ArXiv preprint, abs/2507.06261, 2025. URL https://arr wwwlii..rgggbbbb./ org/abs/2507. 06261.
Thibault Le Sellier de Chezelles, Maxime Gasse, Alexandre Lacoste, Massimo Caccia, Alexandre Drouin, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han LU, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Graham Neubig, Quentin Cappart, Russ Salakhutdinov, and Nicolas Chapados. The browsergym ecosystem for web agent research. Transactions Machine Learning Research, 2025. ISSN 2835-8856. URL https: /ttps:///peerevee// http:///peeeeeeeeee.eeeoooooooaa
Expert Certification.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samual Stevens, Boshi Wang. Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http://papers... nips. cc/ pacppppprra. wttppaaalleee//////alaaaaaaaeeep wwwps//ppper///// pttppppppp/////aa///p/ 60 Awstract--ataeaaeeeaeeeee and Benchmarks. html.
Rumnan Fang, Yuan Liang, Xiaobin Wang, Jialong Wu, Shuofei Qiao, Pengjun Xie, Fei Huang. Huajun Chen, and Ningyu Zhang. Memp: Exploring agent procedural memory. ArXiv preprint, 0bs//208.06433,, 2025. URL nttps://arxee hwt.:/orgga////////ee/e 2505//0088.........
Zafeirios Fountas, Martin Benfeghoul, Adnan Oomerjee, Fenia Christopoulou, Gerasimos Lampouras, Haitham Bou Ammar, and Jun Wang. Human-inspired episodic memory for infinite context L.Mss. In The Thirteenth International Conference on Learning Representations, 2025. URL wwwph:tts///ott http::::///ppeevvti//ee eviee.. www./eeooonddd-tiniiiiiiooo
Yao Fu, Dong-Ki Kim, Jackyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bac, and Honglak Lee. Autoguide: Automated generation and selection of context- aware guidelines for large language model agents. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URI. http: /ttp:rr/ppper..tt wtwp.cc/ppppprrrreeeccpppppppppp www.ff//paeee/////eeees hatppr//224//aaeee d8efbb5dd4
html.
Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Qihan Ren, Yiran Wu, Hongru WANG, Han Xiao, Yuhang Zhou, Shaokun Zhang, Jiayi Zhang, Jinyu Xiang, Yixiong Fang, Qiwen Zhao, Dongrui Liu, Cheng Qian, Zhenhailong Wang, Minda Hu, Huazheng Wang, Qingyun Wu, Heng Ji, and Mengdi Wang. A survey of self-evolving agents: What, when, how, and where to evolve on the path to artificial super intelligence. Transactions on Machine Learning Research, 2026. ISSN 2835-8856. URL Survey Certification.
https://openreww
Alireza Ghafarollahi and Markus J Buchler. Sciagents:
automating scientific discovery through bioinspired multi-agent intelligent graph reasoning, Advanced Materials, 37(22)2443523,, 2025. URL htw.h:tts:///eet /ooi. org/10. 1002/ adma. 202413523.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. ArXiv preprint, abs/2411.15594, 2024. URL https://aaxiv https////axvv..o//ab///////s
orgg/bs///111.1.... 15594.
Izzeddin Gur, Hiroki Furuta, Austin V. Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: https:// //opnreviwe..r.........
12

---

Published as a conference paper at ICLR 2026
Zexue He, Yu Wang, Churan Zhi, Yuanzhe Hu, Tzu-Ping Chen, Lang Yin, Ze Chen, Tong Arthur Wu, Siru Ouyang. Zihan Wang, Jiaxin Pei, Julian McAuley, Yejin Choi, and Alex Pentland. Memoryarena: Benchmarking agent memory in interdependent multi-session agentic tasks, 2026. URL https::p////o..m //arxiv org hts/abs/22600 2602. 16313.
Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. HiAgent: Hierarchical working memory management for solving long-horizon agent tasks with large language model. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 12779-32798, Vienna, Austria, 2025a. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/1120225..mm acl- acl-oog...ll-.....55. 1575. URL https://acllanthlloog
wwg. acl acl-long. 1575/.
Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in LLM agents via incremental multi-turn interactions. In ICML 2025 Workshop on Long-Conten, Foundation Models, 2025b.
URL httph:::////pereeieeee wwwp://etffffriiee nett/forummioo72000021111
Yuyang Hu, Shichun Liu, Yanwei Yue, Guibin Zhang, Boyang Liu, Fangyi Zhu, Jiahang Lin, Honglin Guo, Shihan Dou, Zhiheng Xi, et al. Memory in the age of ai agents. ArXiv preprint, abs/2512.13564, 2025c. URL httpss://axiiii orgg/bs/////0 13564.
Wei-Chich Huang, Weizhi Zhang, Yueqing Liang, Yuanchen Bei, Yankai Chen, Tao Feng, Xinyu Pan, Zhen Tan, Yu Wang, Tianxin Wei, Shanglin Wu, Ruiyao Xu, Liangwei Yang, Rui Yang, Wooseong Yang, Chin- Yuan Yeh, Hanrong Zhang, Haozhen Zhang, Siqi Zhu, Henry Peng Zou, Wanjia Zhao, Song Wang, Wujjang Xu, Zixuan Ke, Zheng Hui, Dawei Li, Yaozu Wu, Langzhou He, Chen Wang, Xiongxiao Xu, Baixiang Huang, Juntao Tan, Shelby Heinecke, Huan Wang, Caiming Xiong, Ahmed A. Metwally, Jun Yan, Chen-Yu Lee, Hanqing Zeng, Yinglong Xia, Xiaokai Wei, Ali Payani, Yu Wang, Haitong Ma. Wenya Wang, Chenguang Wang, Yu Zhang, Xin Wang, Yongfeng Zhang, Jiaxuan You, Hanghang Tong, Xiao Luo, Xue Liu, Yizhou Sun, Wei Wang, Julian McAuley, James Zou, Jiawei Han, Philip S. Yu, and Kai Shu. Rethinking memory mechanisms of foundation agents in the second half: A survey. ArXiv preprint, abs/2602.06052, 2026. URL https: //arxiv..vvvv/// orggabs///el 06052.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofrr Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. /RRlhhttps:/opperreeee
net www.:n/e//ooummidiiFFFF... 8YNMM66..
Can Jin, Hongwu Peng, Qixin Zhang, Yujin Tang, Dimitris N Metaxas, and Tong Che. Two heads are better than one: Test-time scaling of multi-agent collaborative reasoning. ArXiv preprint, abs/2504.09772. 2025. URL https::/arxx...rr//b/////55
//tp///axvvvv//aa////a org/ http/abb//00400777...
Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Kinose, Koki Oguri, Felix Wick, and Yang You. Rap: Retrieval-augmented planning with contextual memory for multimodal I1m agents. ArXiv preprint, abs/2402.03610, 2024. URL
https://arxiv.ct wttpii..orrgbb////22// bttp://0ab//22220611.1
https://arkiv.orr//a///222.20.
Yi Kong, Dianxi Shi, Guoli Yang, Chenlin Huang, Xiaopeng Li, Songchang Jin, et al. Mapagent: Trajectory-construece memory-augmented planning for mobile task automation. ArXiv preprint, abs/2507.21953, 2025. URL https: //axxiv..//////////a///a org/abs/ 250..21953.
Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gustavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, et al. Gemini embedding: Generalizable embeddings from gemini. ArXiv preprint, abs/2503.0789) 2025. URL https://aarxv..rr//ab//222 wwt.:oog/bbbb//....oooo 07891.
Dacheng Li, Shiyi Cao, Chengkun Cao, Xiuyu Li, Shangyin Tan, Kurt Keutzer, Jiarong Xing, Joseph Gonzalez, and lon Stoica. S*: Test time scaling for code generation. ArXiv preprint, abs/2502. 14382, 2025a. URL https:///aree http:////aaxvv.oooaa//////////// arg/ 14382.
13

---

Published as a conference paper at ICLR 2026
Zhiyu Li. Shichao Song, Hanyu Wang, Simin Niu, Ding Chen, Jiawei Yang. Chenyang Xi, Huayi Lai, Jihao Zhao, Yezhaohui Wang, et al. Memos: An operating system for memory-augmentt generation (mag) in large language models. ArXiv preprint, abs/2505.22101, 2025b. URL https: //arxiv.org hwwlv.org//bbb hwnggabs//555 2500. 2211.22111.......
Xuechen Liang, Yangfan He. Yinghui Xia, Xinyuan Song, Jianhui Wang, Meiling Tao, Li Sun, Xinhang Yuan, Jiayi Su, Keqin Li, et al. Self-evolving agents with reflective and memory- augmented abilities. ArXiv preprint, abs/2409.00872, 2024. URL https://arxiv.orrg org/ab/// 00872.
Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, et al. Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems. ArXiv preprint, abs/2504.01990, 2025a. URL https: /ttp:s/////xvovv// org abgg//////0/..... 000.5.0......... 01990.
Yitao Liu, Chenglei Si, Karthik R Narasimhan, and Shunyu Yao. Contextual experience replay for self-improvement of language agents. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14199, Vienna, Austria, 2025b. Association for Computational Linguisties. ISBN 979-8-89176-251-00 doi: 10.18653/w1/2025 or wwwg/2222...2 acl-long.600 694/.
acl-long.694. URL
Yuxuan Liu, Hongda Sun, Wei Liu, Jian Luan, Bo Du, and Rui Yan. Mobilesteward: Integrating multiple app-oriented agents with self-evolution to automate cross-app instructions. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.I. KDD '25, pp. 883-893, New York, NY. USA, 2025c. Association for Computing Machinery. ISBN 97984007 12456.. doi: 10.1445366002443799777 URL https: //oii org/10.1145/
3690624. 3709171.
Elias Lumer, Anmol Gulati, Vamse Kumar Subbiah, Pradeep Honaganahalli Basavaraju, and James A Burke. Memtool: Optimizing short-term memory management for dynamic tool calling in Ilm agent multi-turn conversations. ArXiv preprint, abs/2507.21428, 2025. URL
wwt.://///bb//. http:://araiv.orr/aa///00r 214288 21428.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye. Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 212appr//023//ahh
10 16. 2023, 2023. URL http::/papperee nips. cc/ Cowweeettteeoooeeeeeeeeeeeeeeeee html.
Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of LLM agents. Len-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13851-13875 Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10.18653/v 1/2.633/vl/22..cllll lol- 747. URL https://acclanthologg. org.org/////// 2024. acl-long. .
Atsuyuki Miyai, Zaiying Zhao, Kazuki Egashira, Atsuki Sato, Tatsumi Sunada, Shota Onohara, Hiromasa Yamanishi, Mashiro Toyooka, Kunato Nishina, Ryoma Maeda, et al. Webchorearena: Evaluating web browsing agents on realistic tedious web tasks. ArXiy preprint, abs/2506.01952, 2025. URL //axxii otggobs/ 2ttps//556.000.22
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candes, and Tatsunori Hashimoto. s1: Simple test- time scaling. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.). Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 20275-20321, Suzhou, China, November 2025. Association for Computational
14

---

Published as a conference paper at ICLR 2026
Linguistics. ISBN 979-8-89176-332-- doi: 10. 10.:8653w1122225cmmmpmmm //aclanthology org/2025. eenns-mminnlnnnnmm 1025/.
URL https:
Kolby Nottingham, Bodhisattwa Prasad Majumder, Bhavana Dalvi Mishra, Sameer Singh, Peter Clark, and Roy Fox. Skill set optimization: reinforcing language model behavior via transferable skills. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMg,.org, 2024.
Charles Packer, Vivian Fang. Shishir. G Patil, Kevin Lin, Sarah Wooders, and Joseph_E E Gonzalez. Memgpt: Towards 1lms as operating systems. ArXiv preprint, abs/2310.08560) 2023. URL pctps://arxile wwwp/bs///////ee////eee/e//gg/gt 08560.
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. First Conference on Language Modeling, 2024.
URL https: https/////ppeeeeeereeeee Det/foormm2ddNNNP26FKKSS
Cheng Qian, Shihao Liang. Yujia Qin, Yining Ye, Xin Cong, Yankai Lin, Yesai Wu, Zhiyuan Liu, and Maosong Sun. crestigate-consooldat--cc evolution. ArXiv preprint, 0abs/2401.13999 2024. URL https:///rrxiv org/ ab2//401 13996.
A general strategy for inter-task agent self-
Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification or RL suboptimal. Forty-second International Conference on Machine Learning, 2025. URL
Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen Yih, Pang Wei Koh, and Luke Zettlemoyer. Reason]R: Training retrievers for reasoning tasks. In Second Conference on Language Modeling, 2025. URL
https::/openreeveeeenett////ee
Junhong Shen, Hao Bai, Lunjun Zhang, Yifei Zhou, Amrith Setlur, Shengbang Tong, Diego Caples, Nan Jiang, Tong Zhang, Ameet Talwalkar, et al. Thinking vs. doing: Agents that reason by scaling test-time interaction. ArXiv preprint, abs/2506.07976, 2025. URL https://arxiv.. org/ abs/2506.0790 07976.
Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https::/openneviiii eeepennrevie.... net/ Forum??d=44WAwZtGGG
Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan O Arik. Learn- by-interact: A data-centric framework for self-adaptive agents in realistic environments. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openreview.nee wwtpreewwwwweeeeeeeeeeeeeeeeeeee http://wwwww.eeeooooormddddddddd /tt///ormmmmmm33K00CW
Zeyi Sun, Ziyu Liu, Yuhang Zang, Yuhang Cao, Xiaoyi Dong, Tong Wu, Dahua Lin, and Jiaqi Wang. Seagent: Self-evolving computer agent with autonomous learning from experience. 04700.
ArXiv preprint, abs/2508.04700, 2025. URL
Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. Dynamic cheatsheet: Test-time learning with adaptive memory. ArXiv preprint, abs/2504.07952, 2025.
URL www.hhts://///tm www.aiiwiw../abbbeeeeeooeeeeeeee hwrpg/abs/2504 259.0995...
Zhen Tan, Jun Yan, I-Hung Hsu, Rujun Han, Zifeng Wang, Long Le, Yiwen Song, Yanfei Chen, Hamid Palangi, George Lee, Anand Rajan lyer, Tianlong Chen, Huan Liu, Chen- Yu Lee, and Tomas Pfister. In prospect and retrospect: Reflective memory management for long- term personalized dialogue agents. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8416 8439, Vienna, Austria, 2025. Association for Computational Linguistics. ISBN acl-long.413 URL https:t/accaateeee /aclss/thlllaalooogooooo wwtp:oo../22222222/ aww 413//
doi: 1/18855311/225..
15

---

Published as a conference paper at ICLR 2026
Xiangru Tang, Tianyu Hu, Muyang Ye, Yanjun Shao, Xunjian Yin, Siru Ouyang, Wangchunshu Zhou, Pan Lu, Zhuosheng Zhang, Yilun Zhao, Arman Cohan, and Mark Gerstein. Chemagent: Self-updating memories in large language models improves chemical reasoning. In The Thirteenth International Conference on Learning Representations 2025a. URL https: httpp:: /openrevie......e... net/forum2id-kuhhqqee
Xiangru Tang, Tianrui Qin, Tianhao Peng, Ziyang Zhou, Daniel Shao, Tingting Du, Xinming Wei, Peng Xia, Fang Wu, He Zhu, et al. Agent kb: Leveraging cross-domain experience for agentic problem solving. ArXiv preprint, abs/2507 abs/2507.06229, 2025b. URL httttt//s///axiii wwtp:/ii.ooo//oooo wrg./bs//5/... 06229.
Haozhe Wang, Qixin Xu, Che Liu, Junhong Wu, Fangzhen Lin, and Wenhu Chen. Emergent hierarchical reasoning in lims through reinforcement learning. ArXiv preprint, abs/2509.03646, 2025a. URL https: /tap::://axxv....// org/ absaaas////// 2509. 03646..
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang. Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 1866),186355, 2024. doi: 10.1007/s11700 1704-024-40231-11 URL
https://doi. org/r/////// 10. 000007/s1104-224--4000
Yu Wang, Dmitry Krotov, Yuanzhe Hu, Yifan Gao, Wangchunshu Zhou, Julian McAuley, Dan Gutfreund, Rogerio Feris, and Zexue He. M+: Extending memoryLLM with scalable long- term memory. In Forty-second International Conference on Machine Learning, 2025b. URL https://openreviee http:///wweeeeeeeeeeeoooooonoon/ dd-0ccbkROe8...
Zora Zhiruo Wang, Apurva Gandhi, Graham Neubig, and Daniel Fried. Inducing programmatic skills for agentic tasks. ArXiv preprint, abs/2504.06821, 2025c. URL https://arxxv..rrg// org/abs/ 2504. 06821.
Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. In Forty-seconl International Conference Machine Learning, 2025d. URL https://
Cheng-Kuang Wu, Zhi Rui Tam, Chieh- Yen Lin, Yun-Nung Chen, and Hung-y yi Lee. Streambench: Towards benchmarking continuous improvement of language agents. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.). Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December hatper/2224//hate
10 15, 2024, 2024a. URL http: / /papers//pppppp/e/////rse nips. cc/ paper and Benchmarks Track. html.
Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. Longmemeval: Benchmarking chat assistants long-term interactive memory. In The Thirleenth International Conference on Learning Representations, 2025. URL
pZiyCaVuti.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models. ArXiv preprint, abs/2400.000224 2024b. URL htttp:://aaae http://axxvvvoorrrrsr/asa//
org/abs//400 00724.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NewrIPS 2024, Vancouver, BC, Canada, December 15, 2024, 2024. URL http://papers.nntte nips. cc/ wwtpccpppppeeeeeeeeeeeeeeeee. /appes//pper//22/////// 5d41 N4413e44f84cc AAW.A4F8444612244e5555555 244b66455011118855415 Aast and hra.rrastttt.aadneeeeenrr.rr.... feechharkks.Trckk html.
Amir Globersons, Lester
Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-mem: Agentic memory for Ilm agents. ArXiv preprint, abs/2502. 12110, 2025. URL https: //axxiv org/abs/ 2502, 12110.
16

---

Published as a conference paper at ICLR 2026
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng 7hang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http: http:::/pppppss...pee cc/ wc/paper_sae _files/ 0ttee//aahh/5ac44475555 hetcccl133ccccccoc 328ccc5230 wwweeeeeeeeeeatrattoooooo
html.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL / wttp://www..eee..deeeee.....
Shaozhe Yin, Jinyu Guo, Kai Shuang, Xia Liu, and Ruize Ou. Learning wisdom from errors: Promoting lim's continual relation learning through exploiting error cases. ArXiv preprint, abs/2508. 12031, 2025. URL https: /arxiv wttpvvv.orr/bb/5555555 12031.
Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma. Jingjing Liu, Mingxuan Wang, et al. Memagent: Reshaping long-context Ilm with multi-conv rl-based memory agent. ArXiv preprint, abs/2507.02259, 2025a. URL https:
//arxiv.org/abs /00gg 022599.
Xiao Yu, Baolin Peng, Vineeth Vajipey. Hao Cheng, Michel Galley, Jianfeng Gao, and Zhou Yu. ExACT: Teaching Al agents to explore with reflectiveMMCTS and exploratory learning. In The Thirteenth International Conference on Learning Representations, 2025b. URL https: //
Zhaojian Yu, Yinghao Wu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. Z1: Efficient test- time scaling with code. ArXiv preprint, abs/2504.00810, 2025c. URL /ttps://arxiv..
abs/2504.00 00810.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https:t/////ot hoap::///pperevieww mett/Forumiid==0sgy077eee
Tianjun Zhang, Aman Madaan, Luyu Gao. Steven Zheng, Swaroop Mishra, Yiming Yang, Niket Tandon, and Uri Alon. In-context principle learning from mistakes. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024.
OpenReview.net, 2024. URL /ttper///pprreviiew. wwwpreewwwwwwwwweeoooeoeeeeeeeee www.pp/rwwwee...eeeeooooodr/drod httt://foommm///poAaaa/3
Xinliang Frederick Zhang. Nick Beauchamp, and Lu Wang. Prime: Large language model personalization with cognitive memory and thought processes. ArXiv preprint, abs/2507.04607, 2025a. URL https:://aaate .rgg wog./oo////////eeo abs/2507. 04607.
Zeyu Zhang, Quanyu Dai. Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model-based agents. ACM Trans. Inf. Syst., 43(6), September 2025b. ISSN 1046-8188, doi: 10.1145/3748302. URL https://doi.or org/10 000.//101111 11 1145/374300..
Andrew Zhao, Daniel Huang. Quentin Xu, Matthiu Lin, Yong-Jin Liu, and Gao Huang. Expel: LLM agents are experiential learners. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty- Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium Educational Advances Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 19632-19642. AAAI Press, 2024. doi: 10.1609/AAAL 2000//AAAALV3811.29999923
URL https:///oo: org/10. (609//aaii www.131111... 29936..
Longtao Zheng. Rundong Wang, Xinrun Wang. and Bo An. Synapse: Trajectory-as- as-exemplar prompting with memory for computer control. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
URL http:/tpps//pprrreeweee//ffrnnnn
17

---

Published as a conference paper at ICLR 2026
Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. In Michael J. Wooldridge. Jennifer G. Dy, and Sriraam Artificial Intelligence. AAAI 2024, Thirty- Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 19724-19731. AAAI Press, 2024. doi: 11609//AALLL338117...2222 URL https://doi. org/10....// aaao./aaai..m v39117.2994...
Natarajan (eds.), Thirty-Eighth AAAI Conference
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
URL https: wwwp//veew..ee.ee///ep/ee net/ forfmmmidooknn9cyytt..
Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang. Meml: Learning synergize memory and reasoning for efficient long-horizon agents. ArXiv preprint, abs/2506. 15841, 2025, URL https:://axke org/abs/2506 15841.
King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, et al. Scaling test-time compute for Ilm agents, ArXiv preprint, abs/2506. 12928, 2025. URL
12928.
18

---

Published as a conference paper at ICLR 2026
Contents of Appendix
A Experiment Details
20
A.1 Prompts Used for REASONINGBANK
20
A.2 Implementation Details
20
A.3 MATTS Details
22
Details for Experiment Settings
22
B.1 Web Browsing....
22
B.2 Software Engineering
24
C Additional Analyses
24
C.1 Number of Retrieved Experiences
24
C.2 Inference Cost Study
25
C33 Pass@k Analysis
25
C,4 Results with Smaller Open-source LLMs
25
C.5 Case Study
25
D Future Directions
26
Limitations
28
F Use LLMs
28
19

---

Published as a conference paper at ICLR 2026
A EXPERIMENT DETAILS
This section details the implementation of REASONINGBANK with agent systems mentioned in Section 4.1 for web browsing tasks including WebArena and Mind2Web. We first present all the prompts used for memory extraction in Appendix A.I, and then we provide the technical details for memory extraction, retrieval, and consolidation in Appendix A.2.
A.1 PROMPTS USED FOR REASONINGBANK
System Instruction
System Instraction the comapenncing trujectary
You are an export in web nedggttonn You will lor giver uae query, the comoggonding Unjectay
reprosutt
agent sucomslad eccemplahhed
You expart in and You si
that reprosnta how an agent uttempted revolve the bakk but fulled.
Guicelinaa
Guidalras
You nead to ecrest und summancc uaifdl traiht the format
agents sucredtul trsjectory
gual
memory y
helpful geonoslaab
iamt based the
ageat's foiled tojestery
TTo gea sumworeed memory keers to holpfil gone of cub for tu dinday taaks
genarokubble inaghta.
You mue tos shase why the mijectory is axceesdul and more oumnaatie the inagher
You octner at mort 3 memory ziees from ropemoyy
importen
ovarlapping not mamiir speaiic walactan,
sting contants, but buth foca
memsio speciicc websitee ger end Kuble naight.
moo
have leareed steetggen privent the farure in the future.
extrect momer memeyy y from mageenory.
first reflect
ovarlapping iring rtants, but
why
felled, and
what flosors
Ourput Format
en Curp Forman
Memory hoon:
monory of the memon Hame
Menory lurn
49 Description sentenas
4a Contant <1-S auranocs ceacriiin the inaigtt leatned acccuukllly ancomplakinge
en Content <1-3 semeeocs descrit the in gits isimed to succearkull excomplitiing the
Descrppt sentence ofe memory item-
Prompt
Ioput Prompt
Query sieer qury?
Quary: querys
crojemoy complesss
Figure 9: System instructions for extracting memory items from agent trajectories: the left panel targets successful trajectories (summarizing why they succeed), while the right targets failed trajectories (reflecting on failure and deriving lessons).
Memory Extraction. Figure 9 illustrates the system instructions we used to guide the extraction of memory items from agent trajectories mentioned in Section 3.2. We will first obtain correctness signals from LLM-as-a-Jugge (Gu et al., 2024) using the same backbone LLMs. When the trajectory corresponds to successful case (left panel), the instruction emphasizes analyzing why the trajectory led to success and summarizing transferable reasoning strategies. Conversely, when the trajectory represents a failed case (right panel), the instruction requires reflecting on the causes of failure and articulating lessons preventive strategies. In both settings, the output format is constrained to at most three memory items expressed in a structured Markdown format, ensuring that the resulting insights are concise, non-redundant, and generalizable across tasks rather than tied to specific websites or queries.
LLM-as-a-Judee for Correctness Signals. Figure 10 displays the instruction used for self- evaluation used to get binary signals for both successes and failures. Given the current user query, trajectory in resolving the query, final state of the website, and model output, the LLM is required to output the state of "Success" or "Failure" of whether the trajectory given successfully resolved the query or not.
A.2 IMPLEMENTATION DETAILS
Memory Extraction. We use an LLM-based extraction pipeline to convert raw trajectories into structured memory items. Specifically, we design a prompt template that asks the model to distill reasoning patterns into three components: title, description, and content previously mentioned in Appendix A. The backbone LLM of the extractor is set to the same as the agent system with temperature 1.0. For each trajectory, most 3 memory items could be extracted. Crucially,
20

---

Published as a conference paper at ICLR 2026
System Instruction
You evalugting the performance a web ravigation agent. The agent designed to help human navigate website complete task. Given dacda whether the agent's execution is successful or not.
intent, the agent's action history, the state the webpage. agent's response to the user, your goa
are three types
obtain certain information from the wiepage, such information product, comparioon map routes, etc: The bot's response must contain the information the user wants, or explictly state that the information not wosilable. Otherwse, agg the bot encounters is an escept on and respond with the error content, the task is considered a feilure Besides, be careful about the sufficieccy of agent's actions. For example, when asked to the top searched iterss shop, the agent should order the items likely to
Information seoking: The
map
number searches,
return top
ordering
missing, the
2. Site navigation: The
navigate speciiic page. Carefully examine the bot's action history and the final state the wabpage determine whether the bot successfally completes the task.
to considar bot's response.
3. Content mod The stat the webpage to determine whether bot sucoessfully ly completes the task, No need consider the bot's respons..
modify the content a webpage configuration. Canefully examine the bot's action history and the final
"MPORTANT*
Format your response lines shown be Thoughts: dyour thoughts and ressoning processs"
Status: "success" "faluree
Input Prompt
User Intent: (nttet)
Trajectory: (rraectory)
The detailed final state of the webpage: "'nd (tap) "M
Bot response user: (response respornse else "N/A
Figure 10: System instructions for obtaining binary signals indicating success or failures of the current trajectory.
System Instruction
You are an export web novigation. You will be given user query and multiple trajectories showing how an agent attempted the task. Some vajectories may be successfu, and others may have falled.
Firs-time Instruction
Guidelines
and contrast these Use self-cante cel-contrast reasoning:
Your
denofy the useful and gonerclizable
importent: Let carefully reveramine protin trojectry. including your reasnnigg stepe and actions
Identify patterns and ctringiee that consieently only led nucresx.
Pay special attention whather you used the cornect elements on the page, and whather your nepponte aeddressss the usar query If you find inconsistencive, correct them. everything acems comeet, confirm your final answer.
identify mistbbe infficienciee from feild mejectories and formulate precentaaive
Prafer that za beyond apac fie wext wording.
Output must in the same <thinio...s/iiiiieee.. watiiioooooooiiiooo actions" format as previous tra tegectorics.
You Why memory
while lalled?
trajectorios
lapping
mention spocifi
generaliable behaviors
reaeoning natheme
Make wach iten ceptures actionable and tranaferable imegite
Follow-u Check Instruction
Your strictly Markdiaen
Lot's chock again.
<the memory the Contont
Output must raay in the same "athinks. -cmiinicssatiiosss actione" format as provious trajctoriis.
the lcarned successfully sccompishing the tasko
Input Prompt
Ouwry: queryts
Trajectories:
cirsjectoyy Iilnsrrjeeoory cirajecctryytttorr
Figure 11: System instructions for memory-awan test-time scaling: the left panel shows parallel scaling (comparing multiple trajectories to extract generalizable insights), while the right panel shows sequential scaling (iteratively re-checking trajectory to refine the final answer).
induce items from both successful and failed trajectories, Successes provide validated strategies, while failures supply counterfactual pitfalls that act as negative signals. To determine success or failure, we adopt an LLM-based binary classifier following (Pan et al., 2024; Wang et al., 2025d). The classifier is prompted with the trajectory and the given user query, and asked to output categorical judgment (Success or Failure) as shown in Figure 10. Similarly, the backbone of the classifier is set to the same the agent system, with decoding temperature setting to 0.0 for determinism.
Memory Retrieval and Response Generation. For retrieval, we embed each task query using (Lee al., 2025), accessed via Vertex AL. Similarity search is conducted
htttps: htttt:://ai..gogle.dee wwwggee.de//gemiiiiii httpa/aapppppdddcd/emeiiooooo
21

---

Published as a conference paper at ICLR 2026
the memory pool using cosine distance. We select memory items of the top-k most similar experiences (default 1; ablation study in $5). The retrieved items are concatenated into the agent's 's system prompt with simple formatting template (each item represented by its title and content) and instruction:
Below are some memory items that I accumulated from past interaction from the environment that may be helpful to solve the task. You can use it when you feel it's relevant. In each step, please first explicitly discuss you want to use each memory item or not, and then take action.
Memory Consolidation. After finishing each new query, the trajectory is processed by the extraction pipeline to produce memory items, which are appended into the memory pool. We adopt a minimal consolidation strategy: newly generated items are directly added without additional pruning. This choice highlights the contribution of REASONINGBANK itself without introducing confounding factors from complex consolidation algorithms. Nevertheless, more advanced consolidation mechanisms (e.g., merging, forgetting) be incorporated in future work.
REASONINGBANK Storage We maintain REASONINGBANK in JSON format, and each entry of REASSNNNGGGAN consists of a task query, the original trajectory, and the corresponding memory items. All memory items are stored with the schema (title, description content). The embedding is pre-computed for each given query and stored in another JSON file for efficient similarity search. We persist the memory pool for each independent run, enabling continual accumulation of experiences throughout test-time learning.
A.3 MATTS DETAILS
Prompt Used for MATTS Figure 11 illustrates the system instructions used in our MATTS framework mentioned in Section 3.3. In the parallel scaling setting (left), multiple trajectories for the same both successful and failed-are provided, and the model is instructed to perform the LLM to act
self-contrast reasoning. Instead of relying the model is guided to directly compare and contrast trajectories, identifying patterns that lead to success and mistakes that cause failure. This provides contrastive signal that grounds the memory extraction process in observable differences between outcomes, yielding more reliable and transferable insights. In the sequential scaling setting (right), the model repeatedly re-examines its own trajectory with check instructions, ensuring consistency and correction over iterations without appealing to external judgment.
external judge of quality,
Best-of-N Calculation Details. Given the task query and N trajectories from the agent system, we leverage an LLM and selects the best answer from the trajectories. The LLM is initiated as then
the same backbone LLM as the agent system (e.g., if the agent system uses Gemini-2.5-flass the model also uses Gemini-2.5 2.5-fii-.5f. --fassh). We feed all the trajectories to the model at once and use a carefully curated prompt shown in Figure 12, asking the model to select the best answer.
B DETAILS FOR EXPERIMENT SETTINGS
B. WEB BROWSING
In this section, we detail the experiment settings used for web browsing agents mentioned in Section 4.1.
Datasets. We test REASONINGBANK three agentic datasets for benchmarking web browsing and coding agents. Specifically, we conduct experiments on WebArena (Zhou et al., 2024) which features general web navigation across diverse domains, spaning shopping, administration. coding (Gitlab), and forums (Reddit). Another benchmark we used is Mind2Web (Deng et al., 2023), which provides playground test the generalization of agents versatile operations and environments, including cross-task, cross-website, and cross-domain settings. There are 684 and 1341 test instances in total for WebArena and Mind2Web, respectively. For WebArena, the number of
22

---

Published as a conference paper at ICLR 2026
System Instruction
You are an expert evalusting web navigation agen: trajectories, You will be g ven the user query, and [N] cancidate trajectoriss each reprasenting sequence of steps for solving the same task. Your job is to select the single best trajectory that most offective y and efficendly solves the task, and explain your reasoning.
ee Input Format:
Fach trajectory consists muilippl steps. For step, you be provided: trajactory.
step_rum: Step
action output: The action the egent takes (click, type, scroll, etc). think output The agents reasoning or plan before taking the action.
e# Evaluation Crizeria:
498 Progress Toward Goal 1. How well the trajectory advances toward completing the user's task. 2. Reward tangible, meaningful progress; penslize minimal or no advancement. 3. Consider both individual step contributions and overal progress.
#et Tejectory Efficiency 1. How efficiently the trajectory achieves progress given the number and complesity of steps. 2. Reward sigrificant progress in fewer steps. 3. Favor better value tu--oo-depth ratios. 4, Reward efficient search space exploration.
Repeating dertical observations and actions with no added vel Benign Repettions: Siiht variations that sill yield new information. J. Penalize real loops heavily, penalize benign repetitions only if they waste effort.
een Loop Detection: idertffy loops or redundant actios. 1. Real Loops:
een Enor Severity and Stabiiity Acsess saverity of arrors: 1. Fata//Blockngg Major panalty 2. Sign ficant Moderate pansity. 3. Minar/aacoverable Minor panalty 4. Penalize unstable ls or incoherent model reasoning. S. Consider whether errors prevent goal complesion.
esa Overal Tejectory Quality 1. ogiiall flow of steps, clarity of strategy, and ccherence. 2. Ralanced exploration vs, exploitation 3. Closeness to final
goal. 4, Reward consistant progress and coherant planning.
eP Cusput Format:
Retur the evaluation as ISON object: "" { "index" : |beat trajectory index] "anolysia": "Deaaled reasoring explaining why this trajectiry is the beat,
referencing progress, efficiency, loop detection, enror severity, and overall quality"
Input Prompt
Query: (quarry)
Trajectory 1: (rajectory V//Trajectory 2: (unjectory Setictorryy2y....... 2ln......TTTjjeeee InTrajectory N:
(trejcttory.N N;
Figure 12: System instructions for obtaining the best answer from N candidate trajectories for BoN calculation.
instances for different domains are Shopping (187), Admin (182), Gitlab (180), Reddit (106), and Multi (29). For Mind2Web, the number of different settings are Cross-Task (252), Cross-Website (177), and Cross-Domain (912).
Baselines. We compare REASONINGBANK against several representative
approaches: (i) Vanilla, the backbone LLM agent without any memory module, serving as reference point; (ii) Synapse (Zheng et al., 2024), a representative work that organizes past trajectories in-context memory, and (iii) AWM (Wang et al., 2025d), which further abstracts common patterns from trajectories into reusable workflows. Together, these baselines span a progression from agents without memory, those that directly reuse past trajectories, and finally to methods that distill higher-level structures, providing a comprehensive comparison for evaluating REASONINGBANK. To ensure fair comparison, the baselines are implemented with the same "Memory Retrieval" and "Memory Consolidation" mechanisms. The only difference is about "Memory Extraction", which is exactly how REASONINGBANK different from baselines in terms of memory formulations.
Implementation Details. We build our agents upon several state-of-the-art LLMs accessed via the Vertex AI A including Gemini-2.5-Flash. Gemini-2.5-Par (Comanici et al., 2025), and Claude- 3.7-Sonnet (Anthropic, 2025). These choices allow us to investigate both cross-family (Gemini, Claude) and intra-family (Flash, Pro) variations. BrowserGym (de Chezelles et al., 2025) is used as the execution environment for WebArena, where we set a maximum step limit of 30 per query. The agent is implemented in ReAct (Yao et al., 2023) style, and iterates until the model predicts the stop action or reaches a task termination condition. We use the decoding temperature of 0.7 for model generations for both WebArena and Mind2Weh.
Evaluation Metrics. For WebArena benchmark, we evaluate all methods across two key dimensions: effectiveness and efficiency. For effectiveness, we report the success rate (SR). A task is marked as "successful" only if the agent's final output or state precisely matches the pre- defined ground-truth goal, which is measured by the evaluation protocol from the corresponding benchmarks. SR is the total number of successful tasks divided by the total number of tasks evaluated, formally, it is calculated SR _े Σ isSuccess(a)). where isSuccess(g)) is the binary function that returns 1 if task qi is successful and otherwise. For efficiency, we measure
'https://clon wowgle.com/vele..eoo
23

---

Published as a conference paper at ICLR 2026
the average number of steps (AS) taken by the agent to complete each query, which reflects the computational and interaction cost incurred during task completion. A single step is defined as one complete agent-env interaction cycle following the ReAct loop, which typically involves observing the current state, generating thought, and a subsequent action. AS is calculated as the total number of steps taken in the trajectory when solving task qi divided by the total number of tasks, specifically, AS N Di=1 St)ps(q.). For Mind2Web dataset, each task in has a predefined fixed number of steps; at each step, the agent needs to predict an action, which is evaluated by: elemen! accuracy: to check if the correct page element selected, action F/ to check if the action taken on the element correct. Aggregating element accuracy and action Fl yields step success rate which checks that both element and action selection are correct at the current step. Lastly, after completing every stop in the given task, the last metric task-level success rate measures if all intermediate steps are successfully conducted for this task, i.e., all steps for this task score 1.0 under metric step success rate.
B.2 SOFTWARE ENGINEERING
Dataset. To benchmark agentic coding tasks, we evaluate on SWE-Bench-Verine (Jimenez al., 2024), repository-level issue resolution benchmark. The dataset consists of 500 high-quality test instances that have been manually verified. Each instance requires generating patch to address the underlying bug described in the input issue. The objective is to modify the relevant portions of the codebase such that all provided test scripts execute successfully.
Metrics. We report the issue resolution rate on SWE-Bench- Verified as the primary evaluation metric. The resolution rate measures the percentage of issues successfully fixed across all data points, where an issue is deemed resolved if the submitted patch passes all test scripts. To evaluate the patch application rate, we attempt to apply the generated patches to the repository using the standard patch program, counting only successful applications. Our implementation follows the official evaluation scripts.? For efficiency, we additionally report the average number of steps performed by the agent per instance, following web- web-browsing tasks.
Implementation. We implement REASONINGBANK for SWE-Bench following the setting of mini- SWE-Agent (Yang et al., 2024), which enforces the Bash-Only environment with tools and no special scaffold structure. assumes a simple ReAct agent loop (Yao et al., 2023). Similar to previous experiments, compare REASONINGBANK against (i) No memory and (ii) Synapse."
C ADDITIONAL ANALYSES
C.I NUMBER OF RETRIEVED EXPERIENCES
We conduct another ablation study on different number of retrieved experiences using Gemini-2.5-ftan on WebArena- Shopping subset. As shown in Figure 13, we found that incorporating relevant memory significantly boosts performance (from 39.0 without memory to 49.7 with one experience). However, as the number of experiences increases, the success gradually declines (46.0 with 2, 45.5 with 3, and 44.4 with 4). This suggests that while memory provides valuable guidance, excessive experiences may introduce conflicts or noise. Hence, the relevance and quality of memory are more crucial than sheer quantity for effective performance.
50
49.7
Success Rate
47
45.5
0
3
Number experiences
Figure 13: Ablation results for us- ing various number of experiences.
Phttps://wwe /www. wwtp:wwwwwbenhh.com wwaccch..ccm//ee Beom//WE-beenhh/ /harness//
'We exclude AWM here because the action space in Mini-SWE-Ase AWenttentt is open-ended (arbitrary Bash commands), making it difficult to extract the common routines fixed workflows that AWM requires for cross-task generalization.
24

---

Published as a conference paper at ICLR 2026
C.2 INFERENCE COST STUDY
In this section, provide comprehensive view on inference cost for REASONINGBANK and baselines to facilitate real-world deployment. We report break- down of the averaged total token consumption for each trajectory in addition to number of interaction steps mentioned in Section 4.1. The results are shown in Table 5.
Table 5: Breakdown results of total token consumption required for each task.
Action Generation
LLM-as -a-Judge
Memory Extraction
Methods
Total
No Memory
50847.4
50847.4
Synapse
55920.5
2594.2
58514.7
AWM
53819.6
2479.1
3074.1
59372.8
REASONINGBANK
49306.1
2186.3
1562.1
53054.5
From the table we can see that compared with "No memory", while the total token consumption is increased only by around 4.3%, the overall performance is boosted by 20.5%. Other memory baselines such as Synapse and AWM will greatly increase the computation overhead while achieving less performance gains compared with REASONINGBANK, demonstrating the cost-effectiveness of REASONINGBANK.
C.3 PASS@K K ANALYSIS
Memory-aware scaling improves sample efficiency and sustains stronger performance gains. Pass @ analysis under parallel scaling on WebArena-Shopping subset with Gemini-2.5-flash (Figure 14) reveals two distinct effects. First, MATTS w/o aggregation (Vanilla TTS) already makes test- time learning behave similarly to RL training: instead of inflating @ at large k; it improves sample efficiency by guiding exploration. For example,
MaTTS aggregation
MaTTS w/o memory
MaTTS
72
62.1
63
58.8
Pass@k
54.5
49.7
51.3
2, MATTS w/o
0
50.0
52.9
51.3
52.4
aggregation achieves 50.8 compared to 47.6 from MATTS w/o memory, extracting more value from each rollout as noted in (Yue et al., 2025). Second, equipping TTS with memory- aware scaling pushes performance further. MATTS not only preserves efficiency sustains strong growth with scaling, reaching 62.1 at 5, compared to only 52.4 for MATTS w/o memory. Overall, see that MATTS unlocks more potential of agent systems and encourages diversified generation for better pass @ performance.
45
49.7
47.6
49.7
39.0
small & (51.3 at k 2) but also
Scaling factork k
Figure 14: Pass@k under parallel scaling with REASONINGBANK.
C.4 RESULTS WITH SMALLER OPEN-SOURCE LLMS
To evaluate the generalizability of REAsONINGBANK across different model scales and architectures, we extend our evaluation to smaller, publicly available open-source models. Specifically, conduct experiments using Deemma-3-12B-nnstraed subset. As summarized Table 6, REASONINGBANK continues to achieve consistent performance gains even when deployed on more compact models. While the baseline success rate without memory is 17.1%, REASONINGBANK improves this to 24.1%, outperforming other memory- based methods such as Synapse (16.0%) and AWM (21.4%). Furthermore, REASONINGBANK demonstrates superior efficiency by requiring the fewest average interaction steps (1 1.8 steps) compared to all baselines. These results suggest that the benefits of our memory induction approach are not limited to proprietary frontier models but effectively scale down to smaller open-source LLMs.
the WebArena-Shopping
C.5 CASE STUDY
To better illustrate the benefits of our approach, we present three representative case studies.
hhttps://hugginggacee wttp://o/oooooo///ee/oo/occeeeeo 1arpgglleggmma22200
12b-it
25

---

Published as a conference paper at ICLR 2026
Table 6: Performance on WebArena-Shopping using Gemma-3-12B-
No memory
Synapse
AWM
REASONINGBANK
Success Rate (%) Average Steps
17.1
16.0
21.4
24.1
13.7
14.0
12.5
11.8
What is the date when made my first purchase on this site?
Baseline (No memory)
Answer (YYour first purchose this made 3/11/3.))
information regarding cortain purchase, use memory item see link 'My Orders as mentioned the memory.
Stort on homepage
dlic ('y Account)
View Rocont Orders table
Reasoning Bank
Answar ('Yurr first purchase on made March 2022))
Start on homoopage
click (My Account)
My table
Page
Figure 15: REASONINGBANE enables the agent recall and apply past reasoning hints, guiding to the full order history and yielding the correct first purchase date, unlike the baseline that fails with only recent orders.
Effectiveness. Figure 15 highlights the effectiveness REASONINGBANK in leveraging related previous experiences memory items. While the baseline agent (without memory) only checks the "Recent Orders" table and mistakenly outputs the most recent purchase date, REASONING recalls from past reasoning hints to explore the full purchase history and correctly identifies the carliest order.
Efficiency. Figure 16 demonstrates the efficiency gains. a navigation- heavy shopping task, the baseline requires 29 steps due to repeated inefficient browsing- stucks and struggles to find the correct place filter for "Men". In contrast, REASONING BANK leverages stored reasoning about category filtering, enabling the agent to directly reach the relevant items and complete the task in only 10 steps.
Reflection on Successes and Failures. Figure 17 shows how memory items induced by REASONINGBANK through reflecting past trajectories helps to prevent similar errors from happening again, which enables emergent improvement. In this case, the original trajectory actually fails because of the imprecise search query that leads numerous returned items, and irrelevant is able to first reflect on the trajectory, pinpoint the key reason of failure, and extract valuable strategies that would avoid similar errors such search query optimization and using the functionality filters.
objects. REASONING
D FUTURE DIRECTIONS
In this section, we briefly discuss the potential future directions following REASONINGBANK and MATTS.
Modular and Compositional Memory. Our current framework distills each experience into multiple memory items, and when query arrives, retrieve similar experiences and reuse all associated items independently. This design highlights the effect of memory content but does not consider how items could be composed into higher-level strategies. Future work could
26

---

Published as a conference paper at ICLR 2026
Buy the best rating product from "Men's shoe" category with at least reviews and the product is least expensive
Baseline (No memory)
Inn wwwmmiitttateet for naviglitien Lec.aauiiing
Iwil click "Proceed Checkout
stans
29 Staps in total.
homepagee
Cennot fiter by Men'. need scroil down
Select Price
Reasoning Bank
wwll cloc "Poosed Checkout" the
10 Steps in total.
Start on homepage
Hover (Shoes), Hover (Men)
Select Price
identified with at least 5 reviews
According Memory Item 11, correctly navigate certain categories,
products....
Figure 16: REASONINGBANK improves efficiency by leveraging past reasoning hints, reducing the navigation from 29 steps to 10 steps compared to the baseline without memory.
User Query: Provide me with the complete names of Bluctoath headphones from Sony, and also share the prize range for the available models.
Search results Bluetooch headphones Sany
Diagnose the trajectory:
"The agent spent endless steps on 'page which exhausts the interaction limits without providing the requested information from
Too many browse
Distilled strategies Reasoning
"Search query optimization" ization" to avoid inelevance and ensure preciseness "Adjust number of itemss displayed per page" to accelerate
and
IIrelevant objects
"Use filters available"
Figure 17: Memory items induced by REASONINGBANK unlocks emergent improvement of REASONINGBANK, which help avoid similar errors from happening again.
explore composition-awee aware retrieval and consolidation with based on modular memory extraction, enabling the agent combine complementary items form reusable macros, thereby yielding richer strategies and stronger generalization in long-horizon tasks. For example, memory could be extracted with respect to "planning memory", "tool-use memory", "'pperatioaal memory", "user- centric memory", etc. In this way, memory extracted would be more fine-grained and memory retrieval could unlock compositional and complementary power, not just task similarity.
Advanced Memory Architectures. Our current system design is intentionally minimal; a natural next step is to build a layered, product-level memory stack that integrates mature paradigms e.g., episodic traces (Fountas et al., 2025) for per-task context, short-term "working" memory (Lumer al., 2025) for within-session state, and long-term (Wang et al., 2025b) consolidated knowledge with decay/refresh policies. The philosophy of REASONINGBANK are compatible with the above different memory angularities. Additionally, the current memory retrieval could also move beyond embedding- based similarities to reasoning-intensive controllers (Shao et al., 2025) that decompose queries, plan multi-hop lookups across tiers, and condition selection on uncertainty, recency, and cost. Learning- based routers and consolidation policies could also automate this process. This integration would turn REASONINGBANK with MATTS into a deployable memory service that scales across domains and teams.
27

---

Published as a conference paper at ICLR 2026
E LIMITATIONS
While REASONINGBANK demonstrates strong empirical performance and introduces practical paradigm for memory a scaling dimension, also comes with several limitations that suggest directions for future research.
Focus on memory content. Our study emphasizes how curate and utilize memory content (e.g.- integrating failure trajectories, constructing distilled reasoning cues). For this reason, we did not extensively compare with other memory architectures such as episodic or hierarchical memory. These designs address orthogonal concerns (memory form/structure), while our contribution targets what should be stored and reused. Exploring their combination would be an interesting future direction.
Simplicity in memory retrieval and consolidation. We intentionally adopt simple embedding- based retrieval and straightforward consolidation to better isolate the effect of content quality. More sophisticated strategies (e.g., adaptive retrieval, hierarchical consolidation) are compatible with our framework and could further enhance performance, but are not the focus of this work. This choice ensures that the observed gains can be attributed directly to the design of reasoning-orionn memory content.
Dependence on LLM-as-a-judge for correctness signals. our implementation, success and failure signals for trajectories are determined by LLM-as-a-judge. While this automatic labeling enables scalable evaluation without ground-truth feedback, it may introduce noise when tasks ambiguous when the judge model itself remains robust under such noise, future work could incorporate stronger verifiers, human-in-the- loop feedback, ensemble judgment enhance the reliability of memory induction.
While results suggest the framework
F
USE OF LLMS
We used LLMs as a general-purpose writing assist tool during the preparation of this submission. Specifically, LLMs were employed for polishing the clarity and readability of text (e.g., refining sentence structure, improving grammar, and shortening overly verbose phrasing). All research ideas, methodology design, experiments, analyses, and final writing decisions were conceived, implemented, and validated solely by the authors.
28


Reviews:
[{"review_id": "VnqBm9DgWU", "reviewer": "Reviewer_u2my", "summary": "The paper proposes ReasoningBank, a structured and interpretable memory for continual test-time learning in LLM agents that distills reusable strategies from both successful and failed trajectories. It further introduces memory-aware test-time scaling (MATS), combining parallel and sequential exploration to generate richer, more generalizable memory entries. On WebArena, Mind2Web, and SWE-Bench-Verified, the approach delivers consistent performance and efficiency gains over memory and non-memory baselines, with analyses on memory evolution, the value of failures, and compute efficiency.", "questions": "- **Self-Judgment Robustness.** How consistent are the LLM’s success/failure self-labels used for induction? Please analyze misjudged episodes and quantify how such errors propagate to memory quality and downstream performance.  \n\n- **Scalability**: Could you provide more detail or share experiments on how ReasoningBank scales as the number of memory items grows—e.g., in lifelong deployment exceeding thousands or millions of tasks?\n\n- **Failure Memory Impact.** Have you observed cases where failure-derived entries bias or degrade behavior (e.g., overfitting to pathological patterns)? What safeguards (filters, calibration, de-duplication) ensure only constructive failures persist?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Clear Problem Motivation and Context.** The work identifies that existing memory modules limited learn from failures and disproportionately weight successes, and it clearly argues why this imbalance undermines long-term agent adaptability.\n\n- **Experimental Breadth and Depth.** Evaluation covers diverse agentic tasks (WebArena, Mind2Web, SWE-Bench), several LLM backbones, and robust memory baselines, assessed via effectiveness, step efficiency, and transfer/generalization. Performance comparisons are clear in Tables 1 and 3; Table 4 provides a focused breakdown of success vs. failure step efficiency.\n\n- **Concrete Technical Design.** This method organizes self-verified trajectory analyses into a structured memory of compact, modular strategies, improving both interpretability and reusability.", "weaknesses": "- **Ambiguity in Self-Judgment/Verification.** Because trajectory labels rely on the model’s own self-assessment, robustness is uncertain: calibration, error propagation, and blind-spot risks are not quantitatively evaluated. Targeted ablations (e.g., noisy/adversarial self-judgments, third-party or rule-based verifiers) would clarify how sensitive the curation process is. \n\n\n- **Scalability and Long-Term Lifelong Learning**: Although the paper claims to address \"lifelong agents,\" the experiments are limited to several hundred tasks, and do not simulate truly open-ended deployment or demonstrate the maintenance/pruning of memory over extended horizons. No experiments address issues like memory interference, long-tail drift, or resource constraints on memory expansion.\n\n- **Missing Failure/Robustness/Noise Analysis.** Although Figure 7 illustrates handling of failure trajectories, the work lacks a deeper study of negative cases, when failure-derived memories degrade behavior, and offers limited analysis of noise, redundancy, and their accumulation over time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EPaKy4Y1cj", "reviewer": "Reviewer_RD5S", "summary": "The authors introduce two key enhancements to existing memory systems. First, they design a memory framework that performs knowledge distillation not only from correct trajectories but also from potentially incorrect ones. Second, during test-time scaling, they propose distilling knowledge from multiple parallel or refinement trajectories (denoted as k trajectories). Their approach is empirically validated through extensive experiments on web navigation and software engineering (SWE) problem-solving benchmarks, demonstrating superior performance over baseline methods. Additionally, ablation studies highlight the contribution of learning from \"incorrect\" traces.", "questions": "Are there known cases where the LLM judge misclassifies correct versus incorrect traces? If so, what strategies or mechanisms could mitigate such misclassifications?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper presents a memory system that leverages all available logs (both correct and incorrect) and thoughtfully explores its integration with test-time scaling methods.\n2. The authors conduct a comprehensive set of experiments and analyses to substantiate their claims.\n3. The paper is clearly written and easy to follow.", "weaknesses": "1. The proposed memory system may incur high costs since it relies on multiple LLM calls per task trajectory to derive the memory item. Reporting the cost of each method, including memory updates, would help justify the method’s benefits.\n2. Because memory updates depend on several LLM calls, the system risks compounding errors—for instance, if the LLM judge incorrectly classifies trajectories or if knowledge distilled from different trajectories fails to generalize.\n3. The experiments are limited to the Gemini-2.5 family of models; testing on smaller, publicly available models could strengthen the generalizability of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h16CPI3ZnF", "reviewer": "Reviewer_WYBc", "summary": "The paper presents a memory framework for agents called REASONINGBANK and a complementary memory-aware test-time scaling (MATTS) technique. Unlike previous agent memory frameworks, REASONINGBANK (a) extracts generalized learnings (so-called memory items) from previous reasoning trajectories (i.e., attempts at solving a problem) instead of simply memorizing the entire raw trajectories and (b) learns by comparing successful and unsuccessful trajectories instead of only learning from the successful ones. MATTS complements REASONINGBANK by generating multiple reasoning trajectories, either in sequence or in parallel and extracting memory items from across all trajectories. The authors empirically demonstrate that REASONINGBANK substantially outperforms other agent memory frameworks and a baseline without memory in both task-specific success rate as well as efficiency (measured by the number of reasoning steps). They further show that REASONINGBANK and MATTS work particularly well in combination.", "questions": "Based on the mentioned weaknesses, the following questions can be raised to the authors:\n1. How can memory items gradually evolve? What is the underlying mechanism?\n2. How does “MATTS w/o aggregation” work? A short explanation may be useful for readers unfamiliar with vanilla TTS.\n3. How large are the confidence intervals of the results, especially those results that are only presented for a subset of the LLMs and datasets and how does that affect their reliability and generalizability?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality: While the key ideas (e.g., generalizing learnings, incorporating learnings into prompts, multiple reasoning trajectories, etc.) behind REASONINGBANK and MATTS are not entirely novel and have also been used in similar papers in the last 2 years, the exact way how these techniques are put together and implemented is sufficiently original and this apparently leads to a substantial outperformance of existing approaches.\n\nQuality: The implementation details (e.g., prompts) presented in the main part and the appendix appear very well-crafted. The chosen evaluation experiments are rigorous and suitable to test various aspects of the presented methods.\n\nClarity: The vast majority of the paper is very clearly presented in both text and visuals. The paper offers easily accessible explanations to all parts of the framework and makes suitable references to the appendix which includes an even more comprehensive documentation. Key ideas (e.g., learning from successful and failed experiences alike) are being repeated in several places, because they are so important.\n\nSignificance: The paper presents a well-performing solution to a timely and relevant problem. It further makes extensive references to recent related work and similar techniques. The advantage of the presented methods over their baselines is so substantial and consistent that it can hardly be ignored.", "weaknesses": "In some areas the paper could have been more precise or elaborate with some explanations, e.g.,\n- The REASONINGBANK framework includes a memory consolidation step. While the name suggests that memories are actually being consolidated (i.e., transforming a set of memories into a somewhat more information-dense, better structured, or simply smaller set of memories), the used implementation does nothing of that but simply appends new memory items to the set of existing ones. This appears like a waste of potential. However, the authors offer a justification for their choice in the appendix (lines 927ff.) arguing that advanced consolidation mechanisms could have led to unwanted confounding factors, and leaving such mechanisms for future work. For their paper, the authors may want to rename “memory consolidation” into something that doesn’t use the word consolidation (which suggests that something more complex is going on) and/or making clearer why they decided to keep their implementation simple in the main part, not just the appendix.\n- Line 260 mentions the two main performance metrics “success rate” and “average steps”. While the readers may already have a good intuition about these metrics, it could have been even more clear if the authors had given a more precise definition of how the success rate is calculated and what constitutes a single step.\n- The concept of “rollouts” is mentioned in line 81 but not further explained. It is not immediately clear what rollouts are and how they relate to other parts of the method.\n\nSome aspects of the framework remain somewhat unclear to me, for instance:\n- Lines 428f. mention that memory items can gradually evolve over time. It is not clear to me how this is possible when no actual consolidation of memory items is happening but when new memory items are simply added. Perhaps the authors mean that memory items evolve into new and somewhat more advanced memory items? The sentence, however, would suggest that REASONINGBANK includes a mechanism that alters the same existing memory items. If so, this should be better explained.\n- Line 341 presents “MATTS w/o aggregation” as one of the methods tested in one of the experiments. At this point, it was not entirely clear to me, what the aggregation is and how it works. Perhaps the authors could make this more explicit in some of the previous explanations.\n\nThe paper makes some claims that are not entirely credible and could benefit from additional evidence:\n- Lines 294ff. make the claim that REASONINGBANK employs a memory extraction strategy that is superior to those of other frameworks. Unlike most other claims in the paper, this claim seems not entirely backed by evidence, as there are no more detailed experiments capable of exactly attributing REASONINGBANK’s outperformance to the memory extraction component. While this appears likely, it could have been other factors such as the memory retrieval, the memory consolidation, or the way that memory items are structured as well.\n- The authors present several experiments exploring various aspects of REASONINGBANK and MATTS. The numbers generally look very favourable for their methods, yet some of the experiments are only performed on subsets (e.g., subset of the tested LLMs, subset of the datasets). Not performing all experiments on the full set of all possible combinations is understandable due to budget limitations, however, it is unclear if these restrictions somehow affect the reliability and generalizability of the results. Perhaps, the authors could add at least some error bars or confidence markers to their results to provide an indication of the degree of noise and fluctuation of their results and hence their reliability.\n\nThere are also minor issues with the presentation and cosmetics:\n- Section 3.1 presents a mathematical problem formalization. I see that the authors likely added this, because some reviewers expect a mathematical formalization in every paper. While the formalization is clear and useful here, it is not used anywhere else in the paper and thus appears as it was simply added pro forma.\n- The paper includes a small number of typos, e.g., lines 233 and 238 appear to have been changed and now include incorrect phrases “we generates” and “we refines”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DfgfuMZ5su", "reviewer": "Reviewer_sUDi", "summary": "This work addresses a key limitation of large language model agents: their inability to learn from accumulated experience in ongoing tasks like web browsing and software engineering. The authors propose two innovations. REASONINGBANK is a structured memory that extracts generalizable reasoning strategies from both successes and failures, operating in a loop of retrieval, construction, and integration. MATTS combines REASONINGBANK with test-time expansion: memory guides exploration while diverse expansions enrich memory and improve performance. The main contributions are a memory framework that leverages failures for generalizable reasoning, a synergy with test-time expansion, and empirical validation of memory-driven agent self-improvement.", "questions": "1. Has any subset of trajectories been manually labeled to verify the consistency between LLM Critic judgments and human labels? If there is a discrepancy (e.g., LLM misclassifies 15% of failed trajectories as successful), how would this affect the quality of REASONINGBANK memories and downstream agent performance?\n2. How does REASONINGBANK perform when the memory bank grows large (e.g., over 500 entries)? Could pruning or merging strategies, such as removing low-similarity memories relative to the current query, maintain or improve performance? Which metrics are used to evaluate the effect of such optimizations?\n3. In sequential expansion, how is the termination condition determined (e.g., fixed k=5 steps versus stopping when no new insights emerge)? Do intermediate optimization notes (e.g., “recheck filtering labels”) provide unique value to memory, or are they largely redundant compared with final trajectory insights?\n4. Can memory entries from web browsing tasks transfer to software engineering tasks (e.g., using a “web form data consistency check” strategy for code debugging)? If not, how can the framework be adapted to support cross-task memory transfer?\n5. Are failures categorized by type (e.g., “operational errors” versus “reasoning errors,” such as clicking the wrong button versus misinterpreting a user query), and are separate insights extracted for each type? Would such granularity improve the practical usefulness of the memory?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The problem definition is novel, focusing on extracting reasoning from experience rather than mere experience storage, and highlighting the value of failed experiences.\n2. The experiments are extensive, with diverse benchmarks and thorough ablations, providing strong evidence for the effectiveness of REASONINGBANK.", "weaknesses": "1. The paper evaluates generalization only within subsets of WebArena and Mind2Web. The claimed generalization gains may simply reflect similarities between tasks within the same benchmark, as human or LLM patterns are often consistent internally. Stronger evidence would require cross-benchmark tests, such as using WebArena Shopping as an experience pool to evaluate completely unrelated Mind2Web tasks, or testing SWE-Bench with WebArena experiences.\n2. Success and failure of trajectories are judged by the LLM, but the accuracy of these judgments is not analyzed. Using the LLM as a critic may propagate errors.\n3. Experiments focus only on historical task-level memory, without considering temporal or user-level memory, which are more valuable in practice. Task-level memory yields only modest metric gains that could also be achieved with additional SFT data or RL, but at the cost of efficiency, storage, and increased framework complexity. Temporal or user-level memory cannot be solved as easily and would provide more significant practical value, yet these aspects are not studied.\n4. The analysis of emergent behavior is limited to changes in the memory bank. It does not examine whether prior failed tasks can be corrected through reflection on successes and failures, which would demonstrate true emergent capability. Currently, the memory module mainly helps stabilize model outputs via complex multi-step processes, increasing engineering complexity, lowering scalability, and reducing efficiency. SFT or RL may achieve similar gains more efficiently. Showing experiments that capture emergent improvement would significantly increase the work’s impact.\n5. Only strong closed-source models are evaluated. Open-source models and models of different sizes are not tested.\n6. Experiments are limited to web navigation and software engineering agents, with insufficient evaluation on more general tool-use agents such as Taubench or BFCL. Moreover, the experience extraction prompt appears tailored to web navigation, implying that each new scenario may require a custom prompt and tuning, reducing usability and scalability.\n7. While the paper claims efficiency gains in terms of fewer steps, actual runtime may be higher. The memory module adds significant inference overhead, particularly on the TTFT, which may degrade user experience and hinder practical deployment in real-world agent scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)