Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
Authors: Yuanzhe Hu
Yu Wang
Julian McAuley
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Agents; Agents with Memory; Memory Agents Benchmark; Evaluation for Memory
Primary area: datasets and benchmarks
TL;DR: We introduce a unified evaluation framework designed for memory agents.

Abstract: Recent benchmarks for Large Language Model (LLM) agents primarily focus on evaluating reasoning, planning, and execution capabilities, while another critical component—memory, encompassing how agents memorize, update, and retrieve long-term information—is under-evaluated due to the lack of benchmarks. We term agents with memory mechanisms as memory agents.  In this paper, based on classic theories from memory science and cognitive science, we identify four core competencies essential for memory agents: accurate retrieval, test-time learning, long-range understanding, and selective forgetting. Existing benchmarks either rely on limited context lengths or are tailored for static, long-context settings like book-based QA, which do not reflect the interactive, multi-turn nature of memory agents that incrementally accumulate information. Moreover, no existing benchmarks
cover all four competencies.  We introduce MemoryAgentBench, a new benchmark specifically designed for memory agents. Our benchmark transforms existing long-context datasets and incorporates newly constructed datasets into a multi-turn format, effectively simulating the incremental information processing characteristic of memory agents. By carefully selecting and curating datasets, our benchmark provides comprehensive coverage of the four core memory competencies outlined above, thereby offering a systematic and challenging testbed for assessing memory quality. 
We evaluate a diverse set of memory agents, ranging from simple context-based and retrieval-augmented generation (RAG) systems to advanced agents with external memory modules and tool integration. Empirical results reveal that current methods fall short of mastering all four competencies, underscoring the need for further research into comprehensive memory mechanisms for LLM agents.

Published as a conference paper at ICLR 2026
EVALUATING MEMORY IN LLM AGENTS VIA INCRE- MENTAL MULTI-TURA INTERACTIONS
Yuanzhe Hu Yu Wang*1, Julian McAuley University of California, San Diego
{yuh127, yuw164, jmcaul Bucsyyyyuuu..oooo edu
Datasets
Source Code
ABSTRACT
Recent benchmarks for Large Language Model (LLM) agents primarily fo- cus on evaluating reasoning, planning, and execution capabilities, while another critical component-mentte update, and retrieve long-term information is under-evaluated due to the lack of benchmarks. We term agents with memory mochanisms as mem- ory agents. In this paper, based on classic theorics from memory sci- ence and cognitive science, we identify four core competencies essential for memory agents: accurate retrieval, lest-Lime learning, long-range un- derstanding, and selective forgetting. Existing benchmarks either rely on limited context lengths or are tailored for static, long-conted settings like book-based QA, which do not reflect the interactive, multi-turn nature of memory agents that incrementally accumulate information. Morcover, no cxisting benchmarks cover all four competencies. We introduce Memo- ry AgentBenche new benchmark specifically designed for memory agents. Our benchmark transforms existing long-context datasets and incorporates newly constructed datasets into multi-turn format, effectively simulating the incremental information processing characteristic of memory agents. By carefully selecting and curating datasets, benchmark provides prehensive coverage of the four core memory competencies outlined above, thereby offering a systematic and challenging testbed for assessing memory quality. We evaluate a diverse set of memory agents, ranging from sim- ple context- based and angrenal-angmeenedd generation (RAG) systems advanced agents with external memory modules and tool integration. Em- pirical results reveal that current methods fall short of mastering all four competencies, underscoring the need for further research into comprehen- sive memory mechanisms for LLM agents.
encompassing how agents memorize,
1 INTRODUCTION
Large Language Model (LLM) agents have rapidly transitioned from proof-of-concept chatbots to end-to-end systems that can write software (Wang al. 2024c), control browsers Müäller & Zunid 2024). and reason over multi-modal inputs. Frameworks such as MANUS, OWL (Hu et al.
2025). OPENHANDS (Wang et al. 2024ch, and CODEX routinely solve complex, tool-rich tasks and achieve state-of-the-art results on agentic benchmarks like GAIA (Mialon et al. 2023) and SWE-Bench (Jimenez et al. (023). Yet these eval- uations focus almost exclusively on reasoning (planning, tool using, code synthesis) and leave the equally important question of memorization (abstraction, storing, updating, re- trieving) largely under-explored. Recent memory-centric architectures ranging from para- metric memory systems like MemnoryLLM (Wang el. al. 2024d), SELF-PARAM (Wang et al.), and M+ (Wang et al. as MEMGPT Paaker et al.
2025) to commercial token-level memory solutions such
2023 Lin et al. 2025 MEM0 CChikara et al.) 2025
"These authors contributed equally to this work.
1 Joint corresponding authors.

---

Published as a conference paper at ICLR 2026
went the zoo and saw elephants.
Ag is class 1; Bg is class 2;
(Long long conversation) What did the zoo?
A₂ class Bg is class
(Long long conversation)
Accurate Retrieval
Which class A₂? Which class Bg?
Test-Time Learning
I'm reading this book: Harry Potter gripped his wand, ready for whatever came next
love pear.
(Long long conversation)
(Long long conversation)
say love pears? That might be
typa. don't like fruits. I love peas.
Help me summarize the story.
Do I love pears?
Long Range Understanding
Selective Forgetting
Figure 1: Four complementary competencies that memory agents should have.
COGNEE Markovic et al. 2025;: ZEP [Rasmussonn et al. 2025 and MIRIX (Wang & Chen 2025) employ diverse strategics for storing and retricving past information. Despite grow- ing interest, their real-world effectiveness remains largely aneodolal, and there is currently no unified benchmark for systematically evaluating the quality of memory in agents. In this paper, refer to agents equipped with memory mechanisms Memory Agents, where memory can take various forms, including parameters, vectors, textual histories, or external databases. In this paper, primarily focus on memory agents that utilize textual histo- ries and external databases, as these approaches are most commonly deployed in real-world applications. In contrast, memory encoded in model parameters (Wang et al. 2024d 2025
Yin et al.) 2024) remains largely within academic research and is typically less capable than proprietary memory systems equipped on closed-sourced API models.
Based on some classic theories in memory and cognitive science (James 1890 McClelland
et al. 1995 Anderson & Neely 1996 Wimber et al. 2015), we identify four complementary
competencies (Examples shown in F Figure ! to evaluate memory agents: (1) Accurate Retrieval (AR): The ability to extract the correct snippet response to a query. This can involve one-hop or multi-hop retrieval, as long as the relevant information can be accossed with a single query. (2) Test-Time Learning (TTL): The capacity to incorporate new behaviors acquire new skills during deployment, without additional training. (3) Long- Range Understanding (LRU): The ability to integrate information distributed across extended contexts (2 100k tokens) and answer questions requiring global understanding of the entire sequence. (4) Selective Forgetting (SF): The skill to revise, overwrite, or remove previously stored information when faced with contradictory evidence, aligning with goals in model editing and knowledge unlearning tasks (Meng et al., 2023 Wang et al. 2024e). For these four competencies, we provide more detailed definitions Appendix
Previous datasets developed to evaluate memory in language models have notable lim- itations. Early benchmarks such LOCOMO (Maharana al.
2024)
(~9K tokens),
LooGLEfLi el. al., 2023 (~24k tokens), and LongBench [Bai et al.) 2023) (~ 20k Lokens) feature relatively short contexts that no longer challenge current models. More recent datasets like NovelQA MovelQA(Wangaa et al.) 2024a) (~200k tokens), MOCHA((arpingg)
et al. (2024]
(~127k tokens), Loong(Wang et al. 2024b) (~100k tokens), and o-Bench(Zhamg et al. 2024] (~150k tokens) extend the context length to evaluate global reasoning and retricval capabilities. However, these datasets were primarily designed for evaluating long-context language models rather than memory agents. The reason that long-context benchmarks cannot be directly used to evaluate memory agents is as follows. There is a fundamental distinction between memory and long context: memory serves compressed and distilled representation of past information. Rather than storing all historical content verbatim, memory selectively extracts salient details, removes irrelevant. information, and often in-
2

---

Published as a conference paper at ICLR 2026
corporates new inferences derived from prior experiences. Consequently, memory agents are designed to process context incrementally- stracting and consolidating information over time, generating new inferences, and learning novel rules from accumulated history. For this reason, datasets that provide the entire con- text. in single block are not directly applicable to evaluating memory agents. A more recent effort, LONG (Wu et al. 2025), seeks to address this limitation by using synthetic long-form conversations, which can be injected into memory gradually, session by session. Nonetheless, its evaluation framework remains constrained by limited topical diver- sity and less realistic interaction patterns, reducing its applicability to real-world memory agent scenarios.
imput piece by piece, ab-
To address these limitations, we introduce unified benchmark, Memory A Agent specifically designed to evaluate broad spectrum of memory mechanisms in agent systems. We also provide a framework for memory agent evaluation. In this framework, agents are presented with sequences of textual inputs that simulate multi-turn interactions with users. We reconstructed existing datasets originally developed for long-context LLM evaluation by segmenting and reconstructing inputs into multiple dialogue chunks and feeding them incrementally to the agent in a time order. However, since these datasets do not fully capture all four targeted memory competencies, we also introduce two new datasels: Event and Factoooooiiiation, designed to evaluate accurate retrieval and selective forgetting, respectively. Our benchmark includes evaluations of state-of-the-an commercial memory agents (such as MIRIX and MemGPT), contecontext agents that treat the full input as memory, and RAG agents that extend their memory through retrieval methods. We examine how techniques developed for long-context models and RAG transfer to the memory agent setting. By providing a consistent cvaluation protocol across diverse agent architectures and datasets, Memory AgentBench delivers comprehensive insights into agent performance across the four core memory competencies. In Table [ we compare the MemoryAgen Bench with previous representative benchmarks across multiple dimensions.
Our contributions are summarized as follows:
Datasets: We reconstruct existing datasets and create two new datasets to construct a comprehensive benchmark, covering four distinct memory competencies.
Framework: We provide unified evaluation framework, and open-source the codebase and datasets to encourage reproducibility and further research.
We implement various simple agents with diverse memory mecha- nisms, adopt commercial agents, and evaluate these agents on our proposed benchmark. Our results demonstrate that existing memory agents, although effective in some tasks, still face significant challenges in certain aspects.
Empirical Study:
RELATED WORK
2.1 BENCHMARKS ON LONG-CONTENT AND MEMORY
In this section, we review prior work on evaluation benchmarks, categorizing them into three domains: long-context understanding, cetreeval-uugmenteed generation, and memory agents. Benchmarks for Long-Context LLMs. Early benchmarks designed for long-context evaluation include LongBench Bai et al.) 2023) and LooGLE et al. 2023). with aver- age input lengths of approximately ly 20k and 24k tokens, respectively. More recent bench-
marks such as oo-Bench (Zhang et al. 2024). HELMET((en el al. 2024), RULER (Hseh et al.) 2024). NOCHA (arriiska et al. 2024]. NoLiMa (Modarressi et al. 2025) and Long- Bench V2 et al. 2024) extend context lengths to over 100k tokens. While these bench- marks effectively assess the model's ability to process extensive information in a single pass, they are primaríly intended for static long-context. reading comprehension and do not reflect the incremental, multi-turn nature of memory agents.
Benchmarks for Retrieval-Augmente
Generation. Beyond pure long-context eval- uation, a line of benchmarks targets retrieval-augmeented generation (RAG) for knowledge- intensive tasks such as open-domain QA, fact checking, and document ranking over fixed
3

---

Published as a conference paper at ICLR 2026
Table 1: A comparison between AemoryAgent and existing long-term memory QA benchmarks. #Q denote the total number of questions. Context depth is defined as the number of tokens in the history. *Not reported in the paper, based on our approximation. The context depth of StoryBench is not reported in paper. We compare these datasets in terms of their ability to comprehensively and effectively evaluate the each capability dimension that we propose. We also compare prior work in terms of their evaluation coverage of memory Sesnntt-specifccalfya whether they provide comprehensive assessments across different categories of memory methods: Long- Mag-Cootext Agents (LCA), RAG Agents, and Agentic Memory (AM).
Benchmark
#Q
Context Depth
Core Memory Competencies
TTL LRU
SF
Evaluation Coverage RAG
LCA
AM
Memnory
Zhong et al
2023n
194
5k
X
X
LoCoMa (Ma ma
7512
10k:
PurLTQAA Dy M
012
8583
1M
X
X
V
HealTTalk (Let 11
2024
728
375k"
V
X
V
X
LougMemEval
500
115k, 15M
StoryBench
(Win R a1 20.9
86
X
V
x
MemaoryAgentBeu.
2071
103k H4M
2021 and BEIR Thakur et al. 2021 More recent work explicitly evaluates end-to-end RAG systems under long-context or application-speciin scenarios, including LaR.A Li et al.. LONG*RAG QQi et al. 2024), FRAMES (Krishnal let al. 2025), and CRUD-RAC Lyu et al.] 2025 Large-scale benchmarks such as RAGG Bench (Fiill al. 2024 RAGTruth (Niu al. 2024h, FreshLLM (Vu et al.) 2024), and Te-RAGBench (Strich et al. 2025 further exlend the evaluation space to industrial manu- als, hallucination detection, time-sensitive web QA, and text-and-table financial reports, re- spectively. However, existing RAG benchmarks typically assume a static or slowly changing knowledge basc and short-lived interactions, emphasizing retrieval accuracy and grounding over the continual updating and selective forgetting of information that is central to memory agents.
corpora, e.g., KILT PPetroni et al.
Benchmarks for Memory Agents More recently, benchmarks such as LOCOMO Ma- harana et al. 2024). LongMemEval (Wu et al. 2025), RealTalk (Lee et al.) 2025 and Sto- ryBench (Wan & Ma 2025 have been proposed specifically for evaluating memory agents. While promising, LOCOMO still features relatively short conversations (~9k), and Long- MemEval uses synthetic conversations with limited topical diversity, making the dialogues less realistic and potentially less representative of real-world memory use cases. Mcanwhile, the evaluation scope of the above benchmarks is not sulficient to comprehensively assess the four core competencies Accurate Retrieval, Test-Time Learning, Long-Range Understand- ing, and Selective Forgetting that are essential for robust memory agents.
2.2 AGENTS WITH MEMORY MECHANISMS
Memory mechanisms are attracting more and more at attention lately (Wang et al. 2025 /02 Recent advancements in LLMs have demonstrated the capability process extended context lengths. ranging from 100K to over 1 million tokens. For instance, models such as GPT- 4o AT 2025b) and Claude 3.7 (Anthropic) 2025) can handle inputs of approximately 100K to 200K tokens, while models like Gemini 2.0 Pro (Deep.ind 2025) and the GPT-4.11 series extend this capacity beyond 1 million tokens. These strong long-context capabilities enable a simple yet effective form of memory: storing information directly within the context window. However, this approach is inherently constrained by a hard limit-on once the context. window is exceeded, carlier information must be discarded.
In parallel, RAG continues to serve as dominant paradigm for managing excessive con- text. By retrieving relevant information from earlier context and feeding it to the LLM, RAG allows systems to overcome context length limitations. For example, Open recent mem- ory functionality combines explicit user preference tracking with retrieval-based methods that reference prior interactions. RAG methods can be broadly classified into three cate- gories: Simple RAG: These methods rely on string matching techniques such as TF-IDE,

---

Published as a conference paper at ICLR 2026
BM25 (Robertson & Walker 1994). and BMX (Li al. 2024 which are entirely non-neural berbbedding--aseee RAG: This class leverages neu- ral encoders, primarily transformers, to map text into dense vector representations (Wu 2022 Early methods like DPR (Karpukhin et al. 2020) and Contriever [lzaaard let. al.
and operate on string-level similarity.
et al. 2021 are based BERT DDvvlin et al. 2019), while more recent models such as 2025) achieve significantly improved retrieval performance. Structure- Augmented RAG:: These approaches enhance retrieval with structural repre-
Qwen3 Embeddinn (Zhang et al.
Structure-Augmenteer
sentations such as graphs or trees. Representative systems include GraphRAG (Edge et al.,
2024.. RAPTOR (Sarthi et al.) 2024). HippoRAG-Ve V2 (Gutiérrez et al.) 2025 Cognce,
Zep Rasmussen et al. 2025) ). MemoRAG (Qian el al. 2025), Memu [Chhikkra el al. 2025
Memory (Kang et al. 2025). Memary (kingjulio8233 & Memary contributors) 2024) and
Memobase (meroodb-io & Memobase contributors 2025/. Despite their ellectiveness, RAG- based methods face challenges with ambiguous queries, multi-hop reasoning, and long-range comprehension. When questions require integrating knowledge across entire session or learning from long, skill-encoding inputs, the retricval mechanism-limin to the top-k most relevant passages- may fail to surface the necessary information. To address these limitations, Agentic Memory Agents introduce an iterative, decision-driven framework. Rather than relying on a single-pass retrieval, these agents dynaricalll process the query, retrieve evidence, reflect, and iterate through multiple retrieval and reasoning cycles. Ex- amples include MemGPT (Packer al. 2023), Self-RAG (Asai et al. 2023), Auto-RAG (Yu et al. 2024), A-MEM (Xu et al. 2022 Meml (Zhou et al. 20250. MemAgent (Yu et al. 2025). and MIRIX (Wang & Chen 2025 J. This agentic design is particularly effective for resolving ving ambignous or multi-step queries. Nonetheless, these methods remain fundamen- tally constrained by the limitations of RAG namely, the inability to fully understand or learn from range context that is inaccessible via retrieval alone.
Table 2: Overview of evaluation datasets. We select datasets that cover various important long-context capabilities, In the table, we underline the datasets we constructed ourselves. AvgL.: Average Context Length (measured using the GPT-4o-min mo-mmni model's tokenizer).
Category
Dataset
Metrics
AvgL.
Description
SH-Doc QA
197K
Siagle-Hop Gold passage zetrieval QA.
Accurate
MH-Doc QA
421K
Multiple-Hop Gold passage retrieval QA
Retrieval
LongMemEsal (8*)
Accuracy
355K
Dialogues based QA.
EvealQA
531K
multiple-choice QA on characters evente.
BANKING77
Banking intent clussiication, 77 labels.
CLINCI50
Intent chassficaatin, 151 Jnhels.
Test-time
NLU
Accuracy
103K
Task intent elassification, 68
Learning
TREC Coarse
Question type cluseification, 6 labels.
TREC Fine
Question type cleesiiicaiin, labels.
Movie Recommendation
Recallis5
1.44M
Recommend movies bused on provided dialogues exmmples.
Long Range
BBench-Sum
F1-Score
172K
Novel sunmarization with cutity replacement.
Understauding
Detective QA
Accuaracy
121K
Long range reasoning QA on delective novels.
Selective
DacsCooolidationsS SH
Single hop reasoning in facts judgnent. Multiple hop reasoning facts judgment.
Aecuracy
262K
Forgetting
Fact MasCooolddationMM
3
MEMORY AEE
3.1 DATASET PREPERATION
In this section, we describe how we reconstruct existing datasets and build new ones for evaluating each competency aspect. All datasets with their categories are shown in Table We introduce the details in datasets curation in Appendix B
2
Datasets for Accurate Retrieval (AR) We adopt four datasets to evaluate the accu- rate retrieval capability of memory agents. Three are reconstructed from existing bench- marks, and one is newly created: (1) Document Question Answering: This is a NIAH- style QA task where long passage contains single (SH-QA) or multiple (MH-QA) docu- ments answering the input question. The agent must identify and extract relevant snippets from the extended context. (2) Lonal This benchmark evaluates memory agents on long dialogue histories. Although task types like information extraction (IE) or multi- session reasoning are included, most tasks can be reformulated single-retrieval problems
5

---

Published as a conference paper at ICLR 2026
requiring agents to retrieve the correct segments spanning long multi-turn conversation. We reformulated chat history into five long dialogues (~355K tokens) with 300 questions (LongMemEval (S*) in Table 2 We create LongMemEval (S*) specifically for increasing the number of questions per context, mitigating the exhaustive needs of reconstructing the memory for each question. (3) Event (ours): We introduce Event this reasoning style NIAH task to evaluate agents' ability to recall and reason about temporal sequences in long-form narratives. In this dataset, the agent is required to read a novel and select the correct event from scries of candidates after recciving up-to five previous events. Unlike other long-range narrative text datasets that require extensive manual annotation (Zhang el al. 2024 Xu et al. 2024), our dataset is built through fully automated pipeline, making the process more elficient and scalable. Moreover, this pipeline can be directly applied to other novel-style texts.
Datasets for Test-Time Learning (TTL) We evaluate TTL via two task categories: (1) Multi-Class Classification (MOC): We reconstructed five classification datasets used in prior TTL work (Bertsch ct al., 2024 Yen al. 2024 BANKING77 (Casamueva ct al.
2020). CLINC150 [Larson ct al. 2019p. TREC-Coarse, TREC-Fine (Li & Roth 2002), and NLU [Luu al., 2019). Each task requires the agent to map sentences to class labels, Based on the setup from (Li et al. 2018: He et al.) 2023), we construct dataset to evaluate movie recommendation via dialogue history. The agent is exposed to thousands of movie- related dialogue turns and is asked to recommend twenty relevant movies based on the long interaction history.
leveraging previously seen labeled examples in context. (2) Recommendation
Datascts for Long Range Understanding (LRU) We evaluate LRU via two tasks: (1) Novel Summarization (Summ.): We adopt the Summarization task En Sum from oo-Bench (Zhang et al., 2024). The agent is required to analyze and organize the plot and characters of the novel, and then compose summary of 1000 to 1200 words. (2) Detective QA (Det QA): We also create a difficult question set from Detective QA (Xu et al. 2024). which include ten novels with 71 questions and these questions require agents to do reasoning over a longer narrative range.
Datasets for Selective Forgetting (SF) To assess whether an agent can forget out of date memory and reason over them, we construct a new dataset called FactCon- solidation. Specifically, We build this benchmark using counterfactual edit pairs from MQUAKE (Zhong et al., 2023b). Each pair contains a true fact and a rewritten, con- tradictory version. These are ordered such that the rewritten (new) fact appears after the original, simulating a realistic update scenario. We concatenate multiple such edit pairs to create long contexts of length 6K, 32K, 64K, 262K. We then adpot MQUAKE's origi- nal questions and categorize them into: (1) PactConoolidttinnnSS (Ours) (SH means Single-Hop), requiring direct factual recall (e.g., "Which country was tool A created in?"), and (2) Facttonsslidation-.
(Ours) (MH refers to Multi-Hop), requiring inference over multiple facts (e.g., "What is the location of death of the spouse of person D?"). Agents are prompted to prioritize later information in case of conflict and reason based on the fi- nal memory state. This sctup directly evaluates the strength and consistency of seloctive forgetting over long sequences.
3.2 DIFFERENT CATEGORIES OF MEMORY AGENTS
We evaluate three major types of memory agents that reflect common strategies for handling long-term information: Long-Conted Agents, RAG Agents, and Agentic Memory Agents. These approaches differ in how they store, retrieve, and reason over past inputs.
(1) Long Context Agents Modern language models often support extended context win- dows ranging from 128K to over 1M tokens. A straightforward strategy for memory is to maintain a context buffer of the most recent tokens. For example, a model with a 128K- token limit, the agent concatenates all incoming chunks until the total exceeds the window size. Once the limit is reached, the earliest chunks are evicted in a FIFO (first-in, first-out) manner. This agent design relies solely positional recency and assumes the model can attend effectively over the current context window. (2) RAG Agents RAG-based agents address context limitations by storing past information in an external memory pool and
6

---

Published as a conference paper at ICLR 2026
trieving relevant content needed. We consider three RAG variants: Simple RAG Agents: All input chunks are stored as raw text. During inference, a keyword or rule-based string matching mechanism retrieves relevant passages. Embedding- -hasedd RAG Agents: Each imput chunk is embedded and saved. At query time, the agent embeds the query and performs retrieval using cosine similarity between embeddings. Structure-uugenntae After ingesting all input chunks, the agent constructs structured representation (e.g., knowledge graph or event timeline). Subsequent queries are answered based on this struc- tured memory. (3) Agentic Memory Agents Agentic memory agents extend beyond static memory stores by employing agentic loops-itcratis
RAG Agents:
reasoning cycles in which the agent may reformulate questions, perform memory lookups, and update its working mem- ory. These agents are designed to simulate a more human-like process of recalling, verifying, and integrating knowledge.
3.3 DATASETS AND AGENTS FORMULATION
Datasets Formulation We standardize all datasets into the format: C2," (chunks), 91, 92, ... 9m (questions), and d1, 0g.... Om (answers), where C, denotes the i-th chunk wrapped to construct a user message with instructions of memorizing the content in a sequential input, and C1, "2, On represents a single conversation. Each chunk accom- panied by instructions prompting the agent to memorize its contents. Example prompts are provided in Appendix D.I When curating datasets like EventQA and FactConsslidation, deliberately design scenarios where multiple questions follow a single context. This allows us to probe the model's memory multiple times with one sequential injection. For example, in LME (S*), five contexts paired with 300 questions (shown in Table in Appendix B This design choice reflects a key trend: as LLMs support increasingly long context windows and memory agents become more capable of handling extended inputs, evaluation datasets must also scale accordingly. Injecting 1M tokens for just one question is resource-inefficiens whereas associating the same input with many questions provides significantly higher utility,
Prompt Formulation and Interaction Protocol Unlike standard long-context evalu- ations that input raw text, we wrap all input chunks within a simulated User-Assistan dia- logue to explicitly trigger the agent's memory mechanism. Each input chunk is preceded by a memorization instruction (e.g., "Please memorize and I will ask some questions...") to establish clear intent for information storage. Simultancously, for each specific dataset, we carcfully curated the instructions to ensure agents accurately comprehend the task in- tent and execute the required actions. Crucially, for the Selective Forgetting competency, we introduce explicit guardrails in the prompt. We explicitly instruct agents that facts are indexed by serial numbers, and that "newer facts have langer serial numbers." ". The agents are mandated to solve conflicts by finding the newest fact (see Appendix D for full templates).
Agents Formulation In our framework, all agents are required to take the chunks one by one, absorb them into memory, and incrementally update the memory. After seeing all the chunks, we ask the agent to answer the related questions. To guarantee fair comparison, we employed standardized prompt templates across all agents within each evaluation category, with only minimal adaptations where necessary.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
The datasets are split into four categories and the statistics of all datasots are also shown in Table 6 The evaluation metrics for all datasets are shown in Table 2 along with more dataset details. For the agents, as described in Section 3.2 we consider three categories of agents: Long-Contert Agents, RAG agents and Agentic Memory Agents, where RAG Agents can be further split into Simple RAG Agents, Embedding-bass based RAG Agents and AiructureAAugeent RAG Agents. We give the detailed introduction of each memory agent in Appendix C For chunk size settings, we choose chunk size of 512 for the SH-Doc QA,
7

---

Published as a conference paper at ICLR 2026
Table 3: Overall Performance Comparison. In the absence of a specified model, All RAG agents and commercial memory agents use GPT-4o-mini as the backbone. Thus we highlight the performance of GPT-44-mini as the reference. FC-SH and FC-MH mean FactConsoli- dation Single Hop and Consoliddii Multi Hop, respectively. Best viewed in colors.
AR
TTL
LRU
SF
Overall
Agenl Type
SH-QA MH-QA LME(S")
EvenI Avg.
MCC
Avg.
Summ.
DelQA Avg. Avg||FCSSH [FCSS FC-MH Avg. Scoree
Long
Agents
GPT-40
72.0
51.0
32.0
17.2
58.
$7.6
12.3
50.0
32.2
17.5
54.9
60.0
5.0
32.5
48.8
mini
64.0
13.0
30.7
58.0
19.2
82.0
15.1
05.6
28.9
63.4
66.2
45.0
5.0
25.0
42.2
CPT-4, I-mini
$0.0
66.0
55.7
71.8
75.6
16.7
46.2
41.9
56.3
49.1
36.0
5.0
46.9
Gemini-2 2.0Falah
87.0
65.1
8.7
25.0
50.2
3.0
42.4
Clande-3. 7-Somnes
77.0
53.0
34.0
74.6
59.7
80.4
18.3
63.9
52.5
71.8
02.2
43.0
2.0
22.5
40.6
CPT-4o-min
64.0
43.0
30.7
49.2
15.1
48.6
28.9
63.4
46.2
5.0
25.00
42.3
Simple:
R.AG
Agimts
BM25
66.0
36.0
45.3
74.6
60.5
75.4
13.6
44.5
19.0
02.1
35.6
48.0
3.0
25.5
41.5
Eimledddiog
Agente
Contriever
22.0
31.0
16.7
66.8
333
70.6
15.2
42.9
17.2
42.3
29.88
18.0
7.0
12.5
29.8
Text Embed-3 3 Small
60.0
11.0
J5.3
63.0
53.8
70.0
15.3
12.7
17.7
51.99
36.3
28.0
3.0
15.5
37
Text Embeed-3 3-3 Large
54.0
44.0
50.3
70.0
54.6
72.1
16.2
11.3
18.22
56.3
37.3
28.0
4.0
16.0
38.
Qwend. Embedding
57.0
12.2
14.8
59.2
3.0
Structure Augmenteed
RAG
Agenits
RAPTOR
29.0
38.0
34.3
45.8
36.5
10.4
12.3
15..9
13..4
42.3
27.9
14.0
1.0
7.5
27.0
GraghRAG
47.0
47.0
35.0
31.4
10.9
39.5
9.8
24.8
0.4
39.4
14.0
2.0
23.4
McmoRAG
29.0
33.0
77.0
13.1
45..
9.2
50.7
21.0
7.0
30.9
Iipp2RAAG-v2
76.0
66.0
50.2
67.6
65.1
81.4
10:2
35.8
14.6
57.7
36.2
54.0
5.0
29.5
41.6
Memi
25.0
32.0
36.00
37.5
332 6
32..4
10.0
21:2
4.8
36-6
201.7
18.0
2.0
10.0
21.1
Cognce
31.0
26.0
29.3
28 3
10.1
2.3
29.6
28.0
3.0
15.5
20.6
Zep
44.0
38.3
37.5
62.8
12.1
4.2
28.2
3.0
24.0
Agenbbe
Meconor
Ageata
SHERAG
35.0
42.0
25.7
31.8
33.6
11.6
12.8
12.2
0.9
35.2
18.1
19.0
3.0
11.0
18.7
MemGPP
41.0
MIRIX
62.0
61.0
37.3
29.8
47.5
38.4
9.8
24.1
9.9
40.8
25.4
14.0
2.0
26.2
MIRIX (L1-mini)
73.0
75.0
51.0
$3.0
63.0
61.0
10.3
35.7
18.0
62.0
40.5
20.0
3.0
11.5
37.7
MH-Doc QA, and LME(S*) tasks in AR, as well as for all tasks in SF. This is mainly because these tasks are composed of long texts synthesized from multiple short texts. For other tasks,
chunk size of 4096. Considering computational overhead and API cost, we uniformly use a chunk size of 4096 for Mem0, Cognee, Zep, and MIRIX. We report the
detailed settings of the chunk size in Table 15 Appendix E
4.2 OVERALL PERFORMANCE COMPARISON
Table 3 presents the overall performance across different benchmarks. We summarize the key findings follows: (1) Superiority of RAG methods in Accurate Retrieval Tasks. Most RAG Agents are better than the backbone model "GPT-4o-min"" in the tasks within the Accurate Retrieval Category. This matches our intuition where RAG agents typ- ically excel at extracting a small snippet of text that is crucial for answering the question. Models in Test-Time Learning and Long- Range
(2) Superiority of Understanding. Long-context models achieve the best performance on TTL and LRU. This highlights fundamental limitation of RAG methods and commercial memory agents, which still follow an agentic RAG paradigm. These systems retrieve only partial informa- tion from the past context, lacking the ability to capture a holistic understanding of the input- let alone perform learning across it. (3) Limitation of All Existing Methods on Selective Forgetting. Although being well-discussed task in model-editing com- munity (Mitchell et al.) 20022; Fang et al.) 2024), forgetting out-of-date memory poses significant challenge on memory agents. We observe that all methods fail on the multi-hop situation (with achieving at most 7% accuracy). Only long context agents can achieve fairly reasonable results on single-hop scenarios. In Section 4.3.4. we show that current reasoning models can have much better performance, while it does not change the conclusion that. Selective Forgetting still poses a significant challenge to all memory mechanisms.
4.3 ANALYSIS AND ABLATION STUDY
In this section, we present experiments and analysis along five dimensions: input chunk size, retrieval top-k, backbone model, and dataset validation. Additional results are provided in the appendix, including compuational latency (Appendix E.5), context length analysis (Ap-
pendix F.4), latency and GPU memory usage comparisons (Appendix E.5 E.6), further de-
8

---

Published as a conference paper at ICLR 2026
BM25 Senbeeiiii
HppoRAC-v2 +2 KemdPP
BM25 Embedding 48
HapoRM--22
Model Based FI
Accurcy (%6
10
512
1024
2048
4096
512
1024
2048
4096
Chunk Size
Chunk Size
(a) SH-Doc QA performance
(b) BeBech-Sum performance
Figure Performances on SH-Doc QA and oc-Bench-Sum with different chunk sizes.
Text-Embed-3LLarq EventQA
HippoRAG-V2
MH-Doc QA
Multi-Class Classification
Accuracy (%)
40
20
Top-K
Top-K
Top-K
Figure 3: The accuracies on different benchmarks when varying the retrieval top-k to be 2, 5 and 10.
tails on chunk size and top-k ablations (Appendix E.2 estimation (Appendix M
E.3
as well as the Cost-Performance
4.3.1 ABLATION STUDY INPUT CHUNK SIZE
To understand how chunk size impacts performance, particularly for RAG methods and agentic memory agents, conduct additional analysis where vary the chunk size while fixing the number of retrieved chunks to 10. The results are presented in Figure 2 From the figure, observe that when resources permit, using smaller chunk sizes and in- creasing the number of retrieval calls during memory construction can improve performance on Accurate Retrieval (AR) tasks. Finer-grained segmentation enhances the relevance of retrieved information, particularly for embedding- basedd methods. However, for tasks requir- ing Long-Range Understanding (LRU), varying the chunk size hurts the performance. This is likely because RAG methods are inherently less suited for tasks that demand integration of information across a large, coherent context.
4.3.2 ABLATION STUDY RETRIEVAL ToPK
In our experiments, although we report most results with the number of retrieved chunks set to 10 in Table 3 we also conducted ablation studies with varying retricval sizes. A subset of these results is visualized in Figure 3 with the full results provided in Table g in Appendix E. The results indicate that increasing the number of retrieved chunks generally improves performance across most tasks. It is worth noling that, with a chunk size of 4096 tokens, retrieving 10 chunks already yields an input of approximately 40k tokens. This places significant demands on model capacity. Due to this high token volume, we do not evaluate settings with 20 retrieved chunks.
4.3.3 ABLATION STUDY BACKBONE MODEL
To investigate how different backbone models impact the performance various memory agents, experimented with three different backbone models and selected four represen- tative methods from both the RAG Agents and Agentic Memory categories. The complete experimental results presented in Table Our findings show that for RAG Agents, once

---

Published as a conference paper at ICLR 2026
Table 4: Performance comparison on three different backbone LLMs and four representative memory agents. We choose one datasct from every competency to evaluate agent perfor- mance.
Agent Type
Backbone Model
EventQA
Recor
ceBenchSum
FaerCon-SI1
Avg.
GPT-4n-mini
74.6
13.6
19.0
48.0
38.8
BM25
GPT-4.1-mini
76.4
14.0
19.4
51.0
40.2
Gemini-2.0-Flash
70.8
10.0
18.9
47.0
36.7
GPT-4o-miti
63.0
15.3
17.7
28.0
31.0
Text Embed Small
GPT-4.1mmin
62.0
15.5
17.9
30.0
31.4
Gemini-2.0-Flash
64.0
10.3
17.2
27.0
29.6
GPT-1o-mini
3144
9.8
0.4
14.0
14.7
GraphRAG
GPT-4.1-minl
39.0
10.3
1.2
16.0
16.6
Gemnini-2.0-Flan
36.2
7.2
0.8
13.0
14.3
MIRIX
GPT-4o-mini
29.8
9.8
9.9
14.0
15.9
GPT-4.1-mini
(23.21)
10.3 (0.5t)
(9.07)
20.0
(6.01)
25.6 (9.7t)
the backbone is sufficiently strong, it no longer serves as the main performance bottleneck. Compared to the default setup, upgrading to more powerful model like GPT-441-min yiclds only marginal improvements. In contrast, the main results in Table 3 for the MIRIX method under the Agentic Memory category, using stronger backbone leads to substantial performance gains. This suggests that future advances in backbone models could further boost the effectiveness of Agentic Memory hodh
4.3.4
VALIDATION OF DATASET FACTCONSOLIDATION
As the performance of different models on this dataset remains drastically low, we turn to the stronger reasoning model o4-mini and val- idate our dataset by checking the performance of o4-mini smaller version of this datasct.
Table 5: Performances of reasoning models the dataset FactConsolidation.
Fact
FactCon-MII 6K
6K
32K
32K
The results are shown in Table 5 We found that the 6K version of the Fact dataset, both models perform well and are generally able complete the task effectively. However, their performance drops when the context length increases to 32K. Similarly, on the 6K version of the FactCon-MH dataset, the stronger O4-mini reasoning model achieves a decent score of 80.0, but its performance significantly drops to 14.0 when the context window reaches 32K. This indicates that our dataset solvable under short-context scttings, but current memory agents still lack strong long-range reasoning capabilities, making them unable to handle the task when presented with longer historical inputs.
GPT-4o
92.0
88.0
28.0
10.0
O4-mini
100.0
61.0
80.0
14.0
CONCLUSION
In this paper, we introduce Memory Agent a unified benchmark designed to evalu- ate memory agents across four essential competencies: accurate retrieval, test-time learning, long-range understanding, and selective forgetting. While prior benchmarks focus largely on skill execution or long-context question answering, Aenory fills critical gap by assessing how agents store, update, and utilize long-term information across multi- turn interactions. To build this benchmark, restructure existing datasets and propose two new ones EventQA and Consoliiiii tailored to stress specific memory be- haviors often overlooked in prior work. We evaluate wide spectrum of agents, including long-context models, RAG-based systems, and commercial memory agents, under a consis- tent evaluation protocol. Our results reveal that, despite recent advances, current memory agents st ill exhibit. substantial limitations when faced with tasks requiring dynamic memory updates and long-range consistency. One limitation of our work is that due to budget con- straints, so we could only conduct experiments on some relatively representative Memory Agents, As future work, we aim to provide more evaluation results for more memory agents.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics and associated author guidance; we assess potential impacts and document mitigations accordingly. We evaluate memory in LLM agents using dialogs liconse-compliant corpora; no personally identifiable information or data from minors were collected. To reduce dual-use risks, we release only safety-screened prompts and provide usage notes discouraging survellance-orientel applications. We will release code under the MIT License and third-party materials retain their original licenses.
artifacts under CC BY 4.0;
REPRODUCIBILITY STATEMENT
Upon acceptance, will open-source all code and data used in this paper. The reposi- tory will include (i) training/evallaaten /evaluaaiion scripts, configuration files, and exact prompts; (ii) dataset releases and generation scripts with seeds to fully regencrate interactions; and (iii) end-to-end run recipes. We will pin software dependencies and provide containerized en- vironment (Dockerfile plus conda/ punda//rqqirements.... and OS details to support deterministic re-runs, in line with community guidance on repro- ducibility statements and artifact preparation.
and report hardware, CUDA/cuDNN,
REFERENCES
Michael C. Anderson and James H. Neely. Interference and inhibition in memory retrieval. In Elizabeth Ligon Bjork and Robert A. Bjork (eds.), Memory, Handbook of Perception and Cogmition, pp. 237-313. Academic Press, San Diego, CA, 2 edition, 1996. URL
https::/nemorrcoonrro..oo net htt.r//11nee//1169661 .df
Anthropic. Claude 3.7 sonnet, 2025. URL /ttps:://ww... centhropic.com//iee 3-aude-3-7-sonee This announcement introduces Claude 3.7 Sonnet, described as Authropic's ' most intelligent model to date and the first hybrid reasoning model generally available on the market.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannanch Hajishirzi. Self-rag: Learn- ing to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Represenlations, 2023.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint Xiv:22300. 14508, 2023.
Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng. Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, et al. Longbench v2: Towards deeper understanding and reasoning on realistic long-context, multitasks. arXiv preprint orXin:2412 12. 12..2222..2,, 15204, 2024.
Amanda Bertsch, Maor Tvgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R Gorm- ley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. arXiv preprint Xic: 00200, 2024.
lnigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulié. Ef- ficient intent detection with dual sentence encoders. In Tsung- Hsien Wen, Asli Celikyil- maz, Zhou Yu, Alexandros Papangelis, Mihail Eric, Amuj Kumar, Inigo Casanueva, and Rushin Shah (eds.), Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pp. 38 45, Online, July 2020. Association for Computational Linguis- URL https: htap::s:/aallathhlogg.o. orrpge20.g/00...
lics. doi: 10.18653/v / nlp4convai-1.5/
Prateek Chhikara, Dev Khant, Saket Aryan, Taranject Singh, and Deshraj Yadav. Mem0: Building production-ready ai agents with scalable long-term memory. arXiv preprint 1Ti::5504119445 2025.
11

---

Published as a conference paper at ICLR 2026
DeepMind. Gemini pro, 2025. URL httpps///aappnnc...gooo////ooohs pro/
Jacob Devlin, Ming-Wae Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Yiming Du, Hongru Wang, Zhengyi Zhao, Bin Liang, Baojun Wang, Wanjun Zhong, Zezhong Wang, and Kam-Fai Wong. Perltqa: A personal long-term memory dataset for memory classification, retrieval, and fusion in question answering. In Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10)), pp. 152 164, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https://aclanthoolgg. hrt.://4./////aaaaaarraaa sighan-i..1.1.
Hermann Ebbinghaus. Memory: A contribution experimental psychology. Annals of Neurosciences, 20(4):155-15) 156, 2013. doi: 022214//s.00972.5511111
URL https:
//pubmed. ncbii. ncbi. nlm. mih. gov/2520604//
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From lo- cal to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404. 16130, 2024.
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. Xiv preprint arXiv:2410. 02355, 2024.
Robert Friel, Masha Belyi, and Atindriyo Sanyal. Ragbench: Explainable benchmark for retrieval-augumented generation systems. arXiv preprint arXie:2407. 11t0/07..110000 11005, 2024.
Bernal Jiméncz Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. From rag to memory: Non-parametric continual learning for large language models. arXiv preprint Akix:2502.14800 2025.
Zhankui He, Zhouhang Xie, Rahul Jha, Harald Steck, Dawen Liang, Yesu Feng, Bod- hisattwa Prasad Majumder, Nathan Kallus, and Julian McAuley. Large language models as zcro-shot conversational rocommendors. In Proceedings of the 32nd ACM international conference on information and knowledge management, pp. 720 730, 2023.
Cheng-Pine Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What's the Real Context Size of Your Long- Context Language Models?, August 2024. URL http://arxiv org/ arXiv:2404.06654 [cs).
Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Zeyu Zhang, Yifeng Wang, Qianshuo Ye, Ping Luo, and Guohao Li. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation, 2025. URL https://githhnn http///tttb.coooaaaaaaaeeee comm/came-ai/ozl)
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Ar- mand Joulin, and Edouard Grave. Unsupervised dense information retricval with con- trastive learning. arXiv preprint arXiv:2112.091 2021.
William James. The Principles of Psychology, volume 1. Macmillan, London, 1890. URL https://books.. .ow.iag.goooge...ooo cctm///oootts7---JUULLBBBCCCCC
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasinhan. Swe-bench: Can language models resolve real-world github issues? arXio preprint arXiv:2310.06777 067770 2023.
Jiazheng Kang, Mingming Ji, Zhe Zhao, and Ting Bai. Memory os of ai agent. arXiv preprint arXiv:2506. 06326, 2025.
12

---

Published as a conference paper at ICLR 2026
Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit lyyer. One thousand and one pairs: A' novel' challenge for long-contex: language models. arXiv preprint (.kt::2406112200 2024.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP (1), pp. 6769-666 6781, 2020.
kingjulio8238 and Memary contributors. Memary: The open source memory layer for ai agents, 2024. URL https: gi thub. com//kingjullo8828//eeaaaa GitHub repository.
Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarez, Adam Stam- bler, Shyam Upadhyay, and Manaal Faruqui. Fact, fetch, and reason: A unified evaluation of retrieval-angmenned generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4745 4759, 2025.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. An evaluation dataset for intent classification and out-of-scope prediction. In Kentaro Inui, Jing Jiang, Vincent. Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Melhods in Natural Language Processing and the 9th In- ternational Joint Conference 1311-1316, Hong Kong, China, November 2019, Association for Computational Linguis- tics. doi: 10.18653/v1/DIS 1131. 1131. URL
Natural Language Processing (EMNLP-LJCNAL
pp.
http:///caaattoooooooorr////////
Dong- Lee, Adyasha Maharana, Jay Pujara, Xiang Ren, and Francesco Barbieri. Realtalk: A 21-day real-world dataset for long-term conversation. ar preprint ar arvi::2002.132 2025.
Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts? arXiv preprint (Xxv:3311.04980 2023.
Kuan Li, Liwen Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Shuai Wang, and Minhao Cheng, Lara: Benchmarking eerrieval-ugmmented generation and long-context llms no silver bullet for Ic or rag routing. In Forty-second International Conference on Machine Learning.
Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Char- lin, and Chris Pal. Towards deep conversational recommendations. Advances in neural information processing systems, 31, 2018.
Xianming Li, Julius Lipp, Aamir Shakir, Rui Huang, and Jing Li. Bix: Entropy-weightee similarity and semantic-enhanced lexical search. arXie preprint arXiv:2408.0000 06643, 2024.
Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th Inter- national Conference on Computational Linguistics, 2002. URL https haap:as:/aaaaathoogggg/ org/C02-1150
Kevin Lin, Charlie Snell, Yu Wang, Charles Packer, Sarah Wooders, Ion Stoica, and Joseph E Gonzalez. Sleep-time compute: Beyond inference scaling at test-time. arXiv preprint orXiv:2504.13111 2025.
Xingkun Liu, Arash Eshghi, Pawel Swictojanski, and Verena Rieser. Benchmarking natural language understanding services for building conversational agents, 2019. URL https:
//arxiv. org/ waabbb//3.. 05566
Yuanjie Lyu, Zhiyu Li, Simin Niu, Feiyu Xiong, Bo Tang, Wenjin Wang. Hao Wu, Huany-
ong Liu, Tong Xu, and Enhong Chen. Crud-rag: A comprehensive chinese benchmark for aenrieval-nugmeented generation of large language models. ACM Transactions on In- formation Systems, 13(2):1-322 2025.
13

---

Published as a conference paper at ICLR 2026
Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of Ilm agents. arXiv preprint orXiv:2402.11755 17753 2024.
Vasilije Markovie, Lazar Obradovic, Laszlo Hajdu, and Jovan Pavlovic. Optimizing the interface between knowledge graphs and Iims for complex reasoning. arXiv preprint arXic:2505.244 24478, 2025.
James L. McClelland, Bruce L. McNaughton, and Randall C. O'Reilly. Why there are complementary learning systems the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 411 457, 1995, doi: 10.1037 ncbi. mlm. mih. glh.gov/7624455/
10.1037/0033- /0037-00332222 2010/0000300000000 102.3.4291.... URL https: //pummed.
memodb-io and Memobase contributors. Memobase: Profile-based long-term memory for GitHub reposi-
ai applications, 2025. URL tory.
Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in transformer. In ICLR. OpenReview.nes 2023.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. Memory-based model editing at scale. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 15817-15831. PMLR, 2022.
Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A Rossi, Se- unghyun Yoon, and Hinrich Schütze. Nolima: Long-context evaluation beyond literal matching. arXiv preprint arXiv:2502.0510 2025.
Magnus Müller and Gregor Zunič. Browser use: Enable ai to control your browser, 2024.
URL htttps://githiie com wwt.boowoeeereeeeeeeeoeeweeeeeee wwwpbbrrwwwweeeeeeoeeeeeeeeeeeee wwwpeereeeeeebbeeeereeeee
Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. Ragtruth: A hallucination corpus for developing trustworthy augieeallaugmmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10862 10878, 2024.
OpenAl New embedding models and api updates, 2024. URL https://openai.com. seeetiiinngmddelsssee
new-embedding models anddapii-udates
OPenAL Introducing gpt-4.1 in the api, 2025a. URL Attps://openat.comm/
4-1-4-1/
OpenAl. Gpt-4o system card, 2025b. URL https://openai.ccom cominnexx/ got-4o- system- This report outlines the safety work carried out prior to releasing GPT-4o including external red tcaming, fronticr risk ovaluations according to our Preparedness Framework, and an overview of the mitigations we built in address key risk areas.
Charles Packer, Vivian Fang, Shishir G Patil, Kevin Lin, Sarah Wooders, and Joseph E Gonzalez. Memgpt: Towards 11ms as operating systems. 2023.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thore,, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. Kilt: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2523 2544, 2021.
14

---

Published as a conference paper at ICLR 2026
Zehan Qi, Rongwu Xu, Zhijiang Guo, Cunxiang Wang, Hao Zhang, and Wei Xu. Long2rag: Evaluating long-context & long-form retrieval-angmented generation with key point recall. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 4852-
4872, 2024.
Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Defu Lian, Zhicheng Dou, and Tiejun Huang. Memorag: Boosting long context processing with global memory-enhande retrieval augmentation. In Proceedings of the ACM on Web Conference 2025, pp. 2366- 2377, 2025.
Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef. Zep: A temporal knowledge graph architecture for agent memory. arXiv preprint arXiv:2501. 189566 2025.
Stephen E Robertson and Steve Walker. Some simple effective approximations to the 2- poisson model for probabilistic weighted retrieval. In SIGIR '94: Proceedings of the Sew- enteenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, organised by Dublin City University, pp. 332-241. Springer, 1994.
Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Manning. Raptor: Recursive abstractive processing for tree-organized retrieval. In The Twelfth International Conference on Learning Representations, 2024.
Jan Strich, Enes Kutay Isgorur, Maximilian Trescher, Chris Biemann, and Martin Semmann. T 2-ragbench: Text-and-table benchmark for evaluating retrieval-augmmened generation. arXiv preprint arxiv:2506. (atn:2206.12270 12071, 2025.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. Xiv preprind ar Xrv:v:2104 086663, 2021.
Tu Vu, Mohit Tyyer, Xuczhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun- Hsuan Sung, Denny Zhou, Quoc Lc, et al. Freshllms: Refreshing large language models with search engine augmentation. In Findings of the Association for Computational Lin- guistics: ACL 2024, pp. 13697 13720, 2024.
Luanbo Wan and Weizhi Ma. Storybench: A dynamic benchmark for evaluating long-term memory with multi turns. arXic preprint arXiv:2506. 13356, 2025.
Cunxiang Wang, Ruoxi Ning, Boqi Pan, Tonghui Wu, Qipeng Guo, Cheng Deng, Guang- sheng Bao, Qian Wang, and Yue Zhang, Novelqa: A benchmark for long-range novel question answering. arXie preprint 203. 12766, 2024a.
Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, al. Leave no document behind: Benchmark- ing lont-context lims with extended multi-doc qa. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5627-5646, 2024b.
Xingyao Wang, Boxuan Li, Yufan Song, Frank Xu, Xiangru Tang, Mingehen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference on Learning Representations, 2024c.
Yu Wang and Xi Chen. Mirix: Multi-agent memory system for Ilm-based agents. arXiv preprint wxXi:2507..79557 2025.
Yu Wang, Xinshmang Liu, Xiusi Chen, Sean O'Brien, Junda Wu, and Julian McAuley, Self- updatable large language models by integrating context into model parameters. Tn The Thirleenth International Conference on Learning Representations.
Yu Wang. Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, Xian Li, Bing Yin, et al. Memoryllm: Towards updatabd large language models. or Xio preprint vrxi:22422.04622 2024d.
15

---

Published as a conference paper at ICLR 2026
Yu Wang, Ruihan Wu, Zexue He, Xiusi Chen, and Julian McAuley. Large scale knowledge washing. arXiv preprint arXiv:2405.1677 16720, 2024e.
Yu Wang, Dmitry Krotov, Yuanzhe Hu, Yifan Gao, Wangchunshu Zhou, Julian McAuley, Dan Gutfreund, Rogerio Feris, and Zexue He. M+: Extending memoryLIM with scalable long-term Lang-term memory, In Forty-second International Conference on Machine Learning, 2025.
URL https://openreeeee
met/foruunid--cggieee
Yu Wang. Chi Han, Tongtong Wu, Xiaoxin He, Wangchunshu Zhou, Nafis Sadeq. Xiusi Chen, Zexue He, Wei Wang, Gholamreza Haffari, Heng Ji, and Julian J. McAuley. Towards lifespan cognitive systems. TMLR, 2025/02.
Maria Wimber, Arjen Alink, Ian Charest, Nikolaus Kriegeskorte, and Michael C. Ander- son. Retrieval induces adaptive forgetting of competing memories via cortical pattern suppression. Nature Neuroscience, 18( ():22 589, 2015. doi: 10.1038/mn.333 URL https://wwwt ftttp::/www.nature..
nature. http:/attrr..cmmarrricleeeeeee/e nn. 39973 3973
Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai- Chang, and Dong Yu. Long- memeval: Benchmnrrkngg chat assistants on long-term interactive memory. In The Thir- leenth International Conference on Learning Representations, 2025.
Qiyu Wu, Chongyang Tao, Tao Shen, Can Xu, Xiubo Geng, and Daxin Jiang. Pel: Peer- contrastive learning with diverse augmentations for unsupervised sentence embeddings. ar preprint or 22x::2201.1200 2022.
Wujiang Xu, Kai Mei, Hang Gao, Juntao Tan, Zujie Liang, and Yongieng Zhang. A-mem: Agentic memory for Ilm agents. arXiv preprint arxin2502. 12110, 2025.
Zhe Xu, Jiasheng Ye, Xiaoran Liu, Xiangyang Liu, Tianxiang Sun, Zhigeng Liu, Qipeng Guo, Linlin Li, Qun Liu, Xuanjing Huang, et al. Detectiveqa: Evaluating long-context reasoning detective novels. arXiv preprint arXiv:2400. 2Xii:2409.024000 09.00000..6.. 2024.
Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. Helmet: How evaluate long-context language models effectively and thoroughly. arXiv preprint arXiv:24 10. 02694, 2024.
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Zhiyuan Zeng, Qinyuan Cheng, Xipeng Qiu, and Xuan-Jing Huang. Explicit memory learning with expectation maxinization. In Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 16618-16635, 2024.
Hongli Yu, Tinghong Chen, Jianglao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, et al. Memagent: Reshaping long- context IIm with multi-conv rl-based memory agent. arXiv preprint arXim:2507.02255 02259 2025.
Tian Yu, Shaolei Zhang, and Yang Feng. Auto-rag: Autonomous retrieval-augmmened gen- eration for large language models. arXiv preprint arXiv:2411.1944,, 2024.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. oobench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15262 15277, 2024.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXie preprint arXiv:2506. Xxx::2506.00166, 2025.
Wanjun Zhong, Lianghong Guo, Qiqi Gao, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. Xiv preprint arXiv:2305. 102000 2023a.
16

---

Published as a conference paper at ICLR 2026
Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions. arXiv preprint arXiv: 14 795, 2023b.
Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang Meml: Learning to synergize memory and reasoning for efficient long-horizon agents. arXiv preprint arxiv:2506. 15841, 2025.
17

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMs)
In this paper writing process, we used an LLM to assist with content polishing for example, identifying grammatical errors and suggesting revisions for sentences that unclear potentially ambiguous. Additionally, we used the LLM to generate character icons, which were later used in the creation of main plot visualization.
B DETAILS OF DATASET
Here we provide detailed introduction to the datasets used for evaluating the four core competencies, including the dataset curation, corresponding metrics, average context length, and brief description. Details are shown in Table 2
B.1 ACCURATE RETRIEVAL (AR)
B.1.1 DEFINITION OF AR
The task of accurately retricving information has been extensively explored in prior work. In the domain of long-context modeling, the Needle-in-a-Haaystad (NIAH) task is widely used to evaluate model'ss ability locate the specific value based on a given key within lengthy input. In the RAG setting, this corresponds to document- based QA, where the model must identify and extract relevant snippets from one or more documents to answer a query. These snippets might reside in single location or be distributed across multiple documents. In this paper, we focus on agentic settings, where the "long context" " or "multiple documents" become long-form conversations. We define Accurate Retrieval (AR) as the ability of an agent to identify and retrieve important information that may be dispersed throughout long dialogue history.
B.1.2 DETAILS ON AR DATASETS
We use four datasets to evaluate the accurate retrieval capability of memory agents.
(1) Document Question Answering We improved two QA datasets from (Hsih et al, 2024.. These datasets provide multiple syuthetic contexts of varying lengths, ranging from 3K to over 200K tokens. We select 100 questions from the datascts with shorter context length. For each of these 100 questions,
collect the context and remove duplicate short documents, and then shuffle and concatenate them to create new long documents of 197K 421K tokens, making sure the new context containing the gold passages. Since most answers are short informational entities, such as years, names, or yes/no responses, we use substring exact match (SubEM) calculate the accuracy of QA. SubEM measures whether the predicted answer exactly matches the gold answer as a sub-string, which is common standard in question answering systems.
(2) Long This is dialogue-basse QA dataset. For LME(S*), we multi- ple historical conversation data segments, arrange them in chronological order, and finally concatenate them to create five long conversation histories, each with length of approxi- mately 355K tokens. Since some of the questions have open-ended answers, we adopt the approach used previous work and employ the GPT-4o model to assess whether the agent's responses meet the requirements.
a response is deemed satisfactory, it is marked as True. Finally, we calculate the proportion of satisfactory responses as the evaluation metric. Wu et al. (2025) reported in Table 6 that promet-engineered GPT-4o judge achieves 98.0% accuracy and demonstrates very high stability.
(3) EventQA Using five books from oc-Bench (each contains more than 390K tokens, counted using the gpt-4o-mini tokenizer), we identify the ten most frequently mentioned characters via SpaCy NER. We extract 101 events experienced by key characters using got-4o. For each event, we construct a 6-way multiple-choice question by pairing the true event. with five distractors generated via gpt-4o. The agent receives up-to five previous
18

---

Published as a conference paper at ICLR 2026
events and must identify the correct continuation. We report the mean accuracy 100 such questions per book, and ultimately present the average accuracy across all five books.
B.2 TEST-TIME LEARNING (TTL)
B.2.1 DEFINITION OF TTL
An essential capability for real-world agents is the ability to acquire skills dynamically through interaction with users. This mirrors the concept of In-Context Learning (ICL) in LLMs, where the model learns from a prompt containing small number of examples, often framed as few-shot classification tasks. Ideally, performance improves with additional examples in the prompt. In the conversational agent setting, prompts are replaced by dialogue histories. We define Test-Time Learning (TTL) as the agent's 's ability to learn to perform new tasks directly from the conversation. This property is crucial for enabling self-evolving agents that can continuously adapt and improve in real-world deployments.
B.2.2 DETAILS TTL DATASETS
We evaluate TTL two task categories:
(1) Multi-Class Classification (MCC) We adopt five classification datasets used in prior TTL work. For dataset curation, we use thousands of sentence samples from different categories, with each type of sample assigned a mumber as its label. Following the format ([sentence) \nn Label: (label) \nn, we concalenate all the sentences inlo a long context and shuflle them lo prevent samples of the same lype from being too concentrated. In this lask, the agent needs to refer to a long context and correctly classify the input content. Therefore, average accuracy as the evaluation metric.
(2) Recommendation (Recom.) We concatenate multiple short dialogues about movie recommendations from the original dataset, remove duplicate dialogues, and create long context containing over a thousand recommendation instances. In this task, the agent is required to recommend 20 movies based on the content of the dialogue. We evaluate the recommendations by calculating Recall@5, which measures the overlap between the top recommended movies and the ground truth.
B.3 LONG-RANGE UNDERSTANDING (LRU)
B.3.1 DEFINITION OF LRU
Long-range understanding refers to the agent's ability to form abstract, high-level compre- hension over extended conversations. For example, when user narrates a long story, the agent should retain the content and derive a holistic understanding rather than just recall isolated facts. We define Long-Rand Range Understanding (LRU) as the ability reason about long- long-form inputs and answer high-level questions that require an understanding of the over- all content, rather than detailed recall. An example question might. be: "Summarize the main experiences of Harry Potter."
B.3.2 DETAILS ON LRU DATASETS
We evaluate LRU via the Summarization task En.Sum from co-Bench (Zhang et al. 2024). We follow the settings from (Yen et al. (024) and use the GPT-4o model in evaluating the sunmarized text. In this process, we assess the fluency of the input text (scored as or 1) and use the dot product of this score with the F1 score the final evaluation metric.
B.4 SELECTIVE FORGETTING (SF)
B.4.1 DEFINITION OF SF
In long-term interactions, agents often face evolving or conflicting information teobmattoonwwheheer whether about the external world (e.g., changes in political leadership) user-specific facts (e.g.,
19

---

Published as a conference paper at ICLR 2026
new occupation). This challenge is closely related to model editing (Meng et al. 2023 Fang et al. 2024) and knowledge unlearning (Wang et al. 2024e), which tocus on moditying or removing factual knowledge from language models. We define Selective Forgetting (SF) as the agent's 's ability to detect and resolve contradictions wetween out of date knowledge and newly acquired information, ensuring the agent remains aligned with current realities and user states. SF is distinct from Abstractive Retrieval (AR) in two key ways. (1) Certain questions requiring SF cannot be answered solely through AR. As illustrated in Figure an agent that retrieves all facts related to pears may fail to identify the updated information in the second message. (2) In AR, carlier messages remain relevant and should be retained, even when multiple pieces of evidence are required. In contrast, SF involves identilyingg ouidated or incorrect information and discarding it. That is, AR requires preservation of all related content, whereas SF requires overwriting prior facts to reflect the most up-to-date truth.
B.4.2 DETAILS ON SF DATASETS
We counterfactual edit pairs from the MQUAKE (Zhong et al. 2023b) sentence containing information was assigned a number. For each edit pair, the sentence representing ouldated informat (the distractor) is given a smaller number, while the sen- lence representing more recent. information (the one containing the answer) is given larger number. We then concatenate these sentences into long context in order according to their assigned numbers. We evaluate the SF via two datasets: Single-Hop Fact idation and Multi-Hop FactConsolidation. In these tasks, the agent's responses are mostly informational entities. Therefore, we also use SubEM (Substring Exact Match) as the evaluation metric to calculate the accuracy of QA.
dataset. Each
B.5 JUSTIFICATION FOR COMPETENCIES BASED ON COGNITIVE SCIENCE
Accurate retrieval is central to human memory rescarch, as evidenced by classical forgetting curves and recall tests that foreground fidelity of recall (Ebbinghaus] 2013). However, sole focus on accuracy obscures another fundamental axis: the timescale of learning and consolidation. Ebbinghaus observed that an initial, fleeting grasp rarely endures without reinforcement. (Ebbinghaus) 2013), and James (1890) distinguished primary (immediate) from secondary (enduring) memory. These classic distinctions ground our notions of test- time learning (incorporation of new information via memory) and long-rang' understanding (durable, integrated knowledge). Consistent with this, the Complementary Learning Sys- tems (CLS) framework delineates hippocampal system for rapid episodic learning and neocortical system for gradual, structured knowledge accumulation, underscoring the need Lo assess both quick memorization and long- hng-horizon retention (McClelland el al. 1995).
ion axis, another equally fundamental challenge is selective forgetting. Overlapping contradictory traces can impede retrieval, and interference has long been recognized as a primary driver of forgetting in cognitive psychology (Anderson & Neely 1996.. Neurocognitive evidence further shows that the brain engages targeted control mechanisms to resolve such interference at retrieval time (Wimber et al. 2015). We therefore include selective forgetting- the ability handle interference and contradictions as a core dinension.
Beyond the acquisition-cons/o
In sum, our four pategories-aconn accurate retrieval, test-time learning, long-range understand- ing, and selective forgetting-aliis with key dimensions of memory identified in cognitive science and Al memory systems, covering the essential capabilities that any robust memory mechanism must support in practice.
C DETAILED MEMORY AGENTS DESCRIPTION
We give detailed description of the memory agents used in experiments in this section.
20

---

Published as a conference paper at ICLR 2026
Table 6: Datasets categorized by the specific aspects of evaluation. Here 1K 1024.
Capability
Tasks
# of Sequences : QAs
Avg Len
SH-Doc QA
1: 100
197K
Accurate Retrieval
MH-Doc QA
1: 100
421K
LongMemEval (S*)
5 300
355K
EventQA
5 500
534K
BANKING-77
1 100
CLINC-150
1 100
Test-Time
NLU
1 100
103K
Learning
TREC (Coarse)
100
TREC (Fine)
1: 100
Movie-Rec Redial
1 200
1.4MM
Long-Rand
coBench-Sum
100 100
172K
Understanding
Detective QA
10 71
124K
FactConsolidation-SSt
1 100
Selective Forgetting
262K
FactCousolidation-Me
1 100
C.1 LONG-CONTENT AGENTS
We evaluate five modern long-context LLMs: GPT-4o (OpenAl 2025b) serves the high- performance, low-latency model with better cost efficiency than prior generations. While GPT-4o-mini is lightweight, budget-friendly alternative that enables large-scale evalua- tions by delivering faster responses and lower per-token costs. Notably, the GPT-4.1 (Ope- nAI 2025a) family strengthens instruction following and maintains strong performance at very large context windows (reported up to 1M tokens). Considering the higher token cost, evaluation. Gemini-2.0-Flast
(DeepMind] (2225) targets high throughput and the use of built-in tools, offering 1M token context window for ef- ficient long-context processing. Claude-3.7-Sonnee (Anthropic 2025) is a hybrid-reasoning model with optional visible "extended thinking," " strong mathhcodingg skills, and developer- controlled thinking budgets.
we choose the GPT-4.1-mint
C.2 RAG AGENTS
We consider three RAG variants: Simple RAG Agents, Embedding-based RAG Agents, and Struclure-Augmentte RAG Agents.
(1) Simple RAG Agents We implement a BM25 (Robertson & Walker 1994] retriever as strong lexical bascline: it scores documents by term frequency with saturation and inverse document frequency, with length normalization controlled by parameters K and b. BM25 remains competitive for exact-match queries and complements dense retrievers with robust precision on keyworded questions.
unsuper- vised dense retriever trained via contrastive learning on large text corpora, enabling semantic matching without labeled pairs. Text- -en-E-eeddinn--S-allll----------
(2) Himbedding-based RAG Agents Contriever (Izacard al. 2021
(OpenAl 2024 are Ope-
nAl's general-purpose embedding models offering cost quality trade-ofl (e.g.. 1,536 vs. 3,072 dimensions) for search and retrieval. Beeen---abedding-11 (Zhang al. 2025
4B- iB-parameter embedding/ heagg//rrriinggg model family geared toward multilingual retrieval and long-text understanding.
(3) Structure- Strueture-Augmentter RAG Agents RAPTOR (Sarthi et al. 2024 is method building a hicrarchical tree of recursive summarics (bottom-up clustering and abstraction) and retrieves across levels for long-document QA. GraphRAG (Edge et al. 2024 extracts a knowledge graph and community hierarchy, then performs graph-aware retrieval and summarization. MemoRAG (Qian al. 2025) introduces dual-system pipeline with
21

---

Published as a conference paper at ICLR 2026
light "global-memory"" model to guide retrieval and stronger model for final answers. HippoRAG-v2 (Gutiérrez et al, 2025 extends hippocampal-inspitee factual, sense-making, and associative memory tasks over standard RAG. We also evaluate three open-sourced memory agents: Memo, Cognee and Zep. Memo (Chhikara et al. 2025) provides persistent agent memory layer for storing ceahiingg/rrrrrtiee retrieving user-specific knowledge to enhance personalization. Cognee (Markovic et al.) 2025) is an open-source memory engine that builds structured (graph-native) memories via ECL pipelines power graph-aware RAG. Zep (Rasmussen et al., 2025) is temporal sowledge-graple memory platform for agents, designed to assemble and retrieve long-term conversational and business context.
retrieval to improve
C.3 AGENTIC MEMORY AGENTS
For Agentic Memory Agents, We evaluate the Self-RAG Asai et al. 2023 McmGPT (Packer al. 2023), and MIRIX (Wang & Chen 2025) on our benchmark. Self-RAG use LLMs as the agent to decide when/wihe /hhat to retrieve and to critique its own outputs. MemGPT operates the hierarchical memory management, paging relevant snip- pets between short-term and long-term stores and using event-driven interrupts to maintain coherence and evolvability over extended interactions. MIRIX adopts a multi-agent mem- ory architecture with six specialized memory types (Core, Episodic, Semantic, Procedural, Resource, Knowledge Vault) and coordinator that orchestrates updates/retrieva across agents.
For comparability, we standardize prompts, tool access, and settings (like retricval TopK and imput chunk size) across above all systems.
D PROMPTS
We introduce the examples of prompt used memory construction and task execution in this section.
D.1 INSTRUCTIONS FOR MEMORY CONSTRUCTION
When processing long-context inputs, we split the content into chunks of a specified size and feed these chunks into the agent memory. The agent can then extract relevant information from its memory based on the query to assist with query execution. This chunking approach helps organize and manage large amounts of contextual information, making retrieval and reasoning more efficient. In Figure 4 we provide several example instructions that require the agent to memorize the corresponding context.
D.2 INSTRUCTIONS FOR TASK EXECUTION
In Figure 5 we provide the examples of instructions used on different of datascts when handling the input queries. For some cxisting datasets, we refer the prompt settings from previous work such (Hsieh et al. 2024 Wu et al. 2025). For the dataset oc Bench- Sum, we also insert two answer examples as (demo) in the prompt. help the agent better understand the questions and standardize its outputs.
E DETAILED EXPERIMENTAL RESULTS
In this section, we provide detailed versions of the results presented in the main text.
E.1 DETAILED RESULTS ON TTL
We give detailed results Multi-Class Classification (MCC) task in Table 7 For all three types of tasks, bas-b agents generally underperform compared to their respective GPT-4o-mini backbones. This observation highlights certain limitations inherent to the RAG approach. For instance, in TTL tasks, RAG- based methods often struggle to more accurately retrieve context from memory that is closely associated with the input.
22

---

Published as a conference paper at ICLR 2026
Prompts Used for Memory Construction on Varions Tasks
Document
Question Answering (SH-Doc QA or MH-Doc QA):
Here a dislogne between User and Assistant on (time):
(User); The following context consists the documents have read: (chunk). Please memorize and will ask some questions based in future. (Assistant): Sure! have learned the documents and will answer the question you ask.
LME(S*):
Here a dialngue between User and Assistant (time):
(User): The following context is the dialogue history that I have talked with ChatBoT: (chunk). Please memorize it and I will ask some questions based on it in future.
(Assistant): Sure! have memorized the dialogue history and will answer the question you ask.
EventQA:
Hore is a dialugue between User and Assistant on (lime): (User): The following context is sections in a novel that have read: (chunk). Please memorize and will ask some questions based on future.
(Assistant): Sure! have memorized novel sections and will answer the question you ask.
Multi- Class Classification (MCC):
Here dialogue between User and Assistant (time):
(User): The following context is the examples have learned: (chunk)). Please memorize it and I will ask some questions based on it in future.
(Assistant): Sure! have memorized the examples and will answer the question you ask.
Recommmnndation (Recom):
Here is a dialugue between User and Assistant on (time): (User): The following context is the dialogue history that I have talked with recommender system: (chunk). Please memorize it and I will ask some questions based on it in future. (Assistant): Sure! have memorized the dialogues and will answer the question you ask.
Novel Summarization:
Here a dialogue between User Assistant on (time): (User): The following context. is sections novel that have read: (chunk). Please memorize and I will need you to summarize based it in future. (Assistant): Sure! have memorized the novel sections and I will summarize them when you ask.
Detective QA (Det Q4):
Here is a dialogue between User and Assistant (time): (User): The lollowing context is the sections in a novel that I have read: (chunk). Please memorize it and I will need you to answer the questions based on the entire novel.
(Assistant): Sure! have memorized the novel sections and I will answer the questions you ask.
Selective Forgelling (SF):
Here a dialogne between User and Assistant on (time): (User): The following context is facts that have learned: (chunk). Please memorize and will need you answer questions based on the order (Assistant): Sure! have memorized facts and answer the questions you ask.
Figure 4: The prompts we use for the agents create the memory.
E.2 RESULTS ON INPUT CHUNK SIZE ABLATION STUDY
In Table g we report the detailed results on evaluating the RAG-based Agents on different: chunk sizes and datasets. We selected chunk sizes from the two sets (512, 4096) and (512, 1024, 2048, 4096).
E.3 RESULTS ON RETRIEVAL TOPK ABLATION STUDY
In Table 9 we report the detailed results of the selected RAG-based Agents evaluated on five datasets. We choose different TopK ranging from (2, 5, 10).
23

---

Published as a conference paper at ICLR 2026
Examples of Prompts Used for Task Execution on Various Dataset
Document Question Answering (SH-Doc QA or MH-Doc Q4)
The context given as below: (memory). (n Answer the question based on the context. Only give me the answer and do not output any other words. \n Now Auswer Question: (question) \n Answer:
LME(S**)
Here the context of dialogne history: (memory) )n. Based on the relevant chat history, answer the question concisely, using single phrase if possible. \ Current Date: (question_ date), Now Answer the Question: (question) \n Answer:
EventQA
The context is given as below; (memory). \n Based on the context above, complete the task below: in (question) \n Your task to choose from the above events which event happens next based on the book excerpt. In your response to me, only include the answer without anything else. (n The event that happens next is:
Multi- Class Classification (MCC)
The context is given as below: (memory). \n Use the provided mapping examples from the context to numerical label assign numerical label to the context. Ouly output 'label: {[abel)} } and nothing else. \n Question: (question) label:
Recommmendation (Recom)
Here the context of dialogue history; (memory). in Pretend you movie recommender system. You need to recommend movies based the above dialogue history. Now I will give you new conversation between user and you (a recommender system). Based on the conversation, you reply me with 20 recommendations without, extra sentences. in For Examplesk [Conversation] in The recommendatiinn are: An Lmoviel in 2.movie2 \n ...Au Here is the conversation: (question) \n The recommendations are:
Novel Summarization
The book is given as below: (memory) in You are given a book above and you are tasked to summarize Write summary about 1000 to 1200 words. Only write about the plot and characters the story. not discuss themes or background the book. not provide any analysis or commentary. in (demo) (n Now summarize the book.
Detective QA (Det QA)
The context given below: (memory). In Based the context above, complete the task below: You required to answer the question based on the strict output format.) (question) \n
Fact Consolidation
Here is a knowledge pool with lots of new facts: (memory). \n Pretend you are a knowledge manage- ment system. Each fact in the knowledge pool is provided with a serial number at the beginning, and the newer fact has larger serial number. (n You need to solve the conflicts of facts in the knowledge pool by finding the newest fact. You need to answer a question based on this rule, You should give a very concise answer without saying other words for the question only from the knowledge pool you have memorized rather than the real facts in real world. \n For example: \n IKoowldgee Pool] 'nn Question:: Based the provided Knowledge Pool, what the name of the current president of Country Şu Auswer: Person D. inn Now Answer the Question: Based the provided Knowlodge Pool, (question) (n Answer:
The example prompts for the Memory Agents in Table refers to the accumulated text from the sequential inputs.
Figure 5:
Here (memory)
E.4 RESULTS ON DIFPERENT CONTEXT LENGTH ABLATION STUDY
In Table 10 we report the performances of different agents when scaling the input length. We measure the average context length via the tokenizer of GPT-4o-mini and here 1K is 1024. For Long-Conted performance at relatively small context lengths (c.g., around 50K tokens). However, as the context length increases, the performance of these agents declines accordingly. In contrast, for the RAG-based agents Memo and Cognee, their performance significantly lower than that of their backbone, GPT-4o-mini, even when the context. length is relatively small.
Agents, tasks in the AR series generally achieve satisfactory
24

---

Published as a conference paper at ICLR 2026
Table 7: Overall performance comparison on the datasets for TTL. All RAG agents and commercial memory agents use GPT-4o-min as the backbone.
Agent Type
BANKING CLINIC
NLU
TREC C
TREC
Long Content Agents
GPT-4o
96.0
96.0
90.0
87.0
69.0
GPT-Io-mini
93.0
93.0
87.0
73.0
66.0
GP.--4.1-min
93.0
82.0
85.0
68.0
50.0
Gemini--2.0-Fam
91.0
90.0
84.0
88.0
67.0
Slaude-3.7-Suntt
97.0
98.0
86.0
87.0
79.0
GPT-1o-minl
93.0
93.0
87.0
73.0
66.0
Simple RAG Agends
BM25
89.0
89.0
81.0
62.0
53.0
Emblddinng RAG Agenit
Contriever
80.0
88.0
80.0
55.0
41.0
Text Sanbed-3-Small
88.0
89.0
83.0
54.0
36.0
Text- Embod-3-- LaLage
90.0
91.0
80.0
55.0
46.0
Qwen3-Fune EmbeeddineenBB
88.0
86.0
67.0
59.0
Structare- Aug.trree--gmmteee RAG
Agents
RAPTOR
78.0
75.0
73.0
48.0
23.0
GrahhRAG
64.0
54.0
49.0
24.0
6.0
MemoRAG
90.0
87.0
86.0
66.0
56.0
HippoRAA-v22
81.0
86.0
73.0
38.0
29.0
Memo
35.0
37.0
35.0
29.0
26.0
Cognee
31.0
12.0
42.0
11.0
18.0
Zep
Agentic Memoor Agents
Self-RAG
15.0
MenGIPT
89.0
83.0
79.0
56.0
31.0
MIRIX
49.0
(IRIX(4.1-mini)
65.0
83.0
69.0
73.0
25.0
Table 8: Performance comparison on different datasets and chunk sizes. Here we choose chunk sizes from {512, 1024, 2048, 4096) and we use k=10 for RAG-basod methods.
SH-Doc QA
MH-Doc QA
SoBench-Sumn
512
1024
2048
4096
512
1024
2048
4096
512
1024
2048
4096
BM25
66.0
67.0
68.0
66.0
56.0
54.0
52.0
56.0
11.5
13.2
15.2
19.0
57.0
53.0
52.0
50.0
47.0
44.0
40.0
38.0
7.9
9.4
13.2
14.8
HippoRAG-v2
76.0
70.0
57.0
49.0
66.0
63.0
51.0
38.0
10.5
14.6
MemGPT
41.0
32.0
24.0
27.0
38.0
33.0
37.0
4.2
2.5
Table 9: Performance comparison on different retrieve number.
SH-Doe QA
MH-Doc QA
EventQA R
TTL (MCO)
R=2
R=
R=
10
R=2
R=
R=
10
R
R
10
R
2
R=
R 10
BM25
50.0
60.0
66.0
49.0
54.0
56.0
66.6
71.2
74.6
67.8
74.6
75.4
Contriever
17.0
20.0
22.0
22.0
27.0
31.0
54.4
66.8
56.0
63.0
70.0
70.6
Laat-
36.0
47.0
51.0
37.0
41.0
44.0
51.8
70.0
69.4
72.4
RAPTOR
22.0
27.0
29.0
36.0
38.0
45.8
41.8
40.4
56.3
57.4
59.4
HippoRAG-v2
60.0
69.0
76.0
53.0
60.0
66.0
58.8
67.6
67.4
58.8
61.4
61.4
Self-RAG
27.0
33.0
35.0
34.0
39.0
42.0
28.2
30.6
31.8
9.0
11.6
11.6
Table 10: Performance comparison on different context length.
SIL-Doe QA
MIL-Doe QA
EventQAA 108K
PartCon-SII
FactCon-MII
51K
194K
197K
51K
104K
421K
51K
534K
32K
64K
282K
32K
64K
262K
CPT-4o
91.0
84.0
72.0
72.0
08.0
51.0
96.88
94.0
77.2
88.0
85.0
60.0
10.0
13.0
5.0
GPT-to-mini
15.0
GPT-4. I-mini
83.0
86.0
83.0
72.0
75.0
66.0
97.0
93.8
82.6
82.0
72.0
36.0
7.0
9.0
5.0
Gemini- 2.0-Flaa
92.0
87.0
87.0
69.0
61.0
59.0
93.1
55.6
67.2
49.0
62.0
31.0
7.0
9.01
3.0
Clando-3.1 T-Sonne
77.0
59.0
45.0
Menfl
31.0
25.0
25.0
36.0
29.0
32.0
60.8
17.0
37.5
22.0
8.0
18.0
3.0
2.0
2,0
Cognee
38.0
42.0
31.0
36.0
38.0
26.0
53.1
39.0
26.8
39.0
31.0
28.0
4.0
5.0
3.0
E.5 RESULTS ON COMPUTATIONAL LATENCY ANALYSIS
To illustrate the latency of various memory agents in terms of (1) Memory Construction (M.C.); (2) Query Execution (Q.E.,, we report the latency of various memory agents on MH-QA and LME (S*). This part of experiments is done server with Four NVDIA
25

---

Published as a conference paper at ICLR 2026
Table 11: Computational latency (in seconds) comparison on Long- Context Agents.
MH-QA
LME (S*)
GPT-4o
17.0
20.1
GPT-4o-mini
4.9
5.4
GPT-4.1-mini
9.0
7.4
Gemini-2.0-Flasl
12.4
10.1
Claudo-3.7-Sonnct
23.3
22.7
Table 12: Computational latency (in seconds) comparison on RAG based agents. M.C.
means Memory Construction and Q.E. means Query Execution. *Indicates that the time is obtained through estimation.
MH QA
LME (5*)
512
4096
512
4096
M.C.
Q.E.
M.C.
Q.E.
M.C.
Q.E.
M.C.
Q.E.
BM25
0.12
0.47
0.11
1.7
0.09
1.1
0.08
1.9
Contriever
7.4
0.59
1.7
2.0
6.9
0.92
1.6
1.9
Text-Bmmbed-3-Large
6.1
0.46
5.0
1.7
6.5
0.62
5.8
1.8
Eonud-Embddding-- 4B
367
0.49
470
1.9
293
0.71
372
RAPTOR
193
0.41
161
0.67
108
0.60
104
0.53
GraphRAG
91.9
7.8
HippoRAAG-22 G-x2
1089
0.71
380
1.71
544
188
3.5
Memo
10804
0.79
1334
0.65
18483
1.6
2946
1.7
Cognee
11890
58.7
1185
4.8
4728
7.7
738
4.1
Self-RAG
11.4
3.1
8.1
2.4
5.3
0.82
5.2
1.0
MemGPT
433
9.4
101
10.3
392
11.7
85.5
12.3
MIRIX
29000
20171
14.1
12600
3258
8.7
MIRIX (CPTT441-min))
28800*
21361
16.9
9000
2512
9.2
Table 13: Peak GPU memory usage of embedding models (MB). We measure the memory usage on MH-QA dataset with different chunk size.
Agents / Chunk Size
512
4096
HippoRAG-v2 (NV-Fmbed-v2)
27674
60205
Owen:-Emmbedding--e
16671
41262
I.40 GPU and AMD EPYC 7713 64-Core CPU. We use the NV-Embed-v2 (7B) as the embedding model in HippoRAG-v2. We show the results in Table [ and 12 From the table, find that using smaller chunk size requires significantly more time for memory construction, especially for methods such as HippoRAG-v2, Mem0, Cognce, and McmGPT. Meanwhile, methods such MemO, Cognce and MIRIX need extremely high resources when constructing the memory.
E.6 GPU MEMORY USAGE COMPARISON
In main experiments, we mostly use the LLM API as the backbone models which do not need local GPUs. In experiments, the HippoRAG-v2 (NV-Embed-v2) and Qwen3- Embedding-4B require running the embedding model on GPU. We report their peak GPU memory usage in Table 13 where all experiments conducted single A100 80GB GPU.
F EXPERIMENTAL SETTINGS
In this section, present the experimental settings in evaluation.
26

---

Published as a conference paper at ICLR 2026
Table 14: Maximum output token limits for various tasks
Task
Max Output Tokens
SIIQQA / MIIQAA LMB(S*) EventQA
100
40
MCC
Movie Recommnenaation
do Bench-Sum Detective QA
1,200
500
Cacttonnoliiatiin
10
Table 15:
The choice of chunk size for different datasets.
Chunk Size
512
4096
SH-QA, MH-QA
Benchh--Sum MCC, Recom
Dataset
FactCon-SHI FactCon-MH LME(S*)
EventQA, Detective QA
F.1 MAX OUTPUT TOKENS
We provide the token number limitation for each task in Table
14
F.2 SETTINGS OF THE RAG AGENTS
For the embedding model selection in Strueture-Aummenteed RAG Agents and Agentic Mem- ory Agents, most approaches utilize OpenAT's embedding models, such as Text-Embod-3. Small. While for the HippoRAG-v2 method, we follow the same experimental setting as in Gutiérrez et al. (2025). employing the NV-Embed-v2 model.
We implement three open-sourced memory agents in our main experiments. (1) For Memo, we use memory.add() function to add the message with the content from each context chunk into the agent's 's memory repository during memory consolidation. During query execution, the relevant memory elements are retrieved through memory.search() function. The retrieved memories are then integrated into the query before being processed by the GPT-4o-mini backbonc model to complete the requested tasks. (2) For McmGPT, we employ the insert (ascer__assaae()) function during the memory consolidation phase to inject long context chunks into the Archival Memorry structure. During query execution, this agent. processes requests via the send message() function which generates appropriate responses based on the archived information. (3) For Cognee, we utilize the cognee.add()
and cognee.cognine cognify(() functions construct the memory graph from input chunks wherein the memory consolidation phase. During query execution, the cognee.search() function used retrieve contextually relevant information from the memory graph based on the input query.
F.3 SETTINGS OF THE CHUNK SIZE
We use smaller chunk size (512) for synthetic context used in AR and SF. For some tasks based on continuous text, such as xoBench and EventQA, we used a larger chunk size (4096). For tasks such MCC and Recom, considering the characteristics of these tasks and the computational cost, we also chose a larger chunk size (4096). For the memory construction methods that are more time-consuming and requiring more API cost, Mem0, Zep, Cognee and MIRIX, we uniformly used a chunk size of 4096 across all datasets. The detailed settings presented in Table 15
27

---

Published as a conference paper at ICLR 2026
G TASK RATIONALE AND JUSTIFICATION FOR SELECTIVE FORGETTING TASK
While the Selective Forgetting task may appear specialized or even synthetic at first glance, is designed to address a fundamental, universal challenge in long-term memory systems: maintaining context efficiency and mitigating interference between outdated and updated information. We justify the design, novelty, and validity of this task along four tightly linked dimensions:
1. Theoretical Necessity: For any memory system biological or artificial storage capacity is inherently finite. The ability autonomously discard (or selectively forget) outdated, redundant, superseded information is not niche requirement, but core prerequisite for keeping memory representations concise, robust, and free from conflicting signals. Our task is designed a controlled proxy to evaluate this underexplored yet critical capability.
2. Distinction from Previous Settings (e.g., Knowledge Updating): While prior work has explored knowledge updating (which focuses overwriting old facts with new, conflicting ones), our work uniquely emphasizes the explicit, proactive re- moval of non-essential information to free up cognitive and contextual space. This distinguishes our task from cxisting fact-updating benchmarks, and we view this framework a foundational step toward evaluating more complex memory man- agement behaviors in real-world settings.
3. Justification for the Controlled Synthetic Sctting:
We acknowledge that the current task setup includes synthotic clements, and this design choice is deliberate and methodologically justified. Constructing a naturalistic dataset with long-term interaction history (over 100K tokens) and unambiguous, precise ground-truth an- nolations for what. information should be forgotten is inherently challenging, as real-world forgetting decisions are often ambiguous and context-dependent. Our controlled synthetic setting isolates the selective forgetting capability from con- founding factors, enabling reproducible, apples-to-apples comparisons across differ- ent memory agent architectures.
4. Validity and Feasibility of the Task: Critically, we confirm that our proposed task, despite its synthetic design, is fully valid and solvable. As demonstrated in our ablation study, long-context agents with strong reasoning capabilities achieve near-perfect performance on short-context (6K tokens) versions of the dataset. This result directly confirms that performance degradation on long-context full-length task inpuls stems not from flaws the task definition itself, but from the fundamen- tal limitations of current memory agents in long-range reasoning specifically, their inability to accurately identify and discard outdated information across extended interaction histories.
H SUPPLEMENTARY VALIDATION FOR
LEARNING
This section provides supplementary justification for the terminology and design of our Test-Time Learning (TTL. cvaluation paradigm, as well as additional zcro-shot bascline experiments that validate the core premise of the TTL task.
H.1 TERMINOLOGY AND DESIGN RATIONALE
We use the term Test- Time Learning (TTL) describe the class of tasks that evaluate an agent's 's ability to acquire task-specific skills and rules incrementally from interaction history, and apply these learned patterns to unseen inputs at inference time. The terminology and task design are justified by three core principles:
1. Distinguishing Skill Acquisition from Static Information Retrieval: We explicitly differentiate TTL from standard retrieval-focused tasks (e.g., Accurate Retrieval, Long-Range Understanding) to highlight its unique focus on learning,
28

---

Published as a conference paper at ICLR 2026
rather than fact recall. In retrieval tasks, the agent's core goal is to fetch pre-defined, static facts from the interaction history, In contrast, TTL tasks (e.g., multi-class classification Banking77, personalized movie recommendation) require the agent to induce latent, classification rules, preference patterns, and task schemas from sequential labeled examples in the dialogue history, then generalize these patterns to completely novel, out-of-sample inputs. This operationalization directly aligns with the canonical definition of learning: updating behavioral policy based on accumulated experience, which occurs exclusively during test-time interaction with no pro-training or fine-tuning on the target task.
2. Controlled Operationalization of Online Learning: While a fully dynamic on- line learning sctting would involve interleaved loop of memory update, retrieval, task execution, and feedback-driven memory refinement, such a setup introduces significant confounding variables that make impossible isolate an agent's 's mem- ory capacity from unrelated reasoning, planning, or execution errors. To enable robust, reproducible, and unbiased evaluation of pure memory-enables learning ca- pability, we adopt two-stage protocol that retains the core of online learning while eliminating confounds:
The agent incrementally processes a sequence of labeled task examples, simulating the experience accumulation process of real-world agent interactions over time.
Acquisition Phase:
The agent tested on its ability to apply the rules and patterns acquired in the acquisition phase to held-out, unseen inputs.
Evaluation Phase:
This design ensures that performance differences between models can be directly attributed to their ability to leverage long-term interaction history for learning, rather than spurious factors from dynamic feedback loops.
3. Foundational Framework for Agent Memory Research: As research on memory-enabled L.M agents remains in nascent stage, our TTL evaluation frame- work provides a standardized, reproducible operationalization of in-situ learning for memory agents. While our current protocol simplifies the fully online learning loop for evaluation stability, it successfully captures the core competence required for real-world self-improving agents: the ability to improve task performance solely by memorizing and generalizing from past interactions. We will extend this framework to more complex, fully interleaved ouline learning scenarios in future work.
H.2 ZERO-SHOT BASELINE VALIDATION EXPERIMENTS
A core premise of our TTL task design is that performance improvements in the full memory setting are driven by the agent's 's ability to learn from historical examples, rather than prior knowledge encoded in the base LLM's pre-training. To validate this premise, we conducted zero-shot baseline evaluations, where models were tested on the TTL tasks with no access to the historical example sequence (i.e., no opportunity for test-time learning).
Table presents the zero-shot performance of three mainstream LLMs on our two core TTL tasks: Multi-Class Classification (MCC, Banking77) and personalized Movie Recommenda- tion (Recom.). We also include the full memory performance of GPT-4o-mini for direct comparison, to quantify the performance degradation when test-time learning is disabled.
The results confirm our core hypothesis: all models exhibit cear-chance performance in the zero-shot setting, with average accuracy below 4% across both tasks. In contrast, GPT- 4o-mini achieves 48.6% average accuracy when provided with the full historical example sequence, representing a 45.2 percentage point absolute improvement. This stark perfor- mance gap demonstrates that the base LLMs have meaningful prior knowledge to solve these long-tail tasks out of the box, and all performance gains in the full memory setting are explicitly driven by the agent's ability to learn from the provided interaction history. This validates that our TTL benchmark accurately measures test-time learning capability, rather than pre-training knowledge spurious pattern matching.
29

---

Published as a conference paper at ICLR 2026
Model
MCC
Recom.
Avg.
Zero-Shot Setting (No Access Historical Enamples)
GPT--o-mini
0.6
6.1
3.4
GPT--411mmin
0.8
5.7
3.3
Gemini-2.0-Flash
0.0
5.5
2.8
Full Memory Setting (With Access to Historical Examples)
GPT--4o-miii w/ full context
82.0
15.1
48.6
Table 16: Performance on Test-Time Learning (TTL) tasks under the zero-shot setting,
compared to the full memory setting. All metrics are task accuracy, with higher values indicating better performance.
I COSt-PENFORRANCE ANALYSIS
I.1 METHODOLOGY
To provide realistic asscssment of the practical limitations of cach agent architecture, our cost calculation assumes Context Caching is enabled (a standard feature in modern APIs like OpenAl), which significantly reduces the cost for Long-Context Context (LC) models processing shared histories. We compare three representative architectures: Long Context (LC) models, RAG Agents, and the agentic memory system MIRIX.
Pricing Basis: Costs are calculated based on OpenAl's pricing as of November 2025:
GPT-44-minn: $0.15 / 1M imput tokens, $0.60 / 1M output tokens
GPT-4.1-mini: $0.40 / 1M input tokens, $1.60 1M output tokens
Context Caching: We apply cached input pricing (50% discount for GPT-4o-mini, 75% discount for GPT-4.1-mini) for Long-Contexe Agents, assuming sequential questioning on shared contexts.
Settings: RAG agents use Top-K-10. Embedding indexing costs are excluded as one-time expenses.
1.2 DOST-PERFORMANCE RESULTS
We report the amortized inference cost per query (in USD) and the corresponding per- formance metric (Accuracy /Score) across four representative datasots that vary context length and reasoning complexity in Table 17
Modkl/Anhiileetrr
Mil-Doc QA
MCC
Detective Q4
PCSU
(USD)
(US03)
GPP-4o-man
50.0
43.6
$0.008
52.0
30.01
63.4
$0.01
45.0
GPTT-4 Lintal
$0.003
66.0
$t.013
70.6
91/11
56.1
50.027
J6.0
HAG Agrnts (UAES5 Somin
<20 001
56.0
$0.00
75.6
90/06
M8.1
<90.001
(L.imini))
SAALO
02.0
Table 17: Estimated Amortized Cost vs. Performance per Query. Costs are amortized over the question set sharing the same context. Performance scores use the metrics defined in the main paper.
I.3 KEY INSIGHTS
1. The Brcciency-Reasoninng Tradc-off in RAG: RAG agents like BM25 are cost- efficient. (< $0.001 $0.006 per query) but suffer in tasks requiring global reasoning (e.g., Detective QA, scoring 52.1 vs. 63.4 for LC models). This limits their utility in complex analytical scenarios.
2. Cost Prohibitiveness of Long-Context Scaling: While powerful, Long- Context models face steep cost increases with stronger backbones. Upgrading from GPT-4o-mini to GPT-4.1-mini roughly quadruples the cost (from $0.010 to $0.043
30

---

Published as a conference paper at ICLR 2026
for MH-Doc QA), making high-end long-context deployment expensive even with caching.
3. Agentic Memory (MIRIX) as the Optimal Middle Ground: A critical finding is that MIRIX (with GPT-4.1-mini) achieves lower cost ($0.016) than the raw GPT-4.1-mini Long-Context setup ($0.043) on memory-intensive tasks like MH- Doc QA, while delivering superior performance (75.0 vs. 66.0). This demonstrates that agentic memory mechanisms can successfully decouple performance from linear context costs, offering a scalable solution for high-performance applications.
STRICT COMPUTE-MATCHED COMPARATIVE EXPERIMENTS
J.1 EXPERIMENTAL SETUP
To address concerns about budget effects confounding the comparison between architec- lures, we conducted a strict compute-matched ablation study on Banking77 (TTL) and Book Summarization (LRU) using the strongest backbone (GPTT441-mini). We define three budget levels:
Low ( 4K tokens): Constrain Long-Context (LCC) models to truncated contexts; limit RAG and MIRIX to Top-K=1 chunks to match token counts.
Medium ( 40K tokens): Moderate budget setting for all architectures; limit RAG and MIRIX to Top-K=10 chunks.
High ( 100K+ tokens): Scale RAG and MIRIX to retrieve high Top-K chunks match the LC full-context budget.
For Book Summarization, we evaluate on random 30-book subset for computational ef- ficiency. We compare total compute load via total processed tokens, rather than strictly equalizing forward passes, as forcing agentic models to single forward pass would strip them of their core reasoning capabilities.
Tack
Budget Secting
Long- CouteCt (GPT-L.1-miai)
RAG (BM25)
Agentic (MIRIX)
Low 1K Top-K=1)
71.0
$3.0
52.0
TTL (Banking77)
Medium 40K Top-K=10)
90.0
66.0
High 101K)
93.0
$8.0
67.0
Low 4K Top-K-1)
8.2
7.9
8.4
LRU (Book Summarization)
Mediom 40K Top-K=10)
16.4
15.8
18.7
High 113K)
39.7
38.0
38.8
Table 18: Computo-matched experiment results on TTL and LRU tasks (Accuracy /Scora /core %).
We summarize our key findings as follows:
The Efficiency-Capaciie Trade-off: At the Low ( 4K) budget, RAG (83.0) significantly outperforms LC models (74.0), confirming RAG's suporior struc- tural efficiency for pattern matching via precise retrieval of relevant examples, while LC models suffer heavily from truncation. As the budget increases to Medium ( 40K), performance equalizes 90.0)). At the High budget, LC models scale to 93.0, whereas RAG saturates and slightly degrades (88.0) due to retrieved noise. This demonstrates that while RAG efficient, LC models have a higher capacity coiling when the budget permits.
TTL::
LRU: The Information Threshold Effect: For global reasoning tasks, per- formance exhibits a clear threshold behavior. At Low and Medium budgets, all architectures fail (<20.0 score), confirming that partial information is insufficient for full-book summarization regardless of the method. Only at the High budget (full text access) do all models achieve meaningful performance 39.0), with R.GG nearly matching LC models. This proves that for LRU tasks, success determined by meeting the full information threshold, not by the architecture itself.
31

---

Unlike standard long-context evaluations that imput raw text, we wrap all input chunks within simulated User-Assistani dialogue to explicitly trigger the agent's memory mecha- nism. Each imput chunk preceded by a memorization instruction (e.g, 'Please memorize the following information for future questions') to establish clear intent for information Published as a conference paper at ICLR 2026
K PROMPT DESIGN AND OVERWRITE POLICY ABLATION EXPERIMENTS
K.1
PROMPT DESIGN SPECIFICATION
storage. For each specific dataset, we carefully curated task instructions to ensure agents accurately comprehend the task intent and execute the required actions.
Crucially, for the Selective Forgetting competency, we introduce explicit guardrails in the prompt. We explicitly instruct agents that facts are indexed by serial numbers, and that newer facts have larger serial numbers. Agents are mandated to resolve conflicts by pri- oritizing the newest fact (full prompt templates are provided in the supplementary code repository).
We clarify that while prompt deltas can shift outcomes, our benchmark applies a unified and standardized prompt template across all evaluated agents (Long-context, RAG, and Agentic). This ensures that the performance gaps observed (e.g.; the failure of RAG agents in multi-hop Selective Forgetting) are attributed to the limitations of their memory mechanisms rather than prompt inconsistency.
K.2 OVERWRITE POLICY ABLATION EXPERIMENTS
To rigorously test whether explicit instructions can mitigate the forgetting/overwriin /vvrwwiting issue, we conducted additional ablation studies using explicit overwrite prompts on the GPT--41- mini baseline. We test two policy settings:
Treat the facts as a chrono- logical updale stream. If there is ANY conflict between facts, you must ALWAYS overwrite the earlier fact with the one having the larger serial number.'
Policy A (Always Prefer Later): "Crucial Rule:
Policy B (Conservative/ /Explicit Negation): "Crucial Rule: Be conservative with updates. ONLY discard overwrite earlier fact if the fact with the larger serial number explicitly negates or explicitly states the previous information is incorrect."
K.2.1 ABLATION RESULTS
The results of the overwrite policy ablation are presented in Table 19
Model/Setting
FC-SH
FC-MH
Avg.
GPT-4.1-mini (Baseline)
36.0
5.0
20.5
GPT-4.1-mini (Policy A)
40.0
4.0
22.0 (+1.5)
GPT-4.1-miní (Policy B)
28.0
4.0
16.0
Table 19: Overwrite policy ablation results on Selective Forgetting tasks (Accuracy %).
K.2.2 KEY INSIGHTS
1. Limited Generalization with Aggressive Updates: While Policy slightly improves performance on single-hop tasks (FC-SH score increases from 36.0 to 40.0), it fails to generalize to complex multi-hop reasoning (FC-MH drops to 4.0). This suggests that while prompting helps with simple fact retrieval, it cannot effectively propagate updates through multi-step reasoning chains.
2. Performance Degradation with Conservative Constraints: Policy B sig- nificantly degrades average performance (-4.5 points). The complex conditional
32

---

Published as a conference paper at ICLR 2026
instruction (checking for explicit negation) increases the model's cognitive load and induces overly cautious behavior, preventing the retrieval of valid updates.
These findings serve as a sanity check that validates our core motivation: Selective Forgetting cannot be solved by prompt engineering alone, and requires dedicated memory mechanism design.
L SUPPLEMENTARY NOTES FOR LLM-AS-A-JUDGE AND INPUT FORMAT
L.1 VALIDITY OF LLM-AS-A-JUDGE EVALUATION
We acknowledge that model-based evaluation may not be optimal for all task contexts. How- ever, we adoptod LLM-as-a-judge scoring on the LongMemEval and coBench-Sum datasets to remain consistent with prior work, and we validate its appropriateness as follows:
For LongMemEval, the questions admit clear, objective ground-truth answers with minimal subjectivity. Wu et al. (2025) report that a prompl-engineered GPT-4o judge achieves 98.0% agreement with human annotations, demonstrating very high stability.
Yen et al. (2024) validate that GPT-4o judgments mostly align with human eval- ualions for long-context summarization tasks.
These findings confirm that our LLM-as-a-judge setup reliably reflects human evaluation for the tasks in our benchmark.
L.2 RATIONALE FOR CHUNKED INPUT FORMAT
We acknowledge that real-world personal assistants may receive streaming input (e.g,, con- tinuous user interactions or real-time data streams). However, in practice, it is necessary to quantize real-world inputs before feeding them into a language model (e.g., accumulating continuous input into discrete chunks for inference). Thus, inputting chunks into the agent is natural and realistic strategy to handle streaming input in real-world deployments.
Additionally, the chunked input format simulates the incremental, multi-turn nature of real- world user-agent interactions, where information arrives sequentially over time rather than a single full document. This setup critical for evaluating memory agents, which are designed to process information incrementally, rather than the static full-docunent input used in standard long-context benchmarks.
33


Reviews:
[{"review_id": "FDayACmaND", "reviewer": "Reviewer_nPYL", "summary": "This paper introduces a benchmark (datasets and evaluation metrics) for evaluating so-called \"memory agents\" - essentially sophisticated versions of RAG-like tasks that need retrieval before (maybe some reasoning and then) generation.\n\nThere are 4 categories of tasks; 3 of them somewhat standard, and one of them (\"Selective Forgetting\") that seems newly made in this paper. The datasets therein are a mix of re-purposed (about 70%) existing, and new (about 30%), datasets. \n\nThe authors evaluate several existing methods on this framework.", "questions": "Why devise the \"selective forgetting\" question ? Seems like a boutique task, so why was this particular boutique task chosen in this paper ?\n\nIs it possible to have a simple clear motivating example that shows where existing benchmarks fail to capture a specific particular way modern \"memory agents\" work ? \n\nWhat exactly is a \"memory agent\" ? The term seems to have been left somewhat intentionally vague so as to be broad, but it would be better to clearly explain it.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Tests memory beyond simple retrieval, supporting diverse tasks, multi-turn access etc.\n\nThe authors have made a comprehensive evaluation of a large number of different styles of RAG systems and agents. \n\nThe authors have spent the effort to make some new datasets, and re-purpose existing work into a uniform framework.", "weaknesses": "There should be a subsection called \"RAG Benchmarks\" in Section 2, covering related work in that area as well. \n\nThe \"Selective Forgetting\" task seems like a boutique, somewhat fabricated one. It should be justified why measuring this is specifically important.\n\nThe \"test-time-learning\" title seems a bit of a misnomer, since the actual task is just learning from long histories/context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uR3qANeEDh", "reviewer": "Reviewer_zBBu", "summary": "This paper introduces MemoryAgentBench, a benchmark suite designed to systematically evaluate memory agents along four core competencies: accurate retrieval, test-time learning, long-range understanding, and selective forgetting. This benchmark repurposes existing long-context datasets and introduces two new datasets to address shortcomings in existing memory benchmarks. A wide range of agents, including long-context models, RAG-based agents, and commercial memory systems, are evaluated using a unified protocol. The results highlight that current memory agents exhibit significant limitations, particularly in selective forgetting and long-range understanding.", "questions": "1. For test-time learning, did you measure a baseline performance without providing any in-context examples, to confirm that it improves significantly only when it learns from the given in-context examples? Understanding how novel the tasks are to the model would help interpret whether improved performance indeed reflects on-the-fly learning and not just prior knowledge or spurious pattern-matching.\n\n2. Can the authors also report the associated API costs for conducting different experiments? This will offer insights on the practical limitations of each agent architecture under the benchmark conditions.\n\n3. Minor Typo: Line 91: “fuor” to “four”", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles a timely task and an under-evaluated aspect on LLM-based agents. It introduces a broad, multi-faceted benchmark that evaluates memory beyond traditional long-context QA, encompassing lesser-explored but crucial capabilities like selective forgetting.\n- Introduces two new datasets, EventQA and FactConsolidation, to evaluate accurate retrieval and selective forgetting. Selective existing long-context datasets are carefully reconstructed into multi-turn interactions, simulating more realistic memory use in incremental settings.\n- Comprehensive empirical evaluation across diverse agent architectures under a unified framework.", "weaknesses": "- Most benchmark tasks are language-focused QA or summarization; it lacks memory evaluations in action-heavy or procedural domains (e.g., coding tasks like SWE-Bench or tool-use environments).\n\n- While covering four competencies, some of the chosen evaluation tasks may inherently favor certain types of models, raising questions of whether all approaches are on equal footing. For instance, the test-time learning tasks (e.g. few-shot text classification and movie recommendation through a long dialogue) essentially test in-context learning ability. Large context models with expansive memory windows can simply absorb all provided examples or dialogue turns, whereas retrieval-based agents might struggle to learn new tasks since they mainly fetch past knowledge. The result that long-context models outperform others on these TTL tasks is unsurprising, but it underscores that the benchmark might be conflating memory length with learning ability. In practice, an agent with a smaller context but a clever memory update algorithm might still be disadvantaged in such a setup. It just shows that current RAG systems are not very good at adapting. But it also suggests that the TTL evaluation might be favoring raw context length over true adaptation. A related concern is whether the information or skills being learned at test time were truly unknown to the base model; if not carefully controlled, stronger LLMs might answer correctly without genuine on-the-fly learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gc1zCo7ekH", "reviewer": "Reviewer_Mcjo", "summary": "This paper introduces MemoryAgentBench, a new benchmark for evaluating the memory management capabilities of large language model agents in multi-turn interactions.  The authors outline four essential competencies for memory agents: accurate retrieval, test time learning, long range understanding, and conflict resolution. To test these competencies, the authors adapt several existing datasets and create two new ones, EventQA and FactConsolidation, to expose weaknesses in current memory systems. They evaluate a wide range of memory types, including context-based models, retrieval augmented systems, and agentic memory agents. The results show that while some methods perform well on isolated tasks, no agent performs consistently across all four competencies. This suggests that memory management remains a significant open challenge for the next generation of intelligent agents.", "questions": "1.  How confident should we be that the evaluation metrics and the usage of LLM-as-a-judge in this paper are appropriate (e.g., for summarization or recommendation)?\n\n2. Why does this benchmark primarily use chunking as the main interaction method between the user and the Memory agent? Does this accurately reflect real-world interaction scenarios?\n\n3.  Could you explain the difference between the dialogue-based task designed in the article and the direct document question answering task from the perspectives like task definition and model performance? \n\n\tFor example, considering the document Question Answering task in the Accurate Retrieval category, what is the essential difference between transforming it into a dialogue and directly answering based on the original dataset? Is there a difference in the downstream model's performance?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Compared to previous benchmarks, this paper has a more comprehensive advantage in terms of data token length, problem diversity, and evaluation diversity.\n2. The paper presents a comprehensive and systematic framework that evaluates four memory-related competencies of LLM agents, including a novel aspect of Selective Forgetting. This enables consistent testing across a variety of scenarios by combining existing and newly created datasets.\n3. The paper conducts a broad comparison across various types of memory under a consistent evaluation protocol, clarifying the strengths and weaknesses of each approach, and also provides detailed analyses with ablation studies on various memory settings, offering practical insights into the efficiency and scalability of memory construction and query processing for  agents.\n4. The paper is clearly written and well-structured, with informative tables and figures. The inclusion of detailed prompts and ablations supports reproducibility.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nhV045liUm", "reviewer": "Reviewer_RnwS", "summary": "The paper proposes MemoryAgentBench, a benchmark for LLM memory agents that emphasizes incremental and multi-turn information intake. It operationalizes memory along four competencies: Accurate Retrieval, Test-Time Learning, Long-Range Understanding, and Selective Forgetting, grounding these choices in classic memory science. It converts or reconstructs several long-context datasets into chunked, time-ordered interactions and adds two new datasets, EventQA and FactConsolidation, to close gaps around temporal recall and conflict resolution. The study evaluates long-context buffers, three classes of RAG, and agentic memory systems including commercial offerings, under a unified protocol.\n\nThis paper offers an important contribution because it reframes memory evaluation around how agents actually encounter information, chunk by chunk, across sessions, rather than flooding a context window and calling it memory. The four-competency map separates skills that are often conflated, the datasets stress realistic patterns like temporal ordering and conflict updates, and the results give a clean and actionable picture. The framework is reproducible and extensible, so it can anchor future work on agent memory.", "questions": "(1) Under equalized prompt tokens, generated tokens, and forward passes, how do the best RAG, best long-context, and best agentic memory settings compare on TTL and LRU? \n\n(2) Can you expose the exact overwrite prompts and test a tiny policy grid, for example “always prefer later facts” or “prefer later only if explicit negation”?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The conversion of long-context resources into incremental dialogues is the key design choice and it is done consistently. The inclusion of commercial memory agents alongside RAG and long-context baselines provides a market-realistic snapshot. \n\nEventQA probes temporal recall under narrative structure, and FactConsolidation isolates resolution and forgetting. The dataset validation with shorter contexts is a nice touch that shows solvability in principle and surfaces context-length brittleness in practice.", "weaknesses": "Some claims would be stronger with strict compute-matched comparisons across families. For example, matching total retrieved tokens, prompt tokens, number of model calls, and wall-clock between a tuned RAG loop and a long-context buffer would reduce confounds when declaring winners per competency. Current results trend in the right direction but still leave room for “budget effects.” \n\nMoreover, the benchmark tells agents to “memorize” chunks. The exact prompt templates and any guardrails for overwriting or abstention deserve more foregrounding in the main paper, since small prompt deltas can shift Selective Forgetting outcomes. Much of this is in the appendix; a concise main-text summary would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)