Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs
Authors: Mohammad Tavakoli
Alireza Salemi
Carrie Ye
Mohamed Abdalla
Hamed Zamani
J Ross Mitchell
Venue: ICLR 2026 Poster (2026)
Keywords: long-term memory
conversation
retrieval-augmented generation
Primary area: datasets and benchmarks
TL;DR: We introduce BEAM, a multi-domain benchmark of long (100K–10M token) conversations with comprehensive memory probes, and LIGHT, a cognitive framework that improves LLM memory by 3.5%–12.69% over the strongest baselines.

Abstract: Evaluating the abilities of large language models (LLMs) for tasks that require long-term memory and thus long-context reasoning, for example in conversational settings, is hampered by the existing benchmarks, which often lack narrative coherence, cover narrow domains, and only test simple recall-oriented tasks. This paper introduces a comprehensive solution to these challenges. First, we present a
novel framework for automatically generating long (up to 10M tokens), coherent, and topically diverse conversations, accompanied by probing questions targeting a wide range of memory abilities. From this, we construct BEAM, a new benchmark comprising 100 conversations and 2,000 validated questions. Second, to enhance model performance, we propose LIGHT–a framework inspired by human cognition that equips LLMs with three complementary memory systems: a long-term episodic memory, a short-term working memory, and a scratchpad for accumulating salient facts. Our experiments on BEAM reveal that even LLMs with 1M token context windows (with and without retrieval-augmentation) struggle as dialogues lengthen. In contrast, LIGHT consistently improves performance
across various models, achieving an average improvement of 3.5%–12.69% over the strongest baselines, depending on the backbone LLM. An ablation study further confirms the contribution of each memory component.

Published as a conference paper at ICLR 2026
BEYOND A MILLION TOKENS: BENCHMARKING AND ENHANCING LONG-TERN MEMORY IN LLMS
Mohammad Tavakoli', Alireza Salemi Carrie Yet, Mohamed Abdalla', Hamed Zamani?, J. Ross Mitchell
Universsity Alberta Uniiversity of Massachusetts Amherst ftavako15, cye, mabdal12, jmitche2] teatthh2eealbbrtaa... {asalemi, zamani (Ccs umass .ddu
ABSTRACT
Evaluating the abilities of large language models (LLMs) for tasks that require long-term memory and thus long.context settings, is hampered by the existing benchmarks, which often lack narrative herence, cover narrow domains, and only test simple recall-oriented tasks. This paper introduces comprehensive solution these challenges. First, we present novel framework for automatically generating long (up to 10M tokens), coherent, and topically diverse conversations. accompanied by probing questions targeting a wide range of memory abilities. From this, we construct BEAM, a new bench- mark comprising 100 conversations and 2,000 validated questions. Second, to enhance model performance, we propose LIGHT-a framework inspired by hu- man cognition that equips LLMs with three complementary memory systems: a long-term episodic memory, short-term working memory, and scratchpad for accumulating salient facts. Our experiments on BEAM reveal that even LLMs with IM token context windows (with and without gle as dialogues lengthen. In contrast, LIGHT consistently improves performance across various models, achieving average improvement of 3.5%-12.66%% over the strongest baselines, depending on the backbone LLM. An ablation study fur- ther confirms the contribution of each memory component.
for example in conversational
augmentation) strug-
INTRODUCTION
Large language models (LLMs) have been deployed across diverse applications, including open- domain conversational agents (Laban et al. 2025 Chen et al. 2025), retrieval-auugeeeeeee -augmented generation (RAG) for open-domain question answering and fact checking (Lewis et al., 2220 Salemi et al.
2025 Salemi & Zamani 2025: Kim al. 2024b), long-document and code analysis (Li al. 2025
Jelodar et al. 2025; Fang et al. 2024). and scientific or legal research (Rueda et al.) 2025 Nguyen al. 2025). Many of these tasks demand models capable of processing long inputs, motivating [LMMs such as Gemini (DeepMind 2025) with input windows up to 1M tokens. Among these domains, conversational systems present an intuitive and critical need for extended context, users often engage in protracted, multi-session dialogues that require consistent memory across lengthy
interactions (Zhong al. 2024 Xu al. 2022 Du et al. [0244] Tan al. 2025). This highlights the importance of evaluating how well LLMs can reason over and utilize long conversational histories.
While there are many prior efforts on studying and evaluating long-term memory of LLMs (Kim) ct al. 2024a; Xu et al. 202T: Maharana et al.) 2024: Zhong et al. [2024] Xu et al. 2022; Du et al.] 2024 Tan al. (2025). existing benchmarks have fundamental limitations. Most extend conversa- tion length by artificially concatenating short sessions of different users, producing dialogues with abrupt topic shifts and weak narrative coherence. Such a construction artificially simplifies eval- uation because distinct segments are easily separable, reducing the need for true long-range rea- soning. Furthermore, these datasets typically target narrow domains often limited to personal- leaving many real-world application areas underrepresented. Finally, they emphasize simple context recall, overlooking other critical memory abilities such as contradiction resolution, recognizing evolving information, and instruction following.

---

Published as a conference paper at ICLR 2026
Joon Trare dereppe
Laer
Alerway Ao by
PPoking Camdioaa 6S)
Suapant
Avantel
Inaid
Narratu Ge serater
LLS
Hure
Deracion
Viiid
Conerraan Man
Dinaga Prncer
Butch
Baidn
Ned Quarict
Use
Queliio
Chack
User hesitari
10
Condera
Quesoor
LLM
Haed
Uod hrewer
Soarce
Figure 1: Overview of BEAM generation process. In the first stage, conversation plans are created for each chat seed. In the second stage, user utterances are generated from the conversation plans. In the third stage, assistant responses are produced. the fourth stage, probing questions are generated based the targeted memory abilities and corresponding conversation plans. In the final stage, invalid probing questions are filtered out, and nuggets are created for the validated set.
To address these limitations, this paper presents a framework for automatically generating long coherent conversations between user and an AI assistant-Scaling up to 10M tokens diverse domains-with a set of probing questions designed to evaluate diverse memory abilitics of any LLM on the generated dialogues. An overview of the data generation framework is shown in Figure This framework begins by defining a high-level conversation plan-a narrative for a particular do- main and a simulated user with generated attributes-thal outlines the overall flow of the dialogue. This plan is recursively decomposed into finer sub-plans that specify the storyline and its progres- sion. From these sub-plans we generate chronologically ordered user turns, which are then expanded with corresponding assistant responses. To increase realism, the system injects follow-up questions and clarifications from both sides. Finally, we automatically create set of probing questions that target ten distinct memory dimensions, with a focus on complicated and multi-hop reasoning, which are then validated by human annotators to ensure high quality. Using this pipeline, we construct the BEAM dataset: 100 diverse conversations ranging from 100 K to 10 M tokens each, accompanied by 2000 probing questions to evaluate the memory capabilities of LLMs.
[
To improve LLM performance probing questions, we introduce the LIGHT framework (Figure 2). which is applicable to both open-source and proprictary LLMs, inspired by research in human cog- nitive science and human's 's memorization and recall process (Sridhar et al. 2023 Binder & Desai 2011). This framework integrates three complementary memories: (1) episodic memory, a long- term index of the full conversation used for retrieval; (2) working memory, capturing the most recent user-assistant turns; and (3) a scratchpad, where after each turn the model reasons over the dialogue and records salient facts for future use. At inference, the LLM draws jointly on retrieved episodic content, the working memory, and the accumulated scratchpad to generate accurate answers.
To evaluate LLM memory capabilities and the effectiveness of our method, we conduct experiments on the constructed dataset, BEAM. using both open-source and proprictary models. Results show that even LLMs with long context windows perform substantially worse as conversation length increases. Our method improves the LLM's performance in answering the probing questions by 32.6%12.69% on average over the best-performing baseline, depending on the backbone model and conversation length. An ablation study further reveals the contribution of each LIGHT component on the performance. To support future work, we release all code, data, and evaluation scripts |
2 BEAM: BENCHMARKING MEMORY CAPABILITIES OF LLMS
2.1 PROBLEM FORMULATION
Let D = [T denote a collection of [D] conversations between users and a conversational agent
л. Each conversation is represented as T [t.) where t; T corresponds to the 7th utterance
Available at: http://ghh...commmmmmmmmaaallaaa
2

---

Published as a conference paper at ICLR 2026
(turn) in the dialogue. The objective of this work is to systematically evaluate a predefined set of memory abilities M exhibited by т across conversations. For each memory ability m E M, construct a probing dataset of size N, denoted 2m { 9:)) where 2; is probing question and yi is the corresponding ground-truth answer set. Fach probing question (x. E 2m appended as the (17) + 1)th turn in the dialogue, and the system generates response I = x <(;;; T) based on the conversation. The generated response is then evaluated using an ability-specific scoring function Um. producing a performance score s fmm (m, y/ 9). The goal of this work is quantity the performance of conversational systems on each memory ability in M.
2.2 BENCHMARK CREATION
Our goal is to evaluate how well LLMs can answer questions that depend on long-term conversa- tional memory. We measure performance across ten complementary abilities, seven drawn from prior benchmarks and three newly introduced here- Instruction Following, Event Ordering, and Contradiction Resolution (see Table 2 in Appendix [.I]. Abstention evaluates whether model withholds answers when evidence is missing. Contrudiction Resolution tests the capacity to detect and reconcile inconsistent statements across widely separated turns, maintaining global coherence. Event Ordering assesses whether a model can recognize and reconstruct the sequence of evolving information in the dialogue. Information Extraction measures recall of entities and factual details in long histories. Instruction Following examines sustained adherence to user- user-specified constraints over long contexts. Information Update evaluates revising stored facts as new ones appear. Multi-hop Reasoning probes inference that integrates evidence across multiple, non-adjacent dialogue seg- ments. Preference Following captures personalized responses that adapt to evolving preferences. Summurization assesses the ability to abstract and compress dialogue content, while Temporal Rea- soning tests reasoning about explicit and implicit time relations. Together, these abilities evaluate a system's 's capacity maintain, update, and manipulate information throughout extended conversa- tions (see Appendix [B.6 for examples of each ability). Given these abilities and the formulation in Section 2.1 the benehmark requires three components: 1) user-assistant conversation, 2) probing questions targeting key memory abilities, and 3) an evaluation methodology to assess the model's responses. The overall statistics of the constructed benchmark are summarized in Table 3 in Ap- pendix B.I
The rest of this section details the process used to construct these components.
The overview of our framework for creating conversations, probing questions, and the evaluation strategy is illustrated in Figure [ The process begins by generating simulated conver- sation between a user and an assistant. Structured conversation plans are first produced to guide the flow of the synthetic interactions. Each plan specifies sufficient information generate both user and assistant turns, ensuring a coherent and natural conversational trajectory. While a typical exchange consists of user question followed by an assistant response, realistic dialogues often involve follow-ups for clarification, elaboration, or related subtopics. To capture this, we incorpo- rate two interaction-control modules. The question-detection module identifies whether an assistant response includes a query that requires a user reply; if triggered, the system generates the corre- sponding user response. The follow-up detection module determines when the user would naturally pose a clarifying or elaborative question; if triggered, it produces an additional user query for the assistant. Together, these mechanisms produce conversations that exhibit interactive, bidirectional behavior beyond simple turn-taking. After the conversation is generated, an automated procedure constructs a candidate set of probing questions, each tailored to the specific memory abilities in the benchmark. These candidates are then reviewed by human evaluator, who selects valid questions and formulates the associated evaluation rubrics used for subsequent benchmarking. A case study and example of the different generated components of a conversation is provided in Appendix E
Overview:
2.2. 1 CONVERSATION PLAN GENERATION
A conversation plan serves as the scaffold for each dialogue, providing a coherent storyline that un- folds chronologically. Each plan is generated using an LLM based on seed information, including: the conversation domain; a title and theme; subtopics outlining specific topics; set of narratives defining evolving aspects (e.g., career progression, goals); a user profile with attributes such as name, age. gender, location, profession, and personality traits sampled from the Myers- Briggs Type Indicator (MBTI); a relationship graph linking the user to family, friends, and acquaintances, con- strained for realism (e.g., age gaps); and an explicit fimeline specifying the span of the conversation.
3

---

Published as a conference paper at ICLR 2026
To generate candidate titles and themes, human annotators specify target domains, then GPT-4.1 (OpenAl 2025a) generates candidate titles, themes, and subtopics using Listing 22 Human review- refine outputs for topical diversity. For each conversation,
generate 15-20 narratives using
the open-source L...MA3.3 70B model (AI, 2024) with the prompt in Listing 23 (Appendix H). Given the conversation seed, this model produces narrative elements capturing the evolving story- line, forming the backbone of a coherent conversation.
Conversation plans consist of N sub-plans, each representing a distinct stage in the conversation. Each sub-plan contains M bullet points, defined by narrative, a descriptive statement of its role in the storyline, and a time anchor. For conversations of 128K, 500K, and 1M tokens, a single plan is generated (line 4 in Algorithm|| [ Appendix (B3.5) by conditioning the LLM on the conversa- tion seed, profile, relationship graph, timeline, and specified counts of sub-plans, bullet points, and narratives (prompt in Listing 24 Appendix H). The number of sub-plans varies with domain and target length to meet the token requirement; e.g., coding domains generally require fewer turns than broader domains. For 10M-token conversations, one plan cannot capture the scope, so create ten interlocking plans forming a coherent longer narrative. The process begins with a global seed defining the overall topic and theme, but single seed is insufficient; instead, we derive ten distinct seeds one per plan so the narrative can evolve across stages. We propose two strategies:
Sequential Expansion: The global seed defines the initial point in the conversation' 's chronology. Subsequent seeds represent successive events (e.g., a trip. job search, later milestones). Using the prompt in Listing 28 (Appendix H. each new seed is generated from the main seed, profile, and timeline. Plans are then produced sequentially (line 12] A Algorithm| D Appendix [33.5) with each plan conditioned on its predecessor to maintain continuity. Core relationships (e.g., parents) remain fixed, while new acquaintances are gradually introduced to reflect the evolving context.
Hierarchical Decompositions The main seed is decomposed into ten sub-seeds, each represent- ing a distinct topical and temporal segment. Together, these sub-seeds span the full storyline (e.g.. an international trip: first three for preparation, next five for trip events, final two for reflections). Similar to sequential expansion, the user's core relationships remain constant, while new acquain- tances are introduced to reflect the evolving context. These ten sub-seeds are generated using the prompt in Listing 29 (Appendix H conditioned on the main seed, profile, and timeline.
Each conversation plan is assigned explicit topical and temporal houndaries-encomd seed-to avoid redundancy and ensure sub-themes appear in the right narrative stage. For coher- ence, the LLM conditions on summaries of prior plans and future seeds when producing a new plan, allowing anticipation of upcoming events (e.g., reserving tickets for travel dates). This pro- cedure is implemented in line of Algorithm [ (Appendix B.3.5). Plans are generated using the prompt in Listing (Appendix H conditioned on the main seed, current sub-seed, number of sub- plans, narrative set. user profile, core and new relationships. preceding and subsequent sub-seeds, previous plan, a summary of earlier plans, current sub-seed index, and a binary flag for the first plan (triggering user introduction). Since initial plans may not sufficiently test three key memory abilitics- contradcioon resolution, information update, and instruction followinggoooooo-oooo stage augmentation: first generate the base plan, then use GPT-4.1 (Listing (7) to augment each sub-plan with three targeted bullet points. Performing augmentation separately improves coverage and fidelity. The refinement follows the prompt in Listing 27 (Appendix H). which takes plan as input and outputs the revised version. This stage corresponds to the first module in Figure [I] which forms the first step of the overall data-generation pipeline. The detailed process for plan generation is reported in Appendix B.3.2
in the
apply a two-
2.2.2 USER UTTERANCE GENERATION
Once conversation plans are constructed, user utterances are synthesized from the sub-plans. Each sub-plan contains M bullet points, which are divided sequentially into K contiguous batches of equal size. Batching narrows the LLM'' focus, reducing repetition and low-quality outputs that can occur when conditioning on the entire sub-plan. For each batch, the LLM generates I user questions (line 6 in Algorithm 2 in Appendix B33.5) using the prompt in Listing 32 (Appendix H), conditioned on the conversation seed, the current batch, preceding batches, and context from earlier lub-plans. Each generated user question constitutes user turn in the dialogue, ensuring coherence and continuity across extended conversations. Values of K and I are manually specified based on domain and target conversation length to meet the token budget, with configurations reported in
4

---

Published as a conference paper at ICLR 2026
Table 6 (Appendix B This provides fine-grained control over user interaction density, preventing muder-generation or redundancy. To balance quality and cost, question generation uses the open- source LLaMA-333 70B model (AI 2024), which produces high-quality outputs efficiently as the backbone I.M.. This user-utterance construction aligns with the second stage in Figure The details of this procedure for user utterance generation are provided in Appendix B.3.3
2.2.3 ASSISTANT UTTERANCE GENERATION
Assistant-side responses are generated iteratively in a role-playing setup. where one LLM assumes the assistant role and another the user role. For each sub-plan, the assistant LLM is conditioned on the conversation seed (Section 2.2.T). prior sub-plans, summary of the last M turns, and compressed summary of earlier ones (using the prompt in Listing (37 in Appendix [B; for 10M- token conversations, additional summaries of prior plans are provided. The assistant first generates a response to the user's ' most recent question (line i Algorithm B Appendix
which is an- alyzed by question-detection detection module (line 11 in Algorithm in Appendix B.3.5] using the prompt in Listing 35 Appendix B) to determine the presence of counter-question. If detected, the response passed to the user L.LM, which generates a contextually consistent reply based on the current and prior sub-plans, relevant history, and conversation summaries (using the prompt in Listing 38 in Ap- pendix B line 14jin Algorithm in Appendix [B3.5). This loop continues until no further assistant questions detected or the threshold 81 2 is reached, balancing realism and avoiding infinite cycles. In addition, a follow-up detection module (line i Algorithm 3 Appendix [B3.33 using the prompt in Listing 36 in Appendix [B) evaluates whether a clarifying elaborative user follow- up is warranted, based on factors such as subject complexity, ambiguity, or incomplete responses. When required, the module generates follow-up query conditioned on the seed, current and prior sub-plans, the most recent M turns, and earlier summaries (using the prompt in Listing 39 in Ap- pendix B), which is then passed back to the assistant LLM. The number of follow-up exchanges limited by threshold 82 = 2, analogous to 81. Together, these modules yield dialogues with bidi- rectional dynamics, contextual referencing, and realistic clarifications, approximating human-Al interactions. This assistant-sich generation maps to the third module in Figure [] The details of this procedure are provided in Appendix B.3.4]
2.3 PROBING QUESTIONS GENERATION
After constructing conversations, we generate probing questions to evaluate memory abilities. The pipeline combines automated synthesis with human validation: an LLM first produces candidate probes, which annotators review to select valid ones. Probes are derived from both the conversation plan and chat to ensure each targets a specific ability, is grounded in dialogue turns, and includes explicit provenance. The process begins by passing the plan to GPT-4.1-mine (OpenAl 20256), which selects candidate bullet points conditioned on the ability under evaluation. For example, mhowledge-updatt probes require bullet pairs encoding an initial fact and its later revision, while summarization and event-ordering probes span multiple bullets. Each bullet is linked to its corre- sponding user and assistant turns through indices introduced during user-assistant turn generation, enabling retrieval of the precise dialogue segments in which the content was created. Candidate bullet selection is performed using prompts [ one per memory ability. For abstention, candi- date selection is unnecessary; probes are created directly from the plan using the prompt shown in Listing 14 Appendix[[]]
Given the selected bullet points and aligned dialogue snippets, GPT-4.1-min generates the prob- ing question, a candidate answer, and source identifiers citing the specific messages containing the answer. For 10M-token dialogues, candidate selection and synthesis are performed with a sliding window across the ten interlocking plans, processing a limited number at time to preserve topical locality and scalability. Probe generation uses prompts 10 19 for each memory ability, mapping candidate bullet points and contexts into fully formed questions. Finally, a human evaluator re- views the generated candidates and selects those that are valid and consistent with the conversation. Samples of probing questions are provided in Appendix D items
10)
2.4 EVALUATION
We evaluate LLMs on the probing questions using nugget evaluation, a common approach for long- form text assessment (Pradeep et al. 2024 2025). Each probing question is manually validated:
5

---

Published as a conference paper at ICLR 2026
User-assistant chat
Key-Values
Vector database
Key-Value
Embedding Model
Episodic memory
Turn M-1
Scratchpad
Buffer
Last N peirs
Scratchpad Generator
N
Summanizer
Semantic memory
Last pair Turn M
total takens
Working memory
Retriver Model
Scratchpad
Keep
Query
Filtering Nolse
memory
Response
Figure 2: Overview of the LIGHT framework. The system combines (i) episodic retrieval, (ii) seratchpad and (iii) working memory buffer. At inference, relevant items from the index and scratchpad, together with the full working memory, are integrated to generate the final response.
invalid or unsupported questions are discarded, and minor inconsistencies are corrected. From the validated set, two questions per memory ability chosen for each conversation, yielding 20 prob- ing questions per conversation. Rubric nuggets are then derived for each question. A nugget is atomic, self-contained criterion that system response the ideal reference answer into minimal semantic units, ensuring each nugget is both atomic and self-contained. System responses are scored against these nuggets by an LLM judge (Listing 20 Appendix H). which assigns 0 (unsatisfied), 0.5 (partially satisfied), or (fully satisfied). Scores are averaged across nuggets to produce ability-level metrics. This nugget-based based procedure applies to nine memory abilities; the exception is event ordering. where quality depends on both recall and correct sequence. We evaluate event ordering using the Kendall tau-b coefficient (Kendall) I945), which considers both order and presence. To apply this metric, an LLM equivalence detector (using the prompt Listing 2T Appendix [H) aligns events in system responses with nuggets, outputting yes if two snippets denote the event/topic and no otherwise. Kendall tau-b is then com- puted over the aligned sequences, capturing both recall and ordering fidelity. Examples of nugget construction for each memory ability are provided in Appendix
satisfy. Annotators decompose
LIGHT: IMPROVING MEMORY CAPABILITIES LLMS
Inspired by research in human cognitive science (Sridhar et al., 2023 Binder & Desail 2011), hu- mans employ two primary mechanisms for remembering and using knowledge: episodic memory, the ability to recall specific personal experiences along with their context, and working memory, the capacity to retain and manipulate information about recent events over short periods. In addi- tion. maintaining notes on scratchpad provides an external record that supports long-term recall and later retrieval. Since answering questions in long-context conversations similarly requires in- tegrating past experiences and accumulated knowledge, we introduce method that emulates these strategies by combining episodic recall, short-term working memory, and an external scratch-pal pad mechanism.
Overview: An overview of method is shown in Figure 2 Given question about versation [, where [T] the total number of turns, the framework first queries a re- trieval model R to obtain & relevant segments from Т. simulating recall from episodic memory: E R(x, e k. T). Next, the most recent z dialogue pairs of the conversation are selected to form the working memory, W (177)1)) In parallel, pre-constructed scratchpad. S S contains up m

---

Published as a conference paper at ICLR 2026
salient notes. A filtering function f retains only the items pertinent to yielding S₂ = f( ( x). Finally, the LLM क generates the answer by conditioning on the question and these three mem- ory components, y (x, E, W. S2) using the prompt shown in Listing 44 in Appendix [H] The remainder of this section details the construction and logic of each component in this pipeline.
3.1 RETRIEVAL. FROM THE CONVERSATION
After each user-assisiin assistant turn (Figure top), we apply Qwen2.5- 32B-AWQ (Team 2024) with the prompt in Listing 40 (Appendix HD extract key-value pairs and a summary of the interaction. Keys represent entities and values capture attributes or descriptive details. providing fine-grained, event-level indices analogous hippocampal memory traces (Tyler & DiSccennal 1986). These key-value pairs and summaries are embedded using the BAAl/bge-- small-en-vI.5 embedding model (or Artificial Intelligence, 2023) and stored in vector database as keys, while the original dialogue segments are kept as values to ensure faithful grounding.
Indexing the Conversation:
To retrieve information from the conversation as episodic memory, we embed the question x using the same embedding model and compare it against the stored keys in the index, and the original dialogue segments corresponding to the top k: nearest neighbors are returned.
Retrieval from the Index:
3.2 SCRATCHPAD FORMATION AND UTILIZATION
Construction: In addition to episodic memory (Figure 2 middle pathway), we build a higher-level representation that preserves information beyond individual dialogue events. It integrates semantic knowledge (facts and concepts), autobiographical details (life events), prospective memory (future intentions). and contextual metadata (time, place, acquisition context) (Binder Desai 201 1). For each dialogue pair, we use Sivn2.5-322-AWW
with the prompt in Listing 41 (Appendix H to rea-
son over the current and preceding turn and extract salient content. The resulting "scratchpad" is iteratively merged with earlier versions; once content exceeds a 30K-token threshold-substantiaee substanially shorter than the raw conversation-ie is compressed into 15K-token summary by GPT--411nang us- ing the prompt in Listing 42 This process maintains efficiency and long-term coherence, analogous to the gradual abstraction of semantic memory in humans. Unlike the episodic index, the scratchpad not stored in a retrieval database but is provided directly contextual input during inference.
Filtering Scratchpad (function f): During inference, the scratchpad is selectively filtered with respect the question. is first divided into semantically coherent chunks using semantic chunking. Each chunk is evaluated by Dwen2.5-32B-AWO with the prompt in Listing 43 (Appendix H). which assigns binary relevance label (yes/no). Only the chunks judged relevant are retained, producing a condensed representation of scratchpad that is passed to the response generator.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Baselines: We evaluate our approach against two types of baselines: long-context LLMs and RAG method. For long-context LLMs, the entire conversation history is provided, followed by the probing question. We include two proprictary LLMs (GPT-4.1-name) Gemini-2.0-flash, both IM context), and two open-source models (Qwen2.5-32B-AWQ -lanr---Maerrick-fp)) For long- context experiments, Qwen2.5-32B-AWW
is evaluated with 128K context length, while for the RAG baseline and our proposed method a 32K context length is used. At the 10M-token, since none of the four models support this length, they evaluated on the largest recent dialogue segment fitting their window [ For RAG baselines, each user-assistant turn pair is treated as a document, embedded and stored vector database. At inference, the top five most similar documents are retrieved and passed to the LLM using the prompt in Listing 44 (Appendix H).
2 Ssemant t i cChunker in LangChain is used, which segments into variable-length passages based on semantic rather than fixed token windows. "mmnng available models, only 1loma-4-Scour supports 10M-token context windows; however, due to its extreme computational requirements, we were unable include it in our experiments.
7

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of different LLMs and methods across conversation lengths and memory abil-
ities using the created benchmark. Methods with the best performance per evaluation are bolded.
Length
Memory
Ability
Vanilla
Owen 2.5
R:30
Durs
Vanilla
Llama Maverick
Dars
Vanilla
Gemim Flash
RAG
Chirs
Vanilla
GPT-4. I-nano
R:3G
Durs
Absiention
0,300
0.650
0.475
0.200
0.800
0.660)
0.800
03.00]
0.875
0.475
0.800
0.575
Contradiction Resolution
0.031
0.025
0.037
0.025
0
0.031
0.006
0.050
0.018
0.012
0.018
0.031
Event Ordering
0.192
0.201
0.166
0.181
0.169
0.177
Information Extraction
0.425
0.338
0.479
0.510
0.392
0.518
0.333
0.341
0.464
0.273
0.362
0.538
Instruction Following
0.400
0.375
0.362
0.412
0.375
0.412
0.275
0.287
0.362
0.425
0.350
0.400
100K
Knowledge U'pdate
0.437
0.275
0.362
0.300
0.350
0.450
0.125
0.325
0.300
0.275
0.375
0.375
Multi-Hop Reaconing
0.222
0.203
0.152
0.353
0.200
0,148
0.225
0.178
0.263
0.365
Preference Fallowing
0.554
0.339
0.566
0,450
0.512
0.625
0.330
0.416
0.462
0.437
0.550
0.625
Summarization
0.128
0.074
0.232
0.065
0.111
0.238
0.018
0.093
0.139
0.028
0.083
0.202
Temporal Reasoning
0.112
0.187
0.112
Average
0,280
0.269
0.311
0.240
0.323
0,358
0.242
0.280
0.294
0,239
0.309
0.345
Absicution
0,314
0,728
0.571
0,185
0.785
0.628
0.714
0,800
0.685
0.557
0.828
0.600
Contradiction Resolution
0,053
0.028
0.042
0
0.017
0.025
0
Event Ortering
0.185
0.23]
0.244
0,209
0.186
0.197
0.215
0.189
0,200
0.188
0.180
0.204
Information Extraction
0,166
0.400
0.506
0.608
0.402
0.535
0.469
0.343
0.478
0.142
0.382
0.491
Instruction Following
0.304
0,350
0.295
0.403
0.447
0.390
0,133
0.334
0,280
0.214
0.286
0.342
500K
Knowlodge Updaie
0.111
0.226
0.264
0,171
0,180
0,223
0.107
0.288
Multi-Hop Reasoning
0.125
0.187
0.214
0.219
0.313
0.350
0.198
0.135
0,157
0.070
0.233
0.266
Preference Following
0.567
0:477
0.571
0.560
0.525
0.623
0.579
0.427
0.533
0,450
0.577
0.684
Summarization
0.137
0.187
0.344
0,266
0.197
0.373
0.136
0.165
0,250
0.109
0.184
0.334
Temporal Reasoning
0,035
0.114
0.121
0,064
0.078
0.190
0.150
0.078
0,0W2
0.057
0.161
0.154
Average
20200
0.291
0,316
0.283
0.330
0,359
0.257
0.261
0.292
0.191
0.311
0.335
Abstention
0.342
0.650
0,500
0.221
0.742
0.435
0.642
0.750
0.735
0.492
0,778
0.678
Contradiction Resolution
0.035
0.035
0.021
0,046
0.828
0,042
0.010
0.028
0.007
0.050
0.028
0.021
Event Ordering
0,183
0.195
0.200
0.214
0.179
0.193
0.190
0.198
0.185
0.191
0.179
0.211
Information Extraction
0.138
0.407
0,366
0,489
0.431
0.474
0,574
0.380
0.341
0.153
0.399
0.410
Instruction Following
0,383
0.133
0.120
0.380
0.226
IM
Knowlodge Updaic
0.064
0.378
0.357
0.164
0,342
0.414
0.107
0,278
0
0.150
0.342
0.392
Multi Multi-Hop Reasening
0. 0.102
0.163
0.209
0.174
0.245
0.270
0.083
0.134
0.147
0.091
0.293
0.278
Preference Fullowing
0.186
0.191
0,551
0.535
0.514
0.610
0,273
0.170
0.472
0.135
0,513
0.576
Summarization
0.122
0.157
0,207
0,145
0.315
0,091
0,125
0,224
0.060
0.152
Temporal Reasoning
0.073
0.078
0.154
0.097
0.107
0.176
0.104
0.057
0.085
0.061
0.064
0.107
Average
0.193
0.285
0.309
0.259
0.307
0.336
0.199
0.27
0,284
0.191
0.302
0.336
Abstention
0,250
0,030
0.450
0,650
0,450
0.650
0,400
Contradiction Resolution
0.050
0.000
0.012
0,025
0.000
0.000
0.000
0.025
0.000
0,000
0.012
0.025
Event Ordering
0,150
0.221
0.197
0.190
0.220
0.176
0.220
0.266
0.193
0.215
0.201
0.173
Information Extraction
0,100
0.350
0,050
Instrutton Following
0.175
0,200
0,350
0,250
0,350
0.500
0,025
0.075
0.175
0,250
10M
Knowledge Update
0,100
0,300
0.275
0.100
0.375
0.525
0.050
0.325
0,200
0,050
0.325
0.300
Multi-Hop Rensoning
0,125
0.050
0.125
0,000
0,075
0.125
0.000
0.125
0.125
0.012
0.191
0.135
Preference Following
0.241
0.483
0.175
Summarization
0.114
0.106
0.220
0.065
0.053
0.277
0.000
0.045
0.136
0.020
0.063
0.179
Temporal Reasening
0,000
0,000
0.000
0,000
0.025
0.025
0.025
0.025
0.075
0,050
0.000
0.025
Average
0.133
0,238
0.266
0,216
0.109
0
0.226
For inference, we use Nucleus (Holtzman et al. 2020) with temperature O, ex- cept for conversation plan, user-turn, and assistant-turn generation, where temperature is 0.1 to encourage diversity. All open-source LLMs are served via VLLM for efficient inference. For Llama3,3-70B, we set the maximum output length to 6K tokens during user-turn generation, while for other LLMs we adopt their default maximum output length. For experiments involving both the RAG baseline and our proposed method, we employ FAISS as the vector database (Douze al., 2024).
Inference Setup:
For dense retrieval, we use the embedding model BAAlfbge-smalleen--155 (Xiao et al. 2023
4.2 EMPIRICAL FINDINGS
Across all four conversation lengths (100K-10M tokens), our method consistently Main Results:
outperforms both long-context LLMs and RAG baselines (Table D. At shorter contexts (100K), we observe strong gains, such as +49.1 for Llama-4-Maverick and +44.3% for GPT-4.1-nane over long-context baselines, showing that structured memory helps even when full history can be processed. The benefits grow with context length: at IM tokens. improvements reach +75.9% for GPT-4.1-nane and +60.1 % for Qwen2.5-32B At 10M Lakens-wheer no baseline natively supports the full context-one our method achieves dramatic improvements, including + +155.7% for Llama-4- Maverick and +107.3% for GPT-4.1-aaoo The only exception Gemini-220-ffaa at 10M, where method surpasses the long-context baseline (+57.3%) but slightly trails RAG, likely due to model-specific retrieval behavior. Overall, these findings underscore the scalability and robustness of our framework across diverse architectures and extreme context lengths.
When evaluated across the ten memory abilities, our method shows the largest relative gains in summarization (+160.6%), multi-hop reasoning (+27.2%) and preference following (+76.5%). Strong improvements are also observed in information extraction (456.7%) instruction following (+39.5%) ()), and temporal reasoning (+56.3%). These results highlight that our method is particularly
8

---

Published as a conference paper at ICLR 2026
100K
500K
10M
0.40
0.25
0.30
0.31
0:31
5.33
4.9
Perfo (8)
9.26
0.20
Lomponent
Rase
0.15
Wio Episodic Memory
0.10
Scratchpad
Working Mamory
0.05
Wio Noise Filtering
0.00
Figure 3: Ablation study illustrating the contribution of each component in LIGHT (retrieval,
scratchpad, working memory, and noise filtering) across different conversation lengths.
effective for tasks requiring long-range recall and integration of dispersed information. In contrast, all methods--inlludng ours-perform strongest in abstention and weakest contradiction resolu- tion, indicating that contradiction detection remains challenging open problem.
Ablation: We conduct an ablation to assess the role of each cpmponenttepissoiic memory, scratchpad, working memory, and noise filtering- across conversation lengths (Figure 3). At 100K, removing retrieval does not change performance and it remains steady, since the scratchpad alone suffices, while removing scratchpad or noise filtering reduces performance (-1.1%, -2.2%). Work- ing memory also degrades results here (-1.6%). At 500K, removing any component reduces per- formance except working memory, where removal enhances performance very slightly. At 1M, retrieval, scratchpad, and noise filtering remain beneficial, but removing working memory slightly improves performance, again reflecting its limited usefulness when few questions depend on the most recent turns. By 10M, all components are essential, with removals leading to large drops (-8.5% for retrieval, -3.7% for scratchpad, -5.7% for working memory, -8.3% for noise filtering). Overall, the ablations show that each module contributes increasingly context length grows, and the full architecture consistently achieves the hest performance. Detailed results across all memory abilities are provided Table
Effect of Retrieval Budget: We exam-
ine the effect of retrieval budget (K), test- ing 5, 10, 15, and 20 documents (Fig- uref4). Performance consistently improves when increasing K from 5 to 15, with the best results at K=15 (+7.39%. +10.75%, +6.79%, and +6.3% at 100K, 500K, 1M, and 10M). Increasing further to K=20 slightly degrades performance, likely due to noisy context. Results at K=10 are mixed-helpful 100K, 500K and 1M but harmful at 10M-indicating additional documents sometimes add noisy informa- tion. Full results across memory abilities are shown in Table 9 We also conducted complementary experiments analyzing the effect of retriever choice, where we ob- served that at 100K, 500K, and 1M token lengths, using a sparse retriever improves performance, whereas at 10M tokens, the dense retriever achieves better results. The full results and discussion provided in Appendix C.2
0.350
0,00
Chee
107K
500K
0.775
Retrieval Budges
Figure 4: Effect of varying retrieval budget (K) on performance. The plot shows how the number of re- trieved documents shapes the balance between recall and noise, highlighting different behaviors short and long conversation lengths.
Case Study A case study demonstrating the usefulness of the scratchpad is provided in Ap- pendix

---

Published as a conference paper at ICLR 2026
Additional analyses embedding choice, indexing setup, and supplementary baseline are pro- vided in Appendix C,3
Human Evaluation: We conducted a human evaluation to assess the quality of the generated conversations. Three dimensions were considered: Coherence and Flow, Realism, and Complexity and Depth, each rated on 5-point Likert scale (1 = lowest, highest). The average scores across all conversations were 4.53, 4.57, and 4.64, respectively, indicating consistently high quality. Details of the annotation protocol and inter-annotator agreement provided in Appendix [8.2] We further present a qualitative error analysis characterizing LIGHT 's failure modes across memory abilities in Appendix G
5 RELATED WORK
The detailed related work is provided in Appendix here we present a concise summary.
Context windows of LLMs have expanded dramatically, from early limits of 512-2K tokens (GPT-
2/3; (Radford et al. 2019 Brown et al. 2020 128K-IM (Claude-3, GPT-4-Turbo, Gemini 2.0;
[DeepMind] 2025 Anthropic 2025 OpenAl 2025a)) and even 10M (Llama 4; (Meta-Al 2025
This growth is driven by advances in effiiintt attention (sparse, linear, memory-optimized kemels;
Beltagy et al., 2020: Wang et al. 2020 Dao et al. 2022)). ). improved positional encodings (relative, rotary with scaling, ALiBi; (Dai et al. 2019 Peng et al. 2023b)), long-context training strategies (continued-training: curriculum learning: (Xiong et al. 2023 Ding et al., 2024)), and inference
optimizations such as paged attention, KV-cache compression, and distributed attention (Kwon et al.)
2023 Zhang et al. 2023) Li et al. 2024 Liu et al. 2023). Such capabilities are especially valuable for applications involving conversational histories, the main focus of our work.
Beyond expanding context windows, models incorporate additional mechanisms for persistent mem- ory. These include recurrence and compression (Transformer-XX Compressive Transformer; (Dai
et al.
2019 Rae et al.
2019
2019)), state-space architectures (RWKV, Mamba, Hyena; (Peng et. [2023a]
Gu & Dao 2023 Poli et al. 2023)), ) external memory modules (Memformer, RETRO, RMT; (Wu
al., 20200] Borgeaud
al. 2022 Fan al. 2024)), context summarization (AutoCompressor,
[Chevalier el al. 2)23)), and retrieval-aus -aggented generation (REALM, RAG, HippoRAG; (Guu
al.) 20020 Lewis al.) 2020: Jimenez Gutierrez et al.. 2024)). These approaches complement
larger windows by enabling scalable and persistent long term reasoning.
Existing benchmarks such as DialSim, MSC, LoCoMo, MemoryBank, DuLcMon, PerLTQA. Long-
MemEval, and MemBench (Kim et al. 2024a] Xu et al. 2021 Maharana et al. 2024 Zhong et al.) 2024 Xu et al., 2022 Du et al. 2024 Tan et af. 2025) evaluate recall, temporal reasoning, and multi-session reasoning, but typically span narrow domains, exhibit shallow dependencies, and con- catenate separate user sessions to simulate long context, reducing realism. Recent work such as
MemoryCode (Rakotonirina et al. 2025) generates multi-session dialogues from template-driven instruction seeds to assess long-conter reasoning, but focuses on a single domain. Our benchmark instead scales to 10M tokens across diverse topics and introduces new tasks such as contradiction resolution, event ordering, and instruction following, generating coherent, single-user conversations that preserve narrative continuity for more faithful assessment of long-term conversational mem- ory.
CONCLUSION
This paper addresses the shortcomings of existing benchmarks for evaluating long-tern memory in conversational systems. We introduce scalable framework generate BEAM, a new benchmark with long, coherent dialogues (up to 10M tokens) and diverse memory probes. To improve LLMS performance, we develop LIGHT, cognitive-inspired framework combining episodic, working, and scratchpad memories. Our experiments show that while standard LLMs' performance degrades over long contexts, LIGHT provides substantial improvements, boosting memory performance by an average of 3.5%- 1..%%-1.6694 12.69%. By offering more robust evaluation and effective memory en- hancement technique, this work helps the development of more reliable long-context conversational systems.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Ross Mitchell is the Alberta Health Services Chair Artificial Intelligence in Health and is sup- ported by CIFAR, the University Hospital Foundation, the Alberta Machine Intelligence Institute (Amii), and the Canada Foundation for Innovation. Mohamed Abdalla is supported by CIFAR AI Chair. This research is supported by the Canadian Institutes of Health Research (FRF 196047). Car- rie Ye is supported by the CRAF (CI-24-0013). This research supported in part by the Center for Intelligent Information Retrieval, in part by NSF grant #2143434, in part by the Office of Naval Research contract and with support from Google.org. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the spon- sors.
Society Canada Clinician Investigator Award

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint (aXXxx:23330877222 2023.
Meta AL Llama 3.3 model cards and prompt formats. https:// www. 11ama.mmm...mmmmmmmmmmmmm..mmmmm com/docs/ 2024.
Anthropic. Claude 3 model card. Technical report, Anthropic PBC, 2024. URL https://
www-cdn. -
Model_Carde Claud__aaaa__e_ee 3.paf
Anthropic. Claude 4 model card (claude opus & sonnet 4). Technical report, Anthropic PBC, May
2025.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.055 150, 2020.
Jeffrey R Binder and Rutvik H Desai. The neurobiology of semantic memory. Trends in cognitive sciences, (5(11):527-55) 2011.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, rean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp. 2206-22644 2240. PMLR, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1991,,022 2020.
Zhiliang Chen, Xinyuan Niu, Chuan-Sheng Foo, and Bryan Kian Hsiang Low. Broaden your SCOPE! efficient multi-turn conversation planning for LLMs with semantic space. In The Thirteenth International Conference on Learning Representations, 2025. URL https://
openreview wewenrrveeweeee//eoeeeeeee www.:ee/fffooo mett//orrumm1033cg0333333
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. arXiv preprint arXiv:2305. 14788, 2023.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint 2rXiv:2009.14794, 2020.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv: 028600 2019.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in neural information processing systems, 15116344-16359, 2022.
Google DeepMind. Gemini 2.0 flash: A multimodal model with million token con- text window. hattps://coon google. wttm///eteeeeaaaggeeeaaaaaeeee/a 2025.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186 2019.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang. Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond 2 million tokens. arXiv preprint arXiv:2402. 13753, 2024.
12

---

Published as a conference paper at ICLR 2026
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv preprint arXiv:2401.08281, 2024.
Yiming Du, Hongru Wang. Zhengyi Zhao, Bin Liang, Baojun Wang, Wanjun Zhong, Zezhong Wang, and Kam-Fai Wong. Perltqa: A personal long-term memory dataset for memory classification, retrieval, and fusion in question answering. In Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN- 100), pp. 152-164, 2024.
Qihang Fan, Huaibo Huang, Mingrui Chen, Hongmin Liu, and Ran He. Rmt: Retentive networks meet vision transformers. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 5641-5651, 2024.
Chongzhou Fang, Ning Miao, Shaurya Srivastav, Jialin Liu, Ruoyu Zhang, Ruijie Fang, Asmita, Ryan Tsang, Najmeh Nazari, Han Wang, and Houman Homayoun. Large language models for code analysis: do iims really do their job? In Proceedings of the 33rd USENIX Conference Security Symposium, SEC '24 USA, 2024. USENIX Association. ISBN 978-1-939133-44-1.
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. Splade v2: Sparse lexical and expansion model for information retrieval. In Proceedings of the 45th In- ternational ACM SIGIR Conference Research and Development Information Retrieval, pp. 127-137, 2022.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 2023.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL h
//openreview. net wett:/ormm.dddyygeepp
Hamed Jelodar, Mohammad Meymani, and Roozbeh Razavi-Far. Large language models (Ilms) for source code analysis: applications, models and datasets, 2025. URL https://arxii... org/
abs/2503.17500
Bernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobi- ologically inspired long-term memory for large language models. Advances in Neural Information Processing Systems, 37:59532-59569, 2024.
Adam Tauman Kalai, Ofir Nachum, Santosh S Vempala, and Edwin Zhang. Why language models hallucinate. arXiv preprint arXiv:2509.0466 2025.
Maurice G Kendall. The treatment of ties in ranking problems. Biometrika, 33(3):239-255 1945.
Jiho Kim, Woosog Chay, Hyeonji Hwang, Daeun Kyung, Hyunseung Chung, Eunbyeol Cho, Yohan Jo, and Edward Choi. Dialsim: A real-time simulator for evaluating long-term dialogue under- standing of conversational agents. arXiv e-prints, pp. arXiv-2406, 2024a.
To Eun Kim, Alireza Salemi, Andrew Drozdov, Fernando Diaz, and Hamed Zamani. Retrieval- enhanced machine learning: Synthesis and opportunities, 2024b. uttpttps::/ http:://axxxv.orrr/.
abs bbss/407.. 12982
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princi- ples, pp. 611 611-626, 2023.
Philippe Laban. Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. Lims get lost in multi-turn conversation, 2025. URI.https:: /tttps://ariiii org abs 72005. 06120
13

---

Published as a conference paper at ICLR 2026
Kuang-Huce -uei Lee, Xinyun Chen, Hiroki Furuta, John Canny, and lan Fischer. A human-inspired reading agent with gist memory of very long contexts. arXiv preprint arXiv:2402.09727, 2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmente gener- ation for knowledge-intenn intensive nlp tasks. Advances in neural information processing systems, 33: 9459-9474, 2020.
Minghan Li, Miyang Luo, Tianrui Lv, Yishuai Zhang, Siqi Zhao, Ercong Nie, and Guodong Zhou. A survey of long-document retrieval in the pim and Ilm era, 2025. URL https: //arxiv. arg/abs/2500 httpb///50990755555
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Lim knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947-29770,, 2024.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context. arXiv preprint arXiv:2310.0188 2023.
Adyasha Maharana, Dong-Ho Lee. Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of Ilm agents. arXiv preprint 1rXi::2402.17753. 2024.
Meta-AI. The llama herd: The beginning of a new era of natively multimodal ai innovation. Meta AI Blog, April 2025. URL https://at wamta.com/ba..
1ama-
Ha Thanh Nguyen, Wachara Fungwacharakorn, May Myo Zin, Randy Goebel, Francesca Toni, Kostas Stathis, and Ken Satoh. Lims for legal reasoning: A unified framework and future perspectives. Computer Law & Security Review, 58:106165, 2025. ISSN 2212-473X. doi: 106165. URL rt com/
sciencedirect......cccc.o.oc
Beijing Academy of Artificial Intelligence. Buni/hge-smallen-v11.. Hugging Face model, 2023. MIT License; embed-
Hugging
http:///wgggnaaaa.....o//////ee/ ding model.
OpenAl. Introducing gpt-4.1 in the api. http:://ppeaa..com/////////444// 2025a.
OpenAl. I-min model card.
got-4-1-mini] 2025b. Accessed: 2025-09-11.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung. Matteo Grella, al. Rwkv: Reinventing mas for the transformer era. arXiv preprint arXiv:2305.13044 2023a.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.0007/ 2023b.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference Machine Learning, pp. 28043-28078 PMLR, 2023.
Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. Initial nugget evaluation results for the trec 2024 rag track with the autonuggetizer framework. arXiv preprint arXiv:2411.0960 2024.
Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, and Jimmy Lin. The great nugget recall: Automating fact extraction and rag evaluation with large language models. In Proceedings of the 48th International ACM SIGIR Conference Research and Development Information Retrieval, pp. 180-190, 2025.
14

---

Published as a conference paper at ICLR 2026
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. 12409, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, et al. Improving language under- standing by generative pre-training: 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint
2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-6)) 2020.
Nathanaël Carraz Rakotonirina, Mohammed Hamdy, Jon Ander Campos, Lucas Weber, Alberto Testoni, Marzieh Fadaee, Sandro Pezzelle, and Marco Del Tredici. From tools teammates: Evaluating Ilms in multi-session coding interactions. arXiv preprint arXiv:2502. 13791, 2025.
Alice Rueda, Mohammed S. Hassan, Argyrios Perivolaris, Bazen G. Teferra, Reza Samavi, Sirisha Rambhatla, Yuqi Wu, Yanbo Zhang, Bo Cao, Divya Sharma, Sridhar Krishnan, and Venkat Bhat. Understanding Ilm scientific reasoning through promptings and model's explanation on the an- swers, 2025. URL
Alireza Salemi and Hamed Zamani. Learning to rank for multiple retrieval-augmentt mod- cls through iterative utility maximization. In Proceedings of the 2025 International ACM SI- GIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR), ICTIR '25, pp. 183-193, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400718618, doi: 10.1145/3731110..74445 0731120.374452
URL
Alireza Salemi, Chris Samarinas, and Hamed Zamani. Plan-and-refine: Diverse and comprehensive retrieval-augmennte generation, 2025. URL http::::/aaxx..oor/obbb///////// 07794
Sruthi Sridhar, Abdulrahman Khamaj, and Manish Kumar Asthana. Cognitive neuroscience per- spective on memory; overview and summary. Frontiers in human neuroscience, 17:1217093, 2023.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 127063,, 2024.
Haoran Tan, Zeyu Zhang, Chen Ma, Xu Chen, Quanyu Dai, and Zhenhua Dong. Membench: Towards more comprehensive evaluation on the memory of 1lm-based agents. arXiv preprint arXiv:2506.2160S. 2025.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. htRp::/:: //wp:///wwmmmmmmmmmmm.mmm.m github. farbub..o/blog// http://///ooggg/...ttcom qwen2.wn... 55]
Timothy J Teyler and Pascal DiScenna. The hippocampal memory indexing theory. Behavioral
neuroscience, 100,2):147, 1986.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang. and Hao Ma. Linformer: Self-attention with linear complexity. anXiv preprint wwXiv:2006.04760 2020.
Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. Longmemeval: Benchmarking chat assistants on long-term term interactive memory. arXiv preprint arXiv:2410. 10813, 2024.
15

---

Published as a conference paper at ICLR 2026
Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, and Zhou Yu. Memformer: A
transformer for sequence modeling. arXiv preprint arXiv:0010.6889,,
2020.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309. 100.3999..009, 2023.
Jing Xu, Arthur Szlam, and Jason Weston. Beyond goldfish memory: Long-term open-domain conversation. XrXiv preprint arXiv:2107.07567, 2021.
Xinchao Xu, Zhibin Gou, Wenquan Wu, Zheng- Yu Niu, Hua Wu, Haifeng Wang, and Shihang Wang. Long time no see! open-domain conversation with long-term persona memory. arXiv preprint arXiv:2203.05797, 2022.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33: 33:17283-17297 2020.
Zhenyu Zhang, Ying Sheng. Tianyi Zhou, Tianlong Chen, Lianmin Zheng. Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient gen- erative inference of large language models. Advances Neural Information Processing Systems, 36:34661-34710 2023.
Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19724-19731, 2024.
16

---

Published as a conference paper at ICLR 2026
A DETAILED RELATED WORK
Long-Context Large Language Models. The context window of LLMs has expanded from
120-2.048 tokens in early models (GPT-1/2/3, BERT, T5; (Radford et al., 2018 2019 Brown et al.)
2020 Devlin et al., 2019 Raffel et al.) 2020 to 128K-IM tokens in recent systems (Claude-3,
GPT--4Turbe Gemini 1.5 Pro. Gemini 2.0 Flash, Claude-4, GPT-4.1: (Anthropic 2024, [Achiam] ct al. 2023 Team et al., 2024 Deed 2025 Anthropic 2025 OpenAl| 2025a)), with some
reaching 10M tokens (Llama 4 Scout; (Meta-Al 2025)). This growth has been enabled by imno-
vations that address the quadratic cost of self-attention, including sparse mechanisms (Longformer, BigBird: (Beltagy et al., 20200 Zaheer et al. 2020), linear approximations (Linformer, Performer; [Wang et 2 et al., 2020p) ) and memory-efficient kernels (tlashAttention; (Dao et al. 2022)) Advances in positional encoding, such relative encodings (Transformer-Xll (Dai
ct al. ) rotary embeddings (RoPE: (Su et al.] (024)) with scaling methods (YaRN, NTK;
(Peng et al. 2023b)), and linear biases (ALiBi; (Press et 2021)) )), have extended usable context
lengths. Training strategies like continued pre-training and curriculum learning (e.g., LLaMA-2-
Long (Xiong et al., 2023). LongRoPE (Ding et al.] (024g) further expand capabilities, while in-
ference optimizations such as PagedAttentiio (Kwon et al. 2023). KV-cache compression (H20,
SnapKV: (Zhang et al. 2023 Li et al. 2024)) and distributed approaches (Ring Attention; (Liu
al. 2023 enable practical deployment at scale.
Long-Term Memory Methods. Researchers have developed approaches to enhance long- term memory beyond simply extending context windows. Architectural modifications include Transformer-Xe (Dai et al., 2019), which introduced segment-level recurrence, and Compressive Transformer (Rae et al.] 2019), which stored both recent states and compressed older information. State-space models such as RWKV (Peng et al., 2023a), Mamba (Gu & Dao 2023), and Hyena (Poli et al. 2023) replace attention with recurrent dynamics, allowing linear scaling and theoreti- transformers such as Memformer (Wu et al.) 2020). RETRO (Borgeaud al. 2022) and RMT (Fan et al./ 2024) add external memory slots for explicit storage and recall. Context compression offers an orthogonal strategy by summarizing past infor- mation rather than storing it verbatim, as AutoCompressor (Chevalier et al., 2023). which learns compact, information preserving representations to reduce token usage. Retrieval-augmente gen- eration (RAG) scales further by maintaining external knowledge stores: REALM (Guu et al.. (0220) and RAG (Lewis et al., 2020) pioneered dense retrieval, RETRO (Borgeaud et al. 2022) integrated retrieval into transformers, and HippoRAG (Jimenez. Gutierrez et al. 2024) incorporated structured knowledge graphs
cally unbounded memory.
information-presevving
Building on these foundations, we propose a novel retrieval-augmented method that shows substan- tial improvements over basclines in long-memory evaluation.
Long-Tewe Term Memory Benchmarks. Several benchmarks have emerged to evaluate long-term mem- ory capabilities in LLMss DialSim (Kim et al. 2024a) derives evaluation data from multiparty television scripts. producing dialogues extending to 350K tokens with naturalistic patterns but lim- ited topical diversity. MSC (Xu et al.
2021) introduces multisession human-assistant conversations testing memory across session boundaries, though with brief sessions and shallow dependencies. LoCoMo (Maharana et al., (024) presents 50 conversations averaging 9K tokens in 35 sessions, while MemoryBant Bank (Zhong et al. 2024) provides 300 sessions with 194 probing questions evaluat- ing recall and temporal reasoning.
DuLeMon (Xu et al. 2022) focuses on dialogue-level memory and forgetting curves, PerLTQA (Du et al. 2024) targets memory classification and retrieval, and LongMemEval (Wu et al., 2024) constructs multisession evaluations with 500 questions testing in- formation extraction and temporal reasoning. More recently. MemBench (Tan et al. 2025) evaluates the memory of M.L.Ma agents by assessing their performance on information extraction, multi- hop reasoning, knowledge updating. preference following, and temporal reasoning. Recent work such as MemoryCode (Rakotonirina et al.. 2025) generates multi-session dialogues from template- driven instruction seeds to assess long-context reasoning, but focuses on single domain.
As summarized in Table the existing benchmarks are largely based on concatenated short sessions with limited coherence, narrow personal and casual domains, and few memory abilities. They also lack realistic bidirectional interactivity. In contrast, our benchmark spans diverse domains, scales up resolution, event order-
to 10M tokens, and introduces three additional
17

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of our benchmark with existing long-term memory benchmarks. Memory abilities: IE = Information Extraction, MR = Multi-hop Reasoning, KU Knowledge Update, TR TTmmpora Reasoning, ABS Abstention, CR Contradiction Resolution, EO Event Ordering, IF
Instruction Following, PF Preference Following, SUM Summarization.
Memory Abilities
Benchmark
Domain
Chat Length
IE
MR
KU
TR
ABS
CR
EO
IF
PF
SUM
MSC (Xu
2021
Casual
~
X
DuLeMon Xu ct al.
2022
Casual
~1
X
K
X
X
X
X
X
X
MemoryBank Zhong et
2024
Personal life
~5K
/
X
X
X
X
X
PerLTQQ Du
2024
Personal life
N/A
X
LoCoMo Maharana
al
2024
Personal life
10K
X
X
X
X
DialSim
Kim et ml. 2024.
TV/iilm scripts
350K
/
X
V
X
X
X
X
LongMemEval Wu
2024
Personal life
115K, IM
V
X
X
MemBench
[Tan et
Personal life
100K
X
X
X
X
Multi-domain:
Coding, Math,
128K, 500K,
BEAM (This work)
/
/
/
/
/
Hcalth. Finance.
1M. 10M
Personal life...
ing, and instruction following yielddngg more comprehensive framework for evaluating long-term memory in conversational systems.
B BENCHMARK DESIGN
B.1 DATASET STATISTICS
Table summarizes the statistics of the generated dataset, including averages of user messages, assistant messages, assistant and user follow-up questions, and dialogue turns across different chat sizes.
Table 3: Statistics of the dataset. Reported values averages per chat in each chat size. User Messages and # Assistant Messages denote the average number of utterances from the user and assistant, respectively. Answer Assistant Questions is the number of times the assistant posed question that the user answered. # Followup Questions the number of follow-up questions asked by the user. # Turns refers the total number of dialogue turns.
# Answer Assistant # Followup Questions
Chat Size # User Messages
# Assistant Messages
# turns
Questions
128K
144
144
27
216
107
500K
544
544
79
51
416
IM
1067
1067
105
120
842
10M
10435
10435
1151
1528
7757
B.2
BENCHMARK QUALITY EVALUATION
To evaluate the quality of the generated conversations, we conducted a human assessment across all conversations. Two annotators rated each conversation three dimensions using 5-point Likert scale (1 = lowest, 5 = highest); Coherence and Flow, Dialogue Realism, and Complexity and Depth.
Annotation Setup and Procedure. We used an internal system for the annotators. The annota- tions were completed by two annotators with a computer science background. Human involvement follows:
Probing Question Validation. The annotators were first provided with guidelines and instructions how to perform the task, and training session was held. They went through
18

---

Published as a conference paper at ICLR 2026
calibration set together, completed multiple examples, and discussed their opinions
each example.
Rubric Design. The annotators given guidelines on how to design rubrics along with examples and templates to make the task easier and more unified. A training and calibration session was then held, during which annotators reviewed many samples together, asked questions, and received clarification on ambiguous and edge cases.
Conversation Quality Evaluation. For this task, the annotators were again provided with guidelines on how to evaluate the generated conversations and assign scores. Because the conversations are very long and impractical to read them full, annotators were structed to read the initial 25 dialogue turns to understand what the conversation is about and to become familiar with the theme and flow of the chat. They were instructed to check the follow-up questions proposed by both the user and the assistant, the adherence of the conversation to the conversation plan and user utterances, and whether any hallucinations unexpected behaviors occur. After that, they were asked to read several random turns between the initial 25 turns and the middle of the conversation to assess whether the conver- sation is progressing according to the conversation plan. They then had to read 25 dialogue turns in the middle of the conversation, followed by additional random turns between the middle and the end, and finally 25 dialogue way, annotators read different parts of each conversation and ensured good coverage.
the end of the conversation. In this
For every conversation, annotators rated the three quality dimensions described below using a 5- point Likert scale.
Coherence and Flow: Conversation continuity (each turn follows naturally from the pre- vious one), smooth transitions across topics and responses, and thread consistency without abrupt or jarring shifts.
Dialogue Realism: Naturalness of user queries (messages sound authentic), realistic pro- gression of topics over time, human-like interactions (appropriate clarifications, follow-ups, etc.), and believability of scenarios.
Complexity and Depth: Handling multi-layered, interconnected topics, progressive increase in difficulty, and demonstration of domain expertise when required.
Inter-Annotator Agreement. Before performing the full evaluation, annotators independently sessed set of 20 conversations. Agreement was measured using Cohen's 's Kappa. The observed agreement levels were:
Coherence and Flow: k= = 0.7044
Dialogue Realism: к 0.7391
Complexity and Depth: к = 0.7849
The aggregated evaluation results are reported in Table
Table 4: Conversation quality human evaluation results (I-5 scale). Higher is better.
Chat Size
Coherence and Flow
Dialogue Realism
Complexity and Depth
128K
4.4
4.55
4.35
500K
4.49
4.4
4.63
IM
4.66
4.54
4.6
10M
4.6
4.8
5
Average
4.53
4.57
4.64
B.3 BENCHMARK CREATION DETAILS
B.3.1
DOMAIN COVERAGE OF THE DATASET
To ensure broad coverage and realism, dataset spans diverse set of domains. The collection includes both technical and non-technical conversations, ranging from specialized domains such as coding, mathematies, financial investment and health to personal and social domains such as therapy,
19

---

Published as a conference paper at ICLR 2026
lifestyle, and trip planning. In total, designed 100 multi-tum chats distributed across 19 domains, each represented by a set of distinct titles that capture the thematic scope of the dialogues. The full list of domains and their associated chat titles is provided in Table
Table 5: Domains and associated chat titles in our dataset (100 total chats).
Domain Coding
Chat Titles
Designing a Large-Scale Retrieval-Augmente Generation (RAG) Sys- tem for Enterprise Search Creating a Self-Driving Car Simulation En- vironment Developing Multi-Agent Al Research Platform Build- ing a Multi-Language Al Chathot with Contextual Memory Develop- ing a Personalized News Aggregator with AI Summarization Creating an Autonomous Stock Trading Bot Implementing Custom Image Captioning Model Building a Multiplayer Online Game with Real- Time Physics Building Real-Time Chat Application with Node.js and Socket.io e Creating an Al-Powered Resume Analyzer with Python and NLP Developing Computer Vision App for Real-Time Ob- ject Detection e Creating a Restaurant Recommendation System Au- tomating Social Media Posts with Python Building Personal Budget Tracker Web App in Python and Flask Creating Command- To-Do List Manager in Go Developing a Weather Forecast App in JavaScript with Open Weather API Training Spam Email Classi- fier Using Python and Scikit-learn
Building a Portfolio Website with
HTML, CSS, and Bootstrap
Math
Partial Differential Equations (PDEs) in Depth Functional Analysis and nnfinite-Dimensional Spaces Solving Ordinary Differential Equa- tions (ODEs) Deep Dive into Number Theoory Advanced Probability and Combinatorics * Exploring Non-Fuclidean Geometry Studying Multivariable Calculus Diving into Analytic Geometry Developing Skills in Mathematical Induction Exploring Conic Sections in Depth Understanding Sequences and Series Mastering Basic Differential Calculus Exploring the Geometry of Triangles Understanding the Basics of Probability Mastering Algebraic Equations for Everyday Problem Solving Learning the Foundations of Trigonometry Mas- tering Fractions, Decimals, and Percentages
Writing Assistant & Learning
Building a Portfolio-Ready Resume that Passes Any Applicant Track- ing System Mastering the Art of Persuasive Academic Essay Writing Crafting a Standout Cover Letter for Competitive Job Markets De- signing Multi-Purpose Personal Statement for Global Opportunities Developing Self- Belf-Editing System for Lifelong Writing Improvement
Therapy & Emo-
Recovering from Workplace Burnout and Chronic Stress Healing Af- ter the Loss of Loved One e Overcoming Childhood Trauma and Re- building Self-Trust Coping with Post- Bost-Breakup Emotional Pain and Relationship Trauma
tional Support
Career & Profes- sional Development
Advancing from Mid-Level to Senior Leadership Roles Building a Powerful Professional Network from Scratch Landing Your Next Job: From Resume to Job Offer Designing a 5- Year Career Growth Plan Positioning Yourself for a Promotion
Financial Investment
Building a Long-Term Stock Market Investment Strategy e Getting Started in Real Estate Investing Navigating the World of Cryptocur- rency Creating a Balanced Investment Portfolio
Health & Wellness
Creating a Personalized Nutrition and Meal Planning System e Design- ing a Sustainable Fimess Routine Improving Sleep Quality for Better Health Understanding and Managing Chronic Illness Recognizing Symptoms and Seeking Medical Help Early
20

---

Published as a conference paper at ICLR 2026
Domain
Chat Titles
Relationship Fam- ily
Strengthening Communication Romantic Relationships Parenting Through Different Life Stages Navigating In-Law and Extended Fam- ily Relationships Rebuilding Relationships After Trust Has Been Bro- ken
Education & Learn- ing
Learning to Play a Musical Instrument from Scratch Mastering New Language for Real-Worde Communication Becoming a Skilled Pho- tographer Exploring Performing Arts: Acting, Theater, and Dance
Home & Real Estate
Buying Your First Home with Confidence Renting Home or Apart- ment Without Stress Selling Your Home for Maximum Value DIY Home Improvement and Repairs Making Your Home More Comfort- able and Functional
Lifestyle
Designing a Daily Routine That Boosts Productivity and Well-Being Building Healthy and Sustainable Lifestyle Habits Balancing Social Life and Personal Time
Cooking
Mastering Quick and Healthy Weeknight Dinners Baking Like a Pro Home Exploring Global Cuisines from Your Kitchen Cooking for Special Diets and Allergies Meal Prepping for the Week Ahead
Business & En- trepreneurship
Starting Business from Scratch Growing and Scaling Your Small Business Building a Successful Startup
Trip Planning
Preparing for a Week-Long Hiking and Camping Adventure in Patago- nia Organizing a Cross-Country USA Road Trip Planning a Cultural Immersion Trip to Japan Planning a Budget Backpacking Trip Across Southeast Asia Arranging Luxury Honeymoon in the Maldives
Sport
Soccer Playing, Watching, and Supporting the World's Most Popular Game * Basketball From Street Courts to the NBA Volleyball Indoor, Beach, and Competitive Play Hockey Ice, Field, and Global Competitions Tennis From Local Courts to Grand Slams
Event Planning
Planning Surprise 30th Birthday Party for a Close Friend Coordinat- ing a Destination Beach Wedding for 100 Guests Organizing a Week- end Community Food and Music Festival Planning Cozy Christmas Eve Dinner for Extended Family
Asking dation
Recommen-
Finding the Perfect Smartphone for Photography and Gaming e Choos- ing a Lightweight Laptop for Work, Travel, and Entertainment Select- ing a Must-Read Fiction Series for Winter Evenings Finding the Best Streaming Movies for a Family Weekend Choosing Comfortable and Stylish Sneakers for Daily Wear
Legal & Administra- tive
Filing for Marriage Beariage-BBsedd Green Card in the United States Creating a Legally Valid Will and Estate Plan Applying for a Patent to Protect a New Invention
Philosophical & Eth- ical Discussion
Deciding Whether to Use AI Automate Hiring My Company Considering Whether to Believe in and Live by the Idea of Free Will
21

---

Published as a conference paper at ICLR 2026
B.3.2 CONVERSATION PLAN GENERATION
A conversation plan serves the central scaffold of each conversation, providing coherent story- line that evolves chronologically. The process of constructing conversation plans is anchored by a seed that specifies the domain of the dialogue (e.g.. sports, finance, programming. mathematics), a ritle representing the high-level topic, and a theme that provides a more detailed instantiation of the title. The seed also includes a set of subtopics, which enumerate finer-grained subtopics and details to ensure topical diversity. However, title, theme, and subtopics alone are insufficient to support detailed and information-rich conversations. To enrich the narrative, we introduce narratives set that define the evolving aspects of conversation (c.g., career progression, goals, relationships). Each narrative is paired with descriptive details that specify its scope and trajectory.
In addition to the seed and narrative set, each conversation incorporates user profile, relationship graph, and an explicit timeline. The user profile includes attributes such as name, age, gender, location, profession, and personality traits. To avoid redundancy, personality traits are grounded in the Myers-Briggs Type Indicator (MBTI). Specifically, we randomly select six MBTI types, provide their descriptions, and instruct LLM to synthesize a composite trait profile, enabling the creation of 8,008 unique user profiles. Relationship graphs are then constructed, linking the main user to family members (parents, partner, children), friends, and acquaintances, subject to constraints (c.g., plausible age gaps) to preserve realism. The timeline specifies the temporal span of the conversation, defining the range between its beginning and end.
In order to generate titles and themes of the chats, target domains are first specified by human. Given these domains, GPT-4.1 (OpenAI 2025a) is prompted using the prompt shown in Listing 22 in Appendix H to produce candidate titles, themes, and subtopics. These candidates are refined by human to ensure topical diversity by removing the similar chat titles and selecting diverse chat titles. Finally, for each conversation,
generate 15-20 narratives using open-source LLaMA-3.3 70B [A] 2024) with the prompt shown in Listing 23 to save cost. In this prompt, given the conversation seed input, the LLM produces narratives that capture evolving aspects of the storyline, providing the backbone for constructing coherent conversation plans.
Conversation plans structured as sequence of N sub-plans, where each sub-plan corresponds to a distinct stage of the conversation. Each sub-plan contains a fixed number of M bullet-points, and each bullet-point is defined by a narrative and descriptive statement specifying how that narrative unfolds in the storyline. To maintain temporal coherence, each sub-plan also includes a time anchor specifying concrete date or period.
For conversations of sizes 128K, 500K, and 1M tokens, single conversation plan is generated, as shown in line 4 of Algorithm [ in Appendix B.3.5 The plan is produced by conditioning the LLM on the conversation seed, user profile, relationship graph, timeline, the number of sub-plans, the number of bullet points within each sub-plan and narrative set, using the prompt shown in Listing [4] in Appendix|i H The number of sub-plans is not fixed but varies with both the domain and the target conversation length, in order to adhere to the length budget. For instance, domains such as coding typically require fewer dialogue turns to reach the same token budget compared to more general
domains.
For 10M-token conversations, a single plan cannot adequately capture the scope and continuity required at this scale. To address this, we construct ten distinct yet interlocking conversation plans that together produce a coherent long-term narrative. While the process begins with main seed that defines the global topic and theme of the conversation, a single seed is insufficient for producing ten plans. Instead, we generate ten distinct conversation seeds-one for each plan-so so that the narrative can unfold across multiple stages. The procedure for deriving these seeds and the plans that follow-differs depending on the strategy. We propose two strategies for constructing them:
Sequential Expansion:
The conversation seed is used as the first seed in the sequence. The remaining seeds are generated to represent successive stages of the user's life, extending the sto- ryline chronologically. For instance, if the main seed concerns an international trip, the first plan covers the trip itself, the second covers the period after returning (e.g., job search), and subse- quent seeds correspond to later milestones. We generate these seeds using the prompt shown in Listing 28 which conditions on the main seed, user profile, and timeline to produce a sequence of temporally aligned seeds. Each conversation plan is then generated sequentially, with every plan conditioned on its predecessor to maintain continuity, as specified in line 12 of Algorithm
22

---

Published as a conference paper at ICLR 2026
The plans generated using the prompt shown in Listing 30 yielding temporally ordered series of interconnected narrative arcs. To maintain realism, the user's 's core relationships (e.g., parents, children, partner) remain fixed across plans, while new acquaintances are gradually introduced.
in Appendix B.3.5
Hierarchical Decomposition: Instead of extending the seed chronologically, the main seed is decomposed into ten sub-seeds, each corresponding to a distinet topical temporal slice of the overall storyline. Together, these seeds span the full narrative. For example, if the main seed concerns an international trip, the first three seeds may cover preparation steps (e.g., reservations, document gathering), the next five capture events during the trip, and the final two represent post-trip activities (e.g., reflections, recounting experiences). Like in Sequential Expansion, the user's core relationships (c.g., parents, children, partner) remain fixed across plans, while new acquaintances are gradually introduced. We generate these ten sub-seeds using the prompt shown in Listing 29 which takes the main seed, user profile, and timeline, and outputs ten derived seeds.
Each plan is assigned explicit topical and temporal boundaries to prevent redundancy or thematic overlap. ensuring that sub-themes unfold in the correct stage of the narrative. These boundaries are encoded in the conversation seed itself. For coherence, summaries of all prior plans are provided to the LLM when generating a new plan, allowing contextual references to past events. Moreover, when generating each plan, future seeds are also supplied, encoding their own topical and temporal boundaries. This design allows earlier plans to anticipate upcoming events with consistent references (e.g., booking tickets for the correct travel dates before the trip actually occurs). This strategy is implemented in line 20 of Algorithm [ in Appendix B.3.5 Conversation plans are generated using the prompt shown in Listing 31, which takes as input the main seed, the current sub-seed, the number of sub-plans, the narrative set, the user profile, core and newly introduced relationships, the preceding and subsequent sub-seeds, the previous plan, the summary of all previous plans, the index of the current sub-seed, and binary indicator specifying whether the plan is the first in the sequence (in which case the introduction of the user is included). The output is a fully specified conversation plan.
After the conversation plan is constructed, it is expanded into user-turn questions and subsequently assistant responses, yielding complete dialogues that can be used to evaluate memory abilities. How- ever, in its initial form, the plan may not include sufficient information to evaluate three critical mem- ory abilities: contradiction resolution, knowledge update, and instruction following. To address this, after the initial plan generation, we pass the plan to GPT-4.1 to generate high-quality plans and aug- ment each sub-plan with additional bullet points specifically designed to enable evaluation of these abilities. Importantly, this augmentation is performed in a second stage rather than during the initial plan generation, since incorporating such information directly in a single-pass generation leads to lower quality and less reliable coverage of these abilities. The augmentation is implemented using the prompt shown in Listing 27 which takes an existing conversation plan as input and outputs a revised version where each sub-plan includes three additional bullet points targeting these abilities.
B.3.3 USER UTTERANCE GENERATION
Once conversation plans are constructed, user turns are synthesized directly from them. Each sub- plan within conversation plan consists of M bullet-points, which are partitioned into K contiguous batches of equal size. Partitioning is performed sequentially, such that each batch corresponds to consecutive segment of the sub-plan. Partitioning necessary because conditioning the LLM on an entire sub-plan at once tends to yield repetitive or low-quality questions; batching mitigates this by narrowing the focus of generation. For each batch, the LLM produces I user questions (line of Algorithm in Appendix B.3.5) using the prompt presented in Listing 32 The model is conditioned on the conversation seed, the current batch specification, preceding batches within the same sub-plan, and contextual information from earlier sub-plans. This setup ensures that generated questions remain grounded in prior context, yiclding conversations that are coherent and continuous over extended spans.
The values of K and I vary depending on the domain and the target conversation length, in order to adhere to the overall length budget. We specify the values for K and I manually. The specific configurations of K and I across domains and conversation sizes are reported in Table 6 This provides fine-grained control over the density of user interactions and helps prevent both under-
23

---

Published as a conference paper at ICLR 2026
generation and excessive redundaney. Additonally, to better capture domain-speciffr conversational patterns, we incorporate domain- -enain-specific features during question generation:
Programming: To reflect realistic developer- aaveloper-assiitantt assistant interactions, incorporate questions that involve sharing code snippets. These include (i) buggy code requiring debugging assistance, (ii) correct code seeking optimization, and (iii) natural language descriptions of desired functionality for which code is requested. We use the prompt shown in Listing 33 to generate questions specific the programming domain.
Mathematics. To capture authentic problem-solving dynamics, we incorporate questions that involve sharing mathematical work, requesting corrections, asking for the next logical step in solution, or introducing problems to be solved. We use the prompt shown Listing 34 to generate questions specific to the mathematics domain.
To reduce computational cost while maintaining generation quality, question generation is per- formed using the open-source LLaMA-3.3 70B model (AI. 2024), which produces high-quality questions.
B.3.4 ASSISTANT UTTERANCE GENERATION
After generating user-side questions, assistant-side responses are generated in an iterative, role- playing framework where one L.M assumes the assistant role and another assumes the user mll. For each sub-plan, the assistant LLM is conditioned on the seed as explained in Section [.2.1] prior sub-plans of the conversation plan, a summary of the most recent M dialogue turns, and compressed summary of older turns (generated using the prompt shown in Listing 37p. For 10M- token conversations, additional summaries of prior plans are also provided.
The response generation process unfolds as an iterative interaction between the assistant and user roles. First, the assistant LLM produces answer to the user's most recent question (line]), This output is then analyzed by a question-detection module, which determines whether the assistant's response contains a counter-question directed at the user (line [T], using the prompt shown in List- ing 35 that takes the assistant response as input and outputs yes question is present and no otherwise. If such counter-question is detected, the response-together with the current and pre- vious sub-plans, relevant past context, and conversation summaries- passed to the user LLM, which generates realistic reply that reflects the storyline and contextual details using the prompt shown in Listing (Iine 14p. This new user reply is subsequently passed back to the assistant LLM, continuing the conversation. This loop repeats until further assistant questions are detected the predefined threshold ô2 (which is set to two) is reached, preventing infinite cycles. For ò1 we tested values and which we selected 2 as it produces more realistic dialogues.
Beyond direct question-answer exchanges, follow-up detection module (line 21 evaluates whether, in a realistic setting, the user would naturally ask clarifying or elaborative follow-up. The need for follow-up is determined using the prompt shown in Listing 36 which takes as input the seed, dialogue history, and the assistant's most recent response, and outputs yes or no. This decision is guided by factors such subject complexity, ambiguity in the assistant's answer, or in- completeness of the response. When a follow-up is required, the module conditions on the seed, the current and prior sub-plans the most recent M turns, and summaries of carlier turns to generate the follow-up query using the prompt shown in Listing 33 The generated query is then passed back to the assistant LLM for resolution. As with the assistant-question loop, strict threshold dg (which is set to two like 61) limits the number of follow-up exchanges, preventing unbounded cycles.
Through the interaction of these two chreshold-controlleed modules, the system produces conversa- tions that exhibit naturalistic bidircctional dynamics, rich contextual references, and realistic clarifi- cation behaviors characteristic of human-Al dialogues.
24

---

Published as a conference paper at ICLR 2026
Algorithm Conversation plan generation.
Input: domain length budget L, title 0, theme т. subtopics Σ, user profile user relationships p. timeline r, number of conversation sub-plans N, number of bullet-points in each conversation
sub-plan M, generator G
Output: Conversation plan set
I: S (c, T.
Initialize seed
2: if I. E (128K, 500K, 1M} then
3:
A G(S)
Generate narratives using Listing 23 Generate a single conversation plan with Listing
4:
P + G(S, u, p. p,T,N,M,A,
5:
else if L 10M then
6:
P <- {
> Initialize set of plans
7:
if Sequential Expansion then
8:
S' + Greets (S.I)
Generate sequential sub-seeds with Listing 28
9:
for each 8 E S" do
10:
A, G(s))
b Generate narratives for sub-seed Binary indicator: if first plan, else Generate plan with Listing 30
11:
b 1 1i= 0]
12:
P. + G(,,, p, P,-,,, P+ PU(P]
13:
14:
end for else o -ierarchica Decomposition then
15:
16:
S' + Gascompose r)
Decompose seed with Listing 29
17:
for each s/ E 5' do
A; + ((s))
Generate narratives for sub-seed
19:
6 - 1 1i=
20:
P. < T,, N, A,, u, Pi-1, 1,, Po Po...... i,,,,,))
D. Generate plan with
Listing
21:
+ PU(P]
22:
end for
23:
end if
24: end if
25: return P
Algorithm User questions generation.
Input: seed S, conversation plan p. number of questions per iteration generator G
Output: Question set Q p {PI.... .PI.....PN
b Conversation plan with N sub-plans
2: Q 0
> Initialize empty question set
3: for each p: E do
4: p. {pı1.... P.K
5: for each Ptj E Pi do
6:
(SS,Pij (.g---.,PG-i)) (PI,,...,PPI))... Generate 1 questions using
Listing
7:
Q
(Q4)
Append generated questions to the question set
8: end for
9: end for
10: return Q
25

---

Published as a conference paper at ICLR 2026
B.3.5 ALGORIHTM
Algorithm Answer generation.
Input: question set Q = {Qi..... QN), seed S, conversation plan set P, thresholds 81, 02,
assistant-question detector Φ follow-up detector d/, generator G
Output: conversation list 7
1: T 5 0
Initialize empty conversation list
2: for each Qi E do
3:
(9.......0 Qi=(qn......) {J}
Questions in sub-plan
4: for each q3 E Qi do
5:
t t {
b Initialize turn sequence
H recent-M turn window at turn
6:
H. + summary of turns prior to (M()
7:
p(v)) + summaries of conversation plans preceding p
8:
Gassistan (1((-1,, H(M) H,,, p(Pp)
9:
0zj 37
Generate assistant response with
Listing
t {ai
>> Add assistant's response to current dialogues turn
11:
isQ (a.j. H (M)
(M)
H.)
> Cheekks if assistant response contains question from user
with Listing 35
12:
count. <-0 0
13:
while isQ and count < 81 do
Udj < Guser (S, Pt, P1:(1-1); pl H He, aij)
(M)
14:
Generate user's response
assistant question with Listing 38
+ uU {
Add user's response to current dialogues turn
agj + Gussistant (S, S. 1): H!
(M) H (())
Generate assistant's response Add assistant's response to current dialogues turn
17:
t faijt
18:
count + count +
isQ < M(aig H [1)
H(M) H.))
19:
20:
end while
needFU < (aagg H((a)) S) Checks user need to ask followup question with
21:
Listing 36
22:
fu.count + 0
while needFU and fu.count < 82 do
Uij f- Guser (S, PP:((1: (P)) HH(M (M) H aig)
24:
Generate user's followup
question with Listing 39
25:
t+ {uij}
(M) H (<P))
26:
asj +- Gussistant (S, P1 1):
Generate assistant's 's response to
user's followup question
27:
++ 1- {ai;} fu_count t- foncaunl +
28:
needFU 1h baiis ((M) H,, end while
29:
30:
31:
TUTU
32:
end for
33:
end for
34: return
26

---

Published as a conference paper at ICLR 2026
B.4 USER UTTERANCE GENERATION HYPERPARAMETERS
Table 6: Batching configuration by chat size and domain category for user-turn question generation. SUBPLANS denotes the number of conversation sub-plans, K the number of batches per sub-plan, and the number of questions generated per batch.
Chat Size
Category
NUM.SUBPLANS
K
I
General
5
10
2
128K
Coding
3
23
Math
3
25
General
10
10
4
500K
Coding
10
10
3
Math
10
10
4
General
10
10
9
IM
Coding
10
10
Math
10
10
General
10
10
9
10M
Coding
10
10
Math
10
10
B.5 CREATED PROBING QUESTIONS DISTRIBUTION
We measure which parts of the dialogue contain the information required to answer the probing questions. To this end, each conversation is divided into ten equal segments, and we record the seg- ment(s) where the supporting evidence for each probing question resides. The detailed methodology for aligning probing questions with dialogue segments described Section 2.3 The resulting dis- tributions across conversation lengths are reported in Table
Table 7: Percentage distribution of created probing questions across ten equal chat segments (deciles) for different chat sizes. Each row corresponds to a segment of the dialogue, moving from the beginning (Segment 1) the end (Segment 10).
Chat Segment (Decile)
100K
500K
1M
10M
0.00%
0.65%
0.19%
0.00%
11.05%
23.70%
21.60%
10.24%
14.83%
15.91% %
20.11%
16.27%
12.79%
14.45%
15.83%
15.06%
5
13.08%
7.95%
9.50%
14.46%
13.37%
9.09%
8.01 %
9.64%
11.92%
6.33%
5.96%
10.24%
8.14%
5.52%
5.21%
13.25%
9
9.59%
4.55%
4.47%
8.43%
10
5.23%
11.85%
9.12%
2.41%
27

---

Published as a conference paper at ICLR 2026
B.6 MEMORY ABILITIES EXAMPLES
To illustrate how our benchmark evaluates different aspects of long-term conversational memory. provide representative probing questions and their ideal answers for each of the memory abilities. These examples demonstrate how each ability is operationalized in practice.
1. Abstention (withholding answers when information is missing)
What specific advice did Manuel give about property management companies during the March Investors Meetup? Ideal Answer:
Probing Question:
Based on the provided chat, there is no information related to the specific advice Manuel gave about property management companies.
2. Contradiction Resolution (detecting and reconciling inconsistencies)
Probing Question: Have ever attended any real estate webinars investor meetups? Ideal Answer:
notice you've mentioned contradictory information about this. You said you have never attended any real estate webinars investor meetups, but you also mentioned attending webinar about Turkey's rising demand for multi-family rentals. Which statement correct?
3. Event Ordering (reasoning about chronological sequence)
Probing Question: How did my focus different aspects of property investment and management develop throughout our conversations in order? Mention ONLY and ONLY ten items.
Your focus developed in this sequence: 1) Agent interaction and prepara- tion, 2) Financial evaluation, 3) Financing and mortgage, 4) Contractor management, 5) Zoning and permits, 6) Renovation planning, 7) Viewing preparation, 8) Negotiation and cash flow, 9) Renovation completion, 10) Commercial property considerations.
Ideal Answer:
4. Information Extraction (recalling factual details)
Probing Question: What amount did say had set aside as my initial capital for invest-
ing?
Ideal Answer: You said you had set aside $50,000 your initial capital.
5. Instruction Following (adhering to explicit user constraints)
Probing Question: How should I allocate my funds for the upcoming quarter?
The response should include an itemized list of costs, category-by- category breakdown, and detailed cost analysis, reflecting the instruction to always pro- vide detailed financial breakdowns when budgeting.
Ideal Answer:
6. Information Update (revising prior facts when updated)
Probing Question: How much initial capital have allocated for real estate investing?
Ideal Answer: $60,000
7. Multi-hop Reasoning (integrating evidence across turns)
Probing Question:
How many different banks did I consider for my mortgage options across my conversations?
Ideal Answer: Two banks: Halkbank and Ziraat Bank.
Preference Following (adapting evolving user preferences)
28

---

Published as a conference paper at ICLR 2026
Probing Question: I'm looking at several properties and want to understand how to approach comparing them effectively. What steps would you suggest I take? Ideal Answer: A structured, step-by-step process that emphasizes data collection and analysis, breaking comparisons into measurable factors, reflecting the user's preference for methodical, data-driven decisions.
Summarization (abstraction and compression of content)
Probing Question: Can you give me a comprehensive summary of my journey and decision-making process around investing in rental properties, including how my budget, property choices, management considerations, and financing plans have developed over time?
Your journey began with an initial capital of $50,000, followed by ex- ploration of market conditions, renovation planning, property type tradeoffs, financing through Halkbank and Ziraat Bank, and structured plan for purchase and management.
Ideal Answer:
10. Temporal Reasoning (reasoning about durations and timelines)
Probing Question: How many days are there between my first property viewing with Mehmet Yilmaz and the last one I scheduled?
Ideal Answer: There are 2 days between the first property viewing on March 25 and the last March 27.
29

---

Published as a conference paper at ICLR 2026
C DETAILED EXPERIMENTS
C.1 ABLATION STUDY
In this section, present the complete results of our ablation experiments. All ablations ducted using Qwen2.5-32B-AN as the base model. We evaluate the contribution of individual components in proposed module shown in table
Table 8: Ablation study showing the impact of removing key memory components (retrieval, scratchpad, working memory, and noise filtering) performance across various conversation lengths (100K-10M).
Tength
Memory Ability
Hase
win Ketrieval From Index
w/o Scratchpud
w/o Working Memory
w/o Noise Filtering
Abstention
0.475
0.725
0.600
0.575
0,700
Contradiction Resolution
0.037
0,043
0.012
0.043
0.018
Ordering
0.190
0.194
Information Extraction
0.479
0.329
0.510
0.451
0.485
Instruction Following
100K
Knowledge Update
0.362
0,237
0.350
0,362
0.312
Multi-Hop Rcasoning
0.281
0.201
0.248
0.303
0.181
Preference Following
0.566
0.675
0,533
0,579
0.491
Summarization
0.232
0.232
0.143
0.223
0.103
Temporal Reasoning
0.112
0.075
0. 1.125
0. 125
0.087
Average
0
0,311
0.300
0.327
0.289
Abstention
0.71
0.571
0.385
0.637
0.585
Contradiction Resolution
0.017
0.007
0.014
0.017
0.014
Event Ordering
0.244
0.222
0.266
0.262
0.229
Information Extraction
0.506
0.254
0.466
0.485
0.464
Instruction Following
0.295
0.307
0.316
0.334
0.286
500K
Knowledge Update
0.278
0.192
0.285
0.235
0.314
Multi Hop Remsoning
0.214
0.104
0.2277
0.192
0.247
Preference Following
0.571
0.553
0.450
0.547
0.465
Summarization
0.312
0.225
0.203
Temporal Reasoning
0.121
0,042
0.116
0.114
0.130
Average
0.2955
Abstention
0.500
1.M64
0.600
0.557
0.507
Contradiction Resolution
0.021
0.035
0.042
0.032
Event Ordering
0.200
0.215
0.221
0.227
0.199
Information Extraction
Instruction Following
0.419
0.427
0.335
0.384
0.351
Knowledge Update
Multi-He Hop Reasoning
0.209
0.129
0.227
0.221
0.169
Preference Following
0.551
0.602
0.536
0.597
0.540
Summarization
0.316
0.310
0. 169
0.330
0.128
Temporal Reasoning
0.154
0.050
0.111
0.121
0.111
Average
0.309
0.285
0.295
0.328
0.269
Abstention
0.550
0.800
0.650
0.650
0.600
Contradiction Resolution
0.012
0.000
0.012
0.000
0.000
Event Ordering
0.197
0.199
0.199
0.209
0.181
Information Extraction
0.350
0,000
0.200
0.150
0.200
Instruction Following
0.350
0.175
0.175
0.175
0.050
10M
Knowledge Update
0.275
0.050
0,300
0.150
0.225
Multi-Hog Reasoning
0.125
0.000
0.125
0.125
0.075
Preference Following
0.191
0.241
0.175
Summarization
0.220
0.119
0.068
0.008
0.050
Temporal Reasoning
0.000
0.050
0.000
Average
0.238
0.153
0.202
0.181
0.155
C.2 RETRIEVAL BUDGET
We investigate the impact of the retrieval budget through sets of experiments: (i) varying the retrieval depth by setting the number of retrieved documents K E (5, 10, 15, 20), and (ii) comparing dense retriever against sparse retriever (SPLADE).
The full results examining the effect of different retrieval depths (number of retrieved documents) presented in Table 9
30

---

Published as a conference paper at ICLR 2026
Table Effect of retrieval depth on performance across conversation lengths (100K-10M) and memory abilities. Results are shown for different numbers of retrieved documents (K E (5, 10, 15, 20)).
Length
Memory Ability
K=5
K=10
K=15
K=20
Abstention
0,475
0.500
0.625
0.625
Contradiction Resolution
0.037
0.025
0.025
0.031
Event Ordering
0.205
0.191
0.218
0.210
Information Extraction
0.479
0.450
0.412
0.391
Instruction Following
0.362
0.362
0.475
0.462
100K
Knowledge Update
0.362
0.375
0.350
0.300
Multi-Hop Reasoning
0.281
0.322
0.321
0.309
Preference Following
0.566
0.591
0.562
0.575
Summarization
0.232
0.231
0.218
0.213
Temporal Reasoning
0.112
0.162
0.137
0.137
Average
0.311
0.321
0.334
0.325
Abstention
0.571
0.514
0.614
0.642
Contradiction Resolution
0.017
0.021
0.071
0.071
Event Ordering
0.244
0.229
0.238
0.247
Information Extraction
0.506
0.531
0.503
0.507
Instruction Following
0.295
0.341
0.390
0.373
500K
Knowledge Update
0.278
0.307
0.326
0.326
Multi Multi-Hop Reasoning
0.214
0.188
0.234
0.213
Preference Following
0.571
0.597
0.628
0.607
Summarization
0.344
0.354
0.375
0.376
Temporal Reasoning
0.121
0.128
0.121
0.135
Average
0.316
0.321
0.350
0.350
Abstention
0.500
0.521
0.600
0.585
Contradiction Resolution
0.021
0.021
0.057
0.053
Event Ordering
0.200
0.224
0.240
0.242
Information Extraction
0.366
0.398
0.377
0.391
Instruction Following
0.419
0.476
0.439
0.446
IM
Knowledge Update
0.357
0.350
0.400
0.407
Multi-Hop Reasoning
0.209
0.189
0.209
0.190
Preference Following
0.551
0.596
0.535
0.514
Summarization
0.316
0.317
0.325
0.351
Temporal Reasoning
0.154
0.154
0.119
0.199
Average
0.309
0.325
0.330
0.330
Abstention
0.550
0.600
0.650
0.600
Contradiction Resolution
0,012
0.012
0.025
0.025
Event Ordering
0.197
0.210
0.213
0.236
Information Extraction
0.350
0.150
0.300
0.300
Instruction Following
0.350
0.150
0.450
0.400
10M
Knowledge Update
0.275
0.200
0.300
0.300
Multi-Hop Reasoning
0.125
0.100
0.5
0.150
Preference Following
0.308
0.175
0.275
0.275
Summarization
0.220
0.089
0.196
0.164
Temporal Reasoning
0.000
0.025
0.000
0.000
Average
0.238
0.171
0.253
0.245
In a complementary experiment, we analyzed the impact of retriever choice. Our base architecture employs dense retriever, which compare against the sparse SPLADE-V2 retriever (Formal et al. 2022). As shown Figure 5 in Appendix C.2 SPLADE yields performance gains 1.7% 100K tokens, 0.7% 500K, and 0.8% at 1M, but results in slight performance drop of 0.7%
10M. On average, the sparse retriever provides modest improvement across conversation lengths.
Complete results comparing the dense retriever with SPLADE presented in Table T0
31

---

Published as a conference paper at ICLR 2026
100K
500K
1M
10M
0.35
0.323
0.317
0.309
Performance (%)
0.30
Component
0.231
Base (Dense retriever)
Splade (Sparse retriever)
0.20
Figure LIGHT.
Performance comparison between dense retrieval and sparse retrieval (SPLADE) in
Table 10:
Comparison of dense and sparse retrieval strategics across conversation lengths (100K-10M) and memory abilities. The table reports performance when using the default dense retriever versus a sparse retriever (SPLADE).
Length
Memory Ability
Base (Dense retriever)
Sparse retriever (SPLADE)
Abstention
0.475
0.525
Contradiction Resolution
0.037
0.43
Event Ordering
0.205
0.181
Information Extraction
0)479
0.596
Instruction Following
0.362
0,400
100K
Knowledge Update
0.362
0.350
Multi-Hop Reasoning
0,281
0.267
Preference Following
0.566
0.562
Summarization
0.232
0,230
Temporal Reasoning
0.112
0.125
Average
0.311
0.328
Abstention
0.571
0.557
Contradiction Resolution
0.017
0.025
Event Ordering
0.244
0.226
Information Extraction
0.506
0.559
Instruction Following
0.295
0.345
500K
Knowledge Update
0.278
0.307
Multi-Hop Reasoning
0.214
0,212
Preference Following
0.571
0.565
Summarization
0.344
0.330
Temporal Reasoning
0.121
0.107
Average
0.316
0.323
Abstention
0,500
0.564
Contradiction Resolution
0.021
0.028
Event Ordering
0.200
0.196
Information Extraction
0.366
0.392
Instruction Following
0.419
0.401
IM
Knowledge Update
0.357
0.371
Multi-Hop Reasoning
0.209
0.193
Preference Following
0.551
0.595
Summarization
0,316
0,300
Temporal Reasoning
0.154
0.133
Average
0.309
0.317
Abstention
0.550
0.700
Contradiction Resolution
0.012
0,000
Event Ordering
0.197
0.202
Information Extraction
0.350
0.350
Instruction Following
0.350
0.250
10M
Knowledge Update
0.275
0.375
Multi-Hop Reasoning
0.125
0.125
Preference Following
0.308
0.200
Summarization
320.220
0.090
Temporal Reasoning
0.000
0.025
Average
0.238
0.231

---

Published as a conference paper at ICLR 2026
C.3 EFFECT OF EMBEDDING MODEL CHOICE
We also examined how the choice of embedding model affects the performance of both the RAG baseline and the episodic memory component of LIGHT. In the primary experiments, we used the hal//gg-small-en-v/// embedding model. To assess robustness, we re-ran all experiments using the larger BAAl/hge-larg--en--/
model while keeping the L.M reader fixed GPT-4. I-nano. As shown in Table LIGHT consistently outperforms the RAG baseline under both embedding configurations. Moreover, LIGHT exhibits larger gains when switching to the higher-capacity em- bedding model, achieving an additional 2.08% improvement at the 1M-token setting and 16.37% the 10M-token setting. These results indicate that LIGHT is robust to changes in embedding quality and can effectively leverage stronger embedding models to enhance long-term memory abilities.
Table 1: Effect of the embedding model on performance across conversation lengths -00K))0M) and ten memory abilities. Results are shown for the RAG baseline and LIGHT using two different embedding models.
Length
Memory Ability
RAG (bge-small)
Ours (bge-small)
RAG (bge-large)
Ours (bge-large)
Abstention
0.800
0,575
0.825
0.600
Contradiction Resolution
0.018
0.031
0.012
0.031
Event Ordering
0.169
0.177
0.185
0.171
Information Extraction
0,362
0.538
0.404
0.562
Instruction Following
0.350
0.400
0.337
0.462
100K
Knowledge Update
0.375
0.375
Multi-Hop Reasoning
0,263
0.365
0.224
0.341
Preference Following
0,550
0.625
0.537
0.562
Summarization
0.083
0.202
0.089
0.148
Temporal Reasoning
0.125
0.162
0.112
0.162
Average
0.309
0.345
0.305
0.341
Abstention
0.828
0.600
0.814
0.571
Contradiction Resolution
0.025
0.035
0.028
0.032
Event Ordering
0.180
0.204
0.178
0.202
Information Extraction
0.382
0.491
0.345
0.454
Instruction Following
0.286
0.342
0.303
0.363
500K
Knowledge Update
0.288
0.240
0.380
0.321
Multi-Hop Reasoning
0.233
0.266
0.272
0.282
Preference Following
0.577
0.684
0,571
0,650
Summarization
0.184
0.334
0.153
0,316
Temporal Reasoning
0.161
0.154
0.126
0.126
Average
0.314
0.335
0.317
0.331
Abstention
0.778
0.678
0.771
0.657
Contradiction Resolution
0.028
0.021
0.021
0.025
Event Ordering
0.179
0.211
0.194
0,211
Information Extraction
0.399
0,410
0.360
0.439
Instruction Following
0.271
0,394
0.269
0.421
Knowledge Update
0.342
0,392
0.371
0.378
Multi-Hop Reasoning
0.293
0.278
0.204
0.254
Preference Following
0,513
0.576
0.497
0.598
Summarization
0.152
0.290
0.116
0.296
Temporal Reasoning
0.064
0.107
0.150
Average
0.302
0.336
0.292
0.343
Abstention
0.650
0.400
Contradiction Resolution
0.012
0.025
0.025
0.037
Event Ordering
0.201
0.173
0.203
0.171
Information Extraction
0.300
0.350
0.300
0.450
Instruction Following
0.175
0.250
0.175
0.275
10M
Knowledge Update
0.325
0.300
0.325
0.300
Multi-Hop Reasoning
0.091
0.135
0.066
0.075
Preference Following
0.366
0.425
0.316
0.525
Summarization
0.063
0.179
0.100
0.224
Temporal Reasoning
0.000
0.025
0.000
0.025
Average
0.218
0.226
0.231
0.263
C.4 EFFECT OF INDEXING SETUP
We also investigated the effect of the vector database indexing setup on the performance of LIGHT. In the primary experiments, used IndexFlatIP, and in the experiments below, examined the
33

---

Published as a conference paper at ICLR 2026
effect of switching the indexing setup to IndexHNSWFlat. For this experiment, the reader LLM was GPT-4.1-name The results are shown Table
Table 12: Effect of vector database indexing setup performance across conversation lengths (100K-10M) and ten memory abilities. Results are reported for LIGH'T under two different indexing configurations.
Length
Memory Ability
Ours (IndexFlatIP)
Ours (IndeXHNSWFlatt
Abstention
0.575
0.600
Contradiction Resolution
0.031
0.031
Event Ordering
0,177
0.173
Information Extraction
0,538
0,565
Instruction Following
0,400
0.375
100K
Knowledge Update
0.375
0.400
Multi-Hop Reasoning
0.365
0.285
Preference Following
0.625
0.662
Summarization
0.202
0.217
Temporal Reasoning
0,162
0.162
Average
0.345
0.347
Abstention
0,600
0.528
Contradiction Resolution
0.035
0.032
Event Ordering
0,204
0.207
Information Extraction
0.491
0,503
Instruction Following
0.342
0.332
500K
Knowledge Update
0.240
0,226
Multi-Hop Reasoning
0.266
0.269
Preference Following
0,684
0,666
Summarization
0.334
0.317
Temporal Reasoning
0.154
0.176
Average
0.335
0.325
Abstention
0,678
0.578
Contradiction Resolution
0,021
0.025
Event Ordering
0.211
0.211
Information Extraction
0.410
0,420
Instruction Following
0.394
0.386
IM
Knowledge Update
0,392
0.385
Multi-Hop Reasoning
0.278
0.278
Preference Following
0.576
0.567
Summarization
0,290
0.257
Temporal Reasoning
0.107
0.128
Average
0.336
0.324
Abstention
0,400
0.600
Contradiction Resolution
0.025
0.025
Event Ordering
0.173
0.168
Information Extraction
0.350
0.350
Instruction Following
0.250
0.300
10M
Knowledge Update
0,300
0,225
Multi-Hop Reasoning
0.135
0.075
Preference Following
0.425
0,433
Summarization
0.179
0.194
Temporal Reasoning
0,025
0.000
Average
0.226
0.237
C.5 SUPPLEMENTARY BASELINE EVALUATION
Alongside long- context LLMs and RAG, we also evaluated ReadAgent (Lee et al. 2024), another method designed enhance long-tern memory in LLMs, BEAM and compared it with LIGHT. The results demonstrate that LIGHT consistently outperforms ReadAgent across all four conversa- tion lengths (100K, 500K, 1M, and 10M). The full results shown in Table 13
34

---

Published as a conference paper at ICLR 2026
Table 13: Comparing LIGHT with ReadAgent across conversation lengths (100K-10M) and
memory abilities.
Length
Memory Ability
ReadAgent
Ours (LIGHT)
Abstention
0.850
0.475
Contradiction Resolution
0.000
0.037
Event Ordering
0.200
0.205
Information Extraction
0,066
0.479
Instruction Following
0.237
0.362
100K
Knowledge Update
0.150
0.362
Multi-Hop Reasoning
0.095
0.281
Preference Following
0,425
0.566
Summarization
0,045
0.232
Temporal Reasoning
0.000
0.112
Average
0.206
0.311
Abstention
0.928
0.571
Contradiction Resolution
0,007
0.017
Event Ordering
0,237
0.244
Information Extraction
0.047
0.506
Instruction Following
0.166
0.295
500K
Knowledge Update
0.014
0.278
Multi-Hop Reasoning
0.022
0,214
Preference Following
0.386
0.571
Summarization
0,069
0.344
Temporal Reasoning
0.028
0.121
Average
0.191
0.316
Abstention
0.792
0.500
Contradiction Resolution
0.003
0.021
Event Ordering
0.211
0.200
Information Extraction
0.106
0.366
Instruction Following
0.166
0.419
1M
Knowledge Update
0.014
0.357
Multi-Hop Reasoning
0.105
0.209
Preference Following
0,391
0,551
Summarization
0.041
0.316
Temporal Reasoning
0.033
0.154
Average
0.186
0.309
Abstention
0.750
0.550
Contradiction Resolution
0.000
0,012
Event Ordering
0.205
0.197
Information Extraction
0.000
0.350
Instruction Following
0.300
0.350
10M
Knowledge Update
0.000
0.275
Multi-Hop Reasoning
0.000
0.125
Preference Following
0.166
0.308
Summarization
0.061
0,220
Temporal Reasoning
0.000
0.000
Average
0.148
0.238
35

---

Published as a conference paper at ICLR 2026
D NUGGET DESIGN
In this section, we provide illustrative examples for each memory ability, demonstrating how nuggets derived from the corresponding probing questions.
1. Abstention
Objective: The correct behavior is to acknowledge that the requested information is not present in the provided conversation.
Rubric pattern: Each atomic unit should be in this format: States that, based on the provided chat, there no information about <target topic> Example JSON:
"ques:ion *What spec: advice did Manuel give about property management companies during the March 5 Investors Meetup?", redess_espess"" "Based on the provided chat, chere is no information related the specific advice Manuel gave "oouree_hhatida"" (),
propecty management companies.",
"rubric":
"Raged the prov' ded chat, there is information related the specific advice Manuel gave about property management companies."
2. Contradiction Resolution
Objective:
Correct behavior is that the LLM should detect the contradiction and state both contradictory information while requesting clarification. Rubric pattern:
States there is contradictory information.
Mentions claim <A>
Mentions claim <B>
Requests clarification about which statement is correct
Example JSON:
"question": "Have attendad any real wabinars investar mestups?"
"idesl_answer" notice you' ve mentioned contradictory information about You said have never attended any real estate webinara or inveszor neetupa, but you also mentioned atteading weblnar about Turkey's rising demand for multi-family reatals. Which statement is correct?", poource_chatiddee
"firat_staaemmnnt""
"easond_tsttement"
"rubric":
"LLK response should state: there is contradictory informstior", sttended any real
"ME response shou mentions You you estate webinars or investor meetups",
response shou mention: you alsa mentioned attending about Tuckey'u2019a rising demand for multi-family rentals", "LIK response should mention: which statement is cecrest?"
3. Event Ordering
Objective:
Correct behavior is the model lists sequence of events/topics in the correct
chronological order.
Rubric pattern:
LLM response should mention: <event 1>
36

---

Published as a conference paper at ICLR 2026
LLM response should mention: <event N> Example JSON:
"ques:iea":
"Now did my forus on different aspects of property investment and nanagement develop throughout conversations in order? Mention ONLY ONLY ten items.",
*answer*; "Your rocua or property Investment and management developed this sequence: Initial engagement with the local agent and prepsration for property viewings, 21 Evalustion of prepesty financials including Ro1 and rental income patential, Exploration of financing options and mortgage cencerns, 4) Handling centractor performance and repair negotiaticns, 5) understanding zoning regulations
permit requirements for property converstona, 6) Planning and priorisizing renovations and inveatient risks for multi family properties, 71 Detailed prepazation for property viewings involving beth flow implications related to repeir costs, 5) sinel renevaticn project strategles,
and contracter, 8) Negotiation strategies and cash
completion stepa and portfolia diversificate Consideration of commercial property types and Location factora for long- investment.",
"ordering_teste""
"1st: Agent interaction and viewing prepars:ion",
"2nd: Property finarcisl evalustien",
*3rd: Financing and mortgoga concerns",
"4th: Contractor maragement",
*5th: Zoning and permils",
"6th: Renovat on anning and investment riaks",
"7th: Viewing preparation with agent and contractor",
"8th: Repair cost negotiation and cash flow",
*9th: Kenovation completion and portfolia diversifscationt
*10th: Commercial property and location constderations"
source_chatt_dee (),
"rubrie";
"LIK response should mention: Agent inceraction and viewing preparation",
"LIX response should mention: Property financial evalustion",
Financing and mortgage concerns", "IEX response shou d mention: Contractor management";
"LIK response should
response should
Soning and permits",
"LIX response should mentien: Renovation planning and investment risks",
"IX responae should mention: Viewing preparation with agent and contractor"
"LiK response should mention: Repair cost negotiation and cash flox", Renovation completion and portfolia
*LKK response should
response should mention: Coamercial propecty locatier. considerations",
"Presents the events the correct chronelogical orde."
4. Information Extraction
Objective:
LLM should answer the questioned facts correctly.
Rubric pattern:
Instantiate one criterion per fact directly from the ideal answer, using the
"LLM response should state/mention."
Example JSON:
"quession:; *What amount did say had set aside as my initial capital for investing?",
"idesl_answez"" "You said you had set aside $50, 800 as your initial capital.", "sorrce_chat_idd"; [],
"cubrio";
"LUK response should state: 000*
Instruction Following
37

---

Published as a conference paper at ICLR 2026
Objective: LLM should adhere to format and/or content priorities stated in the conver- sation.
Rubric pattern:
Use instrruction.bbingg..e.. (the explicit instruction) and decompose compliance into atomic criteria
Example JSON:
"question"; "Row should allocate Tunda for the upcoming quarter?", snteraation_being_eee
"Always provide detailed finencial breakdowss when ask about budgeting decisions."
"expacted_ comppl "Responsa should includa itemized costa, spectfic amounts total for different categories, detailed breakdown
estimate*,
"source_chat_idd" (],
rrbbric:
"LUK response should contain: itemized list costs",
"LIK response should contain: sategory-by-cateege breakdosn", "LEX response shou' d contain: detaf led cest anal ystss
6. Knowledge Update
Objective: LLM must reflect updated values when prior values have changed over time.
Rubric pattern:
Derive criteria from the ideal answer, using the stem "LLM response should state/mention." for the updated value(s).
Example JSON:
"question": 2", "answer*; "$60, 000",
initial capital allocated
investing
; *originsl_infer
in
"cubric": "LIK response should state: 360,000*
Multi-hop Reasoning
Objective: LLM must aggregate or compare information spanning multiple sessions. Rubric pattern:
Instantiate criteria from the ideal answer for each required intermediate or ag- gregated fact.
Example JSON:
*questian"; "How many different banks did consider for my mortgage options across my conversations?"
*answez*;
"Two
Dank.",
"source_chat_ida"" [],
"rubric":
"LIK response should state: Two backs", "LiK response should state: Halkbank",
"TI response shou states Airass Bank"
8. Preference Following
38

---

Published as a conference paper at ICLR 2026
Objective: LLM must generate content consistent with user-specified preferences.
Rubric pattern:
Use pre preferannee.bsing..eseee (the user's stated preference) and decom-
pose expected into atomic criteria.
Example JSON:
"questian"; Looking several propersies want understand how approach comparing them fecttivlly. What steps would d you suggest take?", "prefezence_bee wtw__eeeee_____eeeeeeeeeee Leated"; prefez methodical, data driven decisions over impulsive purchases, reflecting analysical nature.",
rexpected_compe compplice"" "Response should ou:line structured, step-by-step
approsch involves gathering amalysing relevant data before making choice, rather than suggesting quick or impulsive actions.", "source_chat_lee (),
rubrice::
"LLK response should contain: provides a cleer, logical process", "LLK response should contain: emphasizes daze collection and analysis", "LIX responae should contain: breaks down compariscn into measurable factora
9. Summarization
Objective: LLM must provide a comprehensive summary covering required content el- ements.
Rubric pattern:
Decompose ideal Rteal.suumary summary into atomic content units; the stem "LLM response should contain:".
Example JSON:
quesstio"; *can you give me a comprehensive summary of my journey and decision- naking process around investing in rental properties, insluding how my budget, property choices, management conatderat ienatdderaionn, ona, and rinancing plans developed over Lime?",
"ideal_ed "idesl_summary"" summazy"; "Your journey toward investing in rental properties began with 650, 200, which you questioned potentially inaufficient for purchasing property within 12 months. Rarly di scusaionn ons highlighted the need to research local market conditions, down payment like
an initial espital
renovations, revealing that typical investments might exceed your initial capital. You explored identifying goed signs such as structural issues and outdated features, emphasizing the impostance of cast-benefit onalysis progressed, you weighed the pros and renovations, of investing close your lccation versus elsewhero, halancing sase of managemant against market diversity growth potential. You single family homes and multi family units, analyzing factors like rental yield, management complexity, similar yields but differing capital needs. Financing options were carefully compared, partteslarls between Halkbank service quality to optimize costa,
requi rementa, and additional
uppez propecties by Learning recognize
your plans
considared the cholce
investment scale,
examples showing
Miratt Bank mortgages
focueing on interest rates, Throughout, you developed step-by-step plan for parchasing your first rental property, including market research, budgeting, inapections, financing, and tenant management, with timelines to reduze anxiety and ensure readiness. this comprehensive process reflects though:ful avolution from initial capital concerns to datailed irvestment stracegies, property evaluation "inancing deciaions, and management planning, all aimed at making informed, belanced real estate Lavestmeat choices.", (),
"uubrio";
"LIK response should contain: iavesting in rental properties began with initial capitel $50, 000""
*LIX response shou d contain: Early discussions highlightan the need to zesearch local market conditions, down payment zequirementa, and additionsl costs like clesing fees",
"LIK responae should contain: You explored ident identifying ifying good fixer-upper propecties by learning recegnize signs such outdated features",
issues
39

---

Published as a conference paper at ICLR 2026
"LEK responae should contain: you weighed the pros and cons of investing versus elsewbere, belancing against market diversity and grewth potential", response shou contain: family homes and multi-Tamily unita, analyzing Tactors like rental yield, menagement complaxity, and investment scale", "LX responae should contain: Pinarcing options
your
managemens
considered
choice between singla-
carefully compared, particulerly between llakkbank and 2irast Dank mortgages, Iocusing on interest rates, fees, and service quality optimize costs", "LIK response should contoin: you devaloged a atep-by-step plan for purchasing your first rental property, inclucing market research, budgeting, inspections, financing, and Lenant management*
10. Temporal Reasoning
Objective: LLM must compute or restate durations and timeline relations correctly. Rubric pattern:
Derive criteria from the ideal answer, using the stem "LLM response should state:"
Example JSON:
"question": "How many days there between first property viewing with Mehmet Yilnaz and the last one scheduled?", days between the first property viewing on March: and the last one on March 27.",
"answer": "There
toalluutoonrr red":
days",
Source_chattidd"" {
"first_evens"; (],
"second_event";
"rbbric"
"LIK response should state: 2 days",
"LX responae should state: from March 25, 2024 till March 27, 2024"
40

---

Published as a conference paper at ICLR 2026
E EXAMPLES FROM DIFFERENT COMPONENTS OF BEAM
In this section, we provide illustrative examples of generating chat in the coding domain. Specif- ically, we include a representative chat seed with its domain, title, theme, and subtopics, followed by the corresponding narratives, where only truncated set is shown for brevity. We then present the user profile and the user's social relationships. Next, we provide excerpts from the conversation plans, showing only subset of bullet points from each sub-plan while preserving their full descrip- tions to maintain clarity. Finally, we provide samples of the generated chat, highlighting exchanges where the user shares or requests code, and including follow-up turns demonstrate the natural- istic back-and-forre flow. Together, these examples illustrate how different components of BEAM interact to form coherent, long-context dialogues.
Chat Seed
Domain: Coding
Title: Automating Social Media Posts with Python
Theme: Scheduling and posting content across multiple platforms Subtopics:
Twitter API integration
Facebook Graph API usage
Instagram automation tools
Scheduling with cron jobs / APScheduler
Image and caption management; hashtag generation
Error handling for failed posts; tracking engagement metrics
Narratives (Truncated)
Technical Problem-Solving: Debugging Twitter OAuth/403/429; fixing hashtag validation;
profiling scheduler bottlenecks.
Learning & Knowledge: API does comprehension (Twitter v2, Facebook Graph v12-15); best practices for Instagram automation; mastering cron/APPSheduuler. Progress Development: Setting up Twitter/Facebook integrations; building Instagram tools; designing scheduling algorithms.
Implementation: Feature implementation and refactoring for efficiency; async migration; retry and backoff strategies.
Framework & Technology: Python libraries (Tweepy, facebook-sdk, requests); APSched- uler/cron: Redis; asyncio.
Testing & QA: Unit/integration/E222 /E2i tests (pytest, Selenium); TDD for schedulers and hash- tag rules.
DevOps & Deployment: CI/CD (GitHub Actions), containerization (Docker), EC2 deploy- ment, blue-green releases.
Data: PostgreSQL schemas, indices, ETL for engagement metries, Redis caching. Integration & APIs: Webhooks, message queues (RabbitMQ), API Gateway, SNS/Lambda.
Performance: Caching, load balancing (HAProxy), CPU/memory targets, throughput goals. Security/Compliance OAuth, token rotation, TLS, GDPR. PM & Workflow: Sprints, reviews, documentation standards.
User Profll
Name: John Brooks
Age:
Gender: Male
Location: Port Charles, Luxembourg Profession: Secretary/Adminnstraae
Personality: He a pillar of his community, always ready lend a helping hand and offer guidance when needed. With strong sense of tradition and order, he values honesty and dedication, often taking on a mentorship role to help others. His diligent and efficient ap- proach to planning and organization makes him a reliable asset to those around him. He has a warm and welcoming demeanor, always willing to open his heart and home to friends, loved ones, and neighbors. Despite his strong convictions, he believes in the power of hospitality and good manners, often going out of his way to make others feel supported and cared for.
41

---

Published as a conference paper at ICLR 2026
With dry sense of humor and quick wit, he can be entertaining to be around, but he's not afraid to speak his mind and challenge the status quo when necessary. His practical and responsible nature makes him respected member of his community, and his ability to stay grounded and logical in stressful situations is a valuable asset to those around him.
Relationships
Parents: Elizabeth (74), Robert (76)
Partner: Shannon (48)
Close Friends: Taylor (51), Teresa (62), Thomas (44), Charles (56), Patricia (46)
Acquainaanes/Colllaguss Wesley (26), Jason (59), Claudia (15), Janice (13), Dana (53)
Conversation Plan (Only few representative bullets from each sub sub-plan) plan)
Subplan March 1, 2024
Project Initialization: I'm setting up a Python 3.10 environment with Tweepy v4.10.1 and Facebook SDK v3.1.0 for API integrations.
Security Compliance Labels: Authentication for Twitter API Integra- tion: Implemented OAuth 1.0a with environment variables TWI TIITRERAPP IEEY and TWITTER AP SECRET securely stored.
Database & Data Management Labels: Database Design for Social Media Posting: Designed PostgreSQL 14 schema with tables for posts, platforms, and scheduling meta- data.
User Instruction: Always include exact API version numbers when I ask about integration details.
Logical Contradiction: have never registered a Twitter Developer account or created any Twitter app.
Subplan 2 - March 20, 2024
Technical Problem-Solving Labels: Debugging Twitter API Integration: Fixed "403
Forbidden" error caused by missing media upload step before tweet creation.
Implementation & Development Labels: Code Refactoring for Performance: Refac- tored twitter.post py to async functions using asyncio, improved throughput by 30%.
Security & Compliance Labels: Authorization for Facebook Graph API: Implemented OAuth 2.0 flow with refresh tokens stored encrypted using Fernet symmetric encryption.
Information Update: The Instagram automation prototype sprint deadline adjusted to April 5, 2024, to allow additional testing of media upload features.
Subplan 3 3- April 5, 2024
Implementation & Development Labels: Implementing Error Handling: Added retry logic with exponential backoff for Instagram API 429 Too Many Requests errors.
Performance & Optimization Labels: Caching Strategies for Image and Caption Management: Implemented Redis caching for resized images, reducing image processing time from 800ms to 200ms.
Debugging & Troubleshooting Labels: Incident Response for Social Media Automa- tion: Responded to March 30, 2024, outage caused by expired Instagram tokens, imple- mented alerting via Slack webhook.
Subplan 4 - April 20, 2024
Implementation Development Labels: Algorithm Optimization for Scheduling: Rewrote scheduling algorithm to use asyne priority queues, reducing average job dispatch latency from 500ms to 150ms.
42

---

Published as a conference paper at ICLR 2026
Framework & Technology Labels: Integrating Twitter API with Python: Upgraded Tweepy from 14.10.1 v4.12.1 to leverage new media upload endpoints.
Security Compliance Labels: Authentication for Twitter API Integration: Rotated Twitter API keys on April 15, 2024, updated environment variables IWITTER and TWITTER API SECRET.
Subplan 5
May 5, 2024
Progress & Development Labels: Building Hashtag Generation Tools: Developed
hashtag generator supporting dynamic keyword extraction using spaCy v3.5.0 NL.P library.
Database & Data Management Labels: Data Warehousing for Engagement Metries: Designed PostgreSQL 14 schema for engagement metrics with partitioning by month for scalability.
Debugging & Troubleshooting Labels: Log Analysis for Facehook Graph API: De- tected "OAuthException: Error validating access token" on May 1, 2024, resolved by token refiesh automation.
Subplan 6 6 May
May 20, 2024
Implementation & Development Labels: Implementing Error Handling: Added cen- tralized error handler middleware in posting API, logging errors with Sentry v1.12.0.
Debugging & Troubleshooting Labels: Error Diagnosis for Twitter API Integration:
Fixed intermittent "ConnectionResetEmror during media upload by adding retry with jitter.
DevOps & Deployment Labels: Containerization for Instagram Automation: Updated
Dockerfile multi-stage builds, reduced image size from 120MB to 85MB.
Subplan 7 - June 5, 2024
DevOps & Deployment Labels: Deploying Social Media Automation Tools: Deployed v1.0.0 release on AWS EC2 t3. 13.medium with 99.9% uptime SLA.
Integration & API Labels: Event-Driven Architecture for Social Media Automation: Implemented AWS SNS topics for post status updates, integrated with Lambda v3.2.1 functions.
User Experience & Interface Labels: Mobile App Design for Social Media Automa- tion: Released beta version of React Native app on Android with basic scheduling and metrics display.
Subplan 8 - June 20, 2024
Progress & Development Labels: Developing Instagram Automation Tools: Imple- mented batch media uploads for Instagram, supporting up 10 images per carousel post.
User Experience & Interface Labels: Responsive Design for Scheduling: Enhanced React 18.2 dashboard for scheduling with drag-and-drop post reordering, tested on Chrome and Safari.
Security & Compliance Labels: Encryption for Instagram Automation: Migrated to- ken storage to AWS KMS encrypted S3 bucket, access controlled via IAM roles.
Subplan 9 July 2024
Security & Compliance Labels: Authentication for Twitter API Integration: Imple-
mented OAuth 2.0 PKCE flow for enhanced security, tested with Postman v50.15.0.
Performance & Optimization Labels: Code Optimization for Twitter API Integra- tion: Reduced memory usage by 20% by optimizing data serialization and connection pooling.
Debugging & Troubleshooting Labels: Error Diagnosis for Twitter API Integration: Fixed "TimeoutError" during high load by increasing HTTP client timeout from 10s to 30s.
43

---

Published as a conference paper at ICLR 2026
Subplan 10 - July 20, 2024
DevOps & Deployment Labels: Deploying Social Media Automation Tools: Com-
pleted production rollout on July 18, 2024, with downtime and 99.95% uptime in first 48 hours.
Integration & API Labels: Event-Driven Architecture for Social Media Automation: Fully implemented AWS SNS event bus with Lambda v3.2.1 for asynchronous post status updates.
Testing & Quality Assurance Labels: End-to-End Testing for Instagram Automation: Verified Instagram automation stability with 1000+ posts over 7 days, no failures reported.
Generated Dialogues (Exemplars, Truncated)
Turn (Mar 1, 2024)
up Python 3.10 environment for automating social media posts, and I've chosen Tweepy v4. 10.1 and Facebook SDK v3.1.0 for API integrations, but I'm not sure how to structure my project for better outreach. Can you help me plan this in of scheduling and deadlines?
User: I'm trying
Assistant:
Certainly! Setting up an automated posting system with Tweepy and Facebook SDK is a great start.
Project Structure:
Project Setup (Mar 1-7) [Truncated ...]
2. Content Creation (Mar 8-14) [Truncated
3. Automation Development (Mar 15-21) [Truncated ...]
4. Deployment and Monitoring (Mar 22 22-28) [Truncated. ...]
User (Follow-up)
That sounds like a great plan! One thing though-could could we add a step for testing posts on a smaller scale before full deployment? Just make everything looks good and functions as expected.
Assistant
Absolutely, adding a step for testing the posts on a smaller scale is a great idea to ensure everything looks good and functions expected before full deployment. Here's how you can incorporate that into the existing plan:
### Updated Project Structure ##### 1. *Project Setup [Trunicateed ...] #### **Testing Phase (March March 24, 2024)**
**DDas 22-23: Conduct small-scale test.
Poost a few test messages to limited audience or a test account.
Verrffy that the posts look good and function expected.
**y 24 Review the test results.
Address any issues found during the test phase.
Make final adjustments to the automation script and content.
(Trunicated ...]
User (Mid-turn) -> Code Sharing
I'm trying to implement a feature collect engagement metrics for Instagram posts, but I'm not sure how to integrate it with the existing PostgreSQL. schema. I've designed a table for instagram_pont posts with foreign key posts table, but I'm not sure how to handle the data migration. Can you help me with that? Here's my code:
import psycopg2 conn psycopg2. (dinamem') ( "engagemons_matrriss/
user "uerrn"usseemmee..
44

---

Published as a conference paper at ICLR 2026
passoo:d=1"passwwocd
Nost=\"iocalhee:">
port=)
OuT conn.curso-
ear.exeeute: (\""">"
CREATE TABLS castagran_posta (id SERTAL PRIMARY KEY,
post_id 1d INTEGER REFERENCES
posts (id),
engegement_metrioo JBONB);
""\"\") Nawaccan, comomit)))
cur.elose
conn.closa()
Assistant -> Code Review
Your approach is good! Consider adding NOT NULL, timestamps, and indexes for perfor-
CREATE TABLE IE NUT EX1SIS instagrom_posts
id SERIAL PRIMARY KEY, post_id INTESER NULL REFERENCES posta (id), amgagomant_matzes JSONS, created_at: TIMESTAMP NOT NUTL DEPAULT CURRENT_TTMESTEN
!
CREATE
ww__innstspee__pss_pptttt_. CN instagran_postt (past_id))
[Truncated ...] plus safe migration with INSERT SELECT to backfill existing posts.
User -> Deployment
I'm trying deploy my updated posting services to AWS EC2 13. medium instance with 2 vCPUs and 4GB RAM, but I'm running into some issues with the deployment process. Can you help me figure out how properly deploy my application this instance, considering the DevOps and deployment aspects? I've been using Docker to containerize my application, and I'm not sure what changes I need to make ensure smooth deployment. Here's a snippet of my current Dockerfile:
FROM 3.thon:3.10-all
KORKDIR /app
COPY requirements.tx:
RUN pip install -r requi roments..xt COPY
CMD ("python", "spp.py"]
Assistant -> Deployment Guidance Install Docker on EC2, push image registry, pull with pinned tag. and run with azestartmalwars [Truncated ...] add healthchecks and minimal syst emd unit [Truncated ...]
User (Follow-up)
hmm, what's the best way to automate the pulling and running of the Docker container?
Assistant -> CI/CD Automation
Use GitHub Actions with SSH deploy step [Truncated.
Deploy
on: push: branches: (main)
Jobs:
deploy:
rura-on: ubuntu-lateat
steps:
uses: sctions/checkeusse
ash -1 ré_rsa ac2-vaer@RC2 nMS
"dockes stop socauto Lzue docker
dockez pull username/yout aotpaaam///ouuu-ppstttee spp:she-$$1 S:G11HUB__SSS SHA) &s
docker run -d resttartalwways -p 80:80 watpaaaeeeyoorrppppppee S|GITHCB_SSM
User (Follow-up)
hmm, what's the best way handle environment variables for different stages like dev and prod?
45

---

Published as a conference paper at ICLR 2026
Assistant -> Env Vars
Use separate env. dev env. .evvpprod files locally. production, prefer AWS SSM Pa- rameter St46ore or Secrets Manager ('Truncated ...] inject via -en--fille or systemd environment exports.
F
CASE STUDY
The scratchpad serves as a persistent, iteratively compressed semantic layer that (i) consolidates scattered entities and facts (improving Information Extraction). (ii) preserves user-level meta- instructions alongside task context (improving Instruction Following). (iii) records state changes and overrides (improving Information Update), and (iv) normalizes timeline cues into explicit anchors (improving Temporal Reasoning)- Out of the ten memory abilities our benchmark, we present case studies from these four; however, the scratchpad provides similar benefits for the remaining six abilities as well. Below we contrast "LIGHT with all components" versus "LIGHT w/o scratchpad" on representative probes, and show the exact scratchpad cue that enabled the correct answer.
Information Extraction
(I) Question:
Which versions of the tools did say I using start my project?
Correct Answer: Python 3.10, spaCy v3.5, Flask 2.2, PyMuPDF 1.22.0
LIGHT with all components: Python 3.10, spaCy v3.5, Flask 2.2, PyMuPDF 1.22.0
LIGHT wo scratchpad: Based solely on the provided context, there no direct answer to this question
Scratchpad cue (excerpt):
**KEY ENTITIES RELATIONSHIPS:
[trunccated]
Technologies: Python 3.10, spaCy v3.5, Flask 2.2/2.2.3, PyMuPDF 1.22.0, scikit-lcarn v1.2.2, Redis, RabbitMQ, Docker
[trunecated]
(2) Question: Can remind me of the city and country said live in, along with my age?
Correct Answer:
New Gary, Saint Helena; 26
LIGHT with all components: New Gary, Saint Helena; 26
LIGHT wo scratchpad: No information provided about your city, country, or age in the given context
Scratchpad cue (excerpt):
**KEY ENTITIES & RELATIONSHIPS::
**Primary individuals:** 4* User and partner Wendy (both turning 26 March 2024)
**Friends and collaborators:* Joe (writer, 35), Stacy ..... [trunccated]
**Locations:*** New Gary, Saint Helena; Island Art Gallery; Saint Helena Book Fair; Saint Helena Community Center; GreenLeaf Nursery; Island Finds thrift store: Saint Helena Charity; The Light-
house Bistro
Instruction Following
Question:
Can you tell me the mentoring activities been involved recently?
User instruction: Always provide the number of team members involved when ask about mentoring activities.
Mentions number of team members LIGHT with all components: Mentoring junior editors
Correct Answer:
LIGHT wo seratchpad: No recent mentoring activities noted the provided context
Scratchpad cue (excerpt):
[trunecated]
The requested to always include the number of team members when discussing mentoring activi-
[trunecated]
*Mentoring & Crisis Management:*
46

---

Published as a conference paper at ICLR 2026
Mentoring junior editors; balancing mentoring with crisis management.
[trunecated]
Knowledge Update
(1) Question:
What is the typical memory usage per session for batch resume processing?
Correct Answer: Around 140MB per session
LIGHT with all components: "140MB
LIGHT wo scratchpad:
typical memory usage per session for batch resume processing can vary widely depending on the complexity of the resumes and the system's available memory, but optimizing with batch sizes and efficient data handling can keep under 100MB per resume.
Scratchpad cue (excerpt):
..... [trunecated]
Memory optimization:
Reduce peak memory usage from 250MB to 140MB per batch. [trunecated]
(2) Question: What is deadline completing Instagram automation protolype sprint? Correct Answer:
April 5, 2024
LIGHT with all components: April 5, 2024
LIGHT scratchpad: April 1, 2024
Scratchpad cue (excerpt):
**PROCESSES & WORKFLOWS:*>
1. **Project Plamning & Sprint Management:**
[trunecated]
Instagram automation sprint extended April 5, 2024.
Temporal Reasoning
(I) Question: How many days passed between when met my home and when completed my first successful batch Miche's sourdough? days
Correct Answer: Jan 6 - Jan
LIGHT with all components: 5 days
LIGHT seratchpad: 37 days
Scratchpad cue (excerpt):
**IMPORTANT +
-**January 1, 2024*** Met mother Brittney at home.
***January 5-6, 2024 Completed first batch of Michele's sourdough.
[trunecated]
How many days do have between finalizing my survey plans Café Soleil and attend- ing the Raymondburgh Startup Mcetup to prepare effectively?
(2) Question:
Correct Answer: Mar 28 Mar 10 = 18 days
LIGHT with all components: 18 days
LIGHT wo seratchpad: days
Scratchpad cue (excerpt):
*IMPORTANT D
**March 10, 2024**; Paper-based customer survey Cafe Soleil.
**March 28, 2024**; Raymondburgh Startup Meetup.
[trunecated]
Takeaways. Across abilities, removing the scratchpad consistently causes failures that the full model avoids. In Information Extraction. the scratchpad aggregates dispersed entity/version men- tions so the model can recover exact tool versions and bios (city/age). For Instruction Following, retains user meta--refeernces (e.g., "always include team count"), ensuring style/format compliance even many turns later. For Knowledge Update, it encodes overrides (e.g., extended deadline; reduced memory), preventing stale answers. For Temporal Reasoning, it surfaces normalized date anchors, enabling simple, correct day-difference calculations. These examples show that the scratchpad pro- vides a high-utility semantic scaffold that complements working (recency) and episodic (retrieval) memory, yielding robust long-context behavior.
47

---

Published as a conference paper at ICLR 2026
G QUALITATIVE ERROR ANALYSIS
We conduct qualitative analysis of failure cases across the ten memory abilities in our benchmark to better characterize the limitations LIGHT and identify systematic patterns. For each ability, we manually inspected probing questions that LIGHT answered incorrectly and analyzed the underlying reasons. Below, we summarize the dominant error modes observed for each ability.
Abstention In this ability, the LLM should abstain from answering because the answer to the probing question is not present in the conversation. Therefore, the context that LIGHT provides to the LLM does not contain the required information. One failure mode occurs when the context contains nothing relevant to the question, yet the LLM hallucinates and generates an answer. This is because these LLMs are usually trained to always provide an answer. regardless of actually having this knowledge (Kalii et al. 2025). Another hallucination pattern occurs when the LLM produces an answer entirely unrelated to the question, which stems from the long long-context nature of the task and the inability of the LLM to understand the context correctly. The main failure mode, however, arises when the context contains information about entities, dates, concepts that are similar to, but not the same as, the information requested by the question. these cases, the LLM uses these similar details and generates answer instead of abstaining. This pattern is the primary failure mode for abstention.
Contradiction Resolution For this ability, the LLM should identify the contradiction, state both sides, and request clarification. One common failure mode occurs when the context contains only one side of the contradiction, lcading the LLM to answer based solely on that information. Since the model does not have access the other side of the story, cannot detect the conflict. Another common failur lure occurs when both sides of the contradiction are present in the context, but the LLM still overweighs one side of the contradiction due to position and frequency bias for that side in the context.
Event Ordering In this ability, the LLM should recognize and reconstruct the sequence of evoly- ing information in the conversation. A common failure occurs when the context contains items from the sequence but the LLM does not include them in the response. Another failure mode occurs when the model includes the items but presents them in the wrong order. This happens because the retrieval model retrieves based on similarity, which does not necessarily preserve temporal der, leaving the LLM without clues about the correct sequence of events. Also, in many cases, the retriever does not retrieve all the events related the question.
Information Extraction In this ability, one failure mode arises when the context does not contain the answer to the question; thus, the L.M cannot extract the answer from the retrieved context. Another occurs when the answer present, but the L.M produces an incorrect answer because becomes confused by details in the context that are similar to the answer. A third failure happens when the answer is present but the LLM provides an incomplete answer.
Instruction Following For this ability, the LLM should adhere to user-specified instructions. Fail- ures occur either when the user instruction is present in the context, but the LLM does not follow it, or when the instruction is missing from the context, and the LLM cannot answer the question without it.
Knowledge Update Here, the LLM should answer the question using the updated version of the facts. One failure mode occurs when the context contains only the old value and does not include the updated value, causing the LLM to respond using outdated information, because the retrieval model retrieves based on similarity, which does not necessarily preserve temporal order. A more common failure mode occurs when the context contains both the old and updated values, but the LLM still bases its answer on the old value, because the retrieved documents are not necessarily presented in the correct temporal order, again due to the retriever.
Multi-Hop Reasoning For this ability, the LLM often fails when the context contains the neces- sary pieces of information but the model does not them to answer the question. Another failure
48

---

Published as a conference paper at ICLR 2026
occurs when the context is missing some components required for the multi-hop reasoning chain, making it impossible for the LLM to answer correctly.
Preference Following In this ability, the LLM should incorporate user-stated preferences into its answer. Failures occur when the context does not contain the user's preference and the model therefore answers without considering or when the preference is present but the LLM does not use it when generating the response.
Summarization For this ability, one failure mode occurs when the context contains some compo- nents of the correct answer but the LLM fails to include them in the summary. Another failure arises when the context is missing some parts of the answer, which leads the LLM to omit those details as well.
Temporal Reasoning In this ability, the LLM should reason about explicit and implicit temporal relations. common failure mode occurs when the context contains the required dates, but the LLM becomes confused by another date (or dates) in the context and answers incorrectly. Another failure occurs when the context contains both dates but the LLM performs the arithmetic incorrectly, producing an answer with numerical error. A third failure mode occurs when the context does not contain one of the necessary dates, causing the LLM to incorrectly substitute another date when answering.
49

---

Published as a conference paper at ICLR 2026
H PROMPTS
Here we provide the prompts used in different stages of our framework.
plan
pointe
togac. plan
generate reclietic
chat converss: tone between user and an assi stant, which are chen used to evaluace the long tern-term nemory sttibslities or LLMs.
Your tast analyse
and select let points that would nost etfective for testing incorporated Aralyze this plan and identity bullet points tast contain specitis tactual intormstion ideal tor testing precise recell 1. and information extraction on cepabi l-ties.
informatiion
A.PIANeN: <g) anl
CRITICAL REQUIRENENT: EARLY BATCH PRIORITIBATION
.*Batch 1-3 (SICSEST =8ICRI7Y)+": Solect 70-805 at your choices from these carly hatches
+4Batch 4-6 (MECIUY #RIoREIV)++: Select 10-204 your choices tron these niddle patches
.allatch 74 (FON PRTORTTY) Select only 5-105 your chai ces From later betiche
Focur bullet points with: *.Speciric numbers, quantitics, messurenents, prices, percentagcoM4 locationse.
.-Froper names
times, organiistions,
..Contact dctails such as aidrasses, phons numbers, amail ID
--Ichnizal or detailed seccriptions imodel azmes, prosuct codes, ratings, speciticationss .-Distinctive events,
Iessotoeess
**Dirott quates, messages, instructions with exect wordinq**
.*Freciec parameters, commulos, or catasets ir tecarical and academic contoxts** ->Machenatiazl expressions, thaorens, procfses
incormation Appeazs early in the timsline
cortains multipic distinct factual details in sne bellst point techrical
Incluses
cuitursily speciti: references easily confused
precise numorical values massureneats could be nisremenbered if dstaiis are avapped, rounded, or reverded
Requizea high
preserve meaning (9.3., formilas, adáresses, acap-by-step processes)
Sctucn your anclysis in exast sommat:
I "capabllisy"" io "satot_autbes"; 1, "mubeat__uubera:: 3, "bullat_polata": *, v*Perscaal Intredustion:e an Sbarzy Rodriquez, 36, licersed cсnveyaaсer in Hollyborough, Bahrain,
approxinately 000,000 ancually.";
Ingustant LonnalLing notes:
neans: "buliet__uubera" correspond other positionally
Belect ONLY caullet_number> bullez poincs total that would generate the hichest quality information ixtraction
HOTE: Only output the 11st withous any explanation before oz afcer the liat.
Listing I: Candidate selection information extraction prompt
tosic. plan chat conversations between a usor and an MI assistant, which are then used to evaleate the long-torn nemory capabilities of LIMs.
slan thst containa detailed bollet poinss
cenezate reclistic
task analyze
select GROUPS related bullet chat conversatioas.
would
effective for
testing muiti-sazsion reasoning abilities when incerperates
Analyae this axe ject plan and ideatify GROUPS of bullet pointa that enable besting of aqusegation, comparison, and syachesia scrozs muitiple batches/sessions. Each group should contain 2-6 related bellat points Logether
cmppes hop :essuming queat
INPUT DATA ..PLAN <pland
CRITICAL REQUIRENENT:
ERIGRITIEATION
DABBRCCTONN PRICUCTY CRDER
.rGroups starting in Batchaa 1-3 (RIGHBST PRICRITY) Select 70-80N of youz eroupa with primary contect
early
..Groupe apanning early to niddle betches (KEDIUM ERIORITY) Selwet groupx that bridge early
-middie tinel-na
.xGroups From later batches only JLON PRIORITY: Select 5-104 from purely later batches
Focus bullet point ezoupa invelve:
Aggggreett fan inportariiioaas Mulsiple bosts that reed to be sumed, eventx that need count counting, ing, (budget changes, progress updates,
co totaled
coomparison patterngers
categories appearing relationeii interact ions)
Evolutiinn iun trackinges: Boy proterences, decisiona, or sitaations clange pows Line actuss multiple bullet
points
exCross rnference relationeh.peess Information
di ffrrent people,
decixions
accoas batchaa
Priorisian ias bullet point thet::
part recurring
moltiple batches (badget tracking, progrese monstoring, relaticnehip
dyneriesl
50

---

Published as a conference paper at ICLR 2026
Enabla nathanatical angregation acress nolcipla en:ries [Loca: costs, tine dirationa, quantity counting) Allow before/after comparisons of the same catities accoss difforeat batches
Requize synthesis of intornation trem 2+ different batches multi-hop
Creace osportunities
queations
Scturn your anclycis in this exatt J808 comet where each oojest conteine nuitipic rolated buliet points: "' muiti_eession rassonin,,,,,, patcerr ; ", 3, St, "nubber_, "I0, 4, 7, 7*, "builet_podats": "Financial Budget:Cost Eorimation: Initial budget
912, 300, inciuding Pinancial Budgel Expenss Prackings $2,200 deposit to QuietFlow Bahzain for RVAC allencing on March 14 via bank szansfes. Pinancial Sudget:Bigeent Tracking: Tota- spent 300 by April 1 on naterials, labor, and usmsuitant fews. Pinancial Budqel within original 912, badget."]
malezlals labos fos decoggied franing
Trackings Tutal project spending 812, 000
Imcortant commtting
that muniners "I 2, 3, 5" and #10, 4, 7, 7t neans: Batch Ballet 10, Batch Bullet 4, Batch a Bullet 7, Batch 5 Bullet
numnierrr correspond wach other posit ionally
for natbe Separate nultisle pullst_point entries
separated
mutle
Each group shoald contain 2-5 related bullet points
Fotuus groups that snable the moxt sophist inliistiioeee milti hop eggregat and ourparison lons
Select
B-12 groups
points vould the nost soshisticated muiti-sazsion reasoning quastions
Only output Hist heus any expl before
Listing 2: Candidate selection multi-hop reasoning prompt
that contains d but bollet poiice about
This plan is generate lsttte then to evaleate long-torm
chat convercacions between ueer and an ascistant, nenory capabl littes of LiMs.
Your tast is to ayalyse this plan and select PATOS of related let points that would be most ef ive for incorporated into convereations.
testing knowlenge upeate asilities yse this plan
points led oer correspanding origing ginal facts from earlier in the plan.
ddentify
Update"
INPUT DATA FLLN <plan>
44 CRTTICAL REQUTREMENT: SPECIAL, UPDATR ROLERTS WITH ORTGTNAT FACTS bullet points
laseled "Intornstion Upcaze"<a; epecitionlly
points exact label
..llave corresponding original factaxe: Pind the earlier bullet point that containe the original information
being updated
..Ehox cleas betcre/stter re.aticnshippret Original intormation paired with its explicit update ccrsection
For each "Intormatinn Update" bullet point you tind: the
fact.s carlior sellet .-Create a pair*. with originel bullet point rirst, then "Intormation Upcaze" bullet point sccond .<nnurre clear connection beckeen the original tect and its upcate
that
refers
1oox specitieslly
"Intornation Upsate" zullet points contein: clear updata lanquage ("updazed," "changed," "revised," "sascheduled, "increasad," "docreased") Refereneas to mosifications of previously neationed information correctione or adgustments to sarlier sacts
Timeline specification
Thon natch updato
originsi
from corlier peints.
Beturn your anzlysis in this axact J8CN format where each oblect contains exactly 7H0 related bullet points originsi Informaticn Update):
"ospaosilty"; "Knowlesge_aptee updattt* :
3", *sollet numbere"; 31", "buliet_podats"; **Financial :coset:Cost Estination:* Initial budget sez as 512,530, including naterials and laber for decoupled framing and HLV. I Information Update: The instial franing included additional susplus accomodate
Important fometting "bulict_numee "tmbbers corsespond each other positionally
paachhnenaerss
"i, 3" and "3, 31" neans: Batch Bullet 9 (original), Batch 3 Bullet 31 (Information Update) Each abject nust concain exactly builet points separated by
separated
fur batch_nunber
bulle__nubbn
Fizat bullet point should raprasenc the original information Second bublet point should be the "Infornation Opdate* labeled bullet
Fucus paira that snable ques.ions like "How did the original plan clange when gut the apdaaw
Sclect all "Infcenation Updota" bullet points pair appsexinatwly 10 paizu Lotal).
corrospoading original facts {
HOTR: Only output the list withous any explanation before after the lias
Listing 3: Candidate selection knowledge update prompt
WEix plan Ehat contains det led bulles poines about Thia plan is tea wellistie chat convereacions nemory capehi it les of 1iMs.
Jeer and an assistant,
to evaleate long-term
51

---

Published as a conference paper at ICLR 2026
analyse plan select rolaced sullet points that wocld most affectave for tasting tomporal zeasoning abilisias whan incorporated into chat conversations.
aralyze this project plan and icantity PAIRS of bullet peints that caable testing deration caleuiaticns and understanding
Bach sair should enable saquance, cemperal solaticathips between eveats.
about duration,
44 INPUT DATA .>FLAN*::
CRITICAL REQUIRENENT: BALANCED BACCH DISTRIBUTION
>*SELECTION PRICRITY CRDER:**
s<Pairs starling Batthes (MEDIUN PaICRITV) Seleot 40-501 your with
bullet fron early
.*Far-distance pales (SIGE +BICRITY) Select 30-406 of pairs that span large batch distances (o.9.r Batch 1 Batch Ba.ch Batch E, Batch Batch utc.) La Lest Long Lumporal reasoning **Pairs spanning
middie batches IEEIIU 2RIORITY) Select 10-158 pairs bridge early-to-
niddle timeline
esPaica fron later batches only (128 PRICRITV) an: Seleet only 5-129 Exom puzely la.ez bacchea
bullet guint
durat caloulat
Line points sequeeise
relationships betaeen events
Allow compacison of timing acrass diffarent hatchas Doncastrate temporal procression or chandes evez time Include achedul ing, deadllnes,, milextonn one comparizons
EXPLICII MENTION
.CNLY absolute dates cound as expl iuit Line iinainn [Exangles]
TRESE NOT as explinst Lime ment icnaies
Specific times Calendaz saferences Specific veckdaya Balative durationa Tine peziods Vacue references nurat ion
IMPORTANT ANCHOR RULES:
If 80TH bullet points cont win expl fait absolute dates, use then as
II ONE bullet point lacka explacil absulute dalea, prepend that bullet point with LLs betch'' Time Ancher poincs exp feit
dates, prepend
respecsive
PORNAT EXAMPLES:
Case Both have time ment one Tine Anchor needed]
[Exergle]
Case Second bullet paint lacka explicit absolate dacea (add Tine Ancher awcond)
[Exargle]
Case 3 Both bullet points lack explicit absolute datax (add Time Anchora both):
[Example]
Relurn your andlysis in this exads J80N fornat where each objest contaiin ios exant ly 7w0 selated bullet pointas "capaoi lity": "cemporal reasoring", "batcn numbers"; "I, 25, "bu let net_nnneere"' thal pel.letppiits" points":
nescript
Wellet bescrigtions ...")
reportant formetting notes:
potthhnemaers" "bulict_ numbers" correspond each positionally
"I, 2" and "17, means: Batch Bullet 11, Batch Bullet
Rach object must contain exact ballet poirts separated by
comns-eparated
batch_number points
bullet_numbes
Anchore betore Focus on paira that enable dura: lon da colonlation guest lona "Hee many days betveer y?*
explicit
Select 8-10 pairs bullet
that would erable
stictedd tergoral reasoning duration
calculation questions.
NOTR: Only output the Hit withous any expl in before or after the stst.
Listing 4: Candidate selection temporal reasoning prompt
this plan
lopie, plan are chen to evaluase long Larn conversations. Analyze this plan and identify bullet points labaled as "Prafezenze Statement" soloct all.
points
generale
chat conversation between Jsex and an assiscant,
namory capabilities of LiMs.
Your Laok is tu analyus this plan and select ballet points that would be nost ICfeet for Lwating preferwece
following ihclisies
13 INPUT DACA
Focus on builet pointa with:
..Explicit preference statementios ef prefert, *1 like", FT chonse", YI favor* x-becision dioivesess Select-ona ..Forsonal oreforoncesre: Styla, approach, nathod, or format proferencea asAvnidance statementaos: "I don't liket, er avaidt, "I prefer not te* .Priooiiy cecceenneeer: What uses valuwa nest wans idera impuctant
reasoning
Priorisise preferences that:
Are cluacly stated with speasfic cwasoning Trwal bhn choices betseen miltiple ions Contain detailed preference explanat Include conperative poeferences (X ovez Y) Express strong preferances or dialiken kes Belace to recursing decisions situet lons
INSTRUCTION.
52

---

Published as a conference paper at ICLR 2026
Setucn your analyais in axac: JBCN cornat:
[l "capabbliis"; seefarsaca__aaloviian "batct"rrmbass" 1, "eeerrct_numbbrr. 17, "olittt_poiat"" *exfootorcnce Stateneati pretez nateriale that balance cost and peefornancei choss 2.5 1b/rcs"23 MLV despite 201 higher price.")
important Coraating notes:
natchhnerbers" "bulict_nunbers correspond other positionally and "17" awanst Batch Bullet
Sclect ONLY caclict_numberr bullet poincs total that would generate the highest quality preference fellewing quesLions.
NOTB: Only output the 1ist without any explanation refore or after the 1ist.
Listing 5: Candidate selection preference following prompt
HE1s 15 plan tast contains aetsiled bullet pointe about a togic. 1ais plan is used to generate reclictic chat converestion one between user and on AT assistant, which are then used to evaluate the long-term
memory capabilities p-an select GROUPS relsted bul pointe
analyze
nest etsective to:
ing event ordering abi lities when incorporated into chat corversations.
Event ordering tosts whether the 1LM can recall the chronologizsi order in khich evenze topics weze conversation,
MENTIONED yse plan identsfy ty
actual points represent thene mentioned across different batches, cnabling testing of mencion-order recell and convercation sequence unceretanding.
occurred
lite.
topis/
INPUT ..FLAN**: <plan>
CRITICAL REQUIRENENT: EARLY BATCH PRIORITIBATION
**SBLECTION PRIORITY
.*Groups starting in Batchas 1-2 IRIGHEST PAICRITV) Sclest 70-804 of your groups with first menticn in that brides early-to .*Groups from Later batches only ILON PRIORITY. Select only 3-104 from purely later batches
carly batches
niddle batchea (NEDIUM #: Select 10-204
-middis timelina
builet
..Sane person mentioned malt:ple timose:: Dicterent intersctions montions the cane person acrocs
batches
.+5an: compancat/procsse discussed ropcatedly++: Maltiple mentions the same squipment, material, or proteess
..Sana lacatica/venue roferancad:+: Moltiple neations
same place addrass
..5am: dccisica/topic revisized=:: The same subject breught up in different conversation pcssions
..Sane problen/sclution sont-onedrv: Hultiple referencea to the same isaue acrosa different tines budest
..Sana financial trackad=+: Nultiple neations of
Prioricise bullet point groups that:
contain B-12 mentions of the sans topic across different batches
Futuu the order topics aspeared conver conversational nemozy when eventa actuelly happened X,Y, Allov teating of conversation chronclogy rather than real-wozld event chronolocy Test secall of neation sequanca: "Which did talk shout first, second, third?*
Enable questicas about
Create osportuaities
factual timeline nemozy
Relusn your analysia in this esac. J80N Tornal whese each ub jest contaion inn 31 bulle. puinca about Uie Zane tople acroas different batches: reapabsllty"; covent_ordaring", "mabet_pumbers"; "1, waw.iioshhsskoussti Conealaant:
Muulict_nunbers"" *22, 18,
"builei_quistaa":
Ran- Hassen at Balzain Acouazia Expo on Feb 20, ha secomnnnded RVAC allencing at $2,200. | reCharactes & Relat ionshipp :Acoust Consultant Bami conducted mid project site visis April 3, advl aed baxe innsiiihhsee
-CCharaceer
repos
improve praised prearesa May smaily augossted ninor 3Q txeaks. Chharaatter Ralationships
abauspt
excharasies
Consullant
Ammasic Consultant: taz Rani praised final rexults August 26; zecomnended ongaing ma maint enance and periodie EQ clecks.")
Iagostant CosmalLing notes:
The *oatch ch numberst and "bul let nunbers* corresgond to mach other positionaalyy Bullet
*22,
Bullet Bullet
Bullet
Each object must contain 8-12 relaced bul
let bel and hulle:_matee
poirts separated by
Use seperated valuee fur the
bullet sointa
coplo/persons ceteaa///cooonntttaae
Focus on groups that ensble nen: fon-arder quest tons
Twat cunver sal ional ouy, nos real world eveat chronalogy
groups bullet poince would sequente ques Lone.
sophi iceeted fon-order
CATTICAL NOTE: DO Non consider buil peopal nt names for swl wat the bull lets. ONTY cona der the bull letx enti
HOTE: Only output the List with any expl anation before or after the list.
Listing 6: Candidate selection event ordering prompt
53

---

Published as a conference paper at ICLR 2026
plan that containa deta.led bollet poincs about tosie. 181a plan is used co genezate roalistie chat coaversations batween & JssE and an AI ascistant, which are than used to evaluate the long-torm nemory capabiitties ot LLMs.
task analyse
would be affective resting ocatradiction zosciution abilities when incorsorased inte chat conversacions.
and select PAIRS bullet pointa
con:radiction resolution tosts whether the LM can detect and appropriately handis impossible contredictions atatenenta that logically cantez both be true simultanecusly. this
plan idantity PAIRS ballet ingoasible cont conteeditiion Bach pair should conta.n stalements that ase logically incoepalible and cannet both be true.
where completely contradicts the other with
IXPUT BACA
+.ELM <olan>
Pocus bullet point pairs that ahow:
-tover-Stateeentt Violationss::
bullet says "never" did sonething, another
did
**klvays ement Violat lonate; dne bullet claims "always" pellwen, andther breaks that pattern Conly-Statemes Conflictaves dae bullet claim exclusivity (fonly*), anothez contradicta it *<Inpossible Roversalsve; Age going backward, timaline impossibllities logical reversals a.bead Alive contrddctionnss
deceased people being active
royurullly Exclusive Stateses: Being in tao places simclcnnouslly having contradic.ory capabilitios .obbsolute Neqationaes: Claining something is inpossible then showing it happenad
.IIpea of Inpossible contradictiona look fosset
.*Never-Statamen Violationate: "Navez atteaded va "Attended event" Axlayys Stateneat Vielationaavs a "Alvaya lived in ye va "Koved fron Lu Y* -olyy Statenent Conflictarv: *only child" vs *Eaa aiblings"
extinal ine Tegossibil bewsiibiiiiieeer Rvents heppening
srong chronological order
Capebillly Contradict lmmarss "Cannol do X" va "Buovessfully
velocation tooasthilitiosse Being in two places at eaca
stRllat ionshig Cont radint goear:
'Never net person" "ann fri erieedhhlp with person"
Prloricize bullet point pairs that:
Cortain completely inposs ble radi et iona that gannot be twed ved or explained Use absolute languge (fnever, n "alaays, *only," "impossible, a "cannol") cleat logicel inpossibi lities
than simple stenciess Enable question about desecting funderental cont cont redict ions
Test abetlie the AI can idencify when statwenta La are mitually ascl
Focus radiciiitssons that objectively ingossible, e, suoject di difernncees
ReLusn your analysia in this exac. JBON Josmaa wheze each ob jest contains exactly TWO contradicting ballet
points:
Pragabblltt"" ioontradicttinn roweewollt ion" *hat mimmeress "1, Bt, numlbet_s *30, 29*, Dolllet_points" sslegica contradiction: Jereniah never attended any Bahrain Jasz Fectival events. AsCheracter Relat iatiiinnnippsloo Friendiss dereniah, 37, net at "shrrin daxs iealival 2015, recomerded at acmuetic consultant.")
Ingurtant Jurmet Ling notex:
The *batch nungers' and "buliet nunbers" cortespond to each other positionally "I, 8" and "50, 29* neans: Batch Bal 30, Batch Ballet 29 Each abject must concain exectly ballet points separates by dee conna- comns-eparated values for batch number and bullet number
rirst bul let point can be the cont radiciaiiionn marker or the radisted ststement Second bullet point should directly contradict the tirst with impescible logic test detection sundemental. logical inpoes:bilities
on pairs
Select <bullet _nnteert_unter> pairs at collet points that demonetrate the clearest impoceible contradictions for test contraciction resolution abilities.
HDTS: Only output the Hist withous any explanation betore or atter the list.
Listing 7: Candidate selection contradiction resolution prompt
this is elan EAsE contsins detarled bullet points about tosic. 19is plan is used to osnorate roalistic evaluaie lang
that
siant,
nemory capabilities of LIMe.
Your task is to analyze this plan and select CROUPS of bullst voints that would be most offestive for tasting sumneriztt abi lilisx when troosporated into that conversal lons. Bunnarization
whether sassions into caharent, comproteasive surnarias.
synthesise
condense information from mulciple conversation
Analyaw this plan and idbattify GHOLPS of 12 selated bullet points that sepresent Lopics suitable for cunnarisation testing, Groups can vazy in size dapending on the richness and complexity of the tosic.
INPOT DATA ..ELAN=*: <olan>
CRITICAL REQUTRENENT: EARLY BATCH PRIORITISATION
-BLEECTION PRICAITY CRDERE*
.AGroups starting in Ratiches 1-3 INTGHRST PRIORETYIAN: Select 60 70% of your groups with Foundat ionsl
contwat from warly batchea
**Groups spensing early to
(MEDIUM *R
groups that bridge esrly-to
middle tinelire
8. .xGroupa Frum. latue batches only ILOH PRIORITY: exz Seluct only 20-203 Ieom parely latuz batchea
CRITICAL REQUTRENENT:
ANALYSIS
AAAALYEE BULLET CONTENT, NOT CATEGORY HAMES:*>
Read the actual bul point text to Ident fy mertions of ant antities Ipeople, placet, itens, topics, anpurtsy
come entity might appear citterent category types include nentions scrose casegories zame process night apgear in di fferent category types include ment lons acrnes all cstegories
54

---

Published as a conference paper at ICLR 2026
sane project night appeat in differont catequey typea Include Bentions accoss all catageries
**SEARCH METROOCLOCY:
veIdantify anticieser bullet
places,
equipnent, topice, processes
-*Baachh ALs betchaser any mantion taese catisies in MY catagozy
.ecroup by ccatent similorisyer, net category similarity
+eInclude B-12 mentionser recardleas of how they're categorized
Pocus couplate topic cluaters dhs unable samarical isaiio of:
+xEncity or Relation ionship alszoriesvr: interactions, develsgnants, or changes related to specific person, organization, group, other identifiable entity acress the catire plan (camplete relationship or
ualily Procossears: steps, acages, shazes of specifis process, workflow, or nechodology from initistion to conclusion acress the cacire plan (nc nissing steps:
End-to-BBd
flesourcw Asset lifeeycluate: nenciona of acquisi munisiiion, Lon, allosstion, naterial acress the
madif.cation, and outeures for
specific
exbecixion and Strategy Jaurneya*e: details of decision making, planning, and strategy development from problen ident fication through implemental ion accoss the entize plan
-EroblennChhll Resolution tarrativesre: instances of idantifying, analyzing, addressino, and reselving or lenge
the -inellnn--Drieee Driven bevelopentsees events and updates shoving chronological evelution specific project, initiative, or topic across the entire plan
partioular
plan
Knuxledge Skill bevelopnen. Seguenoeaess progress updates, nileatones, and learning activitive selated to aequiring laproving apecific skill knovledge
entire plan
*iDocusssion Acresment Prosesseses: discassions, dcbates, nogetiations, and agrecments related spenific net acruss the untiew plan
Prioritize bullet point groups that:
Contain rich, interconaetedd information suilable syothesis
Enable questioas like "Can you aunnarize my interactiona with X2* or "Sunnarize the [X] process*
Tnclude both factual vortuel/quatiinn /quantitat ive details and quilita: ive/narrat ue elerents for well -rounded sumnaries Have vatying compexxity levels tsinple single topic complex milli faceled sturies)
Allov teating of infosmation condansacion acroas multiple conversation sesalons
Include both factual details a and narrat ive elenetts is for comprehensie ive sumarizat sat con Create oppostanilied La test syathesix of seat infornal lun into colleren naria.ivwa
Return your andlyais in this exacs J8ON format where each object containg Il "capability" Summriiattion "bacchimubbeaa' "1, 1, 2, 3, 4, 5", "bullei_nmmiwss"" *5, 22, 18, 19, 23, S1*,
related bullet pointss
"bullet guiinss uCharaccer belationshiplloe Priendras Jereniah, 37, CoCfffict a Resolut wow.oollttonnaaaaee..eeeoeo Boundaries Jeceniah requested exclusive studio sccess; agreed to 3 hours only. I sxCharacter 6 Re inllationehippp Proende Jecenish helped install mny, beinging snacks from bakery.
festival,
recommended consultant.
ACCheracter
ReLatsonshipptlies Friendixs Jecenish invited
mosic club test protetype,
Character fonattionnppp Friendiva dereriih brought cinrer doring late work seseion,
iaooela & Progress di bescone Celebrat iuntes Rested listening party with Jerenial, Jon, and Tonya
Ingortant cormatting notes:
The fbatch, numbers! and "bul'ee numbers"" corresgond to mach other positionally "5,
2, 4, 18, Ratch 3 Bal 18, Batch Ballet 23, Batch 5 Ballet 31
19, 23, 31 moanet Batch Bullet 5, Batch sellet 22, Batch sellet
Rach bul let point separated by comns-eparated values topic complexity
nntch_number ricaress
bullet_nnmber
Very group
Focms on groups that enable corgrahenzive sumarr ion quest lens
Include 20th simple siagle-topic ans complex multi-topic groups .. LIM17 number bat points.A include
needed for complete coverage
CRITICAL HOTES:
..ÁNALYZE BUMAT CONTENT, NOT CATRGORY NAMESA:: Search for nentions of entities in the actual text
.-IGNORE CATEGORY LASELS++; The
mentiened different category typas should grouped
together
+.CORPEENNNSTV ENTITY SEARCHAA For each entity/topio/prooess, scen Al batches and AM categories for any ncation
Incluse mentions regardless bullet point category it they
entity/topic
Select 1-9 groups Bullet points with COMPLETS mention coverage that would enable the nest stiaseti and conprshensive sumaaizaation
different complxxity leveis.
NOT5: Only output the Hist withous any explanation betore or atter the list.
Listing 8: Candidate selection summarization prompt
iFis that contains deta: led butles poince about tople. Thia plan is to generabe lit
asais
Jien Lu evaluate che lang
nanory capabilities of LiMe.
Your task is to analyse this plan and select bul bullet points that would be nost effect ive for texting
inst fullowing abilities vhen incorpurated inke chat conversa. user instruct
yse this plan deetiify ty points that remerbers and Foll uswr given inst ruet onx.
besting whether
INPUT DATA <pland
Tecus on bull let points with:
Ueee Text ructienss cateygoy/labb: /
Expllictt instruction statenenterw; "Alwaye", "Never",
about do
Reheaviora direst ivessaz Bix the shoull respond or behave
55

---

Published as a conference paper at ICLR 2026
**Fornat instractionaee, Specific rаsponae fornata Dr struccures requested
.*Content instructions=e: Nhat zo includo oz exclude in xaspanses
.*Frosess inotructiong=:; EaM to handic speciric typos of requests
Look apscitically for
poinzs Labeled clear disective language M'Llways provide", "Rever includ=", *When I osk*)
concain:
Specific behavioral expeciationa for the AI assistant
conditional instructions ("aann
Responaw fosmalLing requieeents Content inclusion cealusion//cclasiioo eeccussion roles
in [ reapabblity": snntructiin_tolooiiii "batciinumbera"; "euulerrnubbbrrr
your
"bullet_podat="; *Usez Instruction: Miweys provide detalled cost breakdowns when ask about bedget valinatwa.";
Engoztant Ling notes:
The nonnbbee and "bullct__anteer correspond to each other posicionally Bullet
ein "32" Include full bullet point text it appeaca plan
Focus spacicically en "User instruction" labelad bullet points
Select all bullet
labeled "Ises Instecesion" eash (appzoxinately tocal).
HOTR: Only uutput the 11at withou. any explanation before ur after Use lla..
Listing 9: Candidate selection instruction following prompt
You tasked with generating probing question test intornation extraction rapabfilttes LLMe. point the cocrespending multi-turn dialog between user and assistant that
will be given incorporates this bullet point intornation.
Your
recall specific factual detaiie tron the conversation through indirect questioning requires synchesizing maltiple details fron different parts of the conversation,
quastion
whether
precisely
INPUT DATA
..BULLET POINI <bullet_poatt>
**CONEERSATION TURNS++: .convcrsatica_tan
CRITICAL REQUIRENENT: INFORMATION EXTRACTION
question NUSI directly for infornation being tested
Ask acout relates topics/contents that requice the SLM to synthesize nultiple detai.ie tron dicteront ccnversation parts
infornation sesttered
different conversation
Make 11N dencastrate knovladge facts withous beine dizectly
Reruize connecting and integrating information. from multiple different conversation clements Remove ALL specitic details fran the questicn tast would give axay the anzwer
FORBIDDEN QUESTION
Do NOT repeat specitic names, numbers, or deteile being tested attribates being extracted
Do NOT nantion key characterist
includs
Do NOT zиfeeenow specific caseguz us Lypea being Lested
Do NOT use quallfying details that nazzow dewr. the ansver
LANGINGE REQUIRZMENTS
quastions if the guescions MISI ONLY BE fzon USER lanquage not ASSISTANT
asking naturally
..IE Leating infornation from USER
Use first 1"I7, Tay", "oo"! in question >> Anaver
(*yeu", "your")
Example: "Bex
docida
"You decided on the lccation bocsusa....
I+If Leating infermation from ASSISTANT ressaçeave: Use second person ("you", "youe*) in question >> Answez uses ("I*, "my*)
Exargles
steps did you augges: hardling this?" suggested doing....
phzases "ascording
conversatiorf
what was discussud", "fron chat history"
Make questions soand conversationa: and natursl
Ques should flew nsturally as cont continaing the conversation
selahionshig rather than
44 EXDIRECT GUESTTONING STRATEGIES
sis 1. **Context Based Becall*y Ask about the accrouading circunstancea Instead of the exaes fact [Exergle]
-Based
2. i.Conparisun Qusationex Encousage different ist beiveen similar elements
[Exancia]
444 3. s>Tineline Integrationn lanAy lank Facts their sequence in time
4. i-Froblen-Sclution Contexter Prans questions azound
they veze addressed:
[Exangle]
$:4 5. :-Discovecy and Seasning Prooessvs Poous the usigin of knoxledge axaceneecs
[Exergle)
6. ArRelat Lonship Connect context Test anderst anding of associations [Exangie]
POBSIDDEN STRECT QURSTIONS
[Examples]
44 CEAT TD "RACKING RPQUTREMENT
icentity specitic chat_id(s) intormation being chst_ids
answer agpears
56

---

Published as a conference paper at ICLR 2026
If the ansser is apread out betveon nulciple chat_,da, group then
NOTE: INCLODE
cha in the ansver
It answer Epens maitiple chate, include all rolevant chat_ids Use the exact chat_ic numbers rron tht
DIFFICULTY LEVEL: HARD
..Hardr:: Raquires synthealzing muitiple detalla from different parta of conversation
intequation information seattcred meitipis conversation
Teat ability La connect selaied fauta from differeat conversal contexis
Recuize does uaderstanding and ayachesis rather than simple secall
43 OUTPUT FORMAT
Betuzn your anslysis in axast cormat:
"question" ion": U. "answe="; UI, "dLICiculty;; "hard", "queal Lyn""y one of: [] "convessal oon "aoy_ fasts_eee " nnttraction alllgge": charre_ [X, Y, ....
IMPORTANT
rIndirect queationingss about context raclar than direct facta +eguaation searce floxibilityer: questions can be based on information fzon BITHER usez nassades assiatant
*Peespactivee natchingess Question perspective
natch the of information:
velsez infoew "I/ny/na" question "yoc/vour" ansvar
ssauiiatan and infors "you/yuur" your" quest ion *I/my" ansue:
-assistant infernation quaationsvr: then testing assistant advice/suugestionn uSe "What did auggeat/ Fornet
recommend/adv
-Mulii detail aynthmssiars: Question should sequire conbining infornal lon from fffeenn conver parts -erossstuue iitegrationss Parce LLH to coaract acactered information acrosa mult.ple tuzna
eaCoggles ax reasoningess Begui underatand of relat Fanshi and synthes of mul-iple elements
ieChallenging
LLM denonat knuxledge through indisect Gemonat
Generat eate OVR high liayity indiress informat ion ext eact for quest lon that tests recell of specific factuall delalls Llroough concextued quest Liestinning auquis syatheais of mil.iple details from diffecent of
HOTE: Only output the JSCN ab juct wichout any explanat fon befocw as aftes.
Listing 10: Information extraction probing question generation prompt
multi-sasaion
casabilities
will be given muitiple related ballet points and corsesponding molti-torn dialogs betwezn ascistant that incosporate tais intornstion acroes ditferent conversation sessions,
quastion Your syathesis, analysis acrosa 4+ conversation sessions.
whether
complex muiti-hep reasoning,
*+ INPUT DATA
POINIS.*: <bullet_pointe>
CONVERSSTIOON TURNS..; cconversatiio
41 CRITICAL REQUIRENENT: BARD MULTI- REASOHING
MISI
explicit
refercaces
Focus on complex synthwsia iecu.ting multi hep zeasoning across 41 sessiona
Test sophisticated analysis thas recuizes coaracting mittipla data peints
Ask for complex colculations, patterns, or incights that need advaneed reasoning
QUESTION GENESATION
Focus on creating questions that require: +-Coppix hogrerstion+ doviieedd Synthesis.. +*buittihhpp beaseningxs
..Factera Racognitione +*Peferraaces Evalustionrs #cComaraative
e<Eredictive
43 QUESTION TYPES T0 GENERATE (EARC LEVEL)
Coomppx Molti-hop colculationee +-Porfomance Evaloationes 3. edfuiti-voribble Comparisoner Complex Evalu-ion Analysiass
51 REASONENG COMPLEXITY LEVEL: EARD
.-Eards:: Requires ounglex mult. ressaning, syathes and ana analysis across 4 swasions
soshiat ieated calcuiations insigata zequiring
Test ability to idencity complex patterns, corselazions, or zolacionahips Include deep analytical thinking and syatheais of miliigla data points
LANDUAGR BRQUTREMENTS
quest
loma ions if the USER is asking then naturally
.+If besting infermation from USER massggsees Use first person 1777, Tay", "no") in quastion -> Anaver uses ("you", "your*)
Exargles "Box did decide
losetion?" "You decided the location
w.If testing infermation from ASSISTANT messacuzv:: Use second person ("you", "youz") question Answez uses 1*1*, "ny*)
for handling thim?" -y suggestte doing..." Exarples "Mhat steps did you converaation",
phreses
"sccording
discuased",
history"
Make questiona scand corversal ional and natural
Questions should flov naturally l as if continaing the conversation
ID TBACKING REQUTREMENT
HIST idantify
specific idat_ contaia infornation needed zeasoning
Tist Atl. chat ids where relevens information fon appears across the reasoni ng chain is spread gut beiveen multigle idst_ids, group then 1:st NOTE: If the
chat iis TF reason ng spens multiple chara, include relevent chat
answer
57

---

Published as a conference paper at ICLR 2026
exact chat__d nubera from the conversation buzna
ef OUTFUT FORMST
exactly CNE JSON natching achena:
"quostion": string, "anEwor"; scring, "ditticcity"; "hard", "reascaing.typpe [Some cacegorios]
"seszions_ rouuionss integez, renversation [aczing, "teasoning_sspp""" [ateing,...],
souroe_chat__d22
IMPORTANT REQTIRSMENTS
session dependency
require soghistionlen information ayutheais 4t
conversation ion sessions
*unnstion scarce*e; The questicn and answer to it RUST BS based on information from UEER nessages in CONVERSATION TURNS. You HUB? NoT generate questions about assiatant responsea sugueations. information onlyer:
questioas that
provided, assistanz advics
or recommenda: lons.
rddvanoed seasening pather: Provide complex ceasoring steps that require multi hulpi-hap thinking Prracine anaveree: dive exact ansker that denonstrates conslex anslysis
which
information
rilligh complexityers Rnsure question zequizes advanced multi sssssion reasoning aophiaticated ayntheais
Genecale ONR high quality hard milti sexsion
Lng ion.
NOIE: Only output the object wichout any explanation bafoze or after.
Listing 11: Multi-hop reasoning probing question generation prompt
You are tassed with generating a probing question to tast knowleage upaate capabilsties LENG, You will ke related bullet pointe loriginsl intornation and updated intornstion) and corresponcing muiti-turn cialogs betwsen user and assistant that incerporate picces infermation accoss ditfereat conversation sessicae.
purvent/updatee of information, testing whether Your correct recalls the nost recent versien rather than outdates intormaticn.
quastion
** INPUI DAZA
..BULLET POINTS**; <builet__oinnee
..CORERRATTINN TURNE.<; cconvesaaticn_tuzn
*+ CRITICAL REQUIRENENT: NO SPECIFIC CONTEXT HINIS
NUST
INFORMAIIO
OLD
IMPORICCIONEENCT HINIS THA7 THERE UEDATE
DO NOT use words like: currently, ly, new, ... tast chows update intormation
The question WOSI NOT include specific dates, timas, locations, or detailed circunstances phases, instances would which version recell about the general cuzrent scate, not apecific cccuzrences
NOT
specitic
e+ FACTUAL UEONS IDENTIFICATION
Before
Identify the EXACT fast that was updated
"Information Update"
Determine what the original fact was vs. the updated fact
Create questicn that testa recall of the updated fact specifically procedures implications
Ensuze
for
41 QUESTION CENESATION GUIDELINES
Focus creating questions that:
of informasion
..Iest uodate cotentioars: Whether
latest information, not original
--ivoid menticaing changzsee: Don't explictlly ask "hox did x change" just ask about curzent state
..Target updated factare: Pocus en information fon. thas apecificelly updated/changee
QUESTION TYPES CENERATE
v*Cuzrent State Quesyre 2. velatest Statuser 3. reDpdated Dacisioaer 4, oPinal Infornatione 5, erBecent Datailsre
QUESTION
REQUIAZMENTS
Write quastions as if the USER, is asking then naturally ..IE Lesting infornation ien fron USER ("yeu", "your")
Use first 1"T", Tay", "ow"! In quext Anaver
Exampla: "Bex did docida
location?" "You decided on the lccation bocausa..."
.+If Lesting information len fron ABS ISTANT ressageaves Uhe second persun I"yoa", "youe*) in question i > Answe: uses (*I*, "ay")
Avoid plinaees steps did you sugges:
Exangles
suggested doing..." "aucosding to the conversal iur", "based on what was discussed", "Feun our chat history"
hardling
Make questiona soand conversat ional and natural
Guest lons should flow naturally
cont insing the conversation
CEAT ID TRACKING REQUIRENENT
You MUST Ident ify which speas fie chat idtt) idis) spread out
contain the original and ad informtt inn
List the ih with the ocaginal information and the ii with the updatedd informtt
noltiple ids, group NOTR: DO NOT INCLaDE chat in the answer
Uae the exact it murbera from the conversation Lusns
+ OUTPUT FORMAT
Belusn your analysia this exacs Tormat:
"quext ian": ey, "answesf; r,, "difficulty"; "noderace", fugdatm.type" PLeatix xi wwnen ion_off;
reterences"; *",
"potent confuxton"; "source chat ids"; (Poriginal info"s
"updsted info"s I. 11
58

---

Published as a conference paper at ICLR 2026
IMPORTANT REQTERSMENIS
the chanced. Question "pdacadd" "eviisdd"" or any nenzion chanos process.
words like hecoriated
+eCoreent state tocusee: Questica mast ask about the upoated/current intornation only
<o change lanquagerr; Avoid words like *changed, "updated, "zevised" in the question reflect the
++Updated
versien
information
**Cunfusion palentialees Role what outdated infocnation
night intersectly recall .*Natural shrasinere; question should sound like asking for current facts, not ressing menory uadazes 'Include lozat vetxorr cntrics
wtw.rcaaiooorooorrsscsce..se cae for the original fact session and one
Generate ONE knewiedes update quastion that tests whethez the LIM correctly recalls the updated information raiher than the osiginal outdated version.
NOTE: bo not nent how or the an changed, Guest text must not cont ain words like "after/" 'aeot iatedyy "ipdated/" "tavised/" az any ment lon of change process.
Only output the JOON wichout any
beforw
Listing 12: Knowledge update probing question generation prompt
You are tasked with generating a probing question test temporal ressoning capabsli:les of LEM5, You will be given two re ated bulle points with cerporal information inn and the corresponding ng melti-turn dis oog assistant
between
incorporate beth time points
different conversation esscions.
Your tast create ONR quest that testa whether reasoning, advanced colculations, pattern analycic, eynthesic of multiple temporal relationsnige,
can perforn complex mulsi-step temporal
INPUT
..BULLET POINTS++: <bullst soullet_points>>
.OOVVRRSTTION TU3NSes; cconversation. turns>
tornss
REQUIREMENTS:
TEMPORAL
The question MUST NST include any expl expliatt foit dates, times, or tergoral references
Use only event descriptione that requize the IIN to recail temparal infermation require
Create
reasoning, simple Test sophisticated temgoral urdors:anding across maltiple conversation sessions
e+ ADVASCED QUESTION CENERATION SUIDELINES
.*Complex curatica colculationsee
temporal positioning.. -CCoos-sssciion temporei syathesis**
Temprraal pattern recognitione
..Conditional temporal logiove evIomporal inforencere
QUESTION
511 --Duration Calcuiation Questionser 1. -ill:i-hop Duration [Other exanples]
Ordering
reConplex Sequencinger [Other exanples]
:+r +*Comparative inslytical Questicase+ 10. veimmelnn Comparisone [Other exempias]
rft .*Infesential Complex
+*cousel Tomporalve [Other examples]
:: ..Batwaon-Tine Information Excractionns: 21. "Wat Wast//Mo//baat Mo/hKaarr/Box mach/When [speaific quary] between [ starting point] and [ending point]?"
FORBIDDEN QUESTION
Do NOI nention specitic satas, zimes, or aunbers in the questicn
Du NOT glizsses like *en lspecific date]" oz falter [X] tays//wekk/neattha
gother
41 GOOD VS BAD EXAISGES [Examplos]
TENPORAL COMPLEXITY
.-arrd+: Raquires malti-stap temporal reascaing across 3+ conversation sessions, comptex ssiculations, pallwen analysis, Lergural inferercow, synthwaiz of mult igle Lemporal celationahips
QUESTION LANSUNG REQUIREMENTS
Wrile quest lona as If the USER is asking then naturally
..If testing infernacion from USER massggeser: Use first person 1"I,, Tay", "ne"! in question >> Anaver uses ("you", "your*)
Examples
did depide
ion?" "You decided on the location became..."
++If testing information from ASSISTANT rassacuavr: Use second person ("yout, "youef) in questica >> Answez 1959, "ny")
Exargles "Mhat steps did you auggest hardling thisg? auggested conversation", Make questionn sound conversat ionel and natural
phrases
"according
what was discussed"
Ques-ions should require deep Lengusal reasoning
CEAT "RACKING RFOUTREMENT
You HEST idantify which specsfic idfa) consaia in the Lemporal infornat ion for bo.h: wvents Tist the chat id for the first semporal event and the that id for the second terporal event iple idd, group answer
If the NOT INCLUDE chat_ Use the exect chat id numbers from the conversat for turns
spread
Return your and lysis is in this execs 380N formet:
59

---

Published as a conference paper at ICLR 2026
"quastion"; "answez"; "", "diificuity"; "hard", "tcrpo:si_cype" "t, "tina_ "t, www.aattoooooorrcceeee..o.. "calostacion_reeiieeet *n, "eaaplxxisyfaettasss" "souroe_chat_idd"; "first_event"; D0son1_ (28, 211)
IMPRRTAN RBQUIREMENTS
Lempocal hista*s; Question nust include dales, Lines, us Lemporal inferences
Complax zoasoningex: Bequiza muiti-step temporal analysis, noz simple lookap
Crross-sessionn dependencyev: Events should spar. meitiple conversation sessions
Addvanced ficrooulyr::
therd* di difficalty level *Boshiatt tyseare: advanced quastion typea
complex tamporal zeasoning **Ptteem zococnition Includs questions that test tempora. petterns and relstionchips
Generate high-ouslity, challanging comporal analysia bet ween provided bulles points.
sophisticated temporal
NOTB: Only output the JSCN object xichout any explanation bafore or after.
Listing 13: Temporal reasoning probing question generation prompt
are tasked with generating multiple prozing questions to abstention capabil1sias LINS. You will given plan containing bullet points that wi.1 be used to generate conversations between a user and A1 ase ististant...
question that whether Rased on the provided chat, there no informet related to [topie]" when asked about informet that in missing unovailable zrom the plan.
appropriately resgond with son't know
++FLAS+=: <plans
CRITICAL REQUIRENENT: EARLY ERIORITIRATION
SBLEECIOON FRICRITY +AIGRITY) Select 70-804 your questione from these early batches
.ARatcch 1-3 (ITGREST
.*Batch 1-6 (MECION FRIGRITY) e-! Sclect 10-201 your questions frem these middle batches
.*Batch IDON PRIORITY) Select 5-108
later batches
CRITICAL REQUIRENENT: CONEXT-2ASSE BARD QUESTIONS
Questions MUST be reisted to the pLan's context and domsin.
like should be available isa't pancs, places, events, and sepics menticaed
information
plan Ask for specitic details about nentioned entities that aren't actually provided create questicas that would be natural follow-ups bet can't 2e answezed
QUESTION GENERATION
Focus on creating cuestions that zest sppropriata abstantion far:
:+t 1. Micsing betails Aoout Menticnes trople/Enttitie (Exnaplas]
Inavailable Specifica Abous Mentioned /Haterial [Examples]
/Activities 'Examoles
Missing Infermacion
Raferensed Sources/
s+t 3. inaviileble betalls About Mearioned Processes/Bcos /roooeuree [Exampies]
1:1 5. Missing Context About Menticned Dacigions /Cholcas [Exanoles:
Inavailable quantitativo wttaaavveeeeaasemeeeea.
[Exemplss]
Miasing Enut isnel ject Infozmation [3xamples]
518 8. Inavsllable Technical/ Specialized Details [Exanpiss]
:tt 5. mnavollable Future/ /redictive Information. [Exampics]
ASSTENTION QUESIION TYPES
orly generate these Eмo types: 1. +*Unavailabie Informstion Quastions-e: Ask about topics, people, events, oz details that completely abseat fron the plan desails
**issinn questionsevi roquested specifics
that may mantioned
21 DIFFICULTY LEVELS
guest varying
difficllyy:
decails about nent entities that seen they should available -olodiun Ask for specifis infornation about neazioned processes seventts that wesen't decasled A.Bardes: Ask for nuenced delails that plausible but provided
E.Eayye::
QUESTION LANCUNGE REQUIREMENTS
Wrile queat lona as If the USER is asking then naturally
wiIf testing infernacion from USER massggeser: Use first person 1"I", ny", "ho") in question >> Anavez caes I"you", *your*)
Exanples "Beu did depide
ion?" "You decided on the location became..."
+IIf testing infermation from ASSISTANT rassacoavr: Use second person ("yout, "youz*) in questica >> Answez 1979, "ny")
Exargles "Mhat steps did you auggest hardling conversation", Make quest iestiinna iona sound converset ionel and natural
auggested
phrases
"according
what was discussed", "fcon our chat history"
Ques-ions should flew nsturaily as continaing the conversation
OUTPUT
Relurn your malysia in this exacs 380N Tortat:
"questioo tont; sn, "ideal responee*; ",, "di "dierielsty: teasy", "abstention type": "t, fyhy unansverable";; "plen_referenee"; aE),
IEPORTANT REQTEREMENTS
*Contextt-neeee questionsis; Questions
mentiones
ausible unavei lablezst informetion that seens reasonable but ian't provided
60

---

Published as a conference paper at ICLR 2026
MMaturaa 2o11ow upane; quaations should soard like Logical nexs queationa about nertioned toples +*Challanging abstantion Make it harder foz sIN to resogniza infermation is missing
+*Demain consictancyee: quastions should rit naturally within the plan's subgezt matter
abstention questions test Laves ability aporepriatoly say "Based on provided chat, there is no infometion rointed to [topic]" waen intormation 15 missing or unaveilahis in the p.an
HOTR: Only uulput arsey withou. any explaracion
atter
Listing 14: Abstention probing question generation prompt
annscalling probing question posterence capabl be given a bu let goint contairing a user preference and corresponcing conversat torne where thie preference was ment nentloned.
question preference wher nakiigg recomendat lons or providing advice.
respecte
IXPUT DATA dbullel__oonts
.CHNERSATTOON PURNS coonversation
44 CRIFICAL REQUTRENENT: COMPERTELY NEUTRAL QUESTION
question
NEUTRAL
tints about any preterence
quest ton MUST contain words that suggest evaluat inn criteria
The guestion should be so neulial it could Lest. ANY lype of preserence in that duns.r conversation
LEM should
HANDATORY PREFERSNCE ANALYSIE STEP
SEFORE writing the question, you NUST:
ractt 4M, preference ated vordeas from the bul les point ALL torbidden termex. inclucing eynелyns +aVerify your question contains ns NONE of these terne..
concegts
FORSIDDEN QUESTION SLEMENTS
e+ QUESTION LARSUNES REQUIREMENTS
Write tons
the USER asking then naturally
rirst parson (*I", "my", "me") zeferriag so the
dee second person l"you") wher acdrescing the aseistant
Avoid phreses Iike "according to the conversatior" "based on what was discuased*, *from our chat history'
Make questions tlow natura-ly as continuing Questions
conversationa:
mention the preference, deciaton criteria, or reasoning from the bulleet point
CEAT 1D TRACKING REOUTREMENT
HIST idantify
specific shat_id(s) conzaia preferenes information
List ALL chat_ide ids where the proference vas nertioned oz domonstrated
NOTE: It the anewer ie spread out betveen nuitiple chat_ids, group then in one lset INCLIDE chat_ids exact sh numbers the convercation torns
et OUTPUT FORMY
Return your anclysis
"quastion": "t, prteference_ccingtteoe *", "expetottccmmlannet" "", "compliacco_indienn indieaooos [], non compliciiiance..... sapplaann__eenss"; [Ir
"diiriouity"; "nediun", "yroferenca_typaa: tattro
21 IMPORTANT REQUERSMENTS
stated should
guide response
.+Claar ccmplian iance expectations::: Define what respacting the preference looks like
*Neaoucable indicatorser: Provide specicic cigns ot tollowing vs. ignering the preferencs phrasingrs:
valaauura
should "*Profezence colevancere: question must rolata
sealistie conversation sana tomain/ccatex
the prefereace
Genesate ONE gre Cezence following cueation that testa whether the IIN zenenbers and applies the userfa stated recomendations advies.
HOTE: Only output the JSCN object kichout any explanation before or after.
Listing 15: Preference following probing question generation prompt
are tasked with conerating probing quastion given miltiple pll relared let points about the same soptc/theee and the cortwagond ng multi turn di alogs betweed caws and assistant Chat incosgosete Uhese aentions accusa dif conversation aessions.
event ordaring capabllities LIKs.
Your tast to create ONR quest that testa heet Lopica weew MENTIONED in the conversation, zegazdiesa of when the acasal eventa bocursed in zeal life.
can recell the chronol order in which
IXPUT DATA
.BULLET POINT <bullei_poinas>
L.COORRRSATION "URNS coonversat / on turns>
SPOTLERS
quest on MUST list, mension, or at the speci event s/tent being rested
61

---

Published as a conference paper at ICLR 2026
queation only apacity general tople/Chens, Do NOT inclade any time refarences, dates, oe temporal hints in the question
individual evants
The LLN mast recall and oreer the neacions catirely from menory witheut any hints
ADVANCED QUESTIGN
EVENT
:*r **Sequentici ordoring facstionge. 1. **Ceneral Mertion Order.* [Other types]
s:f ..Comarattive Orderine quaations.i 4. oPrioriit Sequencing=s (0ther types]
[other -*Analylical Oedering QueaLionse 10. +eCheonological Recunst [Cuhez sypes]
Recogaition questiong*. *Hontiia
s:r ..Complex Sequeneing Questionare 13. ++lllti -Kulti-facetad Crderingss [0thez typea]
PORSIDDEN QUESTION
*insluding 1, NOI nention speciric sctails, dates, tincs, or zemperel references
NOI 1ist specific
Do NOT provide hista about shat ment ions look for timefranes
Pabruasy", "during sprine", Mearly Do NOT use Lengural words like "first' "then", "after", "befure" guestion
zafezence
sf GOCD VS BAD EXMILESS [Exemples]
s1 ORDERING COMPLEXITY LEVEL: HARD
..Hardas: Either 8 10 neniions requiring chrorologiuell recomat ion or 8 10 neat luns with complex neations zequiring ing sophistioated sequence analysis popaiacicated analyaia
lonal
on advanced sequance reconstruccion
Teat abiliyy yilyy.e La Lrack cusple es mention pelleems auross milt ipls swasions
Include scenarioa zequiring expert level seguence analyais and pattera recognition.
QUESTION LANGUAGE REQUIREMENTS
Write ions as if the USER is asking then naturally (*T*, "my", " *ow") when referring to the phrases ("you*)
Use first
faccording aidressing the assialant conversation",
what was discussed", "feom chat history"
Nake questions sound conversat tonal and natural
Queations luns should flow nsturally as continaing the conversation
terporal
43 HANDATORY QUESTION ENDING REQUIREMENT
ALd quest RUST with the phrase *in ordert if previous didn't mention onn the order
GNLY
This signals Lo
sequencial, ordered seaponse expected
44 CEAT ID "RACKING REQUTREMENT [ONLY for souron_chat__ite filed in JSDN object
specitic chat conteia mention fist chat ids each nention chronologien order in source chat, ids field
icentity
ordering sequence
NOTR: If the ansaer is spread out beiveen milt miltiple that group then in one 1.st
NOTE: DO NOT INCUIDE chat ids the
Map
ita sourcee ch
44 OUTPUT FORMAT
Retucn your anclycis in exact Sommat:
"quest ion": es, "answer*; r*, *di ffieltty;; "sard", "ordering perdering...90 tyge": "*, *tota", menrt Ieettoon"" conversation, sor.ratttoooreeeeeeessssssss [l,
"ordering bested*; *corplexity
Il,
1ds";
IMPORTANT RECUTRSMENTS
any zeferences or tomporal **Highh question varietyse: Uee civeres question types trom the :5 categories provided .cCoppl ex reasoninges: Regui sophisticate icated nertion-order analysia
comporel bintses: Zucation must
*aattor 2ocusse: Test underesanding
conversational ditriculty
and evolution
*avaanzed dicticuleye:: Always
.GGenera) topic Focumes: Asc about breed thenes, not specifia event
stpphistiszted analysisee: Tast expert-level seguence rsconstructicn and pattore. recognition
high-guelity lenging ordering question
sophi
ion-seguence
anslysis without proriding any hints about the speciric cvents or tining.
NOIB:
obfect without expiznation before
Listing 16: Event ordering probing question generation prompt
new Lasked with generaling probing quest Leat cont cont radiot foo resolut fon capabi of MENs. You will be eiven two contradiatory bellet points and the cozrasponding melci c-curn diatogs betveen user and assistant that nncogorate both contradi statenents scross di Fferent converset
Your
create ONE quaation aaks test testiing ing whether the LiM will detect the cont fon and ask for clarificn fon rather than giving definitive
contradicted tople 12 socking factual information,
INPUT
..ULLT POINT Suullei_puin..e
L.COMMERSATION MURNS+A: connversation turns>
QUESTION GENESATION SUIDELINES
Teeus on creating inn quest-ions that:
Aak about. the contradicted Factass Queation the sopio: that has cont redictory informetion
nomal tact-seskinges Test cont redict det ionas: Whacher not ices cinfliittins informtt
there's contreciction
62

---

Published as a conference paper at ICLR 2026
EEppec ctarification requaater; HIN should aax contradlacocy atacement cosrect
-vood zevezling the contradiationn.. question should sean 1ike routing information peruest
CRITICAL REQUIRENENT:
question 87SI general brcad about contradicted topic
Do NOT includs specitic satas, cimes, locations, or detailed ciscumstonces
Do aot zafezence specific eventa or instancea that would narzow down the contradiction specific can detect contradic.ions acruss all related mettions
generai sttiem/behaviorr
This
s1 QUESTION LANEINES REQUIREMENES
Wrile quest
if the first parson (1I", tay", "se") zeferriae to the use:
naturally
doc second person ("you") wher andressing the assistant
Avoid plizases like "according to the comersattior" "based what discusaed", Teom chat histozy" Maka questions sound cenversation
natural
Questions should flow naturally as cont inaing the conversation
et QUESIION FORMAT Generate questicns that ask about the contradisted topic dizectly: [Examples]
EXPECTED
ideal responae should: [Example]
44 CEAT ID "RACKING REQUTREMEN?
MIST idantify
specific chat_id(a) contain contradictozy
List chat_id for the ccncradictory statemenz the shat_id for second ccacradictory stalement
NoI2: If the answer is spread gut beiveon nultiple chat_ida, group then in one 1ist
exact chat_id murbera from the cnnversation Luzna
44 OUTPUT FORMAT
your analysis in
"quest ion"; "t, "ideal 1", "di "difullly "*, *cont rad conr *", "copl Proplec.uee ioueetiionnni on [I. Fort;
("first ststement":
statement";
IMPORTANT REOUT REMENTS
quest phrasingess
should
like
seeking, conticttttion besting
TTppie foquares
disectly about contradio sub subject ject.
Aontt radi et detect ion expectar apectarionaa tonket M/ should not te and ast for clarificatior
Alo hint givingest bon't reveal that therefs mnntrrdiiiinn in the question *cCaaiisicatio seekingsr: Iceal responee saould
statement is
Generabe ONR cunt radiet lan resalusion quest ion that sears whather the LLM will detect the contradion for and appropriately request claritication when asxed
the contraciated topic.
Ba Nor include speciflc dates, Lines, locations, detailed cizound ancex in the question make the question easy. HoTR: Only y the JSCN object wichout explanat for before after.
CRITICAL HOTE:
Listing 17: Contradiction resolution probing question generation prompt
are Lasked with generaling a probing question Leat advanced sumar isation sepabilities will be eiven 6-8 relased bullet peints about the sane tepie/thene and the cozressonding multi-iezn dialogs briwzor user and cosisten: that incorporate this information across different conversation
Your task is to creato ONE quastion that tests whether an LIK con syathesizs and corderse complex, multi- faceled information from across 4t conversation swasicns into aophia loated, comprehenaive
INPUT
..ULLET POINTã.*: duullei_poonns>>
COMEERSTTION TURNS++: ccoversatici_Eorn>>
CRIFICAL REQUIRENENT:
Focus on complex information syathesis fron B-10 bullet points
Twst sumprehensive synthesis requiiing sophi at icated analysis
Requize advanced
construction multiple threads
summerics that domonatrata deep understarding
intogration
43 CRITICAL REQUIRENENT: NEUTRAL SUMARREATTIN PESTING quest
sumeery
questiin neation conly, the ovesarching topic no specific bullet puint details, sultopics,, phasea, oz techaical zerna nay appeas.
The quest fon MUST NCT hint the struuture consens of the expected answer
que Information indapendencly
be
generis,
adentify synthaaise all zelsvant
44 QUESTION CENERATION SUIDELINES creating questionns
exCopplex information synthesiser
cross-pession
condersat iurAr s.Compreeensive coervieass
ddppiitticatee nastalive live cohneuncuer SsSralagic detail prlocitizationv
Compllxx mulationahia Intecaction Summasye Compllate Sequence Event Analysiaxs 3, reBesoarce, effort, & "inel iin Rool ionas
.Multi fackor becixian Process Reviexes soroblen to Resolation Journeys AcChronoloiies
Development
*. Knoowl edge
nisight Integration Surmeryss 8, Coopllex legotiation or agreement Pathar
63

---

Published as a conference paper at ICLR 2026
BINHABEENTION CONFLEXITY LEVEL: 8N5D
.*Harde:: 8-10 bullet points requiring compretsacive syacheris xita detailed progrescion
seshiatioated analyais zequiring underatanding camplex relationships
lest ability to synthesize maltipic nerrative tarsads extensive infermation Include advanced necrative siemento with molti-layered connections and sophisticatsd causation
QUESTION LAMEINGE REQUIREMENTS
the =a esking naturally Use firat pazson (*2%, tay", "me") shen referrine to the use:
queat luna
Uso sccond person ("you*) when acdressing the ascistant "based
1ike "acoording
discusaud", feom chat histery"
Nake questiona scund cenversational
questions should floM ncturally a5 it continuing the coaversation
TRACKING REQUIRENENT
MUST adent ify which specsfic ih(t_id(s) concain information needed the aurmary
List ALL ihat__dd ids where sulevan: sumaasy infornation fon appeara
N0I2: If the answer is spread gut betweon nultiple chat_ids, group then in one 1ist
INCLaDE did the exact shat__d numbrra from the conversation Luzna
44 OUTPUT FORMAT
your anslysis in
Iormat:
"quest iunf; ",, numaay" "e, *diffiuult":; "hard", fsumar Lsp, e,, www.ellaattt_aatttoor D. reyuiers
"bullet polntss__ootttssooereee conversation ...]
IMPORTAV AROUTREMPNTS
should integrate all infornation Crom builes puinta
Boshiatinatee coherencere: Creace complex carracive with maltiple threads advanced logical AAdvanced multi sesaion synt hea ixass Coubine te ir format ion From 41 ponversat raatiiiio fon session
*SLselegic condensat lones: Include exlension important datails while naintiinin aophis leated narsacive
.CCorgile ex quest phrasinggss Question should request comprebensive, sophisticel leated sumarries
Generate OUR advanced surnari sat fon quest that tests the 11Me abil systherize 8-10 bullet pointe sophi
compreher sumnary.
NOTR: Onl y output the JeoN object wichout any espl explanattion for before or after.
Listing 18: Summarization probing question generation prompt
TE1S 18 plan tast contains sets:led bullet pointe about a topic. Iais plan is used =o generate recllctic chat conversstio inversation lona between user and an AT assi stant, which are chen used to evaluate the long-term cspabilitlee of LIMs. analyse
plan select bullet points ruct fon lowing itilities when incorporated into chat converes: anveres:lonn fone.
most
testing
Aralyas this plan and identiff bulle: points that coatain user instrant lons ideal for text ing whellres the LIM remembers lows user-given instructions.
INPUT DATA .. <plan>
bullet points with:
..User instructions cotegocy/labe.
..Ixplicit inat ioction stat enemmas: *blways", "llever', *Vhen esk about X, do should
..Behavioral sirectives::: sow
behave structures requested ..Content instructions.:: Mhat 2o includo or exciude in respanses
Specitic responee
..Frocess instructionee:: ESM to handle specitic types at requests
1ook spscitically for
poinss labeled Clear directive language i'As provide", "Rever include", *When I ask*) Specific fic behavioral expectations for the A1 assistart
contain:
corditionsl instructions ("tneen Response tormatting requirements Content ctluuiion/cccassioo rules
Return anclysis in ("capability": "Lastruction_ instraction_folooidnn "batci_nambera": "bulllst_aubbers
torma::
"bulict_poiate" *User Instruction: Mixeys provise detailed cost breakdowns when ask about bedget. estinates.";
Ingortant tormetting notes:
The "nases__ and "bulict_nanterr"" correspond to each other positionally *in and "32" moans: Batch sulict
full bullet point text appeara plan
Focus specicically on "User instruction" labeled bullet points
Select all bullet
labeled "Ises Instrcesioal
(approxinately tocal).
NOTE: Only output the Hist withous any explanation betore or atter the list.
Listing 19: Instruction following probing question generation prompt
64

---

Published as a conference paper at ICLR 2026
are an eXSeZE specifled RCERIC CRITERIOR.
judging whether respense Benona compliance
EVALIATION IN2UIS
QUESTION
usar a5kad) equescion>
RUBRIC CRITERION (whst to chock!: <rubric_itom>
RESPONSE 70 EVALUATE: clin_response>
EVALIATION RUBRIC:
the subzic decines a apecific requirenant, conatrainz, oz expected behavicr that the 128 zeaponse sheald dononstrate.
:IHECRIANI+ Fay
actention shether ..Fositive somairenents++ Ithings the response SECULD inclads/dol
specifies:
NeggaLiv comstzainisre (Changa the "absent")
SHOULD NOT incluse/do, often indicated by "oo?, "nos", "avoid
41 RESPOMSIVENESS REQUIREMENT (anchored to the QUESTION)
compliant response mast be evon-topic with zespect to the QUESTIOK<* and accempt to anaxer it. address the begative conatraints, bath nuss hold: (a] reaponae zesponaive QUESTION, the prohibited elemant is absent.
If the resporse
stop.
SENANTIC TOLEBANCE RULES:
Oudos nasning, not exact wording.
Accepp upacaparaser and vaayannyaus Chat preserve inceat. r.Case/ ..Case/ #cCase/pnccuattinne wwuctoatiee/siieeeeeee ..Kanbers/currenc
differences mist be ignored. may appear equivelert
(e.g., 1968,00'5, 188K', 188,000 usm "iisty eight thousend dollard' Treal Lhwn as equal when coneriially equivalent. If the rubric expects a number or duration, prafez vonornalized compariaones (extract and compaze values) string met ching.
REUTRALIT jerevents atyle concanination":
ignore tane, gel itensss, length, and Flourish sh less the rubric expl initill requires format/st (e.g-r "ilniized liaf', s', "hu cilaliond'' , bae seatened
penal hedging, voice, verbosity cortent satiaties Only evaluste formet when rubric vesplicitlyus mardatex it.
SCORING SCALE:
(Complete conglianme Pully complies
rubric criterion.
Fasitives required elemet poesent, acturale, properly executed (alloning semantic aquivalenta). iegative; prohibited elenent rasbsent AR resporse is careagonsi vess..
r*0.5 iPartial Compliance) vxy Partially complies.
Positiwe les elenent present but miner cocnrraciss/norrppppeeee execution. Kegalives generally ve and nost avoids the probibited element but with nimus/edge violacions
O. Compli aanpll
comply.
Pasitive: required element miasing ue incorrect.
Negstive: prohinited elenent present e*ar.* resporse is non-responsive/evverre ever the element is absent.
EVALUATION INSTRUCTIONS IONS:
unnderstand the Begui rementaA: AA: Netermine 1 the rubric is asking for sonething to be present (positiive) or
segrsiive/oostreinne
*AParse Compoind Stat=ment the rubric cortains multiple elements connected by pormen,
evaluate snether:
**All elenentses be
for full compliance 11.0)
clementsv present indicatos partial compliance (0.3)
+o ciements** present incicates no compliance 10.0)
+choock Complizncoer:
Foz pasitive requizenents: for presence and quality ot requized
For negative constrainte: look for the absence of the proh elenent
v*Assign Szorevs; Based compliance with epecitic zubeic criterion
seoring
PxProvide Reasoninges: Explain shethez the rubric criterion vas catistied and juctity the score.
FORMI: Return your evalsation in J50N tornat with tvo tields:
"ecore": Iyour score:
0.0),
"coason": "fdetalled explanation of whether the rabris critorion was saristied and why this justified the assigned score]"
NOTE: ONLY output the jeon object, without any explanstion betore or atter that
Listing 20: Rubric scoring for nugget satisfaction prompt
biinary
anippeta describe 3AME event/ fact, ceply v.E....
otherwi se reply ex80se, No extra worde.
no NOT provide ion.anation..
anippets (ffrst peragraph)
65

---

caregories most Published as a conference paper at ICLR 2026
aniopet: [second peraccaph;
Listing 21: Fact equivalence detection prompt
nURSEE) titles, themes, scptopics easegory ceat__atoppary,
("id";
rcategory"; "Irip Flenning",
rciile": "Desicning Year-Long Bound-tha- Itinerary on Shoeatring",
"theme"; "Soquencing flights, overland. iegs, and visas across tive continents in morths", reubtopice"; ('Round-tae-voris tickete", "Back-to-back visa rules", "Seaccnsi climate napping", "Ogen-jav "Budeat forocasting", "Digital-nene" locistics")
NOTS: Gonerate the nost compon ones.
Listing 22: Chat titles generation prompt
You are convez DaEion Franwwork specialist tasked with Identizying the nest zelevant section label categories specific
EXPUT PARAMETERS:
DONAIN: !!
TITLE:
THEKE:
BUBTOPICS: 11
CORE OBJECTIVE:
Arsiyze givea DOMIIN, IITLE, THEME, SUBIOPICS decerninc
selevant and natural for this specific scenas iu.
Genorate 15-20 section label catagories that best fi: this particular context.
SELECTICN:
Here are 5206 examples:
POONIVRRALL CACEGORIES (Always (relationships alxays zelevant)
ChCaracctor Rclationahip Labaisev
..fersunal Enot ional labelau (human wlenent always present)
..bacision Change labelasx (sonversations involve decislons!
TO SPECIFIC CATESORIES [Smlect based on relevance)
.epl anningg Logist "abelats Use when scenario ievolves:
Travel, projects,, Resource nanagenant, schedelina, cocrdination Physi cal pl planning or systemet agproaches
usganiising
exanoles]
ELECTION CASTERIA:
reMuet Include Te ielevvntive
categories
ly
topic
Calagories that would usturally generate diverse converaet ions Catagories that allow for progression and development over time Catwgoriex that create authent ie humen concerrs and interest
AAvvod The ngraa
Categories les that don't naturally fis the acenario Too similar categories thas would overlap Catagories woulde'e generate neaningiul Gererid cabagories that donf specific value
OUTPUT FORMAT:
exautly
label categories this
(Caateooyy Kame] Labelases
Brief explanat iun of why this cetagory is relevant L. Lo the DO0AL///TTEE//TRRR//B..OOO epecitic exarples
category
Example:
anning Logistios Sabeleiee
Essential
involving coordinat
ing, manegemen:
Busget Plenning, Troneportation Strategy, Accommodation Research, Ieinerary Adjustment, Packing ergarizaaiion
QUALITY STANDARDS:
selected category must:
Be direct ly re relevant to the specifle DONATN, TITLEE and THENE
Goncrate naturai, varied conversaticn eppertunities
Allow progression
nuitiple Provide enouch depth for 15-20 bulist points across the ccaversation Juet ouput the Labels, vithout expl igglanatoon at the tirst
cevelopment
ie
person
categorics that create specitic scenario.
naterol, engaging,
conversations for
Listing 23: Narrative generation prompt
66

---

Published as a conference paper at ICLR 2026
are long torn narsacive planning apecialist creating CORENENT PLABE? natural conversational
21ok.
Your task is to generate dotailed batch plans that xill seed reclistic secraaccistant Gialogue.
INPUI
..IOMBIR:: <donain>
..IIFLE:'. <titie>
**SUSTOFICŠ:** cabbiopicss
.<IIHELINE::> <timeline>
.*NUN_BNTCHES: <tun__atth22>> batches
EROFILE:** suser_profilo>
..LSER RELATSCNSHIPS: <BEr__cclsroonhipp
CBJECTIVE
ooherent natral are where real Generate Koun batches> dist inct, non repetitive batch plans
person naturally converses with at. AI assistant.
Each plan most lacroducs НEК stozy elements while maintaining serfect continuity and character consissency.
DETAIL REGUTASMENTS
MANDAIORY SPECIFIC DSTAI
Every batch MUST incinde
concrele detaila Chat enable factual anaverst
"Recuized Catecorins (miniman 5-7 per batch)
..Exact HUAbeeRtAr prices (8X), guantilies, perceniages, amasurenenl distances
..Specific bates/Tines: Por example: "Menth yta*, "xsy Rarar, *next [week dayl", *in weoks", stores, streets, Idineinge, parks, ..Bramd/Product Narma: specifat Ltens, services, companies, Louls, sulleare .YYasilo Sitcations: dacisiona nade, proferences stated, confliets resoived *-Fuent cotexies what hapgened, who won/last, what .ffgecific Pre favor fands, colora, astiviliss,
poosen/reeeataw
*+guantizlable Rasulcases tast acores, rankinea, ratings, complation times
xDeLail Distelbation Buleazss
Avoid vague statementa 'Elacussed opt lons" speclf WHA? opt Instead of ingiidg choices! use "choosing between X, y, and 2
STRUCTORE ARQUTTEMENTS
Gererate exact ly chum batchess plans
Fornat: 'BATCH X PIAN" headers
Each plan contains exactly
(TAFFL, CATRGORY) (LANS) DESCRTPTION :.. [content]] [o-25 worde! NOTE: Bach labsl consists of ca eacegury and desur ipt. Use both for eath bullet point. categories
Rach bul let:
LABELS letpoint
batch begining betch and Liow anchura among betchee should ANOTE. Time anchor muat cocrelat with other dates
CRITICAL ICAl:
Tey,
be increasing and time anchor in eact batch should be betore the dates nentioned in the batches.
PROGRSSSION BABATC (Story Foundation]:
Fi-ss bul et MUST hes +*T-ne Anchor:ss" Second bullet NUET
**Perscnel Introdustion:=" [Nust
persona lity
personelsty.
Ratabl init context with SPECIFTC details lage, locat fon, jeb title, salary rerge)
Introduce all relationsnip with CONCRETE contexts thow long known, waere met) goals,
fiable ceallenges
BATCHES 2-cnun batchers (Story Brolution): Reference user es "1/my/m" tnever repest the fall rane)
Each
the timsline chronclogically upen previously established
Show MEASURARL progression ipronations, relatiorship milestones,, achievement metrice)
++3, RELATIONSEID CONTIMUITY SYSTEM:** (Other details it domain Coding ioRssii
every relationship mention RUST incluse specific interaction dstaiis:
"*Evolution
Requived Details::e
.<In:soduction: "Ret (Nans) at lspociric place] on [sats/tinc]*
.*peve iepentienn Nane] suggested ispecicic actionl which resulted in [outcone]*
.>Beeponing:.> *[aae]] revealed (spcaific informationl during (specific event] -*sturationn:.. "Atter nanchs/yeare),
(spocitie change)"
Intoraction Varisty (zotate - never repeat withir. batches):: :*.
collasorstive, Supportivs, contiictual, Secial, Frotessional, Personal, Transactional, Serendipitous
**Charaste Consistency Rulesivs
Irack specitic preferences for each character (tavorite restaurant, hobby, pet peeve)
Referanca past specific events and their neascrable conseracnces Include relationship bullets
CONPLICT à RESOLITION TRACKING
-.Confiict Bloncatsive
Bach batch include situationa with:
..Binary Desisionster
*Waarurable cutconss::e
Sppecifii 31saeremmenta:*1
Types Rocatole*
67

---

Published as a conference paper at ICLR 2026
Pinancial dacialens apecific amounca
Time menagement with exact deadlines
Relationship toundaries with epecitic incidents Professional choices
options
Pezsonal
specific scenarios
ANII AE1--EEFE VERIPICATION
-*Before MY use specifio, verifiable
Have included
can this generate question with a one-ward anaver?
Docs this show HEASURABLE progzessien from provious menticas? specific?
Ia piace informaticn
detalls?
CONTENT DISTRIBUTION STRATESY
bulleta: Belationehip decelopments with speci incidents
2-4 bulleta: Carzent sicuat fon with neasurable nstrics
bullet: Exact tengoral anchor (specific date/:ime)
bulletx:
bullata: aocislena/pseferenoce
specific choices
bullet: **Frafarencs Statancnz**: implicitly shoving usar creferences
Best: Using zent ining labels wi.h concrete details
SPECIAL BULLET REQUIRENENTS
esl. PREPERENCE STATEKENT (roLaLe e earch batth) :*.
NLs: show preference through cotio//dcision
ABoLaae These Typea Eath Baich
Choics actiona
Nethod inllenentations
Quality decisiona
Timing patterns
S-yle approachez
Priority demmmataat.ion
>xS-ory Progression PacternstAs
..Early Betcliss 11-3]:ax Ratablish bazelinea leaseent salacy, selatienship status, living sit ..Middle Ratcheases Track charges fron baselires with speci Hie metrics
cheszae
beforw/after compari sons
QUALITY STANDARDS
ASSeci Bheckllat:s
Every person
tull
relationsaip
Every event has a date, time, or speal temporal reference Every local ion has name or addrews
Every decision has
specitic detaill
Every
ANNerrative Deptai*e
Include pri prices, parcentages, cistances, dorations
Show cosso-ans-ottee
specitic triggers
Maintain tastual consistency Idon's change establisted numbers/detes)
Be Reference past speci fie evente by name and date
Priaritize concrete details over abst deseript tons Every bullet saould enable at least 2-3 tactusl questions
Include cultoral, financial,
geographic specificity internally consisten:
Ensure details are realistis End immediately atter "BATCH <num_batched PLAN
concration
Listing 24: General domain conversation plan generation prompt
CRITICAL REQUIRENENTS
MANDAIORY SPECIFIC DSTAILS:+
Boery batch MUST Include
concrels, verl Fiable Lechnical. delaila Lhat wnable xingle ward be whoe
factual
aed Detail Catagor (rinamun 5 per batel-):ss
ExExttt Nunbezaser veraion numbers (v2.3.1), port acabers (3000), reaponse Cines (25Cma), 2ile sizea (2.5HB)
exppeci "at=s/Tiness" depl oynent
sprirt deedlines, ing times, ..Naned tuchnologissse apecific Cramworkks, librariws, Louls, services (Reatt 18.2, Postgre801, 14, sanbda)
codes (404, 5001, stack trace snippets depluyneni **Forzommance Moteicstes load tines, querv apeads, nonoey usage, ABI reaponse tines
..Feror Messageszss exact
texts, anglemented,
situalioners
AxConfigur lou aatillsss enviromeen: var variables, es, AFT endpai nta, databbes schenes i "iable Resuliszes Leat covesage (859), aptime (93, uses count. (2, 3CD-), bug count
Ratch bullet must contain AT LEAST une ous verifiable Lechnical detail ABetail Distribut fon Ruleszsa
Avoid statements like 'worked feature" "encountered fundefired Instead "lngroved formance"" use "reduced swapanse 8D0ma Lo
WHl"" feature and ROK
Replace "had bug"
iun"
STRUCTURE RECUTRENENTS
SYSTEM:** fontes
eallevel opnent
68

---

Published as a conference paper at ICLR 2026
Evary cechnical olemen: progression crom previous batches:
-eNeturri Jeveiegncnt Progression Exampicsie
.-Flanning 'I
authantication
cxampies)
IIechnical Complexity Progresaion:*>
batches: Basic impienata:ionn simple
Kiddle batches: Integral challenges, debuyging Lag comples iasues
Later batchas: Parfornance opt antiniaaion,, advanced feacures, production concerns
DONFLICT RESOLUTION
-*aandttoy technical Conflict Elemanta:*.
Each batch nust inclede at lcast 2-3 technissi challerges with:
c.llear Btakeazes what's risk (deplcysent deadlins pe: Eornance SLA, budost cons.tainti
+-Binary Decisionsses
impleneaced Solution **bbasurablle Oatcomms: reduced latency by Xms, seved
tixed bosting, inproved per furmance by
.*Specific Trade ofdee-ofta:rs what saa sacrificed for what gain (asnory for speed, complexicy for (ealures)
<Technical Confliet Types
Pezfornance bettlenecka with specific netrics Architecture decisions with concrete alternatives Intwgrel lun chal lenges with wxternal Security vulnerabiliiie. with scalability issuas with usez numbers Technical dela va Law Iealures
levels
Batch Requirementa
2-3 bullats: Techaical ispl anencation detalls with specific coda elements
2-4 bullets: Current deve ognens status with mesurabble netrice
bullei:
anchos (specific date/Lias)
bullata: bevelognent activicies with verifiable outcones
3-4 bullets: Technical decissons with specific fir alternet ives cons dered
bullet: : ofeeference Statemen.Ass implicilly sbowing developes prwlerentee
Using remsining
ive Batch Plamningsxe
Mach batch should organically Fooue what makes for that deve phase:
Implementai Beavy Batchix
Maltiple impl enentation requests
Architecture decisions
planning
Framework/ threry se ian
MMore
OODING CONVERSATION
Sach bullet should represent realistic developer-Ar interactions:
IImpeementation Requesta:
Exxrgless
-Debbgging
EExapless
--Code
Exarglles
EXECITION NOTES
Use plain, tecanicsl language throughout
Include coalistic tochnical spsciticity:
contiquration details
every sellet contribate
overacching development story
Eneure uni form technical detail qual ity across AM. batches
Vary lotthh cocus organically based developmcat ptase idmpiementation ve dabugging optinizatica) technics.
Prioritize
austract descriptions
Every
qoest
End immediatoly after "BAICH <num tnmm_bttches PLAN"
Listing 25: Coding domain conversation plan generation prompt
CRITICAL BETAIL REQUIRENENTS GANANDATOR SPECIIIC BRTAI" Bowry batch include cunezuus conGeML*, factual anavers:
nathenaticaa. details Uhau wingle word
Beqquisd Categor
per bateh)
.xExatt Numbera:er specific values 3.14), confficients (62x"2 5x 35), dimensions (85Ye1mes76
..Sgecifia Problemarrs complete equations 18x"2 4x 051, spacifle integrala (SVinLLD'S [2xr1] N,d8) (Rychegorear Theoren),
Conceptaies
elinination),
quadrat formule)
..Caloulation Resuita:ss uxact anavers (5x 451, decinal resulta (5)p1 \appeox 3,1415391, Esactiona (8)
(iracc33(4::5)
correctly, net appl inable, Scozw/Grade Netrioates Leat scores (859), huowwork gradea (18/20), quis resulta (3/10 cusewot)
problen
ixfiid, salution exists
a* Tine/bural intpuaattoonna study houre 13 hoursl, problen completion fon time (15 ninutesl, exan dorerion 12 hourst
function cnaracteristion (continuons differen Lablew, netrix propert Mechematiion Propert ieaper Leazss
invercible, symetricc
69

---

Published as a conference paper at ICLR 2026
Each pullet must concain At LEлST one verifiable nathenatical detail Pebotar biacelbacion Bulos:**
Aroid vague statements like "warked probiens" specity NRICH probions and reselte
Replace "studied nath" "campleted quadratic sreblens, selved corsectly"
Instead "impxeved Lnderstarding" use "increazed quiz from
STRUCTURE REQUTREMENTE
++3. KATHENITICAL CONTINUITY
reLearning Phase Evoluciense
Every nathanatica: elenent RUS7 shew progressior. fron provicus batches:
eslatural
.*Consoptual Paaseres nead understand shat derivatives nean..."
Moor champies]
CONFLICT RESCLITION TRACKING
-NNaddatary hathena Conflict Elemmnts:**
Bach batch must include at least 2-3 nachenatioal challenges with:
..llear Stakes:ve What's at risk (exan arade, asziorment deadiine, ceurse prarequisitar
..Binary
appl theuren x used algebraic geutelrie approech
-eMaasurabll Cutcomes: improved accuracy by reduced aolution time by minutes, raised grade from
Sieeifiic St Strapglesze which caused confusion, what concept nisunderstond, wheze coleulatioon wwat
hennae ical Conflles Types to Rotalweas
Conceptual assadderatanding with specific confaasion pointa calculation
Method selection dilenna with prow/ouns
Time pressuze challenges with specific deadliras
Prarequi vite ledge with mi esing concepts
Applical fon difficuliien with world
CONTENT DESTRIBUTION STRATEGY
sxDez Batch Reguizementsne
specific equstione/sslotion
bulleta: Current learning scatus with mesurable netrics
bullwur Exact Lengoral anchos (specific date/Linw)
4-6 bull milleees Mathenation feal activities with firbll outcones
bullets: Learning
wish specific alterra: considered
bullet: ofreference Stat ere implicitly sbowing learning preferencea
Reet: Uhing remsining labels wich concrete bethensrical detai's
tddagtive Planningasr
Rach batch should organicallyy Foeus on what makes sense for that learning phase:
exarples]
NATURAL CONVERSATION
Sach bullet should regresent realietic user-AI interactions:
AProblonn-oovving Roquestoi<s lexamples]
-cConcept Claritication:* lexamples)
-oooltiion Vecitication:e* Texargless
Netehod Explanaticns**
lexenglee es!
QUALITY
>*Chronclogical Consistcacy:+*
Batch learaing teginning/toundataan
progression
Each botch logically follows the previcus learning timcline
-*Wathenatical Authenticity::+
spezitic nathoraticsi detalls: equstion types, names, calculstion methoss MMore exarples
e+user futhonticity:se
Keep personality
provided protile
iMoro examples)
LLarriin Bealisn:es
Follox roclistic nathoratical learning patterns
iMore examрles]
Sppeciiicity Chocklistiee
Every equetion specitic
and variables
Mors cuamoles)
EXECUTION
Usc plain. methoraticsi language throughout
Include realiatie nathematioal apecificity complete equat lons, exact values, specific theorens overarchira mathanatical
Maka sellet
unitorm nathonaticol cuality across
70

---

Published as a conference paper at ICLR 2026
Vazy cocas organically based leatninc plase (undaratand solving applylng) prioritize zoncrste nathoratica: details ever abstrza: descriptions Every buliet should enable at least 2-1 tactuzi mathonaticol questions inmediately after "BATCH <nun_batches> FLAX
Begin generation new.
Listing 26: Math domain conversation plan generation prompt
You specia ridd editor that adde three agecific test bullets exlating batch synthecte far generation,
IXPUT 7ASK
..AAAN <pland
EACE Keep polnsss originel unchanged, exactly positicne rum bul letab+), <nun bul 1etss42,,, <num bu bellets>-3
*+ HANDATORY OUTPUT STRUCTURE
Bullets kml_bullletes Griginal bulleta luncharged)
Bollet <nun bullete>+1; Intornstion Upsate
Rol <ou mulletase2y User Tnat ruct
Pullet mnan_belicte>+] Logical
44 THX THRER SPECTAL RUTERTS
s+r INFORMATION UPSATE (Pulict buinttell
aeIntornation (pcazelva
containing update]
**NANDACCRY FRE-CHECE;**
SCAN 1-<our bullets>-2 EXPLICIT umerrcaa./mmssuree
IDENTIFY exect nuntez, tima, date, neasurement VERIFY value is ciearly stated in criginal text
ONLY THEN create update changing that exact value
18one exarplles les)
CRITICALL: Make INFLICI7 embed
in natural narrativo, son't state is
-+ppaate Cateqorics lsotats through cill:..
Numerica shicts iprices, quantities, messurements) (enpiovnent, raiatienships, Location changes jadaresses, venues, sestinations: Reiationship progressions (social connectione) [5ons other categorics]
>VERIIICCTION:
natching
bullets b-tmetbblllett>2..
ste 2. USER IXSIRUCTION (Bellst coun palte-21)
Inatsuctionsee [action]
[conditisa]
-ANNDAIRR FO Must inciede "uben ask sbout this nakes it testable.
>+Instruction
through
Octput fomnllinng
Concent restrictions
Personal profercaces
Condition zesponzes
Fine-based
[5onc other catcgorics]
.EXNNPLES:**
cHampies!
afe 3. LOSICAL CONTRADICTI3I (Bulist cnun_bullets+3)
:Pusnat:s velggiee cantze iunsvA [ConLradiering only."
**CRITICAL PRE-CAECK:..
SCAN bullets Emurbullless 2 for COMPLTEE ACTIONS PERMANENT STATES;
"yisised", "ate", "craveled", *lived" Pezmanent conditions:
in", "raised az", Absulute statemenis: "neves did xt, talways was ye
PIND exactly ONE target fast to contradict
сaRAтE setup bulles comp bullballeta 2, CHEN cadiit bullei. Soun_balletasis
COMPERTRD
PORSTADEN s000s/ (NEVER (SR):
"Beforw this batal",
batck", [Ollie exerples]
GARORRSA only cont redict COMPETTR: ACTIONS, never plans/ intent lons.
QUALIPTR
tooe exanples]
PORSTODEN M0RSS/DMASES
"Bufore this batel," chis batch" "Foevionaly", 10ther exerples]
*CRTTICAL:** Contradiction event with cifferen: outcones.
original
71

---

Published as a conference paper at ICLR 2026
Conttradi Types luse varsety) :*.
Ago/Tine Reversal: Age going nackward Resurrection:
doing activities
Reves-Statancat Viciatioas: Contracicting "never"
Location impessiatlities: Being in two places simaltarcously
Scatemntt Conflicts: Contradicting exclcsivizy
osieinal 1s COMPLETED ACTION (pas: tense)?
contradiction nakes original INFOSSIBLE, not jact difforenc?
like nosnal, vords that augoest conflict?
hint words?
44 CRITICAL VERIPICATION SIRFS iInfomation Updatei*e
you paint to EXACT nurber/L im//dae bulleta Seur_
changing ONLY that apecific value?
exlogio Cantradictians
bullet criginal fast is CORPLETED ACTION, not plan?
can you soint
being contradicted?
Cuntradiotion is IMPOSSBBLE, no- just difleren.?
FORBIDEEN MORDS/EHRASES neacioned LOGICAL CONTRADICTION section
44 CONMON ERRORS TO AVOIE
Don't place agecial bullets anysheze except snun.cellettrrl, coun_bulletsrr,, saan_bulle:spe1
fy origing Don't skip any of thewe special bulleta
bullets
Don't use other labela for apacial bullets
OUTPUT FORMAT
Betuzn COMPLETE sian where each batch end, When sesup fact is created, inswrt between bulleta krun letas so 2, renumber, and add speciill bullets Knuniulllettrr smundimllesp213, Suunabulleeapte.
original bulleta unshanced exactly additional at
Begin processing the plan now.
Listing 27: Adding special bulletpoints to conversation plan prompt
You are norcative coherence spccisise creating DIEONOLOGICALLY SEQUENCED topic clustere for roallatic generation. Your
ornversationa:
10 inserccanccted form
natural progrossion.
#* INPUI DAZA
..58ED TOPIC..:
..5EE0 THEME+*: thene_
..5250 S1BTOPICS-:: <ubod__ubtspps>>
..U3ER FROF IXCFILE=>; cuser_ prr__ppottis>
<tineline>
CORE OBJECTIVE
Generate 28ON ab jbject containing 10 toples fineladiag the providad seed topic as Topic that forn
CHRONOL2GICALIY COBERENI narrative progression.
follows
roalistie time
e1 CRITICAL REQUIRENENTS
CHROHOLOGICAL
ENDEPENDENCE
.IOOIIC PROGRESSION RULEB:*>
--Iopic S*+: Use the provid:d seed topic EXACTLY as given
..Popics 1-9xx: Each must be COMPLETELY DIFPERENT life domain/category
..N0 EXTENCED SARSATIVES++::
1-9
seed topic's
PROCRESSICN+ eaperleace
topic represents wtat naturally happens complcting the previous
INDEPENDRNS
Each topic must address DIFFERENC area (caresr, relat ionshios, health, educasion, finances, eto.) Topics should shew how one l:fa experience leads to growch in OTHER areas NO should *Pact 2 of previoux Lopic
NATURAL FLOW REQUIREMENTS
AACADAA RELATIONSHIPS WITHOUT CONEINIATIOR:
Topic H+1 INPLUENCED BY Topic but addressen DIPFZRENT 1ife donain
catalyses change Exapll:: Tsavel exprrimncи (Toput 0) Careez teazowasmenn. (7opic belationshap priocit Hople 21 EExanias]
Show growt's
ALIGIMENT
..Danooraphic Consiatencyxs: All topics alien wich userfa aqs, education, carees level, atage ..Pinencial Real smaee Topins must reflect user's actual financial cupacity and constraincs sicoggrahii Logieus: Topies comsider uswets looat and mubilily comstraiots priorities,
.-Value alignnente:: Topice
goals
4:6 4. TOPIC BREADTH REQUIARMENTS
tech
inelude lode letti cineline that:
..Seguent Tininges: Topics
overlap
logical temporal progression
..buration Bealianes Zach topic should span 1-2 nonths authentic deciaion making and implemental
..Ketural Gapars: Taclude realistion time geps between najor Iife rensitions
s.Seasomal Considerat lanaes: Accpunt for nstural tining [job searches, noving
acedenic de (aleddass)
X, Kech topic most be oufficient ly BR0AD to generate 20304 suchent's to conversat fone by including
T-Tinelime
72

---

Published as a conference paper at ICLR 2026
*MMultiple Secision Pointav:: 15-20 na deciaiona pez copis
**Complex Subtopies*e: 9-10 acastantiel subtopics that each zequire extensiva discussion
..ongoing Frocessecer: Topics involving muiti-morta planning, execution, and adjustment phases affecting multiple life
..Cross-
rtr 5. SARRATIVE REALISM
..Katural Tiniager: Reallatic tine between major life dacis-ons
.-Emotionsi Procression=::
smotiensi psychologiea: deveiepment .sFracticnl Domatraintare Tupics mus, acknosladge sea linitations (money, Lame, cesgonsibililien
should
sf OUTFUT FORMI REQUIRDNENTE
Json
"toppis": I("id"; O, "csterory"; "Previded Caterory)", "title"; "[Provided riticl", "chone": "[Provided Thome
1", "aublooops 1/+ Provided Bubcopics Reray v/,, "Limllin"" "[Stast Month, Year End Monch, Year; "), Casegozyl", "title"; "[esscriotive Title]", "chens"; "[character-foce focussd describing the hanpllls///ppprttat inyy)) "subtopius": [ opie 11", "[Bubtoplc 21", (Bubtogle 31", "[uubtoppe 43*, "Subtoppi 51*, "(Bubcopic 61", "[ubloppe 71*, "[Subtopie 81%, "[Subtopie 9;", "[Bubiopic [0]"], "tinollne"; "[Start Month, Year End Meath, Year!"), ure]
("ld"; "cattoorv"";
Lopies
44 CRIFICAL VIMELINE REQUEREMENTS
MEN-OVERLAPPIHG FINELINE
**ABECLITE BULE.*: topie' timaline start one
pECViOUS topic
PATIMELINE CALCULATION PROTCDOL:A>
Toplc 0: Usea provided Lineline exactly
Topiic
Topic
gep
NO overlapping montha beiveun any Lopica Each tosic durations 1-2 nontha
[Fxamplex]
>IIEELINE VERIE ICATICH STEPS:**
tefore Finalixing liing each topic:
Idwatify poevious topic' END nonch
get Veriff NO month appears in mulsigle copies Confien aeallatic gepa for life Lransition
LRAST
possible
nonth gapa represent: JUSTIPICATION:..
Processing ng and integration time after ne jor experierces Natural life shythme and decision naking periods
Realistis pacing signiticant
changes
Time consequences of previ one deal eions to mani fext
TOPIC pRoCRESSION GUIDESINES
Topic Beality (Topies
..Popic lass Bow seed topie experience changes perspecLive on
.-Topic Zest Ripple ettects creating needs in YST ANOTER domein
Phase Muiti-Derain Growth iTopics 3-31
T-Tois 3-5+at Leveraging cumulative growth to address diverse Lite challenges
eft Ehase 3: Intogration Across (Topies 6-8)
..Topics
synthesizing learnings optimize
efr Ehase d: Holistic Vision (Tepic 9)
.-Topic Sext Long-tern life design incorporating all previous growth
AACRITICAL QUESTION FCR EACH IOPIC:*.
PArcer comp itiigg (previous topte), , what DISPERENT area of life would this person natural ly need to address next?"
QUALITY VAL.TATTON OFFCKTTS" Exxxpics]
IXAMPLE
"CGTC
Exxxpic]
FORSIDDEN ELENENTS
..Non-sequential topics+e; Topics could happon in any order
..Protile cnntrescctionse: Topic5 that contradict use='s establissed circunstances
**ihrcelistic jumpse:: Hajor 1ife changes vithout proper coudatiica/ottivat...
Topics topics ..Tenplate responseses;
conversation
unique circumetances
*+ EXECUTION 30I3S
13
Ensure saamless narrative fron Topic through Topic
Prioritize realisn and character consistency over dranstic story-ines
Focus on authaatic life progressions that real people expariance imedistell
closing
*RRIIICALL+: Output your zesvonse in J8CN fornat only. Do not include any explanatozy teat, narkdown ternacting, or adcitional commentery. Provide only the rav JS3N abject.
Genezace complete copic clustes
Listing 28: Ten million sequential seed generation prompt
73

---

Published as a conference paper at ICLR 2026
are creating CERORCLOCICAL aLaroPIC that breaks down main topie anto diverse, nea sepetitivs phases with strict tineline boundaries.
IXPUT DACA
IOPIC:*. <mpin
..M0IX TREME:** thone
..MAIN SIBTOPICS: cnain_Fubccppsss
..USER
ppsr_prof-io>
TINELINE EXTRACTION (MANDATORY FIRETI
Dusation from main topie (e.9.r "x-day doing days)
Action type pwtofct/cwrr///aaaceeeeoooo
Total Limeline ine
nontha
*Calculate Key Datmare:
NATILACTION_ START: When core accion begina
KAIBLACTION_PDD When coze action ends
realistiee prep/ eggal
s1 SUBTOPIC CENERATION RULES
+44 Phese bistrisut ion (10 Top-cet
.-Topics 0-2ex: PREPARATICH (curing main action parlod)
..Iopics
.-Topics 7 9.xt INTEGRATION laf.er accion unds)
PAPACH SUSTOPIC ER UNIQUE:..
No recycling of thenes betwaer aubcopics Rach explores nTFErRNNT
s/chal lenges
Progressive complexity within everlap
phase
Distinct fozus areaa
APREPARATION DIVERSITY (Popica 0 zk2
Topio scovery/ (eeveer//eeeavrh///iiitt Planning
Topic 1: Deciaion Makingg/Besuureee Satbering/Ski Building
Topic 2: Pinal Proparat lons /oonfi manaiaon Launel.
ACTION DIVERSITY (Topics
Topic 3:
Launch Beginningiinnggneeeeeeeeennnnnne inggiiinn/Inniiii IInatiad
Exgeziences
Topic 4: Rarly the angge//Adppaat ionspttttons///orgrr.. ions/Progress
Topic 5t Peak Twe Formence/Deeg Engegement /Hastery
Topic
wwst/Corpietion//ronee
INTEGRATION DIVERSITY (Popics 7 t4
it imesiste Retlection/ Inniiitiii Frocessing
Topio Appl taatio//transsrreeiioo
Topic 9: Long Leen Inpact/Pusure Plamning legaay
44# Regui red Stracture Per Subtopic.
geon ("id"; [D-9), "category"; "[2hase name]" "ibistinet chal shallenge/ wewge//gpoooeentieeeooeooooeeeee OURRI AP with her toples"), "tinlllness "[Date rangel ", "phase. type"" "Ipreparat peppsparatoon//ee fon/oore,eet cocooreeeee act ian/ intogration/",
"[Unique descriptive
REPETITION]
MUST DIFFERENTI "subtopics"; (*10 DIVERSE sub-elerrent
dates":
sype"; "Itypel", phasss relation"; twwffrre/durin//fttte
starte"; "Idatel",
"[dace)","
wein_action lon's "[daration!" ". "ourrent
stass_boundtrreer ("can_mantion": (*Alllved activities"),
(*Foroieden activities"),"
for main action": "I Tusture/presen/ppe
"scy_milastonee"; uniqua milcstcnes"); ruturo_rotezencee [*Setup for coatincisy"!, "eattsasity_tooks" ("Linke to next topio"]
DIVERSITY ENFORCEMENT CHECKLIS? Exargles]
NIIN SUBTOFIC DISTRIBUTION
provides moir_subtopics
strategically:
Stov evelution: besic => intermediate -> advenced nastery
Dicterent angles in each phase iplanning ve doing retlecting!
OUTPUT
Generate ONLY this JSCN structure:
asoni toain__oooi""" "[Iaput]", "hain_theme": "[Iapuz]", "main__abtopica": ["Ioput array"), tinelnei "I Input]",
timeliae"; ("tineline stare"; "[Date]",
end"s Date]", wawp_aaccooooneeeeeeeeeee "Datee""
main_actson_endet "Datta"" boiin_erdoonddssiiooo
"[Duration]',
Ereparatton_phaaee phace": ("start"; "[pate)", tead"; "[Da:e]*, "topizo"; [9, 1, 21
"sose_acttoo_phasaa" ("start"; *:Dase)", tead": "[Date]", tduratiear, "[uuration"" "tapics"s "intoq=ation_hhee" ("stare"; "[rase]", "ead": "[Daze]*; "capizo"; [7, 8, 21),
sustopizs"; [7+ 10 UNIQUE subtopic objects +/])
CRITICAL:
suotopic explcres
aspects
ND shonatid sopetition across 10p1co
Progressive narrative
appertunities
Octput ONLY the JSCN. No explanaticas
Listing 29: Ten million hierarchical seed generation prompt
74

---

Published as a conference paper at ICLR 2026
are long narsacive planning apecialist creating CORERENT STORY PLABE? natural conversation 110w. Your task 1s to generate detailes batch plans that will seed realistic weer-assistant dialoque.
INPUT
..I171S:** <titic>
.IHE@:"+ <thene>
*<IIHELINE;* sLineline>
.:NUN_ENTCHES:: batchas batches
*<LABELS:* sptorided_iabels
<user_ptoi_le>
sooi__oolaciostiip>
..NEM #ELATION3EIPS:** <nex_rciationship
..EREVIOUS PLAN: sprevious__pe plan>
DNCCUDEELNCRODCCIIOOO <Y25/H0>
C0R3 CSJECTIVE
Generate <aun_batches> distinst, nco-ropetitive bateh plans that Zozm ocherent narrativa are where a real person naturally
story
maintaining perfect continaity character consLatsncy.
CRITICAL HARDATIVR PERSPECTIVE **c.*emme*e.... PERSEECTVVE.*
from the USER's perspective (fiest- Use fieat person gerapective Lve Lhroaghout but VARY sen.ennee at suclurees Natural natcal ive flow avoid
parson)
starting acezy bullet with *In
passive
person perspective
CONTENUITY REQUIREMENTS
AACRTTICALANE TF PREVTOUS PLAN provided, MIST:
s-Maintain
Comsistensss viinlegrale Belationshipass
Iemporal Progressiones
vARuil Id Upon Previcux Decia Densa LaPregerve Est abl ished FaCtRAs AxCont inue Belat forship Arcses
PACRITICAL TINELINP PARSTNG IMANDATOY PIRST STRP!:A>
Beforw gensraling ANY conteat, you NUST internally calculate Lineline boundariea.
PASTED
Mrite Tineline Boundar
AACALCULATE
BATRS:*X
EASTEP Create Batch Dat Asxignment:ss
segmente: Days per
Caun,
AABSCLUTE FINELINE RULES:A 1. AVEERYY date swn looed HUST be beiveen START and END datwsas 2, 1.0 Fubure TIMELINE BND Ino month" it timeline ends this
past
A*Teporss MUST opncluci naturally belorw on
progress loginelogicalll
boundariese 5.
ATROOORAA ANCHOR PRQUTRRPAATTSAA
buliet et batch [Nonth] (Bayl, [year), Jevent descriptien!*
temporal acca0E
Format: A*Tepporal Anchorss*
Pach tenporal anchor date HUST be within that bat axeigned date range Batch date Batch
(PORATDERH) ..A
[Examples]
GENERGGNERTTOON CHECKEIST:+4 [Exempios]
CRITICA: DETATI, REMENTS every SPECIFIC DSTAILS:**
SANADDAORY
numecous concrete,
eingle-word
answers:
Reguized Detail Categories (miniman 5-1 sstch)
prices (9X), quantities, percentages, Teasurements, [Sonc categories exanples]
-+Detail Disteibation Each sollot must
one veritiable detail Avoid vague statemeats
STRUCTURE REQUIREMENTE
FORMAT:**
exect ly
Format: 'BAICH & PLAM" headars
Each plan contains exactly 30 pullets
Each bullet: *,
DESCRIFTION]:+ [eantant]* (<=25 words! consists of са:ееооy description.
CAISOSRY]:
Eath
point.
Use only the provided LABELS no custon catepories
* Pirst bullet MUST be Temporal Anchar with the ONLY date reforeaca in the batch
++IP INCLUDR INTROFUCTION YE3:..
.*BAIC (Stary Feundation) :+.
bullet
Introductioas Establish initial
SPECIPIC detaila
Intzoduce zelaticnshipa CONCRETE contexts us neasurable coals, deadlinas, cuantictabie
challenges
INCLUDE, INTROFICTION
75

---

Published as a conference paper at ICLR 2026
(aontiauttion):: parsonal Introdaction ballecs Bedin directly with cussent copie-zolaced content Refersacs es:abiishhed charsetor details from PREVIOUS FLAN
PABACHES 2-cnun_batchea>
Evolution)
Reforance
"I/ay/me" (eases zepeat
ranc)
advances
chronclogically
Build upon ALL previcusly establiched clements
Sbov KEASURABLE pregression (pronotions, relationatip mileatones, schievenent metricst
CONTINUETY SYETEM:**
rrcoze vs. Неx Relationship Hanaqement.er
..OORE SELATCSNSHPSS::: Wazt ranain consisteat acrozs all plans same nanes, established details, orgoing
BELATION3RISS+::
naturally cuzzent topic phasa #sBlaationhhip Intecration
-Relationshls Evolation Handaze:ve
Every relat Lonship ment ND87 specific interact detaila:
-Evolotion Stages with Requived Detailstre [Exengles]
CONFLICT & RESCLITION TRACKING
Mardatary Confliet Plmrenistva
Bach batch include veclear Stakes:ve what's [xxagles]
sicuation with:
(acney amount, deadline, relationship acatos)
Types Rotateres Relationdap houndar les with specifau incidents
decixiors speai
manegement
Psofessional shoices with confrate options Personal valusa with ssecific acenarios
REEECITION VERIPICATION
Exanclas!
CONTEN? DISTRIBUTION STRATEGY
Requi rementases
bulleta: Benatio decelopments with speci incidents (nix core new relatiooshipss 2-4 bulleta: Carseni sicual lun with measueable metrics
let: Exect tergoral anchor (speat fie dste/timel
bullete:
with
bulleta: Becisiona wwt.a///ppppeeeneeceo
apecific choices
Reet: Using rensining labels with concrate details
>*Story Progrecsion Pacternsiar
..Farly Rhtches (1-31 Retabl-sh bazelines Jeurrert salary, re lenehhp status, living eituation, Kiddle Batcheatar Track changes from basel ines with spetifle metr
sslater Batchesses Show cumalative results tetore/atte: comparisons
NATURAL
These plans generate conversations where users seek AI assistance for SPECIFIC situations: [xxagle)
QUALITY STANDARDS
exppecl Ficity Checklists person exneples]
and detined relationship
>ellorrative Deptai**
Include grices, percentages, cistances, durations
Show couso-and-otfes
specific triegers
Nsintain tastual consistency Idon't change establisted numbers/dstes)
Reference past specific evente by name and date
SXECUTION
Prioritize concrete detaile over abst descriptionn
Every bullet saould enable a: least 2-3 factusl questions
Incluse cuitural, tinancial,
speciticity
reslistis
It PREVICUS PLAN provides, inclate 1-5 specidic references to provious eveats per batch End immediately atter "BATCH <num batchegs PLAN
-FINAA TIMELINE
Parse TIMELINE beundaries FIRST
EVERY date most fail within thase beundaries
exceptions timeline linits
Verity
respects
output ONLY the actch plans. No explanstions or additional text.
Begin coneration new.
Listing 30: Ten million sequential conversation plan generation prompt
You new prwaiaion parsacive aschitect generaling TEMPORALLY COBERENT BATCH PLANS will absolute Limeline
Integrity and pheee-agpropriste
NAIN cnain titles | WATN THIME: caain theres 1 TITLE:st 1m THEME: hemeo T-MELTNR: ct iie nex
NUN BATCHES: baun LABELS sprov iddd a> PRCPLLPGPPEEE:: cusess profflee DECTLEAATTTOOTTOTEETETEATTTOOOOO RELATTONSHIPS
SUaTOORC PLANS:<aLL Lallssaal___ttbto weboopiic.t..o...ooooooo p_ai
SUNNARY=<greeionn PRPVTGUS SUMMARY: sprevigus plans sunnary> PRIVTOUS PlAM:sprevi oue plane
76

---

Published as a conference paper at ICLR 2026
CURRENT DAa.sPP httpa:/eeceeeeee.aoo..ecceeeeeee abtopic BUBENN_BUBTOPICCOO 1Dt.TooPoccDrrrrroooooooo a.coopp
06JECTIVE
Genezace snan_patchee" distinst, aca-ropetitive batea pians touning coherent narracive whare acturally ocaverses with AS, Each plan introduces NEW clements while maintairing continuity.
SIRUCTURS [AANATORYT]
Fusita::
heatlers
Exactly 30 aullets per batch
Bullet fornat: *e* (LAEZL сиTеCRк:: [LABSL (LASSE.
DESCRIFTION [content]" (<h30 words)
bullet
excapt semperal [Dale], [cantext)"
other
ALL content in FIRSI-FERSCH (*I/ay/me")
Vazy semtenow atzuclurws, avuid stazting evezy bellet with "I"
REQUTRRMENTS
per batch ninimum]
Exac: Hunbezs: pe prices quan-itles, peccent agea, nessuzements Specific Dates/Pimes: "Honch x yth*, "way P35/224
other exengles]
Roplace vaque with specific: [Exanglex]
TRACKING
TRRCUCHOUT]
Bzack per betch:
Purchases: [Iten, Price, Stoze, Date)
Before EVERY bullet:
check if Fact exists in registry Tf similar exists, ADD NEW 1TMPNGTON TOX jconsequeence//omml fiwst i /on/ perpett iiee/
progressioo)
44 PROGRROSTOV PARTRINS
Batches 1 3 Eatablish baselines initial decisions. celai ionshigg inlcus exBetchee 4 Exxxx Show consequences, compl cations, deegen oelehtineeipss
ches Uneggected decelopments, secondary offects, evnlution BxBatches 9. 10 Long bern impaete, syathesis mectelly, fubuse impl inacions
progressians
Basin eacebli Lshnent Pousth: Beveal impaut Fitthi:
corpl Thirds unlesa dranatic change
44 LABEL ROTATION RULES
Tzack usages Label
FExamgle
RELATIONSHIP RULSS
iish SPECIPICS tage, iona job, selary) suduce relat iondhips wich context. thow long known, whare mel)
First
AANOW 1FS first "ереaтапснеан Inclode relacionehip natursily
context Aftee I't rntrott for, refer
Every re lat tonship ment ian needs specific interaction: Introduct fon Nevel opnent Deepeni ng Naturat fon
[empora] anctor Ispecific date)
2-3 relationehip developments
eituation with
events wita cutcomes
6-5 decisions wita choices
Reet: rema ining labe la with detai
-THRRR--PASS REV11W:s* 1, scPact Uniiqueeess:e 2, snInformation Advancement:4s 3. rrCross-Ratchie
Exargles]
DATE EXIRACTION
Extract trom SURSENT_SUBTOPICC.. SUBTOOUC_BDTTTTT
TENPORAL
spprepraaiio phase:** [Exanple]
ixcoze_action phase:** [Examplc]
intoogrstion
[Example]
(STARE_DATE, DATE]
#+ CONTENT BOUNDARIES
CURSENT_ BABTOC
P.Rulos** ONLY generate fron canumention NEVER generate from cannot_mention ONLY reference current rubtopic activitios Use specified tonso for main action
**Preperation:+ [Exampic]
Accionies [xexple] **Integrstion:+ [Exanple]
#f TINELIKE DISIRIBUTION
Calculato: cun_batchep>: Group batched poz day
batchaa:: Sequential
Daya
Bane-day day di A.fferuntiacinnse Tine progression (evening-)evniig) Activity Focua shiffss Perspective changes
Depth
77

---

Published as a conference paper at ICLR 2026
COMTEXT -MIECRATION
Review Srevicas plans sunmary for established focts 2. Coatinus Irom: previous plan 12 exiats 3. SF
[XCLUDE_ 2XC_JDDE INTRODUCTION Introdace raturally
80: Continue without co-sntroduction Reference prior facts conslatently Show progreasion Cron previous cading
44 VALIDATION GATES [Exampies]
#1 EXECUTION 1. Extract/vezify daces 2. Write 2IRST-2ERSQN 2. Date GNLY in anchos 4, Kaximan detail decsity S. Exactly 20 bullets 6. Validate boundorics
Octpet ONLY batca plans, actar barchess
Listing 31: Ten million hierarchical conversation plan generation prompt
You arw generat reall Tat quesstons that UEE2 would ask an ASSISTANT. Grease queet lons beaed ONLY
specitic details the currert bullet points.
DONATN:
idonain>
TITLE: <title>
GURRENT PoCUs ARMAS ION'Y SOURCE, POR QUESTTONS! <POCUSED, KULARTS>>
COVEREDI <BATCH
CONTEXT REFERSNCE (FOR UNDERSTAKO:NG ONLY1: <PSEVIOUS SUB BATCH PLARSS <EREVIOUS BATCH_ PLANS>
CRITICAL RULES:
PARRORR GENERAT INCE "ist every decail Fron CURRENT FOCUS ARFAS:
Names: (extract all names]
Ages/Suncers: lextract ell nunbers
Locacions: [extract plases]
Facts/Situaticas (extract s.l specitic facts]
USSSE TRACKINE:*. Mark
quastions.
*+* 2. ABSOLUTE SOURCE RESTRICTION
..CNLY ALLONED SOURCE:** Detaiie explicitly vritter, it CURSENT FOCUS AREAS bullet points
CCOMPLETELY FO
nanes, slaces, facts, oz
from
socticns
ANY cspics or content from BATCELACSTOR
CONTEXT REFERENCE BULE: Use CONTEXT REFERENCE only to understand NHO people are or WHAT thinge near. Mten in
REFIRENCE content.
*** J. ZERO REPETITION EMFORCEMENT
AASOLLTT REQUIREIEHT Each specific detail can ONLY menticned ONCE across all quastions.
*ABSCCLTE PROHIBITIONS:**
noze
questions
Nentioning ANY tepis/detall fron BNICH_AISTORY
Reterencing MY content from CONTEXT REFERSICE sections Asking
in bullets
-VERIFICMTION: Befere each quastion, contizm.
sepest previous content.
4:E 4. AMTI AMEI-REETTTION SYSTEM
++DETAIL USMCE
nention: Use full spec-fic
puint
Subsecuent zeferences: Use pronouns (the", "sha", rit", "that", fay choics")
-VERIFICATION: Caeck each question docse't ropeat:
Specifie
elready
Topics fzom SATCH_HISTCRY
Any dctails or ccatent fron referance secticas (CONTEXT REFERSICE)
REALISTIC CONVERSATICN STYLE
**NATUBAL LANCUME:**
Contractioon lons: "I'n", "don'L", "can'L"
casual words: "kinda", "sortà", "gonna"
Fillers: "like", fun", Informal:
*2?",
444 6. QUESTION VARIETY
.YAVOID multipie questioas way
BATTERNS:..
Vary question length and cupplxiit
QUESTION GENTRATION
Norumal questiiune iunes
.*Sock advicers
..ask for helpas
.sfequest clacifioatioons
quidancers
*.Sppeas ennt Tonses ..Validatw decisionars
thought
..Explare opt lanaes
ACUUESTTON S
bullets get
get
Deep dive into complex situacions
Cuick quest tons for simple detsils
dser ini roduci ng hinsel hinsel///eeeee E/heeself should be firm: ques lan
OUTPUT BROUTR BRMPNTS:
78

---

Published as a conference paper at ICLR 2026
Genezaca exactly BATC_ STEES cueationa that:
++08E EVERY 1aTAI fron cussent ballet points exaatly 2aes Sound 1ixe geauine buran requasts for AI help
Poeus asecific
aituations neatiorad
avoid all copetition trae previcus
Show realistic emotional raspanses to bullet situations
Follow natural conversation flos
--EEVER specific details
resmiilioe
chainate in any message
**SUCCESS CRITERIM:*
bullets appwaz appeac ONLY No resecition BAICK_ cooico
age,
qucstions saund 11ke real prople texting foz sivice
All queations Lrace back Lo apeci fie bullet detaila
pronouns/ Subsequent referances REFERRNCE sect 2RAC content from
>+CUIPUT FORMIT:
gusstion,
leuestion Lext! ->-> [bullet_nunber]
AACRITICAL:..
Each
[nunber]" [ranbee]] bullet
based
bullat numbers 1, etc. as they appeaz CURRENT FOCUS AREAS If question nes details fran multiphe bulleis, use Lhe peinasy bullet murbee NrA
If the queaticn
exant ly csnerated from bullespointa,
quest
Fosnat:: One quastion. por lina, natural longth, no numbesing oz extra text.
Listing 32: Question generation general domain prompt
DEVELOFER
ASSISIANI.
specicic detaile
CURSENT FOCUS ARENS (STRICT SCOES ONLY SOURCE FOR QUESTIONS):
XFOCUSED BULLETE>
QUESTIONS ALRSADY COVERED IN IHIS BATCH (NVCD THESE!: (BATCH_HISTORYS
CONTEXT
ONLY
QUESTIONS THIS)
HERBVIOUS SUB, BATCH PLANS>
FEREVIOUS_BAICR BATCC____ABARAAAAE PATCH_LLANSS PLANS>
DETECTION
NANDATORY FIRST STEP CHECK EACH BULLET:*.
It sullet cantains ".4Time Anchorive* -> ABSOLUTELY NO COSS, ONLY predect seejjett/cchduee iing questions IE bullet
PrePersonal Introductions ABSOLUTELY CaLY carsez/perscnal
bellst, GENERATE
AFE 1. CURRENT FOCUS AREAS BULLET TYPE IDENIIFIATTON CHECK FIRST
the Anches:" Litle) >.BEFORE DOING ANYISING: (coat
..Tine Anchor:**
SENERATION
..Fersonat Introductiona bullets (contain "Farsenal Incroduction:" in title) >> NO CODE CENERATION
..Iochnizol bullctsi*e [2l1 others! -> COSE CENERATION REqUIREd
MUNDATOYY
*.STEP 1 - HANDATORY EXIRACTION:* Betore writing ANY questions, you MUST extract and 1ist EVERY SINCLE dctail
Irom CURRENT POCES AREAS:
.>>ancc:*** llist nane deveicper nancs, company nanes, project nanes, client cancs!
SunbeeraVaasiinaaee [llat EVERY version number, date, Line, quantity, post, ID, measuzeneat]
gother catagcries examples]
*.STEP VERIFICNTICN: Count cotal extracted detaila.
1009 than.
*.STEP "RACKING:** As you wrize each quastion, mazk which specific details it uses.
PASTED PINAL CHECK: Before aubnitcing, verify RVEN extracted detail appears in least quest.on. aspear
>XABSCIT7E REQUIRRVENT:*> EXC2PIIONS.
the batch.
s:E 3. ABSOLUTE SOURCE RESTRICTION
++NNTY
expl explici:ly
CURARNT
bullet
..CMMLLTTLYY FOR3TEDEN
ANY content fron SATCE_SISTORY
ANY content from CONTEXT REFRAPNCE sect ions
Genasic programning neat ionad cuzrent
exslicitly
COCTTEXT REFERENCE RULE: Usw only understand WEAT Lechnologise /oompoent when they appess in
4:6 4. DETAIL USAGE PATTERN (ANTI REPETETION]
..Pirst ment oossA full speci Fie detail From bul (exact
..Subseguent referemeess
pranouns
agp", "che
..VERIPCAATONN: Bach specific decail appears CNC across queations
444 5. HANDATORY CONPLEX CODE GENEDATTON
ACRIITIAL: at questions
cubstential snigpets (20-60+ ii
ee'a title ZURRRNT FOCUS AREAS is not tine anchor or personel roduuiiooon
79

---

versions, tiles, Published as a conference paper at ICLR 2026
-ABSCLITE EXCEFTIONS NO COD2 CENERATION:
.*Iine Anchori bellsts NEVER EVER generate any code, programming solutions, technical implenentations
Introodctionse implanacaz1soe
code, programing
tecanical
..FCR PERSONAL ENTRODUCTIOR BULLETS: Generate questions FROH the perepective ot the person introducing Chenselves
The
builet USER asking
ase cuntexiual/personall details, Locinical coding avenac-os
S+STOP AND VERIEY: If the bullet ccncains "Iine lachor:" "Forsonal Introfaction:" in the title, yeu BUSI generate
blucks, programing
scripha, .schnical implemenaal
-OR FINE ANCEC//FERSOML IRTROOUCT:ON BULLEIS:**
Focus pro eot
schedul ing, personal yoals
dazdlines, meeting coordination, profect
NO code blocks, programming solut ions, technical implemental ions Uae natural conversation ien about cining, goals, and concex:
.cODE COMPLEXITY REQUENPMENTS ffur .-Minimun 20- -D-
per blocker
.-Multippe Zunctions/ seattion//nettonn mathodss/casses 14-6 niniman)
A-lealiatie ingurta and dependenc [3 ninima)) landling, val.dazion,
.*Complex busineas logic, database operations, AFI ..Product fon level at acructure and realistic variable nemess
PATTFRNS
coding
onlylies
..bebuggin ing 1425) sur Semerata buggy oude realistic, hard Lo agot eccoes
reCoda Raviev (258):ax Genezate working but achoptinal code needing inproveneats Implamment ion (20R) Generate partial ial implementa: immetaciinn ians with deteiled ronos Semerate ficiiie
FampLional
ANEVER use simple exanples or tucorial style code always production level complaxioyss
klangugges i cont rect lars ("t'a", *donft"), dev slang (flol", "atwfl, fillers ("like", "an"), informal panstuation ("...", "29", "!!")
Anot FonzAe Show genuine feelings frustration with excitenent about features
Natttral Pleazss question lengths, include rambling, thinking load
TTcchicaa buchentiellyse Include actual arsos meszages, fila names, version nurbera bulleta
44F 7. CHRONDEOSICAL CRDER
pointe
provided.
appear questions.
4:4 8. CODING QURSTION STRATEGY
..Implementationi "Relp busld Ispeaicic teature bullet
..Behuggingtss intting
(spect ..Cade Reviewies *Can you review this Ispeli uode] and suggest inprooementa""
zeteiimiaatioois "How can mace sspecitia implenertation) caster?*
OUTPUT
Generace exectly <BUB BATCH SIZE> questions that:
* Use EVERY SINGLE detail FOCUSED BULLETS
specs,
.4851 MUST inc! ude substant aal code anipgets (20 lines) with product fan- level complxxtty. Sound like geauine developer reguests with realistic technical scenarics Follow chronclogical order bullet points
complete, complex code Hatch one at the tour coding patterns with appropriate complexity
simple exampioс*
Stay st-iet within bul po:nt scope submitting,
--AANDAOOYY VERIFICATION:
extracted
*ACUTPUT FORMAT:*N
..CRETICAL: Geacrate ONLY the devolepez neceaçes, nothing clse.*
Do NOT include
Separate comp message organizacions. SEPABATOR----1-
Each message can span muitipic lines and inclede code blocks
+.CRETICAL; Each message NUST end with +-3-> [number]* where |nunber] is the bullet point it's based case #t:*..
RBROUERED FORMAT PA
[ourpu: fornat excлрlс]
PACRETICAL
jou:pu: Founatting rules]
PACRITICAL VEREFICNTICH: Before
**HANDAIORY FIRET CREC:: What sype mansgenent/ee sanspensttscchetuuingg questions, ABSOLUTELY CODE
Time Anchor bullet Generate project
Peesonal Intreduction bullet >> Conerate careez /parsonal questions FROM their parspective, ABBOLUTELY MO
2CDE
Technical
Generate coding questions
Does this use a specific datail from cursent ballets?
Save inclused cusstantial, compiex code (not cimpie examples) for tochnical bollets?
developer An tollexing ene the four coding patterns corractly
this
selps
> FFNAL VERIFICATION BEFCRE SUBMICTING:..
dotails in
100% details CURRENT FOCUS
A852.
80

---

Published as a conference paper at ICLR 2026
Generate exactly <593 BAICR S102> questions ic. the Torma: above...
Listing 33: Question generation coding domain prompt
generating realistio questione
questions
on the speci detaily in CURRENT POCUS AREAS.
44 CURRENT POCUS AREAS (STRICT SCOPE ONLY SOURCE FOR OUESTIONS); XPOCUSES,
44 OURSTIONS ALRSADY COVERED IN THIS BATCH (AVOED THESET:
KRATCH HISTORYS
CONTEXT REFERENCE (POR UNDERSTANDING ONLY DO NOT GENERATE: TORSTTONS AROUT THTS):
REVIOORR PAACHL
KERSVIOUS BATCR PLANS>
CAITICAL RILES:
*+r 1. CURRENT FOCUS AREAS BULLET TYPE ISSMTIFICATION CHECK FIRST
PARRFORE BOTNG AVYTHTNG:A Tdent the let ir GURRENT POCUS ARMAS:
..Iine Anchor:** beliets (caatain Anchor:" title) MACHEMATICAL
..Forsons Introsuction: pallets (contain "Fereensi Intronuction: in title! NO NATHEXATICAL WORK
GENERATION
--.ethenatieel bullete: (all
XATREMATICAL GENERATION REQUIRED
2. HANTATORY
COVERAGE. TRACKING
S 1 MANDATORY EXTRACTION: Betore writing ANY questions, MUST eNtRact and list EVERY SINSLE detail OURRENT FOCUS AREAS:
-EExtratt ALL of these cotogorses:*
..Hanesia* llist EVERY name nensioned people, places, institutions, etc. [list EVERY number, porsentzge, quantity, catagories example]
**STEP VERLEICATICN:: Count setal extracted details, You MUST use 1696 of than.
specitic details it **STEP "RACKINE:** Before subnitcing, verity EVERY extracted dotail appears
*ASTEP FINAL
question.
ABECLUTE REQUIREVSHT:* Every single extracted detail MIST appear in at least one question accoss tae batch.
ABSCLUTE
CNCYY * betails explicitly xritten in CURRENT FOCUS AREAS sullet pointe
COMPLETELY FOR3:DDEN:**
BNSOR_
concent from
REFERENCE sections
Genoric questions about mathemarisal fields
Details not explicitly nentionad in curreat sellets
onl understand
things when they appear CURRENT
ARSAS.
SETAIL
[MIIIREPETTTION)
.-First amnlicas**
apecific detail Ixom ballet point
.*Subsoquont zeferencsa:vr lae proncuns ("it", "chat", *ay honezork")
..VERIFICATOO: Each specifis detail sppears ORLY ONCE across all questions
MAEDATORY MATHENRIICAL
CRRTICCAL Mhen referencing problems, equctions, or natheraticsi work that isp'= explicitly provided in bullets, you RES?
and faclude the complete natlimatioal
CENERATION:**
.-Tine Anchories bullets Conerale quest lona about schaduling, deadlines, Lining w-thout any AATHENATICAL иOаK
fot bullets
PRCM person introducing Chenselves
background, carees, goals withou: any HKTBENATICAL
..EOR PERSONAL ENTREDUCTION BULLETS: + Generata questions FROK the parspective of the parson introdaciag
Chenaelves vea
builet
UBER the questions
a2e contextual/pozoona details, RATHERAFICAL acenarios
POOR FINE ANCERUPERRONAL INTRODUCTION BOLLETS:*> stydy schedal
goals
about exen prsparation, at ady coordinatios, acadenic plamning NO nathenation ical aquations, XO weciinn-oiivnng, но calouiations Use netural converzat ian about simi goala, and acadeni context
RBQUIRED
A.Complete eetlly Stuck :o NO work shown, just destr the what you' te trying bo solvew ..bartially Stack (30%) Show ONLY initial 2-4 where you stuck
ansver/reeuit ..Conceptual Confuxion (109) NO calculat iora, corcept. facused questioons
PAMORK CENERATION BRQUTRPMATTT:A
points Include specifiu fie aurbers, variables, expressiona
nethen
Create realistin in problens users would encounter A.VEERR placwolldess like R... (insert work) AIMAYS generale wot uel nethenet sea work..
AUTHRNTTC STYLE
81

---

Published as a conference paper at ICLR 2026
"Laaquage: contractions 1"2'"", "don"t", cassel slang (M10"", "Bea"), fillers ("Iike", "ua"), Intornal
Functuation ("...", "??", *!!")
-Enotionie Shew genuine reelings confusion, trustration, excitement
reNatural Flouses
langtha, include rambling,
loud
rtt 7. CHROHOLOSICAL CRDER
Process bullet points in exact order provided. Barliez bellet detalls appeaz in earliez questions.
QUESTION GENERATION
s.Froblon-Salvirggs "Help ms solve (specific problen!"
..Concopt Cierification:. "I son't understand [specific conzept;"
ay solution is ..Mechod Bxslanation: "Why (specific nothod]
..foiution Recificationse "Can
44 OUTPUT ARQUERSMENTS:
exactly BETB_BATOK_SIEEE
. Use EVERY SINGLE detail fron NOUSSO__BLLEES at Ceast once [namea, agen, dataa, Lraita, ocals, timefranas, etc.)
Bound like oenuine uses reques:a ieca
include
(eguat celeulations, solution atleepts) reallstic nathenacioal content
Pollow chranclogical order of bulist points
+AAMAYS inel adl complete hena ical problens//quai Jans NEVER use placelaldersss
Stay strictly fouz behavioral
with appropsiate shown
point 2cope
.NANDATORY VERIPICATION: Beform suinitting, ceafirn every excrasted detail appeass in the spestionn
*ACITPUT
..CRITICAL: Gemezale
user mossages, rouking
Do NOT include question numbers, headers, or organizations text message with n. WEASAGE_SFFAAATTO -.. limes include nathenatioal expressions
..MANAATOYY: Separate each
Eath awasege span
..CRITICAL: Each nessaqo NUST
*-2-> [nanbez]* [nunbox] la point
ARRQUERED PORMAT PATTEMMEX
(0urpo: exerple)
ACRITICAL PORMATCING RULES:..
[Ourpo: fornatting roles]
>CRIITIAA +ERIFICATICN: Belure each queation:
Does this use spect fic detail From current lets?
.xlx this Tine Anchar Personal raductt for bullet? If YE5, do NOT incl nathenet ical warks generating questions FROM person's perspective ithey user) Have included actual net hene: work (not glaceholders) for bathenatioon lece?
<*For Personal Intrometionn:
5. Dows this sound like a seal user asking for kelp?
tollexing of the behavioral patterns correctly mathenatiosi bollets?
SURMITTINGE Count how many extracted detaile apgear in your questions, It MUST be 1008 sil details trom CURSENT FOCUS
VERIPICATION
AnMAS.
Gereraae exactly <5u3 BATCH StEE> questions in the torms above...
Listing 34: Question generation math domain prompt
receive a coply. beternine whachar concaina rsdisect, specific quastiones the ussr mast answer next by providing now information, proterenzes, or decisica.
Y.S+.. trip?", "Which optica you profer?"|.
"Wbat's budret
..N0+* for generic or chetorica. prompts (e.g. *Any questions?", "Hould you Like to cive deoper?", "Concider your badget"; that do not demand imeediata, specific ansvar.
ASSISTANT
CRITICAL Respond in Engl Do Include any Chinese.
Octpet exastly ".Ee w.00 noching clse.
Listing 35: Check assistant's response include question prompt
typios
you
ques:
ruspunaee
AACOVRRRAATOO CONTRXTIAA
IITLE: <title>
THENE:: cthert>
SUBTOPICS: SaubLepica>
History:
hietory>
At/x test Resporae: cassist ant responzee
*AASK POL12M-3P ('yea") WREMMA
response
you genuinely proceed
Speci fie steps process you' Lrying follow
Key paraneters Idates, anonss, reguirementst for deciaion you're making ficatiiooion which option applies your apeci situation
AGGnui Con Confusiones: Soneshing unclesr or contredictory
82

---

Published as a conference paper at ICLR 2026
Technical terns used without axplanation that understanding
ionflicting infornation that affeets your raxt action
Amoiquous instroctions where the wreng interprotation has consequenees
Incompicte Practica1 buidarse.::
can't actually yet Lacks spacifica neaded for implenantation
Kissing stogo in procedure
Assunas knewiedes
don't
- FOLLOR-UP (*oo*) MEEN::*
+-Good Enough to Proccedes; You have shat aeed
Only "yes"
Listing 36: Check need for followup prompt
respond only Kegl should be rnly in Engl
any language
responses
Respond to the usezfs neasage by wither:
Pull ansvering ing their question
Provide compeelana
Answwring ing plas asking must follow-up question you need nere
Always hunor these rules:
Do
questions the user already ansvered.
you genuinely need cortext to provide complete, act ionable anover, 3. Keep your maln answes cleas and cungreliensio beforw you aak.
only a quest
the tollexing
inputs:
edonaino
TITLE: <titlee>
TEEME: <thene>
SUBTOPICS: <eubtopice>
PREVICUS PLANS SUMVARY: sprevious plans surmary>
PREVICnS вАTCНES OF THIS ALAN:: <prevt ous bet
CURRENT HISTORY:
<eurrent batconnerssges-
NOTE: Rexpard only in Engl ixh, Bo not include any Chinese.
eACotpot
Setucn exactly saet say user-no taps,
Listing 37: Assistant LLM answer generation prompt
conversation
assistant has just you quastion, and you provide natural, hunan- responae.
rtrt Input Context You'l1 Receiva:
conversation leading .>ionain, ritle, Thans suscopicsre:
..Cursent
quaation
subjecs conversation ..Frevious Flansee; Sumnary of cariier conversation contexts for continuity
..Curzent Flanes: The evarazching narrative direction for this conversation batch assistent
..al's Questicare:
specific
your response
6:88 INPUTS:
e*Cuzrent Batch Nesszge Historyee: current_batchheeeae
>.DOMAIN: edaraia>
>.I:TIE: stitle>
>+TEEN:: <theme>
>.SUBTOPICS: Subteppccs
e*Previcus
see_cossppinn__uumay
>*Previous Batches
Plan*e: sprerious_botcaep> excu:cent Planss: sourrent_plas
..Al's questianer: miisgast_n
CRITICAL: Keep Raspenses BEART
e.Real usezs give brief, to-the-poin: ansvers to AI quastions.r
1:84 Rols Behaviors
are
user with:
Persunal wxper jences, opinions, and inoti
Natural speech patterns and conversational habi:s
Bealistic knowledge liritat Consistent ger sunalily Liaita accoss conversation
curiosity
alaaguuge Authent foltyses "I'a", "don'L",
lots
titl" "that'aa
Include casuel slangs "loi", "bca", "cbb", "Kinda", "sorta", "gonna", "wanna"
Add filler wnrds: "like", *unt, "you know", "f meent, "so", "oelin Uhe infornal panctuationn ian: miltipl periods ....",
narks *22", exclanat lon points mgjn
Tnper fect Natural Speech:.s
Include minos lypos and informal gramas
Add ranbling elerents: *T
Include thinking
loud: "hmmt, "aoruelly", "gl",, "mayber
Anott lonal Aut inntyyy
Show genuine feelingss excitmment, Frustration Lar, uncersainty, hoge
langeage
personal react lones "uph", *ong", "yay", *oof",
83

---

Published as a conference paper at ICLR 2026
sir1 Respense Guidslines:
**STEP Check Flans for Existing intormatioave:
tha Cursent
Provious
the AI's
question
..It cound in planger: 825e your response on that established infernation to mzintair story continuity
.vIf not found In planave: Crease nex answez that aligns with the topic, thena, and existing storyline
S*STEP Answer AI's Queat Diswclly*e;
Keep your zaply focused on answering; do renot** iatrodace nav questiona.
Give direst anaver to what the ai aaked provide unmecessary details
Don
Answez yea would
text conversatior.
Include personal concext or examples when nateral
CRETICALA Beause
doesn't contradici anything watablished in previoua cuzrent plans
+* Consistent
xentext
Kaincain the same parsenality and circumtances theoughout
Keep your responses aliened with the curzent tapic and thame
Response Characterlationn
Ionaee:: Match the convarattinns smotionsl tone and your established personality
A.ku inlyyys Sound like real
Al Leying sound hunan
Critisal Instructions:
<. concisess Real people don't L welle long in ciat
Keep your raply Iocused sn anssering; do renot iatroduce nav queations.
check plans
look any
ior
question
details
.*Maintain consistencyre Nevar contradict information established in cuszent or previcus plans that fit the establisane
..Pill naturallyss If plans dan't have the
ONLY provida your zespense
nota-comentazy
5:ay in cherecter es hunan user chroughout
Answes the ion bui dan feel obil igeled Lo eak question back (Llee AI aaked YOU)
Listing 38: User LLM answer generation prompt
any chinese characters phrases your ressonse. You are real person having conversation with or AI assistant. Based on the conversaticn history and sho A1 "s last responce, ask ONE netural tollow-up questicn.
must raspand voonly Bnqlish*<,
CaCureent Batch Nessage Historys:: couerent courrent_batch..u batch_nessagee
>.D0MMI:: <demzin>
..IIILE: <title>
>*TBEME:
-.SUBIOFICS: <subtepics>
>*Previsus Plans Sumnary+e; sprevicus plans_ plans_sumae sumnary>
exPrevicus
This Planer: spsevious_batcheas
**Cuzrent Flanee: <qurreat_pian>
+*A2's Responsc*e: <2i ios_iogt_
TASK
follow queation aurda) Chat real pezsun naturally ask afcer cecelving the sespense.
43 CRITICAL PREVENT REFECITION
Befoze genarating your question:
.*Saan the Curreat Batch Massag: History** for :ll topics already discussed
.xcheck Previous Batcheart for cuestions already asked
+eeveer
ef you not ics your queation ion seeks infornation lon already provided in the convessation hiatory, STOP and genesace completaly different questior.
REAL PEOPLE PCLCOR-UES
sft Natural conversation starters:
ful: wall..." "han... " "aciuulll.... bbix... fok bat..." taakea
"chacra cool
"yeah about....
#14 hachentie reaut lon palterns:
rrBuilding the last al response:rs
speci topic yet
Connect your peraonal situation Irom Cuezen, Plan
skhhow genuinn renasionassa If guod
"nice!
gother examples]
si4 Queation lypes that fwel natural:
long ueually
(other typex]
44 NATURAL, SPXRCH PATTRENS
Irclude
Contract "dea"L", Lt,
"Munfc", "chatra"
Casuel worde: *kinds", *sorte", " "gonna", "like"
Inotionel reactionss "aghif than", *ob", "yikes"
punctustion:
*??*
84

---

Published as a conference paper at ICLR 2026
CONVERSACION 722M AMARENZSS
Bases on where you axe in the Cuzrent Batch Nessage aiscozy:
Mck broade: exploratory questions
specicic details
about implomentation naxt stops
44 AUTHENFICITY CHECKLIST
CRITICAL ROTE: Respond only in English, ish, Do not Include any Chinese.
RESPONSE:
[Generate
follow-up queszion, else]
Listing 39: User LLM ask followup question prompt
provide you
identiFy
text,
and purput that keys value format.
B.g-:
1: Value
Say
you to provide cummary
tornat: Sunnory;
summari red bext"
tote: key-values
surnary. 20 NOT provide explenation betore atter that.
gutpet Key Key
PPrrvio Contextie (history)
tex:: (text)
Listing 40: Key-value extraction prompt
arG highly analysical asslscanc. structured curmary of key information and insights.
is analyze latest conversation produce
e+Yo Internal
meximum accuracy, you mast
stap-by-stop.
*>Analyze:** Break dovn the user's latest message.
*=Idontify:** Fingolnt all facts, instructions, and updates.
implications the infomation
the conversation.
history.
underlying goal or state?
**Fornat:*> After completing your interasi analysis, format the conciusions into the 'Extracted Facts" structure.
Instruction: Your final output +oNNLee your step-by-stap reasoning or any other text in your response. Strictly rollox the tornst shom in the exanple's outpet.
Facts" black. +DD include
EXANPLE..
CCnneessal
R*Racant Conversation History:**
USER: Hey, need some help with the "Frofect Fhocnix" launch plan.
ASSISTANT: or course. Mhat
need?
USER: launch
respeasible
markating natoricla.
Laccsst Exchaage imclyzs:**
DSER: okay, the final badget for the social media campaigs is 57,500, The clisat, Innovate Corp, jeas approved it. Flease find three case studles of successful 828 saas launches by tonerzow, Auqus: examples.
any ASSISTaNT; Understood. a.l1
competltor case stadies auccasatul competators, and have them for you by tonarzow, Auqust 28th. The approved busqet of 500 for the aucial media campeign has bewn. soled.
sead lacnehas, exclading
EExampl of Correct
Output:*.
The olient's is nane is "Imoval Carp".
The project related to *B28 Saas laurch"
Final budget
campaign is 187,503.
ses for "Lunorsow, August 25th*, User intends to raviow three case studies for the project.
Trat ruct ons Rind three
soudies
include diswet cunget
the exapples.
budget for social media campaion been appzoved by clieat.
The user is under deed' iow and needs the case scudies urgently ly to inform their work on the narket natusials.
AACTUAL TA
Bistory
[llatory)
wxt Exchange AnalyswAAr
user_ messagel
RegreTANT ilatest axsistant, messege:
85

---

Published as a conference paper at ICLR 2026
Extracted Facts:*.
Listing 41: Scratchpad creation prompt
You Lasked with sumarissng and componssing sotasth ped contens flt within specific Loken Timit. Inpup
(oontent)
TsTarget bengthies (tokens limit] tokens
--Your Tock:**
Compress this content by clustering related information, renoving redundancy, and prioricising the nest ingortant details,
clustorr Greup rolated information by tapic, entity, or thens
ingortant intornation Corpppssse::: Ceadense while moiaraining essential meaning and ccatext
Desuplicstee:: Renove redundant
Prioritizeas: Keep
details
ecCutput Pornatiss
compressed conzent organized
PARRY KNTITTES RFIATIONSITPS:
[Nos:
organizations, mentioned]
..CORE DECISIONE PREFERSICES:*.
/critical cecisicn points, requironcnte, constrainte]
>*PROCESSES
[Besential precesurol information and methodologies]]
H+USER FREFOREINES:
[User's
dielikes, pretersed nothode, sectings, choices]
>AUEER INSTRUCTIONE:
1Specitic cirections,
quicance provided
INFCRTART DATE3:**
[Deadlines, nilestones, schadeles events, cime-sensitive intormation]
**CRITICAL CONTEXT:**
[Background infornation neceasary for understanding]
>*ACTIONABLE ITENS:**
[Nex: ataps, panding actions, dead:ines]
ININCRTANTT DEVELOPMENTS:
[Siqnificant
milestones]
S:ay within Hirt)) tokans
Eliminate
essentisl information Lhing
Elininate older when chere never apdated
Maincain chzonolegical contaxt whare Inportant
Prioritize information with orgoing relevance"
CCIIIIAA REQUIKEMENT:
Your zesponse sheald be approxinatcly Itokens, linit] tokens If draft is signi ficanlly shorter than [Lokens linic] Lokens, ADD MORZ DEPAIL
Listing 42: Scratchpad summarization prompt
query You need to decide it the text chank is nesseccory for answering user question.
provide you
It ve need the text chank to snewer the user question, or it the text counk is part at the answer to seer
quest
nat rolevant Output cormat: Return only "yes" or 'no", without any explantion setore or atter that.
text
quastion, seturn "no".
query: (quceyl Smin
Listing 43: Scratchpad noise filtering prompt
You nrw an rastatant that VIET answer quest Tons usinng ONLY the informet fon provided In the context be Tow.
INSTRUCTTONS:
ONLY beaed an the prav ided context
Do 102 use your inteinel Indeledge
CONTEXT:
OURSTION:
Equestion>>
86

---

Published as a conference paper at ICLR 2026
ANSKER REQUIIEMENT::
Be dizect and concise
only sutgut the anewer the qucstion vithout any explanation
SESPONEE:
Listing 44: Answer generation with RAG prompt
I LLM USAGE
We used ChatGPT as writing assistant. Specifically, we first drafted the paper and then employed ChatGPT to refine the text/assist with rephrasing and grammar. The suggestions were manually reviewed and edited before inclusion in the final version.
"hups://chatnnt.com
87


Reviews:
[{"review_id": "fC8Omt5wiw", "reviewer": "Reviewer_WLZS", "summary": "The paper makes two main contributions: a BEAM benchmark for evaluating long-term memory in LLMs and LIGHT framework that augments LLMs with extra tools to better handle long-context tasks. The authors provide a number of empirical results, evaluating a number of models on the new benchmark and ablating the new LIGHT framework components.", "questions": "Is the focus of the paper intended to be on conversation-like tasks only, or is the claim broader (long-context LLM application/evaluation in general)?\n\nMinor suggestion: I believe that including a table of illustrative examples of probing questions into the main paper would greatly aid readability. It will clarify the approach for the readers who want to understand the paper without having to read the appendix.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper focuses on a highly relevant problem.\nThe data generation method is sufficiently original.\nExperimental evaluation reasonably complements the data-generation effort and the ablation studies help better understand the contributions of different LIGHT framework components.", "weaknesses": "My biggest issues with the paper are:\n\n- Insufficient coverage of related work + I'm unsure about how this work fits into the broader research context in this domain.\nThe paper is positioned as addressing the problem of long-term memory, but is focused solely on conversation-like context. The way the authors brush off existing work seems a bit dismissive, especially since there are many works that do not suffer from the limitations the authors quote as disqualifying.\n\nFor example, Narrative QA and Narrative XL datasets do not consist of concatenated unrelated conversations and do require engagement with the material beyond simple retrieval.\n\nThe data generation approach is reasonably original, but I do not think that the authors provide a thorough enough justification for why this approach should be taken over the alternatives.\n\nIf the paper were explicitly targeting conversation-like usage contexts, it might have been more defensible. But still, when there are many datasets that are far from being solved, a lot of attention needs to be devoted to justify introduction of another one, with thorough proof (both empirical and theoretical) that the new dataset catches LLM performance gaps that can't be investigated using already existing datasets.\n\nI found it especially unfortunate that in the related work, the paper focuses on the efforts in developing long-context models, while the primary focus of the paper is on long-context LLM evaluation & benchmarking. The latter is covered very sparsely.\n\n- Empirical ablation results (e.g. Figure 3) don't seem to have confidence intervals, and, more imporantly, the results are not very consistent or strong. I.e. it seems that the proposed LIGHT model does not consistently improve performance, and that the helpfulness of its specific components depends on context length (and likely on many other factors like the nature of the task that were not investigated in as much depth).\n\nOverall, I feel that the paper proposes a number of very promising ideas, but requires a bit more depth to properly place them in the broader research context and to make the results more reliably interpretable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zsNGC9xseI", "reviewer": "Reviewer_z6rF", "summary": "This paper introduces BEAM, a large-scale benchmark for evaluating long-term memory in conversational settings, featuring dialogues up to 10M tokens and 2,000 probing questions across ten distinct memory abilities—including three newly introduced dimensions such as instruction following, event ordering, and contradiction resolution.\nThe paper presents LIGHT, a cognitively inspired framework integrating episodic (retrieval), working (recent context), and scratchpad (salient fact accumulation) memory modules. \nExperiments on multiple open- and closed-source LLMs demonstrate that LIGHT improves accuracy over RAG and vanilla baselines.", "questions": "1. Can you provide error analysis showing common failure patterns for each memory ability and also for each evaluated metrics?\n\n2. At 1M context length experiment, LIGHT doen’s seems to outperform RAG at Abstenation, Contradiction Resolution, and Event ordering. Since RAG is similar to the episodic memory module of LIGHT, any reason why this is the case?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Good scale and diversity: context length in conversation setting up to 10M across multiple domains. The samples are validated by human.\n\n2. New complimentary metrics (instruction following, event ordering, and contradiction resolution.) to evaluate the behaviour of LLM in long context setting\n\n3. Cognitive framework - LIGHT - shows convincing improvement in comparison to strong RAG baseline", "weaknesses": "1. Limited dataset size: only 100 conversations for benchmarking, this could lead to high variance in evaluation results.\n\n2. The paper doesn't deeply analyze why models fail on specific memory abilities. For instance, why do all methods struggle with contradiction resolution? No error analysis or qualitative examination of failure modes beyond the single case study in Appendix F.\n\n3. The combination of retrieval + working memory + external memory is not particularly novel. The main contribution is demonstrating this combination works better than RAG for long conversations. When retrieval is removed (Fig 3), it appears that under 1M context length where the models were trained, the retrieval module doesn’t benefit much but may harm the performance. \n\n4. Unfair claiming at 10M context length evaluation between LIGHT and Vanilla. Models anyways were not trained to perform at 10M context length. It expected that they perform much worse. \n\n5. It’s unclear whether the author will open-source the data and the framework code. Without these, it’s hard to enable the community to gauge the impact of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tEOL4qUU5w", "reviewer": "Reviewer_77HX", "summary": "The paper tackles the challenge of evaluating and improving long-term conversational memory in LLMs by introducing BEAM, a benchmark of 100 coherent, topically diverse dialogues (100K–10M tokens) paired with 2,000 validated probing questions spanning ten memory abilities. It also proposes LIGHT, a cognition-inspired inference framework that combines (i) episodic memory via retrieval over the full conversation, (ii) short-term working memory of recent turns, and (iii) a maintained scratchpad of salient facts filtered for relevance at query time. Experiments across multiple backbones show consistent gains of LIGHT over long-context and RAG baselines; ablations indicate each memory component contributes to performance.", "questions": "- Can you clarify which model you are using for the ablation study? Since the results on Table 8 do not match the results on Table1. \n- Why was BAAI/bge-small-en-v1.5 selected for the episodic index? Was the choice driven by speed/latency, availability, or prior retrieval effectiveness on dialogue passages? Any comparisons versus larger BGE variants, E5, or modern instruct-tuned embedding models like openai text embedding 001.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- BEAM spans multi-domain dialogues at four lengths (128K, 500K, 1M, 10M tokens) and evaluates 10 distinct memory abilities, filling gaps in prior single-domain, recall-heavy datasets.\n- Component-wise ablations substantiate that each memory module contributes meaningfully to performance.\n- Results across context scales (100K – 10M) and models show consistent improvements; the paper also includes component-wise ablations and sensitivity to retrieval budget K.", "weaknesses": "- LIGHT’s episodic memory relies on FAISS with BGE-small embeddings; the paper does not systematically explore robustness to alternative embedding models or indexing setups beyond a brief note.\n- While a human validation step is described for probes and a separate human evaluation of conversation quality is reported, rater pool size/protocols are not clearly specified, making it hard to assess annotation reliability and potential residual biases over very long chats.\n- LIGHT maintains an episodic index and a growing scratchpad over extremely long contexts. The paper varies K but does not report end-to-end latency/memory/GPU budgets for 1M–10M-token settings, limiting practical interpretability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Mzrte3yMF", "reviewer": "Reviewer_dWNe", "summary": "The paper proposes BEAM, a benchmark consisting of 100 synthetic conversations that span up to 10M tokens along with corresponding probing questions. BEAM aims to evaluate the abilities of large language models (LLMs) to reason over long-context input. Unlike existing benchmarks, it is more diverse, more realistic and covers multiple memory subtasks. Moreover, the paper introduces LIGHT, a framework that mimics episodic, working, and scratchpad memories to improve the long-context performance of LLMs. Existing LLMs do not perform well on BEAM. Augmenting LLMs with LIGHT improves their performance compared to standard RAG.", "questions": "- Could you clearly list all the human interventions during dataset creation?\n- Have you manually reviewed the generated conversations (not the probing questions or plans) ? Do you have metrics that asses the quality of these conversations? Do you observe hallucinations (not following the plans) in the generated conversations? \n- What are the main limitations that prevent the LLMs from performing well on this task (even using LIGHT)? What are the specific issues that LIGHT can or cannot address?\n- How does LIGHT compare to other recent long-context baselines other than RAG? This could be done only on the small models if too expensive to run.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- The benchmark contains more realistic conversations, is more diverse, covers many memory subtasks, and contains very long context scenarios. It is challenging even for the best existing LLMs. \n- The paper contains all the required details to understand and possibly replicate the dataset creation. The appendix provides even more details about the dataset creation. \n- The proposed method LIGHT outperforms standard RAG, which is a strong baseline.", "weaknesses": "- There is no qualitative analysis about the kind of errors that the LLMs make.  \n- Regarding readability, some sections of the paper are very dense and harder to read. The level of details describing the dataset creation and the propose method is adequate, but the flow of some paragraphs could be improved using for example more bullet points. \n- All the figures are tiny. They cannot be properly read without zooming. The captions of all figures are very short and do not clearly describe them. Figure 1, which describes dataset creation, and Figure 2, which explains the proposed method, should be highlighted more. The texts and notations used in those figures should be as self-explanatory as possible to allow a quick overview of the contributions.\n- The paper could present a few baselines besides RAG at least for the small models if it is too expensive. \n- Concerning related work, you might want to consider similar work https://arxiv.org/abs/2502.13791 that also relies on plans/templates to generate synthetic conversations in order to evaluate LLMs' ability to follow instructions in long-context. \n- Minor typo: the generated dialogues in the appendix contains typos (\"Trunicated\" in red) .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)